Misleading results

[alexflav23]

Hi Yleun,

I'm updating your implementation of the benchmark, but a few initial suggestions that I will make:

• You are measuring total HTTP response time that includes Play framework latency, which isn't very helpful if people want to side by side compare the drivers. Most of the Play framework errors I saw were failing to parse dates, I am currently dealing with that.

• Methods are being twice in the phantom repository, I really don't understand why, but it's naturally giving everything higher latency. For example:

 def delete(groupId: UUID, id: UUID): Future[Boolean] = {
    for {
      del <- database.groupIds.deleteById(groupId, id)
      del2 <- database.groupIds.deleteById(groupId, id).map(_.wasApplied())
    } yield del2
  }

  def save(gi: GroupId, isNew: Boolean): Future[Boolean] = {
    database
      .groupIds
      .save(gi)
      .flatMap (rs => database.groupIds.save(gi).map(_.wasApplied))
  }

• Quill uses prepared statements, in phantom you've used non-prepared.

• More interestingly, you have an invalid query, because based on the phantom schema you defined both id and groupId as partition keys, which means selecting only by groupId is invalid, so I can only assume you've computed your tests against different keyspaces with a different schema. Your timeouts aren't threading performance, it's because half your queries are invalid...

• There are different implementations of the models used for the two libraries, presumably because quill cannot cope with DateTime I guess.

• The implementation is quite bloated with unnecessary things like Future[Either], Future already pre-wraps a Try with that can be an Exception, so this is just adding CPU cycles.

In short, I'm not sure this benchmark here in its current form is very relevant just yet. I saw you posted it on StackOverflow and while I appreciate you may favour other libs, it's entirely possible you're misleading the audience without meaning to.