Add two more PostgreSQL systems to test: native and no RPC

[hogejo]

Description of Changes

Following your announcement / keynote (https://www.youtube.com/watch?v=C7gJ_UxVnSk), this PR is adding two other PostgreSQL systems to benchmark:

• PostgreSQL benchmarked with a native client (Go) - like you do with SpacetimeDB
• PostgreSQL benchmarked without an RPC server

API and ABI breaking changes

Nothing. This just adds some goodies to your keynote-2 template.

Expected complexity level and risk

Complexity: 2 - some polishing and merge-prep might required

Testing

══════════════════════════════════════════════════════════════════════
  RESULTS
══════════════════════════════════════════════════════════════════════

  postgres       ████████████████████████████████████████    156,034 TPS
  spacetimedb    ███████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     42,107 TPS
  postgres_no_rpc ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     10,796 TPS

• I ran this on my own machine.
• Maybe run this on your infra?

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[FrancescoSaverioZuppichini]

lol grabbing pop corns

[mohammedzeglam-pg]

😂😂😂😂😂😂😂😂 waiting

[misandrie]

Who's gonna tell em

[pykeras]

This was obvious in the video. Every Backend engineers should notice that.
From where the benchmark starts (minute 10 onward) based on the chart legend it was clear that the testing conditions are not equivalent (add more components to maximize the latency, nice try).For others use the full stack (clinet, server, db ) was used with typescript, but for their benchmark they only used (client, db). That alone can skew the results (it's obvious) on top of that they used Rust + optimization flags (Rust 2021 edition) for their client.

Nope, that looks more like number engineering than a neutral benchmark. I'd like to participate in this discussion and see how the Clockwork Labs team responds to these concerns.

Description of Changes

Following your announcement / keynote (https://www.youtube.com/watch?v=C7gJ_UxVnSk), this PR is adding two other PostgreSQL systems to benchmark:

* PostgreSQL benchmarked with a native client (Go) - like you do with SpacetimeDB

* PostgreSQL benchmarked without an RPC server

API and ABI breaking changes

Nothing. This just adds some goodies to your keynote-2 template.

Expected complexity level and risk

Complexity: 2 - some polishing and merge-prep might required

Testing

══════════════════════════════════════════════════════════════════════
  RESULTS
══════════════════════════════════════════════════════════════════════

  postgres       ████████████████████████████████████████    156,034 TPS
  spacetimedb    ███████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     42,107 TPS
  postgres_no_rpc ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     10,796 TPS

* [x]  I ran this on my own machine.

* [ ]  Maybe run this on your infra?

[Ayoree]

It's funny how they ran into a bottleneck problem and decided to write a Rust client only for their own database and not for others. And then they even compared the results.

[zessu]

🍿🍿🍿🍿🍿

[snewell92]

edit - o i am an idiot - this is not a separate bench repo. Can we make it a separate bench repo? xD

Can the CLA be removed from this bench repo? If you want it open and to be a long term source of not only competitor data, but also spacetime's own journey of performance over time, removing all barriers to contribution could help a lot.

[cloutiertyler]

The purpose of our benchmark was to measure the systems as they would actually be deployed. This claims to be a more “apples to apples” comparison, but in reality every deployment of Postgres puts an RPC server in front of it. The point of the benchmark is that SpacetimeDB fills both roles for you by default.

This ends up being more apples to oranges than the original benchmark, because it compares a full backend to half a backend.

Love the deep dive though! We’ll be posting a response!

[cloutiertyler]

Can the CLA be removed from this bench repo? If you want it open and to be a long term source of not only competitor data, but also spacetime's own journey of performance over time, removing all barriers to contribution could help a lot.

I would love to, but I don't think the tool has the ability to do this. Maybe we should create a separate benchmark repo?

[cloutiertyler]

Also I might be missing it somehow, but I'm not seeing the actual Postgres stored procedure in the PR. Can you please upload that as well?

[CalmProton]

Cool, is there anything you can do to make Convex not look as bad? 😅

[ddwalias]

The benchmark in the video is not apple-to-apple comparison, and this PR is not either.

SpacetimeDB fundamentally operates as both a database and an application server, whereas databases like Postgres require a separate server in between.

However, comparing a Rust-based SpacetimeDB client using WebSockets against JavaScript-based servers utilizing HTTP for the other databases introduces transport and language-level bias, which is completely unfair.

In my opinion, to achieve a true apples-to-apples comparison, the entire benchmark suite should be completely rewritten. The application logic of both SpacetimeDB reducer and the intermediary server for Postgres need to be implemented identically to eliminate language bias. Furthermore, the benchmark must utilize the exact same transport mechanism across all evaluated architectures.

Don't get me wrong, I love the marketing and the tech behind SpacetimeDB. However, I don't think SpacetimeDB need to engineer a biased benchmark to win over the traditional server + db.

EDIT: I just realized that the SpacetimeDB in the benchmark actually uses TS for its reducer. However, my point still stands about the transport. HTTP initialization is more expensive than a persistent WebSocket.

[BRAVO68WEB]

We need to talk about this! This won't age well

[Bechma]

Also I might be missing it somehow, but I'm not seeing the actual Postgres stored procedure in the PR. Can you please upload that as well?

https://github.com/clockworklabs/SpacetimeDB/pull/4457/changes#diff-4dd84f03bdf19d7b2a1fa30f43ca7a0ecf5fbe23022e85b97a300b2ef004a31cR69-R121
That is the stored procedure he's using
https://github.com/clockworklabs/SpacetimeDB/pull/4457/changes#diff-45b0611a7f75380aad8f47b3ed5cf6c38559db2d39102481094c8653ca9569e4R133-R144
Here's the communication

I have the same opinion that ddwalias about this topic btw

[hogejo]

Let me keep the conversation within the bounds of this PR:

• I am not going to sign the CLA
• I am moving my efforts to https://github.com/hogejo/spacetimebenchmark

Regarding the comments:

• No offence taken or given <3
• You can find me at the SpacetimeDB community Discord

The benchmark in the video is not apple-to-apple comparison, and this PR is not either.

100% agree. Anything after this, we can discuss as part of a collaboration (chat, writing code, sending PRs, etc.).

I am closing this PR and hoping it will still be available so people can find my code.
If you want to use it in any way and can't sleep without a licence: I am happy to move it somewhere under GPL.