Modify topK and orderBy to track by key instead of by object reference

[kevin-dp]

This PR modifies the topK and orderBy operators. Previoiusly, the topK was computing a topK per key and it tracked multiplicities for each value associated to that key. We were always keying by null in ts/db and the values were entire rows, i.e. objects. However, the objects are always different objects (even for the same row) so they were always treated as a new value.

As reported by community members, this led to a bug where the results of an orderBy query with e.g. limit: 10 would end up with just a single item in the result. This would happen in the following case:

• initial query run, results in 10 rows (because of the limit)
• update to a row in that top 10: ts/db splits it into a delete of the old row and an insert of the new row
• topK operator sees the delete for the old row, but the object reference is different from the object reference of the old row it has seen before, so instead of finding the multiplicity for the old row (which is 1), it doesn't find it and thinks it hasn't seen it before so multiplicity is 0. It decrements it by 1, yielding -1. It concludes that since it was 0 before, this row was not visible and is still not visible. Nothing happens. The row is in fact never deleted.
• topK operator sees the insert for the new (updated) row. It stores this new row with multiplicity 1.
• Now the old row is in the top 10 and the new row is in the top 10. But ts/db materializes the query results by key so you will only see it once (with the new value) in the results. However, since internally it's twice in the top K it kicked out another row, so there are only 9 distinct rows in the query result. If you repeat this update 8 more times, you end up with a single row in the query results

To solve this problem i modified the topK operator such that it takes a keyed stream, i.e. a stream of tuples: [Key, Value]. It computes the topK across all keys (i.e. not per key as it did before) and uses the key to identify values. As such, when a row is received we lookup its multiplicity based on the key instead of the object reference.

[changeset-bot]

🦋 Changeset detected

Latest commit: beace86

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
@tanstack/db-ivm	Patch
@tanstack/db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

More templates

• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@405

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@405

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@405

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@405

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@405

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@405

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@405

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@405

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@405

commit: beace86

[github-actions]

Size Change: 0 B

Total Size: 58.5 kB

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/change-events.js	1.13 kB
./packages/db/dist/esm/collection.js	9.85 kB
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/errors.js	2.98 kB
./packages/db/dist/esm/index.js	1.51 kB
./packages/db/dist/esm/indexes/auto-index.js	689 B
./packages/db/dist/esm/indexes/base-index.js	605 B
./packages/db/dist/esm/indexes/btree-index.js	1.47 kB
./packages/db/dist/esm/indexes/lazy-index.js	1.25 kB
./packages/db/dist/esm/local-only.js	827 B
./packages/db/dist/esm/local-storage.js	2.03 kB
./packages/db/dist/esm/optimistic-action.js	294 B
./packages/db/dist/esm/proxy.js	4.19 kB
./packages/db/dist/esm/query/builder/functions.js	575 B
./packages/db/dist/esm/query/builder/index.js	3.79 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	890 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.48 kB
./packages/db/dist/esm/query/compiler/expressions.js	631 B
./packages/db/dist/esm/query/compiler/group-by.js	2.03 kB
./packages/db/dist/esm/query/compiler/index.js	1.74 kB
./packages/db/dist/esm/query/compiler/joins.js	1.56 kB
./packages/db/dist/esm/query/compiler/order-by.js	677 B
./packages/db/dist/esm/query/compiler/select.js	655 B
./packages/db/dist/esm/query/ir.js	318 B
./packages/db/dist/esm/query/live-query-collection.js	2.45 kB
./packages/db/dist/esm/query/optimizer.js	2.44 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/transactions.js	2.29 kB
./packages/db/dist/esm/utils.js	419 B
./packages/db/dist/esm/utils/btree.js	5.93 kB
./packages/db/dist/esm/utils/comparison.js	718 B
./packages/db/dist/esm/utils/index-optimization.js	1.62 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 1.05 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	152 B
./packages/react-db/dist/esm/useLiveQuery.js	902 B

_{compressed-size-action::react-db-package-size}

[samwillis]

There is a failed test in the react-db package - need to look at that.

[kevin-dp]

There is a failed test in the react-db package - need to look at that.

Seems to be a flake. CI is green now.

[samwillis]