Skip to content

Feat: Transaction batch upload - #7

Draft
Chriztiaan wants to merge 1 commit into
feat/initfrom
feat/write-batches
Draft

Feat: Transaction batch upload#7
Chriztiaan wants to merge 1 commit into
feat/initfrom
feat/write-batches

Conversation

@Chriztiaan

Copy link
Copy Markdown
Collaborator

For the current DemoConnect, the client uploads one transaction per attempt so a queue that built up offline drains one
round-trip at a time. This PR adds POST /api/data/batch (which will eventually merge with /api/data - just separate for illustration), which takes an ordered run of whole transactions from the head of the upload queue and applies each one in its own database transaction.

Batching is opt-in. Leave VITE_BATCH_MAX_TRANSACTIONS unset and the connector keeps the existing
single-transaction path.

How it works

A transaction is never split across batches. The client fills a batch until it hits either
VITE_BATCH_MAX_TRANSACTIONS or VITE_BATCH_MAX_OPERATIONS (default 1000), whichever comes first,
and the backend applies them in order.

The response holds one result per transaction sent, in the same order and always the same length as
the request, so the client never has to infer which transactions were applied. Transactions the
batch never reached come back as not_attempted rather than being omitted.

The client then completes once, at the last result that is success or fatal_error. Completing a
transaction also completes every transaction before it, so a single complete() at that boundary is
enough, and a retry resumes from the failure instead of re-uploading transactions that already
committed.

on_fatal_error controls what a fatal failure does to the rest of the batch:

  • stop (default): the batch ends there; everything after it is reported not_attempted.
  • skip: the failing transaction is dropped and the batch carries on, so a queue blocked by a
    poison operation can still drain. Its result still reports fatal_error with the error
    classification, so the client can record what it discarded. Alternatively, there is room to divert these entries to a dead-letter queue instead of throwing them away.

skip covers fatal failures only. A retryable failure always ends the batch.

Error classification

For a batch to know where to stop, each failure has to be sorted into retryable vs fatal, so this PR
adds per-engine classification for Postgres, MySQL, MSSQL and MongoDB:

  • retryable: the environment misbehaved (connection loss, resource exhaustion). The client uploads the transaction again after a delay.
  • fatal: the data can never be stored (missing required field, constraint violation, malformed
    or out-of-range value, schema mismatch). Where the client discards the transaction.

AI disclosure

This PR was created with the help of Claude Code. Help constitutes assistance in research, planning,
and rough outline of implementation. Beyond having a hand in the implementation, I have also manually
tested this work.

updateBatch: async (batch: CrudEntry[]) => {
// TODO: Use batches & transactions.
// Transactions require a replica set or sharded cluster.
const session = client.startSession();

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took the liberty of changing the mongodb persister to use transactions, this means that users will need to run mongodb with a replica sets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant