fix(bigquery-firestore-export): write run output to deterministic ids

[IzaakGough]

writeRunResultsToFirestore wrote every BigQuery result row with collection.add(), minting a fresh random document id per row, and the run document is only written once all rows are in. processMessages deploys with retry: true, so a run that dies part way through is redelivered and writes the whole result set again under new ids, for as long as the subscription retains the message.

Measured on a live project before the change: republishing one completion notification for a finished 1,000 row run took its output collection from 1,000 to 2,000 documents, and a 3,000 row source caught in the memory-driven retry loop grew through 3,200, 3,400, 3,600, 6,600 and 6,800 documents while still climbing.

Rows now write with set() to an id derived from the row's index within the run, zero padded so ids sort naturally.

After the change the redelivered notification left the count at 1,000, and the 3,000 row run held at exactly 3,000 documents across nine retries. It still runs out of memory, which is tracked separately. 23 unit tests pass.

One thing to weigh: output document ids are now deterministic rather than random, so anything that recorded the old ids will see different ones.

[gemini-code-assist]

Code Review

This pull request updates the BigQuery to Firestore export helper to key output documents by their row index (zero-padded) to prevent duplicate writes on retries. However, the reviewer identifies a critical risk: BigQuery does not guarantee row ordering without an explicit ORDER BY clause, meaning retries could result in silent data corruption due to mismatched row-to-index mappings. The reviewer recommends generating deterministic document IDs by hashing the row content (e.g., using SHA-256) and provides corresponding updates for the helper implementation and test suite.

[gemini-code-assist]

Non-Deterministic Row Ordering Risk

Using the row's index as the document ID is unsafe because BigQuery does not guarantee the order of query results for SELECT * without an explicit ORDER BY clause.

On retries or redeliveries (which this PR aims to handle safely), BigQuery may return the rows in a different order. If the order changes:

1. The row index mapping will change.
2. A retry will overwrite existing documents with different row data.
3. This leads to silent data corruption (some rows will be duplicated under different index IDs, while other rows will be completely lost).

Recommended Solution

Instead of using the row index, generate a deterministic ID by hashing the row content (e.g., using SHA-256). This guarantees 100% idempotency and correctness regardless of the query result order.

Note: You will also need to import crypto at the top of the file: import * as crypto from "crypto";

Suggested change

	function outputDocumentId(index: number): string {
	return String(index).padStart(OUTPUT_DOC_ID_LENGTH, "0");
	}
	function outputDocumentId(row: unknown): string {
	const serialized = JSON.stringify(row, Object.keys(row as object).sort());
	return crypto.createHash("sha256").update(serialized).digest("hex");
	}

[gemini-code-assist]

Update the call site to pass the actual row object rows[j] to outputDocumentId instead of the index j to support deterministic hashing.

Suggested change

	writes.push(
	collection
	.doc(outputDocumentId(j))
	.set(convertUnsupportedDataTypes(rows[j]))
	);
	writes.push(
	collection
	.doc(outputDocumentId(rows[j]))
	.set(convertUnsupportedDataTypes(rows[j]))
	);

[gemini-code-assist]

Update the test assertions to expect the deterministic SHA-256 hashes of the rows instead of the zero-padded index IDs.

Suggested change

	expect([...output().keys()]).toEqual([
	"000000000000",
	"000000000001",
	"000000000002",
	]);
	expect(output().get("000000000001")).toEqual({ id: 1, label: "row-1" });
	const expectedKeys = rows(3).map(row => {
	const serialized = JSON.stringify(row, Object.keys(row).sort());
	return require("crypto").createHash("sha256").update(serialized).digest("hex");
	});
	expect([...output().keys()]).toEqual(expectedKeys);
	expect(output().get(expectedKeys[1])).toEqual({ id: 1, label: "row-1" });

[gemini-code-assist]

Update the rejected row test to dynamically compute the SHA-256 hash of the rejected row, ensuring the test remains correct and passes with the new hashing strategy.

  test("counts a rejected row without dropping the others", async () => {
    const testRows = rows(3);
    const serializedRow1 = JSON.stringify(testRows[1], Object.keys(testRows[1]).sort());
    const rejectedId = require("crypto").createHash("sha256").update(serializedRow1).digest("hex");
    const { db, docs, output } = makeDb([rejectedId]);

    await writeRunResultsToFirestore(
      { db, bigquery: makeBigquery(testRows), config: CONFIG },
      MESSAGE
    );

    const expectedKeys = [testRows[0], testRows[2]].map(row => {
      const serialized = JSON.stringify(row, Object.keys(row).sort());
      return require("crypto").createHash("sha256").update(serialized).digest("hex");
    });
    expect([...output().keys()]).toEqual(expectedKeys);