feat(cli): fold substreams-sink-sql into the substreams CLI#830
Draft
GabrielCartier wants to merge 10 commits into
Draft
feat(cli): fold substreams-sink-sql into the substreams CLI#830GabrielCartier wants to merge 10 commits into
GabrielCartier wants to merge 10 commits into
Conversation
Ported from substreams-sink-sql as-is: historical proto packages (bare `schema`, `sf.substreams.sink.sql.service.v1`) kept for compat with published .spkg files, hence the buf lint ignores. Deprecated sf.substreams.sink.sql.v1 duplicate not ported; its type URL stays accepted at the sink-config layer.
Verbatim port from github.com/streamingfast/substreams-sink-sql under sink/sql/: db_changes (DatabaseChanges mode), db_proto (relational from-proto mode), bytes, proto, services, and the integration test harness. - imports rewritten to in-repo pb/sf/substreams/sink/sql protos; the dead vendored options.pb.go copy was dropped - integration tests (Docker) gated behind SF_SINK_SQL_INTEGRATION_TESTS, CI runners lack Docker - testcontainers bumped to v0.43: wait.ForSQL now takes network.Port instead of nat.Port in helpers_test.go
Fold the substreams-sink-sql binary into the substreams CLI:
- sink sql {run,setup,create-user,generate-csv,inject-csv,tools}
drives the DatabaseChanges mode
- sink from-proto drives the relational-mappings mode
Deliberate changes from the standalone binary:
- DSN is now --dsn (or SUBSTREAMS_SINK_SQL_DSN) instead of a
positional arg, consistent across every subcommand
- metrics/pprof listeners are opt-in flags, no hardcoded :6060
- pgx v4 -> v5 for the inject-csv COPY path
- 'tools cursor delete <hash>' deleted all cursors in the
standalone binary due to variable shadowing; now deletes only
the given hash as documented
- dbt run interval slept nanoseconds instead of seconds - reject dbt config zip entries escaping the extraction dir; the zip comes from untrusted .spkg files - drop dead err param from FileDescriptorForOutputType - port FROM_PROTO.md and point the clickhouse_table_options error at its real anchor (old link 404'd)
'sql' says nothing distinguishing: both sink modes target SQL databases. New name matches the mode's proto (DatabaseChanges) and the db_out module convention; 'db-changes' kept as alias. Review fixes bundled: - DSN env fallback renamed to SUBSTREAMS_SINK_DSN (shared with from-proto) - pprof/delay flags honored by every subcommand, after cheap validation so a bad DSN fails before the delay sleep - metrics via the standard --prometheus-addr sink flag; custom --metrics-listen-addr dropped - block range arg: shared helper, errors on non-range input instead of silently ignoring, supports :stop and start 0 - generate-csv: final-blocks-only forced in config, flag removed from surface instead of silently overridden - from-proto: dead endpoint-resolution block removed (extra manifest read), sink-service run only when a sink config exists - create-user retry via derr.RetryContext (ctx-aware) - load .substreams.env before streaming, like other sink commands - deprecated flag aliases from the standalone binary dropped (--on-module-hash-mistmatch typo, --flush-interval)
- SF_SINK_SQL_INTEGRATION_TESTS set on the ubuntu leg only; macos runners have no Docker - buf lint exemptions for the sink/sql protos narrowed from blanket ignore to the specific historical-naming rules
Examples still showed the standalone binary's positional DSN and a DSN env var the command never reads; copy-pasting them failed with 'no DSN provided'. Also update a stale 'sink sql' panic message.
Transitive dep of the sink/sql port (via wk8/go-ordered-map); Docker Scout blocks the image build on the HIGH out-of-bounds read fixed in 1.1.2.
'sink database-changes' and 'sink from-proto' become 'sink postgres' and 'sink clickhouse'; run auto-detects the mode from the output module type (DatabaseChanges proto -> schema.sql mode, anything else -> relational mappings). - engine-prefixed flags renamed: --cluster, --cursor-file-path, --sink-info-folder, --query-retry-count, --query-retry-sleep; clickhouse flags only exist on clickhouse commands - positional <start>:<stop> range dropped; -s/--start-block and -t/--stop-block like 'substreams run' (inject-csv keeps its positional file range) - DSN scheme validated against the command's engine - generate-csv and inject-csv are postgres-only
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
substreams-sink-sqlbinary now lives in the CLI:substreams sink postgres {run,setup,create-user,generate-csv,inject-csv,tools}substreams sink clickhouse {run,setup,create-user,tools}runauto-detects the mode from the output module type:sf.substreams.sink.database.v1.DatabaseChanges→ schema.sql mode (db_out), anything else → relational mappings (from-proto).Why
One CLI for sinks (follows noop/webhook/protojson precedent). Engine in the command name → per-engine flag surfaces, no mix-and-match.
Changes vs standalone binary
--dsnflag /SUBSTREAMS_SINK_DSNenv, scheme validated against the command's enginesubstreams run:<manifest> [<module>]+-s/--start-block-t/--stop-block; positional<start>:<stop>gone (inject-csv keeps its file range)--cluster,--cursor-file-path,--sink-info-folder,--query-retry-*; clickhouse flags only on clickhouse commands--prometheus-addr, custom--metrics-listen-addrgonetools cursor delete <hash>deleted ALL cursors (shadowing)--flush-interval,--on-module-hash-mistmatchtypo)Notes
SF_SINK_SQL_INTEGRATION_TESTS