Skip to content

feat(sinks): validate all sinks at config-compile time - #26048

Open
thomasqueirozb wants to merge 40 commits into
masterfrom
sinks-validated-all
Open

feat(sinks): validate all sinks at config-compile time#26048
thomasqueirozb wants to merge 40 commits into
masterfrom
sinks-validated-all

Conversation

@thomasqueirozb

Copy link
Copy Markdown
Member

Summary

Extends the validated sink lifecycle to all sinks and makes DynValidatedSink a supertrait of SinkConfig, removing the per-sink as_dyn_validated boilerplate. This also fixes vector validate --no-environment to catch sink configuration errors (including template path-confinement violations) at config-compile time.

Vector configuration

NA

How did you test this PR?

make check-clippy and cargo test -p vector --no-run --features sources-demo_logs.

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

@thomasqueirozb

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

(Some(connstr), None, None) => connstr.inner().into(),

P2 Badge Parse Azure Blob connection strings during validation

For azure_blob configurations that provide a malformed connection_string, this branch just copies the secret into validated state, while ParsedConnectionString::parse and container URL parsing still run later in build_client. Because vector validate --no-environment skips that build path, these pure configuration errors continue to pass compile-time validation; parse and retain the validated connection/container URL state here, deferring only credential construction to build.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/http/config.rs
Comment thread src/sinks/file/mod.rs
Comment thread src/sinks/doris/config.rs Outdated
Comment thread src/sinks/datadog/events/config.rs Outdated
Comment thread src/sinks/elasticsearch/config.rs
Comment thread src/sinks/axiom/config.rs Outdated
Comment thread src/sinks/gcp/cloud_storage.rs Outdated
Comment thread src/sinks/gcp/pubsub.rs Outdated
@github-actions github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Aug 6, 2026
@thomasqueirozb

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0eddcfa6ff

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/influxdb/logs.rs
Comment thread src/sinks/databricks_zerobus/config.rs Outdated
Comment thread src/sinks/aws_kinesis/firehose/config.rs
Comment thread src/sinks/console/config.rs Outdated
Comment thread src/sinks/sematext/logs.rs
@thomasqueirozb
thomasqueirozb marked this pull request as ready for review August 10, 2026 20:18
@thomasqueirozb
thomasqueirozb requested a review from a team as a code owner August 10, 2026 20:18
@thomasqueirozb

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f7215b709c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/azure_monitor_logs/config.rs Outdated
Comment thread src/sinks/redis/config.rs Outdated
Comment thread src/sinks/amqp/config.rs
Comment thread src/sinks/mqtt/config.rs
Comment thread src/sinks/nats/config.rs Outdated
Comment thread src/sinks/socket.rs Outdated
Comment thread src/sinks/websocket/config.rs Outdated
@github-actions github-actions Bot added the domain: sources Anything related to the Vector's sources label Aug 10, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 346586c9cc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/mezmo.rs
Comment thread src/sinks/kafka/config.rs
Comment thread src/sinks/vector/config.rs
Comment thread src/sinks/postgres/config.rs
Comment thread src/sinks/pulsar/config.rs
Comment thread src/sinks/greptimedb/metrics/config.rs
Comment thread src/sinks/azure_blob/config.rs
Comment thread src/sinks/nats/config.rs
Comment thread src/sinks/prometheus/remote_write/config.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c93366efea

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/gcp_chronicle/chronicle_unstructured.rs Outdated
Comment thread src/sinks/aws_s_s/sns/config.rs Outdated
Comment thread src/sinks/http/config.rs
Comment thread src/sinks/socket.rs Outdated
Comment thread src/sinks/elasticsearch/config.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b8ca129e49

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/datadog/logs/config.rs
Comment thread src/sinks/splunk_hec/logs/config.rs
Comment thread src/sinks/greptimedb/logs/config.rs
Comment thread src/sinks/gcp_chronicle/chronicle_unstructured.rs
Comment thread src/sinks/new_relic/config.rs
Comment thread src/sinks/doris/config.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1697d97b20

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

let (sink, healthcheck) = ElasticsearchConfig {
endpoints: vec![endpoint],
let index =
Template::try_from(self.token.inner()).expect("unable to parse token as Template");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Return errors for malformed Sematext tokens

When sematext_logs.token contains invalid template syntax such as foo{{, this expect now runs during sink validation and panics the config compile/vector validate --no-environment path instead of reporting a validation error. Since Template::try_from already returns a Result, propagate that error before building the derived Elasticsearch config so bad user input cannot abort validation.

Useful? React with 👍 / 👎.

// `ElasticsearchCommon::parse_many`).
self.common_mode()?;

let request_limits = self.request.tower.into_settings();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate OpenSearch Serverless prerequisites

When elasticsearch uses opensearch_service_type = "serverless" without AWS auth/region, or with a non-auto api_version, this validation still retains success after only endpoint selection and routing-mode checks. ElasticsearchCommon::parse_config rejects those combinations before API-version probing or network I/O, but vector validate --no-environment now skips that build path; mirror the pure Serverless auth/region/API-version checks here.

Useful? React with 👍 / 👎.

@@ -198,12 +236,41 @@ impl SinkConfig for RemoteWriteConfig {
.transpose()?;

let endpoint = self.endpoint.parse::<Uri>().context(UriParseSnafu)?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require absolute remote-write endpoints

When prometheus_remote_write.endpoint is a relative URI such as /api/v1/write or lacks a scheme/host, http::Uri still parses successfully here, so validation retains an endpoint that the healthcheck/request path later hands to the HTTP client as a non-absolute target. Since the sink docs require the endpoint to include the scheme and write path, check the parsed URI has a scheme and authority during validation instead of accepting a config that deterministically fails when built or healthchecked.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vector validate --no-environment skips template confinement check

1 participant