diff --git a/RELEASE.md b/RELEASE.md index 01fc5b16..abe07755 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -3,15 +3,15 @@ This document outlines how to release a new version of Blip. It is not the [release notes](https://block.github.io/blip/about/release-notes). -Review Blip [versioning](https://github.com/cashapp/blip/blob/main/CONTRIBUTING.md#versioning) guidelines. +Review Blip [versioning](https://github.com/block/blip/blob/main/CONTRIBUTING.md#versioning) guidelines. ## 1. Release Branch First, create a branch to prepare the new version. - [ ] Ensure local main branch is up to date and clean: `git co main && git pull` -- [ ] Create release branch: `git co -b v1.Y.Z` (replace Y and Z with new version) -- [ ] Bump version const [`blip.VERSION`](https://github.com/cashapp/blip/blob/main/blip.go#L21) +- [ ] Create release branch: `git co -b vX.Y.Z` (replace X, Y, and Z with the new version) +- [ ] Bump version const [`blip.VERSION`](https://github.com/block/blip/blob/main/blip.go) ## 2. Documentation @@ -27,17 +27,17 @@ Run `docs/serve.sh` to edit locally. Third, merge the release branch and create a GitHub release. -- [ ] Add, commit, and merge changes in the release branch; commit message "Release v1.Y.Z" or similar -- [ ] [Wait for GitHub Actions](https://github.com/cashapp/blip/actions) to build/publish +- [ ] Add, commit, and merge changes in the release branch; commit message "Release vX.Y.Z" or similar +- [ ] [Wait for GitHub Actions](https://github.com/block/blip/actions) to build/publish - [ ] Update local main branch: `git co main && git pull` -- [ ] Tag local main with new version: `git tag v1.Y.Z` (replace Y and Z) +- [ ] Tag local main with new version: `git tag vX.Y.Z` (replace X, Y, and Z) - [ ] Push tag: `git push --tags` -- [ ] [Draft a new release](https://github.com/cashapp/blip/releases/new) +- [ ] [Draft a new release](https://github.com/block/blip/releases/new) - [ ] Select new version tag - [ ] Click "Generate release notes" button (on the right) - - [ ] Put "Human-readable release notes: ..." preamble before generated release notes (see [v1.0.1 release](https://github.com/cashapp/blip/releases/tag/v1.0.1) for example) + - [ ] Put "Human-readable release notes: ..." preamble before generated release notes (see [v1.0.1 release](https://github.com/block/blip/releases/tag/v1.0.1) for example) - [ ] Clean up generated releases that are noisy or useless - [ ] Make sure "Set as the latest release" is checked (leave pre-release unchecked/clear) - - [ ] Pubish release + - [ ] Publish release Congratulations and thank you for helping develop Blip and monitor MySQL! diff --git a/blip.go b/blip.go index 689e9fba..0e9cdc20 100644 --- a/blip.go +++ b/blip.go @@ -20,7 +20,7 @@ import ( "github.com/aws/aws-sdk-go-v2/aws" ) -const VERSION = "1.2.1" +const VERSION = "2.0.0" var SHA = "" diff --git a/docs/content/about/production-readiness.md b/docs/content/about/production-readiness.md index f7f09b16..42e7456a 100644 --- a/docs/content/about/production-readiness.md +++ b/docs/content/about/production-readiness.md @@ -17,16 +17,23 @@ Stable Feature readiness is documented here to help you make informed decisions about monitoring your databases with Blip. -## v1.x +## v2.x -Blip v1.x is production ready. +Blip v2.x is Stable. Existing production-ready components retain their readiness; collectors introduced in v2.0 are marked New. ### Metric Collectors |Domain|Readiness| |-------|------| +|[autoinc]({{< ref "metrics/domains/autoinc/" >}})|New| |[aws.rds]({{< ref "metrics/domains/aws.rds/" >}})|Production| +|[error.account]({{< ref "metrics/domains/error.account/" >}})|New| +|[error.global]({{< ref "metrics/domains/error.global/" >}})|New| +|[error.host]({{< ref "metrics/domains/error.host/" >}})|New| +|[error.thread]({{< ref "metrics/domains/error.thread/" >}})|New| +|[error.user]({{< ref "metrics/domains/error.user/" >}})|New| |[innodb]({{< ref "metrics/domains/innodb/" >}})|Production| +|[innodb.buffer-pool]({{< ref "metrics/domains/innodb.buffer-pool/" >}})|New| |[repl]({{< ref "metrics/domains/repl" >}})|Production| |[repl.lag]({{< ref "metrics/domains/repl.lag/" >}})|Production| |[size.binlog]({{< ref "metrics/domains/size.binlog/" >}})|Production| diff --git a/docs/content/about/release-notes.md b/docs/content/about/release-notes.md index 107607a2..dc59215a 100644 --- a/docs/content/about/release-notes.md +++ b/docs/content/about/release-notes.md @@ -2,6 +2,57 @@ weight: 0 --- +## v2.0 + +This is a new major version. It added new metric domains, runtime debug controls, extensible AWS Secrets Manager parsing, and a more resilient Datadog delivery path. + +As described in the [Blip versioning guidelines](https://github.com/block/blip/blob/main/CONTRIBUTING.md#versioning), this series is not entirely backwards-compatible with v1.2. Integrations that use the affected exported APIs must be updated before upgrading. + +### Integration API changes + +|Component|v1.2|v2.0| +|---------|----|----| +|AWS secret password helper|`aws.Secret.Password(context.Context) (string, error)`|Removed; use `GetSecret` or `GetSecretPayload`| +|Database credential type|`dbconn.Credentials`|`blip.DbCredentials`| +|Database credential callback|Returned `dbconn.Credentials`|Returns `blip.DbCredentials`| +|Default connection factory|`NewConnFactory(awsConfig, modifyDB)`|`NewConnFactory(awsConfig, modifyDB, ...ConnFactoryOption)`| +|Database-size query helper|Returned `(string, error)`|Returns `(string, []interface{}, error)`| +|Table-size query helper|Returned `(string, error)`|Returns `(string, []interface{}, error)`| +|Table-I/O query helper|Returned `string`|Returns `(string, []interface{})`| +|`heartbeat.BlipReader` values|Comparable|Not comparable| + +To upgrade an integration: + +1. Replace `dbconn.Credentials` with `blip.DbCredentials` and update any `dbconn.CredentialFunc` implementations. +2. Accept the new variadic options argument when storing or wrapping `dbconn.NewConnFactory`; ordinary two-argument calls continue to compile. +3. Capture the parameter slice returned by `DataSizeQuery`, `TableSizeQuery`, and `TableIoWaitQuery`, and pass it to the database query call. +4. Replace `aws.Secret.Password` calls with `GetSecret` for the default JSON object or `GetSecretPayload` plus a password secret parser for custom payloads. +5. Stop comparing `heartbeat.BlipReader` values directly; compare the relevant state exposed by the reader instead. + +### Runtime changes + +AWS Secrets Manager `password-secret` authentication now uses the secret's optional string `username` value instead of always using the configured monitor username. Remove `username` from the secret to retain the v1.2 behavior. + +Default sink HTTP clients now have a 10-second whole-request timeout plus bounded connection and response-header timeouts. Custom HTTP client factories are unchanged. + +### v2.0.0 (7 Aug 2026) + +* Added the `autoinc` domain for auto-increment column utilization. +* Added the `error.account`, `error.global`, `error.host`, `error.thread`, and `error.user` domains. +* Added the `innodb.buffer-pool` domain. +* Added runtime debug toggling through `GET /debug` and `SIGUSR1`. +* Added customizable parsing of AWS Secrets Manager `SecretString` and `SecretBinary` payloads. +* Added bounded and checkpointed Datadog payload submission to avoid oversized requests and resume partially acknowledged batches. +* Prevented sink requests from blocking metric delivery indefinitely. +* Preserved complete metric metadata and isolated counter state in the delta sink. +* Redacted database, sink, and authenticated proxy credentials from debug logs. +* Fixed query construction to use driver parameter interpolation across collectors and heartbeats. +* Fixed nondeterministic `query.response-time` bucket selection and added detailed latency diagnostics. +* Fixed a shutdown panic in signal handling. +* Updated the test matrix from MySQL 5.7 to MySQL 8.4 and refreshed dependencies. + +--- + ## v1.2 This is a new series (new minor version). diff --git a/docs/content/metrics/domains/autoinc/_index.md b/docs/content/metrics/domains/autoinc/_index.md new file mode 100644 index 00000000..4adf932e --- /dev/null +++ b/docs/content/metrics/domains/autoinc/_index.md @@ -0,0 +1,71 @@ +--- +title: "autoinc" +--- + +The `autoinc` domain reports how much of each auto-increment column's numeric range has been used. + +{{< toc >}} + +## Usage + +The collector reads auto-increment columns from `information_schema.COLUMNS` and `information_schema.TABLES`. It reports a `usage` value from 0 to 1 by dividing the table's next `AUTO_INCREMENT` value by the maximum value of the column's signed or unsigned integer type. + +Collect this domain infrequently because auto-increment utilization usually changes slowly. Alert before `usage` reaches 1 so the column can be widened or converted to an unsigned type. + +## Derived Metrics + +### `usage` + +| | | +|---|---| +|**Metric Type**|gauge| +|**Value Units**|ratio| + +The fraction of the auto-increment range used. + +## Options + +### `exclude` + +| | | +|---|---| +|**Value Type**|CSV string of db.table| +|**Default**|`mysql.*,information_schema.*,performance_schema.*,sys.*`| + +A comma-separated list of database or table names to exclude. This option is ignored when `include` is set. + +### `include` + +| | | +|---|---| +|**Value Type**|CSV string of db.table| +|**Default**|| + +A comma-separated list of database or table names to include. This option overrides `exclude`. + +## Group Keys + +|Key|Value| +|---|---| +|`db`|Database name| +|`tbl`|Table name| +|`col`|Column name| +|`data_type`|Signed or unsigned integer type| + +## Meta + +None. + +## Error Policies + +None. + +## MySQL Config + +None. + +## Changelog + +|Blip Version|Change| +|------------|------| +|v2.0.0|Domain added| diff --git a/docs/content/metrics/quick-ref.md b/docs/content/metrics/quick-ref.md index 4a480506..6fb0cc2a 100644 --- a/docs/content/metrics/quick-ref.md +++ b/docs/content/metrics/quick-ref.md @@ -12,18 +12,18 @@ The rest are reserved for future use. |access.index|Index access statistics (`sys.schema_index_statistics`)|| |access.table|Table access statistics (`sys.schema_table_statistics`)|| |aria|MariaDB Aria storage engine|| -|autoinc|Auto-increment column limits|| +|[`autoinc`](domains#autoinc)|Auto-increment column utilization|v2.0.0| |aws|Amazon Web Services|| |[`aws.rds`](domains#awsrds)|[Amazon RDS metrics](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/monitoring-cloudwatch.html#rds-metrics)|v1.0.0| |aws.aurora|Amazon Aurora|| |azure|Microsoft Azure|| |error|MySQL, client, and query errors|| |error.client|Client errors|| -|[error.account](domains#error.account)|Error counts and rates by account [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|TBD| -|[error.global](domains#error.global)|Global error counts and rates [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|TBD| -|[error.host](domains#error.host)|Error counts and rates by host [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|TBD| -|[error.thread](domains#error.thread)|Error counts and rates by thread [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|TBD| -|[error.user](domains#error.user)|Error counts and rates by user [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|TBD| +|[error.account](domains#error.account)|Error counts and rates by account [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|v2.0.0| +|[error.global](domains#error.global)|Global error counts and rates [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|v2.0.0| +|[error.host](domains#error.host)|Error counts and rates by host [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|v2.0.0| +|[error.thread](domains#error.thread)|Error counts and rates by thread [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|v2.0.0| +|[error.user](domains#error.user)|Error counts and rates by user [`Error Summary Tables`](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-error-summary-tables.html)|v2.0.0| |error.query|Query errors|| |error.repl|Replication errors|| |event|[MySQL Event Scheduler](https://dev.mysql.com/doc/refman/8.0/en/event-scheduler.html)|| @@ -33,7 +33,7 @@ The rest are reserved for future use. |gr|MySQL Group Replication|| |host|Host (client)|| |[`innodb`](domains#innodb)|InnoDB metrics [`INFORMATION_SCHEMA.INNODB_METRICS`](https://dev.mysql.com/doc/refman/en/information-schema-innodb-metrics-table.html)|v1.0.0| -|[`innodb.buffer-pool`](domains#innodbbuffer-pool)|InnoDB buffer pool metrics [`INFORMATION_SCHEMA.INNODB_BUFFER_POOL_STAT`](https://dev.mysql.com/doc/refman/8.4/en/information-schema-innodb-buffer-pool-stats-table.html)|TBD| +|[`innodb.buffer-pool`](domains#innodbbuffer-pool)|InnoDB buffer pool metrics [`INFORMATION_SCHEMA.INNODB_BUFFER_POOL_STATS`](https://dev.mysql.com/doc/refman/8.4/en/information-schema-innodb-buffer-pool-stats-table.html)|v2.0.0| |innodb.mutex|InnoDB mutexes `SHOW ENGINE INNODB MUTEX`|| |mariadb|MariaDB enhancements|| |ndb|MySQL NDB Cluster||