diff --git a/openframe/docs/README.md b/openframe/docs/README.md index 74acf3327df..2d841b9a409 100644 --- a/openframe/docs/README.md +++ b/openframe/docs/README.md @@ -50,6 +50,7 @@ The agent has its own switch, `--openframe-mode` / `ORBIT_OPENFRAME_MODE`. | [agent-openframe-mode.md](agent-openframe-mode.md) | OpenFrame agent mode: gateway URL prefix, encrypted bearer-token pipeline (extract / decrypt / refresh), custom osqueryd, `orbit uuid` command. | | [node-key-management.md](node-key-management.md) | Node-key enrollment caching, 401 re-enrollment, Windows file-lock resilience. | | [agent-json-content-type.md](agent-json-content-type.md) | Orbit sets `Content-Type: application/json` on requests with a body (upstream sets none). Without it a WAF cannot JSON-parse the body — Cloud Armor flagged 100% of `/orbit/config` polls as SQLi. Unconditional, not gated on OpenFrame mode. | +| [agent-inventory-waf-shape.md](agent-inventory-waf-shape.md) | The `certificates_darwin`/`certificates_windows` detail queries hex-encode their distinguished-name columns; the ingest decodes them. Raw X.509 DNs are `/`+`=` dense and trip CRS 942431/942432 on every inventory write. Server-side only — no agent upgrade. | ## Database migrations diff --git a/openframe/docs/agent-inventory-waf-shape.md b/openframe/docs/agent-inventory-waf-shape.md new file mode 100644 index 00000000000..387d0d6394e --- /dev/null +++ b/openframe/docs/agent-inventory-waf-shape.md @@ -0,0 +1,76 @@ +# Certificate inventory wire shape + +**Slug:** `OPENFRAME(waf-inventory-shape)` · **File:** [`server/service/osquery_utils/queries.go`](../../server/service/osquery_utils/queries.go) + +The `certificates_darwin` and `certificates_windows` detail queries hex-encode their three +distinguished-name columns (`common_name`, `subject`, `issuer`), aliased with a `_hex` suffix. +`decodeCertificateDNColumns` decodes them at ingest, before anything else reads the row. + +## Why + +An X.509 distinguished name looks like `/C=US/ST=California/O=Acme, Inc./CN=…`. The CRS +`942431`/`942432` signatures are *restricted special-character counters* — they fire when an +argument value exceeds 6 (resp. 2) characters from a punctuation class that includes `/` and `=`. +Every DN exceeds that, so every certificate row in every inventory write matched. + +Measured in `shared-j62b` (dev), 24 h to 2026-08-07: **453** Cloud Armor preview DENYs on +`/tools/agent/fleetmdm-server/api/v1/osquery/*`, all `client-policy`, all UA `osquery/5.9.1`. +`certificates_darwin.*.subject` / `.common_name` was **237 of them (52.3 %)** — the single largest +source, and the only large one that is fixable in code: + +| Source | Events | % | +|---|---:|--:| +| `certificates_darwin` DN columns | 237 | 52.3 % | +| `data.N.hostIdentifier` | 65 | 14.3 % | +| `scheduled_query_stats.*.query` | 59 | 13.0 % | +| `software_windows.*.version` | 49 | 10.8 % | +| `orbit_info.*.last_recorded_error` | 21 | 4.6 % | +| `host_details.*`, `fleet_distributed_query_*` | 22 | 4.9 % | + +Sample matched values: `=com.apple.kerbe`, `=US/ST=Californi`, `local (`. + +Hex specifically, not base64: `[0-9A-F]` contains no punctuation at all. Base64 std (`+/=`) and +base64url (`-_`) both land back inside the restricted class. + +## Rollout + +Detail-query SQL is generated server-side and handed to the agent in the `distributed/read` +response (`detailQueriesForHost`, [`server/service/osquery.go`](../../server/service/osquery.go)). +Both halves of this change live in the server binary, so **the census clears on server deploy — +there is no agent upgrade wait**, unlike [agent-json-content-type.md](agent-json-content-type.md). + +During the upgrade window a host may post results computed from the previous query. Those rows +carry the plain columns and no `_hex` key, so `decodeCertificateDNColumns` skips the hex step and +runs only the `\xHH` unescape — identical to pre-change behaviour. Nothing is dropped. + +## Not covered + +This removes 52 % of the agent-ingest false positives, not all of them. Hardware UUIDs, third-party +version strings and live query results are arbitrary by nature; the only way to change their shape +would be to re-encode the osquery↔Fleet wire format on both ends. `scheduled_query_stats` has a +supported off switch instead — `FLEET_APP_ENABLE_SCHEDULED_QUERY_STATS=false` — so it is not +patched here. + +Enforcing `942431`/`942432` on the agent path therefore still needs a path-scoped rule band in +Cloud Armor. See `openframe-saas-tf` `openframe-saas/*/services/03-shared/armor.tf`. + +## Verifying + +```bash +gcloud logging read \ + 'resource.type="http_load_balancer" AND + jsonPayload.previewSecurityPolicy.outcome="DENY" AND + httpRequest.requestUrl:"/api/v1/osquery/"' \ + --project=shared-j62b --format=json --limit=5000 +``` + +Group by `previewSecurityPolicy.matchedFieldName`; the `certificates_darwin.*` rows should reach +zero within one detail-query refetch interval of the deploy. + +## Before shipping + +Run the modified `SELECT` under `osqueryi` on a real Mac and a real Windows host and confirm +`hex(subject)` round-trips through `hex.DecodeString` + `DecodeHexEscapes` to the same string the +previous query produced. Whether osquery's `\xHH` escaping of non-ASCII sits inside the value that +`hex()` sees decides the decode order, and that is worth confirming against a live host rather than +a fixture. `TestDirectIngestHostCertificatesHexEncodedDN` covers the Go side. diff --git a/openframe/docs/fork-file-manifest.md b/openframe/docs/fork-file-manifest.md index d5a590a6add..5ff58fb1c07 100644 --- a/openframe/docs/fork-file-manifest.md +++ b/openframe/docs/fork-file-manifest.md @@ -273,6 +273,7 @@ server/service/global_policies.go server/service/handler.go server/service/labels_util.go server/service/orbit_client.go +server/service/osquery_utils/queries.go server/service/queries.go server/vulnerabilities/nvd/cpe.go ``` diff --git a/server/service/osquery_utils/queries.go b/server/service/osquery_utils/queries.go index bcd692b2eac..0d48ac42f53 100644 --- a/server/service/osquery_utils/queries.go +++ b/server/service/osquery_utils/queries.go @@ -836,9 +836,12 @@ var extraDetailQueries = map[string]DetailQuery{ DirectIngestFunc: directIngestDiskEncryptionWindows, }, "certificates_darwin": { + // >>> OPENFRAME(waf-inventory-shape): hex-encode the DN columns — a raw X.509 DN is `/` + // and `=` dense, which the edge WAF scores as SQLi (CRS 942431/942432) on every inventory + // write. Decoded in decodeCertificateDNColumns. openframe/docs/agent-inventory-waf-shape.md Query: ` SELECT - ca, common_name, subject, issuer, + ca, hex(common_name) AS common_name_hex, hex(subject) AS subject_hex, hex(issuer) AS issuer_hex, key_algorithm, key_strength, key_usage, signing_algorithm, not_valid_after, not_valid_before, serial, sha1, "system" as source, @@ -849,7 +852,7 @@ var extraDetailQueries = map[string]DetailQuery{ path = '/Library/Keychains/System.keychain' UNION SELECT - ca, common_name, subject, issuer, + ca, hex(common_name) AS common_name_hex, hex(subject) AS subject_hex, hex(issuer) AS issuer_hex, key_algorithm, key_strength, key_usage, signing_algorithm, not_valid_after, not_valid_before, serial, sha1, "user" as source, @@ -858,13 +861,16 @@ var extraDetailQueries = map[string]DetailQuery{ certificates WHERE path LIKE '/Users/%/Library/Keychains/login.keychain-db';`, + // <<< OPENFRAME(waf-inventory-shape) Platforms: []string{"darwin"}, DirectIngestFunc: directIngestHostCertificatesDarwin, }, "certificates_windows": { + // >>> OPENFRAME(waf-inventory-shape): same treatment as certificates_darwin above — + // openframe/docs/agent-inventory-waf-shape.md Query: ` SELECT - ca, common_name, subject, issuer, + ca, hex(common_name) AS common_name_hex, hex(subject) AS subject_hex, hex(issuer) AS issuer_hex, key_algorithm, key_strength, key_usage, signing_algorithm, not_valid_after, not_valid_before, serial, sha1, username, @@ -873,6 +879,7 @@ var extraDetailQueries = map[string]DetailQuery{ certificates WHERE store = 'Personal';`, + // <<< OPENFRAME(waf-inventory-shape) Platforms: []string{"windows"}, DirectIngestFunc: directIngestHostCertificatesWindows, }, @@ -3496,6 +3503,36 @@ func GetDetailQueries( var rxExtractUsernameFromHostCertPath = regexp.MustCompile(`^/Users/([^/]+)/Library/Keychains/login\.keychain\-db$`) +// >>> OPENFRAME(waf-inventory-shape): the certificates detail queries hex-encode their DN columns +// so raw X.509 punctuation never reaches the edge WAF (CRS 942431/942432 count `/` and `=` as SQL +// special characters); this decodes them back — openframe/docs/agent-inventory-waf-shape.md +var certificateDNColumns = []string{"common_name", "subject", "issuer"} + +// decodeCertificateDNColumns normalizes the DN columns of one certificates row in place, so the +// rest of the ingest sees the plain values it always has. A row missing `_hex` came from a +// host still running the pre-encoding query handed out before this server started; its plain +// columns are already in place, so only the \xHH unescape runs. +func decodeCertificateDNColumns(ctx context.Context, logger *slog.Logger, row map[string]string) { + for _, col := range certificateDNColumns { + if encoded, ok := row[col+"_hex"]; ok { + decoded, err := hex.DecodeString(encoded) + if err != nil { + // Empty beats ingesting a hex string as if it were a DN. + logger.ErrorContext(ctx, "decoding hex certificate column", "component", "service", + "method", "directIngestHostCertificates", "column", col, "err", err) + row[col] = "" + } else { + row[col] = string(decoded) + } + } + + // Unescape \xHH sequences for non-ASCII characters (e.g. Cyrillic) in the DN. + row[col] = fleet.DecodeHexEscapes(row[col]) + } +} + +// <<< OPENFRAME(waf-inventory-shape) + func directIngestHostCertificatesDarwin( ctx context.Context, logger *slog.Logger, @@ -3511,11 +3548,10 @@ func directIngestHostCertificatesDarwin( certs := make([]*fleet.HostCertificateRecord, 0, len(rows)) for _, row := range rows { - // Unescape \xHH sequences in fields that may contain non-ASCII - // characters (e.g. Cyrillic) in the certificate's distinguished name. - row["common_name"] = fleet.DecodeHexEscapes(row["common_name"]) - row["subject"] = fleet.DecodeHexEscapes(row["subject"]) - row["issuer"] = fleet.DecodeHexEscapes(row["issuer"]) + // >>> OPENFRAME(waf-inventory-shape): the DN columns arrive hex-encoded from the detail + // query and must be decoded before use — openframe/docs/agent-inventory-waf-shape.md + decodeCertificateDNColumns(ctx, logger, row) + // <<< OPENFRAME(waf-inventory-shape) csum, err := hex.DecodeString(row["sha1"]) if err != nil { @@ -3604,11 +3640,10 @@ func directIngestHostCertificatesWindows( // SHA1 sum + username existsSha1User := make(map[string]bool, len(rows)) for _, row := range rows { - // Unescape \xHH sequences in fields that may contain non-ASCII - // characters (e.g. Cyrillic) in the certificate's distinguished name. - row["common_name"] = fleet.DecodeHexEscapes(row["common_name"]) - row["subject"] = fleet.DecodeHexEscapes(row["subject"]) - row["issuer"] = fleet.DecodeHexEscapes(row["issuer"]) + // >>> OPENFRAME(waf-inventory-shape): the DN columns arrive hex-encoded from the detail + // query and must be decoded before use — openframe/docs/agent-inventory-waf-shape.md + decodeCertificateDNColumns(ctx, logger, row) + // <<< OPENFRAME(waf-inventory-shape) csum, err := hex.DecodeString(row["sha1"]) if err != nil { diff --git a/server/service/osquery_utils/queries_test.go b/server/service/osquery_utils/queries_test.go index 04057ce4153..29e19df73f0 100644 --- a/server/service/osquery_utils/queries_test.go +++ b/server/service/osquery_utils/queries_test.go @@ -2944,6 +2944,133 @@ func TestDirectIngestHostCertificatesDarwinHexEscapes(t *testing.T) { require.True(t, ds.UpdateHostCertificatesFuncInvoked) } +// >>> OPENFRAME(waf-inventory-shape): covers the hex-encoded DN columns the certificates detail +// queries emit, the pre-encoding fallback during a server upgrade, and malformed input — +// openframe/docs/agent-inventory-waf-shape.md +func TestDirectIngestHostCertificatesHexEncodedDN(t *testing.T) { + const ( + commonName = "Ловушка" + subject = `/C=US/ST=California/O=Acme, Inc./CN=Ловушка` + issuer = `/C=US/O=Acme CA/CN=Acme Root` + ) + + // osquery escapes non-ASCII as literal \xHH before hex() sees it. + escape := func(s string) string { + var b strings.Builder + for _, c := range []byte(s) { + if c < 0x80 { + b.WriteByte(c) + } else { + fmt.Fprintf(&b, `\x%02X`, c) + } + } + return b.String() + } + encode := func(s string) string { + return strings.ToUpper(hex.EncodeToString([]byte(escape(s)))) + } + + baseRow := func() map[string]string { + return map[string]string{ + "ca": "0", + "key_algorithm": "rsaEncryption", + "key_strength": "2048", + "key_usage": "Digital Signature", + "serial": "abc123", + "signing_algorithm": "sha256WithRSAEncryption", + "not_valid_after": "1822755797", + "not_valid_before": "1770228826", + "sha1": "aabbccdd00112233445566778899aabbccddeeff", + "source": "system", + "username": "SYSTEM", + "path": "/Library/Keychains/System.keychain", + } + } + + for _, tc := range []struct { + name string + columns map[string]string + wantCommonName string + wantSubjectCN string + wantIssuerCN string + }{ + { + name: "hex encoded", + columns: map[string]string{ + "common_name_hex": encode(commonName), + "subject_hex": encode(subject), + "issuer_hex": encode(issuer), + }, + wantCommonName: commonName, + wantSubjectCN: "Ловушка", + wantIssuerCN: "Acme Root", + }, + { + // A host that received the pre-encoding query before this server started. + name: "plain columns from an in-flight distributed read", + columns: map[string]string{ + "common_name": escape(commonName), + "subject": escape(subject), + "issuer": escape(issuer), + }, + wantCommonName: commonName, + wantSubjectCN: "Ловушка", + wantIssuerCN: "Acme Root", + }, + { + name: "malformed hex is dropped, not ingested raw", + columns: map[string]string{ + "common_name_hex": "zzzz", + "subject_hex": encode(subject), + "issuer_hex": encode(issuer), + }, + wantCommonName: "", + wantSubjectCN: "Ловушка", + wantIssuerCN: "Acme Root", + }, + } { + for _, platform := range []string{"darwin", "windows"} { + t.Run(tc.name+"/"+platform, func(t *testing.T) { + ds := new(mock.Store) + ctx := t.Context() + logger := slog.New(slog.DiscardHandler) + host := &fleet.Host{ID: 1, UUID: "host-uuid", Platform: platform} + + row := baseRow() + for k, v := range tc.columns { + row[k] = v + } + + // parseWindowsDN returns the whole DN as the common name; only parseDarwinDN + // splits out the CN component. + wantSubjectCN, wantIssuerCN := tc.wantSubjectCN, tc.wantIssuerCN + if platform == "windows" { + wantSubjectCN, wantIssuerCN = subject, issuer + } + + ds.UpdateHostCertificatesFunc = func(ctx context.Context, hostID uint, hostUUID string, + certs []*fleet.HostCertificateRecord, origin fleet.HostCertificateOrigin, + ) error { + require.Len(t, certs, 1) + assert.Equal(t, tc.wantCommonName, certs[0].CommonName) + assert.Equal(t, wantSubjectCN, certs[0].SubjectCommonName) + assert.Equal(t, wantIssuerCN, certs[0].IssuerCommonName) + return nil + } + + ingest := directIngestHostCertificatesDarwin + if platform == "windows" { + ingest = directIngestHostCertificatesWindows + } + require.NoError(t, ingest(ctx, logger, host, ds, []map[string]string{row})) + require.True(t, ds.UpdateHostCertificatesFuncInvoked) + }) + } + } +} + +// <<< OPENFRAME(waf-inventory-shape) + func TestDirectIngestHostCertificatesWindows(t *testing.T) { ds := new(mock.Store) ctx := t.Context()