hex-encode certificate DN columns in the osquery detail queries

openframe/docs/README.md

@@ -50,6 +50,7 @@ The agent has its own switch, `--openframe-mode` / `ORBIT_OPENFRAME_MODE`.
 | [agent-openframe-mode.md](agent-openframe-mode.md) | OpenFrame agent mode: gateway URL prefix, encrypted bearer-token pipeline (extract / decrypt / refresh), custom osqueryd, `orbit uuid` command. |
 | [node-key-management.md](node-key-management.md) | Node-key enrollment caching, 401 re-enrollment, Windows file-lock resilience. |
 | [agent-json-content-type.md](agent-json-content-type.md) | Orbit sets `Content-Type: application/json` on requests with a body (upstream sets none). Without it a WAF cannot JSON-parse the body — Cloud Armor flagged 100% of `/orbit/config` polls as SQLi. Unconditional, not gated on OpenFrame mode. |
+| [agent-inventory-waf-shape.md](agent-inventory-waf-shape.md) | The `certificates_darwin`/`certificates_windows` detail queries hex-encode their distinguished-name columns; the ingest decodes them. Raw X.509 DNs are `/`+`=` dense and trip CRS 942431/942432 on every inventory write. Server-side only — no agent upgrade. |
 
 ## Database migrations
 

openframe/docs/agent-inventory-waf-shape.md

@@ -0,0 +1,76 @@
+# Certificate inventory wire shape
+
+**Slug:** `OPENFRAME(waf-inventory-shape)` · **File:** [`server/service/osquery_utils/queries.go`](../../server/service/osquery_utils/queries.go)
+
+The `certificates_darwin` and `certificates_windows` detail queries hex-encode their three
+distinguished-name columns (`common_name`, `subject`, `issuer`), aliased with a `_hex` suffix.
+`decodeCertificateDNColumns` decodes them at ingest, before anything else reads the row.
+
+## Why
+
+An X.509 distinguished name looks like `/C=US/ST=California/O=Acme, Inc./CN=…`. The CRS
+`942431`/`942432` signatures are *restricted special-character counters* — they fire when an
+argument value exceeds 6 (resp. 2) characters from a punctuation class that includes `/` and `=`.
+Every DN exceeds that, so every certificate row in every inventory write matched.
+
+Measured in `shared-j62b` (dev), 24 h to 2026-08-07: **453** Cloud Armor preview DENYs on
+`/tools/agent/fleetmdm-server/api/v1/osquery/*`, all `client-policy`, all UA `osquery/5.9.1`.
+`certificates_darwin.*.subject` / `.common_name` was **237 of them (52.3 %)** — the single largest
+source, and the only large one that is fixable in code:
+
+| Source | Events | % |
+|---|---:|--:|
+| `certificates_darwin` DN columns | 237 | 52.3 % |
+| `data.N.hostIdentifier` | 65 | 14.3 % |
+| `scheduled_query_stats.*.query` | 59 | 13.0 % |
+| `software_windows.*.version` | 49 | 10.8 % |
+| `orbit_info.*.last_recorded_error` | 21 | 4.6 % |
+| `host_details.*`, `fleet_distributed_query_*` | 22 | 4.9 % |
+
+Sample matched values: `=com.apple.kerbe`, `=US/ST=Californi`, `local (`.
+
+Hex specifically, not base64: `[0-9A-F]` contains no punctuation at all. Base64 std (`+/=`) and
+base64url (`-_`) both land back inside the restricted class.
+
+## Rollout
+
+Detail-query SQL is generated server-side and handed to the agent in the `distributed/read`
+response (`detailQueriesForHost`, [`server/service/osquery.go`](../../server/service/osquery.go)).
+Both halves of this change live in the server binary, so **the census clears on server deploy —
+there is no agent upgrade wait**, unlike [agent-json-content-type.md](agent-json-content-type.md).
+
+During the upgrade window a host may post results computed from the previous query. Those rows
+carry the plain columns and no `_hex` key, so `decodeCertificateDNColumns` skips the hex step and
+runs only the `\xHH` unescape — identical to pre-change behaviour. Nothing is dropped.
+
+## Not covered
+
+This removes 52 % of the agent-ingest false positives, not all of them. Hardware UUIDs, third-party
+version strings and live query results are arbitrary by nature; the only way to change their shape
+would be to re-encode the osquery↔Fleet wire format on both ends. `scheduled_query_stats` has a
+supported off switch instead — `FLEET_APP_ENABLE_SCHEDULED_QUERY_STATS=false` — so it is not
+patched here.
+
+Enforcing `942431`/`942432` on the agent path therefore still needs a path-scoped rule band in
+Cloud Armor. See `openframe-saas-tf` `openframe-saas/*/services/03-shared/armor.tf`.
+
+## Verifying
+
+```bash
+gcloud logging read \
+  'resource.type="http_load_balancer" AND
+   jsonPayload.previewSecurityPolicy.outcome="DENY" AND
+   httpRequest.requestUrl:"/api/v1/osquery/"' \
+  --project=shared-j62b --format=json --limit=5000
+```
+
+Group by `previewSecurityPolicy.matchedFieldName`; the `certificates_darwin.*` rows should reach
+zero within one detail-query refetch interval of the deploy.
+
+## Before shipping
+
+Run the modified `SELECT` under `osqueryi` on a real Mac and a real Windows host and confirm
+`hex(subject)` round-trips through `hex.DecodeString` + `DecodeHexEscapes` to the same string the
+previous query produced. Whether osquery's `\xHH` escaping of non-ASCII sits inside the value that
+`hex()` sees decides the decode order, and that is worth confirming against a live host rather than
+a fixture. `TestDirectIngestHostCertificatesHexEncodedDN` covers the Go side.

openframe/docs/fork-file-manifest.md

@@ -273,6 +273,7 @@ server/service/global_policies.go
 server/service/handler.go
 server/service/labels_util.go
 server/service/orbit_client.go
+server/service/osquery_utils/queries.go
 server/service/queries.go
 server/vulnerabilities/nvd/cpe.go
 ```

server/service/osquery_utils/queries.go

@@ -836,9 +836,12 @@ var extraDetailQueries = map[string]DetailQuery{
 		DirectIngestFunc: directIngestDiskEncryptionWindows,
 	},
 	"certificates_darwin": {
+		// >>> OPENFRAME(waf-inventory-shape): hex-encode the DN columns — a raw X.509 DN is `/`
+		// and `=` dense, which the edge WAF scores as SQLi (CRS 942431/942432) on every inventory
+		// write. Decoded in decodeCertificateDNColumns. openframe/docs/agent-inventory-waf-shape.md
 		Query: `
 	SELECT
-		ca, common_name, subject, issuer,
+		ca, hex(common_name) AS common_name_hex, hex(subject) AS subject_hex, hex(issuer) AS issuer_hex,
 		key_algorithm, key_strength, key_usage, signing_algorithm,
 		not_valid_after, not_valid_before,
 		serial, sha1, "system" as source,
@@ -849,7 +852,7 @@ var extraDetailQueries = map[string]DetailQuery{
 		path = '/Library/Keychains/System.keychain'
 	UNION
 	SELECT
-		ca, common_name, subject, issuer,
+		ca, hex(common_name) AS common_name_hex, hex(subject) AS subject_hex, hex(issuer) AS issuer_hex,
 		key_algorithm, key_strength, key_usage, signing_algorithm,
 		not_valid_after, not_valid_before,
 		serial, sha1, "user" as source,
@@ -858,13 +861,16 @@ var extraDetailQueries = map[string]DetailQuery{
 		certificates
 	WHERE
 		path LIKE '/Users/%/Library/Keychains/login.keychain-db';`,
+		// <<< OPENFRAME(waf-inventory-shape)
 		Platforms:        []string{"darwin"},
 		DirectIngestFunc: directIngestHostCertificatesDarwin,
 	},
 	"certificates_windows": {
+		// >>> OPENFRAME(waf-inventory-shape): same treatment as certificates_darwin above —
+		// openframe/docs/agent-inventory-waf-shape.md
 		Query: `
 	SELECT
-		ca, common_name, subject, issuer,
+		ca, hex(common_name) AS common_name_hex, hex(subject) AS subject_hex, hex(issuer) AS issuer_hex,
 		key_algorithm, key_strength, key_usage, signing_algorithm,
 		not_valid_after, not_valid_before,
 		serial, sha1, username,
@@ -873,6 +879,7 @@ var extraDetailQueries = map[string]DetailQuery{
 		certificates
 	WHERE
 		store = 'Personal';`,
+		// <<< OPENFRAME(waf-inventory-shape)
 		Platforms:        []string{"windows"},
 		DirectIngestFunc: directIngestHostCertificatesWindows,
 	},
@@ -3496,6 +3503,36 @@ func GetDetailQueries(
 
 var rxExtractUsernameFromHostCertPath = regexp.MustCompile(`^/Users/([^/]+)/Library/Keychains/login\.keychain\-db$`)
 
+// >>> OPENFRAME(waf-inventory-shape): the certificates detail queries hex-encode their DN columns
+// so raw X.509 punctuation never reaches the edge WAF (CRS 942431/942432 count `/` and `=` as SQL
+// special characters); this decodes them back — openframe/docs/agent-inventory-waf-shape.md
+var certificateDNColumns = []string{"common_name", "subject", "issuer"}
+
+// decodeCertificateDNColumns normalizes the DN columns of one certificates row in place, so the
+// rest of the ingest sees the plain values it always has. A row missing `<col>_hex` came from a
+// host still running the pre-encoding query handed out before this server started; its plain
+// columns are already in place, so only the \xHH unescape runs.
+func decodeCertificateDNColumns(ctx context.Context, logger *slog.Logger, row map[string]string) {
+	for _, col := range certificateDNColumns {
+		if encoded, ok := row[col+"_hex"]; ok {
+			decoded, err := hex.DecodeString(encoded)
+			if err != nil {
+				// Empty beats ingesting a hex string as if it were a DN.
+				logger.ErrorContext(ctx, "decoding hex certificate column", "component", "service",
+					"method", "directIngestHostCertificates", "column", col, "err", err)
+				row[col] = ""
+			} else {
+				row[col] = string(decoded)
+			}
+		}
+
+		// Unescape \xHH sequences for non-ASCII characters (e.g. Cyrillic) in the DN.
+		row[col] = fleet.DecodeHexEscapes(row[col])
+	}
+}
+
+// <<< OPENFRAME(waf-inventory-shape)
+
 func directIngestHostCertificatesDarwin(
 	ctx context.Context,
 	logger *slog.Logger,
@@ -3511,11 +3548,10 @@ func directIngestHostCertificatesDarwin(
 
 	certs := make([]*fleet.HostCertificateRecord, 0, len(rows))
 	for _, row := range rows {
-		// Unescape \xHH sequences in fields that may contain non-ASCII
-		// characters (e.g. Cyrillic) in the certificate's distinguished name.
-		row["common_name"] = fleet.DecodeHexEscapes(row["common_name"])
-		row["subject"] = fleet.DecodeHexEscapes(row["subject"])
-		row["issuer"] = fleet.DecodeHexEscapes(row["issuer"])
+		// >>> OPENFRAME(waf-inventory-shape): the DN columns arrive hex-encoded from the detail
+		// query and must be decoded before use — openframe/docs/agent-inventory-waf-shape.md
+		decodeCertificateDNColumns(ctx, logger, row)
+		// <<< OPENFRAME(waf-inventory-shape)
 
 		csum, err := hex.DecodeString(row["sha1"])
 		if err != nil {
@@ -3604,11 +3640,10 @@ func directIngestHostCertificatesWindows(
 	// SHA1 sum + username
 	existsSha1User := make(map[string]bool, len(rows))
 	for _, row := range rows {
-		// Unescape \xHH sequences in fields that may contain non-ASCII
-		// characters (e.g. Cyrillic) in the certificate's distinguished name.
-		row["common_name"] = fleet.DecodeHexEscapes(row["common_name"])
-		row["subject"] = fleet.DecodeHexEscapes(row["subject"])
-		row["issuer"] = fleet.DecodeHexEscapes(row["issuer"])
+		// >>> OPENFRAME(waf-inventory-shape): the DN columns arrive hex-encoded from the detail
+		// query and must be decoded before use — openframe/docs/agent-inventory-waf-shape.md
+		decodeCertificateDNColumns(ctx, logger, row)
+		// <<< OPENFRAME(waf-inventory-shape)
 
 		csum, err := hex.DecodeString(row["sha1"])
 		if err != nil {

server/service/osquery_utils/queries_test.go

@@ -2944,6 +2944,133 @@ func TestDirectIngestHostCertificatesDarwinHexEscapes(t *testing.T) {
 	require.True(t, ds.UpdateHostCertificatesFuncInvoked)
 }
 
+// >>> OPENFRAME(waf-inventory-shape): covers the hex-encoded DN columns the certificates detail
+// queries emit, the pre-encoding fallback during a server upgrade, and malformed input —
+// openframe/docs/agent-inventory-waf-shape.md
+func TestDirectIngestHostCertificatesHexEncodedDN(t *testing.T) {
+	const (
+		commonName = "Ловушка"
+		subject    = `/C=US/ST=California/O=Acme, Inc./CN=Ловушка`
+		issuer     = `/C=US/O=Acme CA/CN=Acme Root`
+	)
+
+	// osquery escapes non-ASCII as literal \xHH before hex() sees it.
+	escape := func(s string) string {
+		var b strings.Builder
+		for _, c := range []byte(s) {
+			if c < 0x80 {
+				b.WriteByte(c)
+			} else {
+				fmt.Fprintf(&b, `\x%02X`, c)
+			}
+		}
+		return b.String()
+	}
+	encode := func(s string) string {
+		return strings.ToUpper(hex.EncodeToString([]byte(escape(s))))
+	}
+
+	baseRow := func() map[string]string {
+		return map[string]string{
+			"ca":                "0",
+			"key_algorithm":     "rsaEncryption",
+			"key_strength":      "2048",
+			"key_usage":         "Digital Signature",
+			"serial":            "abc123",
+			"signing_algorithm": "sha256WithRSAEncryption",
+			"not_valid_after":   "1822755797",
+			"not_valid_before":  "1770228826",
+			"sha1":              "aabbccdd00112233445566778899aabbccddeeff",
+			"source":            "system",
+			"username":          "SYSTEM",
+			"path":              "/Library/Keychains/System.keychain",
+		}
+	}
+
+	for _, tc := range []struct {
+		name           string
+		columns        map[string]string
+		wantCommonName string
+		wantSubjectCN  string
+		wantIssuerCN   string
+	}{
+		{
+			name: "hex encoded",
+			columns: map[string]string{
+				"common_name_hex": encode(commonName),
+				"subject_hex":     encode(subject),
+				"issuer_hex":      encode(issuer),
+			},
+			wantCommonName: commonName,
+			wantSubjectCN:  "Ловушка",
+			wantIssuerCN:   "Acme Root",
+		},
+		{
+			// A host that received the pre-encoding query before this server started.
+			name: "plain columns from an in-flight distributed read",
+			columns: map[string]string{
+				"common_name": escape(commonName),
+				"subject":     escape(subject),
+				"issuer":      escape(issuer),
+			},
+			wantCommonName: commonName,
+			wantSubjectCN:  "Ловушка",
+			wantIssuerCN:   "Acme Root",
+		},
+		{
+			name: "malformed hex is dropped, not ingested raw",
+			columns: map[string]string{
+				"common_name_hex": "zzzz",
+				"subject_hex":     encode(subject),
+				"issuer_hex":      encode(issuer),
+			},
+			wantCommonName: "",
+			wantSubjectCN:  "Ловушка",
+			wantIssuerCN:   "Acme Root",
+		},
+	} {
+		for _, platform := range []string{"darwin", "windows"} {
+			t.Run(tc.name+"/"+platform, func(t *testing.T) {
+				ds := new(mock.Store)
+				ctx := t.Context()
+				logger := slog.New(slog.DiscardHandler)
+				host := &fleet.Host{ID: 1, UUID: "host-uuid", Platform: platform}
+
+				row := baseRow()
+				for k, v := range tc.columns {
+					row[k] = v
+				}
+
+				// parseWindowsDN returns the whole DN as the common name; only parseDarwinDN
+				// splits out the CN component.
+				wantSubjectCN, wantIssuerCN := tc.wantSubjectCN, tc.wantIssuerCN
+				if platform == "windows" {
+					wantSubjectCN, wantIssuerCN = subject, issuer
+				}
+
+				ds.UpdateHostCertificatesFunc = func(ctx context.Context, hostID uint, hostUUID string,
+					certs []*fleet.HostCertificateRecord, origin fleet.HostCertificateOrigin,
+				) error {
+					require.Len(t, certs, 1)
+					assert.Equal(t, tc.wantCommonName, certs[0].CommonName)
+					assert.Equal(t, wantSubjectCN, certs[0].SubjectCommonName)
+					assert.Equal(t, wantIssuerCN, certs[0].IssuerCommonName)
+					return nil
+				}
+
+				ingest := directIngestHostCertificatesDarwin
+				if platform == "windows" {
+					ingest = directIngestHostCertificatesWindows
+				}
+				require.NoError(t, ingest(ctx, logger, host, ds, []map[string]string{row}))
+				require.True(t, ds.UpdateHostCertificatesFuncInvoked)
+			})
+		}
+	}
+}
+
+// <<< OPENFRAME(waf-inventory-shape)
+
 func TestDirectIngestHostCertificatesWindows(t *testing.T) {
 	ds := new(mock.Store)
 	ctx := t.Context()