Skip to content
27 changes: 27 additions & 0 deletions api/v1alpha1/kollecttarget_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,12 +78,39 @@ type KollectTargetStatus struct {
// +optional
LastExtractionError string `json:"lastExtractionError,omitempty"`

// collectedCount is the number of resources this target was collecting when the
// controller last refreshed the count. It is the machine-readable source of truth
// for collection scale; the Ready condition message restates it as prose for
// backward compatibility only.
//
// A null value means the controller has never computed a count for this target
// (it has not yet reached Ready). Zero means it computed a count and it was zero.
// A target that is Degraded, or whose reconciles are failing, keeps its last known
// count rather than silently reporting a fresh-looking number — see
// collectedCountUpdatedAt for when that measurement was taken.
// +optional
CollectedCount *int64 `json:"collectedCount,omitempty"`

// collectedCountUpdatedAt is when collectedCount last *changed* — not when it was
// last checked. A steady target whose count has not moved keeps an old timestamp
// while still being re-derived every resync, so an old timestamp on its own does
// not mean the number is stale.
//
// Read it together with the conditions: a Ready target with an old timestamp has a
// count that genuinely has not moved, while a Degraded target keeps its last known
// count and the timestamp shows how long ago that measurement was taken.
// +optional
CollectedCountUpdatedAt *metav1.Time `json:"collectedCountUpdatedAt,omitempty"`

CollectionFilterStatus `json:",inline"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=ktgt
// +kubebuilder:printcolumn:name="Collected",type=integer,JSONPath=`.status.collectedCount`
// +kubebuilder:printcolumn:name="Updated",type=date,JSONPath=`.status.collectedCountUpdatedAt`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`

// KollectTarget is the Schema for the kollecttargets API
type KollectTarget struct {
Expand Down
9 changes: 9 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions charts/kollect/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ unchanged (`readOnlyRootFilesystem: true`, capabilities dropped, `/tmp` `emptyDi
| controller.maxConcurrentReconciles.inventory | int | `3` | Max concurrent reconciles for KollectInventory. |
| controller.maxConcurrentReconciles.target | int | `5` | Max concurrent reconciles for KollectTarget. |
| controller.reconcileRateLimit | string | `""` | |
| controller.targetCountResync | string | `""` | How often a Ready KollectTarget is requeued to refresh `status.collectedCount` (empty = manager default 60s). Nothing else re-enqueues a Target when objects enter or leave its matched set, so this bounds how stale the reported count can be. |
| createNamespace | bool | `false` | Create the release namespace if it does not exist. |
| defaultExcludedNamespaces | list | `[]` | Default namespace denylist for Target collection intent (CRD fields on KollectTarget override). |
| defaultIncludedNamespaces | list | `[]` | Default namespace allowlist for Target collection intent (CRD fields on KollectTarget override). |
Expand Down
38 changes: 37 additions & 1 deletion charts/kollect/crds/kollect.dev_kollecttargets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,17 @@ spec:
singular: kollecttarget
scope: Namespaced
versions:
- name: v1alpha1
- additionalPrinterColumns:
- jsonPath: .status.collectedCount
name: Collected
type: integer
- jsonPath: .status.collectedCountUpdatedAt
name: Updated
type: date
- jsonPath: .metadata.creationTimestamp
name: Age
type: date
name: v1alpha1
schema:
openAPIV3Schema:
description: KollectTarget is the Schema for the kollecttargets API
Expand Down Expand Up @@ -354,6 +364,32 @@ spec:
description: activeResourceRules is the number of compiled resourceRules
entries (0 when using legacy fallback).
type: integer
collectedCount:
description: |-
collectedCount is the number of resources this target was collecting when the
controller last refreshed the count. It is the machine-readable source of truth
for collection scale; the Ready condition message restates it as prose for
backward compatibility only.

A null value means the controller has never computed a count for this target
(it has not yet reached Ready). Zero means it computed a count and it was zero.
A target that is Degraded, or whose reconciles are failing, keeps its last known
count rather than silently reporting a fresh-looking number — see
collectedCountUpdatedAt for when that measurement was taken.
format: int64
type: integer
collectedCountUpdatedAt:
description: |-
collectedCountUpdatedAt is when collectedCount last *changed* — not when it was
last checked. A steady target whose count has not moved keeps an old timestamp
while still being re-derived every resync, so an old timestamp on its own does
not mean the number is stale.

Read it together with the conditions: a Ready target with an old timestamp has a
count that genuinely has not moved, while a Degraded target keeps its last known
count and the timestamp shows how long ago that measurement was taken.
format: date-time
type: string
conditions:
description: |-
conditions represent the current state of the KollectTarget resource.
Expand Down
3 changes: 3 additions & 0 deletions charts/kollect/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ spec:
{{- if .Values.controller.reconcileRateLimit }}
- --reconcile-rate-limit={{ .Values.controller.reconcileRateLimit }}
{{- end }}
{{- if .Values.controller.targetCountResync }}
- --target-count-resync={{ .Values.controller.targetCountResync }}
{{- end }}
{{- if .Values.pprof.enabled }}
- --enable-pprof
- --pprof-bind-address={{ .Values.pprof.bindAddress }}
Expand Down
4 changes: 4 additions & 0 deletions charts/kollect/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,10 @@ controller:
informerResyncPeriod: 12h
collectMetricsSampleInterval: 30s
reconcileRateLimit: ""
# -- How often a Ready KollectTarget is requeued to refresh `status.collectedCount`
# (empty = manager default 60s). Nothing else re-enqueues a Target when objects enter
# or leave its matched set, so this bounds how stale the reported count can be.
targetCountResync: ""

resourcesProfile: default

Expand Down
1 change: 1 addition & 0 deletions cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@ func main() {
MaxConcurrentClusterTarget: cfg.maxConcurrentClusterTarget,
MaxConcurrentClusterInventory: cfg.maxConcurrentClusterInventory,
ReconcileRateLimitBase: cfg.reconcileRateLimit,
TargetCountResync: cfg.targetCountResync,
}

if err := (&controller.KollectTargetReconciler{
Expand Down
4 changes: 4 additions & 0 deletions cmd/startup_flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (

ctrl "sigs.k8s.io/controller-runtime"

"github.com/platformrelay/kollect/internal/controller"
"github.com/platformrelay/kollect/internal/inventory"
"github.com/platformrelay/kollect/internal/validation"
)
Expand Down Expand Up @@ -40,6 +41,7 @@ type startupConfig struct {
maxConcurrentClusterTarget int
maxConcurrentClusterInventory int
reconcileRateLimit time.Duration
targetCountResync time.Duration
enablePprof bool
pprofAddr string
watchNamespacesRaw string
Expand Down Expand Up @@ -126,6 +128,8 @@ func bindStartupFlags(fs *flag.FlagSet, cfg *startupConfig) {
"Max concurrent KollectClusterInventory reconciles.")
fs.DurationVar(&cfg.reconcileRateLimit, "reconcile-rate-limit", 0,
"Base delay for per-item exponential reconcile failure rate limiting (0 = controller-runtime default 5ms).")
fs.DurationVar(&cfg.targetCountResync, "target-count-resync", controller.DefaultTargetCountResync,
"How often a Ready KollectTarget is requeued to refresh status.collectedCount (0 = default 60s).")
fs.BoolVar(&cfg.enablePprof, "enable-pprof", false,
"Expose Go pprof on --pprof-bind-address (separate from metrics).")
fs.StringVar(&cfg.pprofAddr, "pprof-bind-address", ":6060",
Expand Down
4 changes: 4 additions & 0 deletions cmd/startup_flags_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"testing"
"time"

"github.com/platformrelay/kollect/internal/controller"
"github.com/platformrelay/kollect/internal/inventory"
"github.com/platformrelay/kollect/internal/validation"

Expand Down Expand Up @@ -51,6 +52,9 @@ func TestBindStartupFlags_Defaults(t *testing.T) {
cfg.collectDispatchQueueSize,
)
}
if cfg.targetCountResync != controller.DefaultTargetCountResync {
t.Fatalf("targetCountResync = %s, want %s", cfg.targetCountResync, controller.DefaultTargetCountResync)
}
if cfg.informerResyncPeriod != 12*time.Hour || cfg.collectMetricsSampleInterval != 30*time.Second {
t.Fatalf(
"unexpected duration defaults: resync=%s sample=%s",
Expand Down
38 changes: 37 additions & 1 deletion config/crd/bases/kollect.dev_kollecttargets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,17 @@ spec:
singular: kollecttarget
scope: Namespaced
versions:
- name: v1alpha1
- additionalPrinterColumns:
- jsonPath: .status.collectedCount
name: Collected
type: integer
- jsonPath: .status.collectedCountUpdatedAt
name: Updated
type: date
- jsonPath: .metadata.creationTimestamp
name: Age
type: date
name: v1alpha1
schema:
openAPIV3Schema:
description: KollectTarget is the Schema for the kollecttargets API
Expand Down Expand Up @@ -354,6 +364,32 @@ spec:
description: activeResourceRules is the number of compiled resourceRules
entries (0 when using legacy fallback).
type: integer
collectedCount:
description: |-
collectedCount is the number of resources this target was collecting when the
controller last refreshed the count. It is the machine-readable source of truth
for collection scale; the Ready condition message restates it as prose for
backward compatibility only.

A null value means the controller has never computed a count for this target
(it has not yet reached Ready). Zero means it computed a count and it was zero.
A target that is Degraded, or whose reconciles are failing, keeps its last known
count rather than silently reporting a fresh-looking number — see
collectedCountUpdatedAt for when that measurement was taken.
format: int64
type: integer
collectedCountUpdatedAt:
description: |-
collectedCountUpdatedAt is when collectedCount last *changed* — not when it was
last checked. A steady target whose count has not moved keeps an old timestamp
while still being re-derived every resync, so an old timestamp on its own does
not mean the number is stale.

Read it together with the conditions: a Ready target with an old timestamp has a
count that genuinely has not moved, while a Degraded target keeps its last known
count and the timestamp shows how long ago that measurement was taken.
format: date-time
type: string
conditions:
description: |-
conditions represent the current state of the KollectTarget resource.
Expand Down
6 changes: 5 additions & 1 deletion docs/adr/0603-performance-scalability.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,11 @@ operator**.
6. **Dispatch pool:** Tunable `--collect-dispatch-workers` / queue; enqueue wait, then block
(backpressure) on the informer goroutine until queue capacity frees or ctx cancels — never
processes inline, to keep worker concurrency the only path that does extract/access-check work.
7. **Resync / metrics sampling:** `--informer-resync-period`; `--collect-metrics-sample-interval`.
7. **Resync / metrics sampling:** `--informer-resync-period`; `--collect-metrics-sample-interval`;
`--target-count-resync` (default **60s**) requeues each Ready `KollectTarget` to refresh
`status.collectedCount` — costs one cluster-wide namespace `LIST` per Target per interval, **two**
when a `KollectScope` is enforced on that Target's namespace, so it is the knob to raise on large
fleets.
8. **Profiling:** Optional `--enable-pprof` on `:6060`; disabled in production Helm values.
9. **Tests:** `load`-tagged tests to **10k** (nightly when 8-core runners exist); 100k manual design proof only.
10. **100k claim gate:** Export sharding enforced + Postgres bulk upsert + **10k nightly green** (once re-enabled).
Expand Down
26 changes: 26 additions & 0 deletions docs/operator-manual/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,32 @@ exponential failure rate limiter (5ms base, 1000s cap). Set a positive duration
`kollect_workqueue_depth` approximates queue pressure as **in-flight reconciles** per controller
(not the internal client-go queue length).

## Reported collection scale

**`KollectTarget.status.collectedCount`** is the machine-readable number of resources a Target is
collecting, surfaced as the `COLLECTED` column of `kubectl get kollecttargets`. The `Ready`
condition message restates it as prose for backward compatibility only.

Objects entering or leaving a Target's matched set do **not** enqueue that Target, so the number is
refreshed by a periodic self-requeue: **`--target-count-resync`** (default **`60s`**, Helm
`controller.targetCountResync`).

**Budget the resync — it is not free.** Each pass costs **one live, cluster-wide, unpaginated
namespace `LIST`** — **two** when a `KollectScope` is enforced on the Target's namespace, because
the scope check resolves the filter status as well. At the default interval, **N** Targets therefore
cost **N** namespace `LIST`s per minute — **2N** under an enforced scope. The rest is cheap: the
engine skips the informer backfill when the Target's state is unchanged, and the status write is
skipped when the number did not move.

**`status.collectedCountUpdatedAt`** records when the number last *changed* — **not** when it was
last checked. A steady Target keeps an old timestamp while still being re-derived every resync, so
an old timestamp on its own does not mean the count is stale.

Judge liveness from the timestamp and the conditions together: a `Ready` Target with an old
timestamp has a count that genuinely has not moved, while a `Degraded` Target keeps its last known
count and the timestamp shows how old that measurement is. Lower the interval for a more responsive
count; raise it to cut reconcile volume on very large fleets.

## Export debouncing

**`KollectInventory.spec.exportMinInterval`** (default **`30s`**) coalesces export to external sinks
Expand Down
8 changes: 8 additions & 0 deletions docs/reference/conditions.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,14 @@ Conditions describe the latest observed generation; use them with Events and con
| `Synced` | The latest observed inventory was exported successfully. |
| `Degraded` | A terminal or partial failure needs attention. |

**`KollectTarget` `Ready.lastTransitionTime` is not a flap signal.** The `Ready` message restates
the live collected count, and the controller treats a changed message as a transition. A busy
Target whose count moves therefore gets a fresh `lastTransitionTime` on every refresh — as often as
`--target-count-resync` (default `60s`) — while `Ready` never leaves `True`. Do not alert on it as
if the Target had flapped, and do not read an *old* `lastTransitionTime` as "collection has
stalled": it only means the count has not moved. Use `status.collectedCountUpdatedAt` and the
`Degraded` condition instead.

`KollectInventory.status.sinkExports[]` records timestamps, checksums, and conditions per sink.
This distinguishes partial fan-out from total failure. Full collected payloads never live in CR
status.
Expand Down
21 changes: 16 additions & 5 deletions internal/collect/engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,12 @@ func NewEngine(

// RegisterTargetOptions carries resolved namespace and rule state for collection filtering.
type RegisterTargetOptions struct {
ScopeCeiling ScopeCeiling
ScopeCeiling ScopeCeiling
// EffectiveNamespaces is the namespace set the caller already resolved. Supplying it
// carries a contract: the caller is responsible for the freshness of the engine's
// namespace metadata cache (call RefreshNamespaces first), because RegisterTarget
// then skips the cluster-wide namespace LIST. Leave it empty to have the engine
// refresh and recompute the set itself.
EffectiveNamespaces []string
}

Expand Down Expand Up @@ -281,10 +286,6 @@ func (e *Engine) RegisterTarget(
return nil
}

if err := e.refreshNamespaceCache(ctx); err != nil {
log.FromContext(ctx).Error(err, "refresh namespace cache")
}

gvr := gvrFromProfile(profile.Spec.TargetGVK)

compiled, err := CompileResourceRules(target.Spec.ResourceRules, e.extractor.celEnv)
Expand All @@ -303,6 +304,16 @@ func (e *Engine) RegisterTarget(
if len(effective) == 0 {
namespaceSource = "recomputed"

// Only the recompute branch reads the namespace cache here, so only it has to
// pay for a live cluster-wide namespace LIST. Callers that supply
// EffectiveNamespaces have already resolved the set from a snapshot they
// refreshed themselves (RefreshNamespaces), and reconcilers re-register on every
// pass — refreshing unconditionally made every resync a LIST for every target
// (PERF-FIX-05 review finding F2).
if err := e.refreshNamespaceCache(ctx); err != nil {
log.FromContext(ctx).Error(err, "refresh namespace cache")
}

e.nsMu.RLock()
matched := MatchIntentNamespaces(
target.Spec.CollectionFilterSpec,
Expand Down
Loading
Loading