[FLINK-40380] Document the runtime configuration and stop jobs from overriding operator config through it

[Dennis-Mircea]

What is the purpose of the change

Follow-up to FLINK-35746, which added runtime configuration observation (reading a job's effective settings from the Flink REST API and layering them over the observed configuration).

Two gaps are addressed. The operator documentation still describes the previous configuration model, to the point of stating the operator cannot observe the running configuration. And the global job parameters taken from the REST response are applied unfiltered, so a job can override operator configuration for its own resource.

Brief change log

Documentation (docs/content{,.zh}/docs/internals/startup.md)

• The Configuration section now lists four configurations rather than three, with a Runtime row describing what it is read from (the JobManager configuration, job execution and checkpoint config REST endpoints) and what it is used for.
• The Observe row states that the runtime configuration is layered on top, matching FlinkResourceContext#getObserveConfig.
• A short paragraph explains why the runtime configuration exists, since a spec is a request rather than a record of what the job ended up running with, followed by its lifecycle: fetched once per job, skipped for globally terminal jobs, cached per resource and job id, and falling back to spec-derived values while a fetch keeps failing.
• The closing warning previously said "None of these is the running configuration", which FLINK-35746 made untrue. It is replaced rather than removed, since the mapping only covers a subset of settings and only while the job runs, so a config.yaml baked into the image is still invisible to the operator.

Filtering (FlinkRuntimeConfigurationUtils)

• Global job parameters whose key is in the operator's own namespaces (kubernetes.operator. and job.autoscaler.) are no longer copied into the runtime configuration, and each dropped key is logged at WARN.
• The check uses the existing KubernetesOperatorConfigOptions.K8S_OP_CONF_PREFIX and AutoScalerOptions.AUTOSCALER_CONF_PREFIX constants so it follows the prefixes if they ever change.

The filter is deliberately limited to those two namespaces. The JobManager configuration endpoint legitimately contributes keys such as rest.*, so filtering more broadly by key name would risk dropping genuinely observed settings, whereas operator and autoscaler keys have no reason to arrive from a job's global parameters at all.

Verifying this change

• AbstractFlinkServiceTest#testMapJobConfigurationDropsOperatorControlledGlobalParameters asserts that an operator key and an autoscaler key set as global job parameters are dropped, while an unrelated parameter and the mapped execution fields are kept.
• The existing testMapJobConfigurationMapsAllExpectedFields continues to pass, so ordinary parameters are unaffected.
• The full flink-kubernetes-operator suite passes (2240 tests).

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changes to the CustomResourceDescriptors: no
• Core observer or reconciler logic that is regularly executed: yes, the runtime configuration mapping used by the job status observer

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? the Internals startup documentation is corrected to describe the runtime configuration added in FLINK-35746

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

[gyfora]

Let's remove this log, it's basically unnecessary and does not communicate anything interesting to the operator. Operator configs are not job configs anyways.

[Dennis-Mircea]

I've removed it now.

[gyfora]

Aside from the too verbose logging, this looks good!