The unified (Analytics Engine) query path built its size limit from
UnifiedQueryContext.Builder's hardcoded default (10000) because
RestUnifiedQueryAction.applyClusterOverrides never forwarded the live
plugins.query.size_limit cluster value. Any configured value was silently
ignored on the AE route while the default pipeline honored it.
Replace the hand-maintained forwardClusterSetting calls with a single
FORWARDED_CLUSTER_SETTINGS allow-list (single source of truth) and add
QUERY_SIZE_LIMIT to it. CALCITE_ENGINE_ENABLED stays excluded since the
unified path forces Calcite on regardless of the cluster value.
Add unit tests asserting the cluster size limit reaches the AE plan context
and that the Calcite-enabled invariant is not overridden.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
Description
plugins.query.size_limit(and, structurally, other planning settings) was silently ignored on the Analytics Engine (unified query) path — the AE route always used the hardcoded default (10000) regardless of the configured cluster value. The default (non-AE) pipeline honored the setting correctly.Root cause.
RestUnifiedQueryAction.applyClusterOverrides()forwarded only a hand-picked subset of cluster settings into theUnifiedQueryContext(PPL_REX_MAX_MATCH_LIMIT,PPL_SYNTAX_LEGACY_PREFERRED,MAX_EXPRESSION_DEPTH) and omittedQUERY_SIZE_LIMIT.UnifiedQueryContext.Builderseeds a default settings map (QUERY_SIZE_LIMIT = SysLimit.DEFAULT.querySizeLimit()), so with no forward the hardcoded default was never overwritten.addQuerySizeLimit()then injected aLogicalSystemLimitbuilt from that stale value.This is a two-lists-drift defect: the builder's default-settings map and the forward list are maintained independently, so any planning setting present in the map but absent from the forward list silently regresses to its default.
Fix
forwardClusterSettingcalls with a singleFORWARDED_CLUSTER_SETTINGSallow-list — one source of truth for which cluster settings the unified path honors — and addQUERY_SIZE_LIMITto it.CALCITE_ENGINE_ENABLEDis deliberately excluded: the unified path is Calcite-based by definition and must force it on regardless of the cluster value.Testing
RestUnifiedQueryActionTest):clusterQuerySizeLimitReachesAnalyticsContext— the live clusterplugins.query.size_limitreaches the AE plan context'ssysLimit. Fails without the fix (expected:<500> but was:<10000>), passes with it.calciteEngineEnabledNotOverriddenByCluster— the Calcite-enabled invariant is not overridden by the cluster value.SettingsIT.testQuerySizeLimit_NoPushdownis not in the parquet exclude list, so it runs on the AE route under-Dtests.analytics.parquet_indices=true. Against a real composite/AE domain it reproduced the bug (expected:<2> but was:<3>), and this fix turns it green — no new IT needed.Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.