Support PPL foreach command

[songkant-aws]

Description

Adds the PPL foreach command (Splunk-compatible), which runs a templated eval for each field in a field list, each element of a multivalue (array) field, or each element of a JSON array.

# multifield: double every matching field
source=idx | foreach value_* [ eval <<FIELD>>_double = <<FIELD>> * 2 ]

# multivalue: fold array elements into an accumulator
source=idx | eval nums = array(1,2,3), total = 0
           | foreach mode=multivalue nums [ eval total = total + <<ITEM>> ]

# json_array: parse JSON array text (literal or field) and iterate
source=idx | eval total = 0
           | foreach mode=json_array '[1,2,3]' [ eval total = total + <<ITEM>> ]

All four Splunk modes are supported: multifield (default), multivalue, json_array, and auto_collections (runtime detection). Placeholders <<FIELD>>/<<MATCHSTR>> (multifield) and <<ITEM>>/<<ITER>> (collection modes) are supported, with fieldstr/matchstr/itemstr/iterstr rename options.

Design

• Multifield mode expands each eval clause once per matching field via placeholder bindings on CalcitePlanContext; <<FIELD>> resolves to the current row field, <<MATCHSTR>> to the wildcard-captured segment.
• Collection modes rewrite each eval clause into a reduce call over the collection. Elements are packed into internal pairs [item, iter, captured-field...] by a foreach_pair_collection UDF so the two-parameter reduce lambda can reach the loop item, the loop index, and any referenced row fields; foreach_pair_item(pair, slot) extracts a slot with its plan-time type attached. Bindings are staged on the context and only activate inside the cloned lambda context, so the reduce call's own arguments resolve against the row normally.
• json_array element typing: json_array(...) calls and string literals are inspected at plan time (mixed string/number arrays rejected); for opaque expressions (a field holding JSON text — the primary Splunk use), the type is inferred from usage: arithmetic on <<ITEM>> selects DOUBLE, otherwise VARCHAR.
• Planning lives in a dedicated ForeachPlanner class rather than growing CalciteRelNodeVisitor.

Behavior verified against Splunk 10.4.0

Scenario	Splunk	This PR
mv field + multivalue	6	6 ✓
JSON-text field + json_array (sum of "[10,20,30]")	60	60 ✓
json_array mode fed a real array	silent no-op	iterates (more permissive, pinned in IT)
multivalue mode fed JSON text	silent no-op	plan-time error (louder, pinned in IT)

Known limitation (documented in tests and user doc): fields whose mapping is a scalar type (e.g. long) holding arrays are rejected by multivalue mode, because OpenSearch mappings do not declare array-ness and the plan-time type is scalar. Nested-typed fields map to ARRAY<ANY> and iterate correctly.

Related Issues

N/A (new PPL command)

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[songkant-aws]

Note for reviewers: two items from the new-command checklist are still pending and I'll push them to this PR shortly:

• PPLQueryDataAnonymizer.visitForeach (query anonymization)
• v2 Analyzer.visitForeach throwing the standard "only for Calcite" exception (currently the v2 path behavior is undefined)

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 0e74b5d)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Possible Issue In wildcardSegments, the method returns null when a pattern does not match, but the caller in multifieldBindings continues iterating patterns after a null result. If multiple patterns are provided and the first does not match but a later one does, the first null result is silently ignored and the loop breaks on the first match. This may cause incorrect behavior if the intent is to validate all patterns or accumulate segments from multiple matches. The break statement at line 144 ensures only the first matching pattern's segments are used, which may be intentional but is not documented. private List<String> wildcardSegments(String pattern, String fieldName) { if (!WildcardUtils.matchesWildcardPattern(pattern, fieldName)) { return null; } if (!WildcardUtils.containsWildcard(pattern)) { return List.of(); } Matcher matcher = Pattern.compile(WildcardUtils.convertWildcardPatternToRegex(pattern)).matcher(fieldName); if (!matcher.matches()) { return null; } List<String> segments = new ArrayList<>(); for (int i = 1; i <= matcher.groupCount(); i++) { segments.add(matcher.group(i)); } return segments; } Possible Issue The eval method returns null when index is out of bounds (line 61), but does not validate that args[0] is actually a list or array before casting. If args[0] is a list but args[1] is not a Number (e.g., due to type coercion failure upstream), the cast at line 59 will throw ClassCastException. Similarly, if args[0] is neither Object[] nor List, the cast at line 60 will fail. The method should validate types before casting or document the assumption that inputs are guaranteed to be well-typed. public static Object eval(Object... args) { if (args.length < 2 || args[0] == null || args[1] == null) { return null; } int index = ((Number) args[1]).intValue(); List<?> pair = args[0] instanceof Object[] array ? Arrays.asList(array) : (List<?>) args[0]; return index < 0 || index >= pair.size() ? null : pair.get(index); } Possible Issue In jsonElementType, when the collection is a field (opaque at plan time), the method infers DOUBLE if itemUsedInArithmetic returns true, otherwise VARCHAR. However, itemUsedInArithmetic only checks if the item placeholder appears inside an arithmetic operator node. If the item is passed to a function that internally performs arithmetic (e.g., a UDF), this heuristic will miss it and incorrectly infer VARCHAR, leading to type mismatch at runtime when the JSON array contains numbers. The heuristic is fragile and may produce incorrect results for non-trivial expressions. */ private SqlTypeName jsonElementType( UnresolvedExpression collection, CalcitePlanContext context, List<ForeachEvalClause> evalClauses, String itemName) { if (collection instanceof Function function && BuiltinFunctionName.JSON_ARRAY .getName() .getFunctionName() .equalsIgnoreCase(function.getFuncName())) { return elementTypeOf( function.getFuncArgs().stream() .map(arg -> rexVisitor.analyze(arg, context).getType()) .map(RelDataType::getFamily) .collect(Collectors.toSet())); } if (collection instanceof Literal literal && literal.getType() == DataType.STRING) { try { List<?> values = gson.fromJson(String.valueOf(literal.getValue()), List.class); if (values != null) { return elementTypeOf( values.stream() .filter(Objects::nonNull) .map(v -> v instanceof Number ? SqlTypeFamily.NUMERIC : SqlTypeFamily.CHARACTER) .collect(Collectors.toSet())); } } catch (JsonSyntaxException ignored) { // Malformed JSON literals return null at runtime; type choice is irrelevant. } return SqlTypeName.VARCHAR; } return itemUsedInArithmetic(evalClauses, itemName) ? SqlTypeName.DOUBLE : SqlTypeName.VARCHAR; } Possible Issue The toList method at line 75 throws IllegalArgumentException if the input is neither a List nor an Object[]. However, the eval method at line 58 only checks if args[0] is null, not whether it is a valid collection type. If a caller passes a non-collection object (e.g., a scalar integer due to a type error upstream), the exception message "foreach pair collection requires an array input" will be thrown, but the root cause (incorrect input type) may not be obvious. The method should validate the input type earlier or provide a more descriptive error. public static Object eval(Object... args) { if (args.length == 0 || args[0] == null) { return null; } List<?> source = toList(args[0]); List<Object[]> pairs = new ArrayList<>(); for (int i = 0; i < source.size(); i++) { Object[] pair = new Object[args.length + 1]; pair[0] = source.get(i); pair[1] = i; for (int j = 1; j < args.length; j++) { pair[j + 1] = args[j]; } pairs.add(pair); } return pairs; } private static List<?> toList(Object value) { if (value instanceof List<?> list) { return list; } if (value instanceof Object[] array) { return List.of(array); } throw new IllegalArgumentException("foreach pair collection requires an array input"); }

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 0e74b5d
Explore these optional code suggestions:

Category	Suggestion	Impact
General	Validate argument type before casting The method assumes args[1] is a Number without validation, which could cause a ClassCastException if an unexpected type is passed. Add type checking before casting to prevent runtime failures. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/ForeachPairItemFunctionImpl.java [55-62] public static Object eval(Object... args) { if (args.length < 2 || args[0] == null || args[1] == null) { + return null; + } + if (!(args[1] instanceof Number)) { return null; } int index = ((Number) args[1]).intValue(); List<?> pair = args[0] instanceof Object[] array ? Arrays.asList(array) : (List<?>) args[0]; return index < 0 || index >= pair.size() ? null : pair.get(index); } Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies a potential ClassCastException if args[1] is not a Number. Adding type checking before casting improves robustness and prevents runtime failures. This is a valid defensive programming practice for a UDF that may receive unexpected inputs.	Medium
	Handle JSON parsing errors explicitly The method silently catches JsonSyntaxException and defaults to VARCHAR for malformed JSON, which may hide errors. Consider logging the exception or throwing a SemanticCheckException to provide clearer feedback when JSON parsing fails during plan time. core/src/main/java/org/opensearch/sql/calcite/ForeachPlanner.java [284-326] private SqlTypeName jsonElementType( UnresolvedExpression collection, CalcitePlanContext context, List<ForeachEvalClause> evalClauses, String itemName) { if (collection instanceof Function function && BuiltinFunctionName.JSON_ARRAY .getName() .getFunctionName() .equalsIgnoreCase(function.getFuncName())) { return elementTypeOf( function.getFuncArgs().stream() .map(arg -> rexVisitor.analyze(arg, context).getType()) .map(RelDataType::getFamily) .collect(Collectors.toSet())); } if (collection instanceof Literal literal && literal.getType() == DataType.STRING) { try { List<?> values = gson.fromJson(String.valueOf(literal.getValue()), List.class); if (values != null) { return elementTypeOf( values.stream() .filter(Objects::nonNull) .map(v -> v instanceof Number ? SqlTypeFamily.NUMERIC : SqlTypeFamily.CHARACTER) .collect(Collectors.toSet())); } - } catch (JsonSyntaxException ignored) { - // Malformed JSON literals return null at runtime; type choice is irrelevant. + } catch (JsonSyntaxException e) { + throw new SemanticCheckException("Invalid JSON array literal: " + literal.getValue(), e); } return SqlTypeName.VARCHAR; } return itemUsedInArithmetic(evalClauses, itemName) ? SqlTypeName.DOUBLE : SqlTypeName.VARCHAR; } Suggestion importance[1-10]: 4 __ Why: The suggestion proposes throwing a SemanticCheckException instead of silently catching JsonSyntaxException. However, the comment in the existing code explicitly states "Malformed JSON literals return null at runtime; type choice is irrelevant," indicating this is intentional behavior. The suggestion would change the semantics by failing at plan time rather than runtime, which may not align with the design intent.	Low
	Remove redundant pattern matching check The method performs redundant pattern matching. After checking matchesWildcardPattern returns true, the subsequent matcher.matches() check is unnecessary and will always succeed. Remove the second match check to avoid duplicate regex evaluation. core/src/main/java/org/opensearch/sql/calcite/ForeachPlanner.java [153-169] private List<String> wildcardSegments(String pattern, String fieldName) { if (!WildcardUtils.matchesWildcardPattern(pattern, fieldName)) { return null; } if (!WildcardUtils.containsWildcard(pattern)) { return List.of(); } Matcher matcher = Pattern.compile(WildcardUtils.convertWildcardPatternToRegex(pattern)).matcher(fieldName); - if (!matcher.matches()) { - return null; - } + matcher.matches(); // Must call matches() to populate groups List<String> segments = new ArrayList<>(); for (int i = 1; i <= matcher.groupCount(); i++) { segments.add(matcher.group(i)); } return segments; } Suggestion importance[1-10]: 3 __ Why: The suggestion correctly identifies that after matchesWildcardPattern returns true, the subsequent matcher.matches() check is redundant. However, the improved code still calls matcher.matches() (just without the conditional), which is necessary to populate the groups. The impact is minor since it only removes an unnecessary null check.	Low

---

Previous suggestions

Suggestions up to commit 93b0892

Category	Suggestion	Impact
Possible issue	Validate input type before casting The cast (List<?>) args[0] assumes args[0] is a List if it's not an array, but this is not validated. If args[0] is neither an array nor a List, a ClassCastException will occur at runtime. Add explicit type validation or handle unexpected types gracefully. core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/ForeachPairItemFunctionImpl.java [55-62] public static Object eval(Object... args) { if (args.length < 2 || args[0] == null || args[1] == null) { return null; } int index = ((Number) args[1]).intValue(); - List<?> pair = args[0] instanceof Object[] array ? Arrays.asList(array) : (List<?>) args[0]; + List<?> pair; + if (args[0] instanceof Object[] array) { + pair = Arrays.asList(array); + } else if (args[0] instanceof List<?> list) { + pair = list; + } else { + throw new IllegalArgumentException("foreach pair item requires an array or list input"); + } return index < 0 || index >= pair.size() ? null : pair.get(index); } Suggestion importance[1-10]: 7 __ Why: The unchecked cast (List<?>) args[0] could cause a ClassCastException at runtime if args[0] is neither an array nor a List. Adding explicit validation improves robustness and provides clearer error messages.	Medium
General	Remove redundant pattern match check The method checks matchesWildcardPattern first, then compiles a regex and calls matcher.matches() again. If the first check passes, the second matcher.matches() should always succeed, making the second null-return unreachable. Remove the redundant check or consolidate the logic to avoid confusion. core/src/main/java/org/opensearch/sql/calcite/ForeachPlanner.java [153-169] private List<String> wildcardSegments(String pattern, String fieldName) { if (!WildcardUtils.matchesWildcardPattern(pattern, fieldName)) { return null; } if (!WildcardUtils.containsWildcard(pattern)) { return List.of(); } Matcher matcher = Pattern.compile(WildcardUtils.convertWildcardPatternToRegex(pattern)).matcher(fieldName); - if (!matcher.matches()) { - return null; + matcher.matches(); // guaranteed to succeed + List<String> segments = new ArrayList<>(); + for (int i = 1; i <= matcher.groupCount(); i++) { + segments.add(matcher.group(i)); } - ... + return segments; } Suggestion importance[1-10]: 5 __ Why: The redundant matcher.matches() check is unnecessary since matchesWildcardPattern already validates the pattern. However, the code still functions correctly, so this is a minor optimization rather than a bug fix.	Low
	Handle empty type family set explicitly When families is empty (all nulls in the JSON array), the method returns SqlTypeName.VARCHAR by default. This silent fallback may mask data issues. Consider explicitly handling the empty set case or documenting this behavior to clarify the intended semantics. core/src/main/java/org/opensearch/sql/calcite/ForeachPlanner.java [357-365] private SqlTypeName elementTypeOf(Set<?> families) { boolean numeric = families.contains(SqlTypeFamily.NUMERIC); boolean character = families.contains(SqlTypeFamily.CHARACTER); if (numeric && character) { throw new SemanticCheckException( "foreach json_array elements must be consistently strings or numbers"); } + if (families.isEmpty()) { + return SqlTypeName.VARCHAR; // default for all-null arrays + } return numeric ? SqlTypeName.DOUBLE : SqlTypeName.VARCHAR; } Suggestion importance[1-10]: 4 __ Why: The suggestion adds explicit handling for empty families sets, but the current implicit behavior (returning VARCHAR) is reasonable for all-null arrays. This is more of a code clarity improvement than a functional issue.	Low

[songkant-aws]

Both pending checklist items are now in (0e74b5d34):

• Analyzer.visitForeach throws the standard only-for-Calcite UnsupportedOperationException on the v2 path, with a UT asserting the message.
• PPLQueryDataAnonymizer.visitForeach renders the command with masked options/targets/eval clauses (collection targets can embed literals such as JSON array strings); ForeachPlaceholder masks like a column reference. Covered by three anonymizer UTs (multifield, multivalue with options, json_array with a literal target).

The new-command checklist is now fully satisfied.

[github-actions]

Persistent review updated to latest commit 0e74b5d