Skip to content

feat(index)!: opt-in track_total_hits + DSL/rebuild fixes (8.0.0-beta.5) - #2

Merged
ykan821 merged 70 commits into
masterfrom
dev
Jun 28, 2026
Merged

feat(index)!: opt-in track_total_hits + DSL/rebuild fixes (8.0.0-beta.5)#2
ykan821 merged 70 commits into
masterfrom
dev

Conversation

@ykan821

@ykan821 ykan821 commented Jun 28, 2026

Copy link
Copy Markdown
Owner

Summary

Serialization correctness fixes, opt-in pagination totals, and engineering/docs cleanup for 8.0.0-beta.5.

Breaking changes

Index now defaults track_total_hits to false, so Elasticsearch omits the hit total:

  • Results::total() / lastPage() return ?int (null when unavailable)
  • Results::toPaginator() throws PaginationTotalUnavailableException

Migration — opt in on indexes that paginate:

class ProductIndex extends Index
{
    protected int|bool $trackTotalHits = true; // or a count cap, e.g. 5000
}

For total-less iteration, use hasMorePages() / chunk() / cursor().

Included

  • DSL: Node::toArray() (and Intervals) throw LogicException for an unset field-keyed field (was an
    uncatchable Error); Agg empty body serializes to {}; RangeSupport rejects positional keys beyond [start, end]; SpanTerm::term() emits {value}.
  • Index/Search: opt-in track_total_hits (Index property, int|bool); total()/lastPage() are ?int;
    hasMorePages(); toPaginator() guard; Rebuild cleans up the orphaned index on swap failure.
  • Docs: README/guide/index (EN + zh) cover the default + opt-in; CLAUDE.md TODO refreshed; static-state warning;
    phpstan stub note.
  • Deprecated: DateHistogram::interval().
  • Removed: composer analyse/cs-check/cs-fix scripts (run the binaries via your Docker workflow).

Checks

  • unit 298 / integration 89 / full 387 pass
  • phpstan · phpmd · php-cs-fixer clean

See CHANGELOG — 8.0.0-beta.5.

ykan821 and others added 30 commits June 8, 2026 23:33
…rrent rebuilds

Implements a distributed lock via ES doc create (op_type=create):
- run() auto-acquires/releases the lock (try/finally)
- forceUnlock() manually releases a stale lock
- isLocked() queries lock state
- ensureLockIndex() auto-creates the .ek_locks system index

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… count() response

first()/scroll()/paginate()/aggregateScalar() failed to restore query
state when doSearch() threw; now wrapped in try-finally to guarantee
restoration.

count() now throws RuntimeException when the ES response lacks the
count field.
- must()/should()/filter() etc. on Boolean/DisjunctionMax/SpanOr/SpanNear now use append semantics
- each clause key holds a single Query(multi=true); closures forward the same instance
- Query instances are stored via addQuery(); buildQuery() flattens them into clauses
- add Query::getQueries(); rename _queryClauses to _queries
- remove Compound::wrapQueryAsClosure()
- add totalRelation() mapping hits.total.relation directly
- aggregations() now returns null by default instead of an empty array for clearer semantics
- confirmed hasMore() has no bug; !empty(hits) is correct for scroll

Co-Authored-By: Claude <noreply@anthropic.com>
…:name()

- rollback() removes aliases from all backing indices, mirroring doRun()
- name() throws when $name is not set in a subclass

Co-Authored-By: Claude <noreply@anthropic.com>
…te TODO

- Node: extract fromKeyValue/fromClosure/fromArrayField/fromScalar
- Query: reuse fromClosure, add fromArray to handle the query key
- constructor becomes a clear dispatch table, each method handles one input form
- update CLAUDE.md TODO status

Co-Authored-By: Claude <noreply@anthropic.com>
…d.import.failed event

- Bulk gains onError(callable); execute() throws on errors by default
- Rebuild drops skipErrors, gains onError(callable) that delegates to Bulk internally
- remove the rebuild.import.failed event; error handling goes through the onError callback uniformly
- batchSize auto-flush errors are also caught via onError, no longer lost
- update docs and TODO

Co-Authored-By: Claude <noreply@anthropic.com>
…rom Index

Index static methods are kept as @deprecated proxies; call sites migrate to the new classes.
Tests swap ReflectionProperty resets for the new classes' reset().
…oc instance entry points

- remove Index::insert(); equivalent functionality is covered by Doc::save()
- remove 6 deprecated proxy methods (listen/dispatch/setPageResolver etc.)
- add on(string $connection), newQuery(), newDoc() instance methods
- add setConnection()/getConnection() instance methods
- query()/doc() now delegate to newQuery()/newDoc()
- ClientManager drops the implicit fallback and gains type declarations
- update tests and README
All 12 files: declare(strict_types=1) + constructor property promotion + readonly + native property/return/param types + union types.

Main changes:
- fix Manager::resolveIndexName() null-alias map causing null->TypeError (array_key_first() ?? $name)
- Doc $id now supports string|int|null: index/create omit the id to let ES auto-generate it, while get/source/exists/update/delete guard with requireId() (closes the hole of silently sending a null id to ES)
- Bulk::execute() gains a json_encode false guard (prevents strlen(false) TypeError under strict_types)
- callable properties (resolver/errorHandler/dataSource) keep a docblock (PHP forbids callable as a property type)

Sync: TestConcreteIndex and docs examples get property types; CLAUDE.md TODO split into Index/DSL layers.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
- strict_types across all 151 files; the 4 atomic properties $_key/$_valueKey/$_fieldKeyed/$_multi get types synced across all leaves; leaf parameter types completed against ES docblocks
- unify $_properties as ?array; add $_raw to carry full pass-through
- rename $_isPropertyField to $_fieldKeyed for naming consistency (aligned with $_multi)
- remove the dead setMulti (Node/Query); multi() handles it uniformly
- remove 30 field() overrides equivalent to the base, keeping Highlight
- clean up redundant properties and toArray in AdjacencyMatrix/Composite; serialization pushed down to resolveProperties

Co-Authored-By: Claude <noreply@anthropic.com>
…ue access

- rename FilterAgg/GlobalAgg/ParentAgg to Filter/Global_/Parent_ (drop the Agg suffix to align with ES keywords; global/parent are PHP reserved words, hence the _)
- rename setFilter()/globalAggregation() to filter()/global()
- remove the $_raw full pass-through mechanism from Node; rename $_rawValue to $_value and add a union type; the scalar construct branch no longer constrains _fieldKeyed
- Query drops the $_raw branch accordingly; add rector.php with modernization rules

Co-Authored-By: Claude <noreply@anthropic.com>
Single-argument public methods in all non-trait classes now use $value
uniformly, avoiding named-argument BC risk. Multi-argument methods and
generic trait methods are unaffected.

- bulk-rename ~350 method parameters (docblocks and bodies kept in sync)
- remove the duplicate Knn::boost()
- fix inconsistent Suggest docblock types (string|null -> ?string)
- fix the PHP_CS_FIXER_IGNORE_ENV deprecation warning
Fill in the missing regexp rule for the intervals query, placed between wildcard and fuzzy to match the ES docs. Add a Regexp leaf (pattern/analyzer/useField).

Co-Authored-By: Claude <noreply@anthropic.com>
Following 2e18121 (leaf setters unified to $value), extend the naming convention to the non-leaf layer — Query/Param/Agg/Function — and all builder traits:

- Param: 26 single-arg setters use $value; sort() becomes pure append; indices_boost switches to append mode (chained calls accumulate instead of overwriting); fix sort docblock
- Agg: $subAggs -> $_subAggs; add type declarations to node/alias/toArray
- traits (Compound/Span/Specialized/FullText/Joining/MatchAll/TermLevel/Aggs): single-arg builders unify on $value; Bucket::filter synced
- Function_: the second arg of the 4 decay methods becomes $value

Co-Authored-By: Claude <noreply@anthropic.com>
- $name -> $connection (ClientManager::set / Index::setClient)
- $newName -> $newIndex (Rebuild, aligning with the <role>Index pattern)
- $document -> $data (index/save/create in Doc/Bulk; unify document payload naming across the library)
- add callable|iterable parameter type to Rebuild::source; fix the \Iterator docblock typo

Co-Authored-By: Claude <noreply@anthropic.com>
- the old cursor() (yielding Results per batch) is renamed chunk()
- add cursor(): yields each full hit (_id/_score/_source), flattening chunk and reusing its scroll cleanup
- keep the low-level scroll/next/clear; the backend may switch to PIT+search_after later with upper-level signatures unchanged
- sync README/docs/IndexTest

**BC:** cursor() return changes from Generator<Results> (per batch) to Generator<hit> (per hit)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
- run() splits try-catch: failures throw doRun's original exception first (lock-release failure is not masked); if lock release fails after success, throw a RuntimeException (noting the rebuild completed and forceUnlock is needed)
- releaseLock drops the \Throwable catch-all and swallows only 404
- forceUnlock is implemented independently and swallows 404 (idempotent success whether the index or doc is missing)
- isLocked returns false only on 404; other ES errors propagate

**BC:** forceUnlock/isLocked no longer swallow non-404 ES errors; run() throws on the success path when lock release fails (previously swallowed silently)

Co-Authored-By: Claude <noreply@anthropic.com>
- phpunit.xml switches to the PHPUnit 10 schema declaration + cacheDirectory
- .gitignore covers .phpunit.cache/
- CLAUDE.md: mark Rebuild exception handling and cursor/chunk API refactor as done

Co-Authored-By: Claude <noreply@anthropic.com>
- Query::buildQuery throws on same-key clauses instead of silently overwriting (extracted mergeClauses)
- ScriptScore::minScore drops the Query::create wrapper so the float serializes correctly
- Pipeline::bucketScript rejects the string shorthand (buckets_path must be a map)

Co-Authored-By: Claude <noreply@anthropic.com>
…support 4 forms, fix empty-bool bug

- remove Node::routeKeyValueClause; fromKeyValue returns to a pure leaf (object values throw)
- ClausesSupport routes via method_exists (no clauseKeys declaration needed)
- array-clause containers (Boolean/DisjunctionMax/SpanOr/SpanNear) use ClausesSupport, supporting closure/array/two-arg/instance
- single-value compounds (Boosting/ConstantScore) use the Node default, no ClausesSupport
- Compound/Span trait methods use create($field, $value) to unify single/two-arg
- empty nodes serialize to a valid shape (bool:{}, must:[]) instead of being omitted, fixing {bool:null}/{must:{}}

Co-Authored-By: Claude <noreply@anthropic.com>
…te rule keys

- toArray previously bypassed Node serialization entirely; inherited methods like boost() and array-constructed $_properties were silently dropped — now merged into the field node
- when merging rules/properties, array_intersect_key detects conflicts; duplicate keys throw RuntimeException instead of being silently overwritten (aligned with Query::mergeClauses)
- private mergeUnique within the class, not polluting Node

Co-Authored-By: Claude <noreply@anthropic.com>
…ndices by default

- delete(bool $resolveAlias = false): when name is an alias, throw RuntimeException by default; resolveAlias=true is required to delete the backing indices
- with resolveAlias=true, delete all backing indices the alias points to (comma-joined), fixing the former array_key_first pitfall of deleting only the first
- removing an alias relationship uses removeAlias(); deleting a real index name is unchanged

**BC:** delete() with no args and an alias name now throws instead of silently deleting backing indices

Co-Authored-By: Claude <noreply@anthropic.com>
…s to a raw-tools contract

Data-loss fix: execute() previously cleared body before checking response['errors'], so callers couldn't retry after a partial auto-flush failure. It now keeps the queue on error — preserved whether there is no handler (throws) or the handler throws; cleared only on success or when the handler returns.

The onError handler contract expands from fn($response) to fn($response, $body, $newbulk): the library does no retry logic, it only hands over three raw materials — the ES response, the full batch (including successful items, native format), and a fresh Bulk on the same index inheriting targetIndex (no handler; throws on its own errors without recursion). Users extract failed items from $body per $response and re-submit them on $newbulk. Rebuild::onError forwarding updated accordingly.

**BC:** onError handler signature changes (old single-arg handlers still work, extra args ignored); execute() error-path behavior changes (keeps the queue instead of clearing).

Co-Authored-By: Claude <noreply@anthropic.com>
create() signature becomes nullable; the body writes _id conditionally like index() (omitted when empty so ES auto-generates it), aligned with Doc::create.

Co-Authored-By: Claude <noreply@anthropic.com>
ykan821 and others added 29 commits June 26, 2026 22:55
- DslTestCase: JSON-only assertQuery; fix ensureIndex exists() asBool (ES validation never actually ran); expose createIndex/seedData/ensureSpecialFields as protected; drop ES connection from unit setUpBeforeClass
- IntegrationTestCase: per-test random index isolation + assertQueryEs/makeIndex, skips without ELASTICKIT_TEST_HOST
- phpunit.xml: integration testsuite; unit excludes tests/Integration
- SmokeTest: verifies ES reachable + seed

Co-Authored-By: Claude <noreply@anthropic.com>
- TermLevel: term/terms/range/exists/prefix/wildcard/fuzzy/ids
- FullText: match/matchPhrase/queryString/matchBoolPrefix
- Compound: bool must/should/filter/mustNot, constantScore
- Geo: geoDistance/geoBoundingBox
- Joining: nested (match/term)

Co-Authored-By: Claude <noreply@anthropic.com>
- Specialized: script, scriptScore
- Aggregation: terms/sum/avg/stats/cardinality/date_histogram

Co-Authored-By: Claude <noreply@anthropic.com>
- SearchContractTest: get/first/count/paginate/scroll/chunk/cursor (real ES)
- DocContractTest: index/get/source/exists/update/upsert/create-conflict(409)/delete/auto-id (real ES)
- Remove mock-based IndexTest and DocTest (replaced by integration)

Co-Authored-By: Claude <noreply@anthropic.com>
…ocks

- BulkContractTest: index/create/update/delete, batchSize auto-flush, onError (real ES; error triggered by create-on-existing id)
- ManagerContractTest: exists/putMapping/alias/refresh/delete (real ES)
- Remove mock-based BulkTest and ManagerTest

Co-Authored-By: Claude <noreply@anthropic.com>
- RebuildContractTest: run creates alias + imports, swap (atomic alias exchange), rejects real index (real ES)
- Remove mock-based RebuildTest

Co-Authored-By: Claude <noreply@anthropic.com>
- StatsSupportContractTest: max/min/sum/avg/stats (real ES aggregations)
- Remove mock-based StatsSupportTest
- tests/Index/ now keeps only EventTest + ResultsTest (library behavior: event dispatch, response parsing)

Co-Authored-By: Claude <noreply@anthropic.com>
- EventContractTest: real ES triggers event dispatch (search/bulk + multiple listeners)
- ResultsTest moved to tests/Integration/ (pure parsing, constructed response data)
- Remove tests/Index/, TestClient stubs, index testsuite, bootstrap require
- Two-layer architecture: unit (DSL build + pure logic) + integration (real ES)
- CLAUDE.md: drop --testsuite index from pre-push checklist

Co-Authored-By: Claude <noreply@anthropic.com>
- IntegrationTestCase: one index per test class (static map), reset between tests via deleteByQuery + re-seed instead of recreate; ~48% faster (132s -> 69s)
- DslTestCase: remove dead $esIndex and empty setUpBeforeClass

Co-Authored-By: Claude <noreply@anthropic.com>
- NodeInvariantsTest: lock down toArray branches (value shorthand, value+property promotion, valueKey conflict, empty fieldKeyed->null, properties-only, float survives) — the P0 data-loss boundaries
- Query::when(): drop doc claim that bare strings are treated as truthy; signature is bool|Closure (strict_types rejects strings), so the documented behavior can't occur

Co-Authored-By: Claude <noreply@anthropic.com>
- Move ClientManager, Event, EventDispatcher, StatsSupport, Pagination under src/Index/Support/
- Domain classes (Index/Search/Results/Doc/Bulk/Manager/Rebuild) stay at the root
- Update all use-declarations and cross-namespace references

Co-Authored-By: Claude <noreply@anthropic.com>
- README.md: English (primary)
- README.zh.md: Chinese translation
- cross-links between the two

Co-Authored-By: Claude <noreply@anthropic.com>
Index::setPaginatorResolver -> Pagination::setPaginatorResolver (where the resolver actually lives after the Support/ reorg)

Co-Authored-By: Claude <noreply@anthropic.com>
…reSpecialFields

- SearchContractTest: testFirst asserts the hit title (was only assertIsArray); testScroll asserts hit count (was only isEmpty)
- ResultsTest moved to tests/ (unit) — pure parsing, belongs in the unit suite
- DslTestCase: remove dead ensureSpecialFields() (integration never called it)

Co-Authored-By: Claude <noreply@anthropic.com>
…k/isLocked/allowEmpty)

- testRollback: alias swaps then rolls back to the previous backing index
- testClean: backing index is deleted
- testForceUnlockIsIdempotent: forceUnlock tolerates a missing lock (404)
- testIsLockedFalseAfterRun: lock released after a successful run
- testEmptySourceThrowsWithoutAllowEmpty: empty source without allowEmpty throws
- testAllowEmpty: allowEmpty permits an empty source
- shared rebuildIndex() helper to cut the anonymous-class boilerplate

Co-Authored-By: Claude <noreply@anthropic.com>
…am signature

- Boolean must/should/filter/mustNot: @PARAM mixed -> @PARAM Query|\Closure|array (4)
- Query::addQuery: @PARAM mixed -> Query|\Closure|array|Node
- Query::when $query/$default: @PARAM mixed -> Query|\Closure|array
- Param::hasParam: add string signature type (was only in docblock)
- Boolean.php: add use ElasticKit\DSL\Query for PHPDoc resolution
- Follows mainstream framework convention: polymorphic builder params keep no signature type, rely on detailed PHPDoc

Co-Authored-By: Claude <noreply@anthropic.com>
- ci.yml: integration job was running --testsuite unit (ES container spinning idle); now --testsuite integration so the 89 real-ES contract tests actually run in CI
- composer.json: remove audit.block-insecure=false to restore composer audit's insecure-package blocking

Co-Authored-By: Claude <noreply@anthropic.com>
… accumulation, first from, audit

- ci.yml: integration job now runs --testsuite integration (was unit; ES container was idle)
- Query::buildClauses: throw on scalar clauses (was silently producing empty must/should/etc)
- Query::aggs: throw on duplicate alias for Agg-instance/array branches (was silently overwriting; closure branch keeps accumulating)
- Param::rescore: accumulate on repeated calls (was last-write-wins; single call still produces a single object for BC)
- Search::first(): reset from=0 (was returning the Nth doc if from() was set)
- composer.json: drop audit.block-insecure=false

Co-Authored-By: Claude <noreply@anthropic.com>
- .php-cs-fixer.php Finder adds tests/
- apply fixes to 14 test files (anonymous class spacing etc.)

Co-Authored-By: Claude <noreply@anthropic.com>
…e accumulated)

- Query::aggs + Agg::aggs closure branch: was reusing existing Agg (silent accumulate); now throws like instance/array branches
- Incremental aggregation building goes through closure-internal or Agg object, not repeated top-level aggs() calls
- Aligns with map semantics: alias is a key (not a list index); duplicate key = error

Co-Authored-By: Claude <noreply@anthropic.com>
- DeepClone: cache ReflectionProperty[] per class (static), avoid new ReflectionClass on every clone
- Search::doSearch: $extra['body'] merges into query body (not replaces)
- first()/paginate()/scroll(): pass size/from override via $extra['body'] instead of clone+restore — all three hot paths are now clone-free

Co-Authored-By: Claude <noreply@anthropic.com>
- New RegistersAgg trait in src/DSL/Support/: registerAgg($alias, $aggs, &$store) consolidates the ~50-line aggs() body shared between Query and Agg
- Query::aggs() and Agg::aggs() now one-liner delegates
- @SuppressWarnings(PHPMD.NPathComplexity) on registerAgg (flat polymorphic dispatch)
- phpstan memory limit via --memory-limit=256M CLI flag (neon/CI/CLAUDE.md/composer.json)

Co-Authored-By: Claude <noreply@anthropic.com>
…set after each)

- Removed resetOptions() that silently cleared retryOnConflict and refresh after each write
- Options now persist until explicitly changed, consistent with Bulk and builder conventions
- Updated docblock: 'next write operation' -> 'subsequent write operations'

Co-Authored-By: Claude <noreply@anthropic.com>
- Node: toArray() throws LogicException when a field-keyed node has no
  field set, replacing an uncatchable typed-property Error
- Agg: empty aggregation body serializes to {} (stdClass), not [] which
  Elasticsearch rejects; aligns with Global_/ReverseNested
- RangeSupport: reject positional elements beyond the [start, end]
  shorthand instead of leaking them as numeric string keys
- SpanTerm: term() emits ES's {value} key (was {term}); delegates to value()
- DateHistogram: mark interval() @deprecated (ES deprecated the key)

Baseline pre-existing RegistersAgg by-ref phpstan errors (proper fix
deferred to the phpstan cleanup).

Co-Authored-By: Claude <noreply@anthropic.com>
Pagination total tracking is now opt-in per index:
- Index: add $trackTotalHits (default false); Search applies it to
  non-scroll requests (ES forbids disabling it in a scroll context)
- Results: total()/lastPage() return ?int, null when the total is
  unavailable (track_total_hits=false); add hasMorePages() (full-page
  heuristic when no total)
- Results: toPaginator() throws PaginationTotalUnavailableException when
  the total is unavailable; isEmpty() docblock points to hasMorePages()
- max_result_window overflow is left to Elasticsearch's own 400

Rebuild: wrap the alias-swap step (refresh + updateAliases/putAlias) in
try/catch and delete the orphaned new index on failure.

**BC:** Index now defaults track_total_hits to false, so total()/lastPage()
return null and toPaginator() throws unless the index sets trackTotalHits=true.

Co-Authored-By: Claude <noreply@anthropic.com>
- CLAUDE.md: replace stale TODO (scroll/bulk/rebuild now tested, infra
  built) with real gaps — Span/Shape integration contracts, Rebuild
  import-failure rollback
- README.md / README.zh.md: warn that Index::setClient / ClientManager /
  EventDispatcher / Pagination hold static state that leaks across
  requests in long-lived workers (Swoole/RoadRunner/Octane); show reset()
- stubs/ClientInterface.stub: document that phpstan does not merge
  interface-stub methods over the vendored interface, so endpoint calls
  stay baseline-suppressed (the Indices class stub works; this one does not)

Co-Authored-By: Claude <noreply@anthropic.com>
…inator

- Intervals::toArray() read \$this->_field without the field-keyed guard that
  Node gained, so (new Intervals())->toArray() still threw an uncatchable
  typed-property Error. Extract a shared Node::wrapFieldKeyed() helper and
  use it in Node, Intervals, and FunctionScore so every field-keyed
  toArray() path is guarded (LogicException).
- Results::hasMorePages(): guard perPage <= 0 to avoid the 0 === 0
  false positive.
- Results::toPaginator(): guard on total() === null (the value it needs)
  rather than totalRelation().

Co-Authored-By: Claude <noreply@anthropic.com>
$trackTotalHits is int|bool (true counts every hit, false omits the total,
an int caps the count) — was typed bool, which rejected a custom cap.
Document the default-false behavior across README/docs/guide (EN + zh):
total()/lastPage() return null and toPaginator() throws unless the index
sets $trackTotalHits = true (or a cap); use hasMorePages/chunk/cursor for
total-less iteration.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@ykan821
ykan821 merged commit 5315fc1 into master Jun 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant