Fix job statistics bugs and improve docs

[bgunebakan]

Summary of the changes / Why this is an improvement

Investigating #871 showed that ctk info jobs and ctk cfr jobstats are not two implementations of the same thing, so this PR does not consolidate them.

ctk info jobs takes a one-shot snapshot and lets CrateDB aggregate it in SQL, while ctk cfr jobstats polls continuously and accumulates call counters, duration histograms, and a decaying average in Python, so statistics outlive the bounded sys.jobs_log.

it is very difficult to understand which command does what from the documentation"*. So this PR keeps both engines, documents what each one is for, and fixes the bugs found while reading them.

Bug fixes

• collect --anonymize <file> was declared is_flag=True, so passing the decoder dictionary path the help text advertised failed outright.

• collect --anonymize fell back to the original statement when anonymization raised, storing in clear text exactly what the option exists to protect. Such statements are now redacted to a stable digest, so the option fails closed.

• view --deanonymize emitted unparseable output.

• view --deanonymize reported every statement twice.

• view mixed up fields, reporting the average duration as the duration histogram, the histogram as the user, and the user as the query type.

• collect counted a job twice when it ended exactly on the watermark millisecond.

• collect derived its watermark from an arbitrary record whenever the watermark table held more than one.

• collect never resumed where a previous run left off. the watermark was written but never read back.

• collect lost statistics when started twice within one process.

• report and ui refused to run whenever statistics had been collected into a schema other than stats.

• top100_count aliased PERCENTILE(..., 0.99) as p90. The value was always the 99th percentile, only the label was wrong. The field is named p99 now, which breaks consumers reading p90.

Checklist

• Link to issue this PR refers to (if applicable): Fixes Improve the two job-statistics implementations (ctk info jobs vs ctk cfr jobstats) #871
• Add changes in CHANGES.md

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: de5b6d5e-0025-47aa-97ba-fba1bd204580

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[florinutz]

good stuff! dropped some comments

[florinutz]

Not quite true. The filter is stmt NOT LIKE '%sys.%', which only drops the sys.jobs_log read, while everything the collector writes still gets recorded. After two collect --once cycles against a live cluster, the statistics table holds the collector's own CREATE TABLE IF NOT EXISTS …jobstats_statements, REFRESH TABLE …, SELECT MAX(last_execution) …, INSERT INTO …jobstats_last and UPDATE …jobstats_last.

[hammerhead]

Thanks, added a few small comments.

[hammerhead]

A question that came to my mind, haven't tried it out: If the same query is submitted by a different user, is username updated, ignored, or a new row gets added?

[bgunebakan]

I checked, it is "ignored", and I think it's a bug. Could you share the results when you try?

[hammerhead]

Suggest adding a short sentence explaining each command:

Suggested change

	Collect statistics, then display or explore them.
	```shell
	ctk cfr jobstats collect
	```
	```shell
	ctk cfr jobstats view
	```
	```shell
	ctk cfr jobstats report
	```
	```shell
	ctk cfr jobstats ui
	```
	Collects statistics on an ongoing basis:
	```shell
	ctk cfr jobstats collect
	```
	Prints collected statistics as a JSON document:
	```shell
	ctk cfr jobstats view
	```
	Shows the top 10 collected statements, sorted by runtime descending:
	```shell
	ctk cfr jobstats report
	```
	Launches a web interface with visualisations for interactive exploration of statistics:
	```shell
	ctk cfr jobstats ui
	```

[bgunebakan]

Thanks, done! ecbbd05

[hammerhead]

The --anonymize part was slightly unclear to me after reading it initially. It wasn't obvious that the decoder dictionary gets automatically generated if missing (and updated once running).

It also prints a warning when running for the first time about not being able to load the encoder dictionary. That warning looks a little scary with the "No such file or directory". Maybe could be a more in the direction of "No existing encoder dictionary found, creating a new one"? Basically not mentioning the file-not-found error.

$ ctk cfr jobstats collect --anonymize
2026-08-14 12:49:08,042 [cratedb_toolkit.cfr.jobstats        ] INFO    : SQL anonymization is enabled, using dictionary: decoder_dictionary.json
2026-08-14 12:49:08,042 [cratedb_toolkit.cfr.jobstats        ] INFO    : Connecting to crate://crate:REDACTED@localhost:4200/
2026-08-14 12:49:08,058 [cratedb_toolkit.cfr.jobstats        ] INFO    : Resuming from recorded watermark: 1786704526754
2026-08-14 12:49:08,058 [cratedb_toolkit.cfr.jobstats        ] INFO    : Recording information snapshot
2026-08-14 12:49:08,058 [cratedb_toolkit.cfr.jobstats        ] INFO    : Reading sys.jobs_log
2026-08-14 12:49:08,064 [cratedb_toolkit.cfr.jobstats        ] WARNING : Could not load encoder dictionary: [Errno 2] No such file or directory: 'decoder_dictionary.json'
2026-08-14 12:49:08,070 [cratedb_toolkit.cfr.jobstats        ] INFO    : Writing statistics to database table: "stats".jobstats_statements
2026-08-14 12:49:08,076 [cratedb_toolkit.cfr.jobstats        ] INFO    : Sleeping for 10.0 seconds

[bgunebakan]

You're right, no need to scare user with warning, the missing file case is handled separately now with "No decoder dictionary found yet, creating a new one", and it's info log.