Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# tokitoki-cli

TokiToki is a small cross-platform uploader for local AI coding usage. Each
Tokitoki is a small cross-platform uploader for local AI coding usage. Each
run reads the configured local agent usage folders, then uploads the discovered
events to the local TokiToki server.
events to the local Tokitoki server.

The CLI can run once or install itself as an OS service.

Expand Down Expand Up @@ -78,7 +78,7 @@ service status Print service status.

## Stable project names

TokiToki normally uses the project name reported by an IDE or local AI agent.
Tokitoki normally uses the project name reported by an IDE or local AI agent.
To give a checkout a stable name across editors, machines, and differently
named local folders, create `.tokitoki` in the project root:

Expand All @@ -90,7 +90,7 @@ release/2026
The first line overrides the project name. The optional second line overrides
the branch. An empty file uses the containing folder's name and preserves any
branch reported by the editor. The nearest project file found by walking up
from the active file wins; for out-of-tree agent files, TokiToki also searches
from the active file wins; for out-of-tree agent files, Tokitoki also searches
the event's reported project path.

Use `{project}` in the first line to include the nearest Git, Mercurial, or
Expand All @@ -107,8 +107,7 @@ folder containing `.tokitoki`.
Only regular files are read: the shared data directory `~/.tokitoki` shares
the name but is a directory, so it is never mistaken for a project file.

Only `.tokitoki` is read. legacy `.legacy-project` and `.toolconfig`
files are not treated as project names by TokiToki.
Only `.tokitoki` is read; other tools' project files are ignored.

Project-file resolution is applied centrally before events enter the local
queue, so it affects IDE heartbeats and AI-agent usage scans consistently. It
Expand Down Expand Up @@ -154,7 +153,7 @@ tokitoki service install --system

## The shared CLI

Every TokiToki front-end — the macOS and Windows apps and every editor
Every Tokitoki front-end — the macOS and Windows apps and every editor
plugin — invokes one shared copy of this CLI. Its location is a contract, not
a suggestion; a plugin that resolves a different path forks the fleet and
stops receiving updates:
Expand Down
14 changes: 7 additions & 7 deletions TODO.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# TokiToki TODO
# Tokitoki TODO

## Known Issues (prioritized, reviewed 2026-07-17)

Expand All @@ -19,7 +19,7 @@ P0 — storage consolidation (done 2026-07-17):
P0 — dead scaffolding after the daemon→CLI refactor:

- [x] Remove the heartbeat chain. It POSTed to `<server>/heartbeats`, which the
TokiToki server does not implement (only `/api/usage-events/batch` exists).
Tokitoki server does not implement (only `/api/usage-events/batch` exists).
Removed: `Heartbeat` type, `RecordHeartbeat`/`Sync`/`postHeartbeats`,
`queue.jsonl` + store queue methods, and the `heartbeat`/`push-heartbeats`
subcommands. (A new `heartbeat` subcommand exists today, but it feeds the
Expand All @@ -34,15 +34,15 @@ P1 — correctness/robustness:
- [x] Upload watermark: each event now has local upload state, so `tokitoki`
uploads only pending/failed events and marks accepted/duplicate server
responses as uploaded.
- [ ] Claude streaming token upsert (see legacy note below) — affects count
- [ ] Claude streaming token upsert (see notes below) — affects count
accuracy.
- [ ] Replace hardcoded `InstallationID: "local-go-agent"` with a per-machine id
persisted under `~/.tokitoki/`, so one user's multiple devices stay distinct.

P2 — missing basics:

- [x] `version` command / `--version`, with the version injected via ldflags.
- [ ] Scan lock + large-transcript protection (see legacy notes below).
- [ ] Scan lock + large-transcript protection (see notes below).

P3 — cross-platform packaging:

Expand All @@ -53,12 +53,12 @@ P3 — cross-platform packaging:

## Scanner Ideas To Revisit

- Add a scan cursor similar to legacy `ai_logs_last_parsed_at` so unchanged provider files are skipped. (The old `source_files` table was dead code and has been removed; a fresh design would add per-file rows to `usage.db`.)
- Add a per-file scan cursor so unchanged provider files are skipped. (The old `source_files` table was dead code and has been removed; a fresh design would add per-file rows to `usage.db`.)
- Add a short global scan lock so startup scan and manual `/usage/scan` cannot parse the same files concurrently.
- Add large transcript protection: bounded scanner buffers, max line size handling, and optional tail scanning for very large session files.
- Fix Claude streaming-update semantics: use a logical event key based on `provider + session_id + request_id + message_id`, then upsert latest token values instead of inserting a new event when token counts grow.
- Review Codex token handling against legacy total-token/delta approach and decide whether TokiToki should store raw token-count events or normalized deltas.
- ~~Add upload queue behavior inspired by legacy offline sync~~ — done 2026-07-17: batch limits, stop-on-first-failure, exponential backoff, permanent rejection, and pruning all live in `usagedb` + `usageupload.SyncPending`.
- Review Codex token handling and decide whether Tokitoki should store raw token-count events or normalized deltas.
- ~~Add offline upload queue behavior~~ — done 2026-07-17: batch limits, stop-on-first-failure, exponential backoff, permanent rejection, and pruning all live in `usagedb` + `usageupload.SyncPending`.
- Add summary-level upload state so server sync sends daily/project/model summaries instead of every local `usage_event`.
- Add source status/debug endpoints for scan counts, last scan time, last error, indexed event count, and pending upload count.

Expand Down
6 changes: 3 additions & 3 deletions cmd/tokitoki/main.go
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Command tokitoki scans local Claude Code/Codex usage files and uploads the
// resulting events to the local TokiToki server.
// resulting events to the local Tokitoki server.
package main

import (
Expand Down Expand Up @@ -547,7 +547,7 @@ func copyProviderDirs(providerDirs map[agentlib.Provider][]string) map[agentlib.
}

func usage() {
fmt.Fprint(os.Stderr, `tokitoki — upload local AI usage to the TokiToki server
fmt.Fprint(os.Stderr, `tokitoki — upload local AI usage to the Tokitoki server

Usage:
tokitoki [--provider-dir PROVIDER=DIR ...] [--check-update]
Expand All @@ -561,7 +561,7 @@ Usage:
tokitoki service <install|uninstall|start|stop|restart|status> [options]

Each invocation scans the provider roots you pass and uploads their usage
events to the TokiToki server (TOKITOKI_BASE_URL, default
events to the Tokitoki server (TOKITOKI_BASE_URL, default
https://tokitoki.dev). By default, tokitoki scans the built-in roots for
claude, codex, copilot, gemini, kimi, qwen, openclaw, pi, amp, droid, kilo,
hermes, codebuff, opencode, and goose. Pass one or more
Expand Down
4 changes: 2 additions & 2 deletions cmd/tokitoki/service_linux.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ func systemdUnitTexts(flags workerFlags, system bool) (string, string, error) {

var service strings.Builder
service.WriteString("[Unit]\n")
service.WriteString("Description=Sync local AI usage to TokiToki\n")
service.WriteString("Description=Sync local AI usage to Tokitoki\n")
service.WriteString("Wants=network-online.target\n")
service.WriteString("After=network-online.target\n\n")
service.WriteString("[Service]\n")
Expand All @@ -145,7 +145,7 @@ func systemdUnitTexts(flags workerFlags, system bool) (string, string, error) {
seconds = 1
}
timer := fmt.Sprintf(`[Unit]
Description=Run the TokiToki usage sync on an interval
Description=Run the Tokitoki usage sync on an interval

[Timer]
OnBootSec=2min
Expand Down
4 changes: 2 additions & 2 deletions cmd/tokitoki/service_other.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,8 @@ func newService(flags workerFlags, userService bool) (daemonservice.Service, err
}
config := &daemonservice.Config{
Name: "tokitoki",
DisplayName: "TokiToki",
Description: "Sync local AI usage to TokiToki.",
DisplayName: "Tokitoki",
Description: "Sync local AI usage to Tokitoki.",
Executable: executable,
Arguments: serviceArguments(flags),
Option: daemonservice.KeyValue{
Expand Down
4 changes: 0 additions & 4 deletions internal/agent/agent.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,6 @@ import (

type Settings struct {
APIKey string `json:"api_key"`
// InstallationID is this install's stable random identity. The server
// keys device rows on it, so it must be unique per machine and constant
// across runs — the store generates it once and persists it.
InstallationID string `json:"installation_id"`
}

type Store interface {
Expand Down
4 changes: 2 additions & 2 deletions internal/agentusage/amp.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"github.com/tokitoki-dev/tokitoki-cli/internal/usage"
)

func loadAmpEntries(paths []string) ([]usage.Entry, error) {
func loadAmpEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
files := make([]string, 0)
for _, path := range paths {
if info, err := os.Stat(path); err == nil && !info.IsDir() {
Expand All @@ -25,7 +25,7 @@ func loadAmpEntries(paths []string) ([]usage.Entry, error) {
}
}
sort.Strings(files)
files = uniqueStrings(files)
files = filterFiles(uniqueStrings(files), filter)

entries := make([]usage.Entry, 0)
for _, file := range files {
Expand Down
4 changes: 2 additions & 2 deletions internal/agentusage/codebuff.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,13 @@ type codebuffUsage struct {
extraTotalTokens uint64
}

func loadCodebuffEntries(paths []string) ([]usage.Entry, error) {
func loadCodebuffEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
files := make([]string, 0)
for _, root := range paths {
files = append(files, collectCodebuffChatFiles(root)...)
}
sort.Strings(files)
files = uniqueStrings(files)
files = filterFiles(uniqueStrings(files), filter)

entriesByID := make(map[string]usage.Entry)
for _, file := range files {
Expand Down
4 changes: 2 additions & 2 deletions internal/agentusage/copilot.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ import (
"github.com/tokitoki-dev/tokitoki-cli/internal/usage"
)

func loadCopilotEntries(paths []string) ([]usage.Entry, error) {
func loadCopilotEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
files := make([]string, 0)
for _, root := range paths {
files = append(files, collectExt(root, ".jsonl")...)
}
sort.Strings(files)
files = uniqueStrings(files)
files = filterFiles(uniqueStrings(files), filter)

entries := make([]usage.Entry, 0)
for _, file := range files {
Expand Down
4 changes: 2 additions & 2 deletions internal/agentusage/droid.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ import (
"github.com/tokitoki-dev/tokitoki-cli/internal/usage"
)

func loadDroidEntries(paths []string) ([]usage.Entry, error) {
func loadDroidEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
files := make([]string, 0)
for _, root := range paths {
files = append(files, collectFiles(root, isDroidSettingsFile)...)
}
sort.Strings(files)
files = uniqueStrings(files)
files = filterFiles(uniqueStrings(files), filter)

entries := make([]usage.Entry, 0)
for _, file := range files {
Expand Down
4 changes: 2 additions & 2 deletions internal/agentusage/gemini.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ import (
"github.com/tokitoki-dev/tokitoki-cli/internal/usage"
)

func loadGeminiEntries(paths []string) ([]usage.Entry, error) {
func loadGeminiEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
files := make([]string, 0)
for _, root := range paths {
files = append(files, collectExt(root, ".json")...)
files = append(files, collectExt(root, ".jsonl")...)
}
sort.Strings(files)
files = uniqueStrings(files)
files = filterFiles(uniqueStrings(files), filter)

entries := make([]usage.Entry, 0)
for _, file := range files {
Expand Down
14 changes: 14 additions & 0 deletions internal/agentusage/helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,20 @@ func collectFiles(root string, match func(string) bool) []string {
return files
}

// filterFiles drops files the filter rejects. A nil filter keeps everything.
func filterFiles(files []string, filter usage.FileFilter) []string {
if filter == nil {
return files
}
kept := files[:0]
for _, file := range files {
if filter(file) {
kept = append(kept, file)
}
}
return kept
}

func collectExt(root, ext string) []string {
ext = strings.ToLower(ext)
return collectFiles(root, func(path string) bool {
Expand Down
4 changes: 2 additions & 2 deletions internal/agentusage/kimi.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ import (
"github.com/tokitoki-dev/tokitoki-cli/internal/usage"
)

func loadKimiEntries(paths []string) ([]usage.Entry, error) {
func loadKimiEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
files := make([]string, 0)
for _, root := range paths {
files = append(files, collectFiles(filepath.Join(root, "sessions"), isKimiWireFile)...)
}
sort.Strings(files)
files = uniqueStrings(files)
files = filterFiles(uniqueStrings(files), filter)

entries := make([]usage.Entry, 0)
seen := make(map[string]bool)
Expand Down
3 changes: 2 additions & 1 deletion internal/agentusage/openclaw.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,13 @@ import (
"github.com/tokitoki-dev/tokitoki-cli/internal/usage"
)

func loadOpenClawEntries(paths []string) ([]usage.Entry, error) {
func loadOpenClawEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
files := make([]string, 0)
for _, path := range paths {
files = append(files, collectFiles(path, isOpenClawSessionFile)...)
}
sort.Strings(files)
files = filterFiles(files, filter)

entries := make([]usage.Entry, 0)
for _, file := range files {
Expand Down
7 changes: 4 additions & 3 deletions internal/agentusage/opencode.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ import (
"github.com/tokitoki-dev/tokitoki-cli/internal/usage"
)

func loadOpenCodeEntries(paths []string) ([]usage.Entry, error) {
func loadOpenCodeEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
entriesByID := make(map[string]usage.Entry)
for _, root := range paths {
rootEntries, err := loadOpenCodeRoot(root)
rootEntries, err := loadOpenCodeRoot(root, filter)
if err != nil {
return nil, err
}
Expand All @@ -36,7 +36,7 @@ func loadOpenCodeEntries(paths []string) ([]usage.Entry, error) {
return entries, nil
}

func loadOpenCodeRoot(root string) ([]usage.Entry, error) {
func loadOpenCodeRoot(root string, filter usage.FileFilter) ([]usage.Entry, error) {
info, err := os.Stat(root)
if err != nil {
return nil, nil
Expand Down Expand Up @@ -73,6 +73,7 @@ func loadOpenCodeRoot(root string) ([]usage.Entry, error) {

files := collectExt(filepath.Join(root, "storage", "message"), ".json")
sort.Strings(files)
files = filterFiles(files, filter)
for _, file := range files {
stem := strings.TrimSuffix(filepath.Base(file), filepath.Ext(file))
if seenIDs[stableOpenCodeMessageID(stem)] {
Expand Down
3 changes: 2 additions & 1 deletion internal/agentusage/pi.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,13 @@ import (
"github.com/tokitoki-dev/tokitoki-cli/internal/usage"
)

func loadPiEntries(paths []string) ([]usage.Entry, error) {
func loadPiEntries(paths []string, filter usage.FileFilter) ([]usage.Entry, error) {
files := make([]string, 0)
for _, path := range paths {
files = append(files, collectExt(path, ".jsonl")...)
}
sort.Strings(files)
files = filterFiles(files, filter)

entries := make([]usage.Entry, 0)
for _, file := range files {
Expand Down
Loading