feat(sync): sync activity log, /sync page, truthful indicators + settings#222
Merged
Conversation
…tus system QRZ auto-sync could never work. It authenticated with username/password against logbook.qrz.com, which only accepts pre-issued logbook API keys, and it recorded results in qrz_sync_status/qrz_sync_date/qrz_logbook_id/ qrz_sync_error columns that exist in no migration or schema — every status UPDATE threw, and backgroundAutoSync swallowed the error. Uploads silently did nothing. - New src/lib/qrz-sync-service.ts: single home for QRZ upload/download, used by auto-sync, the manual per-contact and bulk routes, and (next PR) the sync cron. All paths use the station qrz_api_key and record state in the ADIF-style qrz_qsl_sent/qrz_qsl_rcvd columns the UI reads. 'M' (modified) rows re-upload with OPTION=REPLACE; failures mark 'R' and are retried on every sweep. - Auto-sync now runs inside next/server after() — a bare fire-and-forget promise is frozen on Vercel when the response returns. - Contact edits flag already-uploaded copies 'Y' -> 'M' (flagForReupload) instead of writing the nonexistent qrz_sync_status column, which threw on every edit of core QSO fields. - QRZ downloads keep an incremental cursor in stations.qrz_last_qsl_rcvd_date (previously never written) via MODSINCE. - Delete uploadQSOToQRZ (username/password handshake), updateQrzSyncStatus, dead Contact.matchQSO, dead generateAdifForLoTW, and the phantom fields on ContactData; migration 0004 drops the orphaned qrz_sync_status_enum type. - Rename the misleading contacts query param qrz_sync_status=not_synced to qrz_filter=not_sent (it always filtered by qrz_qsl_sent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cron routes trusted any request whose host/user-agent/x-vercel-id header
mentioned vercel, which disabled the CRON_SECRET check on every *.vercel.app
deployment. Separately, /api/lotw/{upload,download} skipped auth entirely on
a spoofable X-Cron-Job: true header, letting anonymous callers trigger
uploads/downloads for any station_id.
- New src/lib/cron-auth.ts: strict Bearer CRON_SECRET check, no fallbacks.
- Cron routes require the Bearer token unconditionally and fail closed (500)
when CRON_SECRET is unset. Vercel attaches the header automatically when
the env var exists.
- X-Cron-Job is now only a mode discriminator: cron mode additionally
requires the valid Bearer token; the cron routes' internal fetches send it.
- README: "Scheduled sync" section for Vercel + self-hosted crontab setup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the two daily LoTW-only cron routes with a single /api/cron/sync
orchestrator that runs the full cycle per station, in order: QRZ upload
sweep, LoTW upload, QRZ confirmation download, LoTW confirmation download.
Uploads run before downloads so fresh QSOs can confirm in the same cycle,
and QRZ/LoTW legs are serialized so the cross-service 'M' re-upload flags
never race. QRZ finally gets automation — previously upload was on-save or
manual only and confirmation download was manual only.
- QRZ legs iterate active stations with an API key whose owner enabled
qrz_auto_sync, via uploadPendingForStation (250 QSOs/station/run cap,
'R' failures retried every sweep) and downloadConfirmationsForStation
(incremental MODSINCE cursor).
- LoTW legs keep the existing per-station selection, <1h dedupe, and
internal fetch to /api/lotw/{upload,download} with the CRON_SECRET Bearer
header, preserving lotw_*_logs bookkeeping.
- vercel.json: single hourly cron (Vercel Pro), maxDuration 300 for the new
route; old cron routes deleted; README updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ings
Sync failures were invisible: QRZ had no history at all, auto-sync failures
vanished, and the settings UI implied username/password powered logbook
sync. This adds the visibility layer:
- New sync_logs table (migration 0005) + SyncLog model. QRZ upload sweeps,
downloads, manual bulk syncs, and failed on-save auto-syncs all write
rows (trigger manual/auto/cron); no-op cron sweeps are not logged so the
feed stays readable. LoTW keeps its dedicated log tables.
- GET /api/sync/logs: one reverse-chronological feed merging sync_logs with
normalized lotw_upload_logs / lotw_download_logs rows.
- New /sync page ("Sync Activity") listing every run with service,
direction, station, trigger, status, and full error message; linked from
the navbar More menu.
- QRZSyncIndicator + LotwSyncIndicator learn the 'M' (modified - pending
re-upload, amber) and 'I' (excluded, muted) states so flagged rows no
longer render as "upload failed"/"not uploaded".
- Settings truthfulness: profile QRZ section relabeled "XML Lookup
Credentials" with a note that logbook sync uses per-station API keys, and
an amber warning when auto-sync is on but no station has a key; station
form API-key fields relabeled "QRZ.com Logbook API Key (required for
logbook sync)".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an end-user-visible sync activity trail and consolidates scheduled syncing so QRZ/LoTW runs (including failures) are discoverable via a unified /sync UI and a merged /api/sync/logs feed, while tightening cron authentication.
Changes:
- Introduces
sync_logs(migration + model) and a merged/api/sync/logsendpoint (including normalized LoTW log rows). - Adds a new
/sync“Sync Activity” page and updates indicators/settings copy to reflect true QRZ/LoTW states and required API-key configuration. - Replaces separate LoTW cron routes with a unified hourly
/api/cron/syncorchestrator and strictCRON_SECRETauth helpers.
Reviewed changes
Copilot reviewed 33 out of 35 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| vercel.json | Switches scheduled cron to hourly /api/cron/sync and updates function duration config. |
| src/models/SyncLog.ts | Adds SyncLog model for writing/querying sync_logs. |
| src/models/Station.ts | Adds QRZ confirmation cursor field + updater helper. |
| src/models/Contact.ts | Refines pending-QRZ selection, adds reupload flagging, removes phantom QRZ sync-status API. |
| src/lib/qrz.ts | Removes legacy username/password QRZ upload method (API-key path only). |
| src/lib/qrz-sync-service.ts | New unified QRZ upload/download service + sync log writer. |
| src/lib/qrz-auto-sync.ts | Reworks on-save auto-sync to use the service and avoid silent failures. |
| src/lib/lotw.ts | Removes unused LoTW ADIF generation helper. |
| src/lib/cron-auth.ts | Adds strict CRON_SECRET Bearer auth helper. |
| src/components/QRZSyncIndicator.tsx | Updates QRZ indicator to display M/I states truthfully. |
| src/components/Navbar.tsx | Adds “Sync Activity” link under “More”. |
| src/components/LotwSyncIndicator.tsx | Updates LoTW indicator to display M/I states truthfully. |
| src/app/sync/page.tsx | New UI page rendering merged sync activity feed. |
| src/app/stations/new/page.tsx | Clarifies QRZ key label/help text as “Logbook API Key”. |
| src/app/stations/[id]/edit/page.tsx | Clarifies QRZ key label/help text and distinguishes lookup vs logbook credentials. |
| src/app/profile/page.tsx | Clarifies QRZ lookup vs logbook keys; warns when auto-sync enabled but no station has a key. |
| src/app/api/sync/logs/route.ts | New merged feed endpoint for sync_logs + LoTW log tables. |
| src/app/api/lotw/upload/route.ts | Requires valid cron secret for cron-mode requests; prevents header-only bypass. |
| src/app/api/lotw/download/route.ts | Requires valid cron secret for cron-mode requests; prevents header-only bypass. |
| src/app/api/cron/sync/route.ts | New unified cron orchestrator for QRZ+LoTW upload/download cycle. |
| src/app/api/cron/lotw-upload/route.ts | Removes legacy LoTW upload cron route. |
| src/app/api/cron/lotw-download/route.ts | Removes legacy LoTW download cron route. |
| src/app/api/contacts/route.ts | Renames admin filter param and moves auto-sync to after(). |
| src/app/api/contacts/qrz-sync/route.ts | Uses QRZ sync service; writes summary sync log row for manual bulk sync. |
| src/app/api/contacts/qrz-download/route.ts | Uses QRZ sync service for confirmation download. |
| src/app/api/contacts/[id]/route.ts | Flags for reupload on core-field changes and runs auto-sync in after(). |
| src/app/api/contacts/[id]/qrz-sync/route.ts | Uses QRZ sync service for per-contact sync. |
| src/app/admin/page.tsx | Updates admin bulk sync query param to qrz_filter=not_sent. |
| README.md | Documents CRON_SECRET + scheduled /api/cron/sync usage for Vercel and self-hosted. |
| drizzle/schema.ts | Adds sync_logs table definition to canonical schema. |
| drizzle/migrations/meta/0004_snapshot.json | Updates Drizzle snapshot metadata for migration chain. |
| drizzle/migrations/meta/_journal.json | Adds new migration entries (but currently includes an invalid BOM). |
| drizzle/migrations/0005_add_sync_logs.sql | Creates sync_logs table + indexes + constraints. |
| drizzle/migrations/0004_drop_orphaned_qrz_sync_status_enum.sql | Drops orphaned QRZ enum type left from removed phantom schema. |
- Retry failed re-uploads with OPTION=REPLACE: infer a prior ship from qrz_qsl_sent_date so 'R' retries of modified QSOs don't get deduped as inserts and silently never update. - Push the station filter into SQL: findQrzSentNotConfirmed now takes stationId instead of loading all stations' rows and filtering in memory. - Flag re-upload on actual core-field changes: compare payload values against the stored contact instead of truthiness, so clearing a field triggers sync and no-op PUTs don't false-flag 'Y' rows to 'M'. - Reword qrz-auto-sync header: pre-flight skips only log, they don't mark the contact 'R'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The detailed missing-env-var response was reachable by unauthenticated callers, letting them probe deployment configuration. Fail closed on a missing CRON_SECRET (generic message), enforce the Bearer token, then run the remaining env checks. Addresses Copilot review feedback on #221. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Remove the UTF-8 BOM (and CRLF endings) from drizzle/migrations/meta/_journal.json so strict JSON parsers and the Drizzle migrator can read it. - /api/sync/logs now falls back to defaults on non-numeric or negative limit/offset instead of passing NaN into SQL and returning a 500. Addresses Copilot review feedback on #222. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # README.md # src/app/api/cron/lotw-download/route.ts # src/app/api/cron/lotw-upload/route.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The final piece of "never fail silently": every sync run — including failures — now leaves a visible trace.
sync_logstable (migration0005_add_sync_logs) +SyncLogmodel:service(qrznow,lotw/eqslreserved),direction,trigger(manual/auto/cron),status, counts,error_message,detailsjsonb. LoTW keeps its dedicatedlotw_*_logstables (they work and feed/lotw) — no destructive migration.details) all write rows. Hourly no-op sweeps are skipped so the feed stays readable. Logging failures never break the sync itself.GET /api/sync/logs: one reverse-chronological feed —sync_logsUNION ALL normalized LoTW upload/download log rows, with station callsigns joined in./syncpage ("Sync Activity", navbar → More): service/direction/station/trigger/status/started/result columns, full error messages inline, refresh button. Modeled on the/lotwSync History card.QRZSyncIndicatorandLotwSyncIndicatornow render'M'(amber, "Modified — pending re-upload") and'I'(muted, "Excluded from sync") instead of mislabeling those states as failed/not-uploaded.Verification
npm run lint/npm run typecheck/npm run buildclean; migration reviewed (single CREATE TABLE + FKs + indexes).npm run db:migrate(or/api/admin/migrate): log a QSO with a garbage station API key →/syncshows a failedautorow with the QRZ error; run a manual bulk sync → summary row appears; LoTW history rows appear merged.🤖 Generated with Claude Code