feat(sync): unified hourly sync cron for QRZ + LoTW#221
Merged
Conversation
…tus system QRZ auto-sync could never work. It authenticated with username/password against logbook.qrz.com, which only accepts pre-issued logbook API keys, and it recorded results in qrz_sync_status/qrz_sync_date/qrz_logbook_id/ qrz_sync_error columns that exist in no migration or schema — every status UPDATE threw, and backgroundAutoSync swallowed the error. Uploads silently did nothing. - New src/lib/qrz-sync-service.ts: single home for QRZ upload/download, used by auto-sync, the manual per-contact and bulk routes, and (next PR) the sync cron. All paths use the station qrz_api_key and record state in the ADIF-style qrz_qsl_sent/qrz_qsl_rcvd columns the UI reads. 'M' (modified) rows re-upload with OPTION=REPLACE; failures mark 'R' and are retried on every sweep. - Auto-sync now runs inside next/server after() — a bare fire-and-forget promise is frozen on Vercel when the response returns. - Contact edits flag already-uploaded copies 'Y' -> 'M' (flagForReupload) instead of writing the nonexistent qrz_sync_status column, which threw on every edit of core QSO fields. - QRZ downloads keep an incremental cursor in stations.qrz_last_qsl_rcvd_date (previously never written) via MODSINCE. - Delete uploadQSOToQRZ (username/password handshake), updateQrzSyncStatus, dead Contact.matchQSO, dead generateAdifForLoTW, and the phantom fields on ContactData; migration 0004 drops the orphaned qrz_sync_status_enum type. - Rename the misleading contacts query param qrz_sync_status=not_synced to qrz_filter=not_sent (it always filtered by qrz_qsl_sent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cron routes trusted any request whose host/user-agent/x-vercel-id header
mentioned vercel, which disabled the CRON_SECRET check on every *.vercel.app
deployment. Separately, /api/lotw/{upload,download} skipped auth entirely on
a spoofable X-Cron-Job: true header, letting anonymous callers trigger
uploads/downloads for any station_id.
- New src/lib/cron-auth.ts: strict Bearer CRON_SECRET check, no fallbacks.
- Cron routes require the Bearer token unconditionally and fail closed (500)
when CRON_SECRET is unset. Vercel attaches the header automatically when
the env var exists.
- X-Cron-Job is now only a mode discriminator: cron mode additionally
requires the valid Bearer token; the cron routes' internal fetches send it.
- README: "Scheduled sync" section for Vercel + self-hosted crontab setup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the two daily LoTW-only cron routes with a single /api/cron/sync
orchestrator that runs the full cycle per station, in order: QRZ upload
sweep, LoTW upload, QRZ confirmation download, LoTW confirmation download.
Uploads run before downloads so fresh QSOs can confirm in the same cycle,
and QRZ/LoTW legs are serialized so the cross-service 'M' re-upload flags
never race. QRZ finally gets automation — previously upload was on-save or
manual only and confirmation download was manual only.
- QRZ legs iterate active stations with an API key whose owner enabled
qrz_auto_sync, via uploadPendingForStation (250 QSOs/station/run cap,
'R' failures retried every sweep) and downloadConfirmationsForStation
(incremental MODSINCE cursor).
- LoTW legs keep the existing per-station selection, <1h dedupe, and
internal fetch to /api/lotw/{upload,download} with the CRON_SECRET Bearer
header, preserving lotw_*_logs bookkeeping.
- vercel.json: single hourly cron (Vercel Pro), maxDuration 300 for the new
route; old cron routes deleted; README updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR consolidates QRZ + LoTW synchronization into a single hourly cron orchestrator (/api/cron/sync), moving QRZ logic into a shared service layer and tightening cron authentication via a strict CRON_SECRET Bearer token.
Changes:
- Replaces the two LoTW-only daily cron routes with one hourly
/api/cron/syncroute (QRZ upload → LoTW upload → QRZ download → LoTW download), and updatesvercel.jsonaccordingly. - Introduces
src/lib/qrz-sync-service.tsas the single home for QRZ upload/download behavior and rewires the QRZ auto-sync + manual/bulk QRZ endpoints to use it. - Hardens cron-mode auth for LoTW upload/download endpoints via
hasValidCronSecret()and updates docs/migrations to remove legacy/phantom QRZ sync artifacts.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| vercel.json | Removes the two daily LoTW cron entries and adds a single hourly /api/cron/sync cron with a 300s duration limit. |
| src/models/Station.ts | Adds station cursor field support and a helper to advance the QRZ download cursor date. |
| src/models/Contact.ts | Removes legacy QRZ sync fields/APIs, expands “pending QRZ upload” selection, and adds a reupload-flagging helper. |
| src/lib/qrz.ts | Deletes the legacy username/password QRZ logbook upload implementation (API-key path remains). |
| src/lib/qrz-sync-service.ts | New QRZ sync service encapsulating single-contact upload, per-station sweeps, and incremental confirmation downloads. |
| src/lib/qrz-auto-sync.ts | Reworks on-save QRZ auto-sync to use the new service and improves logging/behavior for serverless execution. |
| src/lib/lotw.ts | Removes a now-unused LoTW ADIF generation helper. |
| src/lib/cron-auth.ts | Adds strict cron authentication helper (Authorization: Bearer CRON_SECRET only). |
| src/app/api/lotw/upload/route.ts | Requires CRON_SECRET when X-Cron-Job: true is present (prevents header-only spoofing). |
| src/app/api/lotw/download/route.ts | Same cron-mode auth hardening as LoTW upload. |
| src/app/api/cron/sync/route.ts | Adds the unified sync orchestrator route coordinating QRZ + LoTW upload/download legs. |
| src/app/api/cron/lotw-upload/route.ts | Deletes the legacy LoTW upload cron endpoint (replaced by /api/cron/sync). |
| src/app/api/cron/lotw-download/route.ts | Deletes the legacy LoTW download cron endpoint (replaced by /api/cron/sync). |
| src/app/api/contacts/route.ts | Switches QRZ filter query param naming and runs QRZ auto-sync via after() post-response. |
| src/app/api/contacts/qrz-sync/route.ts | Updates bulk QRZ sync to use qrz-sync-service and adds a station cache for efficiency. |
| src/app/api/contacts/qrz-download/route.ts | Refactors QRZ confirmation download to use qrz-sync-service. |
| src/app/api/contacts/[id]/route.ts | Flags contacts for reupload on core field updates and runs auto-sync via after(). |
| src/app/api/contacts/[id]/qrz-sync/route.ts | Updates per-contact QRZ sync to use station API key + qrz-sync-service. |
| src/app/admin/page.tsx | Updates admin bulk QRZ sync to use the renamed contacts filter query param. |
| README.md | Documents the new scheduled sync endpoint and CRON_SECRET usage (Vercel + self-hosted). |
| drizzle/migrations/meta/0004_snapshot.json | Adds updated schema snapshot including QRZ cursor/date and other migration state. |
| drizzle/migrations/meta/_journal.json | Registers the new migration in Drizzle’s journal. |
| drizzle/migrations/0004_drop_orphaned_qrz_sync_status_enum.sql | Drops the orphaned QRZ sync enum type tied to nonexistent legacy columns. |
- Retry failed re-uploads with OPTION=REPLACE: infer a prior ship from qrz_qsl_sent_date so 'R' retries of modified QSOs don't get deduped as inserts and silently never update. - Push the station filter into SQL: findQrzSentNotConfirmed now takes stationId instead of loading all stations' rows and filtering in memory. - Flag re-upload on actual core-field changes: compare payload values against the stored contact instead of truthiness, so clearing a field triggers sync and no-op PUTs don't false-flag 'Y' rows to 'M'. - Reword qrz-auto-sync header: pre-flight skips only log, they don't mark the contact 'R'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The detailed missing-env-var response was reachable by unauthenticated callers, letting them probe deployment configuration. Fail closed on a missing CRON_SECRET (generic message), enforce the Bearer token, then run the remaining env checks. Addresses Copilot review feedback on #221. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # README.md # src/app/api/cron/lotw-download/route.ts # src/app/api/cron/lotw-upload/route.ts
patrickrb
added a commit
that referenced
this pull request
Jul 13, 2026
…ings (#222) * fix(qrz): rewrite auto-sync onto the API-key path, delete phantom status system QRZ auto-sync could never work. It authenticated with username/password against logbook.qrz.com, which only accepts pre-issued logbook API keys, and it recorded results in qrz_sync_status/qrz_sync_date/qrz_logbook_id/ qrz_sync_error columns that exist in no migration or schema — every status UPDATE threw, and backgroundAutoSync swallowed the error. Uploads silently did nothing. - New src/lib/qrz-sync-service.ts: single home for QRZ upload/download, used by auto-sync, the manual per-contact and bulk routes, and (next PR) the sync cron. All paths use the station qrz_api_key and record state in the ADIF-style qrz_qsl_sent/qrz_qsl_rcvd columns the UI reads. 'M' (modified) rows re-upload with OPTION=REPLACE; failures mark 'R' and are retried on every sweep. - Auto-sync now runs inside next/server after() — a bare fire-and-forget promise is frozen on Vercel when the response returns. - Contact edits flag already-uploaded copies 'Y' -> 'M' (flagForReupload) instead of writing the nonexistent qrz_sync_status column, which threw on every edit of core QSO fields. - QRZ downloads keep an incremental cursor in stations.qrz_last_qsl_rcvd_date (previously never written) via MODSINCE. - Delete uploadQSOToQRZ (username/password handshake), updateQrzSyncStatus, dead Contact.matchQSO, dead generateAdifForLoTW, and the phantom fields on ContactData; migration 0004 drops the orphaned qrz_sync_status_enum type. - Rename the misleading contacts query param qrz_sync_status=not_synced to qrz_filter=not_sent (it always filtered by qrz_qsl_sent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(security): require CRON_SECRET on cron + LoTW sync endpoints The cron routes trusted any request whose host/user-agent/x-vercel-id header mentioned vercel, which disabled the CRON_SECRET check on every *.vercel.app deployment. Separately, /api/lotw/{upload,download} skipped auth entirely on a spoofable X-Cron-Job: true header, letting anonymous callers trigger uploads/downloads for any station_id. - New src/lib/cron-auth.ts: strict Bearer CRON_SECRET check, no fallbacks. - Cron routes require the Bearer token unconditionally and fail closed (500) when CRON_SECRET is unset. Vercel attaches the header automatically when the env var exists. - X-Cron-Job is now only a mode discriminator: cron mode additionally requires the valid Bearer token; the cron routes' internal fetches send it. - README: "Scheduled sync" section for Vercel + self-hosted crontab setup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sync): unified hourly sync cron for QRZ + LoTW Replace the two daily LoTW-only cron routes with a single /api/cron/sync orchestrator that runs the full cycle per station, in order: QRZ upload sweep, LoTW upload, QRZ confirmation download, LoTW confirmation download. Uploads run before downloads so fresh QSOs can confirm in the same cycle, and QRZ/LoTW legs are serialized so the cross-service 'M' re-upload flags never race. QRZ finally gets automation — previously upload was on-save or manual only and confirmation download was manual only. - QRZ legs iterate active stations with an API key whose owner enabled qrz_auto_sync, via uploadPendingForStation (250 QSOs/station/run cap, 'R' failures retried every sweep) and downloadConfirmationsForStation (incremental MODSINCE cursor). - LoTW legs keep the existing per-station selection, <1h dedupe, and internal fetch to /api/lotw/{upload,download} with the CRON_SECRET Bearer header, preserving lotw_*_logs bookkeeping. - vercel.json: single hourly cron (Vercel Pro), maxDuration 300 for the new route; old cron routes deleted; README updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sync): sync activity log, /sync page, truthful indicators + settings Sync failures were invisible: QRZ had no history at all, auto-sync failures vanished, and the settings UI implied username/password powered logbook sync. This adds the visibility layer: - New sync_logs table (migration 0005) + SyncLog model. QRZ upload sweeps, downloads, manual bulk syncs, and failed on-save auto-syncs all write rows (trigger manual/auto/cron); no-op cron sweeps are not logged so the feed stays readable. LoTW keeps its dedicated log tables. - GET /api/sync/logs: one reverse-chronological feed merging sync_logs with normalized lotw_upload_logs / lotw_download_logs rows. - New /sync page ("Sync Activity") listing every run with service, direction, station, trigger, status, and full error message; linked from the navbar More menu. - QRZSyncIndicator + LotwSyncIndicator learn the 'M' (modified - pending re-upload, amber) and 'I' (excluded, muted) states so flagged rows no longer render as "upload failed"/"not uploaded". - Settings truthfulness: profile QRZ section relabeled "XML Lookup Credentials" with a note that logbook sync uses per-station API keys, and an amber warning when auto-sync is on but no station has a key; station form API-key fields relabeled "QRZ.com Logbook API Key (required for logbook sync)". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(qrz): address Copilot review feedback on #219 - Retry failed re-uploads with OPTION=REPLACE: infer a prior ship from qrz_qsl_sent_date so 'R' retries of modified QSOs don't get deduped as inserts and silently never update. - Push the station filter into SQL: findQrzSentNotConfirmed now takes stationId instead of loading all stations' rows and filtering in memory. - Flag re-upload on actual core-field changes: compare payload values against the stored contact instead of truthiness, so clearing a field triggers sync and no-op PUTs don't false-flag 'Y' rows to 'M'. - Reword qrz-auto-sync header: pre-flight skips only log, they don't mark the contact 'R'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cron): authenticate before env validation in unified sync route The detailed missing-env-var response was reachable by unauthenticated callers, letting them probe deployment configuration. Fail closed on a missing CRON_SECRET (generic message), enforce the Bearer token, then run the remaining env checks. Addresses Copilot review feedback on #221. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sync): strip BOM from drizzle journal, guard NaN limit/offset - Remove the UTF-8 BOM (and CRLF endings) from drizzle/migrations/meta/_journal.json so strict JSON parsers and the Drizzle migrator can read it. - /api/sync/logs now falls back to defaults on non-numeric or negative limit/offset instead of passing NaN into SQL and returning a 500. Addresses Copilot review feedback on #222. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the two daily LoTW-only crons with a single
/api/cron/syncorchestrator that runs the full sync cycle for every eligible station, in order:NULL/'N'/'R'/'M') to each station's QRZ logbook, 250/station/run cap, failed rows retried every runlotw_upload_logsbookkeeping is unchangedMODSINCEcursor (stations.qrz_last_qsl_rcvd_date)qso_qslsincecursor added in fix(lotw): confirmations never applied - duplicate qsl_lotw_date assignment (42601) #218Uploads run before downloads so freshly logged QSOs can confirm in the same cycle; QRZ and LoTW legs are serialized so the cross-service
'M're-upload flags never race. Every leg is incremental and idempotent — any cadence is safe.This is the piece that makes QRZ sync automatic: until now QRZ upload ran only on-save or manually, and QRZ confirmation download was manual-only (no cron existed at all).
Config
vercel.json: one hourly cron (0 * * * *, Vercel Pro) replacing the two daily entries;maxDuration: 300for the new route./api/cron/lotw-{upload,download}routes deleted; README "Scheduled sync" updated (self-hosted: one crontab line hitting/api/cron/syncwith theCRON_SECRETBearer header).Verification
npm run lint/npm run typecheck/npm run buildclean.curl -H "Authorization: Bearer $CRON_SECRET" https://<host>/api/cron/syncreturns per-station results for all four legs; without the header it returns 401.🤖 Generated with Claude Code