Skip to content

feat(sync): unified hourly sync cron for QRZ + LoTW#221

Merged
patrickrb merged 8 commits into
mainfrom
feat/unified-sync-cron
Jul 13, 2026
Merged

feat(sync): unified hourly sync cron for QRZ + LoTW#221
patrickrb merged 8 commits into
mainfrom
feat/unified-sync-cron

Conversation

@patrickrb

Copy link
Copy Markdown
Owner

Stacked on #219 and #220 — merge those first; this diff will shrink to just the cron changes once they land.

What

Replaces the two daily LoTW-only crons with a single /api/cron/sync orchestrator that runs the full sync cycle for every eligible station, in order:

  1. QRZ upload sweep — pending QSOs (NULL/'N'/'R'/'M') to each station's QRZ logbook, 250/station/run cap, failed rows retried every run
  2. LoTW upload — existing per-station selection + <1h dedupe, invoked via internal fetch so lotw_upload_logs bookkeeping is unchanged
  3. QRZ confirmation download — incremental via the MODSINCE cursor (stations.qrz_last_qsl_rcvd_date)
  4. LoTW confirmation download — same internal-fetch pattern, now benefits from the qso_qslsince cursor added in fix(lotw): confirmations never applied - duplicate qsl_lotw_date assignment (42601) #218

Uploads run before downloads so freshly logged QSOs can confirm in the same cycle; QRZ and LoTW legs are serialized so the cross-service 'M' re-upload flags never race. Every leg is incremental and idempotent — any cadence is safe.

This is the piece that makes QRZ sync automatic: until now QRZ upload ran only on-save or manually, and QRZ confirmation download was manual-only (no cron existed at all).

Config

  • vercel.json: one hourly cron (0 * * * *, Vercel Pro) replacing the two daily entries; maxDuration: 300 for the new route.
  • Old /api/cron/lotw-{upload,download} routes deleted; README "Scheduled sync" updated (self-hosted: one crontab line hitting /api/cron/sync with the CRON_SECRET Bearer header).

Verification

🤖 Generated with Claude Code

patrickrb and others added 4 commits July 13, 2026 10:18
…tus system

QRZ auto-sync could never work. It authenticated with username/password
against logbook.qrz.com, which only accepts pre-issued logbook API keys, and
it recorded results in qrz_sync_status/qrz_sync_date/qrz_logbook_id/
qrz_sync_error columns that exist in no migration or schema — every status
UPDATE threw, and backgroundAutoSync swallowed the error. Uploads silently
did nothing.

- New src/lib/qrz-sync-service.ts: single home for QRZ upload/download,
  used by auto-sync, the manual per-contact and bulk routes, and (next PR)
  the sync cron. All paths use the station qrz_api_key and record state in
  the ADIF-style qrz_qsl_sent/qrz_qsl_rcvd columns the UI reads. 'M'
  (modified) rows re-upload with OPTION=REPLACE; failures mark 'R' and are
  retried on every sweep.
- Auto-sync now runs inside next/server after() — a bare fire-and-forget
  promise is frozen on Vercel when the response returns.
- Contact edits flag already-uploaded copies 'Y' -> 'M' (flagForReupload)
  instead of writing the nonexistent qrz_sync_status column, which threw on
  every edit of core QSO fields.
- QRZ downloads keep an incremental cursor in
  stations.qrz_last_qsl_rcvd_date (previously never written) via MODSINCE.
- Delete uploadQSOToQRZ (username/password handshake), updateQrzSyncStatus,
  dead Contact.matchQSO, dead generateAdifForLoTW, and the phantom fields on
  ContactData; migration 0004 drops the orphaned qrz_sync_status_enum type.
- Rename the misleading contacts query param qrz_sync_status=not_synced to
  qrz_filter=not_sent (it always filtered by qrz_qsl_sent).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cron routes trusted any request whose host/user-agent/x-vercel-id header
mentioned vercel, which disabled the CRON_SECRET check on every *.vercel.app
deployment. Separately, /api/lotw/{upload,download} skipped auth entirely on
a spoofable X-Cron-Job: true header, letting anonymous callers trigger
uploads/downloads for any station_id.

- New src/lib/cron-auth.ts: strict Bearer CRON_SECRET check, no fallbacks.
- Cron routes require the Bearer token unconditionally and fail closed (500)
  when CRON_SECRET is unset. Vercel attaches the header automatically when
  the env var exists.
- X-Cron-Job is now only a mode discriminator: cron mode additionally
  requires the valid Bearer token; the cron routes' internal fetches send it.
- README: "Scheduled sync" section for Vercel + self-hosted crontab setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the two daily LoTW-only cron routes with a single /api/cron/sync
orchestrator that runs the full cycle per station, in order: QRZ upload
sweep, LoTW upload, QRZ confirmation download, LoTW confirmation download.
Uploads run before downloads so fresh QSOs can confirm in the same cycle,
and QRZ/LoTW legs are serialized so the cross-service 'M' re-upload flags
never race. QRZ finally gets automation — previously upload was on-save or
manual only and confirmation download was manual only.

- QRZ legs iterate active stations with an API key whose owner enabled
  qrz_auto_sync, via uploadPendingForStation (250 QSOs/station/run cap,
  'R' failures retried every sweep) and downloadConfirmationsForStation
  (incremental MODSINCE cursor).
- LoTW legs keep the existing per-station selection, <1h dedupe, and
  internal fetch to /api/lotw/{upload,download} with the CRON_SECRET Bearer
  header, preserving lotw_*_logs bookkeeping.
- vercel.json: single hourly cron (Vercel Pro), maxDuration 300 for the new
  route; old cron routes deleted; README updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 13, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nodelog Ready Ready Preview, Comment Jul 13, 2026 5:43pm

Request Review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR consolidates QRZ + LoTW synchronization into a single hourly cron orchestrator (/api/cron/sync), moving QRZ logic into a shared service layer and tightening cron authentication via a strict CRON_SECRET Bearer token.

Changes:

  • Replaces the two LoTW-only daily cron routes with one hourly /api/cron/sync route (QRZ upload → LoTW upload → QRZ download → LoTW download), and updates vercel.json accordingly.
  • Introduces src/lib/qrz-sync-service.ts as the single home for QRZ upload/download behavior and rewires the QRZ auto-sync + manual/bulk QRZ endpoints to use it.
  • Hardens cron-mode auth for LoTW upload/download endpoints via hasValidCronSecret() and updates docs/migrations to remove legacy/phantom QRZ sync artifacts.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
vercel.json Removes the two daily LoTW cron entries and adds a single hourly /api/cron/sync cron with a 300s duration limit.
src/models/Station.ts Adds station cursor field support and a helper to advance the QRZ download cursor date.
src/models/Contact.ts Removes legacy QRZ sync fields/APIs, expands “pending QRZ upload” selection, and adds a reupload-flagging helper.
src/lib/qrz.ts Deletes the legacy username/password QRZ logbook upload implementation (API-key path remains).
src/lib/qrz-sync-service.ts New QRZ sync service encapsulating single-contact upload, per-station sweeps, and incremental confirmation downloads.
src/lib/qrz-auto-sync.ts Reworks on-save QRZ auto-sync to use the new service and improves logging/behavior for serverless execution.
src/lib/lotw.ts Removes a now-unused LoTW ADIF generation helper.
src/lib/cron-auth.ts Adds strict cron authentication helper (Authorization: Bearer CRON_SECRET only).
src/app/api/lotw/upload/route.ts Requires CRON_SECRET when X-Cron-Job: true is present (prevents header-only spoofing).
src/app/api/lotw/download/route.ts Same cron-mode auth hardening as LoTW upload.
src/app/api/cron/sync/route.ts Adds the unified sync orchestrator route coordinating QRZ + LoTW upload/download legs.
src/app/api/cron/lotw-upload/route.ts Deletes the legacy LoTW upload cron endpoint (replaced by /api/cron/sync).
src/app/api/cron/lotw-download/route.ts Deletes the legacy LoTW download cron endpoint (replaced by /api/cron/sync).
src/app/api/contacts/route.ts Switches QRZ filter query param naming and runs QRZ auto-sync via after() post-response.
src/app/api/contacts/qrz-sync/route.ts Updates bulk QRZ sync to use qrz-sync-service and adds a station cache for efficiency.
src/app/api/contacts/qrz-download/route.ts Refactors QRZ confirmation download to use qrz-sync-service.
src/app/api/contacts/[id]/route.ts Flags contacts for reupload on core field updates and runs auto-sync via after().
src/app/api/contacts/[id]/qrz-sync/route.ts Updates per-contact QRZ sync to use station API key + qrz-sync-service.
src/app/admin/page.tsx Updates admin bulk QRZ sync to use the renamed contacts filter query param.
README.md Documents the new scheduled sync endpoint and CRON_SECRET usage (Vercel + self-hosted).
drizzle/migrations/meta/0004_snapshot.json Adds updated schema snapshot including QRZ cursor/date and other migration state.
drizzle/migrations/meta/_journal.json Registers the new migration in Drizzle’s journal.
drizzle/migrations/0004_drop_orphaned_qrz_sync_status_enum.sql Drops the orphaned QRZ sync enum type tied to nonexistent legacy columns.

Comment thread src/app/api/cron/sync/route.ts Outdated
Comment thread src/app/api/contacts/[id]/route.ts Outdated
patrickrb and others added 3 commits July 13, 2026 12:13
- Retry failed re-uploads with OPTION=REPLACE: infer a prior ship from
  qrz_qsl_sent_date so 'R' retries of modified QSOs don't get deduped
  as inserts and silently never update.
- Push the station filter into SQL: findQrzSentNotConfirmed now takes
  stationId instead of loading all stations' rows and filtering in memory.
- Flag re-upload on actual core-field changes: compare payload values
  against the stored contact instead of truthiness, so clearing a field
  triggers sync and no-op PUTs don't false-flag 'Y' rows to 'M'.
- Reword qrz-auto-sync header: pre-flight skips only log, they don't
  mark the contact 'R'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The detailed missing-env-var response was reachable by unauthenticated
callers, letting them probe deployment configuration. Fail closed on a
missing CRON_SECRET (generic message), enforce the Bearer token, then
run the remaining env checks.

Addresses Copilot review feedback on #221.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts:
#	README.md
#	src/app/api/cron/lotw-download/route.ts
#	src/app/api/cron/lotw-upload/route.ts
@patrickrb
patrickrb merged commit d5de1fb into main Jul 13, 2026
7 checks passed
@patrickrb
patrickrb deleted the feat/unified-sync-cron branch July 13, 2026 20:37
patrickrb added a commit that referenced this pull request Jul 13, 2026
…ings (#222)

* fix(qrz): rewrite auto-sync onto the API-key path, delete phantom status system

QRZ auto-sync could never work. It authenticated with username/password
against logbook.qrz.com, which only accepts pre-issued logbook API keys, and
it recorded results in qrz_sync_status/qrz_sync_date/qrz_logbook_id/
qrz_sync_error columns that exist in no migration or schema — every status
UPDATE threw, and backgroundAutoSync swallowed the error. Uploads silently
did nothing.

- New src/lib/qrz-sync-service.ts: single home for QRZ upload/download,
  used by auto-sync, the manual per-contact and bulk routes, and (next PR)
  the sync cron. All paths use the station qrz_api_key and record state in
  the ADIF-style qrz_qsl_sent/qrz_qsl_rcvd columns the UI reads. 'M'
  (modified) rows re-upload with OPTION=REPLACE; failures mark 'R' and are
  retried on every sweep.
- Auto-sync now runs inside next/server after() — a bare fire-and-forget
  promise is frozen on Vercel when the response returns.
- Contact edits flag already-uploaded copies 'Y' -> 'M' (flagForReupload)
  instead of writing the nonexistent qrz_sync_status column, which threw on
  every edit of core QSO fields.
- QRZ downloads keep an incremental cursor in
  stations.qrz_last_qsl_rcvd_date (previously never written) via MODSINCE.
- Delete uploadQSOToQRZ (username/password handshake), updateQrzSyncStatus,
  dead Contact.matchQSO, dead generateAdifForLoTW, and the phantom fields on
  ContactData; migration 0004 drops the orphaned qrz_sync_status_enum type.
- Rename the misleading contacts query param qrz_sync_status=not_synced to
  qrz_filter=not_sent (it always filtered by qrz_qsl_sent).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): require CRON_SECRET on cron + LoTW sync endpoints

The cron routes trusted any request whose host/user-agent/x-vercel-id header
mentioned vercel, which disabled the CRON_SECRET check on every *.vercel.app
deployment. Separately, /api/lotw/{upload,download} skipped auth entirely on
a spoofable X-Cron-Job: true header, letting anonymous callers trigger
uploads/downloads for any station_id.

- New src/lib/cron-auth.ts: strict Bearer CRON_SECRET check, no fallbacks.
- Cron routes require the Bearer token unconditionally and fail closed (500)
  when CRON_SECRET is unset. Vercel attaches the header automatically when
  the env var exists.
- X-Cron-Job is now only a mode discriminator: cron mode additionally
  requires the valid Bearer token; the cron routes' internal fetches send it.
- README: "Scheduled sync" section for Vercel + self-hosted crontab setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(sync): unified hourly sync cron for QRZ + LoTW

Replace the two daily LoTW-only cron routes with a single /api/cron/sync
orchestrator that runs the full cycle per station, in order: QRZ upload
sweep, LoTW upload, QRZ confirmation download, LoTW confirmation download.
Uploads run before downloads so fresh QSOs can confirm in the same cycle,
and QRZ/LoTW legs are serialized so the cross-service 'M' re-upload flags
never race. QRZ finally gets automation — previously upload was on-save or
manual only and confirmation download was manual only.

- QRZ legs iterate active stations with an API key whose owner enabled
  qrz_auto_sync, via uploadPendingForStation (250 QSOs/station/run cap,
  'R' failures retried every sweep) and downloadConfirmationsForStation
  (incremental MODSINCE cursor).
- LoTW legs keep the existing per-station selection, <1h dedupe, and
  internal fetch to /api/lotw/{upload,download} with the CRON_SECRET Bearer
  header, preserving lotw_*_logs bookkeeping.
- vercel.json: single hourly cron (Vercel Pro), maxDuration 300 for the new
  route; old cron routes deleted; README updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(sync): sync activity log, /sync page, truthful indicators + settings

Sync failures were invisible: QRZ had no history at all, auto-sync failures
vanished, and the settings UI implied username/password powered logbook
sync. This adds the visibility layer:

- New sync_logs table (migration 0005) + SyncLog model. QRZ upload sweeps,
  downloads, manual bulk syncs, and failed on-save auto-syncs all write
  rows (trigger manual/auto/cron); no-op cron sweeps are not logged so the
  feed stays readable. LoTW keeps its dedicated log tables.
- GET /api/sync/logs: one reverse-chronological feed merging sync_logs with
  normalized lotw_upload_logs / lotw_download_logs rows.
- New /sync page ("Sync Activity") listing every run with service,
  direction, station, trigger, status, and full error message; linked from
  the navbar More menu.
- QRZSyncIndicator + LotwSyncIndicator learn the 'M' (modified - pending
  re-upload, amber) and 'I' (excluded, muted) states so flagged rows no
  longer render as "upload failed"/"not uploaded".
- Settings truthfulness: profile QRZ section relabeled "XML Lookup
  Credentials" with a note that logbook sync uses per-station API keys, and
  an amber warning when auto-sync is on but no station has a key; station
  form API-key fields relabeled "QRZ.com Logbook API Key (required for
  logbook sync)".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(qrz): address Copilot review feedback on #219

- Retry failed re-uploads with OPTION=REPLACE: infer a prior ship from
  qrz_qsl_sent_date so 'R' retries of modified QSOs don't get deduped
  as inserts and silently never update.
- Push the station filter into SQL: findQrzSentNotConfirmed now takes
  stationId instead of loading all stations' rows and filtering in memory.
- Flag re-upload on actual core-field changes: compare payload values
  against the stored contact instead of truthiness, so clearing a field
  triggers sync and no-op PUTs don't false-flag 'Y' rows to 'M'.
- Reword qrz-auto-sync header: pre-flight skips only log, they don't
  mark the contact 'R'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cron): authenticate before env validation in unified sync route

The detailed missing-env-var response was reachable by unauthenticated
callers, letting them probe deployment configuration. Fail closed on a
missing CRON_SECRET (generic message), enforce the Bearer token, then
run the remaining env checks.

Addresses Copilot review feedback on #221.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sync): strip BOM from drizzle journal, guard NaN limit/offset

- Remove the UTF-8 BOM (and CRLF endings) from
  drizzle/migrations/meta/_journal.json so strict JSON parsers and the
  Drizzle migrator can read it.
- /api/sync/logs now falls back to defaults on non-numeric or negative
  limit/offset instead of passing NaN into SQL and returning a 500.

Addresses Copilot review feedback on #222.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants