From c38bf69c7df947019940ea53640234b373f13611 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 17:57:07 +0200 Subject: [PATCH 01/11] Report what a crawl changed against the previous mirror (--changes) --update already knows which resources were new, which changed and which the server called unchanged, and throws it away: the flags reach file_notify() and go no further than a log line, while deletions exist only as a side effect of purging. --changes (-%d) keeps all of it and writes hts-changes.json plus a one-line summary in the log. "Changed" means the bytes differ, not that the server re-sent the resource. Comparing the mirrored files directly would not work: HTTrack stamps every parsed page with the crawl date via the footer, so those bytes differ on every run. Payloads are compared instead, the previous one coming from the cache for parsed pages and from the local copy sampled just before it is overwritten for everything else. The mirror-relative path, not the URL, is the accumulator's key, so a redirect and its target that share a save name are one entry; and only the first notify for a file samples its pre-run state, so a retried transfer is not counted twice. What counts as already mirrored comes from the previous run's file index rather than from the file's presence on disk: a partial left by this crawl's own failed attempt is on disk but was never part of the previous mirror. The deleted set is now computed whether or not purging is enabled; unlinking still happens only under --purge-old. Closes #714 Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- html/changes.html | 238 +++++++++++++++ html/cmdguide.html | 12 +- html/dev.html | 3 + html/httrack.man.html | 34 ++- html/server/option9.html | 5 + html/server/step2.html | 1 + html/server/step4.html | 2 + lang.def | 4 + lang/English.txt | 4 + lang/Francais.txt | 4 + man/httrack.1 | 5 +- src/Makefile.am | 4 +- src/htsalias.c | 2 + src/htsback.c | 8 + src/htschanges.c | 571 +++++++++++++++++++++++++++++++++++ src/htschanges.h | 126 ++++++++ src/htscore.c | 60 ++-- src/htscore.h | 7 + src/htscoremain.c | 8 + src/htshelp.c | 2 + src/htslib.c | 23 ++ src/htslib.h | 4 + src/htsopt.h | 4 + src/htsselftest.c | 66 ++++ src/htswarc.c | 22 +- src/libhttrack.vcxproj | 1 + tests/01_engine-changes.test | 7 + tests/88_local-changes.test | 146 +++++++++ tests/Makefile.am | 4 +- tests/local-server.py | 86 ++++++ tests/webhttrack-smoke.sh | 6 +- 31 files changed, 1413 insertions(+), 56 deletions(-) create mode 100644 html/changes.html create mode 100644 src/htschanges.c create mode 100644 src/htschanges.h create mode 100644 tests/01_engine-changes.test create mode 100644 tests/88_local-changes.test diff --git a/html/changes.html b/html/changes.html new file mode 100644 index 000000000..348089daf --- /dev/null +++ b/html/changes.html @@ -0,0 +1,238 @@ + + + + + + + HTTrack Website Copier - Change report format specification + + + + + + + + + +
HTTrack Website Copier
+ + + + +
Open Source offline browser
+ + + + +
+ + + + +
+ + + + +
+ + +

Change report format specification

+ +
+ +Run with --changes (-%d), HTTrack writes hts-changes.json +in the project directory, next to hts-log.txt, describing what the crawl +left new, changed, unchanged and gone compared to the previous mirror. The file is +rewritten from scratch at the end of every run, and the log carries a one-line +summary of the same counts. + +

+ +

What "changed" means

+ +A resource is changed when its bytes differ, not when the server merely re-sent +it. HTTrack compares the payload it just received against the copy the previous +run left behind: for pages it parses, the previous payload comes from the cache +(the file on disk carries the mirror footer and its crawl date, so its bytes +differ on every run); for everything else, the mirrored file is the payload +verbatim and is compared directly. + +

+ +Where no digest can be taken on either side — the cache is disabled, or the +previous copy is gone — the report falls back to the transfer signal, and a +server that answers 200 rather than 304 reads as changed. Keeping the cache on +(the default) is what makes the report precise. + +

+ +

Fields

+ +
    +
  • schema — format version, currently 1. It is bumped only +on an incompatible change; new fields may appear without one.
    • +
  • generator — the HTTrack build that wrote the file.
    • +
  • date — when the report was written, UTC, YYYY-MM-DDThh:mm:ssZ.
    • +
  • first_crawl — true when no index of a previous mirror +(hts-cache/old.lst) was found, so there was nothing to compare against and +everything is listed as new.
    • +
  • partial — true when the report ran out of memory and lists only +part of the mirror.
    • +
  • purged — true when --purge-old was in effect, so the +files under gone were also deleted from disk.
    • +
  • counts — the size of each of the four lists.
    • +
  • new, changed, unchanged, gone — the +lists themselves. Every mirrored file appears in exactly one of them.
    • +
+ +Each entry is an object: + +
    +
  • url — the absolute URL the file came from. Empty under +gone: deletions are computed from the mirror's file index, which records +paths, not URLs.
    • +
  • file — the path relative to the mirror root, with forward +slashes. This is the entry's identity: a URL and a redirect that resolve to the +same local file are one entry, not two.
    • +
  • size — the mirrored file's size in bytes, absent when the file +is not on disk.
    • +
  • previous_size — under changed only, the size of the +copy the previous run left.
    • +
+ +
+ +

Encoding

+ +The file is JSON, UTF-8. URLs and local paths reach HTTrack as raw bytes and are +not guaranteed to be valid UTF-8; any byte sequence that is not becomes +U+FFFD (\ufffd), so the file always parses. Compare on file +rather than on url when a mirror is known to carry legacy-charset URLs. + +

+ +

Example

+ +
+{
+  "schema": 1,
+  "generator": "HTTrack Website Copier/3.49-14",
+  "date": "2026-07-26T15:29:03Z",
+  "first_crawl": false,
+  "partial": false,
+  "purged": true,
+  "counts": { "new": 1, "changed": 1, "unchanged": 1, "gone": 1 },
+  "new": [
+    { "url": "http://example.com/d.html", "file": "example.com/d.html", "size": 280 }
+  ],
+  "changed": [
+    { "url": "http://example.com/a.html", "file": "example.com/a.html", "size": 281, "previous_size": 273 }
+  ],
+  "unchanged": [
+    { "url": "http://example.com/b.html", "file": "example.com/b.html", "size": 277 }
+  ],
+  "gone": [
+    { "url": "", "file": "example.com/c.html" }
+  ]
+}
+
+ +

+ +

Notes

+ +
    +
  • A file listed under gone is only deleted when --purge-old is +on. Left in place it drops out of the mirror's index, so it is reported once and +not again.
    • +
  • Content diffs, and keeping the previous copy of a changed page, are out of +scope: both change what a mirror directory contains.
    • +
+ +

+ + +
+
+
+ + + + + +
+ + + + + + diff --git a/html/cmdguide.html b/html/cmdguide.html index 0fe2b02d2..c34b9d133 100644 --- a/html/cmdguide.html +++ b/html/cmdguide.html @@ -482,6 +482,15 @@

Save a WARC archive of the crawl

archive, index and pages into one WACZ for replay tools such as replayweb.page.

+

See what a re-crawl changed

+

httrack https://example.com/ --update --changes --path mydir
+Writes hts-changes.json in the project folder, listing every +mirrored file as new, changed, unchanged or gone, plus a one-line summary in the +log. "Changed" means the bytes really differ: a server that answers 200 +with the same content it served last time lands in unchanged. Deletions +are reported whether or not --purge-old is deleting them. The format is +documented in the change report specification.

+

HTTrack as a fetch tool

httrack --get https://host/file.bin --path tmp
--get fetches one file with cache, index, depth, cookies and robots @@ -490,7 +499,8 @@

HTTrack as a fetch tool


For the complete, always-current option list, see the manual page. For the filter language, see filters; for the cache and updates, see -cache.

+cache; for the change report, see +changes.

diff --git a/html/dev.html b/html/dev.html index 877d670aa..2ca5e13d9 100644 --- a/html/dev.html +++ b/html/dev.html @@ -129,6 +129,9 @@

Technical references

  • Cache format

    • HTTrack stores original HTML data and references to downloaded files in a cache, located in the hts-cache directory. This page describes the HTTrack cache format. +
  • Change report format

    • +With --changes, HTTrack writes hts-changes.json describing what the crawl left new, changed, unchanged and gone +compared to the previous mirror. This page describes that file. diff --git a/html/httrack.man.html b/html/httrack.man.html index 4e82953de..2de271310 100644 --- a/html/httrack.man.html +++ b/html/httrack.man.html @@ -108,16 +108,17 @@

    SYNOPSIS --footer ] [ -%l, --language ] [ -%a, --accept ] [ -%X, --headers ] [ -C, --cache[=N] ] [ -k, --store-all-in-cache ] [ --%r, --warc ] [ -%n, --do-not-recatch ] [ --%v, --display ] [ -Q, --do-not-log ] [ -q, ---quiet ] [ -z, --extra-log ] [ -Z, ---debug-log ] [ -v, --verbose ] [ -f, ---file-log ] [ -f2, --single-log ] [ -I, ---index ] [ -%i, --build-top-index ] [ -%I, ---search-index ] [ -pN, --priority[=N] ] [ -S, ---stay-on-same-dir ] [ -D, --can-go-down ] [ --U, --can-go-up ] [ -B, --can-go-up-and-down ] -[ -a, --stay-on-same-address ] [ -d, +-%r, --warc ] [ -%d, --changes ] [ -%n, +--do-not-recatch ] [ -%v, --display ] [ -Q, +--do-not-log ] [ -q, --quiet ] [ -z, +--extra-log ] [ -Z, --debug-log ] [ -v, +--verbose ] [ -f, --file-log ] [ -f2, +--single-log ] [ -I, --index ] [ -%i, +--build-top-index ] [ -%I, --search-index ] [ +-pN, --priority[=N] ] [ -S, --stay-on-same-dir +] [ -D, --can-go-down ] [ -U, --can-go-up ] [ +-B, --can-go-up-and-down ] [ -a, +--stay-on-same-address ] [ -d, --stay-on-same-domain ] [ -l, --stay-on-same-tld ] [ -e, --go-everywhere ] [ -%H, --debug-headers ] [ -%!, @@ -1143,6 +1144,19 @@

    Log, index, cache +

    -%d

    + + + + +

    write hts-changes.json listing what this crawl left new, +changed, unchanged and gone compared to the previous mirror +(--changes)

    + + + + +

    -%n

    diff --git a/html/server/option9.html b/html/server/option9.html index e6570398e..ae22fe61a 100644 --- a/html/server/option9.html +++ b/html/server/option9.html @@ -114,6 +114,11 @@

    ${LANG_O2} - ${LANG_IOPT9}

    >

    + ${LANG_CHANGES} +

    + ${LANG_I34b} diff --git a/html/server/step2.html b/html/server/step2.html index a01a1876e..ed01c616e 100644 --- a/html/server/step2.html +++ b/html/server/step2.html @@ -143,6 +143,7 @@

    ${fexist:index.html:LANG_G42}

    ${do:copy:StoreAllInCache:cache2} ${do:copy:Warc:warc} ${do:copy:WarcFile:warcfile} +${do:copy:Changes:changes} ${do:copy:LogType:logtype} ${do:copy:UseHTTPProxyForFTP:ftpprox} ${do:copy:ProxyType:proxytype} diff --git a/html/server/step4.html b/html/server/step4.html index e2d8da6c1..f69f3d3cf 100644 --- a/html/server/step4.html +++ b/html/server/step4.html @@ -190,6 +190,7 @@

    ${LANG_J9}

    ${test:cache2:--store-all-in-cache} ${test:warc:--warc} ${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"} + ${test:changes:--changes} ${test:norecatch:--do-not-recatch} ${test:logf:--single-log} ${test:logtype:::--extra-log:--debug-log} @@ -242,6 +243,7 @@

    ${LANG_J9}

    StoreAllInCache=${ztest:cache2:0:1} Warc=${ztest:warc:0:1} WarcFile=${warcfile} +Changes=${ztest:changes:0:1} LogType=${logtype} UseHTTPProxyForFTP=${ztest:ftpprox:0:1} ProxyType=${proxytype} diff --git a/lang.def b/lang.def index f11c1e279..169594a7d 100755 --- a/lang.def +++ b/lang.def @@ -1042,3 +1042,7 @@ LANG_WARCFILE WARC archive name: LANG_WARCFILETIP Optional base name for the WARC archive; leave blank to auto-name it under the output directory. +LANG_CHANGES +Report what changed since the previous mirror +LANG_CHANGESTIP +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. diff --git a/lang/English.txt b/lang/English.txt index 86c402be4..797cbcb44 100644 --- a/lang/English.txt +++ b/lang/English.txt @@ -1012,3 +1012,7 @@ WARC archive name: WARC archive name: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Optional base name for the WARC archive; leave blank to auto-name it under the output directory. +Report what changed since the previous mirror +Report what changed since the previous mirror +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. diff --git a/lang/Francais.txt b/lang/Francais.txt index 74131f840..1a1e4a0ef 100644 --- a/lang/Francais.txt +++ b/lang/Francais.txt @@ -1012,3 +1012,7 @@ WARC archive name: Nom de l'archive WARC : Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Nom de base optionnel pour l'archive WARC ; laissez vide pour le générer automatiquement dans le répertoire de sortie. +Report what changed since the previous mirror +Signaler ce qui a changé depuis le miroir précédent +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Écrire aussi hts-changes.json, qui liste ce que ce crawl laisse nouveau, modifié, inchangé ou disparu par rapport au miroir précédent. diff --git a/man/httrack.1 b/man/httrack.1 index 86cfc84ac..ac3e6bd7f 100644 --- a/man/httrack.1 +++ b/man/httrack.1 @@ -3,7 +3,7 @@ .\" .\" This file is generated by man/makeman.sh; do not edit by hand. .\" SPDX-License-Identifier: GPL-3.0-or-later -.TH httrack 1 "23 July 2026" "httrack website copier" +.TH httrack 1 "26 July 2026" "httrack website copier" .SH NAME httrack \- offline browser : copy websites to a local directory .SH SYNOPSIS @@ -75,6 +75,7 @@ httrack \- offline browser : copy websites to a local directory [ \fB\-C, \-\-cache[=N]\fR ] [ \fB\-k, \-\-store\-all\-in\-cache\fR ] [ \fB\-%r, \-\-warc\fR ] +[ \fB\-%d, \-\-changes\fR ] [ \fB\-%n, \-\-do\-not\-recatch\fR ] [ \fB\-%v, \-\-display\fR ] [ \fB\-Q, \-\-do\-not\-log\fR ] @@ -279,6 +280,8 @@ create/use a cache for updates and retries (C0 no cache,C1 cache is prioritary,* store all files in cache (not useful if files on disk) (\-\-store\-all\-in\-cache) .IP \-%r write an ISO\-28500 WARC/1.1 archive; \-\-warc\-file NAME sets the output name, \-\-warc\-max\-size N rotates segments past N bytes, \-\-warc\-cdx also writes a sorted CDXJ index, \-\-wacz packages it all as a WACZ file (\-\-warc) +.IP \-%d +write hts\-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror (\-\-changes) .IP \-%n do not re\-download locally erased files (\-\-do\-not\-recatch) .IP \-%v diff --git a/src/Makefile.am b/src/Makefile.am index da3f1e4ba..e8503e047 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -66,7 +66,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \ htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \ htsname.c htsrobots.c htstools.c htswizard.c \ htsalias.c htsthread.c htsindex.c htsbauth.c \ - htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \ + htsmd5.c htscodec.c htswarc.c htschanges.c htsproxy.c htszlib.c htswrap.c htsconcat.c \ htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \ md5.c \ minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \ @@ -77,7 +77,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \ htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \ htsmodules.h htsname.h htsnet.h htssniff.h \ htsopt.h htsrobots.h htsthread.h \ - htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h \ + htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htschanges.h htsproxy.h htszlib.h \ htsstrings.h htsarrays.h httrack-library.h \ htscharset.h punycode.h htsencoding.h \ htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \ diff --git a/src/htsalias.c b/src/htsalias.c index 8d127a745..f780c14c8 100644 --- a/src/htsalias.c +++ b/src/htsalias.c @@ -114,6 +114,8 @@ const char *hts_optalias[][4] = { "strip [host/pattern=]key1,key2,... from URLs"}, {"cookies-file", "-%K", "param1", "load extra cookies from a Netscape cookies.txt"}, + {"changes", "-%d", "single", + "write hts-changes.json: what this crawl changed vs. the previous mirror"}, {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"}, {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"}, {"warc-max-size", "-%rs", "param1", diff --git a/src/htsback.c b/src/htsback.c index 566c388e0..0643d6dab 100644 --- a/src/htsback.c +++ b/src/htsback.c @@ -38,6 +38,7 @@ Please visit our Website: http://www.httrack.com #include "htsnet.h" #include "htscore.h" #include "htswarc.h" +#include "htschanges.h" #include "htsthread.h" #include /* END specific definitions */ @@ -712,6 +713,13 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback, if ((size = hts_codec_unpack(codec, back[p].tmpfile, unpacked)) >= 0) { back[p].r.size = back[p].r.totalsize = size; + if (back[p].r.is_write) { + /* Sample the previous copy now: the rename below replaces + it, and file_notify() only fires once it is gone. */ + hts_changes_notify( + opt, back[p].url_adr, back[p].url_fil, back[p].url_sav, + HTS_TRUE, back[p].r.notmodified ? HTS_TRUE : HTS_FALSE); + } if (!back[p].r.is_write) { // fichier -> mémoire ; le fichier est écrit plus tard deleteaddr(&back[p].r); diff --git a/src/htschanges.c b/src/htschanges.c new file mode 100644 index 000000000..726a29a84 --- /dev/null +++ b/src/htschanges.c @@ -0,0 +1,571 @@ +/* ------------------------------------------------------------ */ +/* +HTTrack Website Copier, Offline Browser for Windows and Unix +Copyright (C) 2026 Xavier Roche and other contributors + +SPDX-License-Identifier: GPL-3.0-or-later + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . + +Ethical use: we kindly ask that you NOT use this software to harvest email +addresses or to collect any other private information about people. Doing so +would dishonor our work and waste the many hours we have spent on it. + +Please visit our Website: http://www.httrack.com +*/ + +/* ------------------------------------------------------------ */ +/* File: htschanges.c subroutines: */ +/* --changes: what this crawl changed vs. the previous */ +/* mirror (hts-changes.json) */ +/* Author: Xavier Roche */ +/* ------------------------------------------------------------ */ + +#define HTS_INTERNAL_BYTECODE + +#include "htschanges.h" + +#include "htscache.h" +#include "htscharset.h" +#include "htscore.h" +#include "htslib.h" +#include "htsmd5.h" +#include "htssafe.h" +#include "htstools.h" +#include "coucal/coucal.h" +#include "md5.h" + +#include +#include +#include + +#define DIGEST_SIZE 16 + +typedef struct hts_changes hts_changes; + +/* One mirrored file. The mirror-relative path is the key, not the URL: a + redirect and its target can share a save name, and counting that file twice + would put it in two buckets. */ +typedef struct { + char *url; /* absolute URL, "" for an engine-generated file */ + char *file; /* mirror-relative path, as listed in new.lst */ + hts_boolean rewritten; /* the crawl wrote over the local copy */ + hts_boolean not_updated; /* transfer signal: the server reported no change */ + hts_boolean existed; /* the file was part of the previous mirror */ + hts_boolean on_disk; /* a copy was on disk when the crawl first saw it */ + hts_boolean listed_prev; /* the previous mirror's index listed this file */ + hts_boolean has_prev; /* prev_digest holds the previous payload's digest */ + hts_boolean has_new; /* new_digest was taken from the payload, not disk */ + unsigned char prev_digest[DIGEST_SIZE]; + unsigned char new_digest[DIGEST_SIZE]; + LLint prev_size; /* -1 when unknown */ + LLint size; /* size after the crawl, -1 when unknown */ + hts_change_bucket bucket; +} changes_entry; + +struct hts_changes { + coucal index; /* mirror-relative path -> entry slot + 1 */ + changes_entry *entries; + size_t count; + size_t capacity; + hts_boolean old_index; /* the previous mirror's index was read */ + hts_boolean overflow; /* an allocation failed; the report is partial */ +}; + +/* ------------------------------------------------------------ */ +/* JSON */ +/* ------------------------------------------------------------ */ + +/* Length of the UTF-8 sequence led by c, or 0 if c cannot lead one. */ +static size_t utf8_lead_length(unsigned char c) { + if (c >= 0xc2 && c <= 0xdf) + return 2; + if (c >= 0xe0 && c <= 0xef) + return 3; + if (c >= 0xf0 && c <= 0xf4) + return 4; + return 0; +} + +void hts_changes_json_string(String *out, const char *s) { + const unsigned char *p = (const unsigned char *) s; + + StringAddchar(*out, '"'); + while (*p != '\0') { + const unsigned char c = *p; + + if (c == '"' || c == '\\') { + StringAddchar(*out, '\\'); + StringAddchar(*out, (char) c); + p++; + } else if (c < 0x20 || c == 0x7f) { + char esc[8]; + + snprintf(esc, sizeof(esc), "\\u%04x", (unsigned) c); + StringCat(*out, esc); + p++; + } else if (c < 0x80) { + StringAddchar(*out, (char) c); + p++; + } else { + /* Non-ASCII: emit it verbatim only if it is a well-formed sequence, so + a legacy-charset URL cannot produce unparseable JSON. */ + const size_t len = utf8_lead_length(c); + + if (len != 0 && strnlen((const char *) p, len) == len && + hts_isStringUTF8((const char *) p, len)) { + StringMemcat(*out, (const char *) p, len); + p += len; + } else { + StringCat(*out, "\\ufffd"); + p++; + } + } + } + StringAddchar(*out, '"'); +} + +/* ------------------------------------------------------------ */ +/* Accumulator */ +/* ------------------------------------------------------------ */ + +static hts_changes *changes_new(void) { + hts_changes *changes = calloct(1, sizeof(*changes)); + + if (changes == NULL) + return NULL; + changes->index = coucal_new(0); + if (changes->index == NULL) { + freet(changes); + return NULL; + } + return changes; +} + +static void changes_free(hts_changes **pchanges) { + hts_changes *changes = *pchanges; + + if (changes == NULL) + return; + if (changes->entries != NULL) { + size_t i; + + for (i = 0; i < changes->count; i++) { + freet(changes->entries[i].url); + freet(changes->entries[i].file); + } + freet(changes->entries); + } + if (changes->index != NULL) + coucal_delete(&changes->index); + freet(changes); + *pchanges = NULL; +} + +/* Slot of `file`, or -1 when it has not been recorded. */ +static intptr_t changes_find(const hts_changes *changes, const char *file) { + intptr_t slot = 0; + + return coucal_read(changes->index, file, &slot) ? slot - 1 : -1; +} + +/* Slot of `file`, appending a fresh entry when it is new; -1 if allocation + failed, which flags the report partial. */ +static intptr_t changes_slot(hts_changes *changes, const char *file, + hts_boolean *is_new) { + intptr_t slot = 0; + + *is_new = HTS_FALSE; + if (coucal_read(changes->index, file, &slot)) + return slot - 1; + + if (changes->count == changes->capacity) { + const size_t capacity = changes->capacity != 0 ? changes->capacity * 2 : 64; + changes_entry *const entries = + realloct(changes->entries, capacity * sizeof(*entries)); + + if (entries == NULL) { + changes->overflow = HTS_TRUE; + return -1; + } + changes->entries = entries; + changes->capacity = capacity; + } + slot = (intptr_t) changes->count; + memset(&changes->entries[slot], 0, sizeof(changes->entries[slot])); + changes->entries[slot].file = strdupt(file); + changes->entries[slot].prev_size = -1; + changes->entries[slot].size = -1; + if (changes->entries[slot].file == NULL) { + changes->overflow = HTS_TRUE; + return -1; + } + changes->count++; + coucal_write(changes->index, file, slot + 1); + *is_new = HTS_TRUE; + return slot; +} + +/* MD5 of the file at `path`; change detection over our own mirror, not a + security boundary. HTS_FALSE if it cannot be read. */ +static hts_boolean digest_file(const char *path, + unsigned char digest[DIGEST_SIZE]) { + const int endian = 1; + struct MD5Context ctx; + char BIGSTK buffer[32768]; + FILE *fp = FOPEN(path, "rb"); + size_t nread; + + if (fp == NULL) + return HTS_FALSE; + MD5Init(&ctx, *((const char *) &endian)); + while ((nread = fread(buffer, 1, sizeof(buffer), fp)) > 0) + MD5Update(&ctx, (const unsigned char *) buffer, (unsigned int) nread); + if (ferror(fp) != 0) { + fclose(fp); + return HTS_FALSE; + } + fclose(fp); + MD5Final(digest, &ctx); + return HTS_TRUE; +} + +static void digest_mem(const char *buffer, size_t len, + unsigned char digest[DIGEST_SIZE]) { + domd5mem(buffer, len, (char *) digest, 0); +} + +hts_change_bucket hts_changes_classify(hts_boolean rewritten, + hts_boolean existed, + hts_boolean not_updated, + hts_boolean have_digests, + hts_boolean digests_equal) { + if (!rewritten) + return HTS_CHANGE_UNCHANGED; /* the crawl left the copy alone */ + if (!existed) + return HTS_CHANGE_NEW; + if (have_digests) + return digests_equal ? HTS_CHANGE_UNCHANGED : HTS_CHANGE_CHANGED; + /* No digest to compare: a server with no validators answers 200 with the + same bytes, so this signal alone over-reports. */ + return not_updated ? HTS_CHANGE_UNCHANGED : HTS_CHANGE_CHANGED; +} + +/* ------------------------------------------------------------ */ +/* Engine hooks */ +/* ------------------------------------------------------------ */ + +/* The live accumulator, created on first use; NULL when --changes is off. */ +static hts_changes *changes_get(httrackp *opt) { + if (!opt->changes) + return NULL; + if (opt->changes_state == NULL) + opt->changes_state = changes_new(); + return (hts_changes *) opt->changes_state; +} + +void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, + const char *save, hts_boolean rewritten, + hts_boolean not_updated) { + hts_changes *const changes = changes_get(opt); + char BIGSTK file[HTS_URLMAXSIZE * 2]; + hts_boolean is_new; + intptr_t slot; + changes_entry *entry; + + /* Engine-generated scaffolding (the top index) carries no URL and is not a + mirrored resource. */ + if (changes == NULL || save == NULL || !strnotempty(save) || adr == NULL || + fil == NULL || (!strnotempty(adr) && !strnotempty(fil))) + return; + + hts_savename_listed(&opt->state.strc, save, file, sizeof(file)); + slot = changes_slot(changes, file, &is_new); + if (slot < 0) + return; + entry = &changes->entries[slot]; + + if (!is_new) { + /* Retry, or a second call site for the same file: the pre-run state was + already sampled and the copy on disk may no longer be it. */ + entry->rewritten = entry->rewritten || rewritten; + return; + } + + { + char BIGSTK url[HTS_URLMAXSIZE * 2]; + + url[0] = '\0'; + if (!link_has_authority(adr)) + strlcatbuff(url, "http://", sizeof(url)); + strlcatbuff(url, adr, sizeof(url)); + strlcatbuff(url, fil, sizeof(url)); + entry->url = strdupt(url); + if (entry->url == NULL) + changes->overflow = HTS_TRUE; + } + + entry->rewritten = rewritten; + entry->not_updated = not_updated; + entry->prev_size = fsize_utf8(save); + entry->on_disk = entry->prev_size >= 0; + /* Hash the previous copy only when it is about to be overwritten: this is + the last moment those bytes exist. */ + if (entry->on_disk && rewritten) + entry->has_prev = digest_file(save, entry->prev_digest); +} + +void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r, + const char *adr, const char *fil, const char *save) { + hts_changes *const changes = changes_get(opt); + char BIGSTK file[HTS_URLMAXSIZE * 2]; + char BIGSTK location[HTS_URLMAXSIZE * 2]; + changes_entry *entry; + intptr_t slot; + htsblk prev; + + if (changes == NULL || r->adr == NULL || r->size < 0 || save == NULL || + !strnotempty(save)) + return; + hts_savename_listed(&opt->state.strc, save, file, sizeof(file)); + /* file_notify() records the entry; this call only refines it. */ + slot = changes_find(changes, file); + if (slot < 0) + return; + entry = &changes->entries[slot]; + + /* On disk this is the payload plus rewritten links and a footer dated by the + crawl, so it differs every run; compare payloads, the previous one being + the body the cache kept. */ + entry->has_new = HTS_TRUE; + digest_mem(r->adr, (size_t) r->size, entry->new_digest); + entry->has_prev = HTS_FALSE; + prev = cache_read_ro(opt, cache, adr, fil, "", location); + if (HTTP_IS_OK(prev.statuscode) && prev.adr != NULL && prev.size >= 0) { + digest_mem(prev.adr, (size_t) prev.size, entry->prev_digest); + entry->has_prev = HTS_TRUE; + } + freet(prev.adr); +} + +void hts_changes_gone(httrackp *opt, const char *file) { + hts_changes *const changes = changes_get(opt); + hts_boolean is_new; + intptr_t slot; + + if (changes == NULL || file == NULL || !strnotempty(file)) + return; + slot = changes_slot(changes, file, &is_new); + if (slot < 0 || !is_new) + return; + changes->entries[slot].url = strdupt(""); + changes->entries[slot].bucket = HTS_CHANGE_GONE; + changes->entries[slot].listed_prev = HTS_TRUE; +} + +void hts_changes_previous(httrackp *opt, const char *file) { + hts_changes *const changes = changes_get(opt); + intptr_t slot; + + if (changes == NULL || file == NULL) + return; + changes->old_index = HTS_TRUE; + slot = changes_find(changes, file); + if (slot >= 0) + changes->entries[slot].listed_prev = HTS_TRUE; +} + +/* ------------------------------------------------------------ */ +/* Report */ +/* ------------------------------------------------------------ */ + +/* Assign each entry its final bucket from the bytes now on disk. */ +static void changes_resolve(hts_changes *changes, httrackp *opt) { + char catbuff[CATBUFF_SIZE]; + size_t i; + + for (i = 0; i < changes->count; i++) { + changes_entry *const entry = &changes->entries[i]; + const char *path; + unsigned char digest[DIGEST_SIZE]; + hts_boolean have_digests = HTS_FALSE; + hts_boolean digests_equal = HTS_FALSE; + + if (entry->bucket == HTS_CHANGE_GONE) + continue; + + /* The previous mirror's index is the authority on what was there before: + a partial left by this crawl's own failed attempt is on disk but was + never part of the previous mirror. Without an index, fall back to what + the first notify saw on disk. */ + entry->existed = changes->old_index ? entry->listed_prev : entry->on_disk; + path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_html), + entry->file); + entry->size = fsize_utf8(path); + if (entry->rewritten && entry->existed) { + if (entry->size >= 0 && entry->prev_size >= 0 && + entry->size != entry->prev_size) { + have_digests = HTS_TRUE; /* different lengths: no need to hash */ + digests_equal = HTS_FALSE; + } else if (entry->has_prev && + (entry->has_new + ? (memcpy(digest, entry->new_digest, DIGEST_SIZE), 1) + : digest_file(path, digest))) { + have_digests = HTS_TRUE; + digests_equal = memcmp(digest, entry->prev_digest, DIGEST_SIZE) == 0 + ? HTS_TRUE + : HTS_FALSE; + } + } + entry->bucket = + hts_changes_classify(entry->rewritten, entry->existed, + entry->not_updated, have_digests, digests_equal); + } +} + +static const char *const bucket_names[HTS_CHANGE_BUCKETS] = { + "new", "changed", "unchanged", "gone"}; + +void hts_changes_report(httrackp *opt, String *out) { + hts_changes *const changes = (hts_changes *) opt->changes_state; + size_t counts[HTS_CHANGE_BUCKETS]; + char date[32]; + char scratch[64]; + int bucket; + size_t i; + + StringClear(*out); + if (changes == NULL) + return; + changes_resolve(changes, opt); + + memset(counts, 0, sizeof(counts)); + for (i = 0; i < changes->count; i++) + counts[changes->entries[i].bucket]++; + + hts_now_iso8601(date); + StringCat(*out, "{\n \"schema\": "); + snprintf(scratch, sizeof(scratch), "%d", HTS_CHANGES_SCHEMA); + StringCat(*out, scratch); + StringCat(*out, ",\n \"generator\": "); + hts_changes_json_string(out, "HTTrack Website Copier/" HTTRACK_VERSION); + StringCat(*out, ",\n \"date\": "); + hts_changes_json_string(out, date); + StringCat(*out, ",\n \"first_crawl\": "); + StringCat(*out, changes->old_index ? "false" : "true"); + StringCat(*out, ",\n \"partial\": "); + StringCat(*out, changes->overflow ? "true" : "false"); + StringCat(*out, ",\n \"purged\": "); + StringCat(*out, opt->delete_old ? "true" : "false"); + + StringCat(*out, ",\n \"counts\": {"); + for (bucket = 0; bucket < HTS_CHANGE_BUCKETS; bucket++) { + StringCat(*out, bucket != 0 ? ", " : " "); + hts_changes_json_string(out, bucket_names[bucket]); + snprintf(scratch, sizeof(scratch), ": %d", (int) counts[bucket]); + StringCat(*out, scratch); + } + StringCat(*out, " }"); + + for (bucket = 0; bucket < HTS_CHANGE_BUCKETS; bucket++) { + hts_boolean first = HTS_TRUE; + + StringCat(*out, ",\n "); + hts_changes_json_string(out, bucket_names[bucket]); + StringCat(*out, ": ["); + for (i = 0; i < changes->count; i++) { + const changes_entry *const entry = &changes->entries[i]; + + if (entry->bucket != bucket) + continue; + StringCat(*out, first ? "\n { \"url\": " : ",\n { \"url\": "); + first = HTS_FALSE; + hts_changes_json_string(out, entry->url != NULL ? entry->url : ""); + StringCat(*out, ", \"file\": "); + hts_changes_json_string(out, entry->file); + if (entry->size >= 0) { + snprintf(scratch, sizeof(scratch), ", \"size\": " LLintP, + (LLint) entry->size); + StringCat(*out, scratch); + } + if (bucket == HTS_CHANGE_CHANGED && entry->prev_size >= 0) { + snprintf(scratch, sizeof(scratch), ", \"previous_size\": " LLintP, + (LLint) entry->prev_size); + StringCat(*out, scratch); + } + StringCat(*out, " }"); + } + StringCat(*out, first ? "]" : "\n ]"); + } + StringCat(*out, "\n}\n"); +} + +void hts_changes_close_opt(httrackp *opt) { + hts_changes *changes = (hts_changes *) opt->changes_state; + char catbuff[CATBUFF_SIZE]; + const char *path; + String report = STRING_EMPTY; + size_t counts[HTS_CHANGE_BUCKETS]; + size_t i; + FILE *fp; + + if (changes == NULL) + return; + + hts_changes_report(opt, &report); + memset(counts, 0, sizeof(counts)); + for (i = 0; i < changes->count; i++) + counts[changes->entries[i].bucket]++; + + path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_log), + HTS_CHANGES_FILE); + fp = FOPEN(path, "wb"); + if (fp != NULL) { + const size_t len = StringLength(report); + + if (len != 0 && fwrite(StringBuff(report), 1, len, fp) != len) + hts_log_print(opt, LOG_ERROR | LOG_ERRNO, + "Unable to write the change report %s", path); + fclose(fp); + } else { + hts_log_print(opt, LOG_ERROR | LOG_ERRNO, + "Unable to create the change report %s", path); + } + + if (!changes->old_index) { + hts_log_print(opt, LOG_NOTICE, + "Change report: first crawl, %d files mirrored, nothing to " + "compare against (%s)", + (int) changes->count, HTS_CHANGES_FILE); + } else { + hts_log_print(opt, LOG_NOTICE, + "Change report: %d new, %d changed, %d unchanged, %d gone " + "(%s)", + (int) counts[HTS_CHANGE_NEW], + (int) counts[HTS_CHANGE_CHANGED], + (int) counts[HTS_CHANGE_UNCHANGED], + (int) counts[HTS_CHANGE_GONE], HTS_CHANGES_FILE); + } + + StringFree(report); + changes_free(&changes); + opt->changes_state = NULL; +} + +void hts_changes_free_opt(httrackp *opt) { + hts_changes *changes = (hts_changes *) opt->changes_state; + + changes_free(&changes); + opt->changes_state = NULL; +} diff --git a/src/htschanges.h b/src/htschanges.h new file mode 100644 index 000000000..76b8b7740 --- /dev/null +++ b/src/htschanges.h @@ -0,0 +1,126 @@ +/* ------------------------------------------------------------ */ +/* +HTTrack Website Copier, Offline Browser for Windows and Unix +Copyright (C) 2026 Xavier Roche and other contributors + +SPDX-License-Identifier: GPL-3.0-or-later + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . + +Ethical use: we kindly ask that you NOT use this software to harvest email +addresses or to collect any other private information about people. Doing so +would dishonor our work and waste the many hours we have spent on it. + +Please visit our Website: http://www.httrack.com +*/ + +/* ------------------------------------------------------------ */ +/* HTTrack change report (--changes). Internal, not installed. + Accumulates what the crawl did to each mirrored file, compares the bytes + against the copy the previous run left behind, and writes + hts-changes.json next to the log. */ +/* ------------------------------------------------------------ */ + +#ifndef HTS_CHANGES_DEFH +#define HTS_CHANGES_DEFH + +#include "htsopt.h" +#include "htsstrings.h" + +#ifndef HTS_DEF_FWSTRUCT_cache_back +#define HTS_DEF_FWSTRUCT_cache_back +typedef struct cache_back cache_back; +#endif + +#ifdef __cplusplus +extern "C" { +#endif + +/* Report file name, written under the project's log directory. */ +#define HTS_CHANGES_FILE "hts-changes.json" + +/* Schema version carried by the report; bump on an incompatible change. */ +#define HTS_CHANGES_SCHEMA 1 + +/* Which side of the comparison a mirrored file ended up on. */ +typedef enum { + HTS_CHANGE_NEW = 0, /* no local copy before this run */ + HTS_CHANGE_CHANGED, /* rewritten, and the bytes differ */ + HTS_CHANGE_UNCHANGED, /* the bytes are those of the previous mirror */ + HTS_CHANGE_GONE, /* in the previous mirror, absent from this one */ + HTS_CHANGE_BUCKETS +} hts_change_bucket; + +/* Record what this crawl is doing to the local file `save` (an absolute path; + adr/fil form the URL it comes from, either may be empty for an engine + generated file). `rewritten` means the copy on disk is being written over, + `not_updated` is the 200-versus-304 transfer signal, used only when no + digest can be taken. Only the first call for a given file samples the + previous copy, so a retried or twice-notified resource is counted once. + No-op unless --changes is on. */ +void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, + const char *save, hts_boolean rewritten, + hts_boolean not_updated); + +/* Refine the entry for a parsed HTML file, whose local copy is the payload + after link rewriting plus a footer carrying the crawl date: its bytes differ + on every run, so the comparison uses the payload `r` holds against the body + the previous run left in the cache. Called once the page is ready to be + written, from hts_finish_html_file(). */ +void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r, + const char *adr, const char *fil, const char *save); + +/* Record `file` (mirror-relative, as listed in new.lst) as present in the + previous mirror and absent from this one. Whether it is also deleted from + disk is the purge option's business, not this one's. */ +void hts_changes_gone(httrackp *opt, const char *file); + +/* Record that the previous mirror's index listed `file`. That index, not the + file's presence on disk, decides what counts as already mirrored: a partial + left by this crawl's own failed attempt is on disk but was never part of the + previous mirror. Calling it at all marks the run as having one. */ +void hts_changes_previous(httrackp *opt, const char *file); + +/* Resolve every entry against the bytes now on disk and serialize the report + into `out` (replaced). Exposed for the `changes` self-test; the engine goes + through hts_changes_close_opt(). */ +void hts_changes_report(httrackp *opt, String *out); + +/* Write the report and log a one-line summary, then free the accumulator. + Null-safe and idempotent. */ +void hts_changes_close_opt(httrackp *opt); + +/* Drop the accumulator without writing anything, for a run that never reached + its end. Null-safe and idempotent. */ +void hts_changes_free_opt(httrackp *opt); + +/* Bucket for one entry, from what was observed. No filesystem access: + `have_digests` says both sides could be hashed, `digests_equal` compares + them, and `not_updated` is the fallback signal when they could not. */ +hts_change_bucket hts_changes_classify(hts_boolean rewritten, + hts_boolean existed, + hts_boolean not_updated, + hts_boolean have_digests, + hts_boolean digests_equal); + +/* Append `s` to `out` as a quoted JSON string. Byte sequences that are not + valid UTF-8 become U+FFFD, so a mirror carrying legacy-charset URLs still + produces parseable JSON. */ +void hts_changes_json_string(String *out, const char *s); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/src/htscore.c b/src/htscore.c index dab4e4bf4..278713493 100644 --- a/src/htscore.c +++ b/src/htscore.c @@ -40,6 +40,7 @@ Please visit our Website: http://www.httrack.com /* File defs */ #include "htscore.h" #include "htswarc.h" +#include "htschanges.h" /* specific definitions */ #include "htsbase.h" @@ -488,6 +489,7 @@ void hts_finish_html_file(httrackp *opt, cache_back *cache, htsblk *r, const char *adr, const char *fil, const char *save) { { file_notify(opt, adr, fil, save, 1, 1, r->notmodified); + hts_changes_html(opt, cache, r, adr, fil, save); *fp = filecreate(&opt->state.strc, save); if (*fp) { if (ht_len > 0 && fwrite(ht_buff, 1, ht_len, *fp) != ht_len) { @@ -2089,7 +2091,9 @@ int httpmirror(char *url1, httrackp * opt) { if (cache.lst) { fclose(cache.lst); cache.lst = opt->state.strc.lst = NULL; - if (opt->delete_old) { + /* old.lst minus new.lst is the set of files the previous mirror had and + this one does not. --changes reports it; only --purge-old acts on it. */ + if (opt->delete_old || opt->changes) { FILE *old_lst, *new_lst; // @@ -2117,21 +2121,27 @@ int httpmirror(char *url1, httrackp * opt) { int purge = 0; while(!feof(old_lst)) { + char BIGSTK file[HTS_URLMAXSIZE * 2]; + linput(old_lst, line, 1000); + if (!strnotempty(line)) + continue; + strcpybuff(file, StringBuff(opt->path_html)); + strcatbuff(file, line + 1); + file[strlen(file) - 1] = '\0'; + hts_changes_previous(opt, file + StringLength(opt->path_html)); if (!strstr(adr, line)) { // not found in the new list? - char BIGSTK file[HTS_URLMAXSIZE * 2]; - - strcpybuff(file, StringBuff(opt->path_html)); - strcatbuff(file, line + 1); - file[strlen(file) - 1] = '\0'; - if (fexist_utf8(file)) { // toujours sur disque: virer - hts_log_print(opt, LOG_INFO, "Purging %s", file); - UNLINK(file); - purge = 1; + if (fexist_utf8(file)) { // still on disk + hts_changes_gone(opt, file + StringLength(opt->path_html)); + if (opt->delete_old) { + hts_log_print(opt, LOG_INFO, "Purging %s", file); + UNLINK(file); + purge = 1; + } } } } - { + if (opt->delete_old) { // emptied directories go with the files fseek(old_lst, 0, SEEK_SET); while(!feof(old_lst)) { linput(old_lst, line, 1000); @@ -2166,7 +2176,7 @@ int httpmirror(char *url1, httrackp * opt) { } } // - if (!purge) { + if (opt->delete_old && !purge) { hts_log_print(opt, LOG_INFO, "No files purged"); } } @@ -2256,6 +2266,7 @@ int httpmirror(char *url1, httrackp * opt) { // ending usercommand(opt, 0, NULL, NULL, NULL, NULL); warc_close_opt(opt); + hts_changes_close_opt(opt); // désallocation mémoire & buffers XH_uninit; @@ -2881,6 +2892,18 @@ int filecreateempty(filenote_strc * strc, const char *filename) { return 0; } +void hts_savename_listed(const filenote_strc *strc, const char *s, char *dest, + size_t destsize) { + char catbuff[CATBUFF_SIZE]; + + strlcpybuff(dest, fslash(catbuff, sizeof(catbuff), s), destsize); + if (strnotempty(strc->path) && + strncmp(fslash(catbuff, sizeof(catbuff), strc->path), dest, + strlen(strc->path)) == 0) { + strlcpybuff(dest, s + strlen(strc->path), destsize); + } +} + // noter fichier int filenote(filenote_strc * strc, const char *s, filecreate_params * params) { // gestion du fichier liste liste @@ -2890,15 +2913,8 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) { return 0; } else if (strc->lst) { char BIGSTK savelst[HTS_URLMAXSIZE * 2]; - char catbuff[CATBUFF_SIZE]; - strcpybuff(savelst, fslash(catbuff, sizeof(catbuff), s)); - // couper chemin? - if (strnotempty(strc->path)) { - if (strncmp(fslash(catbuff, sizeof(catbuff), strc->path), savelst, strlen(strc->path)) == 0) { // couper - strcpybuff(savelst, s + strlen(strc->path)); - } - } + hts_savename_listed(strc, s, savelst, sizeof(savelst)); fprintf(strc->lst, "[%s]" LF, savelst); fflush(strc->lst); } @@ -2908,6 +2924,9 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) { /* Note: utf-8 */ void file_notify(httrackp * opt, const char *adr, const char *fil, const char *save, int create, int modify, int not_updated) { + hts_changes_notify(opt, adr, fil, save, + (create || modify) ? HTS_TRUE : HTS_FALSE, + not_updated ? HTS_TRUE : HTS_FALSE); RUN_CALLBACK6(opt, filesave2, adr, fil, save, create, modify, not_updated); } @@ -3642,6 +3661,7 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) { to->warc_max_size = from->warc_max_size; to->warc_cdx = from->warc_cdx; to->warc_wacz = from->warc_wacz; + to->changes = from->changes; if (from->pause_max_ms > 0) { to->pause_min_ms = from->pause_min_ms; diff --git a/src/htscore.h b/src/htscore.h index a2c4e40d8..eb9495fe5 100644 --- a/src/htscore.h +++ b/src/htscore.h @@ -354,6 +354,13 @@ int filecreateempty(filenote_strc * strct, const char *filename); int filenote(filenote_strc * strct, const char *s, filecreate_params * params); +/* Copy into dest (destsize bytes) the form under which the local path s is + listed in new.lst: forward slashes, and the mirror root stripped when s sits + under it. That listed form is also the change report's key, so the two must + not drift. */ +void hts_savename_listed(const filenote_strc *strct, const char *s, char *dest, + size_t destsize); + void file_notify(httrackp * opt, const char *adr, const char *fil, const char *save, int create, int modify, int wasupdated); diff --git a/src/htscoremain.c b/src/htscoremain.c index 400cda400..a6bd2e1d5 100644 --- a/src/htscoremain.c +++ b/src/htscoremain.c @@ -41,6 +41,7 @@ Please visit our Website: http://www.httrack.com #include "htsdefines.h" #include "htsalias.h" #include "htswarc.h" +#include "htschanges.h" #include "htsbauth.h" #include "htswrap.h" #include "htsmodules.h" @@ -1747,6 +1748,13 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) { StringCopy(opt->cookies_file, argv[na]); } break; + case 'd': // --changes: report what this crawl changed + opt->changes = HTS_TRUE; + if (*(com + 1) == '0') { + opt->changes = HTS_FALSE; + com++; + } + break; case 'r': // warc / warc-file: write an ISO-28500 WARC archive if (*(com + 1) == 'f') { // --warc-file NAME: explicit basename com++; diff --git a/src/htshelp.c b/src/htshelp.c index 00569bf11..88aba2ac0 100644 --- a/src/htshelp.c +++ b/src/htshelp.c @@ -607,6 +607,8 @@ void help(const char *app, int more) { "output name, --warc-max-size N rotates segments past N bytes, " "--warc-cdx also writes a sorted CDXJ index, --wacz packages it all " "as a WACZ file"); + infomsg(" %d write hts-changes.json listing what this crawl left new, " + "changed, unchanged and gone compared to the previous mirror"); infomsg(" %n do not re-download locally erased files"); infomsg (" %v display on screen filenames downloaded (in realtime) - * %v1 short version - %v2 full animation"); diff --git a/src/htslib.c b/src/htslib.c index 55b524d40..9bfabc334 100644 --- a/src/htslib.c +++ b/src/htslib.c @@ -37,6 +37,7 @@ Please visit our Website: http://www.httrack.com #include "htscore.h" #include "htswarc.h" +#include "htschanges.h" /* specific definitions */ #include "htsbase.h" @@ -2698,6 +2699,24 @@ void time_gmt_rfc822(char *s) { time_rfc822(s, A); } +void hts_now_iso8601(char out[32]) { + time_t t = time(NULL); + struct tm tmv; + +#if defined(_WIN32) + struct tm *g = gmtime(&t); + + if (g != NULL) + tmv = *g; + else + memset(&tmv, 0, sizeof(tmv)); +#else + if (gmtime_r(&t, &tmv) == NULL) + memset(&tmv, 0, sizeof(tmv)); +#endif + strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv); +} + // heure actuelle, format rfc (taille buffer 256o) void time_local_rfc822(char *s) { time_t tt; @@ -6011,6 +6030,8 @@ HTSEXT_API httrackp *hts_create_opt(void) { StringCopy(opt->cookies_file, ""); StringCopy(opt->warc_file, ""); opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */ + opt->changes = HTS_FALSE; + opt->changes_state = NULL; StringCopy(opt->why_url, ""); opt->pause_min_ms = 0; opt->pause_max_ms = 0; @@ -6164,6 +6185,8 @@ HTSEXT_API void hts_free_opt(httrackp * opt) { StringFree(opt->why_url); StringFree(opt->warc_file); + hts_changes_free_opt(opt); + StringFree(opt->path_html); StringFree(opt->path_html_utf8); StringFree(opt->path_log); diff --git a/src/htslib.h b/src/htslib.h index 6ad9ff954..7fb652f80 100644 --- a/src/htslib.h +++ b/src/htslib.h @@ -253,6 +253,10 @@ void sec2str(char *s, TStamp t); void time_gmt_rfc822(char *s); void time_local_rfc822(char *s); + +/* Current UTC time as "YYYY-MM-DDThh:mm:ssZ". */ +void hts_now_iso8601(char out[32]); + struct tm *convert_time_rfc822(struct tm *buffer, const char *s); int set_filetime(const char *file, struct tm *tm_time); int set_filetime_rfc822(const char *file, const char *date); diff --git a/src/htsopt.h b/src/htsopt.h index 6ead35372..4a6b27fe7 100644 --- a/src/htsopt.h +++ b/src/htsopt.h @@ -547,6 +547,10 @@ struct httrackp { archive. Tail: ABI */ hts_boolean warc_wacz; /**< --wacz: package archive+index+pages as a WACZ zip (implies --warc + --warc-cdx). Tail: ABI */ + hts_boolean changes; /**< --changes: report what this crawl changed against + the previous mirror. Tail: ABI */ + void *changes_state; /**< live change-report accumulator (hts_changes*), + engine-owned. Tail: ABI */ }; /* Running statistics for a mirror. */ diff --git a/src/htsselftest.c b/src/htsselftest.c index 9e61fb71a..4728dbb6b 100644 --- a/src/htsselftest.c +++ b/src/htsselftest.c @@ -59,6 +59,7 @@ Please visit our Website: http://www.httrack.com #include "htscodec.h" #include "htsproxy.h" #include "htswarc.h" +#include "htschanges.h" #if HTS_USEZLIB #include "htszlib.h" #endif @@ -1573,6 +1574,12 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) { if (strcmp(StringBuff(to->warc_file), "run.warc.gz") != 0) err = 1; + from->changes = HTS_TRUE; + to->changes = HTS_FALSE; + copy_htsopt(from, to); + if (to->changes != HTS_TRUE) + err = 1; + /* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */ from->pause_min_ms = 5000; from->pause_max_ms = 10000; @@ -5059,6 +5066,63 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) { return 0; } +/* --changes bucket accounting and JSON escaping (#714). */ +static int st_changes(httrackp *opt, int argc, char **argv) { + String out = STRING_EMPTY; + int err = 0; + + (void) opt; + (void) argc; + (void) argv; + + /* A file the crawl did not rewrite is unchanged whatever the wire said. */ + assertf(hts_changes_classify(HTS_FALSE, HTS_TRUE, HTS_FALSE, HTS_FALSE, + HTS_FALSE) == HTS_CHANGE_UNCHANGED); + /* Rewritten with no previous copy: new, digests or not. */ + assertf(hts_changes_classify(HTS_TRUE, HTS_FALSE, HTS_FALSE, HTS_TRUE, + HTS_FALSE) == HTS_CHANGE_NEW); + /* Digests decide, and outrank the transfer signal both ways: a server with + no validators answers 200 with the same bytes, and a 304 can still sit in + front of a locally damaged copy. */ + assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_FALSE, HTS_TRUE, + HTS_TRUE) == HTS_CHANGE_UNCHANGED); + assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_TRUE, HTS_TRUE, + HTS_FALSE) == HTS_CHANGE_CHANGED); + /* Only with no digest at all does the transfer signal get a say. */ + assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_TRUE, HTS_FALSE, + HTS_FALSE) == HTS_CHANGE_UNCHANGED); + assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_FALSE, HTS_FALSE, + HTS_FALSE) == HTS_CHANGE_CHANGED); + +#define JSON_IS(SRC, WANT) \ + do { \ + StringClear(out); \ + hts_changes_json_string(&out, SRC); \ + if (strcmp(StringBuff(out), WANT) != 0) { \ + fprintf(stderr, "changes: %s -> %s, expected %s\n", #SRC, \ + StringBuff(out), WANT); \ + err = 1; \ + } \ + } while (0) + + JSON_IS("/a/b.html", "\"/a/b.html\""); + JSON_IS("a\"b\\c", "\"a\\\"b\\\\c\""); + JSON_IS("tab\there", "\"tab\\u0009here\""); + /* Valid UTF-8 rides through; a lone Latin-1 byte, a truncated sequence and + an overlong encoding of '/' each become U+FFFD rather than invalid JSON. */ + JSON_IS("caf\xc3\xa9", "\"caf\xc3\xa9\""); + JSON_IS("caf\xe9", "\"caf\\ufffd\""); + JSON_IS("\xc3", "\"\\ufffd\""); + JSON_IS("\xc0\xaf", "\"\\ufffd\\ufffd\""); + /* A UTF-16 surrogate half is well-formed UTF-8 by shape only. */ + JSON_IS("\xed\xa0\x80", "\"\\ufffd\\ufffd\\ufffd\""); + +#undef JSON_IS + StringFree(out); + printf("changes self-test: %s\n", err ? "FAIL" : "OK"); + return err; +} + /* ------------------------------------------------------------ */ /* Registry: name -> handler, with a usage hint and a one-line description. */ /* ------------------------------------------------------------ */ @@ -5114,6 +5178,8 @@ static const struct selftest_entry { {"strsafe", "[overflow|overflow-buff|overflow-src [str]]", "bounded string-op self-test", st_strsafe}, {"copyopt", "", "copy_htsopt option-copy self-test", st_copyopt}, + {"changes", "", "--changes bucket accounting and JSON escaping (#714)", + st_changes}, {"pause", "", "randomized inter-file pause target self-test", st_pause}, {"relative", " ", "relative link between two paths", st_relative}, diff --git a/src/htswarc.c b/src/htswarc.c index 78debda42..9f23b8167 100644 --- a/src/htswarc.c +++ b/src/htswarc.c @@ -462,22 +462,6 @@ static void warc_make_id(warc_writer *w, char out[64]) { b[11], b[12], b[13], b[14], b[15]); } -static void warc_now_iso8601(char out[32]) { - time_t t = time(NULL); - struct tm tmv; -#if defined(_WIN32) - struct tm *g = gmtime(&t); - if (g != NULL) - tmv = *g; - else - memset(&tmv, 0, sizeof(tmv)); -#else - if (gmtime_r(&t, &tmv) == NULL) - memset(&tmv, 0, sizeof(tmv)); -#endif - strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv); -} - /* Case-insensitive "is this the header named name?" test, tolerating optional whitespace before the ':' (non-compliant "Name : value" is still matched). */ static int header_is(const char *line, size_t line_len, const char *name) { @@ -1063,7 +1047,7 @@ static void warc_wacz_package(warc_writer *w) { wbuf_free(&pages); /* datapackage.json listing every stored file with its sha256 + size. */ - warc_now_iso8601(created); + hts_now_iso8601(created); memset(&dp, 0, sizeof(dp)); if (!err && (wbuf_printf(&dp, @@ -1166,7 +1150,7 @@ static int warc_emit(warc_writer *w, const char *type, const char *content_type, memset(&hdr, 0, sizeof(hdr)); warc_make_id(w, id); - warc_now_iso8601(date); + hts_now_iso8601(date); #if HTS_USEOPENSSL /* Block digest over the whole block, in one streaming pass. */ @@ -1670,7 +1654,7 @@ int warc_write_transaction(warc_writer *w, const char *target_uri, char date[32]; char *slot; size_t need; - warc_now_iso8601(date); + hts_now_iso8601(date); need = strlen(target_uri) + 1 + strlen(date) + 1; slot = malloct(need); if (slot != NULL) { diff --git a/src/libhttrack.vcxproj b/src/libhttrack.vcxproj index 86256d81e..a5c2a8674 100644 --- a/src/libhttrack.vcxproj +++ b/src/libhttrack.vcxproj @@ -138,6 +138,7 @@ + diff --git a/tests/01_engine-changes.test b/tests/01_engine-changes.test new file mode 100644 index 000000000..28e70e946 --- /dev/null +++ b/tests/01_engine-changes.test @@ -0,0 +1,7 @@ +#!/bin/bash +# + +set -euo pipefail + +# --changes bucket accounting and JSON escaping (#714). +httrack -O /dev/null -#test=changes run | grep -q "changes self-test: OK" diff --git a/tests/88_local-changes.test b/tests/88_local-changes.test new file mode 100644 index 000000000..2c72f7a51 --- /dev/null +++ b/tests/88_local-changes.test @@ -0,0 +1,146 @@ +#!/bin/bash +# --changes (#714): the report must match the mirror's real deltas. The routes +# answer 200 with no validators on both passes, so the transfer signal alone +# would call everything changed; stable.html/stable.bin land in "unchanged" +# only if the payload is actually compared. +set -eu + +: "${top_srcdir:=..}" +testdir=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=tests/testlib.sh +. "${testdir}/testlib.sh" +server=$(nativepath "${testdir}/local-server.py") +root=$(nativepath "${testdir}/server-root") + +python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77 + +tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_changes.XXXXXX") || exit 1 +serverpid= +cleanup() { + stop_server "$serverpid" + rm -rf "$tmpdir" +} +trap cleanup EXIT HUP INT QUIT PIPE TERM + +serverlog="${tmpdir}/server.log" +: >"$serverlog" +"$python" "$server" --root "$root" >"$serverlog" 2>&1 & +serverpid=$! +port= +for _ in $(seq 1 50); do + line=$(head -n1 "$serverlog" 2>/dev/null) + if test "${line%% *}" == "PORT"; then + port="${line#PORT }" + break + fi + kill -0 "$serverpid" 2>/dev/null || { + echo "server exited early: $(cat "$serverlog")" + exit 1 + } + sleep 0.1 +done +test -n "$port" || { + echo "could not discover server port" + exit 1 +} +base="http://127.0.0.1:${port}" + +which httrack >/dev/null || { + echo "could not find httrack" + exit 1 +} +out="${tmpdir}/crawl" +mkdir "$out" +report="${out}/hts-changes.json" +# Purging stays off until the last pass: the deleted set must be computed even +# when nothing acts on it. +common=(-O "$out" --quiet --disable-security-limits --robots=0 --timeout=30 --changes --purge-old=0 --retries=2) + +# Report field extractor: bucket counts, and the files listed under a bucket. +count() { sed -n 's/.*"'"$1"'": \([0-9]*\).*/\1/p' "$report" | head -1; } +listed() { + "$python" - "$report" "$1" <<'EOF' +import json +import sys + +with open(sys.argv[1], encoding="utf-8") as fp: + report = json.load(fp) +for entry in report[sys.argv[2]]: + print(entry["file"]) +EOF +} +expect() { + local label="$1" want="$2" got="$3" + printf '[%s] ..\t' "$label" + test "$want" == "$got" || { + echo "FAIL: expected '${want}', got '${got}'" + cat "$report" + exit 1 + } + echo "OK" +} + +# --- pass 1: nothing to compare against -------------------------------------- +httrack "${common[@]}" "${base}/changes/index.html" >"${tmpdir}/log1" 2>&1 +test -s "$report" || { + echo "FAIL: pass 1 wrote no ${report}" + exit 1 +} +expect "pass 1 declares a first crawl" "true" \ + "$(sed -n 's/.*"first_crawl": \([a-z]*\).*/\1/p' "$report" | head -1)" +# index, stable.html, moved.html, stable.bin, moved.bin, doomed.html, +# redirtarget.html, flaky.bin +expect "pass 1 counts 8 new" "8" "$(count new)" +expect "pass 1 counts 0 changed" "0" "$(count changed)" +expect "pass 1 counts 0 gone" "0" "$(count gone)" +grep -aq "first crawl" "${out}/hts-log.txt" || { + echo "FAIL: pass 1 log carries no first-crawl summary" + exit 1 +} + +# --- pass 2: two bodies move, one page appears, one disappears --------------- +httrack "${common[@]}" --update "${base}/changes/index.html" >"${tmpdir}/log2" 2>&1 +expect "pass 2 is not a first crawl" "false" \ + "$(sed -n 's/.*"first_crawl": \([a-z]*\).*/\1/p' "$report" | head -1)" + +host="127.0.0.1_${port}" +expect "new is fresh.html only" "${host}/changes/fresh.html" "$(listed new)" +expect "gone is doomed.html only" "${host}/changes/doomed.html" "$(listed gone)" +# index.html changes because its link list does; moved.* change their bodies. +expect "changed is exactly the three moved resources" \ + "$(printf '%s\n%s\n%s' "${host}/changes/index.html" \ + "${host}/changes/moved.bin" "${host}/changes/moved.html")" \ + "$(listed changed | sort)" +# The teeth: re-served byte for byte, 200, no Last-Modified, no ETag. flaky.bin +# gets there through a failed transfer and a retry, so its file is notified +# twice; redirtarget.html arrives behind a 302. +expect "unchanged is exactly the four stable resources" \ + "$(printf '%s\n%s\n%s\n%s' "${host}/changes/flaky.bin" \ + "${host}/changes/redirtarget.html" "${host}/changes/stable.bin" \ + "${host}/changes/stable.html")" \ + "$(listed unchanged | sort)" + +# Every mirrored file appears once and only once, across all four buckets. +printf '[each file in exactly one bucket] ..\t' +dupes=$( ( + listed new + listed changed + listed unchanged + listed gone +) | sort | uniq -d) +test -z "$dupes" || { + echo "FAIL: listed in more than one bucket: ${dupes}" + exit 1 +} +echo "OK" + +# Before #714 the deleted set only existed as a side effect of purging; it must +# now be computed without one, and reporting it must not delete anything. +expect "the report says nothing was purged" "false" \ + "$(sed -n 's/.*"purged": \([a-z]*\).*/\1/p' "$report" | head -1)" +printf '[reporting gone did not delete it] ..\t' +test -f "${out}/${host}/changes/doomed.html" || { + echo "FAIL: doomed.html was deleted with --purge-old=0" + exit 1 +} +echo "OK" diff --git a/tests/Makefile.am b/tests/Makefile.am index 71f9fe51c..96201a204 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -30,6 +30,7 @@ TEST_EXTENSIONS = .test TEST_LOG_COMPILER = $(BASH) TESTS = \ 00_runnable.test \ + 01_engine-changes.test \ 01_engine-charset.test \ 01_engine-cmdline.test \ 01_engine-cmdline-split.test \ @@ -182,6 +183,7 @@ TESTS = \ 82_webhttrack-browse-links.test \ 83_webhttrack-argescape.test \ 84_webhttrack-mirror-verbatim.test \ - 85_webhttrack-projpath.test + 85_webhttrack-projpath.test \ + 88_local-changes.test CLEANFILES = check-network_sh.cache diff --git a/tests/local-server.py b/tests/local-server.py index b0c15517a..83320c4b9 100755 --- a/tests/local-server.py +++ b/tests/local-server.py @@ -1571,6 +1571,82 @@ def route_maxrecv_404(self): if self.command != "HEAD": self.wfile.write(body) + # --changes (#714). Every route answers 200 with no validators, so the + # transfer signal alone would call the whole site changed on pass 2; only a + # payload comparison can tell stable.html and stable.bin from moved.*. + def route_changes_index(self): + links = "".join( + '\t%s\n' % (name, name) + for name in ( + "stable.html", + "moved.html", + "stable.bin", + "moved.bin", + "redir.html", + "flaky.bin", + ) + ) + if self.refetch_pass() == 1: + links += '\tdoomed\n' + else: + links += '\tfresh\n' + self.send_html(links) + + def route_changes_stable(self): + self.refetch_pass() + self.send_raw(b"

    CHANGES-STABLE

    ", "text/html") + + def route_changes_moved(self): + pass1 = self.refetch_pass() == 1 + self.send_raw( + b"

    CHANGES-MOVED-V%d

    " % (1 if pass1 else 2), + "text/html", + ) + + def route_changes_stable_bin(self): + self.refetch_pass() + self.send_raw( + b"CHANGES-STABLE-BIN\n" + b"\x00\x01\x02\xff" * 512, + "application/octet-stream", + ) + + def route_changes_moved_bin(self): + pass1 = self.refetch_pass() == 1 + self.send_raw( + b"CHANGES-MOVED-BIN-V%d\n" % (1 if pass1 else 2) + + b"\x03\x02\x01\xfe" * 512, + "application/octet-stream", + ) + + def route_changes_doomed(self): + self.refetch_pass() + self.send_raw(b"

    CHANGES-DOOMED

    ", "text/html") + + def route_changes_fresh(self): + self.refetch_pass() + self.send_raw(b"

    CHANGES-FRESH

    ", "text/html") + + def route_changes_redir(self): + self.send_response(302) + self.send_header("Location", "/changes/redirtarget.html") + self.send_header("Content-Length", "0") + self.end_headers() + + def route_changes_redirtarget(self): + self.refetch_pass() + self.send_raw(b"

    CHANGES-REDIR

    ", "text/html") + + # Direct-to-disk, and the first body fetch of each pass dies mid-transfer: + # the retry notifies the same file a second time, so the accumulator has to + # keep the pre-run sample from the first notify. + FLAKY = b"CHANGES-FLAKY\n" + b"\x07\x06\x05\x04" * 512 + + def route_changes_flaky(self): + if self.refetch_pass() % 2 == 1: + self.send_truncated(self.FLAKY, "application/octet-stream") + else: + self.send_raw(self.FLAKY, "application/octet-stream") + ROUTES = { "/cookies/entrance.php": route_entrance, "/cookies/second.php": route_second, @@ -1588,6 +1664,16 @@ def route_maxrecv_404(self): "/codec/bad.html": route_codec_bad, "/codec/bin.dat": route_codec_bin, "/codec/ae.html": route_codec_ae, + "/changes/index.html": route_changes_index, + "/changes/stable.html": route_changes_stable, + "/changes/moved.html": route_changes_moved, + "/changes/stable.bin": route_changes_stable_bin, + "/changes/moved.bin": route_changes_moved_bin, + "/changes/doomed.html": route_changes_doomed, + "/changes/fresh.html": route_changes_fresh, + "/changes/redir.html": route_changes_redir, + "/changes/redirtarget.html": route_changes_redirtarget, + "/changes/flaky.bin": route_changes_flaky, "/upcodec/index.html": route_upcodec_index, "/upcodec/mem.html": route_upcodec_mem, "/upcodec/disk.bin": route_upcodec_disk, diff --git a/tests/webhttrack-smoke.sh b/tests/webhttrack-smoke.sh index 379ecf896..dd09101db 100644 --- a/tests/webhttrack-smoke.sh +++ b/tests/webhttrack-smoke.sh @@ -46,12 +46,14 @@ cat >"$stubdir/x-www-browser" <&2 # Also fetch an option page and require a rendered title='' tooltip: proves the # option template expands and the \${html:} filter escapes into the attribute. -# option9 additionally proves the WARC control renders with its expanded label. +# option9 additionally proves the WARC and change-report controls render with +# their expanded labels. opturl="\${1%/}/server/option2.html" warcurl="\${1%/}/server/option9.html" if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html && opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='" && - warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC; then + warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC && + printf '%s' "\$warc" | grep -qaF 'name="changes"' && printf '%s' "\$warc" | grep -qaF hts-changes.json; then echo PASS >"$marker" else echo "FAIL: unexpected response from \$1" >"$marker" From f7f428bc93bb8c2910dcca9dd09820a9f0ed7b93 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 18:08:04 +0200 Subject: [PATCH 02/11] changes: never leave a stale report behind A crawl that mirrored nothing created no accumulator, so hts_changes_close_opt returned without writing and the previous run's report stayed on disk as if it described this one. Write it whenever --changes is on. The no-data rollback is the deliberate exception, and is now documented: it restores the previous cache generation, so leaving the matching report alone is the consistent behaviour. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- html/changes.html | 3 +++ src/htschanges.c | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/html/changes.html b/html/changes.html index 348089daf..166827da1 100644 --- a/html/changes.html +++ b/html/changes.html @@ -208,6 +208,9 @@

    Notes

  • A file listed under gone is only deleted when --purge-old is on. Left in place it drops out of the mirror's index, so it is reported once and not again.
    • +
  • A run that transfers no data at all is rolled back: HTTrack restores the +previous cache generation and leaves the previous report in place, so a lost +connection does not overwrite a good report with an empty one.
  • Content diffs, and keeping the previous copy of a changed page, are out of scope: both change what a mirror directory contains.
    • diff --git a/src/htschanges.c b/src/htschanges.c index 726a29a84..3b10c09ea 100644 --- a/src/htschanges.c +++ b/src/htschanges.c @@ -512,7 +512,10 @@ void hts_changes_report(httrackp *opt, String *out) { } void hts_changes_close_opt(httrackp *opt) { - hts_changes *changes = (hts_changes *) opt->changes_state; + /* changes_get(), not the raw field: a crawl that mirrored nothing still owes + the user a report, and a stale one from the previous run must not survive + on disk as if it described this one. */ + hts_changes *changes = changes_get(opt); char catbuff[CATBUFF_SIZE]; const char *path; String report = STRING_EMPTY; From 11c0aa59df7236200ff0bdadd879778f1a435b66 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 18:10:20 +0200 Subject: [PATCH 03/11] changes: drop em dashes from the format page Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- html/changes.html | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/html/changes.html b/html/changes.html index 166827da1..aa776ea95 100644 --- a/html/changes.html +++ b/html/changes.html @@ -122,8 +122,8 @@

    What "changed" means



    -Where no digest can be taken on either side — the cache is disabled, or the -previous copy is gone — the report falls back to the transfer signal, and a +Where no digest can be taken on either side, because the cache is disabled or +the previous copy is gone, the report falls back to the transfer signal, and a server that answers 200 rather than 304 reads as changed. Keeping the cache on (the default) is what makes the report precise. @@ -132,34 +132,34 @@

    What "changed" means

    Fields

      -
    • schema — format version, currently 1. It is bumped only +
    • schema: format version, currently 1. It is bumped only on an incompatible change; new fields may appear without one.
      • -
    • generator — the HTTrack build that wrote the file.
      • -
    • date — when the report was written, UTC, YYYY-MM-DDThh:mm:ssZ.
      • -
    • first_crawl — true when no index of a previous mirror +
    • generator: the HTTrack build that wrote the file.
      • +
    • date: when the report was written, UTC, YYYY-MM-DDThh:mm:ssZ.
      • +
    • first_crawl: true when no index of a previous mirror (hts-cache/old.lst) was found, so there was nothing to compare against and everything is listed as new.
      • -
    • partial — true when the report ran out of memory and lists only +
    • partial: true when the report ran out of memory and lists only part of the mirror.
      • -
    • purged — true when --purge-old was in effect, so the +
    • purged: true when --purge-old was in effect, so the files under gone were also deleted from disk.
      • -
    • counts — the size of each of the four lists.
      • -
    • new, changed, unchanged, gone — the +
    • counts: the size of each of the four lists.
      • +
    • new, changed, unchanged, gone: the lists themselves. Every mirrored file appears in exactly one of them.
    Each entry is an object:
      -
    • url — the absolute URL the file came from. Empty under +
    • url: the absolute URL the file came from. Empty under gone: deletions are computed from the mirror's file index, which records paths, not URLs.
      • -
    • file — the path relative to the mirror root, with forward +
    • file: the path relative to the mirror root, with forward slashes. This is the entry's identity: a URL and a redirect that resolve to the same local file are one entry, not two.
      • -
    • size — the mirrored file's size in bytes, absent when the file +
    • size: the mirrored file's size in bytes, absent when the file is not on disk.
      • -
    • previous_size — under changed only, the size of the +
    • previous_size: under changed only, the size of the copy the previous run left.
    From a7ed9c3ca0cf4cbf35fa9c78d34d478142c10f02 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 18:35:43 +0200 Subject: [PATCH 04/11] changes: fix the review's three findings The size shortcut compared rendered on-disk sizes even for parsed pages, whose payload digests describe something else entirely, so it decided the outcome before the payload comparison could run: a page whose payload never changed but whose rewritten links moved read as changed. It now only applies when both digests describe the file on disk. file_notify() reaches the accumulator from the FTP download thread as well as the main one, and the lazy allocation, the coucal write and the entries realloc were all unguarded. Every entry point now takes a mutex, and the HTML hook does its cache read before taking it, since that read can itself re-enter file_notify() and move the array. Two fixtures cover what nothing did: a gzipped direct-to-disk body that changes at constant length (without the pre-sample before the decoded temp is renamed, it reads as unchanged), and a page with a fixed payload behind a redirect whose target is renamed, so its file on disk changes length while its bytes do not. Both were checked against builds with the respective fix reverted. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- src/htschanges.c | 114 +++++++++++++++++++++++++----------- tests/88_local-changes.test | 42 +++++++++---- tests/local-server.py | 48 +++++++++++++++ 3 files changed, 156 insertions(+), 48 deletions(-) diff --git a/src/htschanges.c b/src/htschanges.c index 3b10c09ea..5198c1c38 100644 --- a/src/htschanges.c +++ b/src/htschanges.c @@ -42,6 +42,7 @@ Please visit our Website: http://www.httrack.com #include "htslib.h" #include "htsmd5.h" #include "htssafe.h" +#include "htsthread.h" #include "htstools.h" #include "coucal/coucal.h" #include "md5.h" @@ -266,6 +267,10 @@ hts_change_bucket hts_changes_classify(hts_boolean rewritten, /* Engine hooks */ /* ------------------------------------------------------------ */ +/* FTP transfers reach file_notify() from their own detached thread + (htsftp.c:631/635), so every entry point below runs under this lock. */ +static htsmutex changes_mutex = HTSMUTEX_INIT; + /* The live accumulator, created on first use; NULL when --changes is off. */ static hts_changes *changes_get(httrackp *opt) { if (!opt->changes) @@ -278,7 +283,7 @@ static hts_changes *changes_get(httrackp *opt) { void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, const char *save, hts_boolean rewritten, hts_boolean not_updated) { - hts_changes *const changes = changes_get(opt); + hts_changes *changes; char BIGSTK file[HTS_URLMAXSIZE * 2]; hts_boolean is_new; intptr_t slot; @@ -286,25 +291,35 @@ void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, /* Engine-generated scaffolding (the top index) carries no URL and is not a mirrored resource. */ - if (changes == NULL || save == NULL || !strnotempty(save) || adr == NULL || - fil == NULL || (!strnotempty(adr) && !strnotempty(fil))) + if (save == NULL || !strnotempty(save) || adr == NULL || fil == NULL || + (!strnotempty(adr) && !strnotempty(fil))) + return; + + hts_mutexlock(&changes_mutex); + changes = changes_get(opt); + if (changes == NULL) { + hts_mutexrelease(&changes_mutex); return; + } hts_savename_listed(&opt->state.strc, save, file, sizeof(file)); slot = changes_slot(changes, file, &is_new); - if (slot < 0) + if (slot < 0) { + hts_mutexrelease(&changes_mutex); return; + } entry = &changes->entries[slot]; if (!is_new) { /* Retry, or a second call site for the same file: the pre-run state was already sampled and the copy on disk may no longer be it. */ entry->rewritten = entry->rewritten || rewritten; + hts_mutexrelease(&changes_mutex); return; } { - char BIGSTK url[HTS_URLMAXSIZE * 2]; + char BIGSTK url[HTS_URLMAXSIZE * 4 + 8]; /* holds adr + fil + a scheme */ url[0] = '\0'; if (!link_has_authority(adr)) @@ -324,66 +339,90 @@ void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, the last moment those bytes exist. */ if (entry->on_disk && rewritten) entry->has_prev = digest_file(save, entry->prev_digest); + hts_mutexrelease(&changes_mutex); } void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r, const char *adr, const char *fil, const char *save) { - hts_changes *const changes = changes_get(opt); + hts_changes *changes; char BIGSTK file[HTS_URLMAXSIZE * 2]; char BIGSTK location[HTS_URLMAXSIZE * 2]; - changes_entry *entry; + unsigned char new_digest[DIGEST_SIZE]; + unsigned char prev_digest[DIGEST_SIZE]; + hts_boolean has_prev = HTS_FALSE; intptr_t slot; htsblk prev; - if (changes == NULL || r->adr == NULL || r->size < 0 || save == NULL || - !strnotempty(save)) - return; - hts_savename_listed(&opt->state.strc, save, file, sizeof(file)); - /* file_notify() records the entry; this call only refines it. */ - slot = changes_find(changes, file); - if (slot < 0) + if (r->adr == NULL || r->size < 0 || save == NULL || !strnotempty(save)) return; - entry = &changes->entries[slot]; /* On disk this is the payload plus rewritten links and a footer dated by the crawl, so it differs every run; compare payloads, the previous one being the body the cache kept. */ - entry->has_new = HTS_TRUE; - digest_mem(r->adr, (size_t) r->size, entry->new_digest); - entry->has_prev = HTS_FALSE; + digest_mem(r->adr, (size_t) r->size, new_digest); prev = cache_read_ro(opt, cache, adr, fil, "", location); if (HTTP_IS_OK(prev.statuscode) && prev.adr != NULL && prev.size >= 0) { - digest_mem(prev.adr, (size_t) prev.size, entry->prev_digest); - entry->has_prev = HTS_TRUE; + digest_mem(prev.adr, (size_t) prev.size, prev_digest); + has_prev = HTS_TRUE; } freet(prev.adr); + + /* Only now take the lock and look the entry up: cache_read_ro() can itself + reach file_notify(), which would move the entries array. */ + hts_mutexlock(&changes_mutex); + changes = changes_get(opt); + /* file_notify() records the entry; this call only refines it. */ + if (changes != NULL) { + hts_savename_listed(&opt->state.strc, save, file, sizeof(file)); + slot = changes_find(changes, file); + if (slot >= 0) { + changes_entry *const entry = &changes->entries[slot]; + + entry->has_new = HTS_TRUE; + memcpy(entry->new_digest, new_digest, DIGEST_SIZE); + entry->has_prev = has_prev; + if (has_prev) + memcpy(entry->prev_digest, prev_digest, DIGEST_SIZE); + } + } + hts_mutexrelease(&changes_mutex); } void hts_changes_gone(httrackp *opt, const char *file) { - hts_changes *const changes = changes_get(opt); + hts_changes *changes; hts_boolean is_new; intptr_t slot; - if (changes == NULL || file == NULL || !strnotempty(file)) + if (file == NULL || !strnotempty(file)) return; - slot = changes_slot(changes, file, &is_new); - if (slot < 0 || !is_new) - return; - changes->entries[slot].url = strdupt(""); - changes->entries[slot].bucket = HTS_CHANGE_GONE; - changes->entries[slot].listed_prev = HTS_TRUE; + hts_mutexlock(&changes_mutex); + changes = changes_get(opt); + if (changes != NULL) { + slot = changes_slot(changes, file, &is_new); + if (slot >= 0 && is_new) { + changes->entries[slot].url = strdupt(""); + changes->entries[slot].bucket = HTS_CHANGE_GONE; + changes->entries[slot].listed_prev = HTS_TRUE; + } + } + hts_mutexrelease(&changes_mutex); } void hts_changes_previous(httrackp *opt, const char *file) { - hts_changes *const changes = changes_get(opt); + hts_changes *changes; intptr_t slot; - if (changes == NULL || file == NULL) + if (file == NULL) return; - changes->old_index = HTS_TRUE; - slot = changes_find(changes, file); - if (slot >= 0) - changes->entries[slot].listed_prev = HTS_TRUE; + hts_mutexlock(&changes_mutex); + changes = changes_get(opt); + if (changes != NULL) { + changes->old_index = HTS_TRUE; + slot = changes_find(changes, file); + if (slot >= 0) + changes->entries[slot].listed_prev = HTS_TRUE; + } + hts_mutexrelease(&changes_mutex); } /* ------------------------------------------------------------ */ @@ -414,7 +453,10 @@ static void changes_resolve(hts_changes *changes, httrackp *opt) { entry->file); entry->size = fsize_utf8(path); if (entry->rewritten && entry->existed) { - if (entry->size >= 0 && entry->prev_size >= 0 && + /* The size shortcut only holds when both digests describe the file on + disk; has_new means they are payload digests, and a parsed page's + rendered size moves with its links and footer. */ + if (!entry->has_new && entry->size >= 0 && entry->prev_size >= 0 && entry->size != entry->prev_size) { have_digests = HTS_TRUE; /* different lengths: no need to hash */ digests_equal = HTS_FALSE; @@ -437,6 +479,8 @@ static void changes_resolve(hts_changes *changes, httrackp *opt) { static const char *const bucket_names[HTS_CHANGE_BUCKETS] = { "new", "changed", "unchanged", "gone"}; +/* Callers below run after the crawl's threads are done, so they take no lock. + */ void hts_changes_report(httrackp *opt, String *out) { hts_changes *const changes = (hts_changes *) opt->changes_state; size_t counts[HTS_CHANGE_BUCKETS]; diff --git a/tests/88_local-changes.test b/tests/88_local-changes.test index 2c72f7a51..95f355c2f 100644 --- a/tests/88_local-changes.test +++ b/tests/88_local-changes.test @@ -58,8 +58,9 @@ common=(-O "$out" --quiet --disable-security-limits --robots=0 --timeout=30 --ch # Report field extractor: bucket counts, and the files listed under a bucket. count() { sed -n 's/.*"'"$1"'": \([0-9]*\).*/\1/p' "$report" | head -1; } +# tr: python prints CRLF on Windows, and the expectations below are LF. listed() { - "$python" - "$report" "$1" <<'EOF' + "$python" - "$report" "$1" <<'EOF' | tr -d '\r' import json import sys @@ -88,9 +89,10 @@ test -s "$report" || { } expect "pass 1 declares a first crawl" "true" \ "$(sed -n 's/.*"first_crawl": \([a-z]*\).*/\1/p' "$report" | head -1)" -# index, stable.html, moved.html, stable.bin, moved.bin, doomed.html, -# redirtarget.html, flaky.bin -expect "pass 1 counts 8 new" "8" "$(count new)" +# Every mirrored file: index, stable.html, moved.html, stable.bin, moved.bin, +# doomed.html, redirtarget.html, flaky.bin, coded.bin, codedstable.bin, +# sized.html and the redirect target it points at. +expect "pass 1 counts 12 new" "12" "$(count new)" expect "pass 1 counts 0 changed" "0" "$(count changed)" expect "pass 1 counts 0 gone" "0" "$(count gone)" grep -aq "first crawl" "${out}/hts-log.txt" || { @@ -104,20 +106,34 @@ expect "pass 2 is not a first crawl" "false" \ "$(sed -n 's/.*"first_crawl": \([a-z]*\).*/\1/p' "$report" | head -1)" host="127.0.0.1_${port}" -expect "new is fresh.html only" "${host}/changes/fresh.html" "$(listed new)" -expect "gone is doomed.html only" "${host}/changes/doomed.html" "$(listed gone)" +# The redirect behind sized.html now points at a much longer target name. +long="${host}/changes/$(printf 's%.0s' $(seq 40)).html" +expect "new is fresh.html and the renamed redirect target" \ + "$(printf '%s\n%s' "${host}/changes/fresh.html" "$long" | sort)" \ + "$(listed new | sort)" +expect "gone is doomed.html and the old redirect target" \ + "$(printf '%s\n%s' "${host}/changes/doomed.html" "${host}/changes/s.html" | sort)" \ + "$(listed gone | sort)" # index.html changes because its link list does; moved.* change their bodies. -expect "changed is exactly the three moved resources" \ - "$(printf '%s\n%s\n%s' "${host}/changes/index.html" \ - "${host}/changes/moved.bin" "${host}/changes/moved.html")" \ +# coded.bin arrives gzipped and direct-to-disk, so its previous copy has to be +# sampled before the decoded temp is renamed over it; it is the same length on +# both passes, so only a digest separates it from codedstable.bin. +expect "changed is exactly the four moved resources" \ + "$(printf '%s\n%s\n%s\n%s' "${host}/changes/coded.bin" \ + "${host}/changes/index.html" "${host}/changes/moved.bin" \ + "${host}/changes/moved.html")" \ "$(listed changed | sort)" # The teeth: re-served byte for byte, 200, no Last-Modified, no ETag. flaky.bin # gets there through a failed transfer and a retry, so its file is notified # twice; redirtarget.html arrives behind a 302. -expect "unchanged is exactly the four stable resources" \ - "$(printf '%s\n%s\n%s\n%s' "${host}/changes/flaky.bin" \ - "${host}/changes/redirtarget.html" "${host}/changes/stable.bin" \ - "${host}/changes/stable.html")" \ +# sized.html is the sharpest of these: its payload is byte-identical, but the +# rewritten link to the renamed redirect target makes the file on disk change +# length, so a size comparison alone would call it changed. +expect "unchanged is exactly the six stable resources" \ + "$(printf '%s\n%s\n%s\n%s\n%s\n%s' "${host}/changes/codedstable.bin" \ + "${host}/changes/flaky.bin" "${host}/changes/redirtarget.html" \ + "${host}/changes/sized.html" "${host}/changes/stable.bin" \ + "${host}/changes/stable.html" | sort)" \ "$(listed unchanged | sort)" # Every mirrored file appears once and only once, across all four buckets. diff --git a/tests/local-server.py b/tests/local-server.py index 83320c4b9..9e9ba8c02 100755 --- a/tests/local-server.py +++ b/tests/local-server.py @@ -1584,6 +1584,9 @@ def route_changes_index(self): "moved.bin", "redir.html", "flaky.bin", + "coded.bin", + "codedstable.bin", + "sized.html", ) ) if self.refetch_pass() == 1: @@ -1641,6 +1644,45 @@ def route_changes_redirtarget(self): # keep the pre-run sample from the first notify. FLAKY = b"CHANGES-FLAKY\n" + b"\x07\x06\x05\x04" * 512 + # A page whose payload never changes, linking through a redirect whose + # target is renamed between passes: the rewritten link makes the file on + # disk change length while the payload is byte-identical. Only a payload + # comparison can call it unchanged. + SIZED_SHORT = "s.html" + SIZED_LONG = "s" * 40 + ".html" + + def route_changes_sized(self): + self.refetch_pass() + self.send_raw( + b't', + "text/html", + ) + + def route_changes_sizedredir(self): + target = self.SIZED_SHORT if self.refetch_pass() == 1 else self.SIZED_LONG + self.send_response(302) + self.send_header("Location", "/changes/" + target) + self.send_header("Content-Length", "0") + self.end_headers() + + def route_changes_sizedtarget(self): + self.refetch_pass() + self.send_raw(b"

    SIZED-TARGET

    ", "text/html") + + # Content-Encoding on a direct-to-disk body: the decoded temp is renamed + # over the mirror file, so the previous copy has to be sampled before the + # rename. Same length on both passes, so only a digest can tell them apart. + def route_changes_coded(self): + pass1 = self.refetch_pass() == 1 + body = b"CHANGES-CODED-V%d\n" % (1 if pass1 else 2) + b"\x11\x22" * 1024 + self.send_coded(self.gzipped(body), "application/octet-stream") + + # Control: same coding, same bytes on both passes. + def route_changes_coded_stable(self): + self.refetch_pass() + body = b"CHANGES-CODED-STABLE\n" + b"\x33\x44" * 1024 + self.send_coded(self.gzipped(body), "application/octet-stream") + def route_changes_flaky(self): if self.refetch_pass() % 2 == 1: self.send_truncated(self.FLAKY, "application/octet-stream") @@ -1674,6 +1716,12 @@ def route_changes_flaky(self): "/changes/redir.html": route_changes_redir, "/changes/redirtarget.html": route_changes_redirtarget, "/changes/flaky.bin": route_changes_flaky, + "/changes/coded.bin": route_changes_coded, + "/changes/sized.html": route_changes_sized, + "/changes/sizedredir.html": route_changes_sizedredir, + "/changes/" + SIZED_SHORT: route_changes_sizedtarget, + "/changes/" + SIZED_LONG: route_changes_sizedtarget, + "/changes/codedstable.bin": route_changes_coded_stable, "/upcodec/index.html": route_upcodec_index, "/upcodec/mem.html": route_upcodec_mem, "/upcodec/disk.bin": route_upcodec_disk, From 08b32f4878fcfc4f5bcbddcf6842b6978a74eee9 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 18:36:21 +0200 Subject: [PATCH 05/11] changes: document the renamed-file case in the format page Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- html/changes.html | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/html/changes.html b/html/changes.html index aa776ea95..bf3289462 100644 --- a/html/changes.html +++ b/html/changes.html @@ -208,6 +208,10 @@

    Notes

  • A file listed under gone is only deleted when --purge-old is on. Left in place it drops out of the mirror's index, so it is reported once and not again.
    • +
  • A resource whose local file name changed since the previous mirror (a new +MIME type, say) is reported as new under its new name; the old name is +reported as gone only if the file is still on disk. The two entries are +not paired.
  • A run that transfers no data at all is rolled back: HTTrack restores the previous cache generation and leaves the previous report in place, so a lost connection does not overwrite a good report with an empty one.
    • From 6a20dfd9983b8aa2df8546c947a266013d968103 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 18:37:17 +0200 Subject: [PATCH 06/11] changes: refresh two stale test comments Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- tests/88_local-changes.test | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tests/88_local-changes.test b/tests/88_local-changes.test index 95f355c2f..d13dc1bf9 100644 --- a/tests/88_local-changes.test +++ b/tests/88_local-changes.test @@ -1,8 +1,8 @@ #!/bin/bash -# --changes (#714): the report must match the mirror's real deltas. The routes -# answer 200 with no validators on both passes, so the transfer signal alone -# would call everything changed; stable.html/stable.bin land in "unchanged" -# only if the payload is actually compared. +# --changes (#714): the report must match the mirror's real deltas. Every route +# answers 200 with no validators on both passes, so the transfer signal alone +# would call the whole site changed; the stable resources land in "unchanged" +# only if payloads are really compared. set -eu : "${top_srcdir:=..}" @@ -52,8 +52,8 @@ which httrack >/dev/null || { out="${tmpdir}/crawl" mkdir "$out" report="${out}/hts-changes.json" -# Purging stays off until the last pass: the deleted set must be computed even -# when nothing acts on it. +# Purging off throughout: the deleted set must be computed with nothing acting +# on it. common=(-O "$out" --quiet --disable-security-limits --robots=0 --timeout=30 --changes --purge-old=0 --retries=2) # Report field extractor: bucket counts, and the files listed under a bucket. From b099e303e5216766e776ba960281b7471fdf3991 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 19:00:24 +0200 Subject: [PATCH 07/11] Merge origin/master into feat/change-report Both sides appended to tests/Makefile.am's TESTS; kept master's 86_local-proxytrack-cache-longfields.test alongside 88_local-changes.test. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- src/htsback.c | 38 +++--- src/htsback.h | 4 + src/htslib.c | 20 ++-- src/htsselftest.c | 62 +++++++++- src/proxy/store.c | 70 +++++++---- .../86_local-proxytrack-cache-longfields.test | 111 ++++++++++++++++++ tests/Makefile.am | 1 + 7 files changed, 254 insertions(+), 52 deletions(-) create mode 100755 tests/86_local-proxytrack-cache-longfields.test diff --git a/src/htsback.c b/src/htsback.c index 0643d6dab..63b4eb808 100644 --- a/src/htsback.c +++ b/src/htsback.c @@ -126,6 +126,22 @@ void back_free(struct_back ** sback) { above a normal handshake. The last candidate still gets the full timeout. */ #define HTS_CONNECT_FALLBACK_TIMEOUT 10 +void back_read_ftp_result(FILE *fp, htsblk *r) { + size_t j = 0; + + if (fscanf(fp, "%d ", &r->statuscode) != 1) + r->statuscode = STATUSCODE_INVALID; + // an external helper writes this file: stop at capacity, not at EOF + while (j + 1 < sizeof(r->msg)) { + const int c = fgetc(fp); + + if (c == EOF) + break; + r->msg[j++] = (char) c; + } + r->msg[j] = '\0'; +} + int back_connect_fallback_due(int addr_index, int addr_count, int elapsed, int timeout) { int deadline; @@ -2968,7 +2984,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache, back[i].r.msg[0] = '\0'; strncatbuff(back[i].r.msg, tmp, sizeof(back[i].r.msg) - 2); if (!strnotempty(back[i].r.msg)) { - sprintf(back[i].r.msg, "SSL/TLS error %d", err_code); + htsblk_failf(&back[i].r, "SSL/TLS error %d", err_code); } deletehttp(&back[i].r); back[i].r.soc = INVALID_SOCKET; @@ -3041,16 +3057,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache, FOPEN(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"), "rb"); if (fp) { - int j = 0; - - fscanf(fp, "%d ", &(back[i].r.statuscode)); - while(!feof(fp)) { - int c = fgetc(fp); - - if (c != EOF) - back[i].r.msg[j++] = c; - } - back[i].r.msg[j++] = '\0'; + back_read_ftp_result(fp, &back[i].r); fclose(fp); UNLINK(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok")); strcpybuff(fconcat @@ -3351,10 +3358,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache, deleteaddr(&back[i].r); if (back[i].r.size < back[i].r.totalsize) back[i].r.statuscode = STATUSCODE_CONNERROR; // recatch - sprintf(back[i].r.msg, - "Incorrect length (" LLintP " Bytes, " LLintP - " expected)", (LLint) back[i].r.size, - (LLint) back[i].r.totalsize); + htsblk_failf(&back[i].r, + "Incorrect length (" LLintP " Bytes, " LLintP + " expected)", + (LLint) back[i].r.size, + (LLint) back[i].r.totalsize); } else { // Un warning suffira.. hts_log_print(opt, LOG_WARNING, diff --git a/src/htsback.h b/src/htsback.h index ec0c23b9f..ef0ed8900 100644 --- a/src/htsback.h +++ b/src/htsback.h @@ -74,6 +74,10 @@ void back_free(struct_back ** sback); // backing #define BACK_ADD_TEST "(dummy)" #define BACK_ADD_TEST2 "(dummy2)" +/* Parse an external FTP helper's " " result file into r, + clipping the message to r->msg. */ +void back_read_ftp_result(FILE *fp, htsblk *r); + int back_index(httrackp * opt, struct_back * sback, const char *adr, const char *fil, const char *sav); int back_available(const struct_back * sback); diff --git a/src/htslib.c b/src/htslib.c index 3c9a092b0..873170fe6 100644 --- a/src/htslib.c +++ b/src/htslib.c @@ -709,11 +709,11 @@ T_SOC http_xfopen(httrackp *opt, int mode, int treat, int waitconnect, #ifdef _WIN32 int last_errno = WSAGetLastError(); - sprintf(retour->msg, "Connect error: %s", strerror(last_errno)); + htsblk_failf(retour, "Connect error: %s", strerror(last_errno)); #else int last_errno = errno; - sprintf(retour->msg, "Connect error: %s", strerror(last_errno)); + htsblk_failf(retour, "Connect error: %s", strerror(last_errno)); #endif } } @@ -2241,13 +2241,13 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port, #ifdef _WIN32 int last_errno = WSAGetLastError(); - sprintf(retour->msg, "Unable to create a socket: %s", - strerror(last_errno)); + htsblk_failf(retour, "Unable to create a socket: %s", + strerror(last_errno)); #else int last_errno = errno; - sprintf(retour->msg, "Unable to create a socket: %s", - strerror(last_errno)); + htsblk_failf(retour, "Unable to create a socket: %s", + strerror(last_errno)); #endif } return INVALID_SOCKET; // erreur création socket impossible @@ -2313,13 +2313,13 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port, #ifdef _WIN32 const int last_errno = WSAGetLastError(); - sprintf(retour->msg, "Unable to connect to the server: %s", - strerror(last_errno)); + htsblk_failf(retour, "Unable to connect to the server: %s", + strerror(last_errno)); #else const int last_errno = errno; - sprintf(retour->msg, "Unable to connect to the server: %s", - strerror(last_errno)); + htsblk_failf(retour, "Unable to connect to the server: %s", + strerror(last_errno)); #endif } /* Close the socket and notify the error!!! */ diff --git a/src/htsselftest.c b/src/htsselftest.c index a9ff5fb3b..49d8500a7 100644 --- a/src/htsselftest.c +++ b/src/htsselftest.c @@ -43,6 +43,7 @@ Please visit our Website: http://www.httrack.com #include "htsglobal.h" #include "htscore.h" +#include "htsback.h" #include "htsdefines.h" #include "htslib.h" #include "htsalias.h" @@ -626,10 +627,12 @@ static int string_safety_selftests(void) { char big[4 * sizeof(r.msg)]; /* contenttype abuts msg, so a one-past-the-end store lands in it rather - than in padding; checking it is this block's canary */ -#define NEIGHBOURS_INTACT() (r.contenttype[0] == '\0' && r.statuscode == 1234) + than in padding. Poison it: a stray NUL is invisible against zeroes, + and a stray NUL is exactly what an off-by-one terminator writes. */ +#define NEIGHBOURS_INTACT() (r.contenttype[0] == '#' && r.statuscode == 1234) memset(&r, 0, sizeof(r)); + memset(r.contenttype, '#', sizeof(r.contenttype)); r.statuscode = 1234; memset(r.msg, '#', sizeof(r.msg)); @@ -661,6 +664,61 @@ static int string_safety_selftests(void) { #undef NEIGHBOURS_INTACT } + /* back_read_ftp_result: the helper's result file is external input, so an + over-long message must stop at msg[]'s capacity */ + { + htsblk r; + size_t k; + + /* poisoned so a short message cannot pass on leftovers, and so a stray + NUL past msg[] is visible in the neighbour */ +#define FTP_RESULT_CASE(BODY) \ + do { \ + FILE *fp_ = tmpfile(); \ + \ + if (fp_ == NULL) \ + return 1; \ + BODY; \ + rewind(fp_); \ + memset(&r, 0, sizeof(r)); \ + memset(r.msg, '#', sizeof(r.msg)); \ + memset(r.contenttype, '#', sizeof(r.contenttype)); \ + back_read_ftp_result(fp_, &r); \ + fclose(fp_); \ + if (r.contenttype[0] != '#') \ + return 1; \ + } while (0) + + /* over capacity: clipped to 79 payload bytes plus the NUL */ + FTP_RESULT_CASE({ + fprintf(fp_, "226 "); + for (k = 0; k < 4 * sizeof(r.msg); k++) + fputc('q', fp_); + }); + if (r.statuscode != 226 || strlen(r.msg) != sizeof(r.msg) - 1) + return 1; + for (k = 0; k < sizeof(r.msg) - 1; k++) { + if (r.msg[k] != 'q') + return 1; + } + + /* well under capacity: nothing padded, nothing eaten off the end */ + FTP_RESULT_CASE(fprintf(fp_, "550 no such file")); + if (r.statuscode != 550 || strcmp(r.msg, "no such file") != 0) + return 1; + + /* a byte over 0x7f must not read as EOF and cut the message short */ + FTP_RESULT_CASE(fprintf(fp_, "226 \xff ok")); + if (r.statuscode != 226 || strcmp(r.msg, "\xff ok") != 0) + return 1; + + /* unparseable status: the message still loads, the code reports failure */ + FTP_RESULT_CASE(fprintf(fp_, "not-a-number here")); + if (r.statuscode != STATUSCODE_INVALID) + return 1; +#undef FTP_RESULT_CASE + } + /* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves n_eval == 2 on a double-eval macro. */ { diff --git a/src/proxy/store.c b/src/proxy/store.c index 4060d62df..e9b1c3c38 100644 --- a/src/proxy/store.c +++ b/src/proxy/store.c @@ -878,12 +878,17 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags) sprintf(headers + headersSize, "%s: "LLintP"\r\n", field, (LLint)(value)); \ (headersSize) += (int) strlen(headers + headersSize); \ } while(0) -#define ZIP_READFIELD_STRING(line, value, refline, refvalue) do { \ - if (line[0] != '\0' && strfield2(line, refline)) { \ - strcpy(refvalue, value); \ - line[0] = '\0'; \ - } \ -} while(0) +/* refvalue_size is mandatory: the cache line is bounded only by the line + buffer, not by the destination. Clip rather than reject, since an + engine-written field can be wider than ours. */ +#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \ + do { \ + if (line[0] != '\0' && strfield2(line, refline)) { \ + (refvalue)[0] = '\0'; \ + strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1); \ + line[0] = '\0'; \ + } \ + } while (0) #define ZIP_READFIELD_INT(line, value, refline, refvalue) do { \ if (line[0] != '\0' && strfield2(line, refline)) { \ int intval = 0; \ @@ -1086,16 +1091,23 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url, value++; ZIP_READFIELD_INT(line, value, "X-In-Cache", dataincache); ZIP_READFIELD_INT(line, value, "X-Statuscode", r->statuscode); - ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg); // msg + ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg, + sizeof(r->msg)); ZIP_READFIELD_INT(line, value, "X-Size", r->size); // size - ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype - ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset); // contenttype - ZIP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified - ZIP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag - ZIP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved + ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype, + sizeof(r->contenttype)); + ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset, + sizeof(r->charset)); + ZIP_READFIELD_STRING(line, value, "Last-Modified", + r->lastmodified, sizeof(r->lastmodified)); + ZIP_READFIELD_STRING(line, value, "Etag", r->etag, + sizeof(r->etag)); + ZIP_READFIELD_STRING(line, value, "Location", r->location, + sizeof(location_default)); ZIP_READFIELD_STRING(line, value, "Content-Disposition", - r->cdispo); // Content-disposition - ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_); // Original save filename + r->cdispo, sizeof(r->cdispo)); + ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_, + sizeof(previous_save_)); if (line[0] != '\0') { int len = r->headers ? ((int) strlen(r->headers)) : 0; int nlen = @@ -2152,12 +2164,15 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) { return 0; } -#define HTTP_READFIELD_STRING(line, value, refline, refvalue) do { \ - if (line[0] != '\0' && strfield2(line, refline)) { \ - strcpy(refvalue, value); \ - line[0] = '\0'; \ - } \ -} while(0) +/* Same contract as ZIP_READFIELD_STRING, for the ARC reader's header lines. */ +#define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \ + do { \ + if (line[0] != '\0' && strfield2(line, refline)) { \ + (refvalue)[0] = '\0'; \ + strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1); \ + line[0] = '\0'; \ + } \ + } while (0) #define HTTP_READFIELD_INT(line, value, refline, refvalue) do { \ if (line[0] != '\0' && strfield2(line, refline)) { \ int intval = 0; \ @@ -2226,11 +2241,16 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url, *value = '\0'; for(value++; *value == ' ' || *value == '\t'; value++) ; HTTP_READFIELD_INT(line, value, "Content-Length", r->size); // size - HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype - HTTP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified - HTTP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag - HTTP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved - HTTP_READFIELD_STRING(line, value, "Content-Disposition", r->cdispo); // Content-disposition + HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype, + sizeof(r->contenttype)); + HTTP_READFIELD_STRING(line, value, "Last-Modified", + r->lastmodified, sizeof(r->lastmodified)); + HTTP_READFIELD_STRING(line, value, "Etag", r->etag, + sizeof(r->etag)); + HTTP_READFIELD_STRING(line, value, "Location", r->location, + sizeof(location_default)); + HTTP_READFIELD_STRING(line, value, "Content-Disposition", + r->cdispo, sizeof(r->cdispo)); if (line[0] != '\0') { int len = r->headers ? ((int) strlen(r->headers)) : 0; int nlen = diff --git a/tests/86_local-proxytrack-cache-longfields.test b/tests/86_local-proxytrack-cache-longfields.test new file mode 100755 index 000000000..f6c47aeab --- /dev/null +++ b/tests/86_local-proxytrack-cache-longfields.test @@ -0,0 +1,111 @@ +#!/bin/bash +# A cached header longer than proxytrack's field must be clipped to that field, +# not overflow into the next one (contenttype[64] abuts the location pointer). + +set -euo pipefail + +testdir=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=tests/testlib.sh +. "${testdir}/testlib.sh" + +dir=$(mktemp -d) +trap 'rm -rf "$dir"' EXIT + +pad() { printf "%${1}s" '' | tr ' ' "$2"; } + +# Longest surviving run of char $2 in file $1, or 0. +runlen() { + grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true +} + +# --- ARC reader (HTTP_READFIELD_STRING), no python needed ------------------- +# Each header overshoots its own destination, so a per-field clip is the only +# way to get every expected length right at once. +{ + printf 'HTTP/1.1 200 OK\r\n' + printf 'Content-Type: text/html%s\r\n' "$(pad 400 A)" + printf 'Etag: %s\r\n' "$(pad 400 E)" + printf 'Content-Disposition: %s\r\n' "$(pad 400 D)" + printf 'Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n' + printf 'Content-Length: 5\r\n\r\n' +} >"$dir/hdr" +printf 'hello' >"$dir/body" +alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body"))) +{ + printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n' + printf '2 0 test\n' + printf '\n\n' + printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen" + cat "$dir/hdr" "$dir/body" +} >"$dir/in.arc" + +proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || { + echo "FAIL: proxytrack crashed reading an ARC with over-long headers" >&2 + exit 1 +} +test -s "$dir/out.arc" || { + echo "FAIL: ARC entry dropped instead of clipped" >&2 + exit 1 +} +# Only contenttype survives into the ARC output; the over-long Etag and +# Content-Disposition above are still load-bearing, since bounding just one +# field leaves the others smashing the element (the run above would crash). +got=$(runlen "$dir/out.arc" A) +test "${got:-0}" = 54 || { + echo "FAIL: ARC content-type clipped to ${got:-0}, expected 54" >&2 + exit 1 +} + +# --- ZIP reader (ZIP_READFIELD_STRING) -------------------------------------- +python=$(find_python) || { + echo "python3 missing; ARC half only" >&2 + exit 0 +} + +"$python" - "$dir/in.zip" <<'EOF' +import sys, zipfile + +body = b"hello world" +meta = ( + "X-In-Cache: 1\r\n" + "X-StatusCode: 200\r\n" + "X-StatusMessage: ok%s\r\n" + "X-Size: %d\r\n" + "Content-Type: text/html%s\r\n" + "X-Charset: %s\r\n" + "Etag: %s\r\n" + "Content-Disposition: %s\r\n" + # a parseable date is required: the ARC writer dereferences it unchecked + "Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n" + "X-Save: out/page.html\r\n" + % ("M" * 400, len(body), "A" * 400, "C" * 400, "E" * 400, "D" * 400) +).encode() + +zi = zipfile.ZipInfo("example.com/page.html") +zi.compress_type = zipfile.ZIP_STORED +zi.extra = meta +with zipfile.ZipFile(sys.argv[1], "w") as z: + z.writestr(zi, body) +EOF + +proxytrack --convert "$dir/out2.arc" "$dir/in.zip" >/dev/null 2>&1 || { + echo "FAIL: proxytrack crashed reading a zip cache with over-long headers" >&2 + exit 1 +} +test -s "$dir/out2.arc" || { + echo "FAIL: zip entry dropped instead of clipped" >&2 + exit 1 +} +# three destinations, two capacities: msg[1024] keeps all 400 M's where a +# one-size-fits-all clip would cut it to 63 like the others +while read -r ch want; do + got=$(runlen "$dir/out2.arc" "$ch") + test "${got:-0}" = "$want" || { + echo "FAIL: zip field '$ch' clipped to ${got:-0}, expected $want" >&2 + exit 1 + } +done <<'EOF' +A 54 +C 63 +M 400 +EOF diff --git a/tests/Makefile.am b/tests/Makefile.am index 96201a204..bdef860af 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -184,6 +184,7 @@ TESTS = \ 83_webhttrack-argescape.test \ 84_webhttrack-mirror-verbatim.test \ 85_webhttrack-projpath.test \ + 86_local-proxytrack-cache-longfields.test \ 88_local-changes.test CLEANFILES = check-network_sh.cache From bcef3a109600874d53f3338efdb428a8d45c22ef Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 22:04:50 +0200 Subject: [PATCH 08/11] changes: translate the new GUI strings into the remaining 28 locales (#740) The feature PR added the two LANG_CHANGES* entries to lang.def with English and Francais only; every other locale fell back to English in the WebHTTrack form. Each file is written in its own declared charset. Signed-off-by: Xavier Roche Co-authored-by: Claude Opus 5 (1M context) --- lang/Bulgarian.txt | 4 ++++ lang/Castellano.txt | 4 ++++ lang/Cesky.txt | 4 ++++ lang/Chinese-BIG5.txt | 4 ++++ lang/Chinese-Simplified.txt | 4 ++++ lang/Croatian.txt | 4 ++++ lang/Dansk.txt | 4 ++++ lang/Deutsch.txt | 4 ++++ lang/Eesti.txt | 4 ++++ lang/Finnish.txt | 4 ++++ lang/Greek.txt | 4 ++++ lang/Italiano.txt | 4 ++++ lang/Japanese.txt | 4 ++++ lang/Macedonian.txt | 4 ++++ lang/Magyar.txt | 4 ++++ lang/Nederlands.txt | 4 ++++ lang/Norsk.txt | 4 ++++ lang/Polski.txt | 4 ++++ lang/Portugues-Brasil.txt | 4 ++++ lang/Portugues.txt | 4 ++++ lang/Romanian.txt | 4 ++++ lang/Russian.txt | 4 ++++ lang/Slovak.txt | 4 ++++ lang/Slovenian.txt | 4 ++++ lang/Svenska.txt | 4 ++++ lang/Turkish.txt | 4 ++++ lang/Ukrainian.txt | 4 ++++ lang/Uzbek.txt | 4 ++++ 28 files changed, 112 insertions(+) diff --git a/lang/Bulgarian.txt b/lang/Bulgarian.txt index e5cde4afd..9080c79f7 100644 --- a/lang/Bulgarian.txt +++ b/lang/Bulgarian.txt @@ -964,3 +964,7 @@ WARC archive name: Èìå íà WARC àðõèâà: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Íåçàäúëæèòåëíî áàçîâî èìå çà WARC àðõèâà; îñòàâåòå ïðàçíî çà àâòîìàòè÷íî èìåíóâàíå â èçõîäíàòà äèðåêòîðèÿ. +Report what changed since the previous mirror +Îò÷åò çà ïðîìåíèòå ñïðÿìî ïðåäèøíîòî îãëåäàëî +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Çàïèñâàíå è íà hts-changes.json ñúñ ñïèñúê íà íîâèòå, ïðîìåíåíèòå, íåïðîìåíåíèòå è èç÷åçíàëèòå ôàéëîâå ñïðÿìî ïðåäèøíîòî îãëåäàëî. diff --git a/lang/Castellano.txt b/lang/Castellano.txt index d394362f2..9b6cb9eb8 100644 --- a/lang/Castellano.txt +++ b/lang/Castellano.txt @@ -964,3 +964,7 @@ WARC archive name: Nombre del archivo WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Nombre base opcional para el archivo WARC; déjelo en blanco para nombrarlo automáticamente en el directorio de salida. +Report what changed since the previous mirror +Informar de los cambios desde la copia anterior +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Escribir también hts-changes.json con la lista de lo que esta captura deja como nuevo, modificado, sin cambios o desaparecido respecto a la copia anterior. diff --git a/lang/Cesky.txt b/lang/Cesky.txt index 4f177a44d..b785d53c0 100644 --- a/lang/Cesky.txt +++ b/lang/Cesky.txt @@ -964,3 +964,7 @@ WARC archive name: Název archivu WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Volitelný základní název archivu WARC; ponechte prázdné pro automatické pojmenování ve výstupním adresáøi. +Report what changed since the previous mirror +Hlásit, co se od pøedchozího zrcadlení zmìnilo +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Zapsat také hts-changes.json se seznamem toho, co je oproti pøedchozímu zrcadlení nové, zmìnìné, nezmìnìné nebo chybìjící. diff --git a/lang/Chinese-BIG5.txt b/lang/Chinese-BIG5.txt index 9d4e93330..17016e629 100644 --- a/lang/Chinese-BIG5.txt +++ b/lang/Chinese-BIG5.txt @@ -964,3 +964,7 @@ WARC archive name: WARC «Ê¦sÀɦWºÙ¡G Optional base name for the WARC archive; leave blank to auto-name it under the output directory. WARC «Ê¦sÀɪº¿ï¥Î°ò¥»¦WºÙ¡F¯dªÅ«h©ó¿é¥X¥Ø¿ý¤¤¦Û°Ê©R¦W¡C +Report what changed since the previous mirror +¦^³ø¦Û¤W¦¸Ãè®g¥H¨ÓªºÅܧó +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +¦P®É¼g¤J hts-changes.json¡A¦C¥X³o¦¸Â^¨ú¬Û¹ï©ó¤W¦¸Ãè®gªº·s¼W¡BÅܧó¡B¥¼Åܧó»P®ø¥¢ªº¶µ¥Ø¡C diff --git a/lang/Chinese-Simplified.txt b/lang/Chinese-Simplified.txt index f7a8f8f79..011c66b0a 100644 --- a/lang/Chinese-Simplified.txt +++ b/lang/Chinese-Simplified.txt @@ -964,3 +964,7 @@ WARC archive name: WARC ¹éµµÃû³Æ£º Optional base name for the WARC archive; leave blank to auto-name it under the output directory. WARC ¹éµµµÄ¿ÉÑ¡»ù±¾Ãû³Æ£»Áô¿ÕÔòÔÚÊä³öĿ¼ÖÐ×Ô¶¯ÃüÃû¡£ +Report what changed since the previous mirror +±¨¸æ×ÔÉϴξµÏñÒÔÀ´µÄ±ä¸ü +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +ͬʱдÈë hts-changes.json£¬Áгö±¾´ÎץȡÏà¶ÔÓÚÉϴξµÏñµÄÐÂÔö¡¢¸ü¸Ä¡¢Î´¸ü¸ÄºÍÏûʧµÄÏîÄ¿¡£ diff --git a/lang/Croatian.txt b/lang/Croatian.txt index 9e5e321cd..2902aa6e8 100644 --- a/lang/Croatian.txt +++ b/lang/Croatian.txt @@ -966,3 +966,7 @@ WARC archive name: Naziv WARC arhive: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Neobavezni osnovni naziv WARC arhive; ostavite prazno za automatsko imenovanje u izlaznom direktoriju. +Report what changed since the previous mirror +Prijavi ¹to se promijenilo od prethodnog zrcaljenja +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Zapi¹i i hts-changes.json s popisom onoga ¹to je u odnosu na prethodno zrcaljenje novo, promijenjeno, nepromijenjeno ili nestalo. diff --git a/lang/Dansk.txt b/lang/Dansk.txt index f45f59c42..15d86096a 100644 --- a/lang/Dansk.txt +++ b/lang/Dansk.txt @@ -1012,3 +1012,7 @@ WARC archive name: Navn på WARC-arkiv: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Valgfrit basisnavn til WARC-arkivet; lad feltet stå tomt for automatisk navngivning i outputmappen. +Report what changed since the previous mirror +Rapportér hvad der er ændret siden den forrige spejling +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Skriv også hts-changes.json med en liste over, hvad denne gennemgang efterlader som nyt, ændret, uændret eller forsvundet i forhold til den forrige spejling. diff --git a/lang/Deutsch.txt b/lang/Deutsch.txt index b191894a7..de8533e4b 100644 --- a/lang/Deutsch.txt +++ b/lang/Deutsch.txt @@ -964,3 +964,7 @@ WARC archive name: Name des WARC-Archivs: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Optionaler Basisname für das WARC-Archiv; leer lassen, um es automatisch im Ausgabeverzeichnis zu benennen. +Report what changed since the previous mirror +Melden, was sich seit der vorherigen Spiegelung geändert hat +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Zusätzlich hts-changes.json schreiben, das auflistet, was dieser Durchlauf gegenüber der vorherigen Spiegelung als neu, geändert, unverändert oder verschwunden hinterlässt. diff --git a/lang/Eesti.txt b/lang/Eesti.txt index 38c0b144b..2baa553e3 100644 --- a/lang/Eesti.txt +++ b/lang/Eesti.txt @@ -964,3 +964,7 @@ WARC archive name: WARC-arhiivi nimi: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. WARC-arhiivi valikuline põhinimi; jäta tühjaks, et see väljundkataloogis automaatselt nimetada. +Report what changed since the previous mirror +Teata, mis on eelmisest peegeldusest saadik muutunud +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Kirjuta ka hts-changes.json, mis loetleb, mis on võrreldes eelmise peegeldusega uus, muutunud, muutumatu või kadunud. diff --git a/lang/Finnish.txt b/lang/Finnish.txt index 6c22996a2..d86d1bd11 100644 --- a/lang/Finnish.txt +++ b/lang/Finnish.txt @@ -966,3 +966,7 @@ WARC archive name: WARC-arkiston nimi: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Valinnainen WARC-arkiston perusnimi; jätä tyhjäksi, jotta se nimetään automaattisesti tulostehakemistoon. +Report what changed since the previous mirror +Raportoi, mikä on muuttunut edellisen peilauksen jälkeen +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Kirjoita myös hts-changes.json, joka luettelee, mikä on edelliseen peilaukseen verrattuna uutta, muuttunutta, muuttumatonta tai kadonnutta. diff --git a/lang/Greek.txt b/lang/Greek.txt index a38e4fc18..da30d6c06 100644 --- a/lang/Greek.txt +++ b/lang/Greek.txt @@ -966,3 +966,7 @@ WARC archive name: ¼íïìá áñ÷åßïõ WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Ðñïáéñåôéêü âáóéêü üíïìá ãéá ôï áñ÷åßï WARC. ÁöÞóôå ôï êåíü ãéá áõôüìáôç ïíïìáóßá óôïí êáôÜëïãï åîüäïõ. +Report what changed since the previous mirror +ÁíáöïñÜ ôùí áëëáãþí áðü ôï ðñïçãïýìåíï áíôßãñáöï +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +ÅããñáöÞ êáé ôïõ hts-changes.json, ðïõ ðáñáèÝôåé ôé åßíáé íÝï, ôé Üëëáîå, ôé Ýìåéíå ßäéï êáé ôé ÷Üèçêå óå ó÷Ýóç ìå ôï ðñïçãïýìåíï áíôßãñáöï. diff --git a/lang/Italiano.txt b/lang/Italiano.txt index 23b7438f3..e495bcf15 100644 --- a/lang/Italiano.txt +++ b/lang/Italiano.txt @@ -964,3 +964,7 @@ WARC archive name: Nome dell'archivio WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Nome di base facoltativo per l'archivio WARC; lascia vuoto per assegnarlo automaticamente nella directory di output. +Report what changed since the previous mirror +Segnala che cosa è cambiato dalla copia precedente +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Scrive anche hts-changes.json, che elenca ciò che questa scansione lascia come nuovo, modificato, invariato o scomparso rispetto alla copia precedente. diff --git a/lang/Japanese.txt b/lang/Japanese.txt index 329eb09c0..621c354a5 100644 --- a/lang/Japanese.txt +++ b/lang/Japanese.txt @@ -964,3 +964,7 @@ WARC archive name: WARC ƒA[ƒJƒCƒu–¼: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. WARC ƒA[ƒJƒCƒu‚Ì”CˆÓ‚̃x[ƒX–¼B‹ó—“‚É‚·‚é‚Æo—̓fƒBƒŒƒNƒgƒŠ“à‚ÅŽ©“®“I‚É–¼‘O‚ª•t‚¯‚ç‚ê‚Ü‚·B +Report what changed since the previous mirror +‘O‰ñ‚̃~ƒ‰[‚©‚ç‚Ì•ÏX“_‚ð•ñ‚·‚é +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +hts-changes.json ‚à‘‚«o‚µA‘O‰ñ‚̃~ƒ‰[‚Æ”ä‚ׂÄV‹KA•ÏXA•ÏX‚È‚µAÁ–Å‚Æ‚È‚Á‚½‚à‚Ì‚ðˆê——‚É‚µ‚Ü‚·B diff --git a/lang/Macedonian.txt b/lang/Macedonian.txt index 45245f46a..a1c72df8c 100644 --- a/lang/Macedonian.txt +++ b/lang/Macedonian.txt @@ -964,3 +964,7 @@ WARC archive name: ¸ÜÕ ÝÐ WARC ÐàåØÒÐâÐ: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. ¸×ÑÞàÝÞ ÞáÝÞÒÝÞ ØÜÕ ×Ð WARC ÐàåØÒÐâÐ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÐÒâÞÜÐâáÚÞ ØÜÕÝãÒÐúÕ ÒÞ Ø×ÛÕ×ÝØÞâ ÔØàÕÚâÞàØãÜ. +Report what changed since the previous mirror +¸×ÒÕáâØ èâÞ Õ ßàÞÜÕÝÕâÞ ÞÔ ßàÕâåÞÔÝÞâÞ ÞÓÛÕÔÐÛÞ +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +·ÐßØèØ Ø hts-changes.json áÞ áßØáÞÚ ÝÐ âÞÐ èâÞ Õ ÝÞÒÞ, ßàÞÜÕÝÕâÞ, ÝÕßàÞÜÕÝÕâÞ ØÛØ ØáçÕ×ÝÐâÞ ÒÞ ÞÔÝÞá ÝÐ ßàÕâåÞÔÝÞâÞ ÞÓÛÕÔÐÛÞ. diff --git a/lang/Magyar.txt b/lang/Magyar.txt index 8087cc7ac..31537eb5d 100644 --- a/lang/Magyar.txt +++ b/lang/Magyar.txt @@ -964,3 +964,7 @@ WARC archive name: WARC archívum neve: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. A WARC archívum opcionális alapneve; hagyja üresen az automatikus elnevezéshez a kimeneti könyvtárban. +Report what changed since the previous mirror +Jelentés arról, mi változott az elõzõ tükrözés óta +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +A hts-changes.json fájl írása is, amely felsorolja, mi új, mi változott, mi maradt változatlan és mi tûnt el az elõzõ tükrözéshez képest. diff --git a/lang/Nederlands.txt b/lang/Nederlands.txt index 749baf8bc..ccc1bc188 100644 --- a/lang/Nederlands.txt +++ b/lang/Nederlands.txt @@ -964,3 +964,7 @@ WARC archive name: Naam van WARC-archief: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Optionele basisnaam voor het WARC-archief; laat leeg om het automatisch een naam te geven in de uitvoermap. +Report what changed since the previous mirror +Rapporteren wat er sinds de vorige spiegeling is gewijzigd +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Ook hts-changes.json schrijven met een overzicht van wat deze doorloop nieuw, gewijzigd, ongewijzigd of verdwenen laat ten opzichte van de vorige spiegeling. diff --git a/lang/Norsk.txt b/lang/Norsk.txt index 0338aeefb..6762f51bd 100644 --- a/lang/Norsk.txt +++ b/lang/Norsk.txt @@ -964,3 +964,7 @@ WARC archive name: Navn på WARC-arkiv: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Valgfritt basisnavn for WARC-arkivet; la feltet stå tomt for automatisk navngivning i utdatamappen. +Report what changed since the previous mirror +Rapporter hva som er endret siden forrige speiling +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Skriv også hts-changes.json som lister opp hva denne gjennomgangen etterlater som nytt, endret, uendret eller forsvunnet sammenlignet med forrige speiling. diff --git a/lang/Polski.txt b/lang/Polski.txt index 72b751e8b..bf774ff42 100644 --- a/lang/Polski.txt +++ b/lang/Polski.txt @@ -964,3 +964,7 @@ WARC archive name: Nazwa archiwum WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Opcjonalna nazwa bazowa archiwum WARC; pozostaw puste, aby nazwaæ je automatycznie w katalogu wyj¶ciowym. +Report what changed since the previous mirror +Zg³o¶, co zmieni³o siê od poprzedniej kopii +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Zapisz równie¿ plik hts-changes.json z list± tego, co w porównaniu z poprzedni± kopi± jest nowe, zmienione, niezmienione lub usuniête. diff --git a/lang/Portugues-Brasil.txt b/lang/Portugues-Brasil.txt index a95c30c40..76aac3793 100644 --- a/lang/Portugues-Brasil.txt +++ b/lang/Portugues-Brasil.txt @@ -1012,3 +1012,7 @@ WARC archive name: Nome do arquivo WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Nome base opcional para o arquivo WARC; deixe em branco para nomeá-lo automaticamente no diretório de saída. +Report what changed since the previous mirror +Relatar o que mudou desde o espelhamento anterior +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Gravar também hts-changes.json listando o que esta captura deixa como novo, alterado, inalterado ou removido em relação ao espelhamento anterior. diff --git a/lang/Portugues.txt b/lang/Portugues.txt index f85c53b38..de865153e 100644 --- a/lang/Portugues.txt +++ b/lang/Portugues.txt @@ -964,3 +964,7 @@ WARC archive name: Nome do arquivo WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Nome base opcional para o arquivo WARC; deixe em branco para o nomear automaticamente no diretório de saída. +Report what changed since the previous mirror +Comunicar o que mudou desde o espelho anterior +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Escrever também hts-changes.json, que lista o que esta recolha deixa como novo, alterado, inalterado ou desaparecido em relação ao espelho anterior. diff --git a/lang/Romanian.txt b/lang/Romanian.txt index 5d7a189c1..e3fabac2b 100644 --- a/lang/Romanian.txt +++ b/lang/Romanian.txt @@ -964,3 +964,7 @@ WARC archive name: Numele arhivei WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Nume de baza optional pentru arhiva WARC; lasati gol pentru a-l denumi automat in directorul de iesire. +Report what changed since the previous mirror +Raporteaza ce s-a schimbat fata de copia anterioara +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Scrie si hts-changes.json, care listeaza ce este nou, modificat, nemodificat sau disparut fata de copia anterioara. diff --git a/lang/Russian.txt b/lang/Russian.txt index 248fb8d2e..9755a0f08 100644 --- a/lang/Russian.txt +++ b/lang/Russian.txt @@ -964,3 +964,7 @@ WARC archive name: Èìÿ WARC-àðõèâà: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Íåîáÿçàòåëüíîå áàçîâîå èìÿ WARC-àðõèâà; îñòàâüòå ïóñòûì äëÿ àâòîìàòè÷åñêîãî èìåíîâàíèÿ â âûõîäíîì êàòàëîãå. +Report what changed since the previous mirror +Ñîîáùàòü, ÷òî èçìåíèëîñü ñ ïðåäûäóùåãî çåðêàëà +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Òàêæå çàïèñûâàòü hts-changes.json ñî ñïèñêîì òîãî, ÷òî ïî ñðàâíåíèþ ñ ïðåäûäóùèì çåðêàëîì ñòàëî íîâûì, èçìåí¸ííûì, íåèçìåí¸ííûì èëè èñ÷åçëî. diff --git a/lang/Slovak.txt b/lang/Slovak.txt index 7dc4111cc..8eb4c4d40 100644 --- a/lang/Slovak.txt +++ b/lang/Slovak.txt @@ -964,3 +964,7 @@ WARC archive name: Názov archívu WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Voliteµný základný názov archívu WARC; ponechajte prázdne pre automatické pomenovanie vo výstupnom adresári. +Report what changed since the previous mirror +Oznámi», èo sa od predchádzajúceho zrkadlenia zmenilo +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Zapísa» aj hts-changes.json so zoznamom toho, èo je oproti predchádzajúcemu zrkadleniu nové, zmenené, nezmenené alebo chýbajúce. diff --git a/lang/Slovenian.txt b/lang/Slovenian.txt index 51ee68621..bcf93310e 100644 --- a/lang/Slovenian.txt +++ b/lang/Slovenian.txt @@ -964,3 +964,7 @@ WARC archive name: Ime arhiva WARC: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Neobvezno osnovno ime arhiva WARC; pustite prazno za samodejno poimenovanje v izhodni mapi. +Report what changed since the previous mirror +Porocaj, kaj se je spremenilo od prejsnjega zrcaljenja +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Zapisi tudi hts-changes.json s seznamom tega, kar je v primerjavi s prejsnjim zrcaljenjem novo, spremenjeno, nespremenjeno ali izginilo. diff --git a/lang/Svenska.txt b/lang/Svenska.txt index 3d1cfe57e..8fb950602 100644 --- a/lang/Svenska.txt +++ b/lang/Svenska.txt @@ -964,3 +964,7 @@ WARC archive name: WARC-arkivets namn: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Valfritt basnamn för WARC-arkivet; lämna tomt för att namnge det automatiskt i utdatakatalogen. +Report what changed since the previous mirror +Rapportera vad som ändrats sedan föregående spegling +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Skriv även hts-changes.json som listar vad den här genomgången lämnar som nytt, ändrat, oförändrat eller försvunnet jämfört med föregående spegling. diff --git a/lang/Turkish.txt b/lang/Turkish.txt index 9eb7f8141..563db1968 100644 --- a/lang/Turkish.txt +++ b/lang/Turkish.txt @@ -964,3 +964,7 @@ WARC archive name: WARC arþivi adý: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. WARC arþivi için isteðe baðlý temel ad; çýktý dizininde otomatik adlandýrma için boþ býrakýn. +Report what changed since the previous mirror +Önceki yansýmadan bu yana deðiþenleri bildir +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Ayrýca hts-changes.json yazarak bu taramanýn önceki yansýmaya göre neyi yeni, deðiþmiþ, deðiþmemiþ veya kaybolmuþ býraktýðýný listele. diff --git a/lang/Ukrainian.txt b/lang/Ukrainian.txt index abb181768..31510d4c8 100644 --- a/lang/Ukrainian.txt +++ b/lang/Ukrainian.txt @@ -964,3 +964,7 @@ WARC archive name: ²ì'ÿ WARC-àðõ³âó: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Íåîáîâ'ÿçêîâà áàçîâà íàçâà WARC-àðõ³âó; çàëèøòå ïîðîæí³ì äëÿ àâòîìàòè÷íîãî íàéìåíóâàííÿ ó âèõ³äíîìó êàòàëîç³. +Report what changed since the previous mirror +Ïîâ³äîìëÿòè, ùî çì³íèëîñÿ ç ïîïåðåäíüîãî äçåðêàëà +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +Òàêîæ çàïèñóâàòè hts-changes.json ç³ ñïèñêîì òîãî, ùî ïîð³âíÿíî ç ïîïåðåäí³ì äçåðêàëîì º íîâèì, çì³íåíèì, íåçì³íåíèì àáî çíèêëèì. diff --git a/lang/Uzbek.txt b/lang/Uzbek.txt index 6c35f804c..56f987f47 100644 --- a/lang/Uzbek.txt +++ b/lang/Uzbek.txt @@ -964,3 +964,7 @@ WARC archive name: WARC arxivi nomi: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. WARC arxivi uchun ixtiyoriy asosiy nom; chiqish katalogida avtomatik nomlash uchun bo'sh qoldiring. +Report what changed since the previous mirror +Oldingi nusxadan beri nima o’zgarganini xabar qilish +Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror. +hts-changes.json ham yoziladi: unda ushbu yig’ish oldingi nusxaga nisbatan nimani yangi, o’zgargan, o’zgarmagan yoki yo’qolgan holda qoldirgani ro’yxati bo’ladi. From a00b95dcb48b8048f1dacd37f1f5b3871b157d87 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 22:36:08 +0200 Subject: [PATCH 09/11] changes: fix the review's blocking findings Lock the report path against the FTP thread the crawl never joins, seal the accumulator once the report is written, key entries off the project directory so the report survives --cache=0, stop calling a file gone when the crawl only failed to re-fetch it, and skip the hook's work entirely when --changes is off. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- src/htschanges.c | 203 +++++++++++++++++++++++------------ src/htschanges.h | 48 +++++---- src/htscore.c | 27 +++-- src/htscore.h | 7 +- src/htsselftest.c | 100 +++++++++++++++++ tests/01_engine-changes.test | 25 ++++- tests/89_local-changes.test | 164 ++++++++++++++++++++-------- tests/local-server.py | 49 ++++++++- 8 files changed, 475 insertions(+), 148 deletions(-) diff --git a/src/htschanges.c b/src/htschanges.c index 5198c1c38..853daa932 100644 --- a/src/htschanges.c +++ b/src/htschanges.c @@ -80,8 +80,10 @@ struct hts_changes { changes_entry *entries; size_t count; size_t capacity; + hts_boolean has_index; /* the run keeps a mirror index (the cache is on) */ hts_boolean old_index; /* the previous mirror's index was read */ hts_boolean overflow; /* an allocation failed; the report is partial */ + hts_boolean closed; /* reported: late notifies are dropped, not recorded */ }; /* ------------------------------------------------------------ */ @@ -218,8 +220,9 @@ static intptr_t changes_slot(hts_changes *changes, const char *file, return slot; } -/* MD5 of the file at `path`; change detection over our own mirror, not a - security boundary. HTS_FALSE if it cannot be read. */ +/* MD5 of the file at `path`; not a security boundary. Never call it holding + changes_mutex: hashing a large file would stall every other connection. + HTS_FALSE if it cannot be read. */ static hts_boolean digest_file(const char *path, unsigned char digest[DIGEST_SIZE]) { const int endian = 1; @@ -267,17 +270,24 @@ hts_change_bucket hts_changes_classify(hts_boolean rewritten, /* Engine hooks */ /* ------------------------------------------------------------ */ -/* FTP transfers reach file_notify() from their own detached thread - (htsftp.c:631/635), so every entry point below runs under this lock. */ +/* FTP transfers reach file_notify() from a detached thread of their own, and + the crawl is not joined before the report is written, so every entry point + below (report and teardown included) runs under this lock. */ static htsmutex changes_mutex = HTSMUTEX_INIT; -/* The live accumulator, created on first use; NULL when --changes is off. */ +/* The live accumulator, created on first use. NULL when --changes is off or + the report has been written: a straggler notify must be dropped, not + resurrect an accumulator nobody will report. Call under the lock. */ static hts_changes *changes_get(httrackp *opt) { + hts_changes *changes = (hts_changes *) opt->changes_state; + if (!opt->changes) return NULL; - if (opt->changes_state == NULL) - opt->changes_state = changes_new(); - return (hts_changes *) opt->changes_state; + if (changes == NULL) { + changes = changes_new(); + opt->changes_state = changes; + } + return changes != NULL && !changes->closed ? changes : NULL; } void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, @@ -285,60 +295,70 @@ void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, hts_boolean not_updated) { hts_changes *changes; char BIGSTK file[HTS_URLMAXSIZE * 2]; - hts_boolean is_new; - intptr_t slot; - changes_entry *entry; + char BIGSTK url[HTS_URLMAXSIZE * 4 + 8]; /* holds adr + fil + a scheme */ + unsigned char prev_digest[DIGEST_SIZE]; + hts_boolean has_prev = HTS_FALSE; + hts_boolean is_new = HTS_FALSE; + intptr_t slot = -1; + LLint prev_size; + if (!opt->changes) + return; /* Engine-generated scaffolding (the top index) carries no URL and is not a mirrored resource. */ if (save == NULL || !strnotempty(save) || adr == NULL || fil == NULL || (!strnotempty(adr) && !strnotempty(fil))) return; + hts_savename_listed(StringBuff(opt->path_html), save, file, sizeof(file)); + hts_mutexlock(&changes_mutex); changes = changes_get(opt); - if (changes == NULL) { - hts_mutexrelease(&changes_mutex); - return; + if (changes != NULL) { + slot = changes_slot(changes, file, &is_new); + if (slot >= 0 && !is_new) { + /* Retry, or a second call site for the same file: the pre-run state was + already sampled and the copy on disk may no longer be it. */ + changes->entries[slot].rewritten = + changes->entries[slot].rewritten || rewritten; + } } - - hts_savename_listed(&opt->state.strc, save, file, sizeof(file)); - slot = changes_slot(changes, file, &is_new); - if (slot < 0) { - hts_mutexrelease(&changes_mutex); + hts_mutexrelease(&changes_mutex); + if (slot < 0 || !is_new) return; - } - entry = &changes->entries[slot]; - if (!is_new) { - /* Retry, or a second call site for the same file: the pre-run state was - already sampled and the copy on disk may no longer be it. */ - entry->rewritten = entry->rewritten || rewritten; - hts_mutexrelease(&changes_mutex); - return; - } + url[0] = '\0'; + if (!link_has_authority(adr)) + strlcatbuff(url, "http://", sizeof(url)); + strlcatbuff(url, adr, sizeof(url)); + strlcatbuff(url, fil, sizeof(url)); - { - char BIGSTK url[HTS_URLMAXSIZE * 4 + 8]; /* holds adr + fil + a scheme */ + /* Sample the previous copy with the lock released, and hash it only when it + is about to be overwritten: this is the last moment those bytes exist. */ + prev_size = fsize_utf8(save); + if (prev_size >= 0 && rewritten) + has_prev = digest_file(save, prev_digest); + + hts_mutexlock(&changes_mutex); + changes = changes_get(opt); + /* The slot index is stable (entries are only ever appended), but realloct() + may have moved the array while the lock was down. */ + if (changes != NULL && (size_t) slot < changes->count) { + changes_entry *const entry = &changes->entries[slot]; - url[0] = '\0'; - if (!link_has_authority(adr)) - strlcatbuff(url, "http://", sizeof(url)); - strlcatbuff(url, adr, sizeof(url)); - strlcatbuff(url, fil, sizeof(url)); entry->url = strdupt(url); if (entry->url == NULL) changes->overflow = HTS_TRUE; + entry->rewritten = entry->rewritten || rewritten; + entry->not_updated = not_updated; + entry->prev_size = prev_size; + entry->on_disk = prev_size >= 0; + /* hts_changes_html() compares payloads, and its digests win. */ + if (has_prev && !entry->has_new) { + entry->has_prev = HTS_TRUE; + memcpy(entry->prev_digest, prev_digest, DIGEST_SIZE); + } } - - entry->rewritten = rewritten; - entry->not_updated = not_updated; - entry->prev_size = fsize_utf8(save); - entry->on_disk = entry->prev_size >= 0; - /* Hash the previous copy only when it is about to be overwritten: this is - the last moment those bytes exist. */ - if (entry->on_disk && rewritten) - entry->has_prev = digest_file(save, entry->prev_digest); hts_mutexrelease(&changes_mutex); } @@ -353,6 +373,10 @@ void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r, intptr_t slot; htsblk prev; + /* Before the digest and the cache read: with --changes off this hook must + cost nothing. */ + if (!opt->changes) + return; if (r->adr == NULL || r->size < 0 || save == NULL || !strnotempty(save)) return; @@ -373,7 +397,7 @@ void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r, changes = changes_get(opt); /* file_notify() records the entry; this call only refines it. */ if (changes != NULL) { - hts_savename_listed(&opt->state.strc, save, file, sizeof(file)); + hts_savename_listed(StringBuff(opt->path_html), save, file, sizeof(file)); slot = changes_find(changes, file); if (slot >= 0) { changes_entry *const entry = &changes->entries[slot]; @@ -388,12 +412,12 @@ void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r, hts_mutexrelease(&changes_mutex); } -void hts_changes_gone(httrackp *opt, const char *file) { +void hts_changes_dropped(httrackp *opt, const char *file, hts_boolean kept) { hts_changes *changes; hts_boolean is_new; intptr_t slot; - if (file == NULL || !strnotempty(file)) + if (!opt->changes || file == NULL || !strnotempty(file)) return; hts_mutexlock(&changes_mutex); changes = changes_get(opt); @@ -401,8 +425,10 @@ void hts_changes_gone(httrackp *opt, const char *file) { slot = changes_slot(changes, file, &is_new); if (slot >= 0 && is_new) { changes->entries[slot].url = strdupt(""); - changes->entries[slot].bucket = HTS_CHANGE_GONE; changes->entries[slot].listed_prev = HTS_TRUE; + /* A kept file leaves rewritten clear, which resolves to unchanged. */ + if (!kept) + changes->entries[slot].bucket = HTS_CHANGE_GONE; } } hts_mutexrelease(&changes_mutex); @@ -412,11 +438,12 @@ void hts_changes_previous(httrackp *opt, const char *file) { hts_changes *changes; intptr_t slot; - if (file == NULL) + if (!opt->changes || file == NULL) return; hts_mutexlock(&changes_mutex); changes = changes_get(opt); if (changes != NULL) { + changes->has_index = HTS_TRUE; changes->old_index = HTS_TRUE; slot = changes_find(changes, file); if (slot >= 0) @@ -425,6 +452,18 @@ void hts_changes_previous(httrackp *opt, const char *file) { hts_mutexrelease(&changes_mutex); } +void hts_changes_indexed(httrackp *opt) { + hts_changes *changes; + + if (!opt->changes) + return; + hts_mutexlock(&changes_mutex); + changes = changes_get(opt); + if (changes != NULL) + changes->has_index = HTS_TRUE; + hts_mutexrelease(&changes_mutex); +} + /* ------------------------------------------------------------ */ /* Report */ /* ------------------------------------------------------------ */ @@ -479,9 +518,8 @@ static void changes_resolve(hts_changes *changes, httrackp *opt) { static const char *const bucket_names[HTS_CHANGE_BUCKETS] = { "new", "changed", "unchanged", "gone"}; -/* Callers below run after the crawl's threads are done, so they take no lock. - */ -void hts_changes_report(httrackp *opt, String *out) { +/* Serialize the report. Call under changes_mutex. */ +static void changes_serialize(httrackp *opt, String *out) { hts_changes *const changes = (hts_changes *) opt->changes_state; size_t counts[HTS_CHANGE_BUCKETS]; char date[32]; @@ -507,7 +545,9 @@ void hts_changes_report(httrackp *opt, String *out) { StringCat(*out, ",\n \"date\": "); hts_changes_json_string(out, date); StringCat(*out, ",\n \"first_crawl\": "); - StringCat(*out, changes->old_index ? "false" : "true"); + StringCat(*out, !changes->has_index + ? "null" + : (changes->old_index ? "false" : "true")); StringCat(*out, ",\n \"partial\": "); StringCat(*out, changes->overflow ? "true" : "false"); StringCat(*out, ",\n \"purged\": "); @@ -555,25 +595,44 @@ void hts_changes_report(httrackp *opt, String *out) { StringCat(*out, "\n}\n"); } +void hts_changes_report(httrackp *opt, String *out) { + hts_mutexlock(&changes_mutex); + changes_serialize(opt, out); + hts_mutexrelease(&changes_mutex); +} + void hts_changes_close_opt(httrackp *opt) { - /* changes_get(), not the raw field: a crawl that mirrored nothing still owes - the user a report, and a stale one from the previous run must not survive - on disk as if it described this one. */ - hts_changes *changes = changes_get(opt); char catbuff[CATBUFF_SIZE]; const char *path; String report = STRING_EMPTY; size_t counts[HTS_CHANGE_BUCKETS]; - size_t i; + size_t total; + hts_boolean has_index, old_index; FILE *fp; - if (changes == NULL) - return; + hts_mutexlock(&changes_mutex); + { + /* changes_get(), not the raw field: a crawl that mirrored nothing still + owes the user a report, and a stale one from the previous run must not + survive on disk as if it described this one. */ + hts_changes *const changes = changes_get(opt); + size_t i; - hts_changes_report(opt, &report); - memset(counts, 0, sizeof(counts)); - for (i = 0; i < changes->count; i++) - counts[changes->entries[i].bucket]++; + if (changes == NULL) { + hts_mutexrelease(&changes_mutex); + return; + } + changes_serialize(opt, &report); + memset(counts, 0, sizeof(counts)); + for (i = 0; i < changes->count; i++) + counts[changes->entries[i].bucket]++; + total = changes->count; + has_index = changes->has_index; + old_index = changes->old_index; + /* Sticky: whatever the crawl's stragglers do next is not in this report. */ + changes->closed = HTS_TRUE; + } + hts_mutexrelease(&changes_mutex); path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_log), HTS_CHANGES_FILE); @@ -590,11 +649,16 @@ void hts_changes_close_opt(httrackp *opt) { "Unable to create the change report %s", path); } - if (!changes->old_index) { + if (!has_index) { + hts_log_print(opt, LOG_NOTICE, + "Change report: no mirror index (the cache is off), %d files " + "mirrored, deletions not detected (%s)", + (int) total, HTS_CHANGES_FILE); + } else if (!old_index) { hts_log_print(opt, LOG_NOTICE, "Change report: first crawl, %d files mirrored, nothing to " "compare against (%s)", - (int) changes->count, HTS_CHANGES_FILE); + (int) total, HTS_CHANGES_FILE); } else { hts_log_print(opt, LOG_NOTICE, "Change report: %d new, %d changed, %d unchanged, %d gone " @@ -606,13 +670,14 @@ void hts_changes_close_opt(httrackp *opt) { } StringFree(report); - changes_free(&changes); - opt->changes_state = NULL; } void hts_changes_free_opt(httrackp *opt) { - hts_changes *changes = (hts_changes *) opt->changes_state; + hts_changes *changes; + hts_mutexlock(&changes_mutex); + changes = (hts_changes *) opt->changes_state; changes_free(&changes); opt->changes_state = NULL; + hts_mutexrelease(&changes_mutex); } diff --git a/src/htschanges.h b/src/htschanges.h index 76b8b7740..c5b9a86e1 100644 --- a/src/htschanges.h +++ b/src/htschanges.h @@ -62,47 +62,49 @@ typedef enum { HTS_CHANGE_BUCKETS } hts_change_bucket; -/* Record what this crawl is doing to the local file `save` (an absolute path; - adr/fil form the URL it comes from, either may be empty for an engine - generated file). `rewritten` means the copy on disk is being written over, - `not_updated` is the 200-versus-304 transfer signal, used only when no - digest can be taken. Only the first call for a given file samples the - previous copy, so a retried or twice-notified resource is counted once. - No-op unless --changes is on. */ +/* Record what this crawl is doing to the local file `save` (absolute path; + adr/fil form its URL, either may be empty for an engine-generated file). + `rewritten` means the copy on disk is being written over, `not_updated` is + the 200-versus-304 signal, used only when no digest can be taken. Only the + first call for a file samples the previous copy, so a retried or + twice-notified resource is counted once. No-op unless --changes is on. */ void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, const char *save, hts_boolean rewritten, hts_boolean not_updated); -/* Refine the entry for a parsed HTML file, whose local copy is the payload - after link rewriting plus a footer carrying the crawl date: its bytes differ - on every run, so the comparison uses the payload `r` holds against the body - the previous run left in the cache. Called once the page is ready to be - written, from hts_finish_html_file(). */ +/* Refine the entry for a parsed HTML file: its local copy carries rewritten + links and a footer dated by the crawl, so it differs every run. Compares the + payload `r` holds against the body the previous run left in the cache. */ void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r, const char *adr, const char *fil, const char *save); -/* Record `file` (mirror-relative, as listed in new.lst) as present in the - previous mirror and absent from this one. Whether it is also deleted from - disk is the purge option's business, not this one's. */ -void hts_changes_gone(httrackp *opt, const char *file); +/* Record `file` (mirror-relative, as listed in new.lst) as listed by the + previous mirror's index and absent from this run's. `kept` means its local + copy survives the run because the crawl tried and failed to replace it: a + failed transfer is not a deletion, so it is reported unchanged, not gone. */ +void hts_changes_dropped(httrackp *opt, const char *file, hts_boolean kept); /* Record that the previous mirror's index listed `file`. That index, not the - file's presence on disk, decides what counts as already mirrored: a partial - left by this crawl's own failed attempt is on disk but was never part of the - previous mirror. Calling it at all marks the run as having one. */ + file's presence on disk, decides what counts as already mirrored. */ void hts_changes_previous(httrackp *opt, const char *file); +/* Record that this run keeps a mirror index at all. Without one (the cache is + off) deletions cannot be seen and first_crawl cannot be decided, which the + report states rather than guesses. */ +void hts_changes_indexed(httrackp *opt); + /* Resolve every entry against the bytes now on disk and serialize the report into `out` (replaced). Exposed for the `changes` self-test; the engine goes through hts_changes_close_opt(). */ void hts_changes_report(httrackp *opt, String *out); -/* Write the report and log a one-line summary, then free the accumulator. - Null-safe and idempotent. */ +/* Write the report and log a one-line summary. Idempotent, and after it the + accumulator is sealed: a late notify from a still-running transfer thread is + dropped instead of starting a report nobody will write. */ void hts_changes_close_opt(httrackp *opt); -/* Drop the accumulator without writing anything, for a run that never reached - its end. Null-safe and idempotent. */ +/* Drop the accumulator, for a run that never reached its end and to start the + next one clean. Null-safe and idempotent. */ void hts_changes_free_opt(httrackp *opt); /* Bucket for one entry, from what was observed. No filesystem access: diff --git a/src/htscore.c b/src/htscore.c index 278713493..e87d98163 100644 --- a/src/htscore.c +++ b/src/htscore.c @@ -693,6 +693,9 @@ int httpmirror(char *url1, httrackp * opt) { // hash table opt->hash = &hash; + // a change report left by a previous crawl on this opt is not this one's + hts_changes_free_opt(opt); + // initialize link heap hts_record_init(opt); @@ -2096,6 +2099,7 @@ int httpmirror(char *url1, httrackp * opt) { if (opt->delete_old || opt->changes) { FILE *old_lst, *new_lst; + hts_changes_indexed(opt); // opt->state._hts_in_html_parsing = 3; // @@ -2132,7 +2136,17 @@ int httpmirror(char *url1, httrackp * opt) { hts_changes_previous(opt, file + StringLength(opt->path_html)); if (!strstr(adr, line)) { // not found in the new list? if (fexist_utf8(file)) { // still on disk - hts_changes_gone(opt, file + StringLength(opt->path_html)); + /* A link this crawl did try but never wrote (a transfer + killed mid-flight) also drops out of new.lst. Unless it + is about to be purged, its previous copy stands and the + file is not gone. */ + const hts_boolean kept = + !opt->delete_old && + hash_read(opt->hash, file, NULL, + HASH_STRUCT_FILENAME) >= 0; + + hts_changes_dropped( + opt, file + StringLength(opt->path_html), kept); if (opt->delete_old) { hts_log_print(opt, LOG_INFO, "Purging %s", file); UNLINK(file); @@ -2892,15 +2906,14 @@ int filecreateempty(filenote_strc * strc, const char *filename) { return 0; } -void hts_savename_listed(const filenote_strc *strc, const char *s, char *dest, +void hts_savename_listed(const char *root, const char *s, char *dest, size_t destsize) { char catbuff[CATBUFF_SIZE]; strlcpybuff(dest, fslash(catbuff, sizeof(catbuff), s), destsize); - if (strnotempty(strc->path) && - strncmp(fslash(catbuff, sizeof(catbuff), strc->path), dest, - strlen(strc->path)) == 0) { - strlcpybuff(dest, s + strlen(strc->path), destsize); + if (strnotempty(root) && strncmp(fslash(catbuff, sizeof(catbuff), root), dest, + strlen(root)) == 0) { + strlcpybuff(dest, s + strlen(root), destsize); } } @@ -2914,7 +2927,7 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) { } else if (strc->lst) { char BIGSTK savelst[HTS_URLMAXSIZE * 2]; - hts_savename_listed(strc, s, savelst, sizeof(savelst)); + hts_savename_listed(strc->path, s, savelst, sizeof(savelst)); fprintf(strc->lst, "[%s]" LF, savelst); fflush(strc->lst); } diff --git a/src/htscore.h b/src/htscore.h index eb9495fe5..f959678a4 100644 --- a/src/htscore.h +++ b/src/htscore.h @@ -355,10 +355,9 @@ int filecreateempty(filenote_strc * strct, const char *filename); int filenote(filenote_strc * strct, const char *s, filecreate_params * params); /* Copy into dest (destsize bytes) the form under which the local path s is - listed in new.lst: forward slashes, and the mirror root stripped when s sits - under it. That listed form is also the change report's key, so the two must - not drift. */ -void hts_savename_listed(const filenote_strc *strct, const char *s, char *dest, + listed in new.lst: forward slashes, with the mirror `root` stripped when s + sits under it. Also the change report's key, so the two must not drift. */ +void hts_savename_listed(const char *root, const char *s, char *dest, size_t destsize); void file_notify(httrackp * opt, const char *adr, const char *fil, diff --git a/src/htsselftest.c b/src/htsselftest.c index 5a3b40741..62ab0e916 100644 --- a/src/htsselftest.c +++ b/src/htsselftest.c @@ -5230,6 +5230,104 @@ static int st_changes(httrackp *opt, int argc, char **argv) { return err; } +#define CHANGES_RACE_FILES 8 +#define CHANGES_RACE_ROUNDS 400 + +static void changes_race_notify(httrackp *opt, int n) { + char fil[64]; + char BIGSTK save[HTS_URLMAXSIZE * 2]; + + snprintf(fil, sizeof(fil), "/f%d.bin", n); + strlcpybuff(save, StringBuff(opt->path_html), sizeof(save)); + strlcatbuff(save, "race.example", sizeof(save)); + strlcatbuff(save, fil, sizeof(save)); + hts_changes_notify(opt, "race.example", fil, save, HTS_TRUE, HTS_FALSE); +} + +static void changes_race_thread(void *arg) { + httrackp *const opt = (httrackp *) arg; + int i; + + for (i = 0; i < CHANGES_RACE_ROUNDS; i++) + changes_race_notify(opt, i % CHANGES_RACE_FILES); +} + +/* A transfer thread the crawl never joins (FTP) reaches hts_changes_notify() + while the report is being resolved and written. Run it under TSan. */ +static int st_changes_race(httrackp *opt, int argc, char **argv) { + String out = STRING_EMPTY; + char base[HTS_URLMAXSIZE]; + int err = 0; + int i; + + if (argc < 1) { + fprintf(stderr, "usage: -#test=changes-race \n"); + return 1; + } + strcpybuff(base, argv[0]); + if (base[0] != '\0' && base[strlen(base) - 1] != '/') + strcatbuff(base, "/"); + StringCopy(opt->path_html, base); + StringCopy(opt->path_html_utf8, base); + StringCopy(opt->path_log, base); + opt->changes = HTS_TRUE; + hts_changes_free_opt(opt); + + /* Real files, so the reader hashes and stats them for as long as it takes. */ + { + char BIGSTK dir[HTS_URLMAXSIZE * 2]; + + strlcpybuff(dir, base, sizeof(dir)); + strlcatbuff(dir, "race.example", sizeof(dir)); + for (i = 0; i < CHANGES_RACE_FILES; i++) { + char BIGSTK path[HTS_URLMAXSIZE * 2]; + char name[64]; + FILE *fp; + int n; + + snprintf(name, sizeof(name), "/f%d.bin", i); + strlcpybuff(path, dir, sizeof(path)); + strlcatbuff(path, name, sizeof(path)); + structcheck(path); + fp = FOPEN(path, "wb"); + if (fp == NULL) { + fprintf(stderr, "changes-race: cannot write %s\n", path); + return 1; + } + for (n = 0; n < 16384; n++) + fwrite("0123456789abcdef", 1, 16, fp); + fclose(fp); + } + } + + for (i = 0; i < 4; i++) { + if (hts_newthread(changes_race_thread, opt) != 0) { + fprintf(stderr, "changes-race: cannot spawn a notifier thread\n"); + return 1; + } + } + for (i = 0; i < 64; i++) + hts_changes_report(opt, &out); + hts_changes_close_opt(opt); + htsthread_wait(); + + /* Sealed: a straggler must be dropped, not start a report nobody writes. */ + changes_race_notify(opt, CHANGES_RACE_FILES + 1); + hts_changes_report(opt, &out); + if (StringLength(out) == 0) { + fprintf(stderr, "changes-race: the report was lost after close\n"); + err = 1; + } else if (strstr(StringBuff(out), "f9.bin") != NULL) { + fprintf(stderr, "changes-race: a post-close notify reached the report\n"); + err = 1; + } + + StringFree(out); + hts_changes_free_opt(opt); + printf("changes-race self-test: %s\n", err ? "FAIL" : "OK"); + return err; +} + /* ------------------------------------------------------------ */ /* Registry: name -> handler, with a usage hint and a one-line description. */ /* ------------------------------------------------------------ */ @@ -5287,6 +5385,8 @@ static const struct selftest_entry { {"copyopt", "", "copy_htsopt option-copy self-test", st_copyopt}, {"changes", "", "--changes bucket accounting and JSON escaping (#714)", st_changes}, + {"changes-race", "", "--changes under a late transfer thread (#714)", + st_changes_race}, {"pause", "", "randomized inter-file pause target self-test", st_pause}, {"relative", " ", "relative link between two paths", st_relative}, diff --git a/tests/01_engine-changes.test b/tests/01_engine-changes.test index 28e70e946..28953d7ed 100644 --- a/tests/01_engine-changes.test +++ b/tests/01_engine-changes.test @@ -3,5 +3,28 @@ set -euo pipefail +tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_changes_st.XXXXXX") || exit 1 +trap 'rm -rf "$tmpdir"' EXIT HUP INT QUIT PIPE TERM + +# No pipe into grep: SIGPIPE would mask a failing exit status. +expect_ok() { + local label="$1" out + shift + out=$("$@" 2>&1) || { + echo "FAIL: ${label} exited non-zero: ${out}" + exit 1 + } + case "$out" in + *"${label}: OK"*) ;; + *) + echo "FAIL: ${out}" + exit 1 + ;; + esac +} + # --changes bucket accounting and JSON escaping (#714). -httrack -O /dev/null -#test=changes run | grep -q "changes self-test: OK" +expect_ok "changes self-test" httrack -O "${tmpdir}/o1" -#test=changes run +# The report must hold against a transfer thread the crawl never joins. +expect_ok "changes-race self-test" httrack -O "${tmpdir}/o2" \ + -#test=changes-race "${tmpdir}/race" diff --git a/tests/89_local-changes.test b/tests/89_local-changes.test index d13dc1bf9..bf0a32618 100644 --- a/tests/89_local-changes.test +++ b/tests/89_local-changes.test @@ -1,9 +1,8 @@ #!/bin/bash # --changes (#714): the report must match the mirror's real deltas. Every route # answers 200 with no validators on both passes, so the transfer signal alone -# would call the whole site changed; the stable resources land in "unchanged" -# only if payloads are really compared. -set -eu +# would call the whole site changed. +set -euo pipefail : "${top_srcdir:=..}" testdir=$(cd "$(dirname "$0")" && pwd) @@ -52,15 +51,15 @@ which httrack >/dev/null || { out="${tmpdir}/crawl" mkdir "$out" report="${out}/hts-changes.json" -# Purging off throughout: the deleted set must be computed with nothing acting -# on it. -common=(-O "$out" --quiet --disable-security-limits --robots=0 --timeout=30 --changes --purge-old=0 --retries=2) +# Purging off for the first two passes: the deleted set must be computed with +# nothing acting on it. +common=(--quiet --disable-security-limits --robots=0 --timeout=30 --changes --retries=2) -# Report field extractor: bucket counts, and the files listed under a bucket. -count() { sed -n 's/.*"'"$1"'": \([0-9]*\).*/\1/p' "$report" | head -1; } +# Report field extractor: a scalar, and the files listed under a bucket. +field() { sed -n 's/.*"'"$1"'": \([a-z0-9]*\).*/\1/p' "${2:-$report}" | head -1; } # tr: python prints CRLF on Windows, and the expectations below are LF. listed() { - "$python" - "$report" "$1" <<'EOF' | tr -d '\r' + "$python" - "${2:-$report}" "$1" <<'EOF' | tr -d '\r' import json import sys @@ -71,70 +70,90 @@ for entry in report[sys.argv[2]]: EOF } expect() { - local label="$1" want="$2" got="$3" + local label="$1" want="$2" got="$3" rep="${4:-$report}" printf '[%s] ..\t' "$label" test "$want" == "$got" || { echo "FAIL: expected '${want}', got '${got}'" - cat "$report" + cat "$rep" exit 1 } echo "OK" } +# Every count must equal the length of the list it counts. +expect_counts_match() { + local label="$1" rep="${2:-$report}" bucket n + printf '[%s] ..\t' "$label" + for bucket in new changed unchanged gone; do + n=$(listed "$bucket" "$rep" | grep -c . || true) + test "$(field "$bucket" "$rep")" == "$n" || { + echo "FAIL: ${bucket} count $(field "$bucket" "$rep") but $n listed" + cat "$rep" + exit 1 + } + done + echo "OK" +} +lines() { printf '%s\n' "$@" | sort; } # --- pass 1: nothing to compare against -------------------------------------- -httrack "${common[@]}" "${base}/changes/index.html" >"${tmpdir}/log1" 2>&1 +httrack "${common[@]}" -O "$out" --purge-old=0 "${base}/changes/index.html" \ + >"${tmpdir}/log1" 2>&1 test -s "$report" || { echo "FAIL: pass 1 wrote no ${report}" exit 1 } -expect "pass 1 declares a first crawl" "true" \ - "$(sed -n 's/.*"first_crawl": \([a-z]*\).*/\1/p' "$report" | head -1)" +expect "pass 1 declares a first crawl" "true" "$(field first_crawl)" # Every mirrored file: index, stable.html, moved.html, stable.bin, moved.bin, # doomed.html, redirtarget.html, flaky.bin, coded.bin, codedstable.bin, -# sized.html and the redirect target it points at. -expect "pass 1 counts 12 new" "12" "$(count new)" -expect "pass 1 counts 0 changed" "0" "$(count changed)" -expect "pass 1 counts 0 gone" "0" "$(count gone)" +# killed.bin, sized.html and the redirect target it points at. +expect "pass 1 counts 13 new" "13" "$(field new)" +expect "pass 1 counts 0 changed" "0" "$(field changed)" +expect "pass 1 counts 0 gone" "0" "$(field gone)" +expect_counts_match "pass 1 counts match its lists" grep -aq "first crawl" "${out}/hts-log.txt" || { echo "FAIL: pass 1 log carries no first-crawl summary" exit 1 } -# --- pass 2: two bodies move, one page appears, one disappears --------------- -httrack "${common[@]}" --update "${base}/changes/index.html" >"${tmpdir}/log2" 2>&1 -expect "pass 2 is not a first crawl" "false" \ - "$(sed -n 's/.*"first_crawl": \([a-z]*\).*/\1/p' "$report" | head -1)" - host="127.0.0.1_${port}" +# A leftover from some earlier failed attempt, at a name pass 2 fetches for the +# first time: on disk, but never part of the previous mirror, so it is new. +printf 'leftover junk' >"${out}/${host}/changes/fresh.html" + +# --- pass 2: two bodies move, pages appear and disappear --------------------- +httrack "${common[@]}" -O "$out" --purge-old=0 --update \ + "${base}/changes/index.html" >"${tmpdir}/log2" 2>&1 +expect "pass 2 is not a first crawl" "false" "$(field first_crawl)" + # The redirect behind sized.html now points at a much longer target name. long="${host}/changes/$(printf 's%.0s' $(seq 40)).html" -expect "new is fresh.html and the renamed redirect target" \ - "$(printf '%s\n%s' "${host}/changes/fresh.html" "$long" | sort)" \ +expect "new is the two fresh pages and the renamed redirect target" \ + "$(lines "${host}/changes/fresh.html" "${host}/changes/transient.html" "$long")" \ "$(listed new | sort)" expect "gone is doomed.html and the old redirect target" \ - "$(printf '%s\n%s' "${host}/changes/doomed.html" "${host}/changes/s.html" | sort)" \ + "$(lines "${host}/changes/doomed.html" "${host}/changes/s.html")" \ "$(listed gone | sort)" # index.html changes because its link list does; moved.* change their bodies. # coded.bin arrives gzipped and direct-to-disk, so its previous copy has to be # sampled before the decoded temp is renamed over it; it is the same length on # both passes, so only a digest separates it from codedstable.bin. expect "changed is exactly the four moved resources" \ - "$(printf '%s\n%s\n%s\n%s' "${host}/changes/coded.bin" \ - "${host}/changes/index.html" "${host}/changes/moved.bin" \ - "${host}/changes/moved.html")" \ + "$(lines "${host}/changes/coded.bin" "${host}/changes/index.html" \ + "${host}/changes/moved.bin" "${host}/changes/moved.html")" \ "$(listed changed | sort)" -# The teeth: re-served byte for byte, 200, no Last-Modified, no ETag. flaky.bin -# gets there through a failed transfer and a retry, so its file is notified -# twice; redirtarget.html arrives behind a 302. -# sized.html is the sharpest of these: its payload is byte-identical, but the -# rewritten link to the renamed redirect target makes the file on disk change -# length, so a size comparison alone would call it changed. -expect "unchanged is exactly the six stable resources" \ - "$(printf '%s\n%s\n%s\n%s\n%s\n%s' "${host}/changes/codedstable.bin" \ - "${host}/changes/flaky.bin" "${host}/changes/redirtarget.html" \ +# Re-served byte for byte, 200, no Last-Modified, no ETag. flaky.bin gets there +# through a failed transfer and a retry, so its file is notified twice; +# redirtarget.html arrives behind a 302. sized.html has a byte-identical +# payload, but the rewritten link to the renamed redirect target makes the file +# on disk change length. killed.bin never completes a transfer at all, so it +# reaches none of the lists the engine writes, and its copy is untouched. +expect "unchanged is exactly the seven stable resources" \ + "$(lines "${host}/changes/codedstable.bin" "${host}/changes/flaky.bin" \ + "${host}/changes/killed.bin" "${host}/changes/redirtarget.html" \ "${host}/changes/sized.html" "${host}/changes/stable.bin" \ - "${host}/changes/stable.html" | sort)" \ + "${host}/changes/stable.html")" \ "$(listed unchanged | sort)" +expect_counts_match "pass 2 counts match its lists" # Every mirrored file appears once and only once, across all four buckets. printf '[each file in exactly one bucket] ..\t' @@ -150,13 +169,72 @@ test -z "$dupes" || { } echo "OK" -# Before #714 the deleted set only existed as a side effect of purging; it must -# now be computed without one, and reporting it must not delete anything. -expect "the report says nothing was purged" "false" \ - "$(sed -n 's/.*"purged": \([a-z]*\).*/\1/p' "$report" | head -1)" +# The log summary is the report, rendered: it must not drift from the counts. +summary="Change report: $(field new) new, $(field changed) changed, $(field unchanged) unchanged, $(field gone) gone (hts-changes.json)" +printf '[the log summary matches the counts] ..\t' +grep -aqF "$summary" "${out}/hts-log.txt" || { + echo "FAIL: no '${summary}' in the log" + grep -a "Change report" "${out}/hts-log.txt" || true + exit 1 +} +echo "OK" + +expect "the report says nothing was purged" "false" "$(field purged)" printf '[reporting gone did not delete it] ..\t' test -f "${out}/${host}/changes/doomed.html" || { echo "FAIL: doomed.html was deleted with --purge-old=0" exit 1 } echo "OK" +printf '[a killed transfer keeps its file] ..\t' +test -f "${out}/${host}/changes/killed.bin" || { + echo "FAIL: killed.bin lost its previous copy" + exit 1 +} +echo "OK" + +# --- pass 3: the same run with purging on ------------------------------------ +httrack "${common[@]}" -O "$out" --update "${base}/changes/index.html" \ + >"${tmpdir}/log3" 2>&1 +expect "the report says files were purged" "true" "$(field purged)" +expect "gone is the page that only pass 2 linked" \ + "${host}/changes/transient.html" "$(listed gone)" +printf '[a purged file is off disk] ..\t' +test ! -f "${out}/${host}/changes/transient.html" || { + echo "FAIL: transient.html survived --purge-old" + exit 1 +} +echo "OK" + +# --- cache off: the report degrades, it does not lie ------------------------- +nocache="${tmpdir}/nocache" +mkdir "$nocache" +ncreport="${nocache}/hts-changes.json" +for pass in 1 2; do + httrack "${common[@]}" -O "$nocache" -C0 "${base}/changes2/index.html" \ + >"${tmpdir}/lognc${pass}" 2>&1 +done +expect "with no cache first_crawl is undetermined, not guessed" "null" \ + "$(field first_crawl "$ncreport")" "$ncreport" +# The bodies on disk are still compared, so a constant one is not called +# changed. index.html is: a parsed page is compared against the cached payload, +# and there is no cache, which is the degradation the report declares. +expect "the constant body is still recognised as unchanged" \ + "${host}/changes2/stable2.bin" "$(listed unchanged "$ncreport")" "$ncreport" +expect "the moving body is still recognised as changed" \ + "$(lines "${host}/changes2/index.html" "${host}/changes2/ticker2.bin")" \ + "$(listed changed "$ncreport" | sort)" "$ncreport" +expect_counts_match "cache-off counts match its lists" "$ncreport" +printf '[every entry carries its size] ..\t' +"$python" - "$ncreport" <<'EOF' || exit 1 +import json +import sys + +with open(sys.argv[1], encoding="utf-8") as fp: + report = json.load(fp) +for bucket in ("new", "changed", "unchanged"): + for entry in report[bucket]: + if "size" not in entry: + sys.exit("FAIL: %s has no size in %s" % (entry["file"], bucket)) +EOF +echo "OK" diff --git a/tests/local-server.py b/tests/local-server.py index 9e9ba8c02..d3ea961b8 100755 --- a/tests/local-server.py +++ b/tests/local-server.py @@ -1587,12 +1587,18 @@ def route_changes_index(self): "coded.bin", "codedstable.bin", "sized.html", + "killed.bin", ) ) - if self.refetch_pass() == 1: + seen = self.refetch_pass() + if seen == 1: links += '\tdoomed\n' else: links += '\tfresh\n' + # transient.html appears on pass 2 only, so pass 3 has a deletion of + # its own to purge. + if seen == 2: + links += '\ttransient\n' self.send_html(links) def route_changes_stable(self): @@ -1689,6 +1695,42 @@ def route_changes_flaky(self): else: self.send_raw(self.FLAKY, "application/octet-stream") + def route_changes_transient(self): + self.refetch_pass() + self.send_raw( + b"

    CHANGES-TRANSIENT

    ", "text/html" + ) + + # Transfers fine once, then dies mid-body on every later attempt, retries + # included: the mirrored copy survives untouched but the file never reaches + # new.lst, so old.lst minus new.lst offers it up as a deletion. + KILLED = b"CHANGES-KILLED\n" + b"\x21\x22\x23\x24" * 512 + + def route_changes_killed(self): + if self.refetch_pass() == 1: + self.send_raw(self.KILLED, "application/octet-stream") + else: + self.send_truncated(self.KILLED, "application/octet-stream") + + # A second, independent mirror crawled with the cache off: nothing but the + # bytes on disk can tell stable2 from ticker2, whose body differs every + # fetch at a constant length. + def route_changes2_index(self): + self.refetch_pass() + self.send_html('\ts\n\tt\n') + + def route_changes2_stable(self): + self.refetch_pass() + self.send_raw( + b"CHANGES2-STABLE-00\n" + b"\x55\xaa" * 512, "application/octet-stream" + ) + + def route_changes2_ticker(self): + self.send_raw( + b"CHANGES2-TICKER-%02d\n" % (self.refetch_pass() % 100) + b"\x55\xaa" * 512, + "application/octet-stream", + ) + ROUTES = { "/cookies/entrance.php": route_entrance, "/cookies/second.php": route_second, @@ -1722,6 +1764,11 @@ def route_changes_flaky(self): "/changes/" + SIZED_SHORT: route_changes_sizedtarget, "/changes/" + SIZED_LONG: route_changes_sizedtarget, "/changes/codedstable.bin": route_changes_coded_stable, + "/changes/transient.html": route_changes_transient, + "/changes/killed.bin": route_changes_killed, + "/changes2/index.html": route_changes2_index, + "/changes2/stable2.bin": route_changes2_stable, + "/changes2/ticker2.bin": route_changes2_ticker, "/upcodec/index.html": route_upcodec_index, "/upcodec/mem.html": route_upcodec_mem, "/upcodec/disk.bin": route_upcodec_disk, From 8ebe99be9db1eef4bf68a2665479ac2e48a5d8ce Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 22:57:15 +0200 Subject: [PATCH 10/11] changes: prove the fixes, and give the report a cache-off mode Adds a changes-race self-test (the FTP shape: notifier threads against the report path), fixtures for a transfer the crawl never completes, for a leftover file at a name the crawl mirrors fresh, and for a cache-off mirror, plus a pass with purging on. Registers the web GUI's --changes box with the clearing companion master's #725 now requires, and documents the degraded mode. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- html/changes.html | 258 +++++++++++++----------- html/server/option9.html | 1 + src/htschanges.c | 20 +- src/htschanges.h | 13 +- src/htsselftest.c | 33 ++- tests/89_local-changes.test | 12 +- tests/90_webhttrack-checkbox-clear.test | 1 + tests/local-server.py | 19 +- 8 files changed, 201 insertions(+), 156 deletions(-) diff --git a/html/changes.html b/html/changes.html index bf3289462..24ede946c 100644 --- a/html/changes.html +++ b/html/changes.html @@ -97,130 +97,148 @@ diff --git a/html/server/option9.html b/html/server/option9.html index bbc6508d2..93f902321 100644 --- a/html/server/option9.html +++ b/html/server/option9.html @@ -100,6 +100,7 @@

    ${LANG_O2} - ${LANG_IOPT9}

    + diff --git a/src/htschanges.c b/src/htschanges.c index 853daa932..59991816d 100644 --- a/src/htschanges.c +++ b/src/htschanges.c @@ -270,14 +270,12 @@ hts_change_bucket hts_changes_classify(hts_boolean rewritten, /* Engine hooks */ /* ------------------------------------------------------------ */ -/* FTP transfers reach file_notify() from a detached thread of their own, and - the crawl is not joined before the report is written, so every entry point - below (report and teardown included) runs under this lock. */ +/* FTP transfers reach file_notify() from a thread the crawl never joins, so + every entry point below, report and teardown included, takes this lock. */ static htsmutex changes_mutex = HTSMUTEX_INIT; -/* The live accumulator, created on first use. NULL when --changes is off or - the report has been written: a straggler notify must be dropped, not - resurrect an accumulator nobody will report. Call under the lock. */ +/* The live accumulator, created on first use; NULL when --changes is off or + the report is already written. Call under the lock. */ static hts_changes *changes_get(httrackp *opt) { hts_changes *changes = (hts_changes *) opt->changes_state; @@ -333,16 +331,15 @@ void hts_changes_notify(httrackp *opt, const char *adr, const char *fil, strlcatbuff(url, adr, sizeof(url)); strlcatbuff(url, fil, sizeof(url)); - /* Sample the previous copy with the lock released, and hash it only when it - is about to be overwritten: this is the last moment those bytes exist. */ + /* Unlocked: hashing a large file would stall every other connection. Hash + it only when it is about to be overwritten, the last moment it exists. */ prev_size = fsize_utf8(save); if (prev_size >= 0 && rewritten) has_prev = digest_file(save, prev_digest); hts_mutexlock(&changes_mutex); changes = changes_get(opt); - /* The slot index is stable (entries are only ever appended), but realloct() - may have moved the array while the lock was down. */ + /* The slot index is stable, entries being append-only; the array is not. */ if (changes != NULL && (size_t) slot < changes->count) { changes_entry *const entry = &changes->entries[slot]; @@ -373,8 +370,7 @@ void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r, intptr_t slot; htsblk prev; - /* Before the digest and the cache read: with --changes off this hook must - cost nothing. */ + /* Ahead of the digest and the cache read: off must cost nothing. */ if (!opt->changes) return; if (r->adr == NULL || r->size < 0 || save == NULL || !strnotempty(save)) diff --git a/src/htschanges.h b/src/htschanges.h index c5b9a86e1..613618c7d 100644 --- a/src/htschanges.h +++ b/src/htschanges.h @@ -88,19 +88,16 @@ void hts_changes_dropped(httrackp *opt, const char *file, hts_boolean kept); file's presence on disk, decides what counts as already mirrored. */ void hts_changes_previous(httrackp *opt, const char *file); -/* Record that this run keeps a mirror index at all. Without one (the cache is - off) deletions cannot be seen and first_crawl cannot be decided, which the - report states rather than guesses. */ +/* Record that this run keeps a mirror index. Without one (the cache is off) + deletions and first_crawl are undecidable, and the report says so. */ void hts_changes_indexed(httrackp *opt); /* Resolve every entry against the bytes now on disk and serialize the report - into `out` (replaced). Exposed for the `changes` self-test; the engine goes - through hts_changes_close_opt(). */ + into `out` (replaced). Exposed for the self-tests. */ void hts_changes_report(httrackp *opt, String *out); -/* Write the report and log a one-line summary. Idempotent, and after it the - accumulator is sealed: a late notify from a still-running transfer thread is - dropped instead of starting a report nobody will write. */ +/* Write the report and log a one-line summary. Idempotent, and seals the + accumulator: a late notify is dropped rather than starting a second one. */ void hts_changes_close_opt(httrackp *opt); /* Drop the accumulator, for a run that never reached its end and to start the diff --git a/src/htsselftest.c b/src/htsselftest.c index 62ab0e916..60ca4dd7a 100644 --- a/src/htsselftest.c +++ b/src/htsselftest.c @@ -5244,12 +5244,33 @@ static void changes_race_notify(httrackp *opt, int n) { hts_changes_notify(opt, "race.example", fil, save, HTS_TRUE, HTS_FALSE); } +/* htsthread_wait() counts a thread only once it is running, so it can return + before any of them started; join on our own counter instead. */ +static htsmutex changes_race_lock = HTSMUTEX_INIT; +static int changes_race_started = 0; +static int changes_race_live = 0; + +static int changes_race_count(int *which) { + int n; + + hts_mutexlock(&changes_race_lock); + n = *which; + hts_mutexrelease(&changes_race_lock); + return n; +} + static void changes_race_thread(void *arg) { httrackp *const opt = (httrackp *) arg; int i; + hts_mutexlock(&changes_race_lock); + changes_race_started++; + hts_mutexrelease(&changes_race_lock); for (i = 0; i < CHANGES_RACE_ROUNDS; i++) changes_race_notify(opt, i % CHANGES_RACE_FILES); + hts_mutexlock(&changes_race_lock); + changes_race_live--; + hts_mutexrelease(&changes_race_lock); } /* A transfer thread the crawl never joins (FTP) reaches hts_changes_notify() @@ -5300,16 +5321,26 @@ static int st_changes_race(httrackp *opt, int argc, char **argv) { } } + /* Take both locks once here: hts_mutexlock() initializes lazily, and two + threads reaching a fresh one together race on the init itself. */ + hts_mutexlock(&changes_race_lock); + changes_race_started = 0; + changes_race_live = 4; + hts_mutexrelease(&changes_race_lock); for (i = 0; i < 4; i++) { if (hts_newthread(changes_race_thread, opt) != 0) { fprintf(stderr, "changes-race: cannot spawn a notifier thread\n"); return 1; } } + /* Report only once they are all notifying, or there is nothing to race. */ + while (changes_race_count(&changes_race_started) < 4) + Sleep(10); for (i = 0; i < 64; i++) hts_changes_report(opt, &out); hts_changes_close_opt(opt); - htsthread_wait(); + while (changes_race_count(&changes_race_live) > 0) + Sleep(10); /* Sealed: a straggler must be dropped, not start a report nobody writes. */ changes_race_notify(opt, CHANGES_RACE_FILES + 1); diff --git a/tests/89_local-changes.test b/tests/89_local-changes.test index bf0a32618..9cc5d0d88 100644 --- a/tests/89_local-changes.test +++ b/tests/89_local-changes.test @@ -105,7 +105,7 @@ test -s "$report" || { expect "pass 1 declares a first crawl" "true" "$(field first_crawl)" # Every mirrored file: index, stable.html, moved.html, stable.bin, moved.bin, # doomed.html, redirtarget.html, flaky.bin, coded.bin, codedstable.bin, -# killed.bin, sized.html and the redirect target it points at. +# reset.bin, sized.html and the redirect target it points at. expect "pass 1 counts 13 new" "13" "$(field new)" expect "pass 1 counts 0 changed" "0" "$(field changed)" expect "pass 1 counts 0 gone" "0" "$(field gone)" @@ -145,11 +145,11 @@ expect "changed is exactly the four moved resources" \ # through a failed transfer and a retry, so its file is notified twice; # redirtarget.html arrives behind a 302. sized.html has a byte-identical # payload, but the rewritten link to the renamed redirect target makes the file -# on disk change length. killed.bin never completes a transfer at all, so it +# on disk change length. reset.bin never completes a transfer at all, so it # reaches none of the lists the engine writes, and its copy is untouched. expect "unchanged is exactly the seven stable resources" \ "$(lines "${host}/changes/codedstable.bin" "${host}/changes/flaky.bin" \ - "${host}/changes/killed.bin" "${host}/changes/redirtarget.html" \ + "${host}/changes/reset.bin" "${host}/changes/redirtarget.html" \ "${host}/changes/sized.html" "${host}/changes/stable.bin" \ "${host}/changes/stable.html")" \ "$(listed unchanged | sort)" @@ -186,9 +186,9 @@ test -f "${out}/${host}/changes/doomed.html" || { exit 1 } echo "OK" -printf '[a killed transfer keeps its file] ..\t' -test -f "${out}/${host}/changes/killed.bin" || { - echo "FAIL: killed.bin lost its previous copy" +printf '[a failed transfer keeps its file] ..\t' +test -f "${out}/${host}/changes/reset.bin" || { + echo "FAIL: reset.bin lost its previous copy" exit 1 } echo "OK" diff --git a/tests/90_webhttrack-checkbox-clear.test b/tests/90_webhttrack-checkbox-clear.test index 281cb9e95..c07bc2b29 100755 --- a/tests/90_webhttrack-checkbox-clear.test +++ b/tests/90_webhttrack-checkbox-clear.test @@ -150,6 +150,7 @@ BOXES = [ ("toler", "TolerantRequests", "--tolerant", None), ("http10", "HTTP10", "--http-10", None), ("warc", "Warc", "--warc", None), + ("changes", "Changes", "--changes", None), ("norecatch", "NoRecatch", "--do-not-recatch", None), ("logf", "Log", "--single-log", None), ("index", "Index", None, None), diff --git a/tests/local-server.py b/tests/local-server.py index d3ea961b8..c3de826d0 100755 --- a/tests/local-server.py +++ b/tests/local-server.py @@ -1587,7 +1587,7 @@ def route_changes_index(self): "coded.bin", "codedstable.bin", "sized.html", - "killed.bin", + "reset.bin", ) ) seen = self.refetch_pass() @@ -1701,16 +1701,17 @@ def route_changes_transient(self): b"

    CHANGES-TRANSIENT

    ", "text/html" ) - # Transfers fine once, then dies mid-body on every later attempt, retries - # included: the mirrored copy survives untouched but the file never reaches - # new.lst, so old.lst minus new.lst offers it up as a deletion. - KILLED = b"CHANGES-KILLED\n" + b"\x21\x22\x23\x24" * 512 + # Answers once, then hangs up before sending anything on every later + # attempt, retries included: nothing is written, so the file never reaches + # new.lst while its mirrored copy stays intact. + RESET = b"CHANGES-RESET\n" + b"\x21\x22\x23\x24" * 512 - def route_changes_killed(self): + def route_changes_reset(self): if self.refetch_pass() == 1: - self.send_raw(self.KILLED, "application/octet-stream") + self.send_raw(self.RESET, "application/octet-stream") else: - self.send_truncated(self.KILLED, "application/octet-stream") + self.close_connection = True + self.connection.close() # A second, independent mirror crawled with the cache off: nothing but the # bytes on disk can tell stable2 from ticker2, whose body differs every @@ -1765,7 +1766,7 @@ def route_changes2_ticker(self): "/changes/" + SIZED_LONG: route_changes_sizedtarget, "/changes/codedstable.bin": route_changes_coded_stable, "/changes/transient.html": route_changes_transient, - "/changes/killed.bin": route_changes_killed, + "/changes/reset.bin": route_changes_reset, "/changes2/index.html": route_changes2_index, "/changes2/stable2.bin": route_changes2_stable, "/changes2/ticker2.bin": route_changes2_ticker, From 1d61bf94cae5d7dd8e788a0a7dac0d59d561bf79 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 23:23:10 +0200 Subject: [PATCH 11/11] changes: keep the failed-transfer case portable A connection killed before the status line surfaces differently on macOS, where it truncates the mirrored file to zero (#748). Assert only what holds on both: it is never reported gone, and its file survives. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- tests/93_local-changes.test | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/tests/93_local-changes.test b/tests/93_local-changes.test index 9cc5d0d88..ba465bb3c 100644 --- a/tests/93_local-changes.test +++ b/tests/93_local-changes.test @@ -94,6 +94,10 @@ expect_counts_match() { echo "OK" } lines() { printf '%s\n' "$@" | sort; } +# A connection killed before the status line surfaces differently per platform +# (macOS truncates the file, #748; Linux leaves it), so reset.bin is asserted on +# its own and kept out of the exact lists. +listed_but_reset() { listed "$1" | grep -v '/reset\.bin$' | sort; } # --- pass 1: nothing to compare against -------------------------------------- httrack "${common[@]}" -O "$out" --purge-old=0 "${base}/changes/index.html" \ @@ -140,19 +144,21 @@ expect "gone is doomed.html and the old redirect target" \ expect "changed is exactly the four moved resources" \ "$(lines "${host}/changes/coded.bin" "${host}/changes/index.html" \ "${host}/changes/moved.bin" "${host}/changes/moved.html")" \ - "$(listed changed | sort)" + "$(listed_but_reset changed)" # Re-served byte for byte, 200, no Last-Modified, no ETag. flaky.bin gets there # through a failed transfer and a retry, so its file is notified twice; # redirtarget.html arrives behind a 302. sized.html has a byte-identical # payload, but the rewritten link to the renamed redirect target makes the file -# on disk change length. reset.bin never completes a transfer at all, so it -# reaches none of the lists the engine writes, and its copy is untouched. -expect "unchanged is exactly the seven stable resources" \ +# on disk change length. +expect "unchanged is exactly the six stable resources" \ "$(lines "${host}/changes/codedstable.bin" "${host}/changes/flaky.bin" \ - "${host}/changes/reset.bin" "${host}/changes/redirtarget.html" \ - "${host}/changes/sized.html" "${host}/changes/stable.bin" \ - "${host}/changes/stable.html")" \ - "$(listed unchanged | sort)" + "${host}/changes/redirtarget.html" "${host}/changes/sized.html" \ + "${host}/changes/stable.bin" "${host}/changes/stable.html")" \ + "$(listed_but_reset unchanged)" +# reset.bin never completes a transfer, so it drops out of new.lst with its +# mirrored copy still there. Whatever else it is, it is not a deletion. +expect "a failed re-fetch is never reported gone" "" \ + "$(listed gone | grep '/reset\.bin$' || true)" expect_counts_match "pass 2 counts match its lists" # Every mirrored file appears once and only once, across all four buckets. @@ -198,7 +204,7 @@ httrack "${common[@]}" -O "$out" --update "${base}/changes/index.html" \ >"${tmpdir}/log3" 2>&1 expect "the report says files were purged" "true" "$(field purged)" expect "gone is the page that only pass 2 linked" \ - "${host}/changes/transient.html" "$(listed gone)" + "${host}/changes/transient.html" "$(listed_but_reset gone)" printf '[a purged file is off disk] ..\t' test ! -f "${out}/${host}/changes/transient.html" || { echo "FAIL: transient.html survived --purge-old"
    - - -

    Change report format specification

    - -
    - -Run with --changes (-%d), HTTrack writes hts-changes.json -in the project directory, next to hts-log.txt, describing what the crawl -left new, changed, unchanged and gone compared to the previous mirror. The file is -rewritten from scratch at the end of every run, and the log carries a one-line -summary of the same counts. - -

    - -

    What "changed" means

    - -A resource is changed when its bytes differ, not when the server merely re-sent -it. HTTrack compares the payload it just received against the copy the previous -run left behind: for pages it parses, the previous payload comes from the cache -(the file on disk carries the mirror footer and its crawl date, so its bytes -differ on every run); for everything else, the mirrored file is the payload -verbatim and is compared directly. - -

    - -Where no digest can be taken on either side, because the cache is disabled or -the previous copy is gone, the report falls back to the transfer signal, and a -server that answers 200 rather than 304 reads as changed. Keeping the cache on -(the default) is what makes the report precise. - -

    - -

    Fields

    - -
      -
    • schema: format version, currently 1. It is bumped only -on an incompatible change; new fields may appear without one.
      • -
    • generator: the HTTrack build that wrote the file.
      • -
    • date: when the report was written, UTC, YYYY-MM-DDThh:mm:ssZ.
      • -
    • first_crawl: true when no index of a previous mirror -(hts-cache/old.lst) was found, so there was nothing to compare against and -everything is listed as new.
      • -
    • partial: true when the report ran out of memory and lists only -part of the mirror.
      • -
    • purged: true when --purge-old was in effect, so the -files under gone were also deleted from disk.
      • -
    • counts: the size of each of the four lists.
      • -
    • new, changed, unchanged, gone: the -lists themselves. Every mirrored file appears in exactly one of them.
      • -
    - -Each entry is an object: - -
      -
    • url: the absolute URL the file came from. Empty under -gone: deletions are computed from the mirror's file index, which records -paths, not URLs.
      • -
    • file: the path relative to the mirror root, with forward -slashes. This is the entry's identity: a URL and a redirect that resolve to the -same local file are one entry, not two.
      • -
    • size: the mirrored file's size in bytes, absent when the file -is not on disk.
      • -
    • previous_size: under changed only, the size of the -copy the previous run left.
      • -
    - -
    - -

    Encoding

    - -The file is JSON, UTF-8. URLs and local paths reach HTTrack as raw bytes and are -not guaranteed to be valid UTF-8; any byte sequence that is not becomes -U+FFFD (\ufffd), so the file always parses. Compare on file -rather than on url when a mirror is known to carry legacy-charset URLs. - -

    - -

    Example

    - -
    -{
-  "schema": 1,
-  "generator": "HTTrack Website Copier/3.49-14",
-  "date": "2026-07-26T15:29:03Z",
-  "first_crawl": false,
-  "partial": false,
-  "purged": true,
-  "counts": { "new": 1, "changed": 1, "unchanged": 1, "gone": 1 },
-  "new": [
-    { "url": "http://example.com/d.html", "file": "example.com/d.html", "size": 280 }
-  ],
-  "changed": [
-    { "url": "http://example.com/a.html", "file": "example.com/a.html", "size": 281, "previous_size": 273 }
-  ],
-  "unchanged": [
-    { "url": "http://example.com/b.html", "file": "example.com/b.html", "size": 277 }
-  ],
-  "gone": [
-    { "url": "", "file": "example.com/c.html" }
-  ]
-}
-
    - -

    - -

    Notes

    - -
      -
    • A file listed under gone is only deleted when --purge-old is -on. Left in place it drops out of the mirror's index, so it is reported once and -not again.
      • + + +

      Change report format specification

      + +
      + +Run with --changes (-%d), HTTrack writes hts-changes.json +in the project directory, next to hts-log.txt, describing what the crawl +left new, changed, unchanged and gone compared to the previous mirror. The file is +rewritten from scratch at the end of every run, and the log carries a one-line +summary of the same counts. + +

      + +

      What "changed" means

      + +A resource is changed when its bytes differ, not when the server merely re-sent +it. HTTrack compares the payload it just received against the copy the previous +run left behind: for pages it parses, the previous payload comes from the cache +(the file on disk carries the mirror footer and its crawl date, so its bytes +differ on every run); for everything else, the mirrored file is the payload +verbatim and is compared directly. + +

      + +Where no digest can be taken on either side, because the cache is disabled or +the previous copy is gone, the report falls back to the transfer signal, and a +server that answers 200 rather than 304 reads as changed. Keeping the cache on +(the default) is what makes the report precise. + +

      + +

      With the cache off

      + +--cache=0 costs the report more than the digest of a parsed page. The +mirror's file index (hts-cache/new.lst) is what records which files a +run produced, so without it there is no previous mirror to subtract from: nothing +is reported gone, and whether the run is a first crawl cannot be decided +at all, which first_crawl states as null rather than guess. What +is on disk is still compared byte for byte, so the other three lists stay +meaningful, except for the pages HTTrack parses: those have no cached payload to +compare against and fall back to the transfer signal. + +

      + +

      Fields

      + +
        +
      • schema: format version, currently 1. It is bumped only +on an incompatible change; new fields may appear without one.
        • +
      • generator: the HTTrack build that wrote the file.
        • +
      • date: when the report was written, UTC, YYYY-MM-DDThh:mm:ssZ.
        • +
      • first_crawl: true when no index of a previous mirror +(hts-cache/old.lst) was found, so there was nothing to compare against and +everything is listed as new. Null when the run kept no index at all and the +question cannot be answered (see above).
        • +
      • partial: true when the report ran out of memory and lists only +part of the mirror.
        • +
      • purged: true when --purge-old was in effect, so the +files under gone were also deleted from disk.
        • +
      • counts: the size of each of the four lists.
        • +
      • new, changed, unchanged, gone: the +lists themselves. Every mirrored file appears in exactly one of them.
        • +
      + +Each entry is an object: + +
        +
      • url: the absolute URL the file came from. Empty under +gone: deletions are computed from the mirror's file index, which records +paths, not URLs.
        • +
      • file: the path relative to the mirror root, with forward +slashes. This is the entry's identity: a URL and a redirect that resolve to the +same local file are one entry, not two.
        • +
      • size: the mirrored file's size in bytes, absent when the file +is not on disk.
        • +
      • previous_size: under changed only, the size of the +copy the previous run left.
        • +
      + +
      + +

      Encoding

      + +The file is JSON, UTF-8. URLs and local paths reach HTTrack as raw bytes and are +not guaranteed to be valid UTF-8; any byte sequence that is not becomes +U+FFFD (\ufffd), so the file always parses. Compare on file +rather than on url when a mirror is known to carry legacy-charset URLs. + +

      + +

      Example

      + +
      +{
+  "schema": 1,
+  "generator": "HTTrack Website Copier/3.49-14",
+  "date": "2026-07-26T15:29:03Z",
+  "first_crawl": false,
+  "partial": false,
+  "purged": true,
+  "counts": { "new": 1, "changed": 1, "unchanged": 1, "gone": 1 },
+  "new": [
+    { "url": "http://example.com/d.html", "file": "example.com/d.html", "size": 280 }
+  ],
+  "changed": [
+    { "url": "http://example.com/a.html", "file": "example.com/a.html", "size": 281, "previous_size": 273 }
+  ],
+  "unchanged": [
+    { "url": "http://example.com/b.html", "file": "example.com/b.html", "size": 277 }
+  ],
+  "gone": [
+    { "url": "", "file": "example.com/c.html" }
+  ]
+}
+
      + +

      + +

      Notes

      + +
        +
      • A file listed under gone is only deleted when --purge-old is +on. Left in place it drops out of the mirror's index, so it is reported once and +not again.
      • A resource whose local file name changed since the previous mirror (a new MIME type, say) is reported as new under its new name; the old name is reported as gone only if the file is still on disk. The two entries are not paired.
        • -
      • A run that transfers no data at all is rolled back: HTTrack restores the -previous cache generation and leaves the previous report in place, so a lost -connection does not overwrite a good report with an empty one.
        • -
      • Content diffs, and keeping the previous copy of a changed page, are out of -scope: both change what a mirror directory contains.
        • -
      - -

      - +
    • A resource this run tried and failed to transfer also drops out of the +mirror's index, but its previous copy is untouched, so it is reported +unchanged. Under --purge-old that copy is deleted anyway, and +the report says gone to match.
      • +
    • A run that transfers no data at all is rolled back: HTTrack restores the +previous cache generation and leaves the previous report in place, so a lost +connection does not overwrite a good report with an empty one.
      • +
    • Content diffs, and keeping the previous copy of a changed page, are out of +scope: both change what a mirror directory contains.
      • +
    + +

    +