From 9cc9a36fa6613c5136227b05cbd070d13701ae3a Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 17:38:02 +0200 Subject: [PATCH 01/14] Read sitemap files so URLs nothing links to are found HTTrack finds URLs only by parsing links, so anything a site publishes solely in its sitemap stayed invisible: robots.txt was already parsed, but its Sitemap: lines were ignored and nothing else in the tree touched sitemaps. Adds opt-in --sitemap (-%m), which probes the start host's robots.txt and falls back to /sitemap.xml, and --sitemap-url (-%mu) for an explicit document. Handles and nested , plain or gzipped. Discovered URLs enter with the full depth budget but still go through the wizard, so filters and scope rules decide; a sitemap is not a filter bypass. The parser reads attacker-controlled XML off the network, so it is capped on URL count, index nesting, decompressed size and decompression ratio, and child sitemaps must stay on the host that named them. Closes #712 Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- html/cmdguide.html | 12 + html/httrack.man.html | 17 +- html/server/option8.html | 11 + html/server/step2.html | 2 + html/server/step4.html | 4 + lang.def | 8 + lang/English.txt | 8 + lang/Francais.txt | 8 + man/httrack.1 | 5 +- src/Makefile.am | 4 +- src/htsalias.c | 4 + src/htscore.c | 44 +++ src/htscoremain.c | 20 ++ src/htshelp.c | 2 + src/htslib.c | 4 + src/htsopt.h | 4 + src/htsselftest.c | 163 +++++++++++ src/htssitemap.c | 544 +++++++++++++++++++++++++++++++++++ src/htssitemap.h | 94 ++++++ src/libhttrack.vcxproj | 1 + tests/01_engine-sitemap.test | 10 + tests/84_local-sitemap.test | 53 ++++ tests/Makefile.am | 4 +- tests/local-server.py | 60 +++- tests/webhttrack-smoke.sh | 6 +- 25 files changed, 1083 insertions(+), 9 deletions(-) create mode 100644 src/htssitemap.c create mode 100644 src/htssitemap.h create mode 100644 tests/01_engine-sitemap.test create mode 100644 tests/84_local-sitemap.test diff --git a/html/cmdguide.html b/html/cmdguide.html index 0fe2b02d2..aab16faa2 100644 --- a/html/cmdguide.html +++ b/html/cmdguide.html @@ -163,8 +163,20 @@

2. Scope: how far the crawl reaches

--near (-n)Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere. --ext-depth (-%e)How many levels of external links to follow once the crawl leaves your scope (default 0). --test (-t)Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding. + --sitemap (-%m), --sitemap-url URL (-%mu)Also take start URLs from the site's sitemap, for pages nothing links to. Off by default. +

Link-following only finds what something links to. Anything a site publishes +solely in its sitemap is invisible to HTTrack unless you ask for it. +--sitemap reads the start host's robots.txt for +Sitemap: lines and falls back to /sitemap.xml; +--sitemap-url names one directly. Nested sitemapindex files +and gzipped .xml.gz sitemaps are followed. The URLs found become start +URLs with the full depth budget, but they still go through your filters and +scope rules, so a sitemap cannot widen a crawl you deliberately narrowed. It is +off by default because a sitemap can list thousands of pages nothing links +to.

+

The single most common surprise is "only the home page came down." That is usually not a scope option at all: it is an off-host redirect. A start URL of http://example.com/ that redirects to https://www.example.com/ diff --git a/html/httrack.man.html b/html/httrack.man.html index 4e82953de..c1525ec88 100644 --- a/html/httrack.man.html +++ b/html/httrack.man.html @@ -87,8 +87,8 @@

SYNOPSIS --host-control[=N] ] [ -%P, --extended-parsing[=N] ] [ -n, --near ] [ -t, --test ] [ -%L, --list ] [ -%S, --urllist -] [ -NN, --structure[=N] ] [ -%N, ---delayed-type-check ] [ -%D, +] [ -%m, --sitemap ] [ -NN, --structure[=N] ] +[ -%N, --delayed-type-check ] [ -%D, --cached-delayed-type-check ] [ -%M, --mime-html ] [ -LN, --long-names[=N] ] [ -KN, --keep-links[=N] ] [ -x, --replace-external ] [ @@ -575,6 +575,19 @@

Links options:

<file> add all scan rules located in this text file (one scan rule per line) (--urllist <param>)

+ + + + + +

-%m

+ + + + +

seed the crawl from the site’s sitemap (robots.txt +Sitemap:, then /sitemap.xml); --sitemap-url URL names one +explicitly (--sitemap)

Build options: diff --git a/html/server/option8.html b/html/server/option8.html index 3723ae8af..83ca5ff82 100644 --- a/html/server/option8.html +++ b/html/server/option8.html @@ -132,6 +132,17 @@

${LANG_O2} - ${LANG_IOPT8}



+ ${LANG_SITEMAP} +

+ +${LANG_SITEMAPURL} + +

+ ${LANG_I62b} diff --git a/html/server/step2.html b/html/server/step2.html index a01a1876e..d58af9eb7 100644 --- a/html/server/step2.html +++ b/html/server/step2.html @@ -141,6 +141,8 @@

${fexist:index.html:LANG_G42}

${do:copy:KeepQueryOrder:keepqueryorder} ${do:copy:StripQuery:stripquery} ${do:copy:StoreAllInCache:cache2} +${do:copy:Sitemap:sitemap} +${do:copy:SitemapUrl:sitemapurl} ${do:copy:Warc:warc} ${do:copy:WarcFile:warcfile} ${do:copy:LogType:logtype} diff --git a/html/server/step4.html b/html/server/step4.html index e2d8da6c1..9c96b82d7 100644 --- a/html/server/step4.html +++ b/html/server/step4.html @@ -188,6 +188,8 @@

${LANG_J9}

${test:toler:--tolerant} ${test:http10:--http-10} ${test:cache2:--store-all-in-cache} + ${test:sitemap:--sitemap} + ${test:sitemapurl:--sitemap-url "}${html:sitemapurl}${test:sitemapurl:"} ${test:warc:--warc} ${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"} ${test:norecatch:--do-not-recatch} @@ -240,6 +242,8 @@

${LANG_J9}

KeepQueryOrder=${ztest:keepqueryorder:0:1} StripQuery=${stripquery} StoreAllInCache=${ztest:cache2:0:1} +Sitemap=${ztest:sitemap:0:1} +SitemapUrl=${sitemapurl} Warc=${ztest:warc:0:1} WarcFile=${warcfile} LogType=${logtype} diff --git a/lang.def b/lang.def index f11c1e279..f6399032c 100755 --- a/lang.def +++ b/lang.def @@ -1042,3 +1042,11 @@ LANG_WARCFILE WARC archive name: LANG_WARCFILETIP Optional base name for the WARC archive; leave blank to auto-name it under the output directory. +LANG_SITEMAP +Seed the crawl from the site's sitemap +LANG_SITEMAPTIP +Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL. +LANG_SITEMAPURL +Sitemap address: +LANG_SITEMAPURLTIP +Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml. diff --git a/lang/English.txt b/lang/English.txt index 86c402be4..be31ef6b0 100644 --- a/lang/English.txt +++ b/lang/English.txt @@ -1012,3 +1012,11 @@ WARC archive name: WARC archive name: Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Optional base name for the WARC archive; leave blank to auto-name it under the output directory. +Seed the crawl from the site's sitemap +Seed the crawl from the site's sitemap +Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL. +Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL. +Sitemap address: +Sitemap address: +Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml. +Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml. diff --git a/lang/Francais.txt b/lang/Francais.txt index 74131f840..b20172465 100644 --- a/lang/Francais.txt +++ b/lang/Francais.txt @@ -1012,3 +1012,11 @@ WARC archive name: Nom de l'archive WARC : Optional base name for the WARC archive; leave blank to auto-name it under the output directory. Nom de base optionnel pour l'archive WARC ; laissez vide pour le gnrer automatiquement dans le rpertoire de sortie. +Seed the crawl from the site's sitemap +Partir du plan de site (sitemap) +Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL. +Lire le plan de site (lignes Sitemap: de robots.txt, puis /sitemap.xml) et ajouter chaque URL liste comme adresse de dpart. +Sitemap address: +Adresse du plan de site : +Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml. +Adresse d'un plan de site lire au lieu de sonder le site ; laissez vide pour sonder robots.txt puis /sitemap.xml. diff --git a/man/httrack.1 b/man/httrack.1 index 86cfc84ac..3ea890a52 100644 --- a/man/httrack.1 +++ b/man/httrack.1 @@ -3,7 +3,7 @@ .\" .\" This file is generated by man/makeman.sh; do not edit by hand. .\" SPDX-License-Identifier: GPL-3.0-or-later -.TH httrack 1 "23 July 2026" "httrack website copier" +.TH httrack 1 "26 July 2026" "httrack website copier" .SH NAME httrack \- offline browser : copy websites to a local directory .SH SYNOPSIS @@ -36,6 +36,7 @@ httrack \- offline browser : copy websites to a local directory [ \fB\-t, \-\-test\fR ] [ \fB\-%L, \-\-list\fR ] [ \fB\-%S, \-\-urllist\fR ] +[ \fB\-%m, \-\-sitemap\fR ] [ \fB\-NN, \-\-structure[=N]\fR ] [ \fB\-%N, \-\-delayed\-type\-check\fR ] [ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ] @@ -187,6 +188,8 @@ test all URLs (even forbidden ones) (\-\-test) add all URL located in this text file (one URL per line) (\-\-list ) .IP \-%S add all scan rules located in this text file (one scan rule per line) (\-\-urllist ) +.IP \-%m +seed the crawl from the site's sitemap (robots.txt Sitemap:, then /sitemap.xml); \-\-sitemap\-url URL names one explicitly (\-\-sitemap) .SS Build options: .IP \-NN structure type (0 *original structure, 1+: see below) (\-\-structure[=N]) diff --git a/src/Makefile.am b/src/Makefile.am index da3f1e4ba..aedda1af6 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -66,7 +66,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \ htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \ htsname.c htsrobots.c htstools.c htswizard.c \ htsalias.c htsthread.c htsindex.c htsbauth.c \ - htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \ + htsmd5.c htscodec.c htswarc.c htssitemap.c htsproxy.c htszlib.c htswrap.c htsconcat.c \ htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \ md5.c \ minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \ @@ -77,7 +77,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \ htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \ htsmodules.h htsname.h htsnet.h htssniff.h \ htsopt.h htsrobots.h htsthread.h \ - htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h \ + htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htssitemap.h htsproxy.h htszlib.h \ htsstrings.h htsarrays.h httrack-library.h \ htscharset.h punycode.h htsencoding.h \ htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \ diff --git a/src/htsalias.c b/src/htsalias.c index 8d127a745..858b776ca 100644 --- a/src/htsalias.c +++ b/src/htsalias.c @@ -114,6 +114,10 @@ const char *hts_optalias[][4] = { "strip [host/pattern=]key1,key2,... from URLs"}, {"cookies-file", "-%K", "param1", "load extra cookies from a Netscape cookies.txt"}, + {"sitemap", "-%m", "single", + "seed the crawl from the start host's sitemap (robots.txt, then " + "/sitemap.xml)"}, + {"sitemap-url", "-%mu", "param1", "seed the crawl from this sitemap URL"}, {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"}, {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"}, {"warc-max-size", "-%rs", "param1", diff --git a/src/htscore.c b/src/htscore.c index dab4e4bf4..9246a49a4 100644 --- a/src/htscore.c +++ b/src/htscore.c @@ -39,6 +39,7 @@ Please visit our Website: http://www.httrack.com /* File defs */ #include "htscore.h" +#include "htssitemap.h" #include "htswarc.h" /* specific definitions */ @@ -943,6 +944,21 @@ int httpmirror(char *url1, httrackp * opt) { heap_top()->premier = heap_top_index(); // premier lien, objet-père=objet heap_top()->precedent = heap_top_index(); // lien précédent + /* --sitemap: queue the sitemap probe just after the seeds, so its URLs are + injected before the crawl gets far. */ + if (opt->sitemap || StringNotEmpty(opt->sitemap_url)) { + char BIGSTK first[HTS_URLMAXSIZE * 2]; + const char *const eol = strchr(primary, '\n'); + const size_t len = eol != NULL ? (size_t) (eol - primary) : 0; + + first[0] = '\0'; + if (len > 0 && len < sizeof(first)) { + memcpy(first, primary, len); + first[len] = '\0'; + } + hts_sitemap_seed(opt, first); + } + // Initialiser cache { opt->state._hts_in_html_parsing = 4; @@ -1609,6 +1625,28 @@ int httpmirror(char *url1, httrackp * opt) { /* Load file and decode if necessary, after redirect check. */ LOAD_IN_MEMORY_IF_NECESSARY(); + /* Sitemap document: turn its URLs into top-level seeds. They go + through htsAddLink, so the wizard's filters and scope rules decide, and + this link's max depth leaves them the full budget. */ + if (opt->sitemap_state != NULL && + hts_sitemap_pending(opt, urladr(), urlfil())) { + htsmoduleStruct BIGSTK smstr; + + memset(&smstr, 0, sizeof(smstr)); + smstr.opt = opt; + smstr.sback = sback; + smstr.cache = &cache; + smstr.hashptr = hashptr; + smstr.numero_passe = numero_passe; + smstr.ptr_ = &ptr; + smstr.addLink = htsAddLink; + smstr.url_host = urladr(); + smstr.url_file = urlfil(); + smstr.mime = r.contenttype; + hts_sitemap_ingest(opt, &smstr, urladr(), urlfil(), r.adr, + r.adr != NULL && r.size > 0 ? (size_t) r.size : 0); + } + // ------------------------------------------------------ // ok, fichier chargé localement // ------------------------------------------------------ @@ -2256,6 +2294,7 @@ int httpmirror(char *url1, httrackp * opt) { // ending usercommand(opt, 0, NULL, NULL, NULL, NULL); warc_close_opt(opt); + hts_sitemap_free(opt); // désallocation mémoire & buffers XH_uninit; @@ -3643,6 +3682,11 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) { to->warc_cdx = from->warc_cdx; to->warc_wacz = from->warc_wacz; + if (from->sitemap) + to->sitemap = from->sitemap; + if (StringNotEmpty(from->sitemap_url)) + StringCopyS(to->sitemap_url, from->sitemap_url); + if (from->pause_max_ms > 0) { to->pause_min_ms = from->pause_min_ms; to->pause_max_ms = from->pause_max_ms; diff --git a/src/htscoremain.c b/src/htscoremain.c index 400cda400..d97fa2ab2 100644 --- a/src/htscoremain.c +++ b/src/htscoremain.c @@ -1795,6 +1795,26 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) { StringCopy(opt->warc_file, WARC_AUTONAME); } break; + case 'm': // sitemap / sitemap-url: seed the crawl from sitemaps + if (*(com + 1) == 'u') { // --sitemap-url URL: explicit sitemap + com++; + if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) { + HTS_PANIC_PRINTF( + "Option sitemap-url needs a blank space and a URL"); + htsmain_free(); + return -1; + } + na++; + if (strlen(argv[na]) >= HTS_URLMAXSIZE) { + HTS_PANIC_PRINTF("Sitemap URL too long"); + htsmain_free(); + return -1; + } + StringCopy(opt->sitemap_url, argv[na]); + } else { // --sitemap: robots.txt probe, then /sitemap.xml + opt->sitemap = HTS_TRUE; + } + break; case 'Y': // why: explain the filter verdict for a URL, no crawl if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) { HTS_PANIC_PRINTF("Option why needs a blank space and a URL"); diff --git a/src/htshelp.c b/src/htshelp.c index 00569bf11..a61a51776 100644 --- a/src/htshelp.c +++ b/src/htshelp.c @@ -525,6 +525,8 @@ void help(const char *app, int more) { (" %L add all URL located in this text file (one URL per line)"); infomsg (" %S add all scan rules located in this text file (one scan rule per line)"); + infomsg(" %m seed the crawl from the site's sitemap (robots.txt Sitemap:, " + "then /sitemap.xml); --sitemap-url URL names one explicitly"); infomsg(""); infomsg("Build options:"); infomsg(" NN structure type (0 *original structure, 1+: see below)"); diff --git a/src/htslib.c b/src/htslib.c index 55b524d40..86fd6e726 100644 --- a/src/htslib.c +++ b/src/htslib.c @@ -36,6 +36,7 @@ Please visit our Website: http://www.httrack.com // Fichier librairie .c #include "htscore.h" +#include "htssitemap.h" #include "htswarc.h" /* specific definitions */ @@ -6010,6 +6011,7 @@ HTSEXT_API httrackp *hts_create_opt(void) { StringCopy(opt->strip_query, ""); StringCopy(opt->cookies_file, ""); StringCopy(opt->warc_file, ""); + StringCopy(opt->sitemap_url, ""); opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */ StringCopy(opt->why_url, ""); opt->pause_min_ms = 0; @@ -6163,6 +6165,8 @@ HTSEXT_API void hts_free_opt(httrackp * opt) { StringFree(opt->cookies_file); StringFree(opt->why_url); StringFree(opt->warc_file); + StringFree(opt->sitemap_url); + hts_sitemap_free(opt); /* backstop: httpmirror's early-return paths */ StringFree(opt->path_html); StringFree(opt->path_html_utf8); diff --git a/src/htsopt.h b/src/htsopt.h index 6ead35372..44fcc2b98 100644 --- a/src/htsopt.h +++ b/src/htsopt.h @@ -547,6 +547,10 @@ struct httrackp { archive. Tail: ABI */ hts_boolean warc_wacz; /**< --wacz: package archive+index+pages as a WACZ zip (implies --warc + --warc-cdx). Tail: ABI */ + hts_boolean sitemap; /**< --sitemap: probe the start host's robots.txt for + Sitemap: lines, else /sitemap.xml. Tail: ABI */ + String sitemap_url; /**< --sitemap-url: sitemap to ingest. Tail: ABI */ + void *sitemap_state; /**< live sitemap ingestion state, or NULL. Tail: ABI */ }; /* Running statistics for a mirror. */ diff --git a/src/htsselftest.c b/src/htsselftest.c index 9e61fb71a..fd5fee999 100644 --- a/src/htsselftest.c +++ b/src/htsselftest.c @@ -58,6 +58,7 @@ Please visit our Website: http://www.httrack.com #include "htssniff.h" #include "htscodec.h" #include "htsproxy.h" +#include "htssitemap.h" #include "htswarc.h" #if HTS_USEZLIB #include "htszlib.h" @@ -3574,6 +3575,166 @@ static int st_robots(httrackp *opt, int argc, char **argv) { return 0; } +/* Collect the URLs a sitemap scan hands out. */ +typedef struct sm_collect { + int n; + char url[8][HTS_URLMAXSIZE]; +} sm_collect; + +static hts_boolean sm_take(void *arg, const char *url) { + sm_collect *const c = (sm_collect *) arg; + + if (c->n < (int) (sizeof(c->url) / sizeof(c->url[0]))) + strcpybuff(c->url[c->n], url); + c->n++; + return HTS_TRUE; +} + +/* Scan `doc` off a heap buffer with no NUL terminator, so a read past the + declared size is an ASan error rather than a silent pass. */ +static int sm_scan(const char *doc, int maxurls, hts_boolean *is_index, + sm_collect *out) { + const size_t len = strlen(doc); + char *raw = malloct(len); + int n; + + memset(out, 0, sizeof(*out)); + assertf(raw != NULL); + memcpy(raw, doc, len); + n = hts_sitemap_scan(raw, len, maxurls, is_index, sm_take, out); + freet(raw); + return n; +} + +static int st_sitemap(httrackp *opt, int argc, char **argv) { + sm_collect c; + hts_boolean idx; + (void) opt; + (void) argc; + (void) argv; + + /* A urlset yields its URLs, in order, unescaped. */ + assertf(sm_scan("" + "http://h.test/a.html" + " https://h.test/b?x=1&y=2\n " + "", + 100, &idx, &c) == 2); + assertf(!idx); + assertf(strcmp(c.url[0], "http://h.test/a.html") == 0); + assertf(strcmp(c.url[1], "https://h.test/b?x=1&y=2") == 0); + + /* A sitemapindex is flagged: its URLs are child sitemaps, not pages. */ + assertf(sm_scan("http://h.test/s2.xml.gz" + "", + 100, &idx, &c) == 1); + assertf(idx); + + /* Root element decides even when the other name appears later as text. */ + assertf(sm_scan("http://h.test/a" + "", + 100, &idx, &c) == 1); + assertf(!idx); + + /* Numeric character references, decimal and hex, decode to ASCII. */ + assertf(sm_scan("http://h.test/a?b=c", + 100, &idx, &c) == 1); + assertf(strcmp(c.url[0], "http://h.test/a?b=c") == 0); + + /* A reference outside printable ASCII stays verbatim, not a control byte. */ + assertf(sm_scan("http://h.test/a b", 100, + &idx, &c) == 1); + assertf(strcmp(c.url[0], "http://h.test/a b") == 0); + + /* is not . */ + assertf(sm_scan("http://h.test/a", 100, + &idx, &c) == 0); + + /* Rejected: relative, non-http scheme, embedded space, empty. */ + assertf(sm_scan("/a.htmlftp://h.test/a" + "javascript:alert(1)" + "http://h.test/a b", + 100, &idx, &c) == 0); + + /* The URL length bound: one under fits, exactly at it is dropped rather than + truncated into a different URL. */ + { + char BIGSTK doc[HTS_URLMAXSIZE * 2]; + char BIGSTK url[HTS_URLMAXSIZE + 1]; + size_t i; + + strcpybuff(url, "http://h.test/"); + for (i = strlen(url); i < HTS_URLMAXSIZE - 1; i++) + url[i] = 'a'; + url[i] = '\0'; + snprintf(doc, sizeof(doc), "%s", url); + assertf(sm_scan(doc, 100, &idx, &c) == 1); + + url[i] = 'a'; + url[i + 1] = '\0'; + snprintf(doc, sizeof(doc), "%s", url); + assertf(sm_scan(doc, 100, &idx, &c) == 0); + } + + /* The URL cap stops the scan. */ + assertf(sm_scan("http://h.test/1http://h.test/2" + "http://h.test/3", + 2, &idx, &c) == 2); + + /* An unterminated at end of buffer must not read past it. */ + assertf(sm_scan("http://h.test/a", 100, &idx, &c) == 0); + assertf(sm_scan("http://h.test/gz.html"; + uLongf zlen = compressBound((uLong) strlen(xml)) + 32; + char *z = malloct((size_t) zlen); + z_stream zs; + + assertf(z != NULL); + memset(&zs, 0, sizeof(zs)); + assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8, + Z_DEFAULT_STRATEGY) == Z_OK); + zs.next_in = (const Bytef *) xml; + zs.avail_in = (uInt) strlen(xml); + zs.next_out = (Bytef *) z; + zs.avail_out = (uInt) zlen; + assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END); + zlen = (uLongf) zs.total_out; + deflateEnd(&zs); + + memset(&c, 0, sizeof(c)); + assertf(hts_sitemap_scan(z, (size_t) zlen, 100, &idx, sm_take, &c) == 1); + assertf(strcmp(c.url[0], "http://h.test/gz.html") == 0); + + /* Truncated gzip: refused, not scanned as plain text. */ + memset(&c, 0, sizeof(c)); + assertf(hts_sitemap_scan(z, 4, 100, &idx, sm_take, &c) == -1); + freet(z); + } +#endif + + /* robots.txt: only Sitemap: records, comments stripped, case-insensitive, + and group-independent (no User-agent line needed). */ + memset(&c, 0, sizeof(c)); + { + const char *const txt = "User-agent: *\nDisallow: /x\n" + "SITEMAP: http://h.test/s1.xml # first\n" + "Sitemap: /relative.xml\n" + "Sitemapper: http://h.test/no.xml\n" + "Sitemap:\thttps://h.test/s2.xml\n"; + + assertf(hts_sitemap_scan_robots(txt, strlen(txt), 100, sm_take, &c) == 2); + assertf(strcmp(c.url[0], "http://h.test/s1.xml") == 0); + assertf(strcmp(c.url[1], "https://h.test/s2.xml") == 0); + } + + printf("sitemap self-test OK\n"); + return 0; +} + /* Connected stream pair over loopback; Windows has no socketpair(). */ static int st_socketpair(T_SOC sv[2]) { struct sockaddr_in sa; @@ -5182,6 +5343,8 @@ static const struct selftest_entry { st_contentcodings}, {"robots", "", "robots.txt RFC 9309 Allow/Disallow precedence self-test", st_robots}, + {"sitemap", "", + "sitemap extraction, caps and robots.txt Sitemap:", st_sitemap}, {"ftp-line", "", "get_ftp_line bounds a hostile FTP reply line", st_ftpline}, {"ftp-userpass", "", "ftp_split_userpass bounds URL userinfo", st_ftpuser}, diff --git a/src/htssitemap.c b/src/htssitemap.c new file mode 100644 index 000000000..09918e707 --- /dev/null +++ b/src/htssitemap.c @@ -0,0 +1,544 @@ +/* ------------------------------------------------------------ */ +/* +HTTrack Website Copier, Offline Browser for Windows and Unix +Copyright (C) 1998 Xavier Roche and other contributors + +SPDX-License-Identifier: GPL-3.0-or-later + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . + +Ethical use: we kindly ask that you NOT use this software to harvest email +addresses or to collect any other private information about people. Doing so +would dishonor our work and waste the many hours we have spent on it. + +Please visit our Website: http://www.httrack.com +*/ + +/* ------------------------------------------------------------ */ +/* File: sitemap ingestion (sitemaps.org 0.9) */ +/* Author: Xavier Roche */ +/* ------------------------------------------------------------ */ + +#define HTS_INTERNAL_BYTECODE + +#include "htscore.h" +#include "htssitemap.h" + +#include "htsbase.h" +#include "htscodec.h" +#include "htslib.h" +#include "htsrobots.h" +#include "htssafe.h" +#include "htstools.h" +#include "htszlib.h" + +#include +#include + +/* One queued sitemap document. `is_robots` marks the robots.txt probe, whose + Sitemap: lines feed this same list. */ +typedef struct sitemap_doc { + char adr[HTS_URLMAXSIZE]; + char fil[HTS_URLMAXSIZE]; + int level; + hts_boolean is_robots; + hts_boolean done; + struct sitemap_doc *next; +} sitemap_doc; + +struct hts_sitemap_state { + sitemap_doc *docs; + int ndocs; /* documents queued, capped by HTS_SITEMAP_MAX_DOCS */ + int nurls; /* URLs seeded, capped by HTS_SITEMAP_MAX_URLS_TOTAL */ + hts_boolean fallback_done; /* the /sitemap.xml fallback was already queued */ +}; +typedef struct hts_sitemap_state hts_sitemap_state; + +/* --------------------------------------------------------------------- */ +/* Document parsing (no engine state: fuzzable and self-testable) */ +/* --------------------------------------------------------------------- */ + +/* Decode the five XML predefined entities plus ASCII-printable numeric refs + in-place; anything else is left verbatim. Never grows the string. */ +static void sitemap_unescape(char *s) { + char *r = s, *w = s; + + while (*r != '\0') { + if (*r != '&') { + *w++ = *r++; + continue; + } + { + char *const semi = strchr(r + 1, ';'); + const size_t len = semi != NULL ? (size_t) (semi - (r + 1)) : 0; + int c = -1; + + if (len == 0 || len > 8) { + *w++ = *r++; + continue; + } + if (len == 3 && strncmp(r + 1, "amp", 3) == 0) + c = '&'; + else if (len == 2 && strncmp(r + 1, "lt", 2) == 0) + c = '<'; + else if (len == 2 && strncmp(r + 1, "gt", 2) == 0) + c = '>'; + else if (len == 4 && strncmp(r + 1, "quot", 4) == 0) + c = '"'; + else if (len == 4 && strncmp(r + 1, "apos", 4) == 0) + c = '\''; + else if (r[1] == '#') { + const int hex = (r[2] == 'x' || r[2] == 'X'); + const char *p = r + (hex ? 3 : 2); + long v = 0; + + if (p < semi) { + for (; p < semi; p++) { + const int d = + hex ? (isxdigit((unsigned char) *p) + ? (isdigit((unsigned char) *p) + ? *p - '0' + : (tolower((unsigned char) *p) - 'a' + 10)) + : -1) + : (isdigit((unsigned char) *p) ? *p - '0' : -1); + + if (d < 0) { + v = -1; + break; + } + v = v * (hex ? 16 : 10) + d; + if (v > 0x7e) + break; + } + /* Only ASCII printables: a URL has no business carrying anything + else, and a wider decode would let a reference smuggle in a + control character. */ + if (v >= 0x20 && v <= 0x7e) + c = (int) v; + } + } + if (c < 0) { + *w++ = *r++; + } else { + *w++ = (char) c; + r = semi + 1; + } + } + } + *w = '\0'; +} + +/* Accept only an absolute http(s) URL with no space or control byte. */ +static hts_boolean sitemap_url_ok(const char *url) { + const char *p; + + if (!strfield(url, "http://") && !strfield(url, "https://")) + return HTS_FALSE; + for (p = url; *p != '\0'; p++) { + if ((unsigned char) *p <= ' ' || (unsigned char) *p == 0x7f) + return HTS_FALSE; + } + return HTS_TRUE; +} + +/* Skip to the character after the next '>' at or after p, or NULL. */ +static const char *sitemap_tag_end(const char *p, const char *end) { + while (p < end && *p != '>') + p++; + return p < end ? p + 1 : NULL; +} + +/* Bounded substring search: the document may hold NUL bytes. */ +static const char *sitemap_memstr(const char *p, size_t len, + const char *needle) { + const size_t nlen = strlen(needle); + + if (nlen == 0 || len < nlen) + return NULL; + for (; len >= nlen; p++, len--) { + if (*p == *needle && memcmp(p, needle, nlen) == 0) + return p; + } + return NULL; +} + +/* Decompress a gzip-framed body into a fresh buffer bounded by both the + absolute cap and the codec ratio budget. Returns NULL on failure. */ +static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) { + const LLint budget = hts_codec_maxout((LLint) size); + size_t cap = budget < (LLint) HTS_SITEMAP_MAX_BYTES + ? (size_t) budget + : (size_t) HTS_SITEMAP_MAX_BYTES; + char *out; + size_t n; + + if (cap == 0) + return NULL; + out = malloct(cap + 1); + if (out == NULL) + return NULL; + n = hts_zhead(body, size, out, cap); + if (n == 0) { + freet(out); + return NULL; + } + out[n] = '\0'; + *outsize = n; + return out; +} + +int hts_sitemap_scan(const char *body, size_t size, int maxurls, + hts_boolean *is_index, hts_sitemap_handler handler, + void *arg) { + char *unpacked = NULL; + const char *doc; + const char *end; + const char *p; + int n = 0; + + if (is_index != NULL) + *is_index = HTS_FALSE; + if (body == NULL || size < 2 || handler == NULL) + return 0; + + /* A .xml.gz body arrives raw here: Content-Encoding gzip was already undone + upstream, so only the gzip container is left to peel. */ + if ((unsigned char) body[0] == 0x1f && (unsigned char) body[1] == 0x8b) { + unpacked = sitemap_gunzip(body, size, &size); + if (unpacked == NULL) + return -1; + doc = unpacked; + } else { + if (size > (size_t) HTS_SITEMAP_MAX_BYTES) + size = (size_t) HTS_SITEMAP_MAX_BYTES; + doc = body; + } + end = doc + size; + + /* Whichever root element comes first classifies the document; the handler + reads the verdict before the first URL, so it must be set up front. */ + if (is_index != NULL) { + const char *const idx = sitemap_memstr(doc, size, "" or "", never "" */ + if (loc + 4 >= end || (loc[4] != '>' && !isspace((unsigned char) loc[4]))) { + p = loc + 4; + continue; + } + val = sitemap_tag_end(loc + 4, end); + if (val == NULL) + break; + for (stop = val; stop < end && *stop != '<'; stop++) + ; + /* No closing tag: the document is truncated (cut short, or clipped by the + decompression cap), so the last value may be a partial URL. Drop it. */ + if (stop == end) + break; + p = stop; + while (val < stop && isspace((unsigned char) *val)) + val++; + while (stop > val && isspace((unsigned char) *(stop - 1))) + stop--; + len = (size_t) (stop - val); + /* Overflow-safe: the untrusted length alone against the room left. */ + if (len == 0 || len >= sizeof(url)) + continue; + memcpy(url, val, len); + url[len] = '\0'; + sitemap_unescape(url); + if (!sitemap_url_ok(url)) + continue; + n++; + if (!handler(arg, url)) + break; + } + + if (unpacked != NULL) + freet(unpacked); + return n; +} + +int hts_sitemap_scan_robots(const char *body, size_t size, int maxurls, + hts_sitemap_handler handler, void *arg) { + size_t bptr = 0; + int n = 0; + + if (body == NULL || handler == NULL) + return 0; + while (bptr < size && n < maxurls) { + char BIGSTK line[HTS_URLMAXSIZE]; + char *comm; + char *a; + + bptr += binput(body + bptr, line, sizeof(line) - 2); + comm = strchr(line, '#'); + if (comm != NULL) + *comm = '\0'; + if (!strfield(line, "sitemap:")) + continue; + a = line + 8; + while (is_realspace(*a)) + a++; + { + size_t l = strlen(a); + + while (l > 0 && is_realspace(a[l - 1])) + a[--l] = '\0'; + } + if (!sitemap_url_ok(a)) + continue; + n++; + if (!handler(arg, a)) + break; + } + return n; +} + +/* --------------------------------------------------------------------- */ +/* Engine glue */ +/* --------------------------------------------------------------------- */ + +static hts_sitemap_state *sitemap_state(httrackp *opt) { + if (opt->sitemap_state == NULL) + opt->sitemap_state = calloct(1, sizeof(hts_sitemap_state)); + return (hts_sitemap_state *) opt->sitemap_state; +} + +static sitemap_doc *sitemap_find(httrackp *opt, const char *adr, + const char *fil) { + hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state; + sitemap_doc *d; + + if (st == NULL) + return NULL; + for (d = st->docs; d != NULL; d = d->next) { + if (strfield2(d->adr, adr) && strcmp(d->fil, fil) == 0) + return d; + } + return NULL; +} + +/* Queue a document and record its link with save="" so the body stays in + memory: a sitemap is ingested, never mirrored. Top priority so its URLs get + the full depth budget through htsAddLink. */ +static hts_boolean sitemap_queue(httrackp *opt, const char *adr, + const char *fil, int level, + hts_boolean is_robots) { + hts_sitemap_state *const st = sitemap_state(opt); + sitemap_doc *d; + + if (st == NULL) + return HTS_FALSE; + if (st->ndocs >= HTS_SITEMAP_MAX_DOCS || level > HTS_SITEMAP_MAX_LEVEL) { + hts_log_print(opt, LOG_WARNING, "Sitemap: cap reached, skipping %s%s", adr, + fil); + return HTS_FALSE; + } + if (strlen(adr) >= sizeof(d->adr) || strlen(fil) >= sizeof(d->fil)) + return HTS_FALSE; + if (sitemap_find(opt, adr, fil) != NULL) + return HTS_FALSE; + d = calloct(1, sizeof(sitemap_doc)); + if (d == NULL) + return HTS_FALSE; + strcpybuff(d->adr, adr); + strcpybuff(d->fil, fil); + d->level = level; + d->is_robots = is_robots; + d->next = st->docs; + st->docs = d; + st->ndocs++; + + if (!hts_record_link(opt, adr, fil, "", "", "", NULL)) + return HTS_FALSE; + heap_top()->testmode = 0; + heap_top()->link_import = 0; + heap_top()->depth = opt->depth + 1; + heap_top()->pass2 = 0; + heap_top()->retry = opt->retry; + heap_top()->premier = heap_top_index(); + heap_top()->precedent = heap_top_index(); + hts_log_print(opt, LOG_INFO, "Sitemap: queued %s%s", adr, fil); + return HTS_TRUE; +} + +void hts_sitemap_seed(httrackp *opt, const char *starturl) { + char BIGSTK url[HTS_URLMAXSIZE * 2]; + lien_adrfil af; + + if (StringNotEmpty(opt->sitemap_url)) { + if (strlen(StringBuff(opt->sitemap_url)) >= sizeof(url)) { + hts_log_print(opt, LOG_ERROR, "Sitemap URL too long"); + } else { + strcpybuff(url, StringBuff(opt->sitemap_url)); + if (strstr(url, ":/") == NULL) + hts_log_print(opt, LOG_ERROR, "Sitemap URL must be absolute: %s", url); + else if (ident_url_absolute(url, &af) >= 0) + (void) sitemap_queue(opt, af.adr, af.fil, 0, HTS_FALSE); + } + } + /* --sitemap: probe the start host's robots.txt, whose Sitemap: lines decide + whether the /sitemap.xml fallback is needed. */ + if (!opt->sitemap || starturl == NULL || starturl[0] == '\0' || + strlen(starturl) >= sizeof(url)) + return; + strcpybuff(url, starturl); + if (ident_url_absolute(url, &af) < 0) + return; + if (sitemap_queue(opt, af.adr, "/robots.txt", 0, HTS_TRUE)) { + /* Claim the host so the parser does not queue robots.txt a second time. */ + if (opt->robotsptr != NULL) + (void) checkrobots_set((robots_wizard *) opt->robotsptr, af.adr, ""); + } +} + +hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr, + const char *fil) { + const sitemap_doc *const d = sitemap_find(opt, adr, fil); + + return d != NULL && !d->done ? HTS_TRUE : HTS_FALSE; +} + +/* Handler context: seeding URLs from one document. */ +typedef struct sitemap_ingest_ctx { + httrackp *opt; + htsmoduleStruct *str; + const char *adr; /* host of the document being ingested */ + int level; + hts_boolean is_index; +} sitemap_ingest_ctx; + +/* A of a : hand it to the wizard as a top-level seed. */ +static hts_boolean sitemap_seed_url(void *arg, const char *url) { + sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg; + hts_sitemap_state *const st = sitemap_state(c->opt); + char BIGSTK buff[HTS_URLMAXSIZE]; + + if (st == NULL || st->nurls >= HTS_SITEMAP_MAX_URLS_TOTAL) { + hts_log_print(c->opt, LOG_WARNING, + "Sitemap: URL cap reached, ignoring the rest"); + return HTS_FALSE; + } + /* Both scanners bound the URL below this, but strcpybuff aborts rather than + truncating, so never let hostile input reach it unchecked. */ + if (strlen(url) >= sizeof(buff)) + return HTS_TRUE; + st->nurls++; + strcpybuff(buff, url); + (void) htsAddLink(c->str, buff); + return HTS_TRUE; +} + +/* A of a , or a robots.txt Sitemap: line. Cross-host + children are dropped: a hostile sitemap must not aim the fetcher elsewhere. + */ +static hts_boolean sitemap_seed_child(void *arg, const char *url) { + sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg; + char BIGSTK buff[HTS_URLMAXSIZE]; + lien_adrfil af; + + if (strlen(url) >= sizeof(buff)) + return HTS_TRUE; + strcpybuff(buff, url); + if (ident_url_absolute(buff, &af) < 0) + return HTS_TRUE; + if (!strfield2(af.adr, c->adr)) { + hts_log_print(c->opt, LOG_WARNING, + "Sitemap: ignoring off-host child sitemap %s%s", af.adr, + af.fil); + return HTS_TRUE; + } + (void) sitemap_queue(c->opt, af.adr, af.fil, c->level + 1, HTS_FALSE); + return HTS_TRUE; +} + +/* hts_sitemap_scan classifies the document before the first callback, so the + urlset/sitemapindex split can be decided here. */ +static hts_boolean sitemap_seed_any(void *arg, const char *url) { + sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg; + + return c->is_index ? sitemap_seed_child(arg, url) + : sitemap_seed_url(arg, url); +} + +void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, + const char *fil, const char *body, size_t size) { + sitemap_doc *const d = sitemap_find(opt, adr, fil); + sitemap_ingest_ctx ctx; + int n; + + if (d == NULL || d->done) + return; + d->done = HTS_TRUE; + ctx.opt = opt; + ctx.str = str; + ctx.adr = adr; + ctx.level = d->level; + ctx.is_index = HTS_FALSE; + + if (d->is_robots) { + hts_sitemap_state *const st = sitemap_state(opt); + + n = body != NULL ? hts_sitemap_scan_robots(body, size, HTS_SITEMAP_MAX_DOCS, + sitemap_seed_child, &ctx) + : 0; + /* Fall back to the well-known location only when robots.txt named none. */ + if (n == 0 && st != NULL && !st->fallback_done) { + st->fallback_done = HTS_TRUE; + (void) sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_FALSE); + } + hts_log_print(opt, LOG_INFO, "Sitemap: %d sitemap(s) declared in %s%s", n, + adr, fil); + return; + } + + n = hts_sitemap_scan(body, size, HTS_SITEMAP_MAX_URLS_DOC, &ctx.is_index, + sitemap_seed_any, &ctx); + if (n < 0) { + hts_log_print(opt, LOG_ERROR, "Sitemap: could not decompress %s%s", adr, + fil); + return; + } + hts_log_print(opt, LOG_NOTICE, "Sitemap: %d URL(s) added from %s%s", n, adr, + fil); +} + +void hts_sitemap_free(httrackp *opt) { + hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state; + + if (st == NULL) + return; + while (st->docs != NULL) { + sitemap_doc *const next = st->docs->next; + + freet(st->docs); + st->docs = next; + } + freet(opt->sitemap_state); + opt->sitemap_state = NULL; +} diff --git a/src/htssitemap.h b/src/htssitemap.h new file mode 100644 index 000000000..bb2ec626a --- /dev/null +++ b/src/htssitemap.h @@ -0,0 +1,94 @@ +/* ------------------------------------------------------------ */ +/* +HTTrack Website Copier, Offline Browser for Windows and Unix +Copyright (C) 1998 Xavier Roche and other contributors + +SPDX-License-Identifier: GPL-3.0-or-later + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . + +Ethical use: we kindly ask that you NOT use this software to harvest email +addresses or to collect any other private information about people. Doing so +would dishonor our work and waste the many hours we have spent on it. + +Please visit our Website: http://www.httrack.com +*/ + +/* ------------------------------------------------------------ */ +/* HTTrack sitemap ingestion (sitemaps.org 0.9). Internal, not installed. + Reads / documents, plain or gzip-framed, and feeds + their URLs to the crawl as top-level seeds. The whole input is + attacker-controlled, so every entry point below is capped. */ +/* ------------------------------------------------------------ */ + +#ifndef HTS_SITEMAP_DEFH +#define HTS_SITEMAP_DEFH + +#include "htsdefines.h" +#include "htsopt.h" + +#ifdef __cplusplus +extern "C" { +#endif + +/* Caps. sitemaps.org allows 50000 URLs and 50 MB uncompressed per document; + the byte cap sits above that so a conformant sitemap always fits. */ +#define HTS_SITEMAP_MAX_URLS_DOC 50000 /* per document */ +#define HTS_SITEMAP_MAX_URLS_TOTAL 200000 /* per mirror */ +#define HTS_SITEMAP_MAX_DOCS 256 /* documents per mirror */ +#define HTS_SITEMAP_MAX_LEVEL 4 /* sitemapindex nesting */ +#define HTS_SITEMAP_MAX_BYTES (64 * 1024 * 1024) /* decompressed document */ + +/* Per-URL handler; returning HTS_FALSE stops the scan. */ +typedef hts_boolean (*hts_sitemap_handler)(void *arg, const char *url); + +/* Scan one sitemap document, plain or gzip-framed, handing every acceptable + absolute http(s) URL to `handler`. Stops after `maxurls` URLs, or when + the handler refuses. `is_index` (optional) reports a , whose + URLs are child sitemaps rather than pages. Returns the number of URLs handed + out, or -1 when the document could not be decompressed within the caps. */ +int hts_sitemap_scan(const char *body, size_t size, int maxurls, + hts_boolean *is_index, hts_sitemap_handler handler, + void *arg); + +/* Same, over a NUL-terminated robots.txt body: hands out the "Sitemap:" URLs + (a group-independent record in RFC 9309, so no user-agent grouping). */ +int hts_sitemap_scan_robots(const char *body, size_t size, int maxurls, + hts_sitemap_handler handler, void *arg); + +/* --- Engine glue (needs a live httrackp). --- */ + +/* Queue the first sitemap document of the mirror: the explicit --sitemap-url, + or the start host's /robots.txt probe for --sitemap. `starturl` is the first + command-line seed. No-op when neither option is set. */ +void hts_sitemap_seed(httrackp *opt, const char *starturl); + +/* HTS_TRUE when (adr,fil) is a queued sitemap document awaiting ingestion. */ +hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr, + const char *fil); + +/* Ingest a fetched sitemap document (or the robots.txt probe): seed its URLs + through the wizard via htsAddLink, and queue nested sitemaps. `str` supplies + the parser context of the document being processed. */ +void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, + const char *fil, const char *body, size_t size); + +/* Release the ingestion state held in opt (NULL-safe, idempotent). */ +void hts_sitemap_free(httrackp *opt); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/src/libhttrack.vcxproj b/src/libhttrack.vcxproj index 86256d81e..15d9c02df 100644 --- a/src/libhttrack.vcxproj +++ b/src/libhttrack.vcxproj @@ -138,6 +138,7 @@ + diff --git a/tests/01_engine-sitemap.test b/tests/01_engine-sitemap.test new file mode 100644 index 000000000..35322c05f --- /dev/null +++ b/tests/01_engine-sitemap.test @@ -0,0 +1,10 @@ +#!/bin/bash +# +# Sitemap parser self-test: extraction, entity decoding, URL and length +# rejections, the URL cap, gzip framing and robots.txt Sitemap: records. + +set -euo pipefail + +out=$(httrack -O /dev/null '-#test=sitemap') +echo "$out" +test "$out" = "sitemap self-test OK" diff --git a/tests/84_local-sitemap.test b/tests/84_local-sitemap.test new file mode 100644 index 000000000..66ef0866d --- /dev/null +++ b/tests/84_local-sitemap.test @@ -0,0 +1,53 @@ +#!/bin/bash +# +# --sitemap seeds the crawl from robots.txt -> sitemapindex -> gzipped urlset. +# start.html links to nothing, so orphan*.html can only arrive through the +# sitemap; deep1.html proves the seeds keep a full depth budget under -r2, and +# the off-host page and child sitemap must both be refused (--errors 0). + +set -eu + +: "${top_srcdir:=..}" + +crawl() { bash "$top_srcdir/tests/local-crawl.sh" "$@"; } + +# robots.txt Sitemap: -> index -> .xml.gz, and the seeds behave like -r2 seeds. +# --rerun also walks the update path over a mirror holding sitemap seeds. +crawl --errors 0 --rerun \ + --found 'sitemapdir/orphan1.html' \ + --found 'sitemapdir/orphan2.html' \ + --found 'sitemapdir/deep1.html' \ + httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 + +# Negative control: without the option nothing but the start page is reached. +crawl --errors 0 \ + --found 'sitemapdir/start.html' \ + --not-found 'sitemapdir/orphan1.html' \ + --not-found 'sitemapdir/orphan2.html' \ + httrack 'BASEURL/sitemapdir/start.html' -r2 + +# Sitemap URLs are not a filter bypass: a -*orphan2* rule still rejects one. +crawl --errors 0 \ + --found 'sitemapdir/orphan1.html' \ + --not-found 'sitemapdir/orphan2.html' \ + httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 '-*orphan2*' + +# A robots.txt naming no sitemap falls back to the well-known /sitemap.xml. +# The test server drops its Sitemap: record for this User-Agent. +crawl --errors 0 \ + --found 'sitemapdir/orphan1.html' \ + --log-not-found 'sitemapdir/index.xml' \ + httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -F 'nositemap-agent' + +# --sitemap-url names the document directly, skipping the robots.txt probe. +crawl --errors 0 \ + --found 'sitemapdir/orphan1.html' \ + --found 'sitemapdir/orphan2.html' \ + --log-not-found 'sitemapdir/index.xml' \ + httrack 'BASEURL/sitemapdir/start.html' -r2 \ + --sitemap-url 'BASEURL/sitemapdir/pages.xml.gz' + +# An unfetchable sitemap is not fatal: the crawl still completes. +crawl --found 'sitemapdir/start.html' \ + httrack 'BASEURL/sitemapdir/start.html' -r2 \ + --sitemap-url 'BASEURL/sitemapdir/missing.xml' diff --git a/tests/Makefile.am b/tests/Makefile.am index e0730fd1f..b816a2513 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -89,6 +89,7 @@ TESTS = \ 01_engine-warc-surt.test \ 01_engine-xfread.test \ 01_zlib-acceptencoding.test \ + 01_engine-sitemap.test \ 01_zlib-warc.test \ 01_zlib-warc-cdx.test \ 01_zlib-warc-wacz.test \ @@ -179,6 +180,7 @@ TESTS = \ 79_local-proxytrack-webdav-mime.test \ 80_engine-crash-symbolize.test \ 81_webhttrack-maxsize.test \ - 83_webhttrack-argescape.test + 83_webhttrack-argescape.test \ + 84_local-sitemap.test CLEANFILES = check-network_sh.cache diff --git a/tests/local-server.py b/tests/local-server.py index b0c15517a..2f3e1efe1 100755 --- a/tests/local-server.py +++ b/tests/local-server.py @@ -542,8 +542,16 @@ def route_gated_secret(self): return self.fail_cookie(name) self.send_html("\tThis is the secret.") + # A User-Agent carrying NO_SITEMAP_UA gets a robots.txt with no Sitemap: + # record, so a test can drive the /sitemap.xml fallback instead. + NO_SITEMAP_UA = "nositemap" + def route_robots(self): - body = b"User-agent: *\nDisallow:\n" + # The Sitemap: record is group-independent; only --sitemap acts on it. + body = "User-agent: *\nDisallow:\n" + if self.NO_SITEMAP_UA not in (self.headers.get("User-Agent") or ""): + body += f"Sitemap: http://{self.headers.get('Host')}/sitemapdir/index.xml\n" + body = body.encode() self.send_response(200) self.send_header("Content-Type", "text/plain") self.send_header("Content-Length", str(len(body))) @@ -551,6 +559,49 @@ def route_robots(self): if self.command != "HEAD": self.wfile.write(body) + # --- sitemap ingestion (issue #712) ------------------------------------ + # start.html links to nothing, so orphan*.html are reachable only through + # the sitemap. deep1.html proves the seeds keep a full depth budget; the + # off-host page must be dropped by the travel scope, and the off-host + # child sitemap by the ingester's same-host rule. The index is served both + # from /sitemapdir/ (named by robots.txt) and from the well-known + # /sitemap.xml (the fallback). + + def route_sitemap_index(self): + host = self.headers.get("Host") + self.send_raw( + '\n' + '' + f"http://{host}/sitemapdir/pages.xml.gz" + "http://sitemap-offhost.invalid/s.xml" + "\n".encode(), + "application/xml", + ) + + def route_sitemap_pages(self): + host = self.headers.get("Host") + xml = ( + '\n' + '' + f"http://{host}/sitemapdir/orphan1.html" + f"http://{host}/sitemapdir/orphan2.html" + "http://sitemap-offhost.invalid/x.html" + "\n" + ).encode() + self.send_raw(gzip.compress(xml), "application/x-gzip") + + def route_sitemap_start(self): + self.send_html("\tNothing links to the sitemap pages.") + + def route_sitemap_orphan1(self): + self.send_html('\tdeeper') + + def route_sitemap_orphan2(self): + self.send_html("\tSecond orphan.") + + def route_sitemap_deep1(self): + self.send_html("\tOne level below an orphan.") + # --- type/extension matrix (issue #267 family) ------------------------- def send_raw(self, body, content_type, extra_headers=()): @@ -1578,6 +1629,13 @@ def route_maxrecv_404(self): "/gated/index.php": route_gated_index, "/gated/secret.php": route_gated_secret, "/robots.txt": route_robots, + "/sitemapdir/index.xml": route_sitemap_index, + "/sitemap.xml": route_sitemap_index, + "/sitemapdir/pages.xml.gz": route_sitemap_pages, + "/sitemapdir/start.html": route_sitemap_start, + "/sitemapdir/orphan1.html": route_sitemap_orphan1, + "/sitemapdir/orphan2.html": route_sitemap_orphan2, + "/sitemapdir/deep1.html": route_sitemap_deep1, "/warcgz/index.html": route_warcgz_index, "/warcgz/page.html": route_warcgz_page, "/warcgz/data.bin": route_warcgz_data, diff --git a/tests/webhttrack-smoke.sh b/tests/webhttrack-smoke.sh index 379ecf896..8f66ce2cb 100644 --- a/tests/webhttrack-smoke.sh +++ b/tests/webhttrack-smoke.sh @@ -46,12 +46,14 @@ cat >"$stubdir/x-www-browser" <&2 # Also fetch an option page and require a rendered title='' tooltip: proves the # option template expands and the \${html:} filter escapes into the attribute. -# option9 additionally proves the WARC control renders with its expanded label. +# option9/option8 additionally prove the WARC and sitemap controls render. opturl="\${1%/}/server/option2.html" warcurl="\${1%/}/server/option9.html" +smurl="\${1%/}/server/option8.html" if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html && opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='" && - warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC; then + warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC && + sm="\$(curl -fsSL --max-time 20 "\$smurl")" && printf '%s' "\$sm" | grep -qaF 'name="sitemapurl"' && printf '%s' "\$sm" | grep -qaF 'name="sitemap"'; then echo PASS >"$marker" else echo "FAIL: unexpected response from \$1" >"$marker" From bb3d8db10350091d14d2104c453d6ab4145da255 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 17:50:13 +0200 Subject: [PATCH 02/14] sitemap: distinguish the sitemapindex log line from a urlset one Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- src/htssitemap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/htssitemap.c b/src/htssitemap.c index 09918e707..a27aaa382 100644 --- a/src/htssitemap.c +++ b/src/htssitemap.c @@ -524,8 +524,12 @@ void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, fil); return; } - hts_log_print(opt, LOG_NOTICE, "Sitemap: %d URL(s) added from %s%s", n, adr, - fil); + if (ctx.is_index) + hts_log_print(opt, LOG_NOTICE, + "Sitemap: %d child sitemap(s) listed by %s%s", n, adr, fil); + else + hts_log_print(opt, LOG_NOTICE, "Sitemap: %d URL(s) added from %s%s", n, adr, + fil); } void hts_sitemap_free(httrackp *opt) { From 1d96350564e25f249de2f186d5c3be5155558d73 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 17:59:19 +0200 Subject: [PATCH 03/14] sitemap: fix an out-of-bounds read, tighten the parser and the tests lienrelatif() walked back from the last character of its current-path argument without checking the path was non-empty, reading one byte before the stack buffer. htsAddLink is the first caller to pass an empty savename, which sitemap documents have because they are ingested rather than mirrored, so ASan caught it on the new crawl test. The parser drops a value whose numeric character reference decodes outside printable ASCII, rather than leaving the reference verbatim and seeding a URL the site never published, and classifies a document by its real root element, so a comment naming the other one no longer flips urlset and sitemapindex. The robots.txt line reader is bounded by the body size instead of relying on a NUL terminator. The self-test moves to 01_zlib-sitemap.test: MSan runs 01_engine-* only, because an uninstrumented libz floods it with false positives. Tests gain the assertions the earlier ones were missing: which of the robots.txt route and the /sitemap.xml fallback was taken, that the sitemap documents stay out of the mirror, that the off-host child sitemap is refused, the sitemapindex nesting cap, the per-document URL cap at its production value, and copy_htsopt coverage for the two new fields. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- src/htsselftest.c | 65 +++++++++++++- src/htssitemap.c | 86 +++++++++++++++---- src/htssitemap.h | 4 +- src/htstools.c | 7 +- ...gine-sitemap.test => 01_zlib-sitemap.test} | 0 tests/86_local-sitemap.test | 39 +++++++-- tests/Makefile.am | 2 +- tests/local-server.py | 26 ++++++ 8 files changed, 194 insertions(+), 35 deletions(-) rename tests/{01_engine-sitemap.test => 01_zlib-sitemap.test} (100%) diff --git a/src/htsselftest.c b/src/htsselftest.c index fd5fee999..49a9d5a23 100644 --- a/src/htsselftest.c +++ b/src/htsselftest.c @@ -457,6 +457,9 @@ static void basic_selftests(void) { // link one level up -> a "../" prefix assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0); assertf(strcmp(s, "../a.html") == 0); + // empty current path: the trim used to walk back off the front of it + assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0); + assertf(strcmp(s, "dir/page.html") == 0); } } @@ -1574,6 +1577,22 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) { if (strcmp(StringBuff(to->warc_file), "run.warc.gz") != 0) err = 1; + /* sitemap pair: the flag latches on, the URL takes the String deep copy */ + from->sitemap = HTS_TRUE; + StringCopy(from->sitemap_url, "http://h.test/sitemap.xml"); + to->sitemap = HTS_FALSE; + StringCopy(to->sitemap_url, ""); + copy_htsopt(from, to); + if (!to->sitemap || + strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0) + err = 1; + from->sitemap = HTS_FALSE; + StringCopy(from->sitemap_url, ""); + copy_htsopt(from, to); + if (!to->sitemap || + strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0) + err = 1; + /* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */ from->pause_min_ms = 5000; from->pause_max_ms = 10000; @@ -3640,10 +3659,23 @@ static int st_sitemap(httrackp *opt, int argc, char **argv) { 100, &idx, &c) == 1); assertf(strcmp(c.url[0], "http://h.test/a?b=c") == 0); - /* A reference outside printable ASCII stays verbatim, not a control byte. */ + /* A reference to a byte no URL can hold drops the whole value: decoding it + would smuggle a control character in, keeping it verbatim would seed a URL + the site never published. */ assertf(sm_scan("http://h.test/a b", 100, - &idx, &c) == 1); - assertf(strcmp(c.url[0], "http://h.test/a b") == 0); + &idx, &c) == 0); + assertf(sm_scan("http://h.test/a�b", 100, &idx, + &c) == 0); + + /* A comment naming the other root element must not flip the verdict. */ + assertf(sm_scan("" + "http://h.test/p", + 100, &idx, &c) == 1); + assertf(!idx); + assertf(sm_scan("" + "http://h.test/s", + 100, &idx, &c) == 1); + assertf(idx); /* is not . */ assertf(sm_scan("http://h.test/a", 100, @@ -3680,6 +3712,25 @@ static int st_sitemap(httrackp *opt, int argc, char **argv) { "http://h.test/3", 2, &idx, &c) == 2); + /* The per-document cap at the value the engine actually uses. */ + { + const int many = HTS_SITEMAP_MAX_URLS_DOC + 10; + const size_t per = 32; + char *big = malloct((size_t) many * per + 32); + size_t off; + int i; + + assertf(big != NULL); + off = (size_t) snprintf(big, 32, ""); + for (i = 0; i < many; i++) + off += (size_t) snprintf(big + off, per + 1, + "http://h.test/%d", i); + memset(&c, 0, sizeof(c)); + assertf(hts_sitemap_scan(big, off, HTS_SITEMAP_MAX_URLS_DOC, &idx, sm_take, + &c) == HTS_SITEMAP_MAX_URLS_DOC); + freet(big); + } + /* An unterminated at end of buffer must not read past it. */ assertf(sm_scan("http://h.test/a", 100, &idx, &c) == 0); assertf(sm_scan(" 0x7e) break; } - /* Only ASCII printables: a URL has no business carrying anything - else, and a wider decode would let a reference smuggle in a - control character. */ if (v >= 0x20 && v <= 0x7e) c = (int) v; + else if (v >= 0) + ok = HTS_FALSE; /* a well-formed reference to a byte no URL holds */ } } if (c < 0) { @@ -137,6 +141,7 @@ static void sitemap_unescape(char *s) { } } *w = '\0'; + return ok; } /* Accept only an absolute http(s) URL with no space or control byte. */ @@ -173,6 +178,41 @@ static const char *sitemap_memstr(const char *p, size_t len, return NULL; } +/* HTS_TRUE when the document's root element is `name`. Skips the XML + declaration, comments and processing instructions first, so a comment + mentioning the other root element cannot decide the document type. */ +static hts_boolean sitemap_root_is(const char *doc, size_t size, + const char *name) { + const size_t nlen = strlen(name); + size_t i = 0; + + while (i < size) { + if (isspace((unsigned char) doc[i])) { + i++; + } else if (doc[i] != '<') { + return HTS_FALSE; /* character data before any element: not XML */ + } else if (i + 4 <= size && memcmp(doc + i, ""); + + if (e == NULL) + return HTS_FALSE; + i = (size_t) (e - doc) + 3; + } else if (i + 2 <= size && (doc[i + 1] == '?' || doc[i + 1] == '!')) { + while (i < size && doc[i] != '>') + i++; + i++; + } else { + return i + 1 + nlen <= size && memcmp(doc + i + 1, name, nlen) == 0 && + (i + 1 + nlen == size || + isspace((unsigned char) doc[i + 1 + nlen]) || + doc[i + 1 + nlen] == '>') + ? HTS_TRUE + : HTS_FALSE; + } + } + return HTS_FALSE; +} + /* Decompress a gzip-framed body into a fresh buffer bounded by both the absolute cap and the codec ratio budget. Returns NULL on failure. */ static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) { @@ -226,15 +266,10 @@ int hts_sitemap_scan(const char *body, size_t size, int maxurls, } end = doc + size; - /* Whichever root element comes first classifies the document; the handler - reads the verdict before the first URL, so it must be set up front. */ - if (is_index != NULL) { - const char *const idx = sitemap_memstr(doc, size, " curr)) a--; // chercher dernier / du chemin courant if (*a == '/') diff --git a/tests/01_engine-sitemap.test b/tests/01_zlib-sitemap.test similarity index 100% rename from tests/01_engine-sitemap.test rename to tests/01_zlib-sitemap.test diff --git a/tests/86_local-sitemap.test b/tests/86_local-sitemap.test index 66ef0866d..5874531d1 100644 --- a/tests/86_local-sitemap.test +++ b/tests/86_local-sitemap.test @@ -3,7 +3,7 @@ # --sitemap seeds the crawl from robots.txt -> sitemapindex -> gzipped urlset. # start.html links to nothing, so orphan*.html can only arrive through the # sitemap; deep1.html proves the seeds keep a full depth budget under -r2, and -# the off-host page and child sitemap must both be refused (--errors 0). +# the off-host page and child sitemap must both be refused. set -eu @@ -11,12 +11,23 @@ set -eu crawl() { bash "$top_srcdir/tests/local-crawl.sh" "$@"; } -# robots.txt Sitemap: -> index -> .xml.gz, and the seeds behave like -r2 seeds. +# robots.txt Sitemap: -> index -> .xml.gz, seeds behaving like -r2 seeds. The +# log assertions pin which route was taken: the two documents are served from +# both /sitemapdir/index.xml and the well-known /sitemap.xml, so "some sitemap +# was read" would pass either way. --not-found pins ingestion-only: the sitemap +# documents feed the crawl but never land in the mirror. # --rerun also walks the update path over a mirror holding sitemap seeds. crawl --errors 0 --rerun \ --found 'sitemapdir/orphan1.html' \ --found 'sitemapdir/orphan2.html' \ --found 'sitemapdir/deep1.html' \ + --not-found 'sitemapdir/index.xml' \ + --not-found 'sitemapdir/pages.xml.gz' \ + --not-found 'robots.txt' \ + --log-found 'added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \ + --log-found 'listed by 127\.0\.0\.1:[0-9]+/sitemapdir/index\.xml' \ + --log-found 'ignoring off-host child sitemap' \ + --log-not-found '/sitemap\.xml' \ httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 # Negative control: without the option nothing but the start page is reached. @@ -36,18 +47,30 @@ crawl --errors 0 \ # The test server drops its Sitemap: record for this User-Agent. crawl --errors 0 \ --found 'sitemapdir/orphan1.html' \ - --log-not-found 'sitemapdir/index.xml' \ + --log-found 'listed by 127\.0\.0\.1:[0-9]+/sitemap\.xml' \ + --log-not-found 'sitemapdir/index\.xml' \ httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -F 'nositemap-agent' -# --sitemap-url names the document directly, skipping the robots.txt probe. +# --sitemap-url alone reads the named document and probes nothing. crawl --errors 0 \ --found 'sitemapdir/orphan1.html' \ --found 'sitemapdir/orphan2.html' \ - --log-not-found 'sitemapdir/index.xml' \ + --log-not-found 'sitemapdir/index\.xml' \ + --log-not-found '/sitemap\.xml' \ httrack 'BASEURL/sitemapdir/start.html' -r2 \ --sitemap-url 'BASEURL/sitemapdir/pages.xml.gz' -# An unfetchable sitemap is not fatal: the crawl still completes. -crawl --found 'sitemapdir/start.html' \ - httrack 'BASEURL/sitemapdir/start.html' -r2 \ +# The two options are additive: an unfetchable --sitemap-url is parsed as an +# empty document and does not stop --sitemap from seeding the crawl. +crawl --errors-content 1 \ + --found 'sitemapdir/orphan1.html' \ + --log-found 'Sitemap: 0 URL\(s\) added' \ + httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 \ --sitemap-url 'BASEURL/sitemapdir/missing.xml' + +# The sitemapindex nesting cap stops the chain before its deepest urlset. +crawl --errors 0 \ + --not-found 'sitemapdir/deepcap.html' \ + --log-found 'Sitemap: cap reached' \ + httrack 'BASEURL/sitemapdir/start.html' -r2 \ + --sitemap-url 'BASEURL/sitemapdir/chain0.xml' diff --git a/tests/Makefile.am b/tests/Makefile.am index 772173ec6..4ac4fb249 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -89,8 +89,8 @@ TESTS = \ 01_engine-warc-surt.test \ 01_engine-xfread.test \ 01_zlib-acceptencoding.test \ - 01_engine-sitemap.test \ 01_zlib-warc.test \ + 01_zlib-sitemap.test \ 01_zlib-warc-cdx.test \ 01_zlib-warc-wacz.test \ 01_zlib-contentcodings.test \ diff --git a/tests/local-server.py b/tests/local-server.py index 2f3e1efe1..cde046a4b 100755 --- a/tests/local-server.py +++ b/tests/local-server.py @@ -18,6 +18,7 @@ import gzip import hashlib import os +import re import sys import time from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer @@ -602,6 +603,28 @@ def route_sitemap_orphan2(self): def route_sitemap_deep1(self): self.send_html("\tOne level below an orphan.") + # chain0..chain5 nest one sitemapindex per level; chain6 is the urlset the + # nesting cap must keep out of reach. + def route_sitemap_chain(self): + host = self.headers.get("Host") + level = int(self.path.rsplit("/", 1)[-1][len("chain") : -len(".xml")]) + if level < 6: + body = ( + '\n' + f"http://{host}/sitemapdir/chain{level + 1}.xml" + "\n" + ) + else: + body = ( + '\n' + f"http://{host}/sitemapdir/deepcap.html" + "\n" + ) + self.send_raw(body.encode(), "application/xml") + + def route_sitemap_deepcap(self): + self.send_html("\tReachable only past the sitemapindex nesting cap.") + # --- type/extension matrix (issue #267 family) ------------------------- def send_raw(self, body, content_type, extra_headers=()): @@ -1636,6 +1659,7 @@ def route_maxrecv_404(self): "/sitemapdir/orphan1.html": route_sitemap_orphan1, "/sitemapdir/orphan2.html": route_sitemap_orphan2, "/sitemapdir/deep1.html": route_sitemap_deep1, + "/sitemapdir/deepcap.html": route_sitemap_deepcap, "/warcgz/index.html": route_warcgz_index, "/warcgz/page.html": route_warcgz_page, "/warcgz/data.bin": route_warcgz_data, @@ -1960,6 +1984,8 @@ def dispatch(self): return True # Match percent-encoded paths (accented #157 route) by their decoded form. handler = self.ROUTES.get(path) or self.ROUTES.get(unquote(path)) + if handler is None and re.fullmatch(r"/sitemapdir/chain\d+\.xml", path): + handler = type(self).route_sitemap_chain if handler is not None: handler(self) return True From dcfc4acef8d9a192f1beb6e1a9ce6dceae8d0488 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 18:00:57 +0200 Subject: [PATCH 04/14] sitemap: state what the decompression cap actually binds on deflate tops out near 1032:1, so hts_codec_maxout never binds before the 64 MiB cap; the old comment implied a ratio guard that cannot fire. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- src/htssitemap.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/htssitemap.c b/src/htssitemap.c index 315ad4aaf..11601bccd 100644 --- a/src/htssitemap.c +++ b/src/htssitemap.c @@ -213,8 +213,9 @@ static hts_boolean sitemap_root_is(const char *doc, size_t size, return HTS_FALSE; } -/* Decompress a gzip-framed body into a fresh buffer bounded by both the - absolute cap and the codec ratio budget. Returns NULL on failure. */ +/* Decompress a gzip-framed body into a fresh buffer. The 64 MiB cap is what + binds in practice; deflate tops out near 1032:1, so the tree's codec budget + only matters as the shared policy for a coding that could go further. */ static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) { const LLint budget = hts_codec_maxout((LLint) size); size_t cap = budget < (LLint) HTS_SITEMAP_MAX_BYTES From 0ce7da19733dd317a29b6e4323dc684cfe8b5f1a Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 18:14:41 +0200 Subject: [PATCH 05/14] sitemap: a child sitemap is a fetch, so filters and robots.txt must gate it An adversarial review found that a , and a robots.txt Sitemap: line, went straight to hts_record_link: the request went out even when a -* rule or robots.txt Disallow covered it. Only the half ran through the wizard. Gate the document itself on the filters and on robots.txt, which is all that can apply: the wizard proper wants a referring link, and its up/down travel rules would judge a child sitemap against the parent sitemap's own directory. The robots.txt probe is exempt, being the request that fetches the rules. A 301 also used to end ingestion silently, since the engine re-queues the target as a fresh link that carried no sitemap marking. That hit any site redirecting http to https. The marking now follows the redirect. The "N URL(s) added" counter reported what the scanner emitted rather than what was taken, which hid both of the above; it now reads "N of M". The fallback to /sitemap.xml keys on the same corrected count, so a robots.txt whose only Sitemap: line is off-host or filtered still falls back. Root classification skips a UTF-8 BOM and an XML namespace prefix, and the doc list is cleared when a mirror starts rather than only when it ends. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- src/htscore.c | 19 ++++-- src/htssitemap.c | 112 ++++++++++++++++++++++++++++++------ src/htssitemap.h | 5 ++ tests/86_local-sitemap.test | 28 ++++++++- tests/local-server.py | 34 +++++++++++ 5 files changed, 175 insertions(+), 23 deletions(-) diff --git a/src/htscore.c b/src/htscore.c index 9246a49a4..6e44167eb 100644 --- a/src/htscore.c +++ b/src/htscore.c @@ -946,6 +946,7 @@ int httpmirror(char *url1, httrackp * opt) { /* --sitemap: queue the sitemap probe just after the seeds, so its URLs are injected before the crawl gets far. */ + hts_sitemap_free(opt); /* an earlier mirror may have left a doc list */ if (opt->sitemap || StringNotEmpty(opt->sitemap_url)) { char BIGSTK first[HTS_URLMAXSIZE * 2]; const char *const eol = strchr(primary, '\n'); @@ -1603,11 +1604,21 @@ int httpmirror(char *url1, httrackp * opt) { stre.maketrack_fp = maketrack_fp; /* Parse */ - if (hts_mirror_check_moved(&str, &stre) != 0) { - XH_uninit; - return -1; - } + { + const int nlinks = opt->lien_tot; + if (hts_mirror_check_moved(&str, &stre) != 0) { + XH_uninit; + return -1; + } + /* A redirect re-queues the target as a fresh link; without carrying + the marking over, a moved sitemap is fetched and then ignored. */ + if (opt->sitemap_state != NULL && opt->lien_tot > nlinks && + hts_sitemap_pending(opt, urladr(), urlfil())) { + hts_sitemap_redirect(opt, urladr(), urlfil(), heap_top()->adr, + heap_top()->fil); + } + } } } // if !error diff --git a/src/htssitemap.c b/src/htssitemap.c index 11601bccd..88c883fcf 100644 --- a/src/htssitemap.c +++ b/src/htssitemap.c @@ -37,6 +37,7 @@ Please visit our Website: http://www.httrack.com #include "htsbase.h" #include "htscodec.h" +#include "htsfilters.h" #include "htslib.h" #include "htsrobots.h" #include "htssafe.h" @@ -186,6 +187,8 @@ static hts_boolean sitemap_root_is(const char *doc, size_t size, const size_t nlen = strlen(name); size_t i = 0; + if (size >= 3 && memcmp(doc, "\xef\xbb\xbf", 3) == 0) + i = 3; /* UTF-8 BOM */ while (i < size) { if (isspace((unsigned char) doc[i])) { i++; @@ -202,10 +205,20 @@ static hts_boolean sitemap_root_is(const char *doc, size_t size, i++; i++; } else { - return i + 1 + nlen <= size && memcmp(doc + i + 1, name, nlen) == 0 && - (i + 1 + nlen == size || - isspace((unsigned char) doc[i + 1 + nlen]) || - doc[i + 1 + nlen] == '>') + size_t j = i + 1; + + /* an optional namespace prefix: is the same element */ + while (j < size && doc[j] != ':' && doc[j] != '>' && + !isspace((unsigned char) doc[j])) + j++; + if (j >= size || doc[j] != ':') + j = i + 1; + else + j++; + return j + nlen <= size && memcmp(doc + j, name, nlen) == 0 && + (j + nlen == size || + isspace((unsigned char) doc[j + nlen]) || + doc[j + nlen] == '>' || doc[j + nlen] == '/') ? HTS_TRUE : HTS_FALSE; } @@ -394,12 +407,45 @@ static sitemap_doc *sitemap_find(httrackp *opt, const char *adr, return NULL; } +/* A sitemap document is fetched like any other resource, so the user's filters + and robots.txt decide whether it may be. The wizard proper is not usable + here: it wants a referring link, and its up/down travel rules would judge a + child sitemap against the parent sitemap's directory. */ +static hts_boolean sitemap_fetch_allowed(httrackp *opt, const char *adr, + const char *fil) { + char BIGSTK l[HTS_URLMAXSIZE * 2], lfull[HTS_URLMAXSIZE * 2]; + int jokdepth = 0; + + if (opt->robots && opt->robotsptr != NULL && + checkrobots((robots_wizard *) opt->robotsptr, adr, fil) == -1) { + hts_log_print(opt, LOG_NOTICE, "Sitemap: robots.txt forbids %s%s", adr, + fil); + return HTS_FALSE; + } + strcpybuff(l, jump_identification_const(adr)); + if (*fil != '/') + strcatbuff(l, "/"); + strcatbuff(l, fil); + strcpybuff(lfull, link_has_authority(adr) ? "" : "http://"); + strcatbuff(lfull, adr); + if (*fil != '/') + strcatbuff(lfull, "/"); + strcatbuff(lfull, fil); + if (fa_strjoker_dual(0, *opt->filters.filters, *opt->filters.filptr, lfull, l, + NULL, NULL, &jokdepth) == -1) { + hts_log_print(opt, LOG_NOTICE, "Sitemap: filter rule #%d refuses %s%s", + jokdepth + 1, adr, fil); + return HTS_FALSE; + } + return HTS_TRUE; +} + /* Queue a document and record its link with save="" so the body stays in memory: a sitemap is ingested, never mirrored. Top priority so its URLs get the full depth budget through htsAddLink. */ -static hts_boolean sitemap_queue(httrackp *opt, const char *adr, - const char *fil, int level, - hts_boolean is_robots) { +static hts_boolean sitemap_queue_(httrackp *opt, const char *adr, + const char *fil, int level, + hts_boolean is_robots, hts_boolean link_it) { hts_sitemap_state *const st = sitemap_state(opt); sitemap_doc *d; @@ -414,6 +460,10 @@ static hts_boolean sitemap_queue(httrackp *opt, const char *adr, return HTS_FALSE; if (sitemap_find(opt, adr, fil) != NULL) return HTS_FALSE; + /* The robots.txt probe is the request that fetches the rules, so it cannot be + judged by them; everything else can. */ + if (!is_robots && !sitemap_fetch_allowed(opt, adr, fil)) + return HTS_FALSE; d = calloct(1, sizeof(sitemap_doc)); if (d == NULL) return HTS_FALSE; @@ -425,6 +475,8 @@ static hts_boolean sitemap_queue(httrackp *opt, const char *adr, st->docs = d; st->ndocs++; + if (!link_it) + return HTS_TRUE; if (!hts_record_link(opt, adr, fil, "", "", "", NULL)) return HTS_FALSE; heap_top()->testmode = 0; @@ -438,6 +490,25 @@ static hts_boolean sitemap_queue(httrackp *opt, const char *adr, return HTS_TRUE; } +static hts_boolean sitemap_queue(httrackp *opt, const char *adr, + const char *fil, int level, + hts_boolean is_robots) { + return sitemap_queue_(opt, adr, fil, level, is_robots, HTS_TRUE); +} + +void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil, + const char *newadr, const char *newfil) { + sitemap_doc *const d = sitemap_find(opt, adr, fil); + + if (d == NULL || d->done) + return; + d->done = HTS_TRUE; /* the body lives at the target now */ + /* The engine already queued the target link, so only the marking moves. */ + (void) sitemap_queue_(opt, newadr, newfil, d->level, d->is_robots, HTS_FALSE); + hts_log_print(opt, LOG_NOTICE, "Sitemap: %s%s redirects to %s%s", adr, fil, + newadr, newfil); +} + void hts_sitemap_seed(httrackp *opt, const char *starturl) { char BIGSTK url[HTS_URLMAXSIZE * 2]; lien_adrfil af; @@ -482,6 +553,7 @@ typedef struct sitemap_ingest_ctx { const char *adr; /* host of the document being ingested */ int level; hts_boolean is_index; + int accepted; /* URLs seeded or documents queued, not merely parsed */ } sitemap_ingest_ctx; /* A of a : hand it to the wizard as a top-level seed. */ @@ -501,7 +573,9 @@ static hts_boolean sitemap_seed_url(void *arg, const char *url) { return HTS_TRUE; st->nurls++; strcpybuff(buff, url); - (void) htsAddLink(c->str, buff); + /* htsAddLink reports the wizard's verdict; only count what it took. */ + if (htsAddLink(c->str, buff)) + c->accepted++; return HTS_TRUE; } @@ -524,7 +598,8 @@ static hts_boolean sitemap_seed_child(void *arg, const char *url) { af.fil); return HTS_TRUE; } - (void) sitemap_queue(c->opt, af.adr, af.fil, c->level + 1, HTS_FALSE); + if (sitemap_queue(c->opt, af.adr, af.fil, c->level + 1, HTS_FALSE)) + c->accepted++; return HTS_TRUE; } @@ -551,6 +626,7 @@ void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, ctx.adr = adr; ctx.level = d->level; ctx.is_index = HTS_FALSE; + ctx.accepted = 0; if (d->is_robots) { hts_sitemap_state *const st = sitemap_state(opt); @@ -558,13 +634,16 @@ void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, n = body != NULL ? hts_sitemap_scan_robots(body, size, HTS_SITEMAP_MAX_DOCS, sitemap_seed_child, &ctx) : 0; - /* Fall back to the well-known location only when robots.txt named none. */ - if (n == 0 && st != NULL && !st->fallback_done) { + /* Fall back to the well-known location when robots.txt queued nothing: a + lone off-host or filtered Sitemap: line leaves us with no sitemap at all. + */ + if (ctx.accepted == 0 && st != NULL && !st->fallback_done) { st->fallback_done = HTS_TRUE; (void) sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_FALSE); } - hts_log_print(opt, LOG_INFO, "Sitemap: %d sitemap(s) declared in %s%s", n, - adr, fil); + hts_log_print(opt, LOG_INFO, + "Sitemap: %d of %d sitemap(s) declared in %s%s", ctx.accepted, + n, adr, fil); return; } @@ -577,10 +656,11 @@ void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, } if (ctx.is_index) hts_log_print(opt, LOG_NOTICE, - "Sitemap: %d child sitemap(s) listed by %s%s", n, adr, fil); + "Sitemap: %d of %d child sitemap(s) listed by %s%s", + ctx.accepted, n, adr, fil); else - hts_log_print(opt, LOG_NOTICE, "Sitemap: %d URL(s) added from %s%s", n, adr, - fil); + hts_log_print(opt, LOG_NOTICE, "Sitemap: %d of %d URL(s) added from %s%s", + ctx.accepted, n, adr, fil); } void hts_sitemap_free(httrackp *opt) { diff --git a/src/htssitemap.h b/src/htssitemap.h index a899d37f6..602e16bac 100644 --- a/src/htssitemap.h +++ b/src/htssitemap.h @@ -74,6 +74,11 @@ int hts_sitemap_scan_robots(const char *body, size_t size, int maxurls, command-line seed. No-op when neither option is set. */ void hts_sitemap_seed(httrackp *opt, const char *starturl); +/* Carry the sitemap marking of (adr,fil) over to the target of a redirect the + engine has already queued, so a moved sitemap is still ingested. */ +void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil, + const char *newadr, const char *newfil); + /* HTS_TRUE when (adr,fil) is a queued sitemap document awaiting ingestion. */ hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr, const char *fil); diff --git a/tests/86_local-sitemap.test b/tests/86_local-sitemap.test index 5874531d1..184a2e370 100644 --- a/tests/86_local-sitemap.test +++ b/tests/86_local-sitemap.test @@ -24,8 +24,8 @@ crawl --errors 0 --rerun \ --not-found 'sitemapdir/index.xml' \ --not-found 'sitemapdir/pages.xml.gz' \ --not-found 'robots.txt' \ - --log-found 'added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \ - --log-found 'listed by 127\.0\.0\.1:[0-9]+/sitemapdir/index\.xml' \ + --log-found '2 of 3 URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \ + --log-found '1 of 2 child sitemap\(s\) listed by 127\.0\.0\.1:[0-9]+/sitemapdir/index\.xml' \ --log-found 'ignoring off-host child sitemap' \ --log-not-found '/sitemap\.xml' \ httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 @@ -64,7 +64,7 @@ crawl --errors 0 \ # empty document and does not stop --sitemap from seeding the crawl. crawl --errors-content 1 \ --found 'sitemapdir/orphan1.html' \ - --log-found 'Sitemap: 0 URL\(s\) added' \ + --log-found 'Sitemap: 0 of 0 URL\(s\) added' \ httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 \ --sitemap-url 'BASEURL/sitemapdir/missing.xml' @@ -74,3 +74,25 @@ crawl --errors 0 \ --log-found 'Sitemap: cap reached' \ httrack 'BASEURL/sitemapdir/start.html' -r2 \ --sitemap-url 'BASEURL/sitemapdir/chain0.xml' + +# A child sitemap is a fetch like any other: a filter rejecting it stops the +# whole subtree, so the page only it lists is never reached. +crawl --errors 0 \ + --not-found 'sitemapdir/gated.html' \ + --log-found 'Sitemap: filter rule #[0-9]+ refuses' \ + httrack 'BASEURL/sitemapdir/start.html' -r2 '-*filtered.xml*' \ + --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml' + +# Control: without that filter the same chain does reach the page. +crawl --errors 0 \ + --found 'sitemapdir/gated.html' \ + httrack 'BASEURL/sitemapdir/start.html' -r2 \ + --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml' + +# A redirected sitemap is still ingested: the marking follows the 301. +crawl --errors 0 \ + --found 'sitemapdir/orphan1.html' \ + --log-found 'moved\.xml redirects to .*pages\.xml\.gz' \ + --log-found 'URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \ + httrack 'BASEURL/sitemapdir/start.html' -r2 \ + --sitemap-url 'BASEURL/sitemapdir/moved.xml' diff --git a/tests/local-server.py b/tests/local-server.py index cde046a4b..67413cb50 100755 --- a/tests/local-server.py +++ b/tests/local-server.py @@ -625,6 +625,36 @@ def route_sitemap_chain(self): def route_sitemap_deepcap(self): self.send_html("\tReachable only past the sitemapindex nesting cap.") + # An index naming a child that filters (or robots.txt) must be able to + # refuse: filtered.xml would list gated.html if it were ever fetched. + def route_sitemap_gatedindex(self): + host = self.headers.get("Host") + self.send_raw( + '' + f"http://{host}/sitemapdir/filtered.xml" + "\n".encode(), + "application/xml", + ) + + def route_sitemap_filtered(self): + host = self.headers.get("Host") + self.send_raw( + '' + f"http://{host}/sitemapdir/gated.html" + "\n".encode(), + "application/xml", + ) + + def route_sitemap_gated(self): + self.send_html("\tListed only by the filtered child sitemap.") + + # A moved sitemap: the marking has to follow the redirect. + def route_sitemap_moved(self): + self.send_response(301) + self.send_header("Location", "/sitemapdir/pages.xml.gz") + self.send_header("Content-Length", "0") + self.end_headers() + # --- type/extension matrix (issue #267 family) ------------------------- def send_raw(self, body, content_type, extra_headers=()): @@ -1660,6 +1690,10 @@ def route_maxrecv_404(self): "/sitemapdir/orphan2.html": route_sitemap_orphan2, "/sitemapdir/deep1.html": route_sitemap_deep1, "/sitemapdir/deepcap.html": route_sitemap_deepcap, + "/sitemapdir/gatedindex.xml": route_sitemap_gatedindex, + "/sitemapdir/filtered.xml": route_sitemap_filtered, + "/sitemapdir/gated.html": route_sitemap_gated, + "/sitemapdir/moved.xml": route_sitemap_moved, "/warcgz/index.html": route_warcgz_index, "/warcgz/page.html": route_warcgz_page, "/warcgz/data.bin": route_warcgz_data, From 3745d8321b3de22ed46d10a7c98f1c88a41a75c1 Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 19:10:01 +0200 Subject: [PATCH 06/14] sitemap: gate each fetch by who asked for it, and anchor travel on the start URL A nine-agent review found a scope escape: the sitemap document was its own `premier`, so the wizard measured travel from wherever the site chose to put its sitemap. A root /sitemap.xml therefore widened a /deep/dir/ crawl to the whole host. The ingester now points the wizard at the crawl's own start link and lets each seeded URL become its own anchor, which is what a command-line seed gets. Robots handling was both mistimed and undifferentiated. The Sitemap: lines are now collected by robots_parse, on the same body in the same fetch, and acted on after the parsed rules are installed rather than before; and the decision comes from a new hts_robots_forbids extracted out of the wizard, so the sitemap path inherits the -s1 filters-win override instead of a stricter hand-rolled check. The four fetches are no longer treated alike: a sitemap the user names is user intent, one the site declares invites the fetch, only the guessed /sitemap.xml obeys a Disallow, and the URLs listed inside stay fully gated. Also: hts_unescapeEntities replaces the private entity decoder, whose guard tests and fuzz corpus it silently forfeited; hts_codec_head replaces hts_zhead, which is only defined under HTS_USEZLIB; the composed URL buffer now fits two maximal components plus a scheme, which a 2046-byte --sitemap-url reached; the bounded search is promoted to htstools as hts_memstr; and the live state moves from httrackp into htsoptstate, leaving two installed fields rather than three. Tests gain the scope escape, the three robots cases, a cap-boundary control, the handler invocation count and a compression-bomb decode. Every one was checked against a deliberately broken build. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- html/cmdguide.html | 6 + html/httrack.man.html | 5 +- man/httrack.1 | 2 +- src/htscore.c | 17 +- src/htshelp.c | 5 +- src/htsopt.h | 2 +- src/htsrobots.c | 17 +- src/htsrobots.h | 6 +- src/htsselftest.c | 105 +++++++--- src/htssitemap.c | 388 +++++++++++++++--------------------- src/htssitemap.h | 19 +- src/htstools.c | 20 +- src/htstools.h | 5 + src/htswizard.c | 55 +++-- src/htswizard.h | 7 + tests/87_local-sitemap.test | 42 +++- tests/local-server.py | 99 +++++++-- 17 files changed, 479 insertions(+), 321 deletions(-) diff --git a/html/cmdguide.html b/html/cmdguide.html index aab16faa2..4ee724d4b 100644 --- a/html/cmdguide.html +++ b/html/cmdguide.html @@ -177,6 +177,12 @@

2. Scope: how far the crawl reaches

off by default because a sitemap can list thousands of pages nothing links to.

+

One surprise worth knowing: a sitemap you name with --sitemap-url, +and one the site itself declares in robots.txt, are fetched even when +robots.txt disallows that path, because naming or declaring a sitemap +is an invitation to read it. Only the guessed /sitemap.xml obeys a +Disallow. The URLs listed inside are gated normally either way.

+

The single most common surprise is "only the home page came down." That is usually not a scope option at all: it is an off-host redirect. A start URL of http://example.com/ that redirects to https://www.example.com/ diff --git a/html/httrack.man.html b/html/httrack.man.html index c1525ec88..a1af61d65 100644 --- a/html/httrack.man.html +++ b/html/httrack.man.html @@ -587,7 +587,10 @@

Links options:

seed the crawl from the site’s sitemap (robots.txt Sitemap:, then /sitemap.xml); --sitemap-url URL names one -explicitly (--sitemap)

+explicitly. A sitemap you name, or one the site declares, is +fetched even under robots.txt Disallow; only the guessed +/sitemap.xml obeys it. The URLs found still pass every +filter and scope rule (--sitemap)

Build options: diff --git a/man/httrack.1 b/man/httrack.1 index 3ea890a52..be9f69399 100644 --- a/man/httrack.1 +++ b/man/httrack.1 @@ -189,7 +189,7 @@ test all URLs (even forbidden ones) (\-\-test) .IP \-%S add all scan rules located in this text file (one scan rule per line) (\-\-urllist ) .IP \-%m -seed the crawl from the site's sitemap (robots.txt Sitemap:, then /sitemap.xml); \-\-sitemap\-url URL names one explicitly (\-\-sitemap) +seed the crawl from the site's sitemap (robots.txt Sitemap:, then /sitemap.xml); \-\-sitemap\-url URL names one explicitly. A sitemap you name, or one the site declares, is fetched even under robots.txt Disallow; only the guessed /sitemap.xml obeys it. The URLs found still pass every filter and scope rule (\-\-sitemap) .SS Build options: .IP \-NN structure type (0 *original structure, 1+: see below) (\-\-structure[=N]) diff --git a/src/htscore.c b/src/htscore.c index 6e44167eb..3200dff63 100644 --- a/src/htscore.c +++ b/src/htscore.c @@ -1613,7 +1613,7 @@ int httpmirror(char *url1, httrackp * opt) { } /* A redirect re-queues the target as a fresh link; without carrying the marking over, a moved sitemap is fetched and then ignored. */ - if (opt->sitemap_state != NULL && opt->lien_tot > nlinks && + if (opt->state.sitemap != NULL && opt->lien_tot > nlinks && hts_sitemap_pending(opt, urladr(), urlfil())) { hts_sitemap_redirect(opt, urladr(), urlfil(), heap_top()->adr, heap_top()->fil); @@ -1639,9 +1639,10 @@ int httpmirror(char *url1, httrackp * opt) { /* Sitemap document: turn its URLs into top-level seeds. They go through htsAddLink, so the wizard's filters and scope rules decide, and this link's max depth leaves them the full budget. */ - if (opt->sitemap_state != NULL && + if (opt->state.sitemap != NULL && hts_sitemap_pending(opt, urladr(), urlfil())) { htsmoduleStruct BIGSTK smstr; + int smptr = ptr; memset(&smstr, 0, sizeof(smstr)); smstr.opt = opt; @@ -1649,7 +1650,7 @@ int httpmirror(char *url1, httrackp * opt) { smstr.cache = &cache; smstr.hashptr = hashptr; smstr.numero_passe = numero_passe; - smstr.ptr_ = &ptr; + smstr.ptr_ = &smptr; /* scratch: the ingester retargets the wizard */ smstr.addLink = htsAddLink; smstr.url_host = urladr(); smstr.url_file = urlfil(); @@ -1868,6 +1869,9 @@ int httpmirror(char *url1, httrackp * opt) { if (strnotempty(savename()) == 0) { // pas de chemin de sauvegarde if (strcmp(urlfil(), "/robots.txt") == 0) { // robots.txt + char BIGSTK sitemaps[8192]; + + sitemaps[0] = '\0'; if (r.adr) { char BIGSTK infobuff[8192]; #ifdef IGNORE_RESTRICTIVE_ROBOTS @@ -1879,7 +1883,8 @@ int httpmirror(char *url1, httrackp * opt) { #endif robots_parse(&robots, urladr(), r.adr, r.size, infobuff, - sizeof(infobuff), keep_root); + sizeof(infobuff), keep_root, sitemaps, + sizeof(sitemaps)); if (strnotempty(infobuff)) { hts_log_print(opt, LOG_INFO, "Note: robots.txt forbidden links for %s are: %s", @@ -1889,6 +1894,10 @@ int httpmirror(char *url1, httrackp * opt) { urladr(), infobuff); } } + /* After robots_parse, so the rules this very body carries already + gate the sitemap fetch. Runs even on a failed probe, which is + what falls back to the well-known location. */ + hts_sitemap_robots(opt, urladr(), sitemaps); } } else if (r.is_write) { // déja sauvé sur disque /* diff --git a/src/htshelp.c b/src/htshelp.c index a61a51776..79e9fffd9 100644 --- a/src/htshelp.c +++ b/src/htshelp.c @@ -526,7 +526,10 @@ void help(const char *app, int more) { infomsg (" %S add all scan rules located in this text file (one scan rule per line)"); infomsg(" %m seed the crawl from the site's sitemap (robots.txt Sitemap:, " - "then /sitemap.xml); --sitemap-url URL names one explicitly"); + "then /sitemap.xml); --sitemap-url URL names one explicitly. A " + "sitemap you name, or one the site declares, is fetched even under " + "robots.txt Disallow; only the guessed /sitemap.xml obeys it. The " + "URLs found still pass every filter and scope rule"); infomsg(""); infomsg("Build options:"); infomsg(" NN structure type (0 *original structure, 1+: see below)"); diff --git a/src/htsopt.h b/src/htsopt.h index 44fcc2b98..0767a7097 100644 --- a/src/htsopt.h +++ b/src/htsopt.h @@ -257,6 +257,7 @@ struct htsoptstate { unsigned int tmpnameid; /**< counter for temporary file names */ int is_ended; /**< mirror has finished */ void *warc; /**< open WARC writer (warc_writer*), or NULL */ + void *sitemap; /**< sitemap ingestion state, or NULL */ }; /* Library handles */ @@ -550,7 +551,6 @@ struct httrackp { hts_boolean sitemap; /**< --sitemap: probe the start host's robots.txt for Sitemap: lines, else /sitemap.xml. Tail: ABI */ String sitemap_url; /**< --sitemap-url: sitemap to ingest. Tail: ABI */ - void *sitemap_state; /**< live sitemap ingestion state, or NULL. Tail: ABI */ }; /* Running statistics for a mirror. */ diff --git a/src/htsrobots.c b/src/htsrobots.c index d723c2ed7..cbdcfc93b 100644 --- a/src/htsrobots.c +++ b/src/htsrobots.c @@ -147,7 +147,8 @@ static void robots_blob_add(char *blob, size_t blobsize, char marker, void robots_parse(robots_wizard *robots, const char *adr, const char *body, size_t bodysize, char *info, size_t infosize, - hts_boolean keep_root_disallow) { + hts_boolean keep_root_disallow, char *sitemaps, + size_t sitemapsize) { size_t bptr = 0; int record = 0; char BIGSTK line[1024]; @@ -156,6 +157,8 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body, blob[0] = '\0'; if (info != NULL && infosize > 0) info[0] = '\0'; + if (sitemaps != NULL && sitemapsize > 0) + sitemaps[0] = '\0'; #if DEBUG_ROBOTS printf("robots.txt dump:\n%s\n", body); #endif @@ -172,7 +175,17 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body, line[llen - 1] = '\0'; llen--; } - if (strfield(line, "user-agent:")) { + if (sitemaps != NULL && strfield(line, "sitemap:")) { + // group-independent record (RFC 9309): collected whatever the group + char *a = line + 8; + + while (is_realspace(*a)) + a++; + if (strnotempty(a) && strlen(a) + 2 < sitemapsize - strlen(sitemaps)) { + strlcatbuff(sitemaps, a, sitemapsize); + strlcatbuff(sitemaps, "\n", sitemapsize); + } + } else if (strfield(line, "user-agent:")) { char *a = line + 11; while (is_realspace(*a)) diff --git a/src/htsrobots.h b/src/htsrobots.h index e2d1fb3e4..84ef58aa5 100644 --- a/src/htsrobots.h +++ b/src/htsrobots.h @@ -56,10 +56,12 @@ int checkrobots(robots_wizard * robots, const char *adr, const char *fil); void checkrobots_free(robots_wizard * robots); int checkrobots_set(robots_wizard * robots, const char *adr, const char *data); /* Parse robots.txt `body` for `adr`, storing the HTTrack group's rules; `info` - gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /". */ + gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /", and + `sitemaps` (optional) collects the Sitemap: URLs, one per line. */ void robots_parse(robots_wizard *robots, const char *adr, const char *body, size_t bodysize, char *info, size_t infosize, - hts_boolean keep_root_disallow); + hts_boolean keep_root_disallow, char *sitemaps, + size_t sitemapsize); #endif #endif diff --git a/src/htsselftest.c b/src/htsselftest.c index ebd42b9a9..c3d47bd32 100644 --- a/src/htsselftest.c +++ b/src/htsselftest.c @@ -3618,7 +3618,7 @@ static int rb_decide(robots_wizard *r, const char *txt, const char *path) { char host[64]; snprintf(host, sizeof(host), "h%d.example", n++); - robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE); + robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE, NULL, 0); return checkrobots(r, host, path); } @@ -3760,13 +3760,16 @@ static int st_sitemap(httrackp *opt, int argc, char **argv) { 100, &idx, &c) == 1); assertf(strcmp(c.url[0], "http://h.test/a?b=c") == 0); - /* A reference to a byte no URL can hold drops the whole value: decoding it - would smuggle a control character in, keeping it verbatim would seed a URL - the site never published. */ + /* A reference decoding to a control byte is dropped: the shared decoder + writes the real character and the URL check refuses it. A reference the + decoder cannot represent (�) stays verbatim, like an unknown entity. */ assertf(sm_scan("http://h.test/a b", 100, &idx, &c) == 0); - assertf(sm_scan("http://h.test/a�b", 100, &idx, + assertf(sm_scan("http://h.test/a b", 100, &idx, &c) == 0); + assertf(sm_scan("http://h.test/a�b", 100, &idx, + &c) == 1); + assertf(strcmp(c.url[0], "http://h.test/a�b") == 0); /* A comment naming the other root element must not flip the verdict. */ assertf(sm_scan("" @@ -3829,9 +3832,58 @@ static int st_sitemap(httrackp *opt, int argc, char **argv) { memset(&c, 0, sizeof(c)); assertf(hts_sitemap_scan(big, off, HTS_SITEMAP_MAX_URLS_DOC, &idx, sm_take, &c) == HTS_SITEMAP_MAX_URLS_DOC); + /* The handler count, not just the return: a call site hardcoding a smaller + cap would still return its own argument. */ + assertf(c.n == HTS_SITEMAP_MAX_URLS_DOC); freet(big); } +#if HTS_USEZLIB + /* A highly compressible document decodes without running away: the ratio + budget cannot bind (deflate tops out near 1032:1), so this pins the + decompression path itself rather than the 64 MiB ceiling. */ + { + const char *const one = "http://h.test/bomb"; + const size_t reps = 40000; + size_t xlen = 8 + reps * strlen(one) + 10, i; + char *x = malloct(xlen + 1); + uLongf zlen; + char *z; + z_stream zs; + + assertf(x != NULL); + { + size_t w = (size_t) snprintf(x, xlen, ""); + + for (i = 0; i < reps; i++) + w += (size_t) snprintf(x + w, xlen - w, "%s", one); + w += (size_t) snprintf(x + w, xlen - w, ""); + xlen = w; + } + zlen = compressBound((uLong) xlen) + 32; + z = malloct((size_t) zlen); + assertf(z != NULL); + memset(&zs, 0, sizeof(zs)); + assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8, + Z_DEFAULT_STRATEGY) == Z_OK); + zs.next_in = (const Bytef *) x; + zs.avail_in = (uInt) xlen; + zs.next_out = (Bytef *) z; + zs.avail_out = (uInt) zlen; + assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END); + zlen = (uLongf) zs.total_out; + deflateEnd(&zs); + /* well over the 4096:1 budget's 1 MiB floor, and far under the 64 MiB cap + */ + assertf(xlen > 1024 * 1024 && (size_t) zlen < xlen / 100); + memset(&c, 0, sizeof(c)); + assertf(hts_sitemap_scan(z, (size_t) zlen, 10, &idx, sm_take, &c) == 10); + assertf(strcmp(c.url[0], "http://h.test/bomb") == 0); + freet(z); + freet(x); + } +#endif + /* An unterminated at end of buffer must not read past it. */ assertf(sm_scan("http://h.test/a", 100, &idx, &c) == 0); assertf(sm_scan(" #include -/* One queued sitemap document. `is_robots` marks the robots.txt probe, whose - Sitemap: lines feed this same list. */ +/* One queued sitemap document awaiting ingestion. */ typedef struct sitemap_doc { char adr[HTS_URLMAXSIZE]; char fil[HTS_URLMAXSIZE]; int level; - hts_boolean is_robots; + hts_sitemap_source src; hts_boolean done; struct sitemap_doc *next; } sitemap_doc; @@ -62,7 +63,12 @@ struct hts_sitemap_state { sitemap_doc *docs; int ndocs; /* documents queued, capped by HTS_SITEMAP_MAX_DOCS */ int nurls; /* URLs seeded, capped by HTS_SITEMAP_MAX_URLS_TOTAL */ + hts_boolean probe_done; /* the robots.txt probe has been answered */ hts_boolean fallback_done; /* the /sitemap.xml fallback was already queued */ + /* The crawl's own start URL. Seeded URLs are judged against it, so a site + cannot widen a subtree crawl by putting its sitemap at the root. */ + char anchor_adr[HTS_URLMAXSIZE]; + char anchor_fil[HTS_URLMAXSIZE]; }; typedef struct hts_sitemap_state hts_sitemap_state; @@ -70,81 +76,6 @@ typedef struct hts_sitemap_state hts_sitemap_state; /* Document parsing (no engine state: fuzzable and self-testable) */ /* --------------------------------------------------------------------- */ -/* Decode the five XML predefined entities and numeric character references - in-place, never growing the string. Returns HTS_FALSE when a reference - decodes outside printable ASCII: the value is then not a URL the site can - have published, and decoding it would smuggle a control byte into the - crawl. An unrecognized "&..." run is left verbatim ('&' is legal in a URL). - */ -static hts_boolean sitemap_unescape(char *s) { - char *r = s, *w = s; - hts_boolean ok = HTS_TRUE; - - while (*r != '\0') { - if (*r != '&') { - *w++ = *r++; - continue; - } - { - char *const semi = strchr(r + 1, ';'); - const size_t len = semi != NULL ? (size_t) (semi - (r + 1)) : 0; - int c = -1; - - if (len == 0 || len > 8) { - *w++ = *r++; - continue; - } - if (len == 3 && strncmp(r + 1, "amp", 3) == 0) - c = '&'; - else if (len == 2 && strncmp(r + 1, "lt", 2) == 0) - c = '<'; - else if (len == 2 && strncmp(r + 1, "gt", 2) == 0) - c = '>'; - else if (len == 4 && strncmp(r + 1, "quot", 4) == 0) - c = '"'; - else if (len == 4 && strncmp(r + 1, "apos", 4) == 0) - c = '\''; - else if (r[1] == '#') { - const int hex = (r[2] == 'x' || r[2] == 'X'); - const char *p = r + (hex ? 3 : 2); - long v = 0; - - if (p < semi) { - for (; p < semi; p++) { - const int d = - hex ? (isxdigit((unsigned char) *p) - ? (isdigit((unsigned char) *p) - ? *p - '0' - : (tolower((unsigned char) *p) - 'a' + 10)) - : -1) - : (isdigit((unsigned char) *p) ? *p - '0' : -1); - - if (d < 0) { - v = -1; - break; - } - v = v * (hex ? 16 : 10) + d; - if (v > 0x7e) - break; - } - if (v >= 0x20 && v <= 0x7e) - c = (int) v; - else if (v >= 0) - ok = HTS_FALSE; /* a well-formed reference to a byte no URL holds */ - } - } - if (c < 0) { - *w++ = *r++; - } else { - *w++ = (char) c; - r = semi + 1; - } - } - } - *w = '\0'; - return ok; -} - /* Accept only an absolute http(s) URL with no space or control byte. */ static hts_boolean sitemap_url_ok(const char *url) { const char *p; @@ -165,20 +96,6 @@ static const char *sitemap_tag_end(const char *p, const char *end) { return p < end ? p + 1 : NULL; } -/* Bounded substring search: the document may hold NUL bytes. */ -static const char *sitemap_memstr(const char *p, size_t len, - const char *needle) { - const size_t nlen = strlen(needle); - - if (nlen == 0 || len < nlen) - return NULL; - for (; len >= nlen; p++, len--) { - if (*p == *needle && memcmp(p, needle, nlen) == 0) - return p; - } - return NULL; -} - /* HTS_TRUE when the document's root element is `name`. Skips the XML declaration, comments and processing instructions first, so a comment mentioning the other root element cannot decide the document type. */ @@ -195,7 +112,7 @@ static hts_boolean sitemap_root_is(const char *doc, size_t size, } else if (doc[i] != '<') { return HTS_FALSE; /* character data before any element: not XML */ } else if (i + 4 <= size && memcmp(doc + i, ""); + const char *const e = hts_memstr(doc + i, size - i, "-->", 3); if (e == NULL) return HTS_FALSE; @@ -242,7 +159,7 @@ static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) { out = malloct(cap + 1); if (out == NULL) return NULL; - n = hts_zhead(body, size, out, cap); + n = hts_codec_head(HTS_CODEC_DEFLATE, body, size, out, cap); if (n == 0) { freet(out); return NULL; @@ -266,8 +183,7 @@ int hts_sitemap_scan(const char *body, size_t size, int maxurls, if (body == NULL || size < 2 || handler == NULL) return 0; - /* A .xml.gz body arrives raw here: Content-Encoding gzip was already undone - upstream, so only the gzip container is left to peel. */ + /* Content-Encoding gzip is undone upstream; only the container is left. */ if ((unsigned char) body[0] == 0x1f && (unsigned char) body[1] == 0x8b) { unpacked = sitemap_gunzip(body, size, &size); if (unpacked == NULL) @@ -280,13 +196,12 @@ int hts_sitemap_scan(const char *body, size_t size, int maxurls, } end = doc + size; - /* The root element classifies the document; the handler reads the verdict - before the first URL, so it must be set up front. */ + /* Set before the first callback: the handler reads the verdict. */ if (is_index != NULL) *is_index = sitemap_root_is(doc, size, "sitemapindex"); for (p = doc; n < maxurls;) { - const char *loc = sitemap_memstr(p, (size_t) (end - p), " 0 && is_realspace(a[l - 1])) - a[--l] = '\0'; - } - if (!sitemap_url_ok(a)) - continue; - n++; - if (!handler(arg, a)) - break; - } - return n; -} - /* --------------------------------------------------------------------- */ /* Engine glue */ /* --------------------------------------------------------------------- */ static hts_sitemap_state *sitemap_state(httrackp *opt) { - if (opt->sitemap_state == NULL) - opt->sitemap_state = calloct(1, sizeof(hts_sitemap_state)); - return (hts_sitemap_state *) opt->sitemap_state; + if (opt->state.sitemap == NULL) + opt->state.sitemap = calloct(1, sizeof(hts_sitemap_state)); + return (hts_sitemap_state *) opt->state.sitemap; } static sitemap_doc *sitemap_find(httrackp *opt, const char *adr, const char *fil) { - hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state; + hts_sitemap_state *const st = (hts_sitemap_state *) opt->state.sitemap; sitemap_doc *d; if (st == NULL) @@ -407,21 +273,23 @@ static sitemap_doc *sitemap_find(httrackp *opt, const char *adr, return NULL; } -/* A sitemap document is fetched like any other resource, so the user's filters - and robots.txt decide whether it may be. The wizard proper is not usable - here: it wants a referring link, and its up/down travel rules would judge a - child sitemap against the parent sitemap's directory. */ +/* Who asked for this document decides how far it is gated. The wizard proper + is not usable here: it wants a referring link, and its up/down travel rules + would judge a child sitemap against the parent sitemap's directory. */ static hts_boolean sitemap_fetch_allowed(httrackp *opt, const char *adr, - const char *fil) { - char BIGSTK l[HTS_URLMAXSIZE * 2], lfull[HTS_URLMAXSIZE * 2]; - int jokdepth = 0; - - if (opt->robots && opt->robotsptr != NULL && - checkrobots((robots_wizard *) opt->robotsptr, adr, fil) == -1) { - hts_log_print(opt, LOG_NOTICE, "Sitemap: robots.txt forbids %s%s", adr, - fil); - return HTS_FALSE; - } + const char *fil, + hts_sitemap_source src) { + /* adr and fil are each capped just under HTS_URLMAXSIZE, and lfull prefixes + a scheme and a slash on top of both: 2 * HTS_URLMAXSIZE does not fit. */ + char BIGSTK l[HTS_URLMAXSIZE * 2 + 16], lfull[HTS_URLMAXSIZE * 2 + 16]; + int jokdepth = 0, jok; + + hts_boolean refused; + + /* The user naming a sitemap is the same intent as naming a start URL, which + the wizard admits unconditionally. */ + if (src == HTS_SITEMAP_SRC_USER) + return HTS_TRUE; strcpybuff(l, jump_identification_const(adr)); if (*fil != '/') strcatbuff(l, "/"); @@ -431,21 +299,32 @@ static hts_boolean sitemap_fetch_allowed(httrackp *opt, const char *adr, if (*fil != '/') strcatbuff(lfull, "/"); strcatbuff(lfull, fil); - if (fa_strjoker_dual(0, *opt->filters.filters, *opt->filters.filptr, lfull, l, - NULL, NULL, &jokdepth) == -1) { + jok = fa_strjoker_dual(0, *opt->filters.filters, *opt->filters.filptr, lfull, + l, NULL, NULL, &jokdepth); + refused = (jok == -1) ? HTS_TRUE : HTS_FALSE; + if (refused) { hts_log_print(opt, LOG_NOTICE, "Sitemap: filter rule #%d refuses %s%s", jokdepth + 1, adr, fil); return HTS_FALSE; } + /* A Sitemap: line, or a sitemapindex entry, is the site inviting the fetch; + a Disallow elsewhere in the same file does not retract it. The well-known + location is only ever a guess, so there a Disallow wins. */ + if (src == HTS_SITEMAP_SRC_GUESSED && + hts_robots_forbids(opt, adr, fil, (jok != 0) ? HTS_TRUE : HTS_FALSE, + refused)) { + hts_log_print(opt, LOG_NOTICE, "Sitemap: robots.txt forbids %s%s", adr, + fil); + return HTS_FALSE; + } return HTS_TRUE; } -/* Queue a document and record its link with save="" so the body stays in - memory: a sitemap is ingested, never mirrored. Top priority so its URLs get - the full depth budget through htsAddLink. */ +/* Record the link with save="" so the body stays in memory: a sitemap is + ingested, never mirrored. */ static hts_boolean sitemap_queue_(httrackp *opt, const char *adr, const char *fil, int level, - hts_boolean is_robots, hts_boolean link_it) { + hts_sitemap_source src, hts_boolean link_it) { hts_sitemap_state *const st = sitemap_state(opt); sitemap_doc *d; @@ -460,9 +339,7 @@ static hts_boolean sitemap_queue_(httrackp *opt, const char *adr, return HTS_FALSE; if (sitemap_find(opt, adr, fil) != NULL) return HTS_FALSE; - /* The robots.txt probe is the request that fetches the rules, so it cannot be - judged by them; everything else can. */ - if (!is_robots && !sitemap_fetch_allowed(opt, adr, fil)) + if (!sitemap_fetch_allowed(opt, adr, fil, src)) return HTS_FALSE; d = calloct(1, sizeof(sitemap_doc)); if (d == NULL) @@ -470,7 +347,7 @@ static hts_boolean sitemap_queue_(httrackp *opt, const char *adr, strcpybuff(d->adr, adr); strcpybuff(d->fil, fil); d->level = level; - d->is_robots = is_robots; + d->src = src; d->next = st->docs; st->docs = d; st->ndocs++; @@ -492,8 +369,8 @@ static hts_boolean sitemap_queue_(httrackp *opt, const char *adr, static hts_boolean sitemap_queue(httrackp *opt, const char *adr, const char *fil, int level, - hts_boolean is_robots) { - return sitemap_queue_(opt, adr, fil, level, is_robots, HTS_TRUE); + hts_sitemap_source src) { + return sitemap_queue_(opt, adr, fil, level, src, HTS_TRUE); } void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil, @@ -504,7 +381,7 @@ void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil, return; d->done = HTS_TRUE; /* the body lives at the target now */ /* The engine already queued the target link, so only the marking moves. */ - (void) sitemap_queue_(opt, newadr, newfil, d->level, d->is_robots, HTS_FALSE); + (void) sitemap_queue_(opt, newadr, newfil, d->level, d->src, HTS_FALSE); hts_log_print(opt, LOG_NOTICE, "Sitemap: %s%s redirects to %s%s", adr, fil, newadr, newfil); } @@ -521,24 +398,81 @@ void hts_sitemap_seed(httrackp *opt, const char *starturl) { if (strstr(url, ":/") == NULL) hts_log_print(opt, LOG_ERROR, "Sitemap URL must be absolute: %s", url); else if (ident_url_absolute(url, &af) >= 0) - (void) sitemap_queue(opt, af.adr, af.fil, 0, HTS_FALSE); + (void) sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_USER); } } - /* --sitemap: probe the start host's robots.txt, whose Sitemap: lines decide - whether the /sitemap.xml fallback is needed. */ - if (!opt->sitemap || starturl == NULL || starturl[0] == '\0' || + if (starturl == NULL || starturl[0] == '\0' || strlen(starturl) >= sizeof(url)) return; strcpybuff(url, starturl); if (ident_url_absolute(url, &af) < 0) return; - if (sitemap_queue(opt, af.adr, "/robots.txt", 0, HTS_TRUE)) { + { + hts_sitemap_state *const st = sitemap_state(opt); + + if (st != NULL && strlen(af.adr) < sizeof(st->anchor_adr) && + strlen(af.fil) < sizeof(st->anchor_fil)) { + strcpybuff(st->anchor_adr, af.adr); + strcpybuff(st->anchor_fil, af.fil); + } + } + if (!opt->sitemap) + return; + /* Answered in hts_sitemap_robots, once the parsed rules are installed. */ + if (hts_record_link(opt, af.adr, "/robots.txt", "", "", "", NULL)) { + heap_top()->testmode = 0; + heap_top()->link_import = 0; + heap_top()->depth = 0; + heap_top()->pass2 = 0; + heap_top()->retry = opt->retry; + heap_top()->premier = heap_top_index(); + heap_top()->precedent = heap_top_index(); /* Claim the host so the parser does not queue robots.txt a second time. */ if (opt->robotsptr != NULL) (void) checkrobots_set((robots_wizard *) opt->robotsptr, af.adr, ""); } } +void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps) { + hts_sitemap_state *const st = (hts_sitemap_state *) opt->state.sitemap; + int queued = 0; + + if (st == NULL || !opt->sitemap || st->probe_done || + !strfield2(st->anchor_adr, adr)) + return; + st->probe_done = HTS_TRUE; + if (sitemaps != NULL) { + const char *p = sitemaps; + + while (*p != '\0') { + const char *const eol = strchr(p, '\n'); + const size_t len = eol != NULL ? (size_t) (eol - p) : strlen(p); + char BIGSTK line[HTS_URLMAXSIZE]; + lien_adrfil af; + + if (len > 0 && len < sizeof(line)) { + memcpy(line, p, len); + line[len] = '\0'; + /* Same host: a Sitemap: line must not aim the fetcher elsewhere. */ + if (sitemap_url_ok(line) && ident_url_absolute(line, &af) >= 0 && + strfield2(af.adr, adr) && + sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_DECLARED)) + queued++; + } + if (eol == NULL) + break; + p = eol + 1; + } + } + if (queued == 0 && !st->fallback_done) { + st->fallback_done = HTS_TRUE; + if (sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_SITEMAP_SRC_GUESSED)) + queued++; + } + hts_log_print(opt, LOG_NOTICE, "Sitemap: %d sitemap(s) queued for %s", queued, + adr); +} + hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr, const char *fil) { const sitemap_doc *const d = sitemap_find(opt, adr, fil); @@ -556,32 +490,38 @@ typedef struct sitemap_ingest_ctx { int accepted; /* URLs seeded or documents queued, not merely parsed */ } sitemap_ingest_ctx; -/* A of a : hand it to the wizard as a top-level seed. */ +/* A of a : hand it to the wizard as a top-level seed. + The wizard is pointed at the crawl's own start URL, not at the sitemap: the + site picks where its sitemap lives, so anchoring travel there would let a + root sitemap widen a subtree crawl to the whole host. The URL then becomes + its own anchor, exactly as a command-line seed does. */ static hts_boolean sitemap_seed_url(void *arg, const char *url) { sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg; - hts_sitemap_state *const st = sitemap_state(c->opt); + httrackp *const opt = c->opt; + hts_sitemap_state *const st = sitemap_state(opt); char BIGSTK buff[HTS_URLMAXSIZE]; + int before; if (st == NULL || st->nurls >= HTS_SITEMAP_MAX_URLS_TOTAL) { - hts_log_print(c->opt, LOG_WARNING, + hts_log_print(opt, LOG_WARNING, "Sitemap: URL cap reached, ignoring the rest"); return HTS_FALSE; } - /* Both scanners bound the URL below this, but strcpybuff aborts rather than - truncating, so never let hostile input reach it unchecked. */ + /* strcpybuff aborts rather than truncating: never feed it unchecked input. */ if (strlen(url) >= sizeof(buff)) return HTS_TRUE; st->nurls++; strcpybuff(buff, url); - /* htsAddLink reports the wizard's verdict; only count what it took. */ + before = opt->lien_tot; if (htsAddLink(c->str, buff)) c->accepted++; + if (opt->lien_tot > before) + heap_top()->premier = heap_top_index(); /* a seed anchors on itself */ return HTS_TRUE; } -/* A of a , or a robots.txt Sitemap: line. Cross-host - children are dropped: a hostile sitemap must not aim the fetcher elsewhere. - */ +/* A of a : cross-host children are dropped, so a hostile + sitemap cannot aim the fetcher elsewhere. */ static hts_boolean sitemap_seed_child(void *arg, const char *url) { sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg; char BIGSTK buff[HTS_URLMAXSIZE]; @@ -598,13 +538,13 @@ static hts_boolean sitemap_seed_child(void *arg, const char *url) { af.fil); return HTS_TRUE; } - if (sitemap_queue(c->opt, af.adr, af.fil, c->level + 1, HTS_FALSE)) + if (sitemap_queue(c->opt, af.adr, af.fil, c->level + 1, + HTS_SITEMAP_SRC_DECLARED)) c->accepted++; return HTS_TRUE; } -/* hts_sitemap_scan classifies the document before the first callback, so the - urlset/sitemapindex split can be decided here. */ +/* The scan classifies the document before the first callback. */ static hts_boolean sitemap_seed_any(void *arg, const char *url) { sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg; @@ -615,12 +555,26 @@ static hts_boolean sitemap_seed_any(void *arg, const char *url) { void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, const char *fil, const char *body, size_t size) { sitemap_doc *const d = sitemap_find(opt, adr, fil); + hts_sitemap_state *const st = (hts_sitemap_state *) opt->state.sitemap; sitemap_ingest_ctx ctx; - int n; + int n, anchor, saved_depth; if (d == NULL || d->done) return; d->done = HTS_TRUE; + /* str->ptr_ is a scratch int owned by the caller, so nothing else moves. */ + anchor = *str->ptr_; + if (st != NULL && st->anchor_adr[0] != '\0' && opt->hash != NULL) { + const int i = hash_read((const hash_struct *) opt->hash, st->anchor_adr, + st->anchor_fil, 1); + + if (i >= 0) + anchor = i; + } + *str->ptr_ = anchor; + /* Borrow the anchor's position but keep a seed's full depth budget. */ + saved_depth = heap(anchor)->depth; + heap(anchor)->depth = opt->depth + 1; ctx.opt = opt; ctx.str = str; ctx.adr = adr; @@ -628,27 +582,9 @@ void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, ctx.is_index = HTS_FALSE; ctx.accepted = 0; - if (d->is_robots) { - hts_sitemap_state *const st = sitemap_state(opt); - - n = body != NULL ? hts_sitemap_scan_robots(body, size, HTS_SITEMAP_MAX_DOCS, - sitemap_seed_child, &ctx) - : 0; - /* Fall back to the well-known location when robots.txt queued nothing: a - lone off-host or filtered Sitemap: line leaves us with no sitemap at all. - */ - if (ctx.accepted == 0 && st != NULL && !st->fallback_done) { - st->fallback_done = HTS_TRUE; - (void) sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_FALSE); - } - hts_log_print(opt, LOG_INFO, - "Sitemap: %d of %d sitemap(s) declared in %s%s", ctx.accepted, - n, adr, fil); - return; - } - n = hts_sitemap_scan(body, size, HTS_SITEMAP_MAX_URLS_DOC, &ctx.is_index, sitemap_seed_any, &ctx); + heap(anchor)->depth = saved_depth; if (n < 0) { hts_log_print(opt, LOG_ERROR, "Sitemap: could not decompress %s%s", adr, fil); @@ -664,7 +600,7 @@ void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr, } void hts_sitemap_free(httrackp *opt) { - hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state; + hts_sitemap_state *const st = (hts_sitemap_state *) opt->state.sitemap; if (st == NULL) return; @@ -674,6 +610,6 @@ void hts_sitemap_free(httrackp *opt) { freet(st->docs); st->docs = next; } - freet(opt->sitemap_state); - opt->sitemap_state = NULL; + freet(opt->state.sitemap); + opt->state.sitemap = NULL; } diff --git a/src/htssitemap.h b/src/htssitemap.h index 602e16bac..3ffc0f6a8 100644 --- a/src/htssitemap.h +++ b/src/htssitemap.h @@ -50,6 +50,15 @@ extern "C" { #define HTS_SITEMAP_MAX_LEVEL 4 /* sitemapindex nesting */ #define HTS_SITEMAP_MAX_BYTES (64 * 1024 * 1024) /* decompressed document */ +/* Who asked for a sitemap document, which decides how far its fetch is gated. + The user naming one is the same intent as a start URL; a site declaring one + invites the fetch; the well-known location is only ever our guess. */ +typedef enum { + HTS_SITEMAP_SRC_USER, /**< --sitemap-url */ + HTS_SITEMAP_SRC_DECLARED, /**< a Sitemap: line or a sitemapindex entry */ + HTS_SITEMAP_SRC_GUESSED /**< the /sitemap.xml fallback */ +} hts_sitemap_source; + /* Per-URL handler; returning HTS_FALSE stops the scan. */ typedef hts_boolean (*hts_sitemap_handler)(void *arg, const char *url); @@ -62,11 +71,6 @@ int hts_sitemap_scan(const char *body, size_t size, int maxurls, hts_boolean *is_index, hts_sitemap_handler handler, void *arg); -/* Same, over a robots.txt body: hands out the "Sitemap:" URLs (a - group-independent record in RFC 9309, so no user-agent grouping). */ -int hts_sitemap_scan_robots(const char *body, size_t size, int maxurls, - hts_sitemap_handler handler, void *arg); - /* --- Engine glue (needs a live httrackp). --- */ /* Queue the first sitemap document of the mirror: the explicit --sitemap-url, @@ -74,6 +78,11 @@ int hts_sitemap_scan_robots(const char *body, size_t size, int maxurls, command-line seed. No-op when neither option is set. */ void hts_sitemap_seed(httrackp *opt, const char *starturl); +/* Act on the start host's robots.txt once its rules are installed: queue the + Sitemap: URLs it names (newline-separated, from robots_parse), or the + well-known /sitemap.xml when it names none. No-op unless --sitemap. */ +void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps); + /* Carry the sitemap marking of (adr,fil) over to the target of a redirect the engine has already queued, so a moved sitemap is still ingested. */ void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil, diff --git a/src/htstools.c b/src/htstools.c index 893d0e56a..b4ea38590 100644 --- a/src/htstools.c +++ b/src/htstools.c @@ -276,6 +276,21 @@ int ident_url_relatif(const char *lien, const char *origin_adr, return ok; } +/* Bounded substring search: bodies and archive records carry NUL bytes, so + strstr() would stop at the first one. */ +const char *hts_memstr(const char *hay, size_t haylen, const char *needle, + size_t nlen) { + size_t i; + + if (nlen == 0 || haylen < nlen) + return NULL; + for (i = 0; i + nlen <= haylen; i++) { + if (hay[i] == *needle && memcmp(hay + i, needle, nlen) == 0) + return hay + i; + } + return NULL; +} + // créer dans s, à partir du chemin courant curr_fil, le lien vers link (absolu) // un ident_url_relatif a déja été fait avant, pour que link ne soit pas un chemin relatif int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) { @@ -308,9 +323,8 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) { // copy only the current path curr = _curr; strlcpybuff(curr, curr_fil, sizeof(_curr)); - if ((a = strchr(curr, '?')) == NULL) { // couper au ? (params) - // an empty path has no last character to walk back from: curr-1 reads - // before the buffer + if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters) + // an empty path has no last character: curr-1 would read before the buffer a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr; } while((*a != '/') && (a > curr)) diff --git a/src/htstools.h b/src/htstools.h index c0a57d2a4..7a6e18495 100644 --- a/src/htstools.h +++ b/src/htstools.h @@ -61,6 +61,11 @@ typedef struct lien_adrfilsave lien_adrfilsave; int ident_url_relatif(const char *lien, const char *origin_adr, const char *origin_fil, lien_adrfil* const adrfil); +/* Bounded substring search over data that may hold NUL bytes; NULL if absent. + */ +const char *hts_memstr(const char *hay, size_t haylen, const char *needle, + size_t nlen); + int lienrelatif(char *s, size_t ssize, const char *link, const char *curr); int link_has_authority(const char *lien); int link_has_authorization(const char *lien); diff --git a/src/htswizard.c b/src/htswizard.c index 30bb37697..2925acace 100644 --- a/src/htswizard.c +++ b/src/htswizard.c @@ -151,6 +151,23 @@ static hts_boolean is_embed_pair(const htspair_t *table, const char *tag, return HTS_FALSE; } +/* The engine's robots.txt verdict for (adr,fil). Under HTS_ROBOTS_SOMETIMES an + explicit filter acceptance overrides the ban, which is why the filter outcome + is an input; the sitemap fetcher asks the same question outside the wizard. + */ +hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil, + hts_boolean filters_decided, + hts_boolean filters_refused) { + if (!opt->robots || opt->robotsptr == NULL) + return HTS_FALSE; + if (checkrobots((robots_wizard *) opt->robotsptr, adr, fil) != -1) + return HTS_FALSE; + if (filters_decided && !filters_refused && + opt->robots == HTS_ROBOTS_SOMETIMES) + return HTS_FALSE; + return HTS_TRUE; +} + static int hts_acceptlink_(httrackp * opt, int ptr, const char *adr, const char *fil, const char *tag, const char *attribute, int *set_prio_to, @@ -576,30 +593,26 @@ static int hts_acceptlink_(httrackp * opt, int ptr, } } // vérifier robots.txt - if (opt->robots) { - int r = checkrobots(_ROBOTS, adr, fil); - - if (r == -1) { // interdiction + if (opt->robots && checkrobots(_ROBOTS, adr, fil) == -1) { #if DEBUG_ROBOTS - printf("robots.txt forbidden: %s%s\n", adr, fil); + printf("robots.txt forbidden: %s%s\n", adr, fil); #endif - // question résolue, par les filtres, et mode robot non strict - if ((!question) && (filters_answer) && - (opt->robots == HTS_ROBOTS_SOMETIMES) && (forbidden_url != 1)) { - r = 0; // annuler interdiction des robots - if (!forbidden_url) { - hts_log_print(opt, LOG_DEBUG, - "Warning link followed against robots.txt: link %s at %s%s", - l, adr, fil); - } - } - if (r == -1) { // interdire - forbidden_url = 1; - question = 0; - hts_log_print(opt, LOG_DEBUG, - "(robots.txt) forbidden link: link %s at %s%s", l, adr, - fil); + if (!hts_robots_forbids(opt, adr, fil, + (!question && filters_answer) ? HTS_TRUE + : HTS_FALSE, + (forbidden_url == 1) ? HTS_TRUE : HTS_FALSE)) { + if (!forbidden_url) { + hts_log_print( + opt, LOG_DEBUG, + "Warning link followed against robots.txt: link %s at %s%s", l, + adr, fil); } + } else { + forbidden_url = 1; + question = 0; + hts_log_print(opt, LOG_DEBUG, + "(robots.txt) forbidden link: link %s at %s%s", l, adr, + fil); } } diff --git a/src/htswizard.h b/src/htswizard.h index 5a65ef9e6..0485c4c68 100644 --- a/src/htswizard.h +++ b/src/htswizard.h @@ -49,6 +49,13 @@ typedef struct httrackp httrackp; typedef struct lien_url lien_url; #endif +/* The engine's robots.txt verdict for (adr,fil): HTS_TRUE when the fetch is + forbidden. `filters_decided`/`filters_refused` carry the filter outcome, + which overrides a ban under -s1 (HTS_ROBOTS_SOMETIMES). */ +hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil, + hts_boolean filters_decided, + hts_boolean filters_refused); + int hts_acceptlink(httrackp * opt, int ptr, const char *adr, const char *fil, const char *tag, const char *attribute, diff --git a/tests/87_local-sitemap.test b/tests/87_local-sitemap.test index 184a2e370..c6461d80d 100644 --- a/tests/87_local-sitemap.test +++ b/tests/87_local-sitemap.test @@ -23,7 +23,6 @@ crawl --errors 0 --rerun \ --found 'sitemapdir/deep1.html' \ --not-found 'sitemapdir/index.xml' \ --not-found 'sitemapdir/pages.xml.gz' \ - --not-found 'robots.txt' \ --log-found '2 of 3 URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \ --log-found '1 of 2 child sitemap\(s\) listed by 127\.0\.0\.1:[0-9]+/sitemapdir/index\.xml' \ --log-found 'ignoring off-host child sitemap' \ @@ -68,13 +67,50 @@ crawl --errors-content 1 \ httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 \ --sitemap-url 'BASEURL/sitemapdir/missing.xml' -# The sitemapindex nesting cap stops the chain before its deepest urlset. +# The sitemapindex nesting cap stops the chain before its deepest urlset. The +# positive control is the page one level inside the cap: without it, a cap +# mutated to 0 (nothing followed) would pass this just as well. crawl --errors 0 \ - --not-found 'sitemapdir/deepcap.html' \ + --found 'sitemapdir/cap3.html' \ + --not-found 'sitemapdir/cap4.html' \ --log-found 'Sitemap: cap reached' \ httrack 'BASEURL/sitemapdir/start.html' -r2 \ --sitemap-url 'BASEURL/sitemapdir/chain0.xml' +# A site cannot widen a subtree crawl by putting its sitemap at the root: the +# page below the start directory is seeded, the one above it is refused. +crawl --errors 0 \ + --found 'deep/dir/below.html' \ + --not-found 'elsewhere/updir.html' \ + httrack 'BASEURL/deep/dir/start.html' --sitemap -r3 -F 'scopesitemap-agent' + +# How far a sitemap fetch is gated depends on who asked for it, so the three +# cases below have to differ: one "robots is honoured" assertion would hide a +# regression in either direction. The harness disables robots, hence -s2 here. + +# We guessed /sitemap.xml, so a Disallow on it wins. +crawl --errors 0 \ + --not-found 'sitemapdir/orphan1.html' \ + --log-found 'Sitemap: robots.txt forbids' \ + httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \ + -F 'denysitemap-agent' + +# The site declared it through a Sitemap: line, which the same Disallow does +# not retract. +crawl --errors 0 \ + --found 'sitemapdir/orphan1.html' \ + --log-not-found 'Sitemap: robots.txt forbids' \ + httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \ + -F 'denydeclared-agent' + +# The user named it, which is the same intent as a start URL: not refused +# either, even though robots.txt forbids that exact path. +crawl --errors 0 \ + --found 'sitemapdir/orphan1.html' \ + --log-not-found 'Sitemap: robots.txt forbids' \ + httrack 'BASEURL/sitemapdir/start.html' -r2 -s2 -F 'denysitemap-agent' \ + --sitemap-url 'BASEURL/sitemap.xml' + # A child sitemap is a fetch like any other: a filter rejecting it stops the # whole subtree, so the page only it lists is never reached. crawl --errors 0 \ diff --git a/tests/local-server.py b/tests/local-server.py index 67413cb50..4118ae0a2 100755 --- a/tests/local-server.py +++ b/tests/local-server.py @@ -546,12 +546,32 @@ def route_gated_secret(self): # A User-Agent carrying NO_SITEMAP_UA gets a robots.txt with no Sitemap: # record, so a test can drive the /sitemap.xml fallback instead. NO_SITEMAP_UA = "nositemap" + # ... and one that additionally Disallows the well-known location, so the + # fallback has to be refused by the rules this very body carries. + DENY_SITEMAP_UA = "denysitemap" + # ... the same Disallow, but with the sitemap also declared: the + # declaration is the site inviting the fetch and must win. + DENY_DECLARED_UA = "denydeclared" + # ... and one that points the sitemap at the site root, to check a subtree + # crawl is not widened by where the site chooses to put its sitemap. + SCOPE_SITEMAP_UA = "scopesitemap" def route_robots(self): # The Sitemap: record is group-independent; only --sitemap acts on it. + ua = self.headers.get("User-Agent") or "" + host = self.headers.get("Host") body = "User-agent: *\nDisallow:\n" - if self.NO_SITEMAP_UA not in (self.headers.get("User-Agent") or ""): - body += f"Sitemap: http://{self.headers.get('Host')}/sitemapdir/index.xml\n" + if self.DENY_DECLARED_UA in ua: + body = ( + "User-agent: *\nDisallow: /sitemap.xml\n" + f"Sitemap: http://{host}/sitemap.xml\n" + ) + elif self.DENY_SITEMAP_UA in ua: + body = "User-agent: *\nDisallow: /sitemap.xml\n" + elif self.SCOPE_SITEMAP_UA in ua: + body += f"Sitemap: http://{host}/scopesitemap.xml\n" + elif self.NO_SITEMAP_UA not in ua: + body += f"Sitemap: http://{host}/sitemapdir/index.xml\n" body = body.encode() self.send_response(200) self.send_header("Content-Type", "text/plain") @@ -585,8 +605,8 @@ def route_sitemap_pages(self): '\n' '' f"http://{host}/sitemapdir/orphan1.html" - f"http://{host}/sitemapdir/orphan2.html" "http://sitemap-offhost.invalid/x.html" + f"http://{host}/sitemapdir/orphan2.html" "\n" ).encode() self.send_raw(gzip.compress(xml), "application/x-gzip") @@ -603,30 +623,40 @@ def route_sitemap_orphan2(self): def route_sitemap_deep1(self): self.send_html("\tOne level below an orphan.") - # chain0..chain5 nest one sitemapindex per level; chain6 is the urlset the - # nesting cap must keep out of reach. + # chainN is a sitemapindex at nesting level N, listing chain(N+1) and a + # urlset capN.xml whose single page is capN.html. Levels up to + # HTS_SITEMAP_MAX_LEVEL are followed, so capN.html appears for N below the + # cap and stops appearing at it: the pair pins the boundary, which a cap + # mutated either way would break. def route_sitemap_chain(self): host = self.headers.get("Host") level = int(self.path.rsplit("/", 1)[-1][len("chain") : -len(".xml")]) - if level < 6: - body = ( + self.send_raw( + ( '\n' f"http://{host}/sitemapdir/chain{level + 1}.xml" + "" + f"http://{host}/sitemapdir/cap{level}.xml" "\n" - ) - else: - body = ( + ).encode(), + "application/xml", + ) + + def route_sitemap_capset(self): + host = self.headers.get("Host") + level = self.path.rsplit("/", 1)[-1][len("cap") : -len(".xml")] + self.send_raw( + ( '\n' - f"http://{host}/sitemapdir/deepcap.html" + f"http://{host}/sitemapdir/cap{level}.html" "\n" - ) - self.send_raw(body.encode(), "application/xml") + ).encode(), + "application/xml", + ) - def route_sitemap_deepcap(self): - self.send_html("\tReachable only past the sitemapindex nesting cap.") + def route_sitemap_cappage(self): + self.send_html("\tReached through a nested sitemapindex.") - # An index naming a child that filters (or robots.txt) must be able to - # refuse: filtered.xml would list gated.html if it were ever fetched. def route_sitemap_gatedindex(self): host = self.headers.get("Host") self.send_raw( @@ -649,6 +679,27 @@ def route_sitemap_gated(self): self.send_html("\tListed only by the filtered child sitemap.") # A moved sitemap: the marking has to follow the redirect. + # A root sitemap naming a page below the crawl's start directory and one + # above it. Only the first may be seeded when the crawl started at /deep/dir/. + def route_sitemap_scope(self): + host = self.headers.get("Host") + self.send_raw( + '' + f"http://{host}/deep/dir/below.html" + f"http://{host}/elsewhere/updir.html" + "\n".encode(), + "application/xml", + ) + + def route_sitemap_deepstart(self): + self.send_html("\tA start page in a subdirectory, linking nothing.") + + def route_sitemap_below(self): + self.send_html("\tBelow the start directory.") + + def route_sitemap_updir(self): + self.send_html("\tAbove the start directory.") + def route_sitemap_moved(self): self.send_response(301) self.send_header("Location", "/sitemapdir/pages.xml.gz") @@ -1689,11 +1740,14 @@ def route_maxrecv_404(self): "/sitemapdir/orphan1.html": route_sitemap_orphan1, "/sitemapdir/orphan2.html": route_sitemap_orphan2, "/sitemapdir/deep1.html": route_sitemap_deep1, - "/sitemapdir/deepcap.html": route_sitemap_deepcap, "/sitemapdir/gatedindex.xml": route_sitemap_gatedindex, "/sitemapdir/filtered.xml": route_sitemap_filtered, "/sitemapdir/gated.html": route_sitemap_gated, "/sitemapdir/moved.xml": route_sitemap_moved, + "/scopesitemap.xml": route_sitemap_scope, + "/deep/dir/start.html": route_sitemap_deepstart, + "/deep/dir/below.html": route_sitemap_below, + "/elsewhere/updir.html": route_sitemap_updir, "/warcgz/index.html": route_warcgz_index, "/warcgz/page.html": route_warcgz_page, "/warcgz/data.bin": route_warcgz_data, @@ -2018,8 +2072,13 @@ def dispatch(self): return True # Match percent-encoded paths (accented #157 route) by their decoded form. handler = self.ROUTES.get(path) or self.ROUTES.get(unquote(path)) - if handler is None and re.fullmatch(r"/sitemapdir/chain\d+\.xml", path): - handler = type(self).route_sitemap_chain + if handler is None: + if re.fullmatch(r"/sitemapdir/chain\d+\.xml", path): + handler = type(self).route_sitemap_chain + elif re.fullmatch(r"/sitemapdir/cap\d+\.xml", path): + handler = type(self).route_sitemap_capset + elif re.fullmatch(r"/sitemapdir/cap\d+\.html", path): + handler = type(self).route_sitemap_cappage if handler is not None: handler(self) return True From 028ac8b5ad6c39b87238649ed47dc28f558d496a Mon Sep 17 00:00:00 2001 From: Xavier Roche Date: Sun, 26 Jul 2026 19:21:06 +0200 Subject: [PATCH 07/14] sitemap: add the fuzz harness the parser was missing, and drop truncated Sitemap: lines The file header called the scanner fuzzable while fuzz/ registered ten harnesses and none for it. fuzz-sitemap feeds it raw XML, gzip-framed bodies and truncated streams off a heap copy of exactly the input size, so an overread is an ASan report rather than a quiet pass, with a four-file seed corpus. 60000 runs clean under ASan+UBSan. robots_parse now drops a Sitemap: line that filled its scratch buffer instead of handing on the half URL it was truncated to. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Xavier Roche --- fuzz/Makefile.am | 7 +- .../00a99f50f02b4dacea6d0089384b65e940b46cac | 1 + .../02f9cead3211d007e4476dca0542ba0642d9215f | Bin 0 -> 6 bytes .../05d392d686a8affc917e9f4c0006470fc9ae10e2 | Bin 0 -> 79 bytes .../060da1aaa324173a3992d401de50ba8a82dca8f6 | 1 + .../0762f2f322356310a6a1415923a8638342ccbab7 | Bin 0 -> 120 bytes .../0840f9901450f24841c81fda32af0830d06805fa | 1 + .../0a3c091488367a3446b8707090ee0ff653b16fe0 | 1 + .../0acdb463ec3964e5a01dbc3b570dfdec8cd40e1e | Bin 0 -> 37 bytes .../0af80f676f9e204f697caee48ab3180ede03ea56 | 1 + .../0c18b306e933712fe4c6bed04ee0962c82201fe6 | 1 + .../0c4a15a3173d6d5292dc4e0104bcca72cc4db27e | Bin 0 -> 48 bytes .../0caed9d0f4542bf2c1bafb49a842a34f09bfcbef | Bin 0 -> 18 bytes .../0cbd954d0871608e3a42043dc0265395dff86c9b | Bin 0 -> 54 bytes .../0d56488006e785281ef1ea0a4cd878cc10cf36de | 1 + .../0de18afb696bf7b4ff8efa5c67a9e0e775344b79 | 1 + .../0e00256a636924a1350f5f6353e954715b5094fb | Bin 0 -> 33 bytes .../0efcc232871dd78f5b8f57944e9b66e77c742246 | Bin 0 -> 131 bytes .../0ff8683336daaa98c6fb44b36d256fc59e22e1ac | 1 + .../10e7c3c57c64025428298b8c163879322f212d8f | 1 + .../119c566502513de0981ae5400b315d02fce93311 | 1 + .../11f6ad8ec52a2984abaafd7c3b516503785c2072 | 1 + .../136825aeb153d97b2ac3f664136dca068e2cdbf3 | 1 + .../13f68f78d292b78efac3f93369af9b7fa67e671c | Bin 0 -> 62 bytes .../143b943867717061de12ae64372cd97a5427e976 | 1 + .../1683f30ee69c20212771f0977158957bc51d06be | 1 + .../16cf9bebe58d0fee1a6c413a08ddf3b39c9974e2 | Bin 0 -> 118 bytes .../1774568b8fe763552d59661d139b3c1c32886b5a | Bin 0 -> 58 bytes .../17cd699ea8085eabee01b683a8abfa32b6f6bc22 | 1 + .../1950639e93b0e273ebfb2b5f47763966685c23fe | 1 + .../1a5d2a45772c53d803def107b67366db0b8e20af | 1 + .../1c27c818f25065e22838264a9a36244b6f551501 | Bin 0 -> 60 bytes .../1cfc2a832330111f356dad19bb5732ce1f1064be | 1 + .../1d35055a95f4093097adb16b83518f50d8a65326 | Bin 0 -> 94 bytes .../1da39276d0e9f4b91b880eee52ab1254fcc957e0 | Bin 0 -> 79 bytes .../2099ce0da61af1b84bdee76acdd62fd2ced42d80 | 2 + .../217aea3e505eb01e6aa3cd317fd53d1d25295228 | 1 + .../22175973e65fa52913d8beaf7b6f8e351fde0ff9 | 1 + .../22a628e7d0e55e1b90500087ef00a14d113594be | Bin 0 -> 31 bytes .../22d12b8efb1cd66561fc123953d9af9ea3b36d19 | Bin 0 -> 129 bytes .../231c83e2cff6c004a7ebbeb56f8020be6f6b9d38 | 1 + .../2342c6bd446e3561a5fca62c1394ac0ed30f7212 | 1 + .../246e6f26840d821990ae19d45fde49c40a6f43e2 | 1 + .../273c10d28ee0022f1ef9e438373c5427991d2c32 | 1 + .../2cdbfcee6990a36146fe6d1a21c7e7184509e4aa | Bin 0 -> 62 bytes .../2e1136584124459e506f747223eacb12e12d4281 | Bin 0 -> 47 bytes .../31fdb7b4526b8161831df86c8a95226c581ff472 | 1 + .../336abd1a87047f61c40dccaf4ad1df4156ec930c | Bin 0 -> 98 bytes .../3535f99e3b54d0ba2284fbcc0759c91664989bce | 1 + .../36a2f4986e72271a3c2e024a78457f25a8d08b45 | 1 + .../36f83a6227244c2f4b8d9e37babcc5c1aaac026d | Bin 0 -> 68 bytes .../38b09ca3dcb5be711a2e40fe957d2349422f1eb0 | 1 + .../3b1b6083420bd7e67cd2300be6ae2c31a56e132a | 1 + .../3e16c0c23128584c7a6e9ffdefbe40a8795cf428 | 1 + .../3e284e0c7aaebfd6e411fe73bcd68c8800d34839 | Bin 0 -> 46 bytes .../3e3785814ac9579f350b899facab62e80d43dcdd | Bin 0 -> 55 bytes .../3e73f5237baee8133c1e086e7cd623f3b9da941c | 1 + .../3f10def7d9e4a04ecda6c41ae28f5c1117082dc7 | Bin 0 -> 101 bytes .../3feeb4edb06a3ed045f6024a0552dff46b9b2d57 | 1 + .../468a8ca4d7592a3a543f01a56f2e5ffedab3a0a8 | 1 + .../47c562b8239a10e6377e1f4141645ad1ec9878fb | 1 + .../48e2a7d7f446fb8db38897441af4387e54e9a7da | 1 + .../49f5de2e50e78251229a579c2f39bedbba35e12e | Bin 0 -> 82 bytes .../4cfc6f83573a5c40b77ec7a89f6c022549e99108 | Bin 0 -> 68 bytes .../4d9c68140b8039b221fc77153f2484e8524cbb0f | 1 + .../4f5f0e6a0036466d373930b0b66ff37162ae4390 | Bin 0 -> 42 bytes .../50bdd2e5fe8ec578126fe54af1b200b34aa1ec7b | Bin 0 -> 49 bytes .../5710370c1503e7d2f6c76d500707b08503602563 | 1 + .../5747bb2536514958c1d090dad9f52a005e2f0d00 | 1 + .../584dd51816e5b705e0b1ee01fe08146cfcdd039f | 1 + .../58995e9e3d5f5f2ad9ade175594b5c358019f144 | 1 + .../58c1f9c5ccda08ac8fb85a2e09e424951524a8db | Bin 0 -> 48 bytes .../597be90d0fda30c3def2baa7135d064827000cce | 1 + .../5a8890576de5cc596d37a68f8a96f79a3056bf3b | Bin 0 -> 22 bytes .../5b9502d842f6f3c9bcc2b79aaa68cc8cc0b873fc | 1 + .../5d4f1a50e41b5044c1a0b01042907b2238809124 | 1 + .../5e9d8fe18524df401151d7bd9ebe05d14e7b167c | Bin 0 -> 82 bytes .../6197e36e9bafff09c9a7a1d3d3a1acfe228d42bb | 1 + .../65fb54c30ea60a48cf5fe9c89c0913d9feaf81e4 | 1 + .../678a87639c0db550bf6410391fae33adeb423816 | 1 + .../67f7ea7cc3966cba35925ab2f569e37b72f20770 | 1 + .../68533b9173b5d26bdc4f7482d3720f8b338d97ec | Bin 0 -> 128 bytes .../68e2689b4a313192bbdb5a10d50950aa44ed9b8c | 1 + .../694e068d470a3d67ba85c6db6dc210d89fcf3b1f | 1 + .../69b20e91038ff742a182007e4fd0b0eea8390ede | 1 + .../6c4982d82648107e62671bcc8534d618a3796cad | 1 + .../6d66650a7220ffcfad7f0c1f4d12590e4969789a | 1 + .../6dc3d7277036e229086c50809d03ffbfca93de13 | 1 + .../6dd1cfa987e900f39c1f65ad4604e537770f2a63 | 1 + .../6e05240a3d1eafde09d3b4592609150136ee5d06 | 1 + .../7236e9de6722ef91534680f34887c625a2fe49da | 1 + .../7327e7b591e2d97b3d0fb981d245a58e2c8c3e57 | Bin 0 -> 25 bytes .../73c66b9ecc06366897053ed1d9be1fcce523f149 | Bin 0 -> 55 bytes .../74a318c0a2c4c5f2686ea8a627f5eb15ee3bd668 | 1 + .../74c1d696fde563ca5abf9e9ae1823dd9ce76c61b | Bin 0 -> 63 bytes .../755c9c71ed51fdd6b853c40cce2d8364a98ca867 | 1 + .../75d69f6d427862dbc1e6b1c2ea7009f92fc55517 | 1 + .../7749e198ec66b0723e8e1b1c0285f5d14d190269 | 1 + .../776ce1aa1e931dd4016b94a4e3c18dcf569cc855 | 1 + .../77d9b43d5bbd08ef9338cf6a8c5f1a786390bd77 | 1 + .../7969e0e43da1b80d48e157309e415aef025ffd84 | Bin 0 -> 61 bytes .../7c99aac7091b9ff1023ebf12897c42c2b5146750 | 1 + .../7d56e311f61b11792334f7d4d57ffe009596a093 | Bin 0 -> 128 bytes .../8133938bc85e7f4cff891db44b4c598b4039d0ae | 1 + .../8596c9456ec637abc8477e42308210956d002e33 | Bin 0 -> 73 bytes .../85f54f06c01b52e7af77e1ea43c13becc0176702 | Bin 0 -> 71 bytes .../863661d323853a881d3a66d0a3834e863ca0b93b | Bin 0 -> 73 bytes .../864d934d6dcd3ad14525d1a59df1d8f7a428c252 | Bin 0 -> 63 bytes .../872faf1a741133a61098cd1cc8a0898ea86248ae | Bin 0 -> 109 bytes .../89e598f54fea7059c8c28b1b9418d7c0e5a8a028 | Bin 0 -> 34 bytes .../8ac6a34d65103279df1eaaaabfb7823be2f75387 | Bin 0 -> 55 bytes .../8add9d8dc27b939c444152727c95801a76093b13 | Bin 0 -> 96 bytes .../8b0798c9bf0a8f1206a8cce5b49c48348a5acb3c | Bin 0 -> 72 bytes .../8cbd6fc35a657c15a117f5fa44b9c5475b5c244e | Bin 0 -> 42 bytes .../8ef128178bd3c31759010e9ed4641d56511a10cc | 1 + .../8f69378809f4cb3b06cd71b4310246aede68be6c | 1 + .../8f874e1efeac088e0fe991ac5fa8dda7e17b06ae | Bin 0 -> 72 bytes .../904b658055ecbe45f30526969f0b4634c2c9f641 | 1 + .../904c672c4c5e3f2a0d875264c6940798e7bcdecd | 1 + .../92d2db22c0000a2d6fa54a2cf128f92db97b9993 | 1 + .../932ff270a6a487aa020b26421534a1c12b4a94f8 | Bin 0 -> 89 bytes .../94609cd811a26c48c711415866b304e214a4e97a | 1 + .../94b3ab30db215ec5b044a7dc24fb5e69e8778659 | 1 + .../956a73fe53021543044dfcfb61173044ca7c2a17 | 2 + .../97619afecb1c73169b02116ff610543e833a1e60 | Bin 0 -> 130 bytes .../97645ac8be26bd22563d7d27fb833d964977a44a | 1 + .../9811960a0551fd86e280eb58ed6446c0ecc74fbb | 1 + .../983772efe5696d5b65d0f006808a477d9225feab | 1 + .../993c132d0386ef7b502bab4e62d26023f0bf0250 | 1 + .../99711e0eee687fb4b9305e8b700996ef49616f20 | 1 + .../9ac709a1a3dcd5f15b0d655ea2a5a84bb7685596 | Bin 0 -> 17 bytes .../9be22a3ec737f88e0e61326f822e65952a6aeb7d | 1 + .../9e7d3f551e039e680769c619f850085179ffef3c | 1 + .../9fbbd38632a5e00322c3c1182b5de821d96d01c4 | 2 + .../a6ca50dcffc4ceb4fde9cb48683e6e6179bc9d17 | Bin 0 -> 67 bytes .../a70841b5461831653714d0086967ba0ee158ae0c | 1 + .../a79b49ba5a2b994879a6040a97910a9837d076da | 1 + .../ab4b1bb206a56cd96dc06daf7782f1880a6cac6e | 1 + .../ae028d22587dc0244e4798b26e2a2bf13167347f | Bin 0 -> 52 bytes .../b4967ae700f5af942180d72ee02b24bf294840dd | Bin 0 -> 52 bytes .../b64f8460e1fad6df6b9e663fa4e60e10db2c9bca | Bin 0 -> 131 bytes .../b721ef14aa0f27fb82d7f1db3020ceaa87377f6c | 1 + .../b8394acb26ab461576b225188103d0dedc45457b | 1 + .../bab6f59b7df9cac3ba92122d377d495c0659e215 | 1 + .../bac8a95096d48861e5ad182e63a9900eb15ade81 | 1 + .../bb603cec6991c99bba03265550e773e92e130f6e | 1 + .../be0f77cd79e4effa6060839f02f1f086dbec05c3 | Bin 0 -> 39 bytes .../c02c945097f3529771b1e4940af36e9bfc40c066 | Bin 0 -> 126 bytes .../c26c7a6dd511884da2655f00dcb5b796170e0526 | 1 + .../c41d6ab840b6a73aabd37e77a511f48f79fee4f6 | 1 + .../c6656a3f7e241754587f35ab64975d630c8f1b16 | Bin 0 -> 85 bytes .../c7fd6658ea9a3951d5f3f66d0eea6455a32a2751 | Bin 0 -> 91 bytes .../c8e34963df67be9d69f5455703f59808c385994b | 1 + .../c8e7d3426661a572c544513d07ccf7a165bb074c | Bin 0 -> 68 bytes .../cec49314e1cef1adfcab5ff6cf3024f5b8962ce4 | Bin 0 -> 62 bytes .../d08fd04e86bce0ee91539aa4e3f88e4857797a45 | 1 + .../d229c3ccf5655abdf7dd507a5423d82c29f958b6 | 1 + .../d2b67f32adc4dfae849a928bec97eba3fda7ba63 | 1 + .../d3c84d21ece3614327c53f6e4f3984870b1b9542 | 1 + .../d4c1d2ac1eabd38c5b340c5ce1698306de40f8c4 | Bin 0 -> 34 bytes .../d9fadcc7f4fae9b160d860ce1acdb21d77aab90b | 1 + .../dac4463bd89dab0ec0879cb6a6f250bec1603ca0 | Bin 0 -> 30 bytes .../dc5a07c652023cfd6c5b2be5503168d5b67a34f7 | 1 + .../dd3778d3276b51e63e8180499e39621a4300a3f6 | 1 + .../dd5fba84018693317bdfe544eeb8d880499c0aa6 | 1 + .../e35ba049599035ebc4c0d46396b63991e9fcd9bd | Bin 0 -> 61 bytes .../e5feb1ad3e6d18f9e10d3475eaadc462633ba30d | 1 + .../e76101ae92d1a510293cd56283804b33783b3a11 | Bin 0 -> 81 bytes .../e76bd0139a9d24b6575a83b6154e1c4a5f89e839 | 1 + .../e8eb9faa5d366c5bd059b1ba22c5fe8cb54ac36b | 1 + .../e9318e9449f610242dbc7ee871974fe1ea933a19 | 1 + .../eccbd84d02c3aeaac60a42ce6c5bad7971db0bf4 | Bin 0 -> 58 bytes .../eda0a1789a40de76a6d4474dbc76f218050ce8b3 | 1 + .../edcf1376941c09176dda0d8b134a8ce9e6e686f2 | 1 + .../ee5c05abfc504dda713cd9368fd6b52538873afc | 1 + .../ef40dcbe031bad1543d0e39ded75d2d503df40b3 | 1 + .../efc62b91a0bd15d95ba261d6bda1e4ee7760cb01 | Bin 0 -> 97 bytes .../f1f0d28d3fedc79f2a302920fa4da7d75ddc8549 | Bin 0 -> 52 bytes .../f25fee6de9c4c840c4e15c80d7cc7165d16964ed | 1 + .../f36385c58a6f76373d071f0050edb338b94ffc41 | 1 + .../f547b6e16cb92fe2408fdeeccea8fde8eba3646a | Bin 0 -> 72 bytes .../f5794f9cf1cf371539ad3bec5561eaf6516b261d | Bin 0 -> 107 bytes .../f620966e26b1267099535a59df7f443db7b3064b | Bin 0 -> 41 bytes .../f82b077bd6e07d706117e02d3872bb881f606493 | 1 + .../f8ba6bbd60fe059c383e42d59b1c0ec054c8ebe5 | 1 + .../f8d1d2ddaf0ba19b56b5acc75d3f2e2abe509c5b | Bin 0 -> 132 bytes .../f96b23bfa34bae6b2344000d4b9feae5a054409e | 1 + .../fbbf0b771191ed9ca496abfaca123a2fdf80fe8c | 1 + .../fbdf3c734f46c16e43038166cd2f05a5162ded27 | 1 + .../fd6ba10987b71497231ab5d7f51618caf600227e | Bin 0 -> 120 bytes .../fdc9015b3be9b81bd64fde25e6a57af1808564cc | Bin 0 -> 31 bytes .../fe92867179fa26acfdbc620381505332e7815b13 | 1 + fuzz/corpus/sitemap/sitemapindex.xml | 1 + fuzz/corpus/sitemap/truncated.xml | 1 + fuzz/corpus/sitemap/urlset.xml | 1 + fuzz/corpus/sitemap/urlset.xml.gz | Bin 0 -> 69 bytes fuzz/fuzz-sitemap.c | 60 ++++++++++++++++++ src/htsrobots.c | 4 +- 198 files changed, 191 insertions(+), 3 deletions(-) create mode 100644 fuzz/corpus/sitemap/00a99f50f02b4dacea6d0089384b65e940b46cac create mode 100644 fuzz/corpus/sitemap/02f9cead3211d007e4476dca0542ba0642d9215f create mode 100644 fuzz/corpus/sitemap/05d392d686a8affc917e9f4c0006470fc9ae10e2 create mode 100644 fuzz/corpus/sitemap/060da1aaa324173a3992d401de50ba8a82dca8f6 create mode 100644 fuzz/corpus/sitemap/0762f2f322356310a6a1415923a8638342ccbab7 create mode 100644 fuzz/corpus/sitemap/0840f9901450f24841c81fda32af0830d06805fa create mode 100644 fuzz/corpus/sitemap/0a3c091488367a3446b8707090ee0ff653b16fe0 create mode 100644 fuzz/corpus/sitemap/0acdb463ec3964e5a01dbc3b570dfdec8cd40e1e create mode 100644 fuzz/corpus/sitemap/0af80f676f9e204f697caee48ab3180ede03ea56 create mode 100644 fuzz/corpus/sitemap/0c18b306e933712fe4c6bed04ee0962c82201fe6 create mode 100644 fuzz/corpus/sitemap/0c4a15a3173d6d5292dc4e0104bcca72cc4db27e create mode 100644 fuzz/corpus/sitemap/0caed9d0f4542bf2c1bafb49a842a34f09bfcbef create mode 100644 fuzz/corpus/sitemap/0cbd954d0871608e3a42043dc0265395dff86c9b create mode 100644 fuzz/corpus/sitemap/0d56488006e785281ef1ea0a4cd878cc10cf36de create mode 100644 fuzz/corpus/sitemap/0de18afb696bf7b4ff8efa5c67a9e0e775344b79 create mode 100644 fuzz/corpus/sitemap/0e00256a636924a1350f5f6353e954715b5094fb create mode 100644 fuzz/corpus/sitemap/0efcc232871dd78f5b8f57944e9b66e77c742246 create mode 100644 fuzz/corpus/sitemap/0ff8683336daaa98c6fb44b36d256fc59e22e1ac create mode 100644 fuzz/corpus/sitemap/10e7c3c57c64025428298b8c163879322f212d8f create mode 100644 fuzz/corpus/sitemap/119c566502513de0981ae5400b315d02fce93311 create mode 100644 fuzz/corpus/sitemap/11f6ad8ec52a2984abaafd7c3b516503785c2072 create mode 100644 fuzz/corpus/sitemap/136825aeb153d97b2ac3f664136dca068e2cdbf3 create mode 100644 fuzz/corpus/sitemap/13f68f78d292b78efac3f93369af9b7fa67e671c create mode 100644 fuzz/corpus/sitemap/143b943867717061de12ae64372cd97a5427e976 create mode 100644 fuzz/corpus/sitemap/1683f30ee69c20212771f0977158957bc51d06be create mode 100644 fuzz/corpus/sitemap/16cf9bebe58d0fee1a6c413a08ddf3b39c9974e2 create mode 100644 fuzz/corpus/sitemap/1774568b8fe763552d59661d139b3c1c32886b5a create mode 100644 fuzz/corpus/sitemap/17cd699ea8085eabee01b683a8abfa32b6f6bc22 create mode 100644 fuzz/corpus/sitemap/1950639e93b0e273ebfb2b5f47763966685c23fe create mode 100644 fuzz/corpus/sitemap/1a5d2a45772c53d803def107b67366db0b8e20af create mode 100644 fuzz/corpus/sitemap/1c27c818f25065e22838264a9a36244b6f551501 create mode 100644 fuzz/corpus/sitemap/1cfc2a832330111f356dad19bb5732ce1f1064be create mode 100644 fuzz/corpus/sitemap/1d35055a95f4093097adb16b83518f50d8a65326 create mode 100644 fuzz/corpus/sitemap/1da39276d0e9f4b91b880eee52ab1254fcc957e0 create mode 100644 fuzz/corpus/sitemap/2099ce0da61af1b84bdee76acdd62fd2ced42d80 create mode 100644 fuzz/corpus/sitemap/217aea3e505eb01e6aa3cd317fd53d1d25295228 create mode 100644 fuzz/corpus/sitemap/22175973e65fa52913d8beaf7b6f8e351fde0ff9 create mode 100644 fuzz/corpus/sitemap/22a628e7d0e55e1b90500087ef00a14d113594be create mode 100644 fuzz/corpus/sitemap/22d12b8efb1cd66561fc123953d9af9ea3b36d19 create mode 100644 fuzz/corpus/sitemap/231c83e2cff6c004a7ebbeb56f8020be6f6b9d38 create mode 100644 fuzz/corpus/sitemap/2342c6bd446e3561a5fca62c1394ac0ed30f7212 create mode 100644 fuzz/corpus/sitemap/246e6f26840d821990ae19d45fde49c40a6f43e2 create mode 100644 fuzz/corpus/sitemap/273c10d28ee0022f1ef9e438373c5427991d2c32 create mode 100644 fuzz/corpus/sitemap/2cdbfcee6990a36146fe6d1a21c7e7184509e4aa create mode 100644 fuzz/corpus/sitemap/2e1136584124459e506f747223eacb12e12d4281 create mode 100644 fuzz/corpus/sitemap/31fdb7b4526b8161831df86c8a95226c581ff472 create mode 100644 fuzz/corpus/sitemap/336abd1a87047f61c40dccaf4ad1df4156ec930c create mode 100644 fuzz/corpus/sitemap/3535f99e3b54d0ba2284fbcc0759c91664989bce create mode 100644 fuzz/corpus/sitemap/36a2f4986e72271a3c2e024a78457f25a8d08b45 create mode 100644 fuzz/corpus/sitemap/36f83a6227244c2f4b8d9e37babcc5c1aaac026d create mode 100644 fuzz/corpus/sitemap/38b09ca3dcb5be711a2e40fe957d2349422f1eb0 create mode 100644 fuzz/corpus/sitemap/3b1b6083420bd7e67cd2300be6ae2c31a56e132a create mode 100644 fuzz/corpus/sitemap/3e16c0c23128584c7a6e9ffdefbe40a8795cf428 create mode 100644 fuzz/corpus/sitemap/3e284e0c7aaebfd6e411fe73bcd68c8800d34839 create mode 100644 fuzz/corpus/sitemap/3e3785814ac9579f350b899facab62e80d43dcdd create mode 100644 fuzz/corpus/sitemap/3e73f5237baee8133c1e086e7cd623f3b9da941c create mode 100644 fuzz/corpus/sitemap/3f10def7d9e4a04ecda6c41ae28f5c1117082dc7 create mode 100644 fuzz/corpus/sitemap/3feeb4edb06a3ed045f6024a0552dff46b9b2d57 create mode 100644 fuzz/corpus/sitemap/468a8ca4d7592a3a543f01a56f2e5ffedab3a0a8 create mode 100644 fuzz/corpus/sitemap/47c562b8239a10e6377e1f4141645ad1ec9878fb create mode 100644 fuzz/corpus/sitemap/48e2a7d7f446fb8db38897441af4387e54e9a7da create mode 100644 fuzz/corpus/sitemap/49f5de2e50e78251229a579c2f39bedbba35e12e create mode 100644 fuzz/corpus/sitemap/4cfc6f83573a5c40b77ec7a89f6c022549e99108 create mode 100644 fuzz/corpus/sitemap/4d9c68140b8039b221fc77153f2484e8524cbb0f create mode 100644 fuzz/corpus/sitemap/4f5f0e6a0036466d373930b0b66ff37162ae4390 create mode 100644 fuzz/corpus/sitemap/50bdd2e5fe8ec578126fe54af1b200b34aa1ec7b create mode 100644 fuzz/corpus/sitemap/5710370c1503e7d2f6c76d500707b08503602563 create mode 100644 fuzz/corpus/sitemap/5747bb2536514958c1d090dad9f52a005e2f0d00 create mode 100644 fuzz/corpus/sitemap/584dd51816e5b705e0b1ee01fe08146cfcdd039f create mode 100644 fuzz/corpus/sitemap/58995e9e3d5f5f2ad9ade175594b5c358019f144 create mode 100644 fuzz/corpus/sitemap/58c1f9c5ccda08ac8fb85a2e09e424951524a8db create mode 100644 fuzz/corpus/sitemap/597be90d0fda30c3def2baa7135d064827000cce create mode 100644 fuzz/corpus/sitemap/5a8890576de5cc596d37a68f8a96f79a3056bf3b create mode 100644 fuzz/corpus/sitemap/5b9502d842f6f3c9bcc2b79aaa68cc8cc0b873fc create mode 100644 fuzz/corpus/sitemap/5d4f1a50e41b5044c1a0b01042907b2238809124 create mode 100644 fuzz/corpus/sitemap/5e9d8fe18524df401151d7bd9ebe05d14e7b167c create mode 100644 fuzz/corpus/sitemap/6197e36e9bafff09c9a7a1d3d3a1acfe228d42bb create mode 100644 fuzz/corpus/sitemap/65fb54c30ea60a48cf5fe9c89c0913d9feaf81e4 create mode 100644 fuzz/corpus/sitemap/678a87639c0db550bf6410391fae33adeb423816 create mode 100644 fuzz/corpus/sitemap/67f7ea7cc3966cba35925ab2f569e37b72f20770 create mode 100644 fuzz/corpus/sitemap/68533b9173b5d26bdc4f7482d3720f8b338d97ec create mode 100644 fuzz/corpus/sitemap/68e2689b4a313192bbdb5a10d50950aa44ed9b8c create mode 100644 fuzz/corpus/sitemap/694e068d470a3d67ba85c6db6dc210d89fcf3b1f create mode 100644 fuzz/corpus/sitemap/69b20e91038ff742a182007e4fd0b0eea8390ede create mode 100644 fuzz/corpus/sitemap/6c4982d82648107e62671bcc8534d618a3796cad create mode 100644 fuzz/corpus/sitemap/6d66650a7220ffcfad7f0c1f4d12590e4969789a create mode 100644 fuzz/corpus/sitemap/6dc3d7277036e229086c50809d03ffbfca93de13 create mode 100644 fuzz/corpus/sitemap/6dd1cfa987e900f39c1f65ad4604e537770f2a63 create mode 100644 fuzz/corpus/sitemap/6e05240a3d1eafde09d3b4592609150136ee5d06 create mode 100644 fuzz/corpus/sitemap/7236e9de6722ef91534680f34887c625a2fe49da create mode 100644 fuzz/corpus/sitemap/7327e7b591e2d97b3d0fb981d245a58e2c8c3e57 create mode 100644 fuzz/corpus/sitemap/73c66b9ecc06366897053ed1d9be1fcce523f149 create mode 100644 fuzz/corpus/sitemap/74a318c0a2c4c5f2686ea8a627f5eb15ee3bd668 create mode 100644 fuzz/corpus/sitemap/74c1d696fde563ca5abf9e9ae1823dd9ce76c61b create mode 100644 fuzz/corpus/sitemap/755c9c71ed51fdd6b853c40cce2d8364a98ca867 create mode 100644 fuzz/corpus/sitemap/75d69f6d427862dbc1e6b1c2ea7009f92fc55517 create mode 100644 fuzz/corpus/sitemap/7749e198ec66b0723e8e1b1c0285f5d14d190269 create mode 100644 fuzz/corpus/sitemap/776ce1aa1e931dd4016b94a4e3c18dcf569cc855 create mode 100644 fuzz/corpus/sitemap/77d9b43d5bbd08ef9338cf6a8c5f1a786390bd77 create mode 100644 fuzz/corpus/sitemap/7969e0e43da1b80d48e157309e415aef025ffd84 create mode 100644 fuzz/corpus/sitemap/7c99aac7091b9ff1023ebf12897c42c2b5146750 create mode 100644 fuzz/corpus/sitemap/7d56e311f61b11792334f7d4d57ffe009596a093 create mode 100644 fuzz/corpus/sitemap/8133938bc85e7f4cff891db44b4c598b4039d0ae create mode 100644 fuzz/corpus/sitemap/8596c9456ec637abc8477e42308210956d002e33 create mode 100644 fuzz/corpus/sitemap/85f54f06c01b52e7af77e1ea43c13becc0176702 create mode 100644 fuzz/corpus/sitemap/863661d323853a881d3a66d0a3834e863ca0b93b create mode 100644 fuzz/corpus/sitemap/864d934d6dcd3ad14525d1a59df1d8f7a428c252 create mode 100644 fuzz/corpus/sitemap/872faf1a741133a61098cd1cc8a0898ea86248ae create mode 100644 fuzz/corpus/sitemap/89e598f54fea7059c8c28b1b9418d7c0e5a8a028 create mode 100644 fuzz/corpus/sitemap/8ac6a34d65103279df1eaaaabfb7823be2f75387 create mode 100644 fuzz/corpus/sitemap/8add9d8dc27b939c444152727c95801a76093b13 create mode 100644 fuzz/corpus/sitemap/8b0798c9bf0a8f1206a8cce5b49c48348a5acb3c create mode 100644 fuzz/corpus/sitemap/8cbd6fc35a657c15a117f5fa44b9c5475b5c244e create mode 100644 fuzz/corpus/sitemap/8ef128178bd3c31759010e9ed4641d56511a10cc create mode 100644 fuzz/corpus/sitemap/8f69378809f4cb3b06cd71b4310246aede68be6c create mode 100644 fuzz/corpus/sitemap/8f874e1efeac088e0fe991ac5fa8dda7e17b06ae create mode 100644 fuzz/corpus/sitemap/904b658055ecbe45f30526969f0b4634c2c9f641 create mode 100644 fuzz/corpus/sitemap/904c672c4c5e3f2a0d875264c6940798e7bcdecd create mode 100644 fuzz/corpus/sitemap/92d2db22c0000a2d6fa54a2cf128f92db97b9993 create mode 100644 fuzz/corpus/sitemap/932ff270a6a487aa020b26421534a1c12b4a94f8 create mode 100644 fuzz/corpus/sitemap/94609cd811a26c48c711415866b304e214a4e97a create mode 100644 fuzz/corpus/sitemap/94b3ab30db215ec5b044a7dc24fb5e69e8778659 create mode 100644 fuzz/corpus/sitemap/956a73fe53021543044dfcfb61173044ca7c2a17 create mode 100644 fuzz/corpus/sitemap/97619afecb1c73169b02116ff610543e833a1e60 create mode 100644 fuzz/corpus/sitemap/97645ac8be26bd22563d7d27fb833d964977a44a create mode 100644 fuzz/corpus/sitemap/9811960a0551fd86e280eb58ed6446c0ecc74fbb create mode 100644 fuzz/corpus/sitemap/983772efe5696d5b65d0f006808a477d9225feab create mode 100644 fuzz/corpus/sitemap/993c132d0386ef7b502bab4e62d26023f0bf0250 create mode 100644 fuzz/corpus/sitemap/99711e0eee687fb4b9305e8b700996ef49616f20 create mode 100644 fuzz/corpus/sitemap/9ac709a1a3dcd5f15b0d655ea2a5a84bb7685596 create mode 100644 fuzz/corpus/sitemap/9be22a3ec737f88e0e61326f822e65952a6aeb7d create mode 100644 fuzz/corpus/sitemap/9e7d3f551e039e680769c619f850085179ffef3c create mode 100644 fuzz/corpus/sitemap/9fbbd38632a5e00322c3c1182b5de821d96d01c4 create mode 100644 fuzz/corpus/sitemap/a6ca50dcffc4ceb4fde9cb48683e6e6179bc9d17 create mode 100644 fuzz/corpus/sitemap/a70841b5461831653714d0086967ba0ee158ae0c create mode 100644 fuzz/corpus/sitemap/a79b49ba5a2b994879a6040a97910a9837d076da create mode 100644 fuzz/corpus/sitemap/ab4b1bb206a56cd96dc06daf7782f1880a6cac6e create mode 100644 fuzz/corpus/sitemap/ae028d22587dc0244e4798b26e2a2bf13167347f create mode 100644 fuzz/corpus/sitemap/b4967ae700f5af942180d72ee02b24bf294840dd create mode 100644 fuzz/corpus/sitemap/b64f8460e1fad6df6b9e663fa4e60e10db2c9bca create mode 100644 fuzz/corpus/sitemap/b721ef14aa0f27fb82d7f1db3020ceaa87377f6c create mode 100644 fuzz/corpus/sitemap/b8394acb26ab461576b225188103d0dedc45457b create mode 100644 fuzz/corpus/sitemap/bab6f59b7df9cac3ba92122d377d495c0659e215 create mode 100644 fuzz/corpus/sitemap/bac8a95096d48861e5ad182e63a9900eb15ade81 create mode 100644 fuzz/corpus/sitemap/bb603cec6991c99bba03265550e773e92e130f6e create mode 100644 fuzz/corpus/sitemap/be0f77cd79e4effa6060839f02f1f086dbec05c3 create mode 100644 fuzz/corpus/sitemap/c02c945097f3529771b1e4940af36e9bfc40c066 create mode 100644 fuzz/corpus/sitemap/c26c7a6dd511884da2655f00dcb5b796170e0526 create mode 100644 fuzz/corpus/sitemap/c41d6ab840b6a73aabd37e77a511f48f79fee4f6 create mode 100644 fuzz/corpus/sitemap/c6656a3f7e241754587f35ab64975d630c8f1b16 create mode 100644 fuzz/corpus/sitemap/c7fd6658ea9a3951d5f3f66d0eea6455a32a2751 create mode 100644 fuzz/corpus/sitemap/c8e34963df67be9d69f5455703f59808c385994b create mode 100644 fuzz/corpus/sitemap/c8e7d3426661a572c544513d07ccf7a165bb074c create mode 100644 fuzz/corpus/sitemap/cec49314e1cef1adfcab5ff6cf3024f5b8962ce4 create mode 100644 fuzz/corpus/sitemap/d08fd04e86bce0ee91539aa4e3f88e4857797a45 create mode 100644 fuzz/corpus/sitemap/d229c3ccf5655abdf7dd507a5423d82c29f958b6 create mode 100644 fuzz/corpus/sitemap/d2b67f32adc4dfae849a928bec97eba3fda7ba63 create mode 100644 fuzz/corpus/sitemap/d3c84d21ece3614327c53f6e4f3984870b1b9542 create mode 100644 fuzz/corpus/sitemap/d4c1d2ac1eabd38c5b340c5ce1698306de40f8c4 create mode 100644 fuzz/corpus/sitemap/d9fadcc7f4fae9b160d860ce1acdb21d77aab90b create mode 100644 fuzz/corpus/sitemap/dac4463bd89dab0ec0879cb6a6f250bec1603ca0 create mode 100644 fuzz/corpus/sitemap/dc5a07c652023cfd6c5b2be5503168d5b67a34f7 create mode 100644 fuzz/corpus/sitemap/dd3778d3276b51e63e8180499e39621a4300a3f6 create mode 100644 fuzz/corpus/sitemap/dd5fba84018693317bdfe544eeb8d880499c0aa6 create mode 100644 fuzz/corpus/sitemap/e35ba049599035ebc4c0d46396b63991e9fcd9bd create mode 100644 fuzz/corpus/sitemap/e5feb1ad3e6d18f9e10d3475eaadc462633ba30d create mode 100644 fuzz/corpus/sitemap/e76101ae92d1a510293cd56283804b33783b3a11 create mode 100644 fuzz/corpus/sitemap/e76bd0139a9d24b6575a83b6154e1c4a5f89e839 create mode 100644 fuzz/corpus/sitemap/e8eb9faa5d366c5bd059b1ba22c5fe8cb54ac36b create mode 100644 fuzz/corpus/sitemap/e9318e9449f610242dbc7ee871974fe1ea933a19 create mode 100644 fuzz/corpus/sitemap/eccbd84d02c3aeaac60a42ce6c5bad7971db0bf4 create mode 100644 fuzz/corpus/sitemap/eda0a1789a40de76a6d4474dbc76f218050ce8b3 create mode 100644 fuzz/corpus/sitemap/edcf1376941c09176dda0d8b134a8ce9e6e686f2 create mode 100644 fuzz/corpus/sitemap/ee5c05abfc504dda713cd9368fd6b52538873afc create mode 100644 fuzz/corpus/sitemap/ef40dcbe031bad1543d0e39ded75d2d503df40b3 create mode 100644 fuzz/corpus/sitemap/efc62b91a0bd15d95ba261d6bda1e4ee7760cb01 create mode 100644 fuzz/corpus/sitemap/f1f0d28d3fedc79f2a302920fa4da7d75ddc8549 create mode 100644 fuzz/corpus/sitemap/f25fee6de9c4c840c4e15c80d7cc7165d16964ed create mode 100644 fuzz/corpus/sitemap/f36385c58a6f76373d071f0050edb338b94ffc41 create mode 100644 fuzz/corpus/sitemap/f547b6e16cb92fe2408fdeeccea8fde8eba3646a create mode 100644 fuzz/corpus/sitemap/f5794f9cf1cf371539ad3bec5561eaf6516b261d create mode 100644 fuzz/corpus/sitemap/f620966e26b1267099535a59df7f443db7b3064b create mode 100644 fuzz/corpus/sitemap/f82b077bd6e07d706117e02d3872bb881f606493 create mode 100644 fuzz/corpus/sitemap/f8ba6bbd60fe059c383e42d59b1c0ec054c8ebe5 create mode 100644 fuzz/corpus/sitemap/f8d1d2ddaf0ba19b56b5acc75d3f2e2abe509c5b create mode 100644 fuzz/corpus/sitemap/f96b23bfa34bae6b2344000d4b9feae5a054409e create mode 100644 fuzz/corpus/sitemap/fbbf0b771191ed9ca496abfaca123a2fdf80fe8c create mode 100644 fuzz/corpus/sitemap/fbdf3c734f46c16e43038166cd2f05a5162ded27 create mode 100644 fuzz/corpus/sitemap/fd6ba10987b71497231ab5d7f51618caf600227e create mode 100644 fuzz/corpus/sitemap/fdc9015b3be9b81bd64fde25e6a57af1808564cc create mode 100644 fuzz/corpus/sitemap/fe92867179fa26acfdbc620381505332e7815b13 create mode 100644 fuzz/corpus/sitemap/sitemapindex.xml create mode 100644 fuzz/corpus/sitemap/truncated.xml create mode 100644 fuzz/corpus/sitemap/urlset.xml create mode 100644 fuzz/corpus/sitemap/urlset.xml.gz create mode 100644 fuzz/fuzz-sitemap.c diff --git a/fuzz/Makefile.am b/fuzz/Makefile.am index 66624d81a..c9ae330f9 100644 --- a/fuzz/Makefile.am +++ b/fuzz/Makefile.am @@ -2,7 +2,7 @@ if FUZZERS noinst_PROGRAMS = fuzz-charset fuzz-meta fuzz-idna fuzz-entities \ fuzz-unescape fuzz-filters fuzz-url fuzz-header fuzz-cachendx \ - fuzz-htsparse + fuzz-htsparse fuzz-sitemap endif AM_CPPFLAGS = \ @@ -27,6 +27,7 @@ fuzz_url_SOURCES = fuzz-url.c fuzz.h fuzz_header_SOURCES = fuzz-header.c fuzz.h fuzz_cachendx_SOURCES = fuzz-cachendx.c fuzz.h fuzz_htsparse_SOURCES = fuzz-htsparse.c fuzz.h +fuzz_sitemap_SOURCES = fuzz-sitemap.c fuzz.h # List corpus files explicitly: automake does not expand EXTRA_DIST globs. EXTRA_DIST = README.md run-fuzzers.sh \ @@ -47,4 +48,6 @@ EXTRA_DIST = README.md run-fuzzers.sh \ corpus/cachendx/regress-overadvance.bin \ corpus/cachendx/regress-truncated-entry.bin \ corpus/htsparse/basic.html corpus/htsparse/script-inscript.html \ - corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html + corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html \ + corpus/sitemap/urlset.xml corpus/sitemap/sitemapindex.xml \ + corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz diff --git a/fuzz/corpus/sitemap/00a99f50f02b4dacea6d0089384b65e940b46cac b/fuzz/corpus/sitemap/00a99f50f02b4dacea6d0089384b65e940b46cac new file mode 100644 index 000000000..17cec566f --- /dev/null +++ b/fuzz/corpus/sitemap/00a99f50f02b4dacea6d0089384b65e940b46cac @@ -0,0 +1 @@ +http://h./ diff --git a/fuzz/corpus/sitemap/02f9cead3211d007e4476dca0542ba0642d9215f b/fuzz/corpus/sitemap/02f9cead3211d007e4476dca0542ba0642d9215f new file mode 100644 index 0000000000000000000000000000000000000000..aff5c400f1a632f5d31a2ce48ecb910edab149ce GIT binary patch literal 6 LcmZQp00IU80oVY9 literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/05d392d686a8affc917e9f4c0006470fc9ae10e2 b/fuzz/corpus/sitemap/05d392d686a8affc917e9f4c0006470fc9ae10e2 new file mode 100644 index 0000000000000000000000000000000000000000..d3efd6070aafa16520a1b32e1953f02227ba7fd7 GIT binary patch literal 79 zcmZR!Db6fO1>szV0-Itxo1FY)yNr^O0xNyJ3{9|jVu4+*9agCVI~)Dt%$n5P#DdJc Jl++44E&xgF8>RpN literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/060da1aaa324173a3992d401de50ba8a82dca8f6 b/fuzz/corpus/sitemap/060da1aaa324173a3992d401de50ba8a82dca8f6 new file mode 100644 index 000000000..53a08390e --- /dev/null +++ b/fuzz/corpus/sitemap/060da1aaa324173a3992d401de50ba8a82dca8f6 @@ -0,0 +1 @@ +hW`J*9%!nA4bUnO1FQ`IiS{k@ literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/0840f9901450f24841c81fda32af0830d06805fa b/fuzz/corpus/sitemap/0840f9901450f24841c81fda32af0830d06805fa new file mode 100644 index 000000000..644ca3a06 --- /dev/null +++ b/fuzz/corpus/sitemap/0840f9901450f24841c81fda32af0830d06805fa @@ -0,0 +1 @@ +https:tseh.//t/b?x=1&amTT\TT:t@seh.//t/b?x=1&aMp;y=2 diff --git a/fuzz/corpus/sitemap/0a3c091488367a3446b8707090ee0ff653b16fe0 b/fuzz/corpus/sitemap/0a3c091488367a3446b8707090ee0ff653b16fe0 new file mode 100644 index 000000000..55026cd72 --- /dev/null +++ b/fuzz/corpus/sitemap/0a3c091488367a3446b8707090ee0ff653b16fe0 @@ -0,0 +1 @@ +si>h/t/b?x=M5≈y=2loc>/t/b?x=M5≈y=emapinde1&a>/t/b?x=M5≈p;y=2 diff --git a/fuzz/corpus/sitemap/0c18b306e933712fe4c6bed04ee0962c82201fe6 b/fuzz/corpus/sitemap/0c18b306e933712fe4c6bed04ee0962c82201fe6 new file mode 100644 index 000000000..22d123286 --- /dev/null +++ b/fuzz/corpus/sitemap/0c18b306e933712fe4c6bed04ee0962c82201fe6 @@ -0,0 +1 @@ +stemapinde1&a>/p/b?si/xg5M≈y=emap5≈tsioc>/xg5M≈y=emap5≈p;y=2inde1&a>/t/b?x> diff --git a/fuzz/corpus/sitemap/0c4a15a3173d6d5292dc4e0104bcca72cc4db27e b/fuzz/corpus/sitemap/0c4a15a3173d6d5292dc4e0104bcca72cc4db27e new file mode 100644 index 0000000000000000000000000000000000000000..cd43e831fbea47af8ed0a875fb514b2126db951f GIT binary patch literal 48 zcmZ?#|NsAfI~)DdB0C$6q8vM$ocv_F4E+-QB>M_mLp5a(PGn$U0LlQ>ye|L%$P5sH literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/0caed9d0f4542bf2c1bafb49a842a34f09bfcbef b/fuzz/corpus/sitemap/0caed9d0f4542bf2c1bafb49a842a34f09bfcbef new file mode 100644 index 0000000000000000000000000000000000000000..17c574cedbdbf5fe11b7cffb04dbb25f0b9bf1cf GIT binary patch literal 18 ZcmZR!DbCDIEXYhvVYtMQZLeRf1pqiD1=Iil literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/0cbd954d0871608e3a42043dc0265395dff86c9b b/fuzz/corpus/sitemap/0cbd954d0871608e3a42043dc0265395dff86c9b new file mode 100644 index 0000000000000000000000000000000000000000..8a626855747654495304803185015a6c68ff67a4 GIT binary patch literal 54 ucmZRyDYCP($;oH12a(BkoKW!OKf`|p1_qnbqGApp4F#n|IfqkA?6?420StEl literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/0d56488006e785281ef1ea0a4cd878cc10cf36de b/fuzz/corpus/sitemap/0d56488006e785281ef1ea0a4cd878cc10cf36de new file mode 100644 index 000000000..dab61392f --- /dev/null +++ b/fuzz/corpus/sitemap/0d56488006e785281ef1ea0a4cd878cc10cf36de @@ -0,0 +1 @@ +http/h.test/b?x2↦https:tseh.//t/b?x=1&x=1&r>httn="1.1&r> diff --git a/fuzz/corpus/sitemap/0e00256a636924a1350f5f6353e954715b5094fb b/fuzz/corpus/sitemap/0e00256a636924a1350f5f6353e954715b5094fb new file mode 100644 index 0000000000000000000000000000000000000000..17da0721b1694547e8785a95d9245491a6c71ee6 GIT binary patch literal 33 ncmZR!DM`&`D9Fr9NwvwzPqxb_DJihh*UKo*1d1mX*x3L8y&VgB literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/0efcc232871dd78f5b8f57944e9b66e77c742246 b/fuzz/corpus/sitemap/0efcc232871dd78f5b8f57944e9b66e77c742246 new file mode 100644 index 0000000000000000000000000000000000000000..ae392e575a1019f43e9e787413f8d4b242a04d4e GIT binary patch literal 131 zcmZ?#|NsAfI~)DdB0C$6q8taCocv_F4E+-QB>M_mBQ<4PLp5bMAIJvr6AuM2Br-5C Pv \ No newline at end of file diff --git a/fuzz/corpus/sitemap/119c566502513de0981ae5400b315d02fce93311 b/fuzz/corpus/sitemap/119c566502513de0981ae5400b315d02fce93311 new file mode 100644 index 000000000..ac53e04b5 --- /dev/null +++ b/fuzz/corpus/sitemap/119c566502513de0981ae5400b315d02fce93311 @@ -0,0 +1 @@ +>https://h.test/b? diff --git a/fuzz/corpus/sitemap/11f6ad8ec52a2984abaafd7c3b516503785c2072 b/fuzz/corpus/sitemap/11f6ad8ec52a2984abaafd7c3b516503785c2072 new file mode 100644 index 000000000..c1b0730e0 --- /dev/null +++ b/fuzz/corpus/sitemap/11f6ad8ec52a2984abaafd7c3b516503785c2072 @@ -0,0 +1 @@ +x \ No newline at end of file diff --git a/fuzz/corpus/sitemap/136825aeb153d97b2ac3f664136dca068e2cdbf3 b/fuzz/corpus/sitemap/136825aeb153d97b2ac3f664136dca068e2cdbf3 new file mode 100644 index 000000000..b894eee47 --- /dev/null +++ b/fuzz/corpus/sitemap/136825aeb153d97b2ac3f664136dca068e2cdbf3 @@ -0,0 +1 @@ +x: \ No newline at end of file diff --git a/fuzz/corpus/sitemap/13f68f78d292b78efac3f93369af9b7fa67e671c b/fuzz/corpus/sitemap/13f68f78d292b78efac3f93369af9b7fa67e671c new file mode 100644 index 0000000000000000000000000000000000000000..1eb2d0c56c83c6fea15ff380368b748dc75537d7 GIT binary patch literal 62 vcmZRS@8)2Tbxg}*`oCFI_mrle?#9iGm&(j6f1WI}00S!!`5ytQej?ZaJ9i^? literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/143b943867717061de12ae64372cd97a5427e976 b/fuzz/corpus/sitemap/143b943867717061de12ae64372cd97a5427e976 new file mode 100644 index 000000000..01545ed92 --- /dev/null +++ b/fuzz/corpus/sitemap/143b943867717061de12ae64372cd97a5427e976 @@ -0,0 +1 @@ +hoev \ No newline at end of file diff --git a/fuzz/corpus/sitemap/1683f30ee69c20212771f0977158957bc51d06be b/fuzz/corpus/sitemap/1683f30ee69c20212771f0977158957bc51d06be new file mode 100644 index 000000000..a7425ce9b --- /dev/null +++ b/fuzz/corpus/sitemap/1683f30ee69c20212771f0977158957bc51d06be @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/fuzz/corpus/sitemap/16cf9bebe58d0fee1a6c413a08ddf3b39c9974e2 b/fuzz/corpus/sitemap/16cf9bebe58d0fee1a6c413a08ddf3b39c9974e2 new file mode 100644 index 0000000000000000000000000000000000000000..aab905f8a9456241f30d2dcf11be4e0b88d7953e GIT binary patch literal 118 zcmdm8!Dv%-$j;6tCqLOPqokyu*h*iYQE$eS+#DMQ5GX3nOfN3cPt?mO0Sf6Cfuzw? XGcXvbC88UFO-*irRi&+wosB*KlQknZ literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/1774568b8fe763552d59661d139b3c1c32886b5a b/fuzz/corpus/sitemap/1774568b8fe763552d59661d139b3c1c32886b5a new file mode 100644 index 0000000000000000000000000000000000000000..2fe3d21a41f5a161135450a5649c3d411d448fd9 GIT binary patch literal 58 ucmZR!Db6fO%}p%G%u7kFu!HgJY;y9G?J|l>7_RE2SJ~);csBaTD!Bl385I)% literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/17cd699ea8085eabee01b683a8abfa32b6f6bc22 b/fuzz/corpus/sitemap/17cd699ea8085eabee01b683a8abfa32b6f6bc22 new file mode 100644 index 000000000..332d717e0 --- /dev/null +++ b/fuzz/corpus/sitemap/17cd699ea8085eabee01b683a8abfa32b6f6bc22 @@ -0,0 +1 @@ +htvps:ts/@t/b;x=1&amTTTTTTTTTTTTpVy=2T diff --git a/fuzz/corpus/sitemap/1950639e93b0e273ebfb2b5f47763966685c23fe b/fuzz/corpus/sitemap/1950639e93b0e273ebfb2b5f47763966685c23fe new file mode 100644 index 000000000..550a2ae56 --- /dev/null +++ b/fuzz/corpus/sitemap/1950639e93b0e273ebfb2b5f47763966685c23fe @@ -0,0 +1 @@ +?=x1&999999290$TiGTzvB1tICqLOPqokz3N?+ebzc{lbH7_N#LfVdt7c>https:rl>{.0rl>httpshtops:tlset> +o> \ No newline at end of file diff --git a/fuzz/corpus/sitemap/217aea3e505eb01e6aa3cd317fd53d1d25295228 b/fuzz/corpus/sitemap/217aea3e505eb01e6aa3cd317fd53d1d25295228 new file mode 100644 index 000000000..2724b2d1d --- /dev/null +++ b/fuzz/corpus/sitemap/217aea3e505eb01e6aa3cd317fd53d1d25295228 @@ -0,0 +1 @@ +*bhp://h.te+st/x \ No newline at end of file diff --git a/fuzz/corpus/sitemap/22a628e7d0e55e1b90500087ef00a14d113594be b/fuzz/corpus/sitemap/22a628e7d0e55e1b90500087ef00a14d113594be new file mode 100644 index 0000000000000000000000000000000000000000..15e0d866910a985c8ecea4f4ff45e9413fab8d6f GIT binary patch literal 31 mcmc~zEXd4Dtzckau*u0!w#z7CWC&wgUi_FbK^6 literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/22d12b8efb1cd66561fc123953d9af9ea3b36d19 b/fuzz/corpus/sitemap/22d12b8efb1cd66561fc123953d9af9ea3b36d19 new file mode 100644 index 0000000000000000000000000000000000000000..107f120b999b82cd58593bed9bc5193a716ab1a1 GIT binary patch literal 129 scmZRyv9DmD8OY61C`&CW&dkqaUhttp://ы/loc>> diff --git a/fuzz/corpus/sitemap/2342c6bd446e3561a5fca62c1394ac0ed30f7212 b/fuzz/corpus/sitemap/2342c6bd446e3561a5fca62c1394ac0ed30f7212 new file mode 100644 index 000000000..b454ce67b --- /dev/null +++ b/fuzz/corpus/sitemap/2342c6bd446e3561a5fca62c1394ac0ed30f7212 @@ -0,0 +1 @@ +< \ No newline at end of file diff --git a/fuzz/corpus/sitemap/246e6f26840d821990ae19d45fde49c40a6f43e2 b/fuzz/corpus/sitemap/246e6f26840d821990ae19d45fde49c40a6f43e2 new file mode 100644 index 000000000..4ea487689 --- /dev/null +++ b/fuzz/corpus/sitemap/246e6f26840d821990ae19d45fde49c40a6f43e2 @@ -0,0 +1 @@ + < \ No newline at end of file diff --git a/fuzz/corpus/sitemap/273c10d28ee0022f1ef9e438373c5427991d2c32 b/fuzz/corpus/sitemap/273c10d28ee0022f1ef9e438373c5427991d2c32 new file mode 100644 index 000000000..a5e9d430d --- /dev/null +++ b/fuzz/corpus/sitemap/273c10d28ee0022f1ef9e438373c5427991d2c32 @@ -0,0 +1 @@ +https:tseh.//t/b?x=1&aMp1a~∓y=2XXXXXX>https:tseh.//t/b?x=1&amTTTTTTTTpVy=c>https:tseh.//t/b?x=1&amTTTTTTTTpVy=2 diff --git a/fuzz/corpus/sitemap/336abd1a87047f61c40dccaf4ad1df4156ec930c b/fuzz/corpus/sitemap/336abd1a87047f61c40dccaf4ad1df4156ec930c new file mode 100644 index 0000000000000000000000000000000000000000..7c993b820b94a29f9fed333a18d4de8292cc6f6d GIT binary patch literal 98 zcmZRyDYCP($;nT)%P1)+D7MnqXVl9m$<4800D+?7%>3dK{X~e6K88{>oeT^{YKgf8 Sm9|E9Hu|MSIfqkA?6?54-5uiq literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/3535f99e3b54d0ba2284fbcc0759c91664989bce b/fuzz/corpus/sitemap/3535f99e3b54d0ba2284fbcc0759c91664989bce new file mode 100644 index 000000000..197458679 --- /dev/null +++ b/fuzz/corpus/sitemap/3535f99e3b54d0ba2284fbcc0759c91664989bce @@ -0,0 +1 @@ +htops:tseh.//t/b?x=0&r>Čitemapindex>h \ No newline at end of file diff --git a/fuzz/corpus/sitemap/36a2f4986e72271a3c2e024a78457f25a8d08b45 b/fuzz/corpus/sitemap/36a2f4986e72271a3c2e024a78457f25a8d08b45 new file mode 100644 index 000000000..e93d49cda --- /dev/null +++ b/fuzz/corpus/sitemap/36a2f4986e72271a3c2e024a78457f25a8d08b45 @@ -0,0 +1 @@ +lE/| \ No newline at end of file diff --git a/fuzz/corpus/sitemap/36f83a6227244c2f4b8d9e37babcc5c1aaac026d b/fuzz/corpus/sitemap/36f83a6227244c2f4b8d9e37babcc5c1aaac026d new file mode 100644 index 0000000000000000000000000000000000000000..6e0092172239f194860e2e213cf63029ec80622a GIT binary patch literal 68 zcmZ?#|NsAfI~)DdB0C$6q8taCocv_F4E+-QB>M_mLp5a(M&~CoFff2{4p`s&0su5d B6ypE@ literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/38b09ca3dcb5be711a2e40fe957d2349422f1eb0 b/fuzz/corpus/sitemap/38b09ca3dcb5be711a2e40fe957d2349422f1eb0 new file mode 100644 index 000000000..327736469 --- /dev/null +++ b/fuzz/corpus/sitemap/38b09ca3dcb5be711a2e40fe957d2349422f1eb0 @@ -0,0 +1 @@ +tema><> #) #) #) https:tseh.//t/b?x=1&aMp;y=2>https:tseh.//t/b?x=1&aMp;y=2 diff --git a/fuzz/corpus/sitemap/3e284e0c7aaebfd6e411fe73bcd68c8800d34839 b/fuzz/corpus/sitemap/3e284e0c7aaebfd6e411fe73bcd68c8800d34839 new file mode 100644 index 0000000000000000000000000000000000000000..480c60c1fdf698abfc5a5b7efb5c04d777fbe2f7 GIT binary patch literal 46 zcmZRS@8)2Tbv&KL^nbIa?kP<_-i@0J884NYS^hj}0R&|h7FHG(|A9bLx^@EqhLaM@ literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/3e3785814ac9579f350b899facab62e80d43dcdd b/fuzz/corpus/sitemap/3e3785814ac9579f350b899facab62e80d43dcdd new file mode 100644 index 0000000000000000000000000000000000000000..1362753fca4a0f78fd4a759651193e614dcbd6f5 GIT binary patch literal 55 ycmZRyv9GWyVL%4Oj6j}}p`M*hPJXgoMoCFQu~mtFl6{4(fm&i|fw_^5z5)RMjthhU literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/3e73f5237baee8133c1e086e7cd623f3b9da941c b/fuzz/corpus/sitemap/3e73f5237baee8133c1e086e7cd623f3b9da941c new file mode 100644 index 000000000..f9cc4e523 --- /dev/null +++ b/fuzz/corpus/sitemap/3e73f5237baee8133c1e086e7cd623f3b9da941c @@ -0,0 +1 @@ +Bs7h/7hx \ No newline at end of file diff --git a/fuzz/corpus/sitemap/3f10def7d9e4a04ecda6c41ae28f5c1117082dc7 b/fuzz/corpus/sitemap/3f10def7d9e4a04ecda6c41ae28f5c1117082dc7 new file mode 100644 index 0000000000000000000000000000000000000000..d60e94e796e8a19c1edcdb5fe2dd5cc56fa22421 GIT binary patch literal 101 zcmZ<~WiXVMmd>0$TiGTzvB1tICqLOPqokz3N?+ebzc{lbH7_N#LfVcCtPZ<00o@rD E03bCWivR!s literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/3feeb4edb06a3ed045f6024a0552dff46b9b2d57 b/fuzz/corpus/sitemap/3feeb4edb06a3ed045f6024a0552dff46b9b2d57 new file mode 100644 index 000000000..a08fc2c32 --- /dev/null +++ b/fuzz/corpus/sitemap/3feeb4edb06a3ed045f6024a0552dff46b9b2d57 @@ -0,0 +1 @@ +l vlo> it https:tsops:tpp/ diff --git a/fuzz/corpus/sitemap/48e2a7d7f446fb8db38897441af4387e54e9a7da b/fuzz/corpus/sitemap/48e2a7d7f446fb8db38897441af4387e54e9a7da new file mode 100644 index 000000000..2a786e93a --- /dev/null +++ b/fuzz/corpus/sitemap/48e2a7d7f446fb8db38897441af4387e54e9a7da @@ -0,0 +1 @@ +<se>.bluse>^(5hXbDd(E% M(0Cjs%fP?@02+!bi2wiq literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/4cfc6f83573a5c40b77ec7a89f6c022549e99108 b/fuzz/corpus/sitemap/4cfc6f83573a5c40b77ec7a89f6c022549e99108 new file mode 100644 index 0000000000000000000000000000000000000000..b7a6b708ea80ee069298c06ab2da83630e0a7c7b GIT binary patch literal 68 zcmZR!Db6fO%}p%G%u7kFu!HgJY;y9G?J`PA3as??GxSPQi%S@;>ZMoN=!3*<^pW*& F0RU8%7;*pr literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/4d9c68140b8039b221fc77153f2484e8524cbb0f b/fuzz/corpus/sitemap/4d9c68140b8039b221fc77153f2484e8524cbb0f new file mode 100644 index 000000000..ad21d05ad --- /dev/null +++ b/fuzz/corpus/sitemap/4d9c68140b8039b221fc77153f2484e8524cbb0f @@ -0,0 +1 @@ +BsBs/7has \ No newline at end of file diff --git a/fuzz/corpus/sitemap/4f5f0e6a0036466d373930b0b66ff37162ae4390 b/fuzz/corpus/sitemap/4f5f0e6a0036466d373930b0b66ff37162ae4390 new file mode 100644 index 0000000000000000000000000000000000000000..875f88ebcd7e4de9f1271b7034afcbd1fad0a4a2 GIT binary patch literal 42 wcmZ>KzuP9W$j+v;D96qwCqLOPqohPX$-ctYP%SZ6*_x3-A1G67qi@Fr08X9_#{d8T literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/50bdd2e5fe8ec578126fe54af1b200b34aa1ec7b b/fuzz/corpus/sitemap/50bdd2e5fe8ec578126fe54af1b200b34aa1ec7b new file mode 100644 index 0000000000000000000000000000000000000000..e697beb690317ffd33d976c41793b45cad43d0dd GIT binary patch literal 49 ucmZRS@8)2Tbxg}*`oCFI_mrle?#9iGm&(j6f1WI}00S!!`5yup7#IMxf)o(| literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/5710370c1503e7d2f6c76d500707b08503602563 b/fuzz/corpus/sitemap/5710370c1503e7d2f6c76d500707b08503602563 new file mode 100644 index 000000000..343a1776b --- /dev/null +++ b/fuzz/corpus/sitemap/5710370c1503e7d2f6c76d500707b08503602563 @@ -0,0 +1 @@ +>http://ы/loc>/ diff --git a/fuzz/corpus/sitemap/58995e9e3d5f5f2ad9ade175594b5c358019f144 b/fuzz/corpus/sitemap/58995e9e3d5f5f2ad9ade175594b5c358019f144 new file mode 100644 index 000000000..7fbc70987 --- /dev/null +++ b/fuzz/corpus/sitemap/58995e9e3d5f5f2ad9ade175594b5c358019f144 @@ -0,0 +1 @@ +>httppppp S^hj(W&s9P78d^@fPsMl0JJI;bN~PV literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/597be90d0fda30c3def2baa7135d064827000cce b/fuzz/corpus/sitemap/597be90d0fda30c3def2baa7135d064827000cce new file mode 100644 index 000000000..337589125 --- /dev/null +++ b/fuzz/corpus/sitemap/597be90d0fda30c3def2baa7135d064827000cce @@ -0,0 +1 @@ +rl>ihttst/x=5&y=2< \ No newline at end of file diff --git a/fuzz/corpus/sitemap/5a8890576de5cc596d37a68f8a96f79a3056bf3b b/fuzz/corpus/sitemap/5a8890576de5cc596d37a68f8a96f79a3056bf3b new file mode 100644 index 0000000000000000000000000000000000000000..4602471ca2ac82dc63673989973d3d565b38603f GIT binary patch literal 22 YcmZRyv9GYn$xpVkVPIfjhp_aS0X-%K761SM literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/5b9502d842f6f3c9bcc2b79aaa68cc8cc0b873fc b/fuzz/corpus/sitemap/5b9502d842f6f3c9bcc2b79aaa68cc8cc0b873fc new file mode 100644 index 000000000..25fc96719 --- /dev/null +++ b/fuzz/corpus/sitemap/5b9502d842f6f3c9bcc2b79aaa68cc8cc0b873fc @@ -0,0 +1 @@ +https:tseh.//t/b?x=1&y=2https:tseh.//t/b?x=1&y=2 diff --git a/fuzz/corpus/sitemap/5e9d8fe18524df401151d7bd9ebe05d14e7b167c b/fuzz/corpus/sitemap/5e9d8fe18524df401151d7bd9ebe05d14e7b167c new file mode 100644 index 0000000000000000000000000000000000000000..aad082de5c363dfe64e623a55073ffb43b3f42be GIT binary patch literal 82 zcmZQ%2q<7+aIIlruxDUku*u0!wzIdhDJm__NiDInDa|MW&j0> Q3o;N&5vme%3o`Wd0JC}-7XSbN literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/6197e36e9bafff09c9a7a1d3d3a1acfe228d42bb b/fuzz/corpus/sitemap/6197e36e9bafff09c9a7a1d3d3a1acfe228d42bb new file mode 100644 index 000000000..f78652218 --- /dev/null +++ b/fuzz/corpus/sitemap/6197e36e9bafff09c9a7a1d3d3a1acfe228d42bb @@ -0,0 +1 @@ +https:tseh.//t/b?x=1&y=3.bluset> diff --git a/fuzz/corpus/sitemap/65fb54c30ea60a48cf5fe9c89c0913d9feaf81e4 b/fuzz/corpus/sitemap/65fb54c30ea60a48cf5fe9c89c0913d9feaf81e4 new file mode 100644 index 000000000..e7df8635b --- /dev/null +++ b/fuzz/corpus/sitemap/65fb54c30ea60a48cf5fe9c89c0913d9feaf81e4 @@ -0,0 +1 @@ +h-top>h \ No newline at end of file diff --git a/fuzz/corpus/sitemap/678a87639c0db550bf6410391fae33adeb423816 b/fuzz/corpus/sitemap/678a87639c0db550bf6410391fae33adeb423816 new file mode 100644 index 000000000..e008c0fe8 --- /dev/null +++ b/fuzz/corpus/sitemap/678a87639c0db550bf6410391fae33adeb423816 @@ -0,0 +1 @@ +:ts./se2>Khttps:tseh.&/t&bTTTTTTPTseh.tseh.&/tht<o \ No newline at end of file diff --git a/fuzz/corpus/sitemap/6c4982d82648107e62671bcc8534d618a3796cad b/fuzz/corpus/sitemap/6c4982d82648107e62671bcc8534d618a3796cad new file mode 100644 index 000000000..4043ed025 --- /dev/null +++ b/fuzz/corpus/sitemap/6c4982d82648107e62671bcc8534d618a3796cad @@ -0,0 +1 @@ + x diff --git a/fuzz/corpus/sitemap/6d66650a7220ffcfad7f0c1f4d12590e4969789a b/fuzz/corpus/sitemap/6d66650a7220ffcfad7f0c1f4d12590e4969789a new file mode 100644 index 000000000..5446b700c --- /dev/null +++ b/fuzz/corpus/sitemap/6d66650a7220ffcfad7f0c1f4d12590e4969789a @@ -0,0 +1 @@ +tema><> #) #) x: \ No newline at end of file diff --git a/fuzz/corpus/sitemap/6e05240a3d1eafde09d3b4592609150136ee5d06 b/fuzz/corpus/sitemap/6e05240a3d1eafde09d3b4592609150136ee5d06 new file mode 100644 index 000000000..6321066be --- /dev/null +++ b/fuzz/corpus/sitemap/6e05240a3d1eafde09d3b4592609150136ee5d06 @@ -0,0 +1 @@ +<hem< \ No newline at end of file diff --git a/fuzz/corpus/sitemap/7327e7b591e2d97b3d0fb981d245a58e2c8c3e57 b/fuzz/corpus/sitemap/7327e7b591e2d97b3d0fb981d245a58e2c8c3e57 new file mode 100644 index 0000000000000000000000000000000000000000..2daacfc1f56cbdbbbf2cf9697af47dc6c6fcd0bc GIT binary patch literal 25 gcmcC>w8_a&wo9@%P*YCKwXUqGWU#Z*FJ+Vm0ASw-2LJ#7 literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/73c66b9ecc06366897053ed1d9be1fcce523f149 b/fuzz/corpus/sitemap/73c66b9ecc06366897053ed1d9be1fcce523f149 new file mode 100644 index 0000000000000000000000000000000000000000..9e9d2a153aaf43b32d8b7804760c5561c6e42a07 GIT binary patch literal 55 lcmZRyDYCQ4$p@ljJ5EfH!C;_fX=$kkQj-iJI5|t~xBwVm3VQ$m literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/74a318c0a2c4c5f2686ea8a627f5eb15ee3bd668 b/fuzz/corpus/sitemap/74a318c0a2c4c5f2686ea8a627f5eb15ee3bd668 new file mode 100644 index 000000000..5df20c8b2 --- /dev/null +++ b/fuzz/corpus/sitemap/74a318c0a2c4c5f2686ea8a627f5eb15ee3bd668 @@ -0,0 +1 @@ +https:'/h.test/b?x=1&99999999999999999&em99999999999999999999999rl>https:tseh.//beth/.t> diff --git a/fuzz/corpus/sitemap/74c1d696fde563ca5abf9e9ae1823dd9ce76c61b b/fuzz/corpus/sitemap/74c1d696fde563ca5abf9e9ae1823dd9ce76c61b new file mode 100644 index 0000000000000000000000000000000000000000..14b265812634cf5965b80933f904f666c19ac551 GIT binary patch literal 63 qcmZRS@8)2Tbxg}*`oCFI_mrle?#9i}7%!EXS^hj(W`P0zqX7T{Ngs3o literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/755c9c71ed51fdd6b853c40cce2d8364a98ca867 b/fuzz/corpus/sitemap/755c9c71ed51fdd6b853c40cce2d8364a98ca867 new file mode 100644 index 000000000..fd78650a9 --- /dev/null +++ b/fuzz/corpus/sitemap/755c9c71ed51fdd6b853c40cce2d8364a98ca867 @@ -0,0 +1 @@ +sitemap;loc>/t/b?x=5⤅y=2https:ts//t/b?x=1&1&r>https:ts//t/b?x=1&y=2rl>htops:tslset> diff --git a/fuzz/corpus/sitemap/7749e198ec66b0723e8e1b1c0285f5d14d190269 b/fuzz/corpus/sitemap/7749e198ec66b0723e8e1b1c0285f5d14d190269 new file mode 100644 index 000000000..8d5313544 --- /dev/null +++ b/fuzz/corpus/sitemap/7749e198ec66b0723e8e1b1c0285f5d14d190269 @@ -0,0 +1 @@ +//h.+exst/x \ No newline at end of file diff --git a/fuzz/corpus/sitemap/776ce1aa1e931dd4016b94a4e3c18dcf569cc855 b/fuzz/corpus/sitemap/776ce1aa1e931dd4016b94a4e3c18dcf569cc855 new file mode 100644 index 000000000..fa40460a7 --- /dev/null +++ b/fuzz/corpus/sitemap/776ce1aa1e931dd4016b94a4e3c18dcf569cc855 @@ -0,0 +1 @@ +s>t@
:/h!tets/x \ No newline at end of file diff --git a/fuzz/corpus/sitemap/77d9b43d5bbd08ef9338cf6a8c5f1a786390bd77 b/fuzz/corpus/sitemap/77d9b43d5bbd08ef9338cf6a8c5f1a786390bd77 new file mode 100644 index 000000000..58b4519ef --- /dev/null +++ b/fuzz/corpus/sitemap/77d9b43d5bbd08ef9338cf6a8c5f1a786390bd77 @@ -0,0 +1 @@ +l  \ No newline at end of file diff --git a/fuzz/corpus/sitemap/7969e0e43da1b80d48e157309e415aef025ffd84 b/fuzz/corpus/sitemap/7969e0e43da1b80d48e157309e415aef025ffd84 new file mode 100644 index 0000000000000000000000000000000000000000..138801190810e040b860503639963e1aa08bedbb GIT binary patch literal 61 qcmZ>^Db6fO%}p%GtV>C?v&qR%w#z7~DF6xNfdp`g=h!eXFaQA4eij!1 literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/7c99aac7091b9ff1023ebf12897c42c2b5146750 b/fuzz/corpus/sitemap/7c99aac7091b9ff1023ebf12897c42c2b5146750 new file mode 100644 index 000000000..229c38c33 --- /dev/null +++ b/fuzz/corpus/sitemap/7c99aac7091b9ff1023ebf12897c42c2b5146750 @@ -0,0 +1 @@ +https:tst/b?x=1&~1&aMp;y=2XXXXXX>2> diff --git a/fuzz/corpus/sitemap/8596c9456ec637abc8477e42308210956d002e33 b/fuzz/corpus/sitemap/8596c9456ec637abc8477e42308210956d002e33 new file mode 100644 index 0000000000000000000000000000000000000000..50a013a9a0941902d8e758dc5ed75fe78b473190 GIT binary patch literal 73 zcmZRyDYCP($;oH1ugJ|&C`&CW&ddk#lkGU+;6Dg}cs7Xzb~cO*4904Sxds3K`#Uf& O#=a`bIhBd(lf0;vm`Y)@jnPK kKmnNZ|9?SdUP@|(olP-VMFGSx6cclb-|ya^T4Kiq0QylZk^lez literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/89e598f54fea7059c8c28b1b9418d7c0e5a8a028 b/fuzz/corpus/sitemap/89e598f54fea7059c8c28b1b9418d7c0e5a8a028 new file mode 100644 index 0000000000000000000000000000000000000000..7e31191116ccef7852b0e2a59b5245eeb507d73c GIT binary patch literal 34 mcmZ=x%}p%G%u7kNv&qR%w#z7~G19BZ&CyG*vS9!LJ1zjw@d_CL literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/8ac6a34d65103279df1eaaaabfb7823be2f75387 b/fuzz/corpus/sitemap/8ac6a34d65103279df1eaaaabfb7823be2f75387 new file mode 100644 index 0000000000000000000000000000000000000000..845906807da78650a6eb2514716c2ed6b3ad62a0 GIT binary patch literal 55 zcmZRyv9HK2v9rm^PtLPdvdb|jGSvfda==1Ql^7TbAhN|J`V|$nhH8nq1=h`40Pd3# A3jhEB literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/8add9d8dc27b939c444152727c95801a76093b13 b/fuzz/corpus/sitemap/8add9d8dc27b939c444152727c95801a76093b13 new file mode 100644 index 0000000000000000000000000000000000000000..ac8fdb8ff7125bebbc590263fc69ae24ab130c3c GIT binary patch literal 96 zcmZR!Db6fOEzZ<0EwZyIEy}U8$;nT)%P4_TB?VUcdKGpy#W3-bg8x8Zm7xjJmYY~$ Um&;IKlY%4-RIdS(WVEvZ0MaNUqyPW_ literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/8b0798c9bf0a8f1206a8cce5b49c48348a5acb3c b/fuzz/corpus/sitemap/8b0798c9bf0a8f1206a8cce5b49c48348a5acb3c new file mode 100644 index 0000000000000000000000000000000000000000..1db541ca23f1eb0f17608bb7c902ed89dbf3713a GIT binary patch literal 72 zcmZRy$+5G^$xpV+C@Cu_wsHislk5%DloNBUD{Cql>}*OZZS9c6jck%RI8yC^(jgFe J{ZdAGE&ws^6(s-w literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/8cbd6fc35a657c15a117f5fa44b9c5475b5c244e b/fuzz/corpus/sitemap/8cbd6fc35a657c15a117f5fa44b9c5475b5c244e new file mode 100644 index 0000000000000000000000000000000000000000..9a2c422b375878a6d85b3d82079a286f98f86d7f GIT binary patch literal 42 xcmZQ%U|^^v \ No newline at end of file diff --git a/fuzz/corpus/sitemap/8f69378809f4cb3b06cd71b4310246aede68be6c b/fuzz/corpus/sitemap/8f69378809f4cb3b06cd71b4310246aede68be6c new file mode 100644 index 000000000..21c4c7797 --- /dev/null +++ b/fuzz/corpus/sitemap/8f69378809f4cb3b06cd71b4310246aede68be6c @@ -0,0 +1 @@ +io< >/t≈y>/t/b?x=M5≈y=2kvE#nzP!Hgh=0&amTTTT0&amTTTTTTPTseh.tseh.&//tb?x=0&amTTTTTTPTseh.//.0rl> diff --git a/fuzz/corpus/sitemap/94b3ab30db215ec5b044a7dc24fb5e69e8778659 b/fuzz/corpus/sitemap/94b3ab30db215ec5b044a7dc24fb5e69e8778659 new file mode 100644 index 000000000..08d013017 --- /dev/null +++ b/fuzz/corpus/sitemap/94b3ab30db215ec5b044a7dc24fb5e69e8778659 @@ -0,0 +1 @@ +htt/b?xp&m=a;y=4t> \ No newline at end of file diff --git a/fuzz/corpus/sitemap/956a73fe53021543044dfcfb61173044ca7c2a17 b/fuzz/corpus/sitemap/956a73fe53021543044dfcfb61173044ca7c2a17 new file mode 100644 index 000000000..370cac205 --- /dev/null +++ b/fuzz/corpus/sitemap/956a73fe53021543044dfcfb61173044ca7c2a17 @@ -0,0 +1,2 @@ +B1ihttp:// +hx \ No newline at end of file diff --git a/fuzz/corpus/sitemap/97619afecb1c73169b02116ff610543e833a1e60 b/fuzz/corpus/sitemap/97619afecb1c73169b02116ff610543e833a1e60 new file mode 100644 index 0000000000000000000000000000000000000000..6c02067c4289ed12a3d82cb98c0b69ec214aefe0 GIT binary patch literal 130 zcmZRwDb6fO%}p%G%u7kFI70zo)+$RaDmK!q$jxDZfnXSefuW$F0LBT(x5)r0E@4P6 H&ddh@x9>k! literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/97645ac8be26bd22563d7d27fb833d964977a44a b/fuzz/corpus/sitemap/97645ac8be26bd22563d7d27fb833d964977a44a new file mode 100644 index 000000000..0846e4553 --- /dev/null +++ b/fuzz/corpus/sitemap/97645ac8be26bd22563d7d27fb833d964977a44a @@ -0,0 +1 @@ +<https://>< diff --git a/fuzz/corpus/sitemap/9811960a0551fd86e280eb58ed6446c0ecc74fbb b/fuzz/corpus/sitemap/9811960a0551fd86e280eb58ed6446c0ecc74fbb new file mode 100644 index 000000000..06ed47784 --- /dev/null +++ b/fuzz/corpus/sitemap/9811960a0551fd86e280eb58ed6446c0ecc74fbb @@ -0,0 +1 @@ +l>hvv> \ No newline at end of file diff --git a/fuzz/corpus/sitemap/993c132d0386ef7b502bab4e62d26023f0bf0250 b/fuzz/corpus/sitemap/993c132d0386ef7b502bab4e62d26023f0bf0250 new file mode 100644 index 000000000..e225ff1ed --- /dev/null +++ b/fuzz/corpus/sitemap/993c132d0386ef7b502bab4e62d26023f0bf0250 @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/fuzz/corpus/sitemap/99711e0eee687fb4b9305e8b700996ef49616f20 b/fuzz/corpus/sitemap/99711e0eee687fb4b9305e8b700996ef49616f20 new file mode 100644 index 000000000..7338c6567 --- /dev/null +++ b/fuzz/corpus/sitemap/99711e0eee687fb4b9305e8b700996ef49616f20 @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/fuzz/corpus/sitemap/9ac709a1a3dcd5f15b0d655ea2a5a84bb7685596 b/fuzz/corpus/sitemap/9ac709a1a3dcd5f15b0d655ea2a5a84bb7685596 new file mode 100644 index 0000000000000000000000000000000000000000..bffc16a7aad2b18492e12ef7f96794f455b7005a GIT binary patch literal 17 YcmZQ%V6e%_PqwqSvnkHWC^FUq03zxHjQ{`u literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/9be22a3ec737f88e0e61326f822e65952a6aeb7d b/fuzz/corpus/sitemap/9be22a3ec737f88e0e61326f822e65952a6aeb7d new file mode 100644 index 000000000..b908c129b --- /dev/null +++ b/fuzz/corpus/sitemap/9be22a3ec737f88e0e61326f822e65952a6aeb7d @@ -0,0 +1 @@ +https:tseh.//t/b?x=1&Vy=2<https:tseh.//t/b?x=1&y=2 diff --git a/fuzz/corpus/sitemap/9e7d3f551e039e680769c619f850085179ffef3c b/fuzz/corpus/sitemap/9e7d3f551e039e680769c619f850085179ffef3c new file mode 100644 index 000000000..37325a786 --- /dev/null +++ b/fuzz/corpus/sitemap/9e7d3f551e039e680769c619f850085179ffef3c @@ -0,0 +1 @@ +><> h diff --git a/fuzz/corpus/sitemap/9fbbd38632a5e00322c3c1182b5de821d96d01c4 b/fuzz/corpus/sitemap/9fbbd38632a5e00322c3c1182b5de821d96d01c4 new file mode 100644 index 000000000..2757688b7 --- /dev/null +++ b/fuzz/corpus/sitemap/9fbbd38632a5e00322c3c1182b5de821d96d01c4 @@ -0,0 +1,2 @@ +>> +//uset> diff --git a/fuzz/corpus/sitemap/a6ca50dcffc4ceb4fde9cb48683e6e6179bc9d17 b/fuzz/corpus/sitemap/a6ca50dcffc4ceb4fde9cb48683e6e6179bc9d17 new file mode 100644 index 0000000000000000000000000000000000000000..594ef69d8435fcf00e2c652278766c86f40a8b84 GIT binary patch literal 67 zcmZRyv9HL@QFw0800cHU`N?+nb|yuo#rpb(>@xK1z#> -- diff --git a/fuzz/corpus/sitemap/ab4b1bb206a56cd96dc06daf7782f1880a6cac6e b/fuzz/corpus/sitemap/ab4b1bb206a56cd96dc06daf7782f1880a6cac6e new file mode 100644 index 000000000..4e3a00a82 --- /dev/null +++ b/fuzz/corpus/sitemap/ab4b1bb206a56cd96dc06daf7782f1880a6cac6e @@ -0,0 +1 @@ +http://h.test/x.bluset> E02GlFZiwcRN?SWegcPRXrHt}i0HxY3kN^Mx literal 0 HcmV?d00001 diff --git a/fuzz/corpus/sitemap/b721ef14aa0f27fb82d7f1db3020ceaa87377f6c b/fuzz/corpus/sitemap/b721ef14aa0f27fb82d7f1db3020ceaa87377f6c new file mode 100644 index 000000000..4dc63560f --- /dev/null +++ b/fuzz/corpus/sitemap/b721ef14aa0f27fb82d7f1db3020ceaa87377f6c @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/fuzz/corpus/sitemap/9ac709a1a3dcd5f15b0d655ea2a5a84bb7685596 b/fuzz/corpus/sitemap/9ac709a1a3dcd5f15b0d655ea2a5a84bb7685596 deleted file mode 100644 index bffc16a7aad2b18492e12ef7f96794f455b7005a..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 17 YcmZQ%V6e%_PqwqSvnkHWC^FUq03zxHjQ{`u diff --git a/fuzz/corpus/sitemap/9be22a3ec737f88e0e61326f822e65952a6aeb7d b/fuzz/corpus/sitemap/9be22a3ec737f88e0e61326f822e65952a6aeb7d deleted file mode 100644 index b908c129b..000000000 --- a/fuzz/corpus/sitemap/9be22a3ec737f88e0e61326f822e65952a6aeb7d +++ /dev/null @@ -1 +0,0 @@ -https:tseh.//t/b?x=1&Vy=2<https:tseh.//t/b?x=1&y=2 diff --git a/fuzz/corpus/sitemap/9e7d3f551e039e680769c619f850085179ffef3c b/fuzz/corpus/sitemap/9e7d3f551e039e680769c619f850085179ffef3c deleted file mode 100644 index 37325a786..000000000 --- a/fuzz/corpus/sitemap/9e7d3f551e039e680769c619f850085179ffef3c +++ /dev/null @@ -1 +0,0 @@ -><> h diff --git a/fuzz/corpus/sitemap/9fbbd38632a5e00322c3c1182b5de821d96d01c4 b/fuzz/corpus/sitemap/9fbbd38632a5e00322c3c1182b5de821d96d01c4 deleted file mode 100644 index 2757688b7..000000000 --- a/fuzz/corpus/sitemap/9fbbd38632a5e00322c3c1182b5de821d96d01c4 +++ /dev/null @@ -1,2 +0,0 @@ ->> -//uset> diff --git a/fuzz/corpus/sitemap/a6ca50dcffc4ceb4fde9cb48683e6e6179bc9d17 b/fuzz/corpus/sitemap/a6ca50dcffc4ceb4fde9cb48683e6e6179bc9d17 deleted file mode 100644 index 594ef69d8435fcf00e2c652278766c86f40a8b84..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 67 zcmZRyv9HL@QFw0800cHU`N?+nb|yuo#rpb(>@xK1z#> -- diff --git a/fuzz/corpus/sitemap/ab4b1bb206a56cd96dc06daf7782f1880a6cac6e b/fuzz/corpus/sitemap/ab4b1bb206a56cd96dc06daf7782f1880a6cac6e deleted file mode 100644 index 4e3a00a82..000000000 --- a/fuzz/corpus/sitemap/ab4b1bb206a56cd96dc06daf7782f1880a6cac6e +++ /dev/null @@ -1 +0,0 @@ -http://h.test/x.bluset> E02GlFZiwcRN?SWegcPRXrHt}i0HxY3kN^Mx diff --git a/fuzz/corpus/sitemap/b721ef14aa0f27fb82d7f1db3020ceaa87377f6c b/fuzz/corpus/sitemap/b721ef14aa0f27fb82d7f1db3020ceaa87377f6c deleted file mode 100644 index 4dc63560f..000000000 --- a/fuzz/corpus/sitemap/b721ef14aa0f27fb82d7f1db3020ceaa87377f6c +++ /dev/null @@ -1 +0,0 @@ -