Sitemap files are never read, so URLs nothing links to are never found#718
Open
xroche wants to merge 16 commits into
Open
Sitemap files are never read, so URLs nothing links to are never found#718xroche wants to merge 16 commits into
xroche wants to merge 16 commits into
Conversation
HTTrack finds URLs only by parsing links, so anything a site publishes solely in its sitemap stayed invisible: robots.txt was already parsed, but its Sitemap: lines were ignored and nothing else in the tree touched sitemaps. Adds opt-in --sitemap (-%m), which probes the start host's robots.txt and falls back to /sitemap.xml, and --sitemap-url (-%mu) for an explicit document. Handles <urlset> and nested <sitemapindex>, plain or gzipped. Discovered URLs enter with the full depth budget but still go through the wizard, so filters and scope rules decide; a sitemap is not a filter bypass. The parser reads attacker-controlled XML off the network, so it is capped on URL count, index nesting, decompressed size and decompression ratio, and child sitemaps must stay on the host that named them. Closes #712 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
# Conflicts: # tests/Makefile.am
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
lienrelatif() walked back from the last character of its current-path argument without checking the path was non-empty, reading one byte before the stack buffer. htsAddLink is the first caller to pass an empty savename, which sitemap documents have because they are ingested rather than mirrored, so ASan caught it on the new crawl test. The parser drops a value whose numeric character reference decodes outside printable ASCII, rather than leaving the reference verbatim and seeding a URL the site never published, and classifies a document by its real root element, so a comment naming the other one no longer flips urlset and sitemapindex. The robots.txt line reader is bounded by the body size instead of relying on a NUL terminator. The self-test moves to 01_zlib-sitemap.test: MSan runs 01_engine-* only, because an uninstrumented libz floods it with false positives. Tests gain the assertions the earlier ones were missing: which of the robots.txt route and the /sitemap.xml fallback was taken, that the sitemap documents stay out of the mirror, that the off-host child sitemap is refused, the sitemapindex nesting cap, the per-document URL cap at its production value, and copy_htsopt coverage for the two new fields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
deflate tops out near 1032:1, so hts_codec_maxout never binds before the 64 MiB cap; the old comment implied a ratio guard that cannot fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
…ate it An adversarial review found that a <sitemapindex> <loc>, and a robots.txt Sitemap: line, went straight to hts_record_link: the request went out even when a -* rule or robots.txt Disallow covered it. Only the <urlset> half ran through the wizard. Gate the document itself on the filters and on robots.txt, which is all that can apply: the wizard proper wants a referring link, and its up/down travel rules would judge a child sitemap against the parent sitemap's own directory. The robots.txt probe is exempt, being the request that fetches the rules. A 301 also used to end ingestion silently, since the engine re-queues the target as a fresh link that carried no sitemap marking. That hit any site redirecting http to https. The marking now follows the redirect. The "N URL(s) added" counter reported what the scanner emitted rather than what was taken, which hid both of the above; it now reads "N of M". The fallback to /sitemap.xml keys on the same corrected count, so a robots.txt whose only Sitemap: line is off-host or filtered still falls back. Root classification skips a UTF-8 BOM and an XML namespace prefix, and the doc list is cleared when a mirror starts rather than only when it ends. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
# Conflicts: # tests/Makefile.am
…e start URL A nine-agent review found a scope escape: the sitemap document was its own `premier`, so the wizard measured travel from wherever the site chose to put its sitemap. A root /sitemap.xml therefore widened a /deep/dir/ crawl to the whole host. The ingester now points the wizard at the crawl's own start link and lets each seeded URL become its own anchor, which is what a command-line seed gets. Robots handling was both mistimed and undifferentiated. The Sitemap: lines are now collected by robots_parse, on the same body in the same fetch, and acted on after the parsed rules are installed rather than before; and the decision comes from a new hts_robots_forbids extracted out of the wizard, so the sitemap path inherits the -s1 filters-win override instead of a stricter hand-rolled check. The four fetches are no longer treated alike: a sitemap the user names is user intent, one the site declares invites the fetch, only the guessed /sitemap.xml obeys a Disallow, and the URLs listed inside stay fully gated. Also: hts_unescapeEntities replaces the private entity decoder, whose guard tests and fuzz corpus it silently forfeited; hts_codec_head replaces hts_zhead, which is only defined under HTS_USEZLIB; the composed URL buffer now fits two maximal components plus a scheme, which a 2046-byte --sitemap-url reached; the bounded search is promoted to htstools as hts_memstr; and the live state moves from httrackp into htsoptstate, leaving two installed fields rather than three. Tests gain the scope escape, the three robots cases, a cap-boundary control, the handler invocation count and a compression-bomb decode. Every one was checked against a deliberately broken build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…ted Sitemap: lines The file header called the scanner fuzzable while fuzz/ registered ten harnesses and none for it. fuzz-sitemap feeds it raw XML, gzip-framed bodies and truncated streams off a heap copy of exactly the input size, so an overread is an ASan report rather than a quiet pass, with a four-file seed corpus. 60000 runs clean under ASan+UBSan. robots_parse now drops a Sitemap: line that filled its scratch buffer instead of handing on the half URL it was truncated to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
A libFuzzer run writes its finds into the first corpus directory, and 191 of them were committed with the harness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
snprintf returns the length it wanted to write, so accumulating it blind lets the next offset and size argument walk past the buffer. Guard each step the way the argv builder above already does, and give the per-URL loop a real remaining-space bound instead of a fixed 33. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <roche@httrack.com> # Conflicts: # tests/Makefile.am
The headers were copied from an existing file and kept its 1998 year. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <roche@httrack.com> # Conflicts: # src/htsselftest.c # tests/Makefile.am
htsoptstate is embedded by value as httrackp.state, so a field at its tail shifts every httrackp member declared after it: an offsetof probe put warc_file at 141752 on master and 141760 on the branch. Move the pointer to httrackp's own tail, where every existing offset holds and copy_htsopt still ignores it. Also renumber the crawl test to 89, master having taken 87 and 90, and give the new option8 checkbox the hidden companion that 90_webhttrack-checkbox-clear requires, plus its row in that test's table. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
added a commit
that referenced
this pull request
Jul 26, 2026
Conflict in tests/Makefile.am only: master took 87 and 90, PR #718 holds 88, so the change-report test moves to 89. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HTTrack finds URLs by parsing links, so anything a site publishes only in its sitemap is invisible to it.
robots.txtwas already fetched and parsed, but itsSitemap:lines were ignored and nothing else in the tree touched sitemaps.--sitemap(-%m) now probes the start host'srobots.txtand falls back to/sitemap.xml;--sitemap-url URL(-%mu) names one directly, and the two are additive. Both read<urlset>and nested<sitemapindex>, plain or gzipped. It stays off by default, because a sitemap can list thousands of pages nothing links to.What is worth reviewing is how much authority a sitemap gets. URLs listed inside one are seeded with a command-line seed's depth budget but still go through the full wizard, and they are judged against the crawl's own start URL rather than against wherever the site chose to put its sitemap; otherwise a root sitemap would widen a
/deep/dir/crawl to the whole host. The documents themselves are gated by who asked for them: one you name is user intent, one the site declares inrobots.txtis an invitation, and only the guessed/sitemap.xmlobeys aDisallow. Child sitemaps must stay on the host that named them, and the<loc>scanner is a small dedicated one in the newsrc/htssitemap.c, capped on URL count, document count, index nesting and decompressed size.Two fields are tail-appended to
struct httrackpin the installedhtsopt.hand one tohtsoptstate; nothing is reordered, and noVERSION_INFObump here.-#test=sitemapcovers the parser, its caps and the compression path, whiletests/87_local-sitemap.testruns twelve loopback crawls: the scope case above, the three robots cases, the nesting cap boundary, a redirected sitemap, and controls with the option off and with a filter refusing a sitemap URL. Every assertion was checked against a deliberately broken build. The option is also wired into the WebHTTrack GUI,--help, the man page and the command-line guide, with the other 28lang/*.txtlocales falling back to French as--warcdid.The branch also carries the one-line
lienrelatif()guard from #729, which its crawl test needs to stay green under ASan. That fix belongs to #729, which explains and evidences it.Closes #712