Commit 3467275
committed
cpp: deduplicate pages by canonical wiki page, not by name
The entries filter kept a process-wide list of entry names and returned no
entries for any page whose name had been seen before. Since a page without
entries is not stored at all, this silently dropped whole documents rather
than just index entries.
Unrelated pages legitimately share a name: every container's erase_if page
is titled "std::erase_if (std::<container>)", which normalises to plain
"std::erase_if", so only the first one crawled survived. Same for std::move
in <algorithm> vs <utility>, and std::optional::operator bool.
The deduplication exists because cppreference serves the same wiki page
under several URLs and the scraper stores pages under the requested URL
rather than the effective one. Key it on the canonical page name from the
"Retrieved from" footer instead, which identifies the underlying wiki page
exactly.
Fixes #2175
Fixes #2190
Fixes #22231 parent f280950 commit 3467275
1 file changed
Lines changed: 27 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
68 | | - | |
69 | | - | |
70 | | - | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
71 | 77 | | |
72 | | - | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
73 | 81 | | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
80 | 96 | | |
81 | 97 | | |
82 | 98 | | |
| |||
0 commit comments