Skip to content

Commit 3467275

Browse files
committed
cpp: deduplicate pages by canonical wiki page, not by name
The entries filter kept a process-wide list of entry names and returned no entries for any page whose name had been seen before. Since a page without entries is not stored at all, this silently dropped whole documents rather than just index entries. Unrelated pages legitimately share a name: every container's erase_if page is titled "std::erase_if (std::<container>)", which normalises to plain "std::erase_if", so only the first one crawled survived. Same for std::move in <algorithm> vs <utility>, and std::optional::operator bool. The deduplication exists because cppreference serves the same wiki page under several URLs and the scraper stores pages under the requested URL rather than the effective one. Key it on the canonical page name from the "Retrieved from" footer instead, which identifies the underlying wiki page exactly. Fixes #2175 Fixes #2190 Fixes #2223
1 parent f280950 commit 3467275

1 file changed

Lines changed: 27 additions & 11 deletions

File tree

lib/docs/filters/cpp/entries.rb

Lines changed: 27 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
module Docs
22
class Cpp
33
class EntriesFilter < Docs::EntriesFilter
4-
@@duplicate_names = []
4+
@@canonical_pages = []
55

66
REPLACE_NAMES = {
77
'Error directive' => '#error directive',
@@ -65,18 +65,34 @@ def format_name(name)
6565
REPLACE_NAMES[name] || name
6666
end
6767

68-
# Avoid duplicate pages, these duplicate page are the same page for
69-
# multiple functions that are organized in the same page because provide
70-
# similar behavior but have different name.
68+
# Avoid duplicate pages: cppreference serves the same wiki page under
69+
# several URLs (redirects), and since the scraper stores pages under the
70+
# requested URL rather than the effective one, each alias would be stored
71+
# as a separate copy of the same document.
72+
#
73+
# Deduplicate on the canonical wiki page name taken from the "Retrieved
74+
# from" footer, not on the entry name -- unrelated pages legitimately
75+
# share a name (e.g. std::erase_if for every container, std::move in both
76+
# <algorithm> and <utility>) and keying on the name silently dropped them.
7177
def entries
72-
entries = []
78+
return [] if duplicate_page?
79+
super
80+
end
7381

74-
if !(@@duplicate_names.include?(name))
75-
@@duplicate_names.push(name)
76-
entries << default_entry if root_page? || include_default_entry?
77-
entries.concat(additional_entries)
78-
build_entries(entries)
79-
end
82+
def duplicate_page?
83+
page = canonical_page
84+
return false if page.nil?
85+
return true if @@canonical_pages.include?(page)
86+
87+
@@canonical_pages.push(page)
88+
false
89+
end
90+
91+
# e.g. "cpp/container/unordered_map/erase_if", from the printfooter link
92+
# to /mwiki/index.php?title=<page>&oldid=<revision>
93+
def canonical_page
94+
href = at_css('.printfooter a').try(:[], 'href')
95+
href && href[/[?&]title=([^&]+)/, 1]
8096
end
8197

8298
end

0 commit comments

Comments
 (0)