Improve scrapers content extraction

output.txt

@@ -0,0 +1,119 @@
+This page will help you plan your courses and academic career as you complete your Bachelor of Cybersecurity degree program.
+
+When selecting your courses for the term, it is helpful to review the   Program Requirements [Link: https://calendar.carleton.ca/undergrad/undergradprograms/computerscience/] and your academic audit [Link: https://carleton.ca/academicadvising/how-to-read-your-audit/] . If you have questions regarding course selection, please contact our undergraduate advisor [Link: https://carleton.ca/scs/our-people/#ug-advisors] team.
+
+At any time you can build a sample timetable within Carleton Central or through the  Public Class Schedule [Link: http://central.carleton.ca/prod/bwysched.p_select_term?wsea_code=EXT] .
+
+Course Descriptions and Outlines
+
+Registration
+
+First Year Registration Guide
+
+Academic Audit
+
+Prerequisites
+
+Before you Register
+
+After you Register
+
+Course Patterns
+
+Electives and Prohibited Courses
+
+Course Selection FAQs
+
+Course Descriptions and Outlines
+
+Course Descriptions [Link: https://carleton.ca/scs/current-students/undergraduate-students/courses-and-registration/]
+
+Course Outlines [Link: https://outline.scs.carleton.ca]
+
+Registration
+
+First Year Registration Guide
+
+Welcome to Carleton and your Bachelor of Cybersecurity degree program! Please read this degree guide prior to registering.
+
+First Year Registration Guide [Link: https://carleton.ca/registration/course-selection-guide/bcyber/]
+
+Academic Audit
+
+The academic audit is the ‘road map’ to academic success. It lists which courses you are required to take in order to complete your degree at Carleton. The audit will inform you of the progress you have already made, as well as show you the courses that you will need to take in the future. Remember to check your audit regularly to ensure you are fulfilling your degree requirements.
+
+Understanding your Academic Audit ↗ [Link: https://carleton.ca/academicadvising/how-to-read-your-audit/]
+
+Prerequisites and Restrictions
+
+A prerequisite is a course that must be successfully completed (with a minimum grade of D- or above unless otherwise noted) before you are able to register into the course. A course may also have additional restrictions that require students be in a particular program, and/or have a specific year standing (first, second, third of fourth) in order to register.
+
+The School of Computer Science strictly enforces calendar prerequisites .  If you do not meet the prerequisite(s) of a COMP or CSEC course , you will be deregistered. More information on deregistration can be found in Section 2.1.7 of the Undergraduate Calendar [Link: https://calendar.carleton.ca/undergrad/regulations/academicregulationsoftheuniversity/registration-evaluation-records/#deregistration] . This also applies to DEF (deferred final exam); once the final grade for that course is posted, you will be deregistered from any current course if the prerequisite(s) is not met. As deferred exams are always held after the registration period closes, if you are deregistered you will not be able to add more courses to your class schedule. Please be aware that deregistration can happen at anytime during the term which may affect your status as a full-time student. Please see this link [Link: https://carleton.ca/academicadvising/reduced-course-load/]  to determine who you should speak with if your student status changes. Please note that students with a GNA (grade not available) in a course may also be affected by this policy.
+
+Before you register
+
+Course registration is done online through Carleton Central [Link: http://central.carleton.ca/] .  It is your one-stop service for registration, student records, admission, and fee information.
+
+Refer to the Registrar’s Office website [Link: https://carleton.ca/registration/] for how-to videos to assist you in navigating the Carleton Central registration system. You can view the registration open and deadline dates on the Registration Dates and Deadlines webpage [Link: https://students.carleton.ca/academic-dates/] .
+
+After You Register
+
+Pay your tuition by the payment deadline each term to avoid penalties or late fees. The Student Accounts office has information on how to make a payment [Link: https://carleton.ca/studentaccounts/fee-payment/] .
+
+Course Patterns
+
+It is recommended that all B.Cyber. students register in courses each term according to the course pattern listed below.
+
+Students should refer to the course pattern of their Catalog Year. In most cases, a student’s Catalog Year will match the term a student was admitted to the B.Cyber. program, but should be confirmed by reviewing your audit. [Link: https://carleton.ca/academicadvising/how-to-read-your-audit/] The Catalog Year is listed in the upper-right hand corner of the audit.
+
+BCYBER Course Map 202630 (pdf) [PDF: https://carleton.ca/scs/wp-content/uploads/sites/260/BCyber-Course-Map-202630-3.pdf] – (admit term Fall 2026)
+
+BCYBER Course Map 202530 (pdf) [PDF: https://carleton.ca/scs/wp-content/uploads/sites/260/FINAL2-BCyber-Course-Map-202530.pdf] – (admit term Fall 2025)
+
+
+
+Electives and Prohibited Courses
+
+Electives are courses required for completion of your 20.0 credit degree program. Many first year students find it difficult to choose electives. Our best advice is to choose a topic you enjoyed in high school or are generally interested in studying.
+
+Find course titles and descriptions  in the courses [Link: https://calendar.carleton.ca/undergrad/courses/] section of the undergraduate calendar.
+
+Read the course descriptions to ensure you have the prerequisites , and then check the Public Class Schedule in Carleton Central to see when the course is offered.
+
+Be aware of  preclusions in the course descriptions. If a course precludes additional credit for another course, that means you cannot get credit for both of those courses. Please note that preclusion does not mean equivalent , though in some cases it may be. Please contact your undergraduate advisor for further clarification.
+
+If you encounter a restriction when trying to register, please submit a Registration Override Request on Carleton Central, and the department owning the course will get back to you.
+
+Free Electives
+
+Free Electives are all courses offered by the Faculty of Arts and Social Sciences, the Faculty of Public Affairs, the Sprott School of Business, the Faculty of Science and the Faculty of Engineering and Design except for courses in the Prohibited Courses category. Free electives can include COMP, CSEC, DATA, MATH and STAT courses.
+
+Prohibited Courses
+
+Prohibited courses cannot be used for credit towards any program requirement (ie. electives, minors, etc.) for students enrolled in a B.Cyber. program. Please always refer to the most recent Undergraduate Calendar [Link: https://calendar.carleton.ca/undergrad/undergradprograms/computerscience/] for the most up-to-date list of prohibited courses , and for all prerequisites and preclusions.
+
+IMPORTANT: Maximum Number of Credits Below the 2000-level.
+
+A student may count a maximum of 7.0 credits below the 2000-level toward fulfillment of graduation requirements.
+
+Credits in excess of this limit will be set aside as Extra to the Degree (ETD), No Credit for the Degree (NCD), or Forfeit. This allows students to increase their CGPA by pushing out low grades below the 2000-level through replacement by higher grades at the same level.
+
+Course Selection FAQs
+
+Here are some frequently asked questions regarding course selection:
+
+What is a prerequisite?
+
+Some courses require you to take one or more prerequisite s in order to get into the course. A prerequisite is a course that must be successfully completed (with a minimum grade of D- or above unless otherwise noted) BEFORE you are able to register into the course. By first taking all of the required prerequisite course(s), this will ensure that your knowledge level is adequate for you to understand the material in the course that you are trying to register into. Always look at the most current Undergraduate Calendar [Link: http://calendar.carleton.ca/undergrad/] in order to determine which prerequisites are required for the course that you are trying to take. Some COMP courses will require a prerequisite grade that is higher than D-. The School of Computer Science strictly enforces prerequisites as outlined in the calendar.
+
+What is a preclusion?
+
+When two courses have very similar (or overlapping) material, the calendar often indicates that the courses will preclude one another. This means that you cannot have both of these courses count as credit towards your degree requirements. Precluded courses are not necessarily equivalent. Do not assume that one course can substitute for another simply because it precludes the other course. You should always check with your Undergraduate Advisors to be sure. Note that if you take a course that precludes another course that you have already taken, the most recent grade will count. Preclusions are listed in the course description in the Undergraduate Calendar [Link: http://calendar.carleton.ca/undergrad/] .
+
+Where are the CSEC courses on the Public Class Schedule?
+
+The Bachelor of Cybersecurity is introducing all new CSEC courses as part of the curriculum. These courses will be developed, and released, as the first cohort of students progress through their program. 2000-level CSEC courses will be released in the 2026-27 Academic Year, 3000-level CSEC courses will be released in the 2027-28 Academic Year, and finally the remaining 4000-level CSEC courses will be released in the 2028-29 Academic Year. As of May 2025 all core CSEC courses will be listed in the Undergraduate Calendar [Link: http://calendar.carleton.ca/undergrad/] , but will not appear on the public class schedule until they are offered for the first time.
+
+Do I have to follow the course pattern for my admit year?
+
+Students are encouraged to follow the provided course pattern towards graduation, as it will guarantee a conflict free schedule each year in addition to ensuring you will meet all prerequisite requirements each term. If you are off pattern (or plan to deviate) from your provided course pattern, you should check in with one of your Undergraduate Advisors [Link: https://carleton.ca/scs/our-people/school-of-computer-science-administration/administrative-staff/undergraduate-advisor/] to discuss some considerations for planning your courses towards graduation.Courses and Registration (B.Cyber.) - School of Computer Science

pyproject.toml

@@ -11,11 +11,11 @@ dependencies = [
     "pgvector",
     "alembic",
     "httpx",
-    "trafilatura",
     "langchain-text-splitters",
     "celery",
     "redis",
     "discord-py>=2.7.1",
+    "beautifulsoup4>=4.15.0",
 ]
 
 [dependency-groups]

src/ingestion/scrapers/html_scraper.py

@@ -1,43 +1,88 @@
-import httpx
-import trafilatura
-
-from src.config.logger import get_logger
-
-log = get_logger(__name__)
-
-
-def scrape(url: str) -> tuple[str, str | None]:
-    """Fetch a URL and extract clean text + title.
-
-    Returns (text, title). Title may be None if not found. Text may be empty
-    if trafilatura couldn't extract anything; the caller should check and skip.
-    """
-    log.info("scrape_started", url=url)
-    with httpx.Client(timeout=30, follow_redirects=True) as client:
-        response = client.get(url)
-        response.raise_for_status()
-
-    # Known limitations (tracked as a separate ticket):
-    #  - Misses content inside aria-hidden="true" accordions (common on FAQ pages)
-    #  - Misses question text but gets answer text on other FAQ pages, we should get both
-    #  - Link formatting is markdown-style and may need cleanup downstream
-    # Improving extraction quality is a tuning ticket, not a blocker for development.
-    text = (
-        trafilatura.extract(
-            response.text,
-            favor_recall=True,
-            include_links=True,
-            include_tables=True,
-            include_formatting=False,
-            include_comments=False,
-            deduplicate=True,
-            output_format="txt",
-        )
-        or ""
-    )
-
-    metadata = trafilatura.extract_metadata(response.text)
-    title = metadata.title if metadata else None
-
-    log.info("scrape_complete", url=url, chars=len(text), title=title)
-    return text, title
+from urllib.parse import urlsplit
+import httpx
+from bs4 import BeautifulSoup
+import re
+
+from src.config.logger import get_logger
+
+log = get_logger(__name__)
+
+_BOILERPLATE_TAGS = ["header", "footer", "nav", "script", "style", "noscript"]
+_BOILERPLATE_SELECTORS = (
+    ".navbar-container, .navbar, .footer, .global-nav, "
+    ".navigation, .topbar, .content__meta, .visuallyhidden, "
+    ".resource__contributors-list, #on-this-page, .resource__sources__heading"
+)
+_BLOCK_TAGS = ["p", "div", "li", "h1", "h2", "h3", "h4", "h5", "h6",
+               "tr", "section", "article", "ul", "ol", "table", "br", "blockquote"]
+
+def _extract_from_html(html: str) -> tuple[str, str | None]:
+    log.info("html_extract_start", html_len=len(html))
+
+    cut_marker = "<!-- close main-wrapper"
+    if cut_marker in html:
+        html = html.split(cut_marker)[0]
+
+    soup = BeautifulSoup(html, "html.parser")
+
+    root = soup.select_one("main, [role=main], #main-content")
+    if root is None:  # CCSS-style pages have no <main>
+        root = soup.body or soup
+
+    # still might need to clean some things up
+    for t in soup(_BOILERPLATE_TAGS):
+        t.decompose()
+    for el in soup.select(_BOILERPLATE_SELECTORS):
+        el.decompose()
+
+    log.info("boilerplate_removed")
+
+    # formatting links after clearing having cleared clutter
+    for link in root.find_all("a"):
+        href = (link.get("href") or "").strip()
+        if href and not href.startswith("#"):
+            text = link.get_text(" ", strip=True)
+            is_pdf = urlsplit(href).path.lower().endswith(".pdf")
+            marker = "PDF" if is_pdf else "Link"
+            link.replace_with(f"{text} [{marker}: {href}] ")
+
+    log.info("links_parsed")
+
+    for tag in root.find_all(_BLOCK_TAGS):
+        tag.append("\n")
+
+    text = root.get_text(" ", strip=False)
+    text = re.sub(r"[ \t]+", " ", text)
+    text = re.sub(r" *\n *", "\n", text)
+    text = re.sub(r"\n{3,}", "\n\n", text)
+    text = text.strip()
+
+    title = soup.title.get_text(strip=True) if soup.title else None
+
+    log.info("html_extract_complete", text_len=len(text), has_title=title is not None)
+    return text, title
+
+
+def scrape(url: str) -> tuple[str, str | None]:
+    """Fetch a URL and extract clean text + title.
+
+    Returns (text, title). Title may be None if not found.
+    """
+    log.info("scrape_called", url=url)
+
+    log.info("http_request_start", url=url)
+    with httpx.Client(timeout=30, follow_redirects=True) as client:
+        response = client.get(url)
+        response.raise_for_status()
+
+    log.info(
+        "http_request_success",
+        url=url,
+        status_code=response.status_code,
+        bytes=len(response.text),
+    )
+
+    log.info("html_parsing_start", url=url)
+    text, title = _extract_from_html(response.text)
+    log.info("scrape_complete", url=url, chars=len(text), title=title)
+    return text, title