Skip to content

feat: Show crawl page progress - #3539

Merged
ikreymer merged 8 commits into
mainfrom
frontend-crawl-page-progress
Aug 1, 2026
Merged

feat: Show crawl page progress#3539
ikreymer merged 8 commits into
mainfrom
frontend-crawl-page-progress

Conversation

@SuaYoo

@SuaYoo SuaYoo commented Jul 24, 2026

Copy link
Copy Markdown
Member

Resolves #3524

Changes

  • Displays progress of page crawled so far in crawl runs list
  • Displays number of found pages in crawl runs list, removing "Run by" column to recover more space
  • Sort superadmin active crawls by "Pages Crawled"
  • Replaces irrelevant QA column of latest crawl stats with "Pages Found"
  • Makes latest crawl in workflow crawl history clickable, removing redundant "Watch crawl" notice
  • Documents crawl run history in user guide

Manual testing

  1. Log in as crawler
  2. Go to "Crawling"
  3. Start a workflow crawl
  4. Go to "Crawl Runs"
  5. Wait for crawl to begin running. Verify page crawl progress is displayed
  6. Go to running workflow
  7. Go to "Crawls". Verify page crawl progress is displayed
  8. Click running crawl. Verify navigation to "Latest Crawl" tab

Screenshots

Page Image/video
Crawl Runs Screenshot 2026-07-24 at 1 38 51 PM
Workflow / Crawls Screenshot 2026-07-24 at 1 46 44 PM
Workflow / Latest Crawl Screenshot 2026-07-24 at 2 21 52 PM
Superadmin / Active crawls Screenshot 2026-07-24 at 1 50 40 PM
User guide / Intro to Crawl Workflows Screenshot 2026-07-24 at 2 08 14 PM
User guide / Running Crawls Screenshot 2026-07-24 at 2 20 51 PM

@mistydemeo mistydemeo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gave this a test locally, and working well! Looks good too. I think this is a nice win for usability.

Do we keep track of how many pages we found after a crawl finishes? It would be interesting to be able to show this for completed crawls too, or at least for stopped crawls where it was manually halted before it got to everything.

@emma-sg emma-sg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@ikreymer
ikreymer merged commit 0dec827 into main Aug 1, 2026
29 of 30 checks passed
@ikreymer
ikreymer deleted the frontend-crawl-page-progress branch August 1, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task]: Show page progress in running crawls list

4 participants