Add Chrome mask test to autowebcompat-repro agent#6226
Conversation
ksy36
changed the title
ksy36
force-pushed
the
autowebcompat-chrome-mask
branch
from
aeeaf14 to
482f326
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds support for running a “Chrome Mask” (Chrome UA spoofing) experiment as part of the autowebcompat-repro agent flow by provisioning a Firefox profile with the Chrome Mask extension installed and enabling the DevTools MCP privileged-context features needed to manage/verify the extension.
Changes:
- Add
setup_profile.pyto build a Firefox profile and preinstall AMO extensions (used for Chrome Mask). - Extend DevTools MCP server config to optionally enable privileged-context tools + system access env var.
- Update agent result schema and system prompt to include/report
chrome_mask_fixedand document the Chrome Mask procedure.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| agents/autowebcompat-repro/hackbot_agents/autowebcompat_repro/setup_profile.py | New helper to download/install AMO XPI(s) into a profile and warm-launch Firefox to register them. |
| agents/autowebcompat-repro/hackbot_agents/autowebcompat_repro/result.py | Adds chrome_mask_fixed field to the structured result model/schema. |
| agents/autowebcompat-repro/hackbot_agents/autowebcompat_repro/prompts/system.md | Updates instructions to run the Chrome Mask test only after baseline reproduction and to verify UA spoofing. |
| agents/autowebcompat-repro/hackbot_agents/autowebcompat_repro/devtools_mcp.py | Adds enable_privileged_context option and wires --enable-privileged-context + env var. |
| agents/autowebcompat-repro/hackbot_agents/autowebcompat_repro/agent.py | Plumbs optional chrome_mask_profile through to DevTools MCP config and privileged-context toggle. |
| agents/autowebcompat-repro/hackbot_agents/autowebcompat_repro/main.py | Installs Firefox Nightly and now builds/passes a Chrome Mask preinstalled profile into the agent run. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| 1. Identify the affected URL and the described broken behavior. | ||
| 2. Navigate to the URL using the Firefox DevTools MCP and try to reproduce the issue. | ||
| 3. Submit your findings via `submit_result` (see "Reporting your result"). | ||
| 2. Baseline: Navigate to the URL with the Firefox DevTools MCP and |
There was a problem hiding this comment.
I think this is OK for now, but a lot of this is a set of deterministic steps. We don't need an LLM to execute things that we can represent directly in code; it's more token-heavy and less deterministic. I'd rather put this control flow in the harness and just use the LLM for the parts that aren't just running through an algorithm.
There was a problem hiding this comment.
Yeah, another option could be to have one profile without Chrome mask and another with Chrome mask but enabled globally. Or maybe it's possible to enable it for a given domain with a script that the agent will run. I can try a few options after we launch this first version.
| Raises ``RuntimeError`` if an extension does not end up registered and | ||
| enabled in the profile. | ||
| """ | ||
| parent = Path(tempfile.mkdtemp(prefix="ff-profile-")) |
There was a problem hiding this comment.
It looks like we end up leaking this temporary directory. Is that expected (e.g. because we end up destroying the entire container) or should there be some teardown code that runs at the end of each agent run?
There was a problem hiding this comment.
I've added a cleanup on error and after a successful run. But yes I assumed a container is destroyed after each run
ksy36
force-pushed
the
autowebcompat-chrome-mask
branch
from
3886492 to
fb3ef6d
Compare
This PR adds a script that creates a profile and installs Chrome mask extension from AMO and passes the profile to devtools MCP.