diff --git a/main.py b/main.py index 016e47b..87d155f 100644 --- a/main.py +++ b/main.py @@ -70,6 +70,17 @@ def run(log_level: str = "CRITICAL"): instructions = """ # Perfecto MCP Server +## Proactive Knowledge Consultation + +- **ALWAYS consult Perfecto Skills and Help tools first** before answering questions about Perfecto features, authoring AI Scriptless tests, interpreting results, troubleshooting, or providing recommendations. +- **Use `perfecto_skills`**: Access specialized knowledge about Perfecto capabilities (starting with AI Scriptless), best practices, and official getting-started guidance. +- **Use `perfecto_help`**: Consult live Perfecto documentation (category_id='perfecto', subcategory_id_list=['ide'] for AI Scriptless). +- **Golden rule**: If you're not 100% certain about something related to Perfecto, consult Skills or Help first, and if you need to search online, prioritize help.perfecto.io. + +## Important Guidelines +- **Batch Operations**: When making multiple calls to the same tool, check if that tool supports a `batch` action and use it instead of separate calls. +- **Don't assume / Don't invent**: If something is unclear, consult Skills/Help before responding. +- **Provide resources**: Include markdown-formatted links to Perfecto help documentation when relevant. """ mcp = FastMCP("perfecto-mcp", instructions=instructions, diff --git a/pyproject.toml b/pyproject.toml index 4d10b99..2a934f8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -36,7 +36,7 @@ where = ["."] include = ["tools*", "config", "models", "formatters", "resources"] [tool.setuptools.package-data] -"resources" = ["*.png"] +"resources" = ["**/*"] [tool.pytest.ini_options] pythonpath = ["."] diff --git a/resources/skills/perfecto-ai-scriptless/SKILL.md b/resources/skills/perfecto-ai-scriptless/SKILL.md new file mode 100644 index 0000000..0e98671 --- /dev/null +++ b/resources/skills/perfecto-ai-scriptless/SKILL.md @@ -0,0 +1,94 @@ +--- +name: perfecto-ai-scriptless +description: Author, run, and troubleshoot Perfecto AI Scriptless (also called Scriptless Mobile) tests — workspace UI, DUT execution, loops/conditions, AI user actions, AI validations, visual comparisons, and Perfecto MCP perfecto_ai_scriptless workflows. Use this skill whenever the user mentions AI Scriptless, Scriptless Mobile, Perfecto scriptless, natural-language test steps, AI commands, AI validation, AI user action, visual comparison, DUT, scriptless lab, or wants to create/edit/run Perfecto scriptless tests via UI or MCP — even if they do not say the exact product name. +--- + +# Perfecto AI Scriptless + +Teach agents how to help users with Perfecto AI Scriptless: product behavior, licenses, +lab UI, and MCP authoring/execution. Prefer this skill over guessing. + +Official docs hub: [AI Scriptless interface](https://help.perfecto.io/perfecto-help/content/perfecto/ide/scriptless-mobile-interface.htm) + +## When this skill loads + +1. Read this `SKILL.md` first. +2. Load only the reference files needed for the user's question (see [Reference files](#reference-files)). +3. For live Perfecto Help pages not covered here, use `perfecto_help` with `category_id='perfecto'` and `subcategory_id='ide'`. +4. Explain **why** a step matters when the rule prevents broken tests (licenses, `step_path`, command policy). + +## Licenses (check early) + +AI Scriptless behavior depends on cloud licenses: + +| Licenses present | UI label | Scope | +| --- | --- | --- | +| Perfecto AI + Desktop Web | **AI Scriptless** | Mobile + desktop web, natural-language commands | +| Missing one or both | **Scriptless Mobile** | Mobile only | + +- AI commands need Perfecto AI **and** admin feature-toggle opt-in. Without them, AI steps stay inactive and dependent tests fail. +- Desktop web as DUT needs Perfecto AI **and** Desktop Web. +- If MCP AI Scriptless calls fail with license/feature errors, tell the user to contact their Perfecto admin — do not invent workarounds. + +## Quick start (UI) + +1. Open Perfecto → **Scriptless Automation** → **Build ai scriptless test** (optional: pick a real device). +2. Lab URL: `https://{cloud}.app.perfectomobile.com/lab/scriptless-mobile/` (also from `perfecto_user` → `read_user`). +3. Orient on the workspace — read [interface.md](references/interface.md) if the user asks about toolbar, sidebar, editor, devices, or widgets. +4. Create/open/save tests, add AI or classic commands, then run with a DUT and open the Single Test Report. + +## Agent rules for MCP authoring + +Follow these when using `perfecto_ai_scriptless` (why in parentheses): + +1. **Consult skills/help before inventing product behavior** — Perfecto Scriptless rules are license- and UI-specific. +2. **Call `list_commands` before `add_command`** and follow the selection policy in the tool `info` field. Prefer primary AI commands: `ai_user-action`, `ai_validation`, `ai_visual-comparison` for natural-language steps. +3. **Call `get_command_definitions` before filling arguments** so parameter names/types match the repository. +4. **Treat `step_path` as ephemeral** (e.g. `0`, `2.0`, `5.b0.1`). Perfecto does not persist paths; they shift after insert/move/delete. Always `view_test_structure` before the next structure edit — never reuse a path from an older mutation response. +5. **Re-`view_test_structure` after every structure mutation** before the next edit. +6. **Validate the device before `execute_test`**: resolve DUT via `perfecto_devices`, check real-device availability, then execute; monitor with `perfecto_execution`. +7. **Do not invent per-test URLs.** Only lab entry exists; open tests in the UI by folder/name from `list_tests`. +8. **Capabilities not in MCP yet** (DataTables, Scheduler, Embedded tests, Object Spy, AI Assistant chat, restore snapshot, download as Appium, etc.): send the user to the lab UI and the matching help/reference — do not fake support. + +Full action catalog: [mcp-tools.md](references/mcp-tools.md). + +### Author + run workflow + +``` +1. list_commands (+ get_command_definitions as needed) +2. create_test OR list_tests → view_test_structure +3. add_command / add_logical_step / add_loop / add_condition … +4. view_test_structure again after each mutation +5. Validate device (real / virtual / desktop) +6. execute_test → list_live_executions / list_report_executions +``` + +## Phrase AI steps well + +When helping write natural-language steps, load the matching reference: + +- User actions → [ai-user-actions-best-practices.md](references/ai-user-actions-best-practices.md) — describe the **goal**, not locators; keep actions ≤ ~15 internal steps / 3 minutes. +- Validations → [ai-validations-best-practices.md](references/ai-validations-best-practices.md) — binary PASS/FAIL only, not open questions. +- Visual comparisons → [ai-visual-comparison-best-practices.md](references/ai-visual-comparison-best-practices.md) — avoid Device / Pixel Diff as failure categories unless intentional. +- FAQ / limits → [ai-commands.md](references/ai-commands.md) + +## Reference files + +Load on demand (one level deep): + +| Need | File | +| --- | --- | +| What it is, licenses, access | [getting-started.md](references/getting-started.md) | +| Workspace UI layout | [interface.md](references/interface.md) | +| New/open/save/delete/versions | [handle-tests.md](references/handle-tests.md) | +| Run / stop / DUT / reports | [execute-tests.md](references/execute-tests.md) | +| Conditions, loops, logical steps | [control-constructs.md](references/control-constructs.md) | +| MCP action map | [mcp-tools.md](references/mcp-tools.md) | +| AI commands FAQ | [ai-commands.md](references/ai-commands.md) | +| AI user-action phrasing | [ai-user-actions-best-practices.md](references/ai-user-actions-best-practices.md) | +| AI validation phrasing | [ai-validations-best-practices.md](references/ai-validations-best-practices.md) | +| AI visual comparison | [ai-visual-comparison-best-practices.md](references/ai-visual-comparison-best-practices.md) | + +## Official documentation + +Live catalog: [AI Scriptless](https://help.perfecto.io/perfecto-help/content/perfecto/ide/get-started-with-scriptless-mobile.htm). Prefer `perfecto_help` for pages under subcategory `ide` when a reference is missing or may be outdated. diff --git a/resources/skills/perfecto-ai-scriptless/evals/evals.json b/resources/skills/perfecto-ai-scriptless/evals/evals.json new file mode 100644 index 0000000..556a8e6 --- /dev/null +++ b/resources/skills/perfecto-ai-scriptless/evals/evals.json @@ -0,0 +1,40 @@ +{ + "skill_name": "perfecto-ai-scriptless", + "evals": [ + { + "id": 1, + "prompt": "I'm new to Perfecto — how do I open AI Scriptless and create my first test? Our cloud might only have Scriptless Mobile, not the full AI label.", + "expected_output": "Explains access path (Scriptless Automation → Build ai scriptless test), lab URL pattern, license labeling (AI Scriptless vs Scriptless Mobile), and a short create/save/run path. Mentions AI license/toggle for AI commands.", + "files": [], + "expectations": [ + "Mentions Perfecto landing page or Scriptless Automation access path", + "Explains AI Scriptless vs Scriptless Mobile license difference", + "Mentions DUT or execute/run step for a first test", + "Does not invent a per-test deep link URL" + ] + }, + { + "id": 2, + "prompt": "Using Perfecto MCP, add an AI validation step that checks the login button is green, then run the test on a free real device named something like Galaxy-S23.", + "expected_output": "Uses perfecto_ai_scriptless with list_commands / get_command_definitions, prefers ai_validation, phrases a PASS/FAIL validation (not an open question), view_test_structure around edits, validates device availability, then execute_test and monitoring via perfecto_execution/devices.", + "files": [], + "expectations": [ + "Calls or describes list_commands before add_command", + "Uses or recommends ai_validation (not a vague open question)", + "Mentions view_test_structure or ephemeral step_path behavior", + "Includes device availability check before execute_test" + ] + }, + { + "id": 3, + "prompt": "Help me phrase an AI user action for checkout: pick VISA and confirm. Also should I fail the visual comparison on Device and Pixel Diff?", + "expected_output": "Gives goal-based phrasing (not locator/coordinates), points at user-action best practices, and advises against Device/Pixel Diff as default failure categories for visual comparison.", + "files": [], + "expectations": [ + "Phrasing describes logical goal rather than mouse coordinates or brittle labels alone", + "Warns against Device and Pixel Diff as default visual-comparison failure categories", + "References or applies AI user-action / visual-comparison guidance from the skill" + ] + } + ] +} diff --git a/resources/skills/perfecto-ai-scriptless/references/ai-commands.md b/resources/skills/perfecto-ai-scriptless/references/ai-commands.md new file mode 100644 index 0000000..f24a02e --- /dev/null +++ b/resources/skills/perfecto-ai-scriptless/references/ai-commands.md @@ -0,0 +1,47 @@ +# AI commands overview and FAQ + +Sources: +- [AI commands FAQ](https://help.perfecto.io/perfecto-help/content/perfecto/ide/sm-ai-commands-faq.htm) +- [AI commands](https://help.perfecto.io/perfecto-help/content/perfecto/sm-commands-and-checkpoints/ai-commands.htm) + +## What AI commands do + +AI commands (Scriptless) and AI functions (Appium) use natural language — in any language — like a conversation. Benefits: + +- Complex validations that may combine multiple traditional steps into one +- Coverage beyond fragile locator-based scripting +- Platform-agnostic steps that adapt better to UI changes + +In the Scriptless UI, AI commands live under the **AI** left-sidebar tab. Prefer Perfecto's AI Assistant in the lab when crafting prompts. + +Primary MCP command IDs (confirm via `list_commands`): + +- `ai_user-action` — perform actions on the DUT +- `ai_validation` — PASS/FAIL assertions +- `ai_visual-comparison` — screen comparison categories + +## Licensing FAQ + +- AI is **not** on by default for every Perfecto feature — only AI commands/functions. +- A **separate Perfecto AI license** and admin **feature-toggle opt-in** are required. +- If the org opts out later, AI-based commands cannot run and dependent tests fail. +- Data use: see Perforce [Generative AI Policy](https://www.perforce.com/generative-ai-policy). + +## Limits and tips + +- One AI command can contain up to **30** internal steps; execution limit is **3 minutes** per call. For best results, target **≤ 15** steps. +- Form fill / text extract with AI works on the **default tab** in web sessions (desktop browser, mobile web, WebView) — not native mobile/desktop apps, and not tabs opened mid-flow (Selenium stays on the default tab). +- If answers are inconsistent: refine with the AI Assistant suggestions, then review best-practice docs. Ambiguous prompts are the usual cause. + +## Example validation conversation pattern + +User: "Is there a sign-up here option for new user?" +Assistant may suggest: "Is there a sign-up option for new users labeled 'Sign Up Here'?" + +Use assistant suggestions as the final validation text when possible. + +## Related best practices + +- [AI user actions](ai-user-actions-best-practices.md) +- [AI validations](ai-validations-best-practices.md) +- [AI visual comparisons](ai-visual-comparison-best-practices.md) diff --git a/resources/skills/perfecto-ai-scriptless/references/ai-user-actions-best-practices.md b/resources/skills/perfecto-ai-scriptless/references/ai-user-actions-best-practices.md new file mode 100644 index 0000000..64831af --- /dev/null +++ b/resources/skills/perfecto-ai-scriptless/references/ai-user-actions-best-practices.md @@ -0,0 +1,69 @@ +# Best practices: AI user actions + +Source: [Best practices for working with AI user actions](https://help.perfecto.io/perfecto-help/content/perfecto/ide/sm-ai-user-action-best-practices.htm) + +Write the action once; run across devices/OS. Describe the **logical goal**, not locators. + +In Scriptless, use AI commands ([AI commands](https://help.perfecto.io/perfecto-help/content/perfecto/sm-commands-and-checkpoints/ai-commands.htm)). For Appium, see [AI functions](https://help.perfecto.io/perfecto-help/content/perfecto/automation-testing/ai-functions.htm). + +## Performance + +- Phrase each User Action so it resolves to **≤ 15** steps and finishes in **under 3 minutes**. +- Enable **Reasoning** when counting, sorting, or ordering needs extra reasoning resources. + +## What works well + +- Common UI: hamburger, help, notifications +- Click buttons, links, icons, FABs +- Type into empty fields or replace text +- Radio buttons, checkboxes, sliders +- Tabs, lists, navbars +- Dismiss interrupting popups/notifications +- Browser navigation (go to, back, forward, refresh, clean) +- Install mobile apps (see install section) + +Still evolving / may be unreliable: inline labels relative to fields, table cell/row actions, color pickers. Not yet fully tested: append text to existing field content. LLMs can make mistakes. + +## Why AI user actions help + +- **Write once, run everywhere** — no device-specific locators +- **Survive UI renames/moves** — goal-based instructions +- **Wait instead of fail** — instruct wait/retry/dismiss modals/progress +- **Focus** — “focus on / zoom into …” for dense UIs + +## Do's and don'ts + +| Don't | Do | Why | +| --- | --- | --- | +| Move mouse to Cart button bottom-right | On the profile page, open the shopping cart | Logical goal, not coordinates/labels | +| Register a new account | Fill the form: Enter "Jill A. Smith" in the name fields… | Enough concrete detail | +| Click the red shoes | From search results, select the red sneakers that cost $30 | Disambiguate similar items | +| Pay by card | Fill the form: In the drop down, select credit card… | Triggers form-fill tooling for dropdowns | + +## Specialized knowledge (`ai.txt`) + +For domain UIs (seat maps, dashboards, canvases): + +1. Write instructions in plain text (max **2,000** characters; larger files are ignored). +2. Save as `ai.txt`. +3. Upload to the **root** of the Perfecto Repository. +4. Reloads on login; applied as general user instructions for AI assistants/commands. + +## Install mobile apps via User Action + +Example prompts: `Install public:/path/myapp.ipa`, `Install ${myApp} on the device, open it and log in`. + +Configure instrumentation in the UI widget (Sensor, WebView, Secured Screen, Resign). iOS notes: + +- Virtual iOS: use `appname.zip` +- Real iOS: Resign Application on by default (mandatory unless WebView changes rules) + +See [Install a mobile app using AI User Actions](https://help.perfecto.io/perfecto-help/content/perfecto/ide/sm-ai-user-action-install-app.htm). + +## Fill forms + +Prefer **one** prompt that fills many fields, then a separate submit action. + +Supported: text/textarea, native `