TestQL — Interface Query Language for Testing

AI Cost Tracking

• 🤖 LLM usage: $30.3028 (135 commits)
• 👤 Human dev: ~$7745 (77.4h @ $100/h, 30min dedup)

Generated on 2026-07-21 using openrouter/qwen/qwen3-coder-next

---

TestQL is a declarative DSL (Domain Specific Language) for testing GUI, REST API, and hardware encoder interfaces. It provides a simple, readable syntax for writing automated tests without programming overhead.

What's new in 1.2.x

• Native Desktop E2E: DESKTOP_* commands with vdisplay mirror→Xvfb capture, img2nl + imgl OCR on Linux (no Wayland portal)
• Artifact Discovery & Topology: Complete codebase inspection with manifest generation
• Web Inspection: Live URL scanning with Playwright browser automation
• Multi-DSL Platform: TestTOON, NL, SQL, Proto, GraphQL adapters with Unified IR
• Autoloop Integration: MCP-driven development with autonomous test loops
• Refactor Plans: Structured refactoring plans with NLP summaries
• Production deployment in c2004: 50-assertion realtime-health scenario running every 60s as part of Prometheus + planfile self-healing pipeline

Recent Improvements (1.2.48)

• c2004 Healing Pipeline: testql-watchdog container, Prometheus metrics, LLM-ready ticket generation on failure
• MCP Server: testql/mcp/server.py — resources for topology, tools for discover/run/explain
• Internal IR: testql/ir/ module with steps, assertions, captures, fixtures, plan
• TestTOON Adapter: full round-tripping with validation layer
• Configuration Management: refactored config system
• Code Analysis Engine: refactored analysis pipeline
• Markdown Output: --output markdown for human-readable reports
• Artifact Discovery: testql discover with topology graph generation
• Web Inspection: --scan-network with headless browser capture, console errors, network logs
• Multi-Format Output: JSON, YAML, TOON for all results and topology
• Browser Testing: Playwright integration for JS-rendered DOM capture
• API Testing: 7 OpenAPI endpoints with FastAPI integration
• CLI/Shell Execution: SHELL, EXEC, RUN commands with assertions
• GUI Execution: Playwright/Selenium support with GUI_START, GUI_CLICK, GUI_INPUT
• Unit Test Execution: UNIT_PYTEST, UNIT_ASSERT, UNIT_IMPORT integration
• Test Coverage: 65% with pyqual.yaml quality pipeline

Installation

# Install from source
pip install -e .

# Install with development dependencies
pip install -e ".[dev]"

Requirements

• Python 3.10+
• HTTPX for API testing
• Chrome/Chromium for browser GUI testing. TestQL installs the Python Playwright API and can also reuse Playwright from the calling project's node_modules automatically.
• Native desktop E2E (optional): vdisplay, img2nl, imgl — see Desktop GUI E2E

Optional: native desktop + vision

pip install -e ".[desktop,vision]" \
  -e ~/github/wronai/vdisplay[pillow] \
  -e ~/github/wronai/img2nl[analyze,similarity,opencv,scan] \
  -e ~/github/semcod/imgl

# System: xvfb x11-apps xdotool xrandr tesseract-ocr

Quick Start

# Run a test scenario
testql scenarios/tests/test-api.testql.toon.yaml --url http://localhost:8101

# Dry-run (parse + validate only)
testql scenarios/views/connect-id-barcode.testql.toon.yaml --dry-run

# JSON output for CI integration
testql scenarios/tests/test-api.testql.toon.yaml --output json

# Run with verbose logging
testql scenarios/tests/test-api.testql.toon.yaml --verbose

Public verification API

Orchestrators should use the versioned Python contract instead of importing CLI internals. TestQL owns path/glob resolution and batch result normalization; the caller receives canonical request/result hashes for audit and replay.

from testql.verification import VerificationRequest, run_verification

request = VerificationRequest(
    file_specs=("scenarios/**/*.testql.toon.yaml",),
    project_dir=".",
    dry_run=True,
)
result = run_verification(request)
print(result.to_dict())  # testql.verification-result.v1

Packaged JSON Schemas are available through verification_contract_schema().

Versioned LLM contracts

The live nlp2env path accepts only the versioned ToolCall 1.0.0 response. Its GBNF, Protobuf, JSON Schema and boundary manifest are packaged under testql/contracts/nlp2env/v1. OpenRouter receives the schema through response_format, Ollama through format, and TestQL validates the complete JSON response again before invoking MCP. Markdown fences, surrounding prose, unknown tools, literal password fields and contract-version drift fail closed.

Set OPENROUTER_APP_NAME to identify TestQL in OpenRouter logs. If it is not set, TestQL uses the current project folder name. See .env.example for the GLM 5.2 defaults.

The optional live nlp2dsl conversation provider similarly uses ConversationFields 1.0.0. It returns only requested missing fields, while preserving the existing plain field mapping passed into llmContext.

Artifact Discovery, Topology, and Web Inspection

TestQL can now inspect codebases, manifests, and opt-in live URLs, then write structured topology and result artifacts for humans, CI, and LLM workflows.

# Discover local artifact types
python3 -m testql.cli discover ./testql --format json

# Build a topology graph from discovered artifacts
python3 -m testql.cli topology ./testql --format toon

# Inspect a live web page and write all data/metadata to .testql
python3 -m testql.cli inspect https://tom.sapletta.com/ \
  --scan-network \
  --out-dir .testql

The .testql/ artifact bundle contains:

metadata.json
topology.{json,yaml,toon.yaml}
result.{json,yaml,toon.yaml}
refactor-plan.{json,yaml,toon.yaml}
inspection.{json,yaml,toon.yaml}
summary.md

Live URL inspection currently extracts:

• HTTP metadata: status code, final URL, content type.
• Page schema: title, links, assets, forms.
• Topology graph: page, link, asset, form, interface, evidence nodes.
• Structured results: web checks for status, title, links, assets, forms.
• Reports: TOON/YAML/JSON plus NLP summary.

Example:

examples/web-inspection-dot-testql/run.sh https://tom.sapletta.com/

Current capabilities:

• Asset classification: script, stylesheet, image, icon, preload, link.
• Bounded link validation: HEAD checks for all internal links (up to 100).
• Bounded asset validation: HEAD checks for all extracted assets.
• Broken resource detection: assets or links returning error status are flagged as findings.
• Bounded sitemap crawl: fetches up to 10 internal subpages, extracts titles and link counts, adds subpage nodes to the topology.
• Sitemap checks: crawl coverage, broken subpage detection, duplicate title warnings.
• Playwright browser inspection (--browser): renders the page in a headless browser, captures console errors, network calls (REST/GraphQL/WebSocket), and JS-rendered DOM.
• Browser checks: render detection, console error count, network call capture, title extraction, link/asset/form enumeration.

Current limitations:

• Per-resource validation uses HEAD requests only; full page content is not fetched for linked pages.
• Screenshots, performance metrics, accessibility checks, and auth flows are planned next.

API Endpoint Detection

TestQL includes advanced endpoint detection for multiple frameworks:

# List all detected endpoints in a project
testql endpoints ./my-project
testql endpoints ./my-project --format json
testql endpoints ./my-project --framework fastapi

# Analyze project structure and generate tests
testql analyze ./my-project
testql generate ./my-project

Supported Frameworks

• FastAPI — Detects routers, decorators, include_router patterns
• Flask — Blueprints, MethodView, route decorators
• Django — URL patterns from urls.py
• Express.js — JavaScript/TypeScript route definitions
• OpenAPI/Swagger — Spec file parsing (JSON/YAML)
• GraphQL — Schema and resolver detection
• WebSocket — WS endpoint detection

Endpoint Detection Features

• Framework Detection: Automatically identifies the web framework
• Handler Names: Extracts function/method names for documentation
• Parameters: Detects path/query/body parameters
• Docstrings: Uses function docstrings for test descriptions
• Test Inference: Analyzes existing tests to discover endpoints

OpenAPI Generation

Generate OpenAPI 3.0 specifications from your code:

# Generate OpenAPI spec (YAML format)
testql openapi ./my-project
testql openapi ./my-project --format json

# Generate with contract tests
testql openapi ./my-project --contract-tests

# Custom output file
testql openapi ./my-project -o ./docs/api-spec.yaml --title "My API"

Contract Testing

TestQL generates contract tests from OpenAPI specs:

# Auto-generated from openapi.yaml
API[25]{method, endpoint, expected_status}:
  GET, /api/v1/users, 200
  POST, /api/v1/users, 201
  ...

ASSERT[3]{field, operator, expected}:
  content_type, ==, application/json
  schema_valid, ==, true
  status, <, 500

Project Echo (AI Context)

Generate AI-friendly project metadata by combining TESTQL scenarios with DOQL system models:

# Generate unified project context
testql echo --toon-path testql-scenarios/ --doql-path app.doql.less

# JSON output for LLM consumption
testql echo --toon-path ./tests --doql-path ./app.doql.less --format json -o context.json

# Text output (human-readable)
testql echo --doql-path ./app.doql.less --format text

Echo Layers

Layer	Source	Content
API Contract	*.testql.toon.yaml	Endpoints, methods, assertions
System Model	*.doql.less	Entities, workflows, interfaces
Unified Context	Combined	Complete project metadata for AI

Example Output

📦 Project: weboql (0.1.2)

🧠 Type:
  • API (fastapi)

🛠️ Workflows:
  • install: pip install -e .
  • test: pytest
  • run: HARDWARE_MODE=mock weboql-server

🌐 API scenarios:
  • API Health Check (api) - 4 endpoint(s)

💡 LLM suggestions:
  • Run tests: task test
  • Start server: task run

Language Reference

Variables

SET api_url "http://localhost:8101"
SET timeout 5000
GET api_url              # Prints variable value

Secrets

Import credentials with GETENV_SECRET rather than SET, so the value never reaches a result artifact:

GETENV_SECRET SUBACTOR_ADMIN_TOKEN admin_token
GUI_INPUT "#adminToken" "${admin_token}"

The value is replaced with ***REDACTED*** in the reported variables and across the whole result payload — step names, messages, values, details, errors and warnings. This matters on failure: a driver echoes back what it was handed, so a Playwright fill timeout reports fill("<value>") in its call log, and that message is what a CI job keeps.

Variables support ${var} and $var interpolation:

SET base_url "http://localhost:8101"
API GET "${base_url}/api/devices"

Logging

LOG "Starting test suite"
LOG "Current value: ${var}"

API Commands

# HTTP methods
API GET "/api/v3/data/devices"
API POST "/api/v3/scenarios" {"id": "ts1", "name": "Test"}
API PUT "/api/v3/devices/123" {"status": "active"}
API DELETE "/api/v3/scenarios/old"

Assertions

# Status code assertions
ASSERT_STATUS 200
ASSERT_OK                  # Shorthand for 2xx status

# Content assertions
ASSERT_CONTAINS "device"
ASSERT_CONTAINS "status": "ok"

# JSON path assertions
ASSERT_JSON data.length > 0
ASSERT_JSON devices[0].id == "dev-001"
ASSERT_JSON status != "error"

Operators: ==, !=, >, >=, <, <=, CONTAINS

Shell output assertions

ASSERT_JSON reads the last API response. To assert on a JSON report printed by a command, use ASSERT_STDOUT_JSON — same paths, same operators, applied to the stdout of the last SHELL / EXEC / RUN:

SHELL "node scripts/audit-dsl-artifacts.mjs --json" 60000
ASSERT_EXIT_CODE 0
ASSERT_STDOUT_JSON ok == true
ASSERT_STDOUT_JSON summary.findings == 0
ASSERT_STDOUT_JSON findings.length == 0
ASSERT_STDOUT_JSON schema == "subactor.dsl-artifact-audit.v1"

Prefer it over ASSERT_STDOUT_CONTAINS for machine-readable output: substring matching is whitespace-sensitive, cannot compare numbers, and matches a nested occurrence of a key just as happily as the intended one. A leading banner line before the JSON is tolerated; unparseable output and a missing path both fail rather than pass silently.

GUI Navigation (Playwright)

Browser-based GUI tests via Playwright. For native OS desktop (monitors, windows, OCR, mirror capture on Wayland), see Desktop GUI E2E.

GUI_START first uses the Python Playwright installed with TestQL. Existing environments that predate this dependency automatically fall back to node_modules/playwright in the calling project and its parent directories. An installed Chrome/Chromium executable is detected automatically. Custom locations can be supplied with TESTQL_NODE_PLAYWRIGHT_PATH, TESTQL_PLAYWRIGHT_SEARCH_PATHS, or TESTQL_BROWSER_EXECUTABLE.

# Navigation
NAVIGATE "/connect-workshop"
NAVIGATE "${base_url}/devices"

# Interaction
WAIT 500
CLICK "[data-action='search']"
INPUT "#search-input" "drager"
CLICK "button[type='submit']"

# Assertions
ASSERT_VISIBLE "[data-testid='results']"
ASSERT_TEXT "#status" "Connected"

Browser performance

GUI_MEASURE_NAVIGATION performs a real Playwright navigation and stores Navigation Timing, resource-transfer and long-task metrics. The ready selector is also detected in same-origin iframes.

GUI_START "about:blank"
GUI_MEASURE_NAVIGATION "http://localhost:8100/connect-scenario" ".scenario-embedded-root" timeout=15000 as=_performance
ASSERT_JSON _performance.ready_ms < 2000
ASSERT_JSON _performance.response_ms < 500
GUI_STOP

The result contains navigation_ms, ready_ms, response_ms, dom_content_loaded_ms, load_event_ms, resource_count, transfer_bytes, script_bytes, long_task_count, long_task_ms, and frame_count.

Native Desktop (Linux)

Real OS desktop E2E via vdisplay mirror→Xvfb (no Wayland portal). Full guide: Desktop GUI E2E.

DESKTOP_MONITORS
DESKTOP_CAPTURE "shot.png" primary
DESKTOP_ASSERT_WINDOW "Toolbox"
DESKTOP_ASSERT_ELEMENTS 1 "shot.png"
DESKTOP_ANALYZE "shot.png" "layout.json"
DESKTOP_FOCUS "Toolbox"

DISPLAY=:0 testql run examples/desktop/gui-e2e-inspect.oql
DISPLAY=:0 bash examples/desktop/run-all.sh

Hardware Encoder Commands

ENCODER_ON                    # Activate encoder mode
ENCODER_FOCUS column1         # Focus specific column
ENCODER_SCROLL 3              # Scroll 3 steps
ENCODER_CLICK                 # Confirm (single click)
ENCODER_DBLCLICK              # Cancel (double click)
ENCODER_PAGE_NEXT             # Next page
ENCODER_PAGE_PREV             # Previous page
ENCODER_STATUS                # Print current encoder state
ENCODER_OFF                   # Deactivate encoder mode

Script Composition

# Include common setup
INCLUDE "common-setup.testql.toon.yaml"

# Include relative paths
INCLUDE "../helpers/auth.testql.toon.yaml"

Control Flow (Planned)

IF status == "ready"
  LOG "System ready"
ELSE ERROR "System not ready"

LABEL start
REPEAT 3 {
  API GET "/api/ping"
}
GOTO start

Project Structure

testql/
├── testql/
│   ├── cli.py           # Command-line interface
│   ├── base.py          # Base test runner
│   ├── commands/        # Command implementations
│   ├── runners/         # Test execution runners
│   └── reporters/       # Output formatters
├── scenarios/           # Sample test scenarios
│   ├── tests/          # Automated test suites
│   ├── views/          # GUI view tests
│   ├── diagnostics/    # System diagnostic scripts
│   ├── examples/       # Usage examples
│   └── recordings/     # Recorded test sessions
├── docs/
│   ├── testql-spec.md  # Full language specification
│   └── recipes/        # Common testing patterns
└── tests/              # Unit tests

Scenario Organization

TestQL scenarios use two formats:

• *.testql.toon.yaml — TestTOON tabular format for test sequences (preserves order)
• *.testql.less — LESS format for shared test configuration (variables, mixins)

Directory	Purpose	Example
scenarios/tests/	API/integration tests	test-api.testql.toon.yaml
scenarios/views/	GUI view tests	connect-id-barcode.testql.toon.yaml
scenarios/diagnostics/	System checks	health-check.testql.toon.yaml
scenarios/examples/	Learning samples	device-identification.testql.toon.yaml
scenarios/recordings/	Recorded sessions	session-recording.testql.toon.yaml

TestTOON Format

Test files use the tabular TestTOON format where each section declares its schema:

# SCENARIO: API Smoke Test
# TYPE: api
# VERSION: 1.0

API[3]{method, endpoint, status}:
  GET,  /api/v3/health,   200
  GET,  /api/v3/devices,  200
  POST, /api/v3/start,    201

ASSERT[2]{field, op, expected}:
  status,  ==, ok
  count,   >,  0

LESS Configuration

Shared test configuration uses LESS for variables and mixins:

// config.testql.less
@api_url: http://localhost:8101;
@timeout_ms: 30000;

.api-test(@method, @endpoint) {
  method: @method;
  endpoint: @endpoint;
  expect_status: 200;
}

CLI Options

testql <file> [options]

Options:
  --url <url>         Base URL for API tests (default: http://localhost:8101)
  --dry-run           Parse and validate only, don't execute
  --output <format>   Output format: text (default), json, junit
  --verbose           Enable verbose logging
  --timeout <ms>      Default timeout for operations

Generate from Topology

Generate executable scenarios from discovered topology paths:

# Generate TestTOON from the first topology trace
testql generate-topology ./project

# Generate IR JSON from a specific trace
testql generate-topology ./project --trace-id trace.001 --format ir-json

# Write to file with live network scanning
testql generate-topology ./project --scan-network -o scenario.testql.toon.yaml

Testing

# Run all tests
pytest

# Run with coverage
pytest --cov=testql

# Run specific test file
pytest tests/test_runner.py -v

Example Scenario

# scenarios/tests/test-api.testql.toon.yaml
# SCENARIO: Device API Test
# TYPE: api
# VERSION: 1.0

CONFIG[1]{key, value}:
  base_url,  http://localhost:8101

API[2]{method, endpoint, status}:
  GET,   /api/v3/data/devices,   200
  POST,  /api/v3/scenarios,      201

ASSERT[2]{field, op, expected}:
  data.length,  >,   0
  data.id,      !=,  null

Real-world deployment — c2004 healing pipeline

Since May 2026 TestQL runs as the active-probing component of the c2004 fleet management monorepo's self-healing pipeline. It's the canonical, production-grade reference for how to wire testql run into Prometheus + Alertmanager + an LLM-agnostic ticket layer.

What c2004 uses TestQL for

Component	Role
realtime-health.testql.toon.yaml	50-assertion scenario covering 20 endpoints across backend, frontend, connect-* services, hw-firmware, and modules
testql-watchdog (Docker container)	Runs the scenario every 60s, exposes /metrics to Prometheus, POSTs failures to a healing webhook
TOON output format	30-60% smaller than JSON in LLM context — critical for fitting failures into a prompt window

Pipeline shape

testql-watchdog (loop)
    │ runs `testql run realtime-health.testql.toon.yaml --output json`
    ▼
50 assertions execute → exposes testql_scenario_pass_total / fail_total
    │ on failure: POST http://healing-webhook:8810/probe-failure
    ▼
healing-webhook → planfile ticket create --label llm-ready
    │
    ▼
LLM agent (Windsurf/Cursor/Claude Code/aider) reads
  `planfile ticket show PLF-XXX` and proposes a fix

Operator flow

Note: The observability stack (including testql-watchdog) does not start automatically with make dev / make run. Those commands only start the application processes (Node.js + Python). You must launch the monitoring stack separately.

# 1. Terminal A — start the application (no monitoring)
make dev                        # starts frontend + backend + encoder

# 2. Terminal B — start observability + self-healing (separate step)
task monitor:up                 # brings up testql-watchdog + 10 other containers

# 3. Verify watchdog is running
task monitor:probe              # ad-hoc run of the scenario (50 assertions)
docker logs c2004-testql-watchdog | tail
# → cycle done exit=0 pass=50 fail=0

# 4. View Prometheus metrics
open http://localhost:9101/metrics   # testql_scenario_pass_total / fail_total

Key commands:

• task monitor:up — start full observability stack
• task monitor:core — start core stack (lighter, without Loki/Blackbox)
• task monitor:down — stop all observability containers
• task monitor:logs:watchdog — tail testql-watchdog logs

Why TestQL was chosen for this role

1. Black-box + business-logic — pinging /health is not enough; TestQL exercises real endpoints (/api/v3/devices, /api/v3/menu/configurations, etc.) with full assertion blocks.
2. TOON format — the scenario file is human-editable and ~40% smaller than the equivalent JSON when sent to an LLM for diagnosis.
3. Single binary — pip install testql and the watchdog container is ~80 MB.
4. JSON output mode — --output json makes Prometheus metric extraction a one-liner in watchdog.py.

See also

• c2004 master doc: docs/planfile-llm-guide.md
• c2004 README section: "🩺 Real-time Monitoring + Planfile Self-Healing Pipeline"
• The actual scenario: testql-testing/scenarios/realtime-health.testql.toon.yaml
• The watchdog: monitoring/testql-watchdog/

---

Examples

Example	Description
API Testing	REST API testing with API GET/POST and assertions
Artifact Bundle	Generate .testql/ bundles via CLI or Python script
Browser Inspection	Headless browser inspection with Playwright
Discovery	Discover artifacts and build project topology
Desktop E2E	Native Linux desktop: vdisplay mirror, img2nl, imgl OCR
GUI Testing	Playwright-based GUI navigation and assertions
Project Echo	Generate AI context from TestQL + DOQL models
Shell Testing	Run shell commands and assert exit codes/output
TestTOON Basics	Tabular TestTOON format walkthrough
Topology	Generate topology graphs from codebases
Web Inspection	Inspect live URLs and generate structured reports
Web Inspection + Bundle	Full web inspection writing .testql artifact bundle

Documentation

• TestQL Specification — Complete language reference
• Desktop GUI E2E — Native Linux desktop testing (vdisplay, img2nl, imgl)
• Desktop examples — Scenario index and run-all script
• Recipes — Common testing patterns and examples

License

Licensed under Apache-2.0.