leo4.tester

A standalone PHP CLI that black-box tests Leo4 (PlusClouds' cloud orchestration platform) by driving real customer journeys against a running instance over HTTP and SSH — not in-process unit tests against Leo's own codebase. Each "feature" is a scenario modeled on something a real customer actually does:

• iaas-image-deployment — provisions a VM from every publicly available repository image, verifies it actually comes up: reachable status, a resolvable IP, working SSH/password login, and a healthy in-guest agent. Tears the VM down afterward.
• crm-sales-pipeline — creates a test customer account and checks whether they were automatically added to the sales pipeline (a CRM Opportunity).

Results are reported per scenario as a table of pass/fail/error checks, and every failed/errored check is also appended to a local JSON-lines log for later triage.

Requirements

• PHP 8.2+
• Composer
• Network access to the Leo4 API base URL you're testing against
• A valid Leo4 API bearer token (see Getting a token)

Installation

composer install
cp .env.example .env

Then fill in .env — see Configuration below.

Configuration

All configuration lives in .env (gitignored — never commit real credentials). .env.example is the template.

Variable	Required	Default	Meaning
LEO_BASE_URL	no	https://apiv4.plusclouds.com	Leo4 API base URL
LEO_API_TOKEN	yes	—	Bearer token used on every request
IAAS_COMPUTE_POOL_ID	yes (for iaas-image-deployment)	—	Compute pool UUID new test VMs are provisioned into
IAAS_NETWORK_ID	no	—	Network UUID to attach; omit to let the platform auto-select
IAAS_STORAGE_VOLUME_ID	no	—	Storage volume UUID; omit to let the platform auto-select
IAAS_CPU_CORES	no	1	vCPUs requested per test VM
IAAS_DISK_GB	no	20	Disk size requested per test VM
IAAS_RAM	no	4	RAM requested per test VM, in GB — this is create-wizard's own unit; note the plain CRUD create endpoint expects ram in MB instead, the two are not consistent (see Known Leo4 platform issues)
IAAS_LAZY_DEPLOY	no	true	Passed through as is_lazy_deploy on create-wizard
IAAS_ONLY_IMAGE_ID	no	—	Restrict iaas-image-deployment to a single image id instead of sweeping every public image — use this while validating credentials/payload before a full run
POLL_INTERVAL_SECONDS	no	10	How often to poll VM status while waiting for it to deploy
POLL_TIMEOUT_SECONDS	no	600	Max time to wait for a VM to reach running before failing the check
SSH_TIMEOUT_SECONDS	no	15	SSH connection timeout
AGENT_HEALTHCHECK_PORT	no	8080	Port the in-guest vm.agent's /healthz is expected on
CLEANUP_AFTER_TEST	no	true	Tear down test VMs after each check
ERROR_LOG_PATH	no	storage/errors.log	Where failed/errored checks are appended as JSON lines
HTTP_TIMEOUT_SECONDS	no	60	HTTP client timeout — create-wizard alone has been observed taking ~30-45s to respond, so this needs real headroom

Getting a token

leo4.tester expects a pre-obtained bearer token — it does not automate Leo's multi-step OAuth login flow. Log in through the normal Leo4 flow (or however your team currently obtains a token) and paste the resulting token into LEO_API_TOKEN. Tokens don't appear to expire quickly in practice, but if requests start failing with auth errors, get a fresh one.

Running tests

Automated, via the CLI

php bin/leo-tester list                       # show available scenarios
php bin/leo-tester run iaas-image-deployment  # run one scenario
php bin/leo-tester run crm-sales-pipeline
php bin/leo-tester run:all                    # run every scenario
php bin/leo-tester run:all --format=json       # machine-readable output for CI

Exit code is non-zero if any check failed or errored, so run:all is CI-friendly as-is.

Before running iaas-image-deployment against every public image, set IAAS_ONLY_IMAGE_ID to a single known-good image id and do one run first. A full sweep provisions and tears down one real VM per public image, which is slow (each VM can take 3-8 minutes to reach running) and, if something in your config is wrong, wastes that time once per image instead of once.

# .env
IAAS_ONLY_IMAGE_ID=64873cfe-d82f-4c1e-9695-e1be959b85bf

php bin/leo-tester run iaas-image-deployment

Once that passes, remove IAAS_ONLY_IMAGE_ID to sweep every public image.

Internal unit tests

These test the tool's own code (HTTP client, result aggregation) against mocked responses — they don't touch a real Leo4 instance:

vendor/bin/phpunit

Manual / exploratory testing against the live API

Sometimes you need to poke at Leo's API directly — to validate a new payload shape, chase down an unexpected result, or check whether something observed by the tool is a real platform issue before filing it. The pattern used throughout this project's own development:

# Load .env into the shell so $LEO_BASE_URL / $LEO_API_TOKEN are available
set -a; source .env; set +a

# Example: list public images
curl -s "$LEO_BASE_URL/iaas/repository-images" \
  -H "Authorization: Bearer $LEO_API_TOKEN" -H "Accept: application/json" \
  | python3 -m json.tool | less

# Example: create a VM directly via create-wizard (note the /leo prefix!)
curl -s -X POST "$LEO_BASE_URL/leo/iaas/virtual-machines/create-wizard" \
  -H "Authorization: Bearer $LEO_API_TOKEN" -H "Accept: application/json" -H "Content-Type: application/json" \
  -d '{
    "name": "manual-test-1",
    "ram": 4,
    "cpu": 2,
    "disk": 20,
    "auto_deploy": true,
    "boot_after_deploy": true,
    "iaas_repository_image_id": "<image-id>",
    "iaas_compute_pool_id": "<pool-id>"
  }'

# Poll status
curl -s "$LEO_BASE_URL/iaas/virtual-machines/<vm-id>" \
  -H "Authorization: Bearer $LEO_API_TOKEN" -H "Accept: application/json" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['status'])"

Always clean up manually created VMs when you're done — see Cleaning up manually below, since DELETE alone is not reliable on this platform.

A note on secrets

Never echo/cat/print a decrypted password or bearer token to your terminal history or into anything that gets logged or shared (chat, tickets, commits). If you need to use a real credential value returned by the API (e.g. from the decrypt-password endpoint) to test something like SSH login, pipe it directly into the next command / a script variable and only print pass/fail results, never the value itself.

Cleaning up manually

DELETE /iaas/virtual-machines/{id} returns 204 even when it doesn't actually delete anything (see #991 below). The reliable sequence:

# 1. If the VM is running, shut it down first
curl -s -X POST "$LEO_BASE_URL/iaas/virtual-machines/<vm-id>/do/shutdown" \
  -H "Authorization: Bearer $LEO_API_TOKEN" -H "Content-Type: application/json" -d '{}'

# 2. Wait for status to become "halted", then delete
curl -s -X DELETE "$LEO_BASE_URL/iaas/virtual-machines/<vm-id>" \
  -H "Authorization: Bearer $LEO_API_TOKEN"

# 3. Verify it's actually gone (a 404 here means success)
curl -s -w "\n%{http_code}\n" "$LEO_BASE_URL/iaas/virtual-machines/<vm-id>" \
  -H "Authorization: Bearer $LEO_API_TOKEN"

If the VM was created via the plain CRUD endpoint and never deployed (status: draft), this sequence does not reliably work either — as of this writing there is no known API-level fix for cleaning up a stuck draft (see #991). It's a harmless orphaned record (no compute allocated), not a billing risk, but there's currently no way to remove it via the API.

leo4.tester itself automates this shutdown-then-delete-then-verify sequence in its own cleanup (see ImageDeploymentTest::cleanup()), so you generally don't need to do this by hand for VMs the tool created — only for ones you provisioned manually while investigating something.

Architecture

bin/leo-tester                     Symfony Console entrypoint
src/
  Config.php                       Loads/validates .env
  Http/LeoClient.php                Guzzle wrapper: base_uri + Bearer auth, json(), pagination walker
  Http/LeoApiException.php
  Feature/FeatureTestInterface.php  Contract: name(), run(LeoClient, OutputInterface): FeatureResult
  Feature/Check.php                 One sub-result: name, status (pass/fail/error), message, duration, details
  Feature/FeatureResult.php         A scenario's Check[] plus pass/fail aggregation
  Feature/FeatureRegistry.php       name => FeatureTestInterface map
  Feature/Iaas/ImageDeploymentTest.php
  Feature/Crm/SalesPipelineTest.php
  Ssh/SshProbe.php                  phpseclib3 wrapper: SSH connect+auth, agent /healthz check
  Console/Command/{List,Run,RunAll}Command.php
  Reporting/{Console,Json}Reporter.php
  Logging/ErrorLogger.php           Appends failed/errored checks to storage/errors.log
tests/                              PHPUnit tests for the tool's own internals (mocked HTTP)

Adding a new scenario

Implement LeoTester\Feature\FeatureTestInterface (name() and run(), returning a FeatureResult built from one or more Checks), then register it in FeatureRegistry::withDefaults(). No other wiring is needed — it automatically becomes available to list, run, and run:all.

Known Leo4 platform issues

Building and running this tool against a real Leo4 instance surfaced several platform bugs, filed in plusclouds/leov4 (tagged bug + ClaudeFinding). Worth knowing about if you're investigating unexpected results:

#	Issue	Status as of writing
#990	SSH password auth rejected even with Leo's own decrypted password	Unresolved — iaas-image-deployment's SSH check will keep failing until fixed
#991	DELETE /iaas/virtual-machines/{id} returns 204 but silently fails (running VMs need shutdown-first; draft VMs never succeed)	Unresolved — see Cleaning up manually
#992	GET .../addresses unreachable due to a trailing-space route bug	Worked around in code (resolveIpAddress() cross-references virtual-network-cards + ip-addresses instead)
#993	filter[...] query params silently ignored on several IAAS index endpoints	Worked around in code (client-side filtering everywhere)
#994	ram unit inconsistent: MB on plain CRUD create, GB on create-wizard	Documented, IAAS_RAM is in GB (create-wizard's unit)
#995	password field leaks raw encrypted ciphertext once a VM is deployed	Documented only, no workaround needed in this tool
#996	Empty collections return 404 instead of an empty list	Documented — a 404 from a list endpoint can mean "genuinely empty"
#997	vm.agent unreachable on port 8080 even after 8+ min uptime	Unresolved — iaas-image-deployment's agent check will keep failing until fixed

Also note: create-wizard's real path is POST /leo/iaas/virtual-machines/create-wizard — easy to miss the /leo prefix, and hitting the wrong (unprefixed) path returns a generic 500 that looks identical to a real server error.

crm-sales-pipeline is expected to FAIL until Leo actually wires account creation to automatic Opportunity creation — see the SalesPipelineTest class docblock. That failure is the tool doing its job, not a bug in it.