diff --git a/dev/microsoft-agents-testing/README.md b/dev/microsoft-agents-testing/README.md index 4acca0c57..11160bf05 100644 --- a/dev/microsoft-agents-testing/README.md +++ b/dev/microsoft-agents-testing/README.md @@ -96,14 +96,18 @@ client.expect().that_for_any(text="~Hello") # assert ``` Every method has an `ex_` variant (`ex_send`, `ex_invoke`, etc.) that returns -the raw `Exchange` objects instead of just the response activities. +the raw `Exchange` objects instead of just the response activities. The fluent +shortcuts are typed by collection: `expect()`/`select()` return +`ActivityExpect`/`ActivitySelect`, while `ex_expect()`/`ex_select()` return +`ExchangeExpect`/`ExchangeSelect`. ## Expect & Select Fluent API for asserting on and filtering response collections. `Expect` raises `AssertionError` with diagnostic context — it shows what was expected, what was received, and which items were checked. Prefix a value with `~` for -substring matching, or pass a lambda for custom logic. The variable named `x` has a special meaning and is passed in dynamically during evaluation. +substring matching, or pass a lambda for custom logic. Lambda parameters named +`x`, `actual`, or `value` receive the resolved value during evaluation. ```python client.expect().that_for_any(text="~hello") # any reply contains "hello" @@ -112,6 +116,19 @@ client.expect().that_for_exactly(2, type="message") # exactly 2 messages client.expect().that_for_any(text=lambda x: len(x) > 10) # lambda predicate ``` +Use `contains` for nested model, dict, and iterable values. It requires a +callable, dictionary filter, or keyword criteria; `contains()` and +`contains({})` are invalid because an unfiltered predicate would match +everything. + +```python +from microsoft_agents.testing.utils import contains + +client.expect().that_for_any( + attachments=contains(content_type="application/vnd.microsoft.card.hero") +) +``` + `Select` filters and slices before you assert or extract: ```python @@ -125,15 +142,14 @@ Select(client.history()).where(type="message").expect().that(text="~hello") Every request and response is recorded in a `Transcript`. When a test fails you can print the conversation to see exactly what happened. -`ConversationTranscriptFormatter` gives a chat-style view; -`ActivityTranscriptFormatter` shows all activities with selectable fields. -Both support `DetailLevel` (`MINIMAL`, `STANDARD`, `DETAILED`, `FULL`) and -`TimeFormat` (`CLOCK`, `RELATIVE`, `ELAPSED`). +`ConversationTranscriptFormatter` gives a chat-style view, +`ActivityTranscriptFormatter` shows a flat JSON activity stream, and +`JsonTranscriptFormatter` shows exchange-grouped JSON. ```python -from microsoft_agents.testing import ConversationTranscriptFormatter, DetailLevel +from microsoft_agents.testing import ConversationTranscriptFormatter -ConversationTranscriptFormatter(detail=DetailLevel.FULL).print(client.transcript) +print(ConversationTranscriptFormatter().format(client.transcript)) ``` ``` @@ -181,9 +197,12 @@ class TestEcho: ... | Document | Contents | |----------|----------| -| [MOTIVATION.md](MOTIVATION.md) | Before/after code comparison | -| [API.md](API.md) | Public API reference | -| [SAMPLES.md](SAMPLES.md) | Guide to the runnable samples | +| [MOTIVATION.md](docs/MOTIVATION.md) | Before/after code comparison | +| [API.md](docs/API.md) | Public API reference | +| [ASSERTIONS.md](docs/ASSERTIONS.md) | Fluent `Expect`, `Select`, predicates, lambdas, and assertion internals | +| [CLI.md](docs/CLI.md) | `agt` command guide | +| [UTILITIES.md](docs/UTILITIES.md) | `contains`, `poll`, `send`, and `ex_send` helper guide | +| [SAMPLES.md](docs/SAMPLES.md) | Guide to the runnable samples | ## License diff --git a/dev/microsoft-agents-testing/docs/API.md b/dev/microsoft-agents-testing/docs/API.md index d3b9cea35..b91fe4574 100644 --- a/dev/microsoft-agents-testing/docs/API.md +++ b/dev/microsoft-agents-testing/docs/API.md @@ -5,7 +5,8 @@ from microsoft_agents.testing import ( AiohttpScenario, ExternalScenario, Scenario, AgentEnvironment, AgentClient, ScenarioConfig, ClientConfig, ActivityTemplate, - Expect, Select, + Expect, Select, ActivityExpect, ActivitySelect, + ExchangeExpect, ExchangeSelect, Transcript, Exchange, ConversationTranscriptFormatter, ActivityTranscriptFormatter, JsonTranscriptFormatter, @@ -166,15 +167,23 @@ await client.send(activity, wait=0.5) | Method | Returns | Description | |--------|---------|-------------| -| `expect(history=False)` | `Expect` | Assert on response activities | -| `select(history=False)` | `Select` | Filter response activities | -| `ex_expect(history=False)` | `Expect` | Assert on exchanges | -| `ex_select(history=False)` | `Select` | Filter exchanges | +| `expect(history=False)` | `ActivityExpect` | Assert on response activities | +| `select(history=False)` | `ActivitySelect` | Filter response activities | +| `ex_expect(history=False)` | `ExchangeExpect` | Assert on exchanges | +| `ex_select(history=False)` | `ExchangeSelect` | Filter exchanges | ```python +from microsoft_agents.testing.utils import contains + # Assert any reply contains "hello" (case-sensitive substring) client.expect().that_for_any(text="~hello") +# Search nested model, dict, and iterable values. +# A callable, dictionary filter, or keyword criteria is required. +client.expect().that_for_any( + attachments=contains(content_type="application/vnd.microsoft.card.hero") +) + # Filter then assert client.select().where(type="message").expect().that(text="~world") ``` @@ -252,8 +261,13 @@ with diagnostic context on failure. ```python Expect(items: Iterable[dict | BaseModel]) +ActivityExpect(items: Iterable[Activity]) +ExchangeExpect(items: Iterable[Exchange]) ``` +`AgentClient.expect()` returns `ActivityExpect`; `AgentClient.ex_expect()` +returns `ExchangeExpect`. + | Method | Passes when… | |--------|-------------| | `that(**kwargs)` | **All** items match | @@ -304,8 +318,13 @@ Chainable filtering over a collection. ```python Select(items: Iterable[dict | BaseModel]) +ActivitySelect(items: Iterable[Activity]) +ExchangeSelect(items: Iterable[Exchange]) ``` +`AgentClient.select()` returns `ActivitySelect`; `AgentClient.ex_select()` +returns `ExchangeSelect`. + | Method | Description | |--------|-------------| | `where(**kwargs)` | Keep items matching criteria | @@ -607,7 +626,7 @@ agt scenario load --url http://localhost:3978/api/messages \ | Option | Default | Description | |--------|---------|-------------| | `--message / -m` | — | Text message to send | -| `--json-file / -j` | — | JSON activity file to send | +| `--json_file / -j` | — | JSON activity file to send | | `--num / -n` | *(required)* | Number of concurrent requests | | `--timeout / -t` | `5000` | Milliseconds per request before it is recorded as a timeout error | diff --git a/dev/microsoft-agents-testing/docs/ASSERTIONS.md b/dev/microsoft-agents-testing/docs/ASSERTIONS.md new file mode 100644 index 000000000..932075660 --- /dev/null +++ b/dev/microsoft-agents-testing/docs/ASSERTIONS.md @@ -0,0 +1,327 @@ +# Fluent Assertions + +The fluent assertion engine is a general-purpose way to assert over collections +of dictionaries and Pydantic models. It is used by `AgentClient`, but it is not +limited to transcripts or agent tests. + +```python +from microsoft_agents.testing import Expect, Select + +items = [ + {"type": "message", "text": "welcome"}, + {"type": "typing"}, +] + +Expect(items).that_for_any(type="message", text="~welcome") +Select(items).where(type="message").expect().that_for_none(text="~error") +``` + +## Core concepts + +| Concept | Use it for | +|---------|------------| +| `Expect` | Assert that items match criteria | +| `Select` | Filter, order, and slice items before asserting or reading them | +| `ActivityExpect` / `ActivitySelect` | Typed wrappers for activity collections | +| `ExchangeExpect` / `ExchangeSelect` | Typed wrappers for exchange collections | + +Most tests should start with `Expect(items)` or `Select(items)`. + +## `Expect` + +`Expect` wraps a collection of dictionaries or Pydantic models and raises an +`AssertionError` when the selected quantifier does not pass. + +```python +Expect(items).that_for_any(type="message") +Expect(items).that_for_none(text="~error") +Expect(items).that_for_one(text="hello") +Expect(items).that_for_exactly(2, type="message") +``` + +| Method | Passes when | +|--------|-------------| +| `that(...)` | All items match | +| `that_for_all(...)` | All items match | +| `that_for_any(...)` | At least one item matches | +| `that_for_none(...)` | No items match | +| `that_for_one(...)` | Exactly one item matches | +| `that_for_exactly(n, ...)` | Exactly `n` items match | + +Collection checks are separate from predicate checks: + +```python +Expect(items).is_not_empty() +Expect(items).has_count(2) +``` + +Quantifier methods return `self`, so assertions can be chained: + +```python +Expect(items) \ + .that_for_any(text="~hello") \ + .that_for_none(text="~error") \ + .has_count(2) +``` + +## `Select` + +`Select` filters a collection and returns a new selection. Use it when you want +to narrow a collection before asserting or extracting items. + +```python +messages = Select(items).where(type="message") + +messages.expect().that_for_any(text="~hello") +latest_message = messages.last().get()[0] +``` + +| Method | Description | +|--------|-------------| +| `where(...)` | Keep matching items | +| `where_not(...)` | Exclude matching items | +| `order_by(key, reverse=False)` | Sort by field name or callable criteria | +| `first(n=1)` | Keep the first `n` items | +| `last(n=1)` | Keep the last `n` items | +| `at(n)` | Keep the item at index `n` | +| `sample(n)` | Randomly sample up to `n` items | +| `get()` | Return the selected items | +| `count()` | Return the selected item count | +| `empty()` | Return whether the selection is empty | +| `expect()` | Switch to assertions over the selection | + +`where()` and `where_not()` use the same matching rules as `Expect`. + +## Matching rules + +Criteria can be provided as keyword arguments, a dictionary, or a root callable. +Multiple criteria on the same assertion must all match the same item. + +```python +# Keyword criteria +Expect(items).that_for_any(type="message", text="hello") + +# Dictionary criteria +Expect(items).that_for_any({"type": "message", "text": "hello"}) + +# Root callable criteria +Expect(items).that_for_any(lambda x: x["type"] == "message") +``` + +### Exact values + +Plain values become equality checks. + +```python +Expect(items).that_for_any(type="message") +Expect(items).that_for_any(channel_id="msteams") +``` + +### Substring values + +String values starting with `~` become case-sensitive substring checks. + +```python +Expect(items).that_for_any(text="~welcome") +Expect(items).that_for_none(text="~error") +``` + +Use a lambda when you need case-insensitive matching or more complex string +logic. + +### Dot-notation paths + +Nested values can be matched with dot-notation keys. + +```python +activities = [ + { + "type": "message", + "conversation": {"id": "conversation-1"}, + "from": {"id": "user-1"}, + } +] + +Expect(activities).that_for_any({ + "conversation.id": "conversation-1", + "from.id": "user-1", +}) +``` + +### Dictionary handling and expansion + +Dictionary criteria can be written either as nested dictionaries or as +dot-notation keys. The assertion engine treats both forms the same way: + +```python +nested_criteria = { + "conversation": {"id": "conversation-1"}, + "from": {"id": "user-1"}, +} + +dot_criteria = { + "conversation.id": "conversation-1", + "from.id": "user-1", +} + +Expect(activities).that_for_any(nested_criteria) +Expect(activities).that_for_any(dot_criteria) +``` + +This makes it possible to choose the form that best matches the test: nested +dictionaries are useful when the criteria mirrors a payload shape, while +dot-notation is concise for one-off nested checks. + +## Lambdas and callable predicates + +Callable predicates let assertions express checks that are awkward as exact or +substring values. + +```python +Expect(items).that_for_any(text=lambda x: len(x) > 10) +Select(items).where(attachments=lambda x: len(x) > 0) +``` + +The current invocation convention is intentionally explicit: + +- use a parameter named `x`, `actual`, or `value` to receive the resolved value +- for root callables, that resolved value is the whole item being evaluated +- parameters with other names are not populated by the framework + +These examples are equivalent: + +```python +Expect(items).that_for_any(text=lambda x: x.startswith("Hello")) +Expect(items).that_for_any(text=lambda actual: actual.startswith("Hello")) +Expect(items).that_for_any(text=lambda value: value.startswith("Hello")) +``` + +For field criteria, the lambda receives the value at that field: + +```python +Expect(items).that_for_any( + text=lambda x: isinstance(x, str) and len(x) > 20 +) +``` + +For root criteria, the lambda receives the whole dictionary or model: + +```python +Expect(items).that_for_any( + lambda x: x["type"] == "message" and x.get("text") +) +``` + +Because only `x`, `actual`, and `value` are populated, other parameter names do +not receive the field value: + +```python +Expect(items).that_for_any(text=lambda text: text.startswith("Hello")) +``` + +Prefer naming the parameter `x`, `actual`, or `value` so the predicate is clear +and portable across `Expect`, `Select`, `contains`, and the backend predicate +APIs. + +## Pydantic models + +The assertion engine accepts Pydantic models as well as dictionaries. Models are +converted to dictionaries for field matching, while root callables receive the +original model object. + +```python +from pydantic import BaseModel + + +class Reply(BaseModel): + type: str + text: str + + +replies = [Reply(type="message", text="welcome")] + +Expect(replies).that_for_any(type="message", text="~welcome") +Expect(replies).that_for_any(lambda x: x.text == "welcome") +``` + +## Typed activity and exchange wrappers + +The typed wrappers are specialized versions of `Expect` and `Select` for common +testing package models: + +```python +from microsoft_agents.testing import ActivityExpect, ActivitySelect + +ActivityExpect(activities).that_for_any(type="message") +ActivitySelect(activities).where(type="message").expect().is_not_empty() +``` + +`ExchangeExpect` and `ExchangeSelect` work the same way for exchange +collections: + +```python +from microsoft_agents.testing import ExchangeExpect, ExchangeSelect + +ExchangeExpect(exchanges).that_for_one(status_code=200) +ExchangeSelect(exchanges).where(status_code=200).expect().is_not_empty() +``` + +## AgentClient shortcuts + +`AgentClient` exposes convenience shortcuts for the current transcript. These +shortcuts return the typed wrappers described above: + +```python +client.expect() # ActivityExpect over response activities +client.select() # ActivitySelect over response activities +client.ex_expect() # ExchangeExpect over exchanges +client.ex_select() # ExchangeSelect over exchanges +``` + +Use these when testing an agent through a scenario: + +```python +await client.send("hello", wait=0.5) + +client.expect().that_for_any(type="message", text="~hello") +client.ex_expect().that_for_one(status_code=200) +``` + +## Nested values and `contains` + +Use `contains` when the value you care about can be nested inside a model, +dictionary, or iterable, such as attachments, entities, channel data, or card +payloads. + +```python +from microsoft_agents.testing.utils import contains + +Expect(activities).that_for_any( + attachments=contains(content_type="application/vnd.microsoft.card.hero") +) +``` + +`contains` accepts the same criteria shapes as the fluent APIs: callable, +dictionary, or keyword criteria. See [UTILITIES.md](UTILITIES.md) for the full +`Contains` guide. + +## Failure descriptions + +When an expectation fails, the assertion engine reports: + +- how many items matched +- which item indexes matched or failed +- which keys failed on each failed item +- the actual value found at each failed key +- expected values for generated equality predicates when available + +This makes exact and substring criteria more diagnosable than opaque custom +predicates, so prefer built-in criteria when they can express the assertion. + +## Sample + +Run the deep-dive assertions sample for an end-to-end walkthrough of this page: + +```bash +python -m docs.samples.deep_dive_assertions +``` diff --git a/dev/microsoft-agents-testing/docs/CLI.md b/dev/microsoft-agents-testing/docs/CLI.md new file mode 100644 index 000000000..a67438b89 --- /dev/null +++ b/dev/microsoft-agents-testing/docs/CLI.md @@ -0,0 +1,113 @@ +# CLI + +The `agt` command is the terminal interface for the testing package. Use it for +quick manual checks, interactive chats, one-off posts, simple load tests, and +scaffolding test harnesses. + +```bash +agt [--env FILE] [--connection NAME] [--verbose] COMMAND +``` + +| Global option | Default | Description | +|---------------|---------|-------------| +| `--env / -e FILE` | `.env` | Environment file to load | +| `--connection / -c NAME` | `SERVICE_CONNECTION` | Named connection for auth credentials | +| `--verbose / -v` | `False` | Print debug output | + +## Choosing an agent target + +Scenario commands use one of two target options: + +| Option | Description | +|--------|-------------| +| `--url / -u URL` | Connect to an agent already running at an HTTP endpoint | +| `--agent / -a NAME` | Use a scenario registered in `scenario_registry` | + +Use `--module MODULE` with scenario commands when the scenario registration +lives in another module and must be imported before lookup. + +```bash +agt scenario chat --url http://localhost:3978/api/messages +agt scenario chat --agent agt.basic +agt scenario chat --module my_tests.scenarios --agent local.echo +``` + +## Environment commands + +```bash +agt env show +agt env help +``` + +`env show` prints the Python/runtime context, loaded environment file, loaded +environment variable names, and registered scenario count. `env help` prints the +expected authentication-related `.env` variable names. + +## Scenario commands + +### List registered scenarios + +```bash +agt scenario list +agt scenario list "agt.*" +``` + +The optional pattern uses the same glob-style matching as +`scenario_registry.discover()`. + +### Interactive chat + +```bash +agt scenario chat --url http://localhost:3978/api/messages +agt scenario chat --agent agt.basic +``` + +Starts a REPL-style session. Type `/exit` or `/quit` to end the chat. + +### Post one activity + +```bash +agt scenario post --url http://localhost:3978/api/messages --message "Hello!" +agt scenario post --url http://localhost:3978/api/messages --json-file activity.json +``` + +`post` sends one text message or JSON activity and prints the resulting +transcript. Use `--timeout / -t` to control how long to wait for responses, in +milliseconds. + +### Load test + +```bash +agt scenario load --url http://localhost:3978/api/messages \ + --message "Hello!" --num 50 --timeout 5000 + +agt scenario load --url http://localhost:3978/api/messages \ + --json_file activity.json --num 20 +``` + +`load` sends the same message or activity concurrently and prints per-request +failures plus aggregate latency statistics. Use either `--message` or +`--json_file`, not both. + +### Run an in-process scenario + +```bash +agt scenario run --agent agt.basic +``` + +`run` starts an in-process scenario as a long-running local server. It is not +available for `ExternalScenario` targets because those agents are already +running. + +## Scaffolding + +```bash +agt init +agt init basic +agt init basic --force +``` + +Omit the preset name to list available presets. Provide a preset name to copy +that harness into the current directory. `--force` overwrites conflicting files +from the preset. + diff --git a/dev/microsoft-agents-testing/docs/README.md b/dev/microsoft-agents-testing/docs/README.md index c7db97205..70c4d2927 100644 --- a/dev/microsoft-agents-testing/docs/README.md +++ b/dev/microsoft-agents-testing/docs/README.md @@ -96,14 +96,18 @@ client.expect().that_for_any(text="~Hello") # assert ``` Every method has an `ex_` variant (`ex_send`, `ex_invoke`, etc.) that returns -the raw `Exchange` objects instead of just the response activities. +the raw `Exchange` objects instead of just the response activities. The fluent +shortcuts are typed by collection: `expect()`/`select()` return +`ActivityExpect`/`ActivitySelect`, while `ex_expect()`/`ex_select()` return +`ExchangeExpect`/`ExchangeSelect`. ## Expect & Select Fluent API for asserting on and filtering response collections. `Expect` raises `AssertionError` with diagnostic context — it shows what was expected, what was received, and which items were checked. Prefix a value with `~` for -substring matching, or pass a lambda for custom logic. The variable named `x` has a special meaning and is passed in dynamically during evaluation. +substring matching, or pass a lambda for custom logic. Lambda parameters named +`x`, `actual`, or `value` receive the resolved value during evaluation. ```python client.expect().that_for_any(text="~hello") # any reply contains "hello" @@ -112,6 +116,19 @@ client.expect().that_for_exactly(2, type="message") # exactly 2 messages client.expect().that_for_any(text=lambda x: len(x) > 10) # lambda predicate ``` +Use `contains` for nested model, dict, and iterable values. It requires a +callable, dictionary filter, or keyword criteria; `contains()` and +`contains({})` are invalid because an unfiltered predicate would match +everything. + +```python +from microsoft_agents.testing.utils import contains + +client.expect().that_for_any( + attachments=contains(content_type="application/vnd.microsoft.card.hero") +) +``` + `Select` filters and slices before you assert or extract: ```python @@ -220,6 +237,9 @@ Use `--agent NAME` instead of `--url` to target a named scenario from `scenario_ |----------|----------| | [MOTIVATION.md](MOTIVATION.md) | Before/after code comparison | | [API.md](API.md) | Public API reference | +| [ASSERTIONS.md](ASSERTIONS.md) | Fluent `Expect`, `Select`, predicates, lambdas, and assertion internals | +| [CLI.md](CLI.md) | `agt` command guide | +| [UTILITIES.md](UTILITIES.md) | `contains`, `poll`, `send`, and `ex_send` helper guide | | [SAMPLES.md](SAMPLES.md) | Guide to the runnable samples | ## License diff --git a/dev/microsoft-agents-testing/docs/SAMPLES.md b/dev/microsoft-agents-testing/docs/SAMPLES.md index 325aea8b7..41393ee5a 100644 --- a/dev/microsoft-agents-testing/docs/SAMPLES.md +++ b/dev/microsoft-agents-testing/docs/SAMPLES.md @@ -8,6 +8,8 @@ Runnable scripts in `docs/samples/`. Each is self-contained. | `interactive.py` | REPL chat with transcript on exit | | `scenario_registry_demo.py` | Registering and discovering named scenarios | | `transcript_formatting.py` | `ConversationTranscriptFormatter`, `ActivityTranscriptFormatter`, `JsonTranscriptFormatter` | +| `deep_dive_assertions.py` | Fluent assertion engine: `Expect`, `Select`, lambdas, typed wrappers, and internals | +| `utilities.py` | `contains`, `poll`, `send`, and `ex_send` helpers | | `pytest_plugin_usage.py` | `@pytest.mark.agent_test`, fixtures | | `multi_client.py` | Multiple users, `ActivityTemplate`, child clients | @@ -24,7 +26,7 @@ async with scenario.client() as client: ``` ```bash -python docs/samples/quickstart.py +python -m docs.samples.quickstart ``` --- @@ -34,7 +36,7 @@ python docs/samples/quickstart.py REPL loop. Type messages, see replies. Prints the full transcript on exit. ```bash -python docs/samples/interactive.py +python -m docs.samples.interactive ``` --- @@ -51,7 +53,7 @@ local = scenario_registry.discover("local.*") ``` ```bash -python docs/samples/scenario_registry_demo.py +python -m docs.samples.scenario_registry_demo ``` --- @@ -73,11 +75,67 @@ from microsoft_agents.testing import ( ) print_conversation(client.transcript) -print(ActivityTranscriptFormatter(model_dump_args={"exclude_none": True}).format(client.transcript)) +print( + ActivityTranscriptFormatter( + model_dump_args={"exclude_unset": True, "exclude_none": True} + ).format(client.transcript) +) +``` + +```bash +python -m docs.samples.transcript_formatting +``` + +--- + +## deep_dive_assertions.py + +Walks through the general-purpose fluent assertion engine without requiring a +running agent. + +- `Expect` and `Select` over dictionaries +- exact, substring, dictionary, dot-notation, and root callable criteria +- lambda predicates and the `x` / `actual` / `value` convention +- Pydantic model support +- `ActivityExpect`, `ActivitySelect`, `ExchangeExpect`, and `ExchangeSelect` +- `contains` + +```python +from microsoft_agents.testing import Expect, Select + +items = [{"type": "message", "text": "welcome"}] + +Expect(items).that_for_any(type="message", text="~welcome") +Select(items).where(type="message").expect().is_not_empty() +``` + +```bash +python -m docs.samples.deep_dive_assertions +``` + +--- + +## utilities.py + +Demonstrates the utility helpers for nested predicates, polling asynchronous +side effects, and one-shot sends to a running agent URL. + +- `contains` — searches nested model, dict, and iterable values +- `poll` — waits until a synchronous condition becomes true +- `send` — returns response `Activity` objects from an agent URL +- `ex_send` — returns full `Exchange` objects from an agent URL + +```python +from microsoft_agents.testing.utils import contains, poll, send, ex_send + +client.expect().that_for_any( + attachments=contains(content_type="application/vnd.microsoft.card.hero") +) +await poll(lambda: state["saved"], timeout=1.0, interval=0.01) ``` ```bash -python docs/samples/transcript_formatting.py +python -m docs.samples.utilities ``` --- @@ -118,7 +176,7 @@ async with scenario.run() as factory: ``` ```bash -python docs/samples/multi_client.py +python -m docs.samples.multi_client ``` --- @@ -148,7 +206,7 @@ agt scenario load --url http://localhost:3978/api/messages \ # Send a custom activity from a JSON file agt scenario load --url http://localhost:3978/api/messages \ - --json-file activity.json --num 20 + --json_file activity.json --num 20 ``` Reports per-request errors and aggregate latency (average, min, max, p90). diff --git a/dev/microsoft-agents-testing/docs/UTILITIES.md b/dev/microsoft-agents-testing/docs/UTILITIES.md new file mode 100644 index 000000000..c5adb6e1e --- /dev/null +++ b/dev/microsoft-agents-testing/docs/UTILITIES.md @@ -0,0 +1,177 @@ +# Utilities + +The `microsoft_agents.testing.utils` package contains small helpers for common +test tasks that do not need the full fluent API every time: + +| Utility | Use it for | +|---------|------------| +| `contains` | Search nested model, dict, and iterable values in `Expect` or `Select` predicates | +| `poll` | Wait for eventually consistent state or asynchronous side effects | +| `send` | Send one activity to a running agent URL and return response activities | +| `ex_send` | Send one activity to a running agent URL and return full `Exchange` objects | + +```python +from microsoft_agents.testing.utils import contains, ex_send, poll, send +from microsoft_agents.testing.utils.contains import Contains +``` + +## `contains` + +`contains` is a convenience factory that returns a `Contains` predicate. +`Contains` is a callable object that walks nested Pydantic models, +dictionaries, and iterables until it finds a matching value. Use it when the +value you need is inside a property such as `attachments`, `channel_data`, +entities, or a nested card payload. + +```python +client.expect().that_for_any( + attachments=contains(content_type="application/vnd.microsoft.card.hero") +) +``` + +### `Contains` class + +Use the `contains(...)` function for most tests. Import `Contains` directly when +you want to name, reuse, or configure the predicate before passing it to +`Expect`, `Select`, or plain Python code. + +```python +from microsoft_agents.testing.utils.contains import Contains + +has_hero_card = Contains(content_type="application/vnd.microsoft.card.hero") + +client.expect().that_for_any(attachments=has_hero_card) +hero_replies = client.select().where(has_hero_card).get() +assert has_hero_card(client.history()[0]) +``` + +`Contains` and `contains` accept the same criteria shapes as the fluent +predicate APIs: + +```python +contains(lambda value: value == "tenant-1") +contains({"content_type": "application/vnd.microsoft.card.hero"}) +contains(content_type="application/vnd.microsoft.card.hero") +contains({"content_type": "thumbnail"}, content_type="hero") +``` + +The same examples work with the class: + +```python +Contains(lambda value: value == "tenant-1") +Contains({"content_type": "application/vnd.microsoft.card.hero"}) +Contains(content_type="application/vnd.microsoft.card.hero") +``` + +Keyword criteria are merged with dictionary criteria, with keyword values taking +precedence for duplicate keys. `contains()` and `contains({})` are invalid +because an unfiltered predicate would match everything. Passing `None` as the +filter is also invalid. + +### Matching behavior + +`Contains` checks the current value first, then recursively visits nested values: + +| Source value | Traversal behavior | +|--------------|--------------------| +| Pydantic model | Visits model field values | +| `dict` | Visits dictionary values | +| Iterable | Visits each item, except strings and bytes | +| Scalar | Stops traversal if the predicate does not match | + +For SDK Pydantic models, criteria can use Python field names such as +`content_type`, even when serialized payloads use aliases such as +`contentType`. + +### Depth limits + +Use `.depth(n)` to limit traversal. The root object is depth `0`; nested model +fields, dictionary values, and iterable items increment the depth by one. +`depth()` returns a new `Contains` instance and does not mutate the original. + +```python +deep_search = contains(content_type="hero") +shallow_search = deep_search.depth(1) +``` + +Depth limits are useful when a broad value predicate could match too deep in a +large payload. For example, `depth(1)` checks the root object and its immediate +children only. + +### Common usage patterns + +Search within a specific property: + +```python +client.expect().that_for_any( + attachments=contains(content_type="application/vnd.microsoft.card.hero") +) +``` + +Search anywhere in an activity: + +```python +client.expect().that_for_any( + contains(lambda value: value == "tenant-1") +) +``` + +Filter before asserting: + +```python +hero_messages = client.select().where( + contains(content_type="application/vnd.microsoft.card.hero") +) +hero_messages.expect().is_not_empty() +``` + +Reuse a named predicate: + +```python +has_error_text = Contains(lambda value: isinstance(value, str) and "error" in value) + +client.expect().that_for_none(has_error_text) +``` + +## `poll` + +`poll` repeatedly evaluates a synchronous condition until it returns `True` or +the timeout expires. It is useful when an agent updates memory, writes a file, or +triggers another asynchronous side effect after the response has been sent. + +```python +await poll(lambda: state["saved"], timeout=2.0, interval=0.05) +``` + +`poll` raises `TimeoutError` when the condition never succeeds. The interval +must be non-negative, and the timeout must be greater than or equal to the +interval. + +## `send` and `ex_send` + +`send` and `ex_send` are convenience helpers for quick checks against an agent +that is already running at an HTTP endpoint. + +```python +replies = await send("Hello!", "http://localhost:3978/api/messages") +print(replies[0].text) + +exchanges = await ex_send( + {"type": "message", "text": "Hello from a dict payload"}, + "http://localhost:3978/api/messages", +) +print(exchanges[0].request.text) +``` + +Both helpers accept a string, a dictionary activity payload, or an `Activity` +instance. Use `send` when you only need response `Activity` objects. Use +`ex_send` when you need the request, response list, invoke response, timing, or +error metadata stored on each `Exchange`. + +## Sample + +Run the utilities sample for an end-to-end demonstration: + +```bash +python -m docs.samples.utilities +``` diff --git a/dev/microsoft-agents-testing/docs/samples/__init__.py b/dev/microsoft-agents-testing/docs/samples/__init__.py index 2f912ef91..f3b56997b 100644 --- a/dev/microsoft-agents-testing/docs/samples/__init__.py +++ b/dev/microsoft-agents-testing/docs/samples/__init__.py @@ -9,11 +9,11 @@ Samples ------- quickstart.py - Minimal example — send a message, print the reply. Shows + Minimal example - send a message, print the reply. Shows AiohttpScenario, scenario.client(), and send_expect_replies(). interactive.py - REPL loop — chat with an in-process agent, print the transcript + REPL loop - chat with an in-process agent, print the transcript on exit. scenario_registry_demo.py @@ -22,11 +22,15 @@ transcript_formatting.py Visualise conversations for debugging: ConversationTranscriptFormatter, - ActivityTranscriptFormatter, DetailLevel, TimeFormat, selectable fields, - and convenience functions. + ActivityTranscriptFormatter, JsonTranscriptFormatter, and convenience + functions. + +deep_dive_assertions.py + Fluent assertion engine walkthrough: Expect, Select, matching rules, + lambdas, typed wrappers, and contains. pytest_plugin_usage.py - Zero-boilerplate pytest tests using @pytest.mark.agent_test — class + Zero-boilerplate pytest tests using @pytest.mark.agent_test - class and function markers, fixtures (agent_client, agent_environment, etc.), registered scenario names, and Select/Expect through the client. diff --git a/dev/microsoft-agents-testing/docs/samples/deep_dive_assertions.py b/dev/microsoft-agents-testing/docs/samples/deep_dive_assertions.py new file mode 100644 index 000000000..aaca72f0b --- /dev/null +++ b/dev/microsoft-agents-testing/docs/samples/deep_dive_assertions.py @@ -0,0 +1,240 @@ +#!/usr/bin/env python +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +"""Deep Dive Assertions - fluent assertion engine walkthrough. + +Features demonstrated: + - Expect and Select over plain dictionaries. + - Exact, substring, dot-notation, dictionary, and root callable criteria. + - Dictionary criteria flattening and result expansion. + - Lambda predicates with the x / actual / value convention. + - Pydantic model support. + - ActivityExpect, ActivitySelect, ExchangeExpect, and ExchangeSelect. + - contains for nested values. + +Run:: + + python -m docs.samples.deep_dive_assertions +""" + +from pydantic import BaseModel + +from microsoft_agents.activity import Activity, Attachment +from microsoft_agents.testing import ( + ActivityExpect, + ActivitySelect, + ExchangeExpect, + ExchangeSelect, + Expect, + Select, +) +from microsoft_agents.testing.core.transport import Exchange +from microsoft_agents.testing.utils import contains + +HERO_CARD = "application/vnd.microsoft.card.hero" + + +class Reply(BaseModel): + """Small Pydantic model used to show model support.""" + + type: str + text: str + score: int + + +def demo_expect() -> None: + """Assert over plain dictionaries.""" + print("-- Expect --") + + items = [ + {"type": "message", "text": "hello world", "score": 10}, + {"type": "typing", "score": 1}, + {"type": "message", "text": "done", "score": 5}, + ] + + Expect(items).that_for_any(type="message", text="~hello") + Expect(items).that_for_none(text="~error") + Expect(items).that_for_exactly(2, type="message") + Expect(items).is_not_empty().has_count(3) + + print("Expect quantifiers passed.") + + +def demo_select() -> None: + """Filter dictionaries before asserting.""" + print("-- Select --") + + items = [ + {"type": "message", "text": "alpha", "score": 1}, + {"type": "message", "text": "bravo", "score": 2}, + {"type": "event", "name": "done", "score": 3}, + ] + + messages = Select(items).where(type="message") + messages.expect().that_for_any(text="alpha") + + latest_message = messages.last().get()[0] + assert latest_message["text"] == "bravo" + + non_events = Select(items).where_not(type="event") + assert non_events.count() == 2 + + print("Select filtering and slicing passed.") + + +def demo_matching_rules() -> None: + """Show exact, substring, dictionary, dot-notation, and root callable criteria.""" + print("-- Matching rules --") + + activities = [ + { + "type": "message", + "text": "welcome back", + "conversation": {"id": "conversation-1"}, + "from": {"id": "user-1"}, + } + ] + + Expect(activities).that_for_any(type="message") + Expect(activities).that_for_any(text="~welcome") + Expect(activities).that_for_any({"type": "message", "text": "~back"}) + Expect(activities).that_for_any( + { + "conversation.id": "conversation-1", + "from.id": "user-1", + } + ) + Expect(activities).that_for_any(lambda x: x["type"] == "message") + + print("All matching forms passed.") + + +def demo_dictionary_expansion() -> None: + """Show nested dictionary criteria and dot-notation expansion.""" + print("-- Dictionary handling --") + + activity = { + "type": "message", + "conversation": {"id": "conversation-1"}, + "from": {"id": "user-1"}, + } + + nested_criteria = { + "conversation": {"id": "conversation-1"}, + "from": {"id": "user-1"}, + } + dot_criteria = { + "conversation.id": "conversation-1", + "from.id": "user-1", + } + + Expect([activity]).that_for_any(nested_criteria) + Expect([activity]).that_for_any(dot_criteria) + + print("Nested dictionary and dot-notation criteria passed.") + + +def demo_lambdas() -> None: + """Show the current lambda invocation convention.""" + print("-- Lambdas --") + + items = [{"text": "Hello from assertions", "score": 42}] + + Expect(items).that_for_any(text=lambda x: x.startswith("Hello")) + Expect(items).that_for_any(text=lambda actual: "assertions" in actual) + Expect(items).that_for_any(text=lambda value: value.endswith("assertions")) + Expect(items).that_for_any(score=lambda x: x > 40) + + print("Lambda predicates passed.") + + +def demo_pydantic_models() -> None: + """Assert over Pydantic models.""" + print("-- Pydantic models --") + + replies = [ + Reply(type="message", text="welcome", score=10), + Reply(type="message", text="done", score=5), + ] + + Expect(replies).that_for_all(type="message") + Expect(replies).that_for_any(text="~welcome") + Expect(replies).that_for_any(lambda x: x.text == "welcome") + + selected = Select(replies).where(score=lambda x: x >= 10).get() + assert selected[0].text == "welcome" + + print("Pydantic model assertions passed.") + + +def demo_typed_wrappers() -> None: + """Use activity and exchange typed wrappers directly.""" + print("-- Typed wrappers --") + + activities = [ + Activity(type="message", text="hello"), + Activity(type="typing"), + ] + + ActivityExpect(activities).that_for_any(type="message", text="hello") + message_activities = ActivitySelect(activities).where(type="message").get() + assert message_activities[0].text == "hello" + + exchanges = [ + Exchange( + request=Activity(type="message", text="hello"), + status_code=200, + responses=[Activity(type="message", text="reply")], + ) + ] + + ExchangeExpect(exchanges).that_for_one(status_code=200) + successful = ExchangeSelect(exchanges).where(status_code=200).get() + assert successful[0].responses[0].text == "reply" + + print("Typed wrapper assertions passed.") + + +def demo_contains() -> None: + """Search nested values inside activity payloads.""" + print("-- contains --") + + activities = [ + Activity( + type="message", + text="card reply", + attachments=[ + Attachment( + content_type=HERO_CARD, + content={"title": "Deep dive"}, + ) + ], + ) + ] + + Expect(activities).that_for_any(attachments=contains(content_type=HERO_CARD)) + ActivitySelect(activities).where( + contains(content_type=HERO_CARD) + ).expect().is_not_empty() + + print("Nested contains assertions passed.") + + +def main() -> None: + print("Deep Dive Assertions Demo\n") + + demo_expect() + demo_select() + demo_matching_rules() + demo_dictionary_expansion() + demo_lambdas() + demo_pydantic_models() + demo_typed_wrappers() + demo_contains() + + print("\nAll assertion demos complete.") + + +if __name__ == "__main__": + main() diff --git a/dev/microsoft-agents-testing/docs/samples/interactive.py b/dev/microsoft-agents-testing/docs/samples/interactive.py index 46b6f8b09..e316c4de3 100644 --- a/dev/microsoft-agents-testing/docs/samples/interactive.py +++ b/dev/microsoft-agents-testing/docs/samples/interactive.py @@ -2,13 +2,13 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -"""Interactive REPL — chat with an in-process echo agent. +"""Interactive REPL - chat with an in-process echo agent. Features demonstrated: - - AiohttpScenario — host an agent in-process, no external server needed. - - AgentClient — send messages & receive replies. - - Transcript — automatic exchange recording. - - ConversationTranscriptFormatter — pretty-print the session on exit. + - AiohttpScenario - host an agent in-process, no external server needed. + - AgentClient - send messages and receive replies. + - Transcript - automatic exchange recording. + - ConversationTranscriptFormatter - pretty-print the session on exit. Run:: @@ -22,12 +22,11 @@ AiohttpScenario, AgentEnvironment, ConversationTranscriptFormatter, - DetailLevel, ) # --------------------------------------------------------------------------- -# 1) Define the agent — a simple echo handler +# 1) Define the agent - a simple echo handler # --------------------------------------------------------------------------- async def init_echo_agent(env: AgentEnvironment) -> None: @@ -65,9 +64,7 @@ async def main() -> None: # Print the full conversation transcript on exit print("\n--- Session transcript ---") - ConversationTranscriptFormatter(detail=DetailLevel.DETAILED).print( - client.transcript - ) + print(ConversationTranscriptFormatter().format(client.transcript)) if __name__ == "__main__": diff --git a/dev/microsoft-agents-testing/docs/samples/multi_client.py b/dev/microsoft-agents-testing/docs/samples/multi_client.py index 216da5cbd..3547b9911 100644 --- a/dev/microsoft-agents-testing/docs/samples/multi_client.py +++ b/dev/microsoft-agents-testing/docs/samples/multi_client.py @@ -2,14 +2,14 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -"""Multi-Client & Advanced Patterns — multiple users, child clients, templates. +"""Multi-Client & Advanced Patterns - multiple users, child clients, templates. Features demonstrated: - - scenario.run() + ClientFactory — create multiple independent clients. - - ClientConfig — per-client auth tokens, headers, templates. - - ActivityTemplate — set default fields on every outgoing activity. - - AgentClient.child() — scoped transcript isolation. - - Transcript hierarchy — parent/child exchange propagation. + - scenario.run() + ClientFactory - create multiple independent clients. + - ClientConfig - per-client auth tokens, headers, templates. + - ActivityTemplate - set default fields on every outgoing activity. + - AgentClient.child() - scoped transcript isolation. + - Transcript hierarchy - parent/child exchange propagation. Run:: @@ -24,14 +24,12 @@ AgentEnvironment, ClientConfig, ActivityTemplate, - Transcript, ConversationTranscriptFormatter, - DetailLevel, ) # --------------------------------------------------------------------------- -# Agent — identifies who is talking +# Agent - identifies who is talking # --------------------------------------------------------------------------- async def init_agent(env: AgentEnvironment) -> None: @@ -51,7 +49,7 @@ async def on_message(ctx: TurnContext, state: TurnState): async def demo_multi_client() -> None: """Create two clients with different identities in the same scenario run.""" - print("── 1. Multiple clients via scenario.run() ──\n") + print("-- 1. Multiple clients via scenario.run() --\n") async with scenario.run() as factory: # Each factory() call creates an independent client. @@ -89,12 +87,12 @@ async def demo_multi_client() -> None: # --------------------------------------------------------------------------- -# 2) ActivityTemplate — set defaults for all outgoing activities +# 2) ActivityTemplate - set defaults for all outgoing activities # --------------------------------------------------------------------------- async def demo_activity_template() -> None: """Show how templates apply default fields automatically.""" - print("── 2. ActivityTemplate defaults ──\n") + print("-- 2. ActivityTemplate defaults --\n") config = ClientConfig( activity_template=ActivityTemplate( @@ -112,7 +110,7 @@ async def demo_activity_template() -> None: replies = await client.send_expect_replies("template test") print(f"Agent replied: {replies[0].text}") - # The template enriched the outgoing activity with defaults — + # The template enriched the outgoing activity with defaults - # we can verify via the transcript's recorded request. exchange = client.ex_history()[0] req = exchange.request @@ -126,12 +124,12 @@ async def demo_activity_template() -> None: # --------------------------------------------------------------------------- -# 3) Child clients — transcript scoping +# 3) Child clients - transcript scoping # --------------------------------------------------------------------------- async def demo_child_client() -> None: """AgentClient.child() creates a scoped transcript branch.""" - print("── 3. Child clients & transcript hierarchy ──\n") + print("-- 3. Child clients & transcript hierarchy --\n") async with scenario.client() as parent: await parent.send_expect_replies("Parent message 1") @@ -140,8 +138,6 @@ async def demo_child_client() -> None: await child.send_expect_replies("Child message 1") await child.send_expect_replies("Child message 2") - await parent.send_expect_replies("Parent message 2") - # Parent transcript sees everything (its own + propagated from child) print(f"Parent transcript exchanges: {len(parent.transcript)}") @@ -149,14 +145,13 @@ async def demo_child_client() -> None: print(f"Child transcript exchanges : {len(child.transcript)}") print("\n--- Parent view ---") - ConversationTranscriptFormatter( - user_label="User", agent_label="Agent", detail=DetailLevel.STANDARD - ).print(parent.transcript) + print(ConversationTranscriptFormatter().format(parent.transcript)) print("\n--- Child view ---") - ConversationTranscriptFormatter( - user_label="User", agent_label="Agent", detail=DetailLevel.STANDARD - ).print(child.transcript) + print(ConversationTranscriptFormatter().format(child.transcript)) + + reply = (await parent.send_expect_replies("Parent message 2"))[0] + print(f"\nParent can continue independently: {reply.text}") print() diff --git a/dev/microsoft-agents-testing/docs/samples/pytest_plugin_usage.py b/dev/microsoft-agents-testing/docs/samples/pytest_plugin_usage.py index 476fa57bc..64b1caadc 100644 --- a/dev/microsoft-agents-testing/docs/samples/pytest_plugin_usage.py +++ b/dev/microsoft-agents-testing/docs/samples/pytest_plugin_usage.py @@ -2,16 +2,16 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -"""Pytest Plugin — use @pytest.mark.agent_test for zero-boilerplate tests. +"""Pytest Plugin - use @pytest.mark.agent_test for zero-boilerplate tests. Features demonstrated: - - @pytest.mark.agent_test(scenario) — class-level and function-level markers. - - agent_client fixture — sends messages, collects replies. - - agent_environment fixture — inspect the agent's internals. - - Derived fixtures — agent_application, storage, adapter, + - @pytest.mark.agent_test(scenario) - class-level and function-level markers. + - agent_client fixture - sends messages, collects replies. + - agent_environment fixture - inspect the agent's internals. + - Derived fixtures - agent_application, storage, adapter, authorization, connection_manager. - - Registered scenario names — pass a string name instead of an object. - - Expect / Select via client — fluent assertions right on the client. + - Registered scenario names - pass a string name instead of an object. + - Expect / Select via client - fluent assertions right on the client. Run:: @@ -25,8 +25,12 @@ from microsoft_agents.hosting.core import TurnContext, TurnState, AgentApplication from microsoft_agents.testing import ( + ActivityExpect, + ActivitySelect, AiohttpScenario, AgentEnvironment, + ExchangeExpect, + ExchangeSelect, scenario_registry, ) @@ -45,7 +49,7 @@ async def on_message(ctx: TurnContext, state: TurnState): # --------------------------------------------------------------------------- -# 1) Class-level marker — every test in the class gets the same scenario +# 1) Class-level marker - every test in the class gets the same scenario # --------------------------------------------------------------------------- @pytest.mark.agent_test(echo_scenario) @@ -71,7 +75,7 @@ def test_derived_fixtures(self, agent_application, storage, adapter): # --------------------------------------------------------------------------- -# 2) Function-level marker — different scenarios per test +# 2) Function-level marker - different scenarios per test # --------------------------------------------------------------------------- class TestFunctionLevelMarker: @@ -84,7 +88,7 @@ async def test_echo(self, agent_client): # --------------------------------------------------------------------------- -# 3) Registered scenario name — look up by string +# 3) Registered scenario name - look up by string # --------------------------------------------------------------------------- # Register the scenario so it can be referenced by name @@ -119,9 +123,19 @@ async def test_expect_shortcuts(self, agent_client): await agent_client.send_expect_replies("BBB") # expect(history=True) asserts over all responses so far - agent_client.expect(history=True).that_for_any(text="Echo: AAA") - agent_client.expect(history=True).that_for_any(text="Echo: BBB") + expect: ActivityExpect = agent_client.expect(history=True) + expect.that_for_any(text="Echo: AAA") + expect.that_for_any(text="Echo: BBB") # select(history=True) lets you filter first - msgs = agent_client.select(history=True).where(type="message").get() + select: ActivitySelect = agent_client.select(history=True) + msgs = select.where(type="message").get() assert len(msgs) >= 2 + + # ex_expect/ex_select assert over full Exchange objects + ex_expect: ExchangeExpect = agent_client.ex_expect(history=True) + ex_expect.that_for_any(status_code=200) + + ex_select: ExchangeSelect = agent_client.ex_select(history=True) + successful = ex_select.where(status_code=200).get() + assert len(successful) >= 2 diff --git a/dev/microsoft-agents-testing/docs/samples/quickstart.py b/dev/microsoft-agents-testing/docs/samples/quickstart.py index 4936a5189..a044fb5ef 100644 --- a/dev/microsoft-agents-testing/docs/samples/quickstart.py +++ b/dev/microsoft-agents-testing/docs/samples/quickstart.py @@ -2,13 +2,13 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -"""Quickstart — the simplest possible agent test, no pytest required. +"""Quickstart - the simplest possible agent test, no pytest required. Features demonstrated: - - AiohttpScenario — in-process agent hosting. - - scenario.client() — async context manager that starts the agent, + - AiohttpScenario - in-process agent hosting. + - scenario.client() - async context manager that starts the agent, yields an AgentClient, and tears everything down. - - send_expect_replies() — send a message and get the inline replies. + - send_expect_replies() - send a message and get the inline replies. Run:: diff --git a/dev/microsoft-agents-testing/docs/samples/scenario_registry_demo.py b/dev/microsoft-agents-testing/docs/samples/scenario_registry_demo.py index 8bde8d8bc..562a53c9a 100644 --- a/dev/microsoft-agents-testing/docs/samples/scenario_registry_demo.py +++ b/dev/microsoft-agents-testing/docs/samples/scenario_registry_demo.py @@ -2,14 +2,14 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -"""Scenario Registry — register, discover, and look up named scenarios. +"""Scenario Registry - register, discover, and look up named scenarios. Features demonstrated: - - scenario_registry.register() — register a scenario under a name. - - scenario_registry.get() — retrieve a scenario by name. - - scenario_registry.discover() — glob-pattern discovery across namespaces. - - Dot-notation namespacing — organise scenarios as "namespace.name". - - load_scenarios() — bulk-register from an importable module. + - scenario_registry.register() - register a scenario under a name. + - scenario_registry.get() - retrieve a scenario by name. + - scenario_registry.discover() - glob-pattern discovery across namespaces. + - Dot-notation namespacing - organise scenarios as "namespace.name". + - load_scenarios() - bulk-register from an importable module. Run:: @@ -66,24 +66,24 @@ async def h(ctx: TurnContext, state: TurnState): async def main() -> None: - # ── get() — retrieve a single scenario by exact name ──────────── + # get() - retrieve a single scenario by exact name echo = scenario_registry.get("samples.echo") async with echo.client() as client: replies = await client.send_expect_replies("World") print(f"Echo agent replied: {replies[0].text}") - # ── discover() — find scenarios matching a glob pattern ───────── + # discover() - find scenarios matching a glob pattern all_samples = scenario_registry.discover("samples.*") print(f"\nDiscovered {len(all_samples)} scenario(s) in 'samples' namespace:") for name, entry in all_samples.items(): print(f" {name:25s} {entry.description}") - # ── Iterate all registered scenarios ──────────────────────────── + # Iterate all registered scenarios print(f"\nAll registered scenarios ({len(scenario_registry)}):") for entry in scenario_registry: print(f" {entry.name:25s} namespace={entry.namespace!r}") - # ── Membership check ──────────────────────────────────────────── + # Membership check assert "samples.echo" in scenario_registry assert "nonexistent" not in scenario_registry diff --git a/dev/microsoft-agents-testing/docs/samples/test_motivation_assertions.py b/dev/microsoft-agents-testing/docs/samples/test_motivation_assertions.py index 0754e42c7..03b9f3a3c 100644 --- a/dev/microsoft-agents-testing/docs/samples/test_motivation_assertions.py +++ b/dev/microsoft-agents-testing/docs/samples/test_motivation_assertions.py @@ -1,7 +1,7 @@ """Verify the assertion failure outputs shown in MOTIVATION.md. Run with: pytest tests/test_motivation_assertions.py -v -Both tests are expected to FAIL — the point is to compare the error messages. +Both tests are expected to FAIL - the point is to compare the error messages. """ import re @@ -44,7 +44,7 @@ class FakeActivity: type="message", channel_id="msteams", locale="en-US", - text="Your order confirmed — Order #123456", + text="Your order confirmed - Order #123456", from_property=ChannelAccount(id="bot-2", name="HelperBot"), # wrong name conversation=ConversationAccount(id="thread-002"), ), diff --git a/dev/microsoft-agents-testing/docs/samples/transcript_formatting.py b/dev/microsoft-agents-testing/docs/samples/transcript_formatting.py index ac826dbe9..6610d3e19 100644 --- a/dev/microsoft-agents-testing/docs/samples/transcript_formatting.py +++ b/dev/microsoft-agents-testing/docs/samples/transcript_formatting.py @@ -2,15 +2,14 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. -"""Transcript Formatting — visualise agent conversations for debugging. +"""Transcript Formatting - visualize agent conversations for debugging. Features demonstrated: - - Transcript / Exchange — automatic recording of every request & response. - - ConversationTranscriptFormatter — chat-style view with custom labels. - - ActivityTranscriptFormatter — field-level view with selectable columns. - - DetailLevel (MINIMAL → FULL) — control how much context is printed. - - TimeFormat (CLOCK / RELATIVE / ELAPSED) — timestamp display styles. - - print_conversation / print_activities — one-liner convenience functions. + - Transcript / Exchange - automatic recording of every request and response. + - ConversationTranscriptFormatter - chat-style transcript view. + - ActivityTranscriptFormatter - flat JSON array of Activity objects. + - JsonTranscriptFormatter - JSON array of Exchange objects. + - print_conversation / print_activities / print_json convenience functions. Run:: @@ -22,181 +21,81 @@ from microsoft_agents.activity import Activity, ActivityTypes from microsoft_agents.hosting.core import TurnContext, TurnState from microsoft_agents.testing import ( - AiohttpScenario, + ActivityTranscriptFormatter, AgentEnvironment, + AiohttpScenario, ConversationTranscriptFormatter, - ActivityTranscriptFormatter, - DetailLevel, -) -from microsoft_agents.testing.transcript_formatter import ( - TimeFormat, - print_conversation, + JsonTranscriptFormatter, print_activities, - DEFAULT_ACTIVITY_FIELDS, - EXTENDED_ACTIVITY_FIELDS, + print_conversation, + print_json, ) -# --------------------------------------------------------------------------- -# Agents -# --------------------------------------------------------------------------- - -async def init_echo(env: AgentEnvironment) -> None: - @env.agent_application.activity("message") - async def h(ctx: TurnContext, state: TurnState): - await ctx.send_activity(f"Echo: {ctx.activity.text}") - - async def init_multi_reply(env: AgentEnvironment) -> None: - """Agent that sends a typing indicator then multiple messages.""" + """Register an agent that sends multiple activity types.""" + @env.agent_application.activity("message") - async def h(ctx: TurnContext, state: TurnState): + async def on_message(ctx: TurnContext, state: TurnState) -> None: await ctx.send_activity(Activity(type=ActivityTypes.typing)) await ctx.send_activity("Processing your request...") await ctx.send_activity(f"Here is your answer about: {ctx.activity.text}") -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - def section(title: str) -> None: print(f"\n{'=' * 60}") print(f" {title}") print(f"{'=' * 60}") -# --------------------------------------------------------------------------- -# Demos -# --------------------------------------------------------------------------- - -async def demo_detail_levels() -> None: - """Show every DetailLevel with the ConversationTranscriptFormatter.""" - section("ConversationTranscriptFormatter — Detail Levels") - - scenario = AiohttpScenario(init_echo, use_jwt_middleware=False) - async with scenario.client() as client: - await client.send_expect_replies("Hello!") - await client.send_expect_replies("How are you?") - await client.send_expect_replies("Goodbye") - transcript = client.transcript - - for level in DetailLevel: - print(f"\n--- {level.name} ---") - ConversationTranscriptFormatter(detail=level).print(transcript) - - -async def demo_custom_labels() -> None: - """ConversationTranscriptFormatter with custom labels.""" - section("Custom Labels (User ↔ Bot)") - - scenario = AiohttpScenario(init_echo, use_jwt_middleware=False) - async with scenario.client() as client: - await client.send_expect_replies("ping") - await client.send_expect_replies("pong") - transcript = client.transcript - - ConversationTranscriptFormatter( - user_label="Human", - agent_label="Bot", - ).print(transcript) - - -async def demo_time_formats() -> None: - """Show CLOCK, RELATIVE, and ELAPSED timestamp styles.""" - section("TimeFormat — CLOCK / RELATIVE / ELAPSED") - - scenario = AiohttpScenario(init_echo, use_jwt_middleware=False) - async with scenario.client() as client: - await client.send_expect_replies("First") - await client.send_expect_replies("Second") - await client.send_expect_replies("Third") - transcript = client.transcript - - for tf in TimeFormat: - print(f"\n--- {tf.name} ---") - ConversationTranscriptFormatter( - detail=DetailLevel.DETAILED, - time_format=tf, - ).print(transcript) - - -async def demo_activity_formatter() -> None: - """ActivityTranscriptFormatter with selectable field columns.""" - section("ActivityTranscriptFormatter — Selectable Fields") - +async def create_transcript(): + """Run the sample agent and return its transcript.""" scenario = AiohttpScenario(init_multi_reply, use_jwt_middleware=False) async with scenario.client() as client: - await client.send_expect_replies("quantum physics") - transcript = client.transcript + await client.send_expect_replies("transcript formatting") + return client.transcript - print(f"\n--- Default fields: {DEFAULT_ACTIVITY_FIELDS} ---") - ActivityTranscriptFormatter().print(transcript) - print(f"\n--- Minimal (type + text only) ---") - ActivityTranscriptFormatter(fields=["type", "text"]).print(transcript) +async def demo_formatters() -> None: + """Show the three transcript formatter outputs.""" + transcript = await create_transcript() - print(f"\n--- Extended fields with timing ---") - ActivityTranscriptFormatter( - fields=EXTENDED_ACTIVITY_FIELDS, - detail=DetailLevel.DETAILED, - ).print(transcript) + section("ConversationTranscriptFormatter") + print(ConversationTranscriptFormatter().format(transcript)) - print(f"\n--- FULL detail ---") - ActivityTranscriptFormatter(detail=DetailLevel.FULL).print(transcript) + section("ActivityTranscriptFormatter") + print( + ActivityTranscriptFormatter( + model_dump_args={"exclude_unset": True, "exclude_none": True} + ).format(transcript) + ) - -async def demo_show_other_types() -> None: - """Toggle visibility of non-message activities (e.g. typing).""" - section("show_other_types — Typing Indicators") - - scenario = AiohttpScenario(init_multi_reply, use_jwt_middleware=False) - async with scenario.client() as client: - await client.send_expect_replies("test") - transcript = client.transcript - - print("\n--- show_other_types=False (default) ---") - ConversationTranscriptFormatter(show_other_types=False).print(transcript) - - print("\n--- show_other_types=True ---") - ConversationTranscriptFormatter(show_other_types=True).print(transcript) + section("JsonTranscriptFormatter") + print( + JsonTranscriptFormatter( + model_dump_args={"exclude_unset": True, "exclude_none": True} + ).format(transcript) + ) async def demo_convenience_functions() -> None: - """print_conversation() and print_activities() one-liners.""" - section("Convenience Functions") - - scenario = AiohttpScenario(init_echo, use_jwt_middleware=False) - async with scenario.client() as client: - await client.send_expect_replies("Quick test") - transcript = client.transcript + """Show the one-line print helpers.""" + transcript = await create_transcript() - print("\n--- print_conversation() ---") + section("print_conversation") print_conversation(transcript) - print("\n--- print_conversation(detail=FULL) ---") - print_conversation(transcript, detail=DetailLevel.FULL) - - print("\n--- print_activities() ---") + section("print_activities") print_activities(transcript) - print("\n--- print_activities(fields=['type', 'text', 'id']) ---") - print_activities(transcript, fields=["type", "text", "id"]) - + section("print_json") + print_json(transcript) -# --------------------------------------------------------------------------- -# Main -# --------------------------------------------------------------------------- async def main() -> None: print("Transcript Formatting Demo") - print("Shows how conversations look with each formatter and option.\n") - await demo_detail_levels() - await demo_custom_labels() - await demo_time_formats() - await demo_activity_formatter() - await demo_show_other_types() + await demo_formatters() await demo_convenience_functions() print(f"\n{'=' * 60}") diff --git a/dev/microsoft-agents-testing/docs/samples/utilities.py b/dev/microsoft-agents-testing/docs/samples/utilities.py new file mode 100644 index 000000000..284be586f --- /dev/null +++ b/dev/microsoft-agents-testing/docs/samples/utilities.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +"""Utilities - contains, poll, send, and ex_send. + +Features demonstrated: + - contains - search nested model, dict, and iterable values. + - poll - wait for asynchronous side effects. + - send - quick activity send against a running agent URL. + - ex_send - send and inspect Exchange metadata. + +Run:: + + python -m docs.samples.utilities +""" + +import asyncio + +from microsoft_agents.activity import Activity, Attachment, DeliveryModes +from microsoft_agents.hosting.core import TurnContext, TurnState +from microsoft_agents.testing import AiohttpScenario, AgentEnvironment, ScenarioConfig +from microsoft_agents.testing.utils import contains, ex_send, poll, send + +HERO_CARD = "application/vnd.microsoft.card.hero" +AGENT_URL = "http://127.0.0.1:3978/api/messages" + + +async def init_agent(env: AgentEnvironment) -> None: + """Register an agent that replies with text and a nested attachment.""" + + @env.agent_application.activity("message") + async def on_message(ctx: TurnContext, state: TurnState) -> None: + await ctx.send_activity( + Activity( + type="message", + text=f"Echo: {ctx.activity.text}", + attachments=[ + Attachment( + content_type=HERO_CARD, + content={"title": "Utility sample"}, + ) + ], + ) + ) + + +scenario = AiohttpScenario( + init_agent, + config=ScenarioConfig(callback_server_port=9379), + use_jwt_middleware=False, +) + + +async def demo_contains() -> None: + """Use contains with Expect and Select.""" + print("-- contains --") + + async with scenario.client() as client: + await client.send_expect_replies("show me a card") + + client.expect().that_for_any(attachments=contains(content_type=HERO_CARD)) + + selected = client.select().where(contains(content_type=HERO_CARD)).get() + print(f"Activities with hero cards: {len(selected)}") + + +async def demo_poll() -> None: + """Wait for a side effect to appear.""" + print("-- poll --") + + state = {"saved": False} + + async def save_later() -> None: + await asyncio.sleep(0.05) + state["saved"] = True + + asyncio.create_task(save_later()) + await poll(lambda: state["saved"], timeout=1.0, interval=0.01) + print("Asynchronous state was saved.") + + +async def demo_send_helpers() -> None: + """Use send and ex_send against a running agent endpoint.""" + print("-- send / ex_send --") + + async with scenario.run(): + replies = await send("hello from send", AGENT_URL, listen_duration=0.2) + print(f"send returned: {replies[0].text}") + + exchange_activity = Activity( + type="message", + text="hello from ex_send", + delivery_mode=DeliveryModes.expect_replies, + ) + + exchanges = await ex_send(exchange_activity, AGENT_URL, listen_duration=0.0) + print(f"ex_send request text: {exchanges[0].request.text}") + print(f"ex_send response count: {len(exchanges[0].responses)}") + + +async def main() -> None: + print("Utilities Demo\n") + + await demo_contains() + await demo_poll() + await demo_send_helpers() + + print("\nAll utility demos complete.") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/__init__.py b/dev/microsoft-agents-testing/microsoft_agents/testing/__init__.py index c01abad58..fbcb8ee3e 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/__init__.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/__init__.py @@ -47,8 +47,14 @@ Sender, Transcript, Exchange, + ExpectBase, + SelectBase, Expect, Select, + ActivityExpect, + ActivitySelect, + ExchangeExpect, + ExchangeSelect, Unset, ) @@ -95,6 +101,12 @@ "Exchange", "Expect", "Select", + "ExpectBase", + "SelectBase", + "ActivityExpect", + "ActivitySelect", + "ExchangeExpect", + "ExchangeSelect", "Unset", "AgentEnvironment", "AiohttpScenario", diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/__init__.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/__init__.py index c47a7e701..0bfff9b88 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/core/__init__.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/__init__.py @@ -14,7 +14,9 @@ """ from .fluent import ( + ExpectBase, Expect, + SelectBase, Select, ModelTemplate, ActivityTemplate, @@ -32,6 +34,13 @@ Sender, ) +from .type_defs import ( + ActivityExpect, + ActivitySelect, + ExchangeExpect, + ExchangeSelect, +) + from .agent_client import AgentClient from ._aiohttp_client_factory import _AiohttpClientFactory from .scenario import Scenario, ScenarioConfig, ClientFactory @@ -47,6 +56,12 @@ __all__ = [ "Expect", "Select", + "ExpectBase", + "SelectBase", + "ActivityExpect", + "ActivitySelect", + "ExchangeExpect", + "ExchangeSelect", "ModelTemplate", "ActivityTemplate", "ModelTransform", diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/agent_client.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/agent_client.py index 3e3524dab..e3e309870 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/core/agent_client.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/agent_client.py @@ -28,6 +28,12 @@ Exchange, Sender ) +from .type_defs import ( + ActivityExpect, + ActivitySelect, + ExchangeExpect, + ExchangeSelect, +) from .utils import activities_from_ex # Default field values applied to all outgoing activities @@ -151,37 +157,37 @@ def clear(self) -> None: ### Utilities ### - def ex_select(self, history: bool = False) -> Select: + def ex_select(self, history: bool = False) -> ExchangeSelect: """Create a Select instance for filtering exchanges. :param history: If True, includes full history; otherwise, recent only. :return: A Select instance for fluent filtering. """ - return Select(self._ex_collect(history=history)) + return ExchangeSelect(self._ex_collect(history=history)) - def select(self, history: bool = False) -> Select: + def select(self, history: bool = False) -> ActivitySelect: """Create a Select instance for filtering activities. :param history: If True, includes full history; otherwise, recent only. :return: A Select instance for fluent filtering. """ - return Select(self._collect(history=history)) + return ActivitySelect(self._collect(history=history)) - def ex_expect(self, history: bool = False) -> Expect: + def ex_expect(self, history: bool = False) -> ExchangeExpect: """Create an Expect instance for asserting on exchanges. :param history: If True, includes full history; otherwise, recent only. :return: An Expect instance for fluent assertions. """ - return Expect(self._ex_collect(history=history)) + return ExchangeExpect(self._ex_collect(history=history)) - def expect(self, history: bool = False) -> Expect: + def expect(self, history: bool = False) -> ActivityExpect: """Create an Expect instance for asserting on activities. :param history: If True, includes full history; otherwise, recent only. :return: An Expect instance for fluent assertions. """ - return Expect(self._collect(history=history)) + return ActivityExpect(self._collect(history=history)) ### ### Sending API diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/__init__.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/__init__.py index 607b543a2..012d48bdc 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/__init__.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/__init__.py @@ -33,8 +33,8 @@ Unset, ) -from .expect import Expect -from .select import Select +from .expect import Expect, ExpectBase +from .select import Select, SelectBase from .model_template import ModelTemplate, ActivityTemplate from .utils import normalize_model_data @@ -52,7 +52,9 @@ "for_n", "ActivityTemplate", "Expect", + "ExpectBase", "Select", + "SelectBase", "ModelTemplate", "flatten", "expand", diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/model_predicate.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/model_predicate.py index 08cf4ff4b..469555b01 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/model_predicate.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/model_predicate.py @@ -9,16 +9,12 @@ from __future__ import annotations -from typing import Callable, cast +from typing import Any, Callable, cast, Sequence from dataclasses import dataclass from pydantic import BaseModel from .transform import DictionaryTransform, ModelTransform -from .quantifier import ( - Quantifier, - for_all, -) @dataclass class ModelPredicateResult: @@ -34,24 +30,32 @@ class ModelPredicateResult: result_dicts: Detailed results per item showing which fields matched. """ - source: list[dict] + source: Sequence[dict] dict_transform: dict result_bools: list[bool] result_dicts: list[dict] - def __init__(self, source: list[dict] | list[BaseModel], dict_transform: dict, result_dicts: list[dict]) -> None: - if isinstance(source, list) and source and isinstance(source[0], BaseModel): - source = cast(list[BaseModel], source) - self.source = cast(list[dict], [s.model_dump(exclude_unset=True, mode="json") for s in source]) + def __init__(self, source: Sequence[dict | BaseModel], dict_transform: dict, result_dicts: list[dict]) -> None: + if isinstance(source, Sequence) and source and isinstance(source[0], BaseModel): + source = cast(Sequence[BaseModel], source) + self.source = cast(Sequence[dict], [s.model_dump(exclude_unset=True, mode="json") for s in source]) else: - self.source = cast(list[dict], source) + self.source = cast(Sequence[dict], source) self.dict_transform = dict_transform self.result_dicts = result_dicts - self.result_bools = [ self._truthy(d) for d in self.result_dicts ] + predicate_paths = list(self.dict_transform.keys()) + self.result_bools = [ + self._truthy(d, predicate_paths=predicate_paths) for d in self.result_dicts + ] - def _truthy(self, result: dict | list) -> bool: + def _truthy( + self, result: dict | Sequence, predicate_paths: Sequence[str] | None = None + ) -> bool: - res: list[bool] = [] + if predicate_paths: + return all(bool(self._get_path(result, path)) for path in predicate_paths) + + res: Sequence[bool] = [] if isinstance(result, dict): iterable = result.values() @@ -66,6 +70,18 @@ def _truthy(self, result: dict | list) -> bool: return all(res) + def _get_path(self, result: dict | Sequence, path: str) -> Any: + if isinstance(result, dict) and path in result: + return result[path] + + current: Any = result + for key in path.split("."): + if not isinstance(current, dict) or key not in current: + return False + current = current[key] + + return current + class ModelPredicate: """Evaluates predicates against models to produce boolean results. @@ -77,14 +93,14 @@ def __init__(self, dict_transform: DictionaryTransform) -> None: self._dt = dict_transform self._transform = ModelTransform(dict_transform) - def eval(self, source: dict | BaseModel | list[dict] | list[BaseModel]) -> ModelPredicateResult: + def eval(self, source: dict | BaseModel | Sequence[BaseModel | dict]) -> ModelPredicateResult: """Evaluate the predicate against one or more models. :param source: A single model or a list of models to evaluate. :return: A ModelPredicateResult with per-item match results. """ - if not isinstance(source, list): - source = cast(list[dict] | list[BaseModel], [source]) + if not isinstance(source, Sequence): + source = cast(Sequence[dict] | Sequence[BaseModel], [source]) res = self._transform.eval(source) return ModelPredicateResult(source, self._dt.map, res) diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/transform.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/transform.py index c7d728cca..fd31084de 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/transform.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/transform.py @@ -10,11 +10,11 @@ from __future__ import annotations import inspect -from typing import Any, Callable, overload, TypeVar, cast +from typing import Any, Callable, overload, TypeVar, cast, Sequence from pydantic import BaseModel -from .types import Unset, SafeObject, resolve, parent +from .types import SafeObject, resolve from .utils import expand, flatten T = TypeVar("T") @@ -85,7 +85,7 @@ def _invoke( """Invoke a predicate function with the resolved value for a key. Uses introspection to determine whether the function expects - its argument as 'actual' or 'x'. + its argument as 'actual', 'x', or 'value'. :param actual: The source dictionary. :param key: The dot-notation key to resolve from the dictionary. @@ -102,6 +102,8 @@ def _invoke( args["actual"] = self._get(actual, key) elif "x" in func_args: args["x"] = self._get(actual, key) + elif "value" in func_args: + args["value"] = self._get(actual, key) return func(**args) @@ -166,8 +168,8 @@ def __init__(self, dict_transform: DictionaryTransform) -> None: @overload def eval(self, source: dict | BaseModel) -> dict: ... @overload - def eval(self, source: list[dict] | list[BaseModel]) -> list[dict]: ... - def eval(self, source: dict | BaseModel | list[dict] | list[BaseModel]) -> list[dict] | dict: + def eval(self, source: Sequence[dict | BaseModel]) -> list[dict]: ... + def eval(self, source: dict | BaseModel | Sequence[dict | BaseModel]) -> list[dict] | dict: """Evaluate the underlying DictionaryTransform against one or more models. Pydantic models are dumped to dictionaries before evaluation. @@ -175,19 +177,19 @@ def eval(self, source: dict | BaseModel | list[dict] | list[BaseModel]) -> list[ :param source: A single model/dict or a list of models/dicts. :return: Evaluation result(s) as dictionaries of boolean outcomes. """ - if not isinstance(source, list): - source = cast(list[dict] | list[BaseModel], [source]) + if not isinstance(source, Sequence): + source = cast(Sequence[dict] | Sequence[BaseModel], [source]) items = source else: - items = cast(list[dict] | list[BaseModel], source) + items = cast(Sequence[dict] | Sequence[BaseModel], source) if len(items) > 0 and isinstance(items[0], BaseModel): - items = cast(list[BaseModel], items) + items = cast(Sequence[BaseModel], items) items = [ item.model_dump(exclude_unset=True, exclude_none=True, by_alias=True) for item in items ] - items = cast(list[dict], items) + items = cast(Sequence[dict], items) results = [] for i, item in enumerate(items): diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py index eb24c308f..322b24d46 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py @@ -12,7 +12,7 @@ from .readonly import Readonly -class Unset(Readonly): +class _Unset(Readonly): """Singleton representing an unset/missing value. All attribute access, item access, and method calls return the Unset @@ -53,4 +53,4 @@ def __iter__(self): """Returns an empty iterator to prevent iteration hangs.""" return iter([]) -Unset = Unset() \ No newline at end of file +Unset = _Unset() \ No newline at end of file diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/expect.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/expect.py index fdd7035f0..03f954ee6 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/expect.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/expect.py @@ -9,7 +9,7 @@ from __future__ import annotations -from typing import Callable, Iterable, Self, TypeVar +from typing import Callable, Iterable, Self, TypeVar, Generic, Sequence from pydantic import BaseModel @@ -27,7 +27,7 @@ ModelT = TypeVar("ModelT", bound=dict | BaseModel) -class Expect: +class ExpectBase(Generic[ModelT]): """ Assertion class that raises on failure. @@ -48,7 +48,7 @@ class Expect: Select(responses).where(type="message").expect.that(text="hello") """ - def __init__(self, items: Iterable[ModelT]) -> None: + def __init__(self, items: Sequence[ModelT]) -> None: """Initialize Expect with a collection of items. :param items: An iterable of dicts or BaseModel instances. @@ -186,4 +186,8 @@ def has_count(self, expected_count: int) -> Self: actual_count = len(self._items) if actual_count != expected_count: raise AssertionError(f"Expected {expected_count} items, found {actual_count}.") - return self \ No newline at end of file + return self + +class Expect(ExpectBase[dict | BaseModel]): + """Concrete Expect class for use with Select and other collections.""" + pass \ No newline at end of file diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/select.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/select.py index f418f34fc..1b73be5b0 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/select.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/select.py @@ -10,15 +10,15 @@ from __future__ import annotations import random -from typing import TypeVar, Iterable, Callable, cast +from typing import TypeVar, Callable, Generic, Sequence, Self from pydantic import BaseModel from .backend import ModelPredicate, DictionaryTransform from .expect import Expect -T = TypeVar("T", bound=BaseModel) +ModelT = TypeVar("ModelT", bound=dict | BaseModel) -class Select: +class SelectBase(Generic[ModelT]): """ Unified selection and assertion for models. @@ -44,24 +44,23 @@ class Select: def __init__( self, - items: Iterable[dict] | Iterable[BaseModel], + items: Sequence[ModelT], ) -> None: - self._items = cast(list[dict] | list[BaseModel], list(items)) + self._items = list(items) def expect(self) -> Expect: """Get an Expect instance for assertions on the current selection.""" return Expect(self._items) - def _child(self, items: Iterable[dict] | Iterable[BaseModel]) -> Select: + def _child(self, items: Sequence[ModelT]) -> Self: """Create a child Select with new items, inheriting selector and quantifier.""" - child = Select(items) - return child + return self.__class__(items) ### ### Selectors ### - def _where(self, _filter: dict | Callable | None = None, _reverse: bool=False, **kwargs) -> Select: + def _where(self, _filter: dict | Callable | None = None, _reverse: bool=False, **kwargs) -> Self: """Filter items by criteria. Chainable.""" mp = ModelPredicate.from_args(_filter, **kwargs) @@ -73,7 +72,7 @@ def _where(self, _filter: dict | Callable | None = None, _reverse: bool=False, * return self._child(filtered_items) - def where(self, _filter: dict | Callable | None = None, **kwargs) -> Select: + def where(self, _filter: dict | Callable | None = None, **kwargs) -> Self: """Filter items matching criteria. Chainable. :param _filter: A dict of field checks or a callable predicate. @@ -82,44 +81,47 @@ def where(self, _filter: dict | Callable | None = None, **kwargs) -> Select: """ return self._where(_filter, **kwargs) - def where_not(self, _filter: dict | Callable | None = None, **kwargs) -> Select: + def where_not(self, _filter: dict | Callable | None = None, **kwargs) -> Self: """Exclude items by criteria. Chainable.""" return self._where(_filter, _reverse=True, **kwargs) - def order_by(self, key: str | Callable | None, reverse: bool = False, **kwargs) -> Select: + def order_by(self, key: str | Callable | None, reverse: bool = False, **kwargs) -> Self: """Order items by a specific key or callable. Chainable.""" dt = DictionaryTransform.from_args(key, **kwargs) return self._child( - sorted( - self._items, - key=dt.eval, - reverse=reverse, + list( + sorted( + self._items, + key=dt.eval, + reverse=reverse, + ) ) ) - def merge(self, other: Select) -> Select: + def merge(self, other: Self) -> Self: """Merge with another Select's items.""" - return self._child(self._items + other._items) + l = self._items + other._items + return self._child(l) def _bool_list(self) -> list[bool]: """Return a list of True values matching the number of selected items.""" return [ True for _ in self._items ] - def first(self, n: int = 1) -> Select: + def first(self, n: int = 1) -> Self: """Select the first n items.""" return self._child(self._items[:n]) - def last(self, n: int = 1) -> Select: + def last(self, n: int = 1) -> Self: """Select the last n items.""" return self._child(self._items[-n:]) - def at(self, n: int) -> Select: + def at(self, n: int) -> Self: """Set selector to 'exactly n'.""" return self._child(self._items[n:n+1]) - def sample(self, n: int) -> Select: + def sample(self, n: int) -> Self: """Randomly sample n items.""" if n < 0: raise ValueError("Sample size n must be non-negative.") @@ -141,4 +143,8 @@ def count(self) -> int: def empty(self) -> bool: """Check if no items are in the current selection.""" - return len(self._items) == 0 \ No newline at end of file + return len(self._items) == 0 + +class Select(SelectBase[dict | BaseModel]): + """Select class for filtering and asserting on model collections.""" + pass \ No newline at end of file diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/core/type_defs.py b/dev/microsoft-agents-testing/microsoft_agents/testing/core/type_defs.py new file mode 100644 index 000000000..5f0393d36 --- /dev/null +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/core/type_defs.py @@ -0,0 +1,26 @@ +from microsoft_agents.activity import Activity + +from .fluent import ExpectBase, SelectBase +from .transport import Exchange + +class ActivityExpect(ExpectBase[Activity]): + """Expect class specifically for asserting on activity collections.""" + pass + +class ExchangeExpect(ExpectBase[Exchange]): + """Expect class specifically for asserting on Exchange model collections.""" + pass + +class ActivitySelect(SelectBase[Activity]): + """Select class specifically for filtering and asserting on activity collections.""" + + def expect(self) -> ActivityExpect: + """Get an ActivityExpect instance for assertions on the current selection.""" + return ActivityExpect(self._items) + +class ExchangeSelect(SelectBase[Exchange]): + """Select class specifically for filtering and asserting on Exchange model collections.""" + + def expect(self) -> ExchangeExpect: + """Get an ExchangeExpect instance for assertions on the current selection.""" + return ExchangeExpect(self._items) diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/utils/__init__.py b/dev/microsoft-agents-testing/microsoft_agents/testing/utils/__init__.py index 2c0d93149..4d11da3df 100644 --- a/dev/microsoft-agents-testing/microsoft_agents/testing/utils/__init__.py +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/utils/__init__.py @@ -1,8 +1,10 @@ from .poll import poll from .send import ex_send, send +from .contains import contains __all__ = [ "poll", "ex_send", - "send" -] \ No newline at end of file + "send", + "contains", +] diff --git a/dev/microsoft-agents-testing/microsoft_agents/testing/utils/contains.py b/dev/microsoft-agents-testing/microsoft_agents/testing/utils/contains.py new file mode 100644 index 000000000..97c5515fc --- /dev/null +++ b/dev/microsoft-agents-testing/microsoft_agents/testing/utils/contains.py @@ -0,0 +1,164 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +"""Contains predicate helper for fluent test assertions.""" + +from __future__ import annotations + +import re +from collections.abc import Callable, Iterable +from typing import Any, cast + +from pydantic import BaseModel + +from microsoft_agents.testing.core.fluent.backend import ModelPredicate + +PredicateFilter = Callable[[Any], bool] | dict[str, Any] +_EMPTY_FILTER = cast(PredicateFilter, object()) + + +def _to_snake_case(value: str) -> str: + value = re.sub("(.)([A-Z][a-z]+)", r"\1_\2", value) + return re.sub("([a-z0-9])([A-Z])", r"\1_\2", value).lower() + + +def _normalize_dict_keys(value: Any) -> Any: + if isinstance(value, dict): + normalized = {} + for key, item in value.items(): + normalized_key = _to_snake_case(key) if isinstance(key, str) else key + normalized[normalized_key] = _normalize_dict_keys(item) + return normalized + if isinstance(value, list): + return [_normalize_dict_keys(item) for item in value] + return value + + +def _build_predicate( + _filter: PredicateFilter = _EMPTY_FILTER, + **kwargs, +) -> Callable[[Any], bool]: + """Build the value predicate used while traversing nested values. + + :param _filter: A callable predicate or dictionary of field checks. + :param kwargs: Additional field checks merged with a dictionary filter. + :raises ValueError: If no filter or keyword checks are provided, or if the + filter is ``None``. + :return: A callable that evaluates one traversed value. + """ + if _filter is None: + raise ValueError("Filter cannot be None.") + if _filter is _EMPTY_FILTER or (isinstance(_filter, dict) and not _filter): + if not kwargs: + raise ValueError("A filter or keyword criteria must be provided.") + _filter = {} + + if callable(_filter) and not kwargs: + return _filter + + model_predicate = ModelPredicate.from_args(_filter, **kwargs) + + def predicate(value: Any) -> bool: + if isinstance(value, list): + return False + + try: + if any(model_predicate.eval(value).result_bools): + return True + if isinstance(value, BaseModel): + field_name_value = value.model_dump( + exclude_unset=True, + exclude_none=True, + by_alias=False, + ) + return any(model_predicate.eval(field_name_value).result_bools) + if isinstance(value, dict): + normalized_value = _normalize_dict_keys(value) + if normalized_value != value: + return any(model_predicate.eval(normalized_value).result_bools) + return False + except (AttributeError, IndexError, KeyError, TypeError, ValueError): + return False + + return predicate + + +class Contains: + """Callable predicate that searches nested model, dict, and iterable values.""" + + def __init__(self, _filter: PredicateFilter = _EMPTY_FILTER, **kwargs) -> None: + """Initialize the predicate with the same criteria accepted by ``contains``. + + ``_filter`` may be a callable applied to each visited value or a + dictionary of field checks evaluated against each visited model or dict. + Keyword checks are merged with dictionary filters, with keyword values + taking precedence for duplicate keys. A filter or keyword criteria must + be provided. + + :param _filter: A callable predicate or dictionary of field checks. + :param kwargs: Additional field checks. + :raises ValueError: If no criteria are provided, the filter is ``None``, + or the filter type is invalid. + """ + self._predicate = _build_predicate(_filter, **kwargs) + self._max_depth = 3 + + def __call__(self, x: Any) -> bool: + return self._contains(x) + + def depth(self, depth: int) -> Contains: + """Return a new predicate with a different maximum search depth. + + Depth starts at the root value as ``0``. Nested model fields, dict + values, and iterable items increment the depth by one. + + :param depth: The maximum depth to search. + :raises ValueError: If ``depth`` is negative. + :return: A new ``Contains`` instance with the requested depth. + """ + if depth < 0: + raise ValueError("Depth must be non-negative.") + contains = Contains(self._predicate) + contains._max_depth = depth + return contains + + def _contains(self, x: Any, depth: int = 0) -> bool: + if depth > self._max_depth: + return False + + try: + if self._predicate(x): + return True + except (AttributeError, IndexError, KeyError, TypeError): + pass + + if depth == self._max_depth: + return False + + if isinstance(x, BaseModel): + it = (value for _, value in x) + elif isinstance(x, dict): + it = x.values() + elif isinstance(x, Iterable) and not isinstance(x, (str, bytes)): + it = x + else: + return False + + return any(self._contains(value, depth + 1) for value in it) + + +def contains( + _filter: PredicateFilter = _EMPTY_FILTER, + **kwargs, +) -> Contains: + """Create a predicate that searches nested model, dict, and iterable values. + + A filter or keyword criteria must be provided. + + :param _filter: A callable predicate or dictionary of field checks. + :param kwargs: Additional field checks. + :raises ValueError: If no criteria are provided, the filter is ``None``, or + the filter type is invalid. + :return: A ``Contains`` predicate. + """ + return Contains(_filter, **kwargs) diff --git a/dev/microsoft-agents-testing/tests/core/fluent/backend/test_model_predicate.py b/dev/microsoft-agents-testing/tests/core/fluent/backend/test_model_predicate.py index 41c847fe8..56cdfe35b 100644 --- a/dev/microsoft-agents-testing/tests/core/fluent/backend/test_model_predicate.py +++ b/dev/microsoft-agents-testing/tests/core/fluent/backend/test_model_predicate.py @@ -189,6 +189,7 @@ class SampleModel(BaseModel): name: str value: int active: bool = True + attachments: list[str] | None = None class NestedModel(BaseModel): @@ -354,6 +355,17 @@ def test_eval_with_root_callable_list(self): result = predicate.eval(models) assert result.result_bools == [True, False] + def test_eval_with_root_callable_returning_empty_list(self): + """eval treats an empty list returned by a root callable as false.""" + predicate = ModelPredicate.from_args(lambda x: x.attachments) + models = [ + SampleModel(name="a", value=1, attachments=None), + SampleModel(name="b", value=2, attachments=[]), + SampleModel(name="c", value=3, attachments=["card"]), + ] + result = predicate.eval(models) + assert result.result_bools == [False, False, True] + def test_eval_with_mixed_value_and_callable(self): """eval works with mixed value and callable predicates.""" predicate = ModelPredicate.from_args( diff --git a/dev/microsoft-agents-testing/tests/core/fluent/backend/test_transform.py b/dev/microsoft-agents-testing/tests/core/fluent/backend/test_transform.py index 02f08e8bc..17b855ae9 100644 --- a/dev/microsoft-agents-testing/tests/core/fluent/backend/test_transform.py +++ b/dev/microsoft-agents-testing/tests/core/fluent/backend/test_transform.py @@ -183,6 +183,14 @@ def test_invoke_with_actual_arg(self): result = transform._invoke(actual, "a", func) assert result == 6 + def test_invoke_with_value_arg(self): + """_invoke passes value as 'value' argument.""" + transform = DictionaryTransform(None) + actual = {"a": 5} + func = lambda value: value + 2 + result = transform._invoke(actual, "a", func) + assert result == 7 + def test_invoke_with_missing_key(self): """_invoke passes Unset for missing keys.""" transform = DictionaryTransform(None) diff --git a/dev/microsoft-agents-testing/tests/core/fluent/test_expect.py b/dev/microsoft-agents-testing/tests/core/fluent/test_expect.py index 392b318e7..6618d9190 100644 --- a/dev/microsoft-agents-testing/tests/core/fluent/test_expect.py +++ b/dev/microsoft-agents-testing/tests/core/fluent/test_expect.py @@ -68,6 +68,16 @@ def test_that_with_callable(self): items = [{"value": 10}, {"value": 20}] Expect(items).that(value=lambda x: x > 5) + def test_that_with_value_callable(self): + """that() passes resolved values to callable predicates named value.""" + items = [{"value": 10}, {"value": 20}] + Expect(items).that(value=lambda value: value > 5) + + def test_that_with_root_value_callable(self): + """that() passes root items to root callable predicates named value.""" + items = [{"value": 10}, {"value": 20}] + Expect(items).that(lambda value: value["value"] > 5) + def test_that_with_callable_fails(self): """that() raises AssertionError when callable fails.""" items = [{"value": 10}, {"value": 2}] diff --git a/dev/microsoft-agents-testing/tests/core/fluent/test_select.py b/dev/microsoft-agents-testing/tests/core/fluent/test_select.py index aab247852..7c5e9cfb9 100644 --- a/dev/microsoft-agents-testing/tests/core/fluent/test_select.py +++ b/dev/microsoft-agents-testing/tests/core/fluent/test_select.py @@ -99,6 +99,30 @@ def test_where_filters_by_callable(self): result = Select(items).where({"value": lambda x: x > 7}).get() assert len(result) == 2 + def test_where_filters_by_value_callable(self): + """where() passes resolved values to callable predicates named value.""" + items = [ + {"name": "a", "value": 10}, + {"name": "b", "value": 5}, + {"name": "c", "value": 20}, + ] + result = Select(items).where({"value": lambda value: value > 7}).get() + assert [item["name"] for item in result] == ["a", "c"] + + def test_where_filters_by_root_callable_truthiness(self): + """where() filters using truthiness from root callable results.""" + class ActivityLike(BaseModel): + name: str + attachments: list[str] | None = None + + items = [ + ActivityLike(name="none", attachments=None), + ActivityLike(name="empty", attachments=[]), + ActivityLike(name="card", attachments=["hero-card"]), + ] + result = Select(items).where(lambda x: x.attachments).get() + assert [item.name for item in result] == ["card"] + def test_where_returns_select(self): """where() returns a Select instance for chaining.""" result = Select([{"a": 1}]).where({"a": 1}) diff --git a/dev/microsoft-agents-testing/tests/core/test_agent_client.py b/dev/microsoft-agents-testing/tests/core/test_agent_client.py index fb0ba51c3..c0295c289 100644 --- a/dev/microsoft-agents-testing/tests/core/test_agent_client.py +++ b/dev/microsoft-agents-testing/tests/core/test_agent_client.py @@ -591,7 +591,7 @@ class TestAgentClientSelectExpect: @pytest.mark.asyncio async def test_select_returns_select_instance(self): """select() returns a Select instance.""" - from microsoft_agents.testing.core.fluent import Select + from microsoft_agents.testing.core import ActivitySelect sender = StubSender() sender.with_responses(Activity(type=ActivityTypes.message, text="Reply")) @@ -600,12 +600,27 @@ async def test_select_returns_select_instance(self): await client.send("Hello") result = client.select() - assert isinstance(result, Select) + assert isinstance(result, ActivitySelect) + + @pytest.mark.asyncio + async def test_expect_returns_activity_expect_instance(self): + """expect() returns an ActivityExpect instance.""" + from microsoft_agents.testing.core import ActivityExpect + + sender = StubSender() + sender.with_responses(Activity(type=ActivityTypes.message, text="Reply")) + + client = AgentClient(sender=sender) + await client.send("Hello") + + result = client.expect() + assert isinstance(result, ActivityExpect) + result.that_for_one(text="Reply") @pytest.mark.asyncio async def test_ex_select_returns_select_with_exchanges(self): """ex_select() returns a Select instance with exchanges.""" - from microsoft_agents.testing.core.fluent import Select + from microsoft_agents.testing.core import ExchangeSelect sender = StubSender() sender.with_responses(Activity(type=ActivityTypes.message, text="Reply")) @@ -614,7 +629,22 @@ async def test_ex_select_returns_select_with_exchanges(self): await client.send("Hello") result = client.ex_select() - assert isinstance(result, Select) + assert isinstance(result, ExchangeSelect) + + @pytest.mark.asyncio + async def test_ex_expect_returns_exchange_expect_instance(self): + """ex_expect() returns an ExchangeExpect instance.""" + from microsoft_agents.testing.core import ExchangeExpect + + sender = StubSender() + sender.with_responses(Activity(type=ActivityTypes.message, text="Reply")) + + client = AgentClient(sender=sender) + await client.send("Hello") + + result = client.ex_expect() + assert isinstance(result, ExchangeExpect) + result.that_for_one(status_code=200) # ============================================================================ diff --git a/dev/microsoft-agents-testing/tests/core/test_type_defs.py b/dev/microsoft-agents-testing/tests/core/test_type_defs.py new file mode 100644 index 000000000..7b16007f9 --- /dev/null +++ b/dev/microsoft-agents-testing/tests/core/test_type_defs.py @@ -0,0 +1,122 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +"""Tests for typed fluent activity and exchange wrappers.""" + +from microsoft_agents.activity import Activity, ActivityTypes +from microsoft_agents.testing.core import ( + ActivityExpect, + ActivitySelect, + Exchange, + ExchangeExpect, + ExchangeSelect, +) + + +def _message(text: str) -> Activity: + return Activity(type=ActivityTypes.message, text=text) + + +def _exchange(text: str, status_code: int = 200) -> Exchange: + return Exchange( + request=_message(text), + status_code=status_code, + responses=[_message(f"Reply: {text}")], + ) + + +class TestActivityExpect: + """Tests for ActivityExpect.""" + + def test_asserts_over_activity_fields(self): + """ActivityExpect can assert directly on Activity fields.""" + expect = ActivityExpect([_message("Hello"), _message("Hi")]) + + result = expect.that_for_all(type=ActivityTypes.message).that_for_any( + text="Hello" + ) + + assert result is expect + + def test_fails_when_activity_criteria_do_not_match(self): + """ActivityExpect raises when activity criteria fail.""" + expect = ActivityExpect([Activity(type=ActivityTypes.typing)]) + + try: + expect.that_for_any(type=ActivityTypes.message) + except AssertionError as error: + assert "Expectation failed" in str(error) + else: + raise AssertionError("Expected ActivityExpect to fail.") + + +class TestActivitySelect: + """Tests for ActivitySelect.""" + + def test_filters_activities_and_preserves_activity_select_type(self): + """ActivitySelect filters Activity items and returns ActivitySelect children.""" + activities = [ + _message("First"), + Activity(type=ActivityTypes.typing), + _message("Second"), + ] + + selected = ActivitySelect(activities).where(type=ActivityTypes.message) + + assert isinstance(selected, ActivitySelect) + assert [activity.text for activity in selected.get()] == ["First", "Second"] + + def test_expect_returns_activity_expect(self): + """ActivitySelect.expect() returns ActivityExpect.""" + selected = ActivitySelect([_message("Hello")]) + + expect = selected.expect() + + assert isinstance(expect, ActivityExpect) + expect.that_for_one(text="Hello") + + +class TestExchangeExpect: + """Tests for ExchangeExpect.""" + + def test_asserts_over_exchange_fields(self): + """ExchangeExpect can assert directly on Exchange fields.""" + expect = ExchangeExpect([_exchange("Hello", 200), _exchange("Fail", 500)]) + + result = expect.that_for_any(status_code=200).that_for_any(status_code=500) + + assert result is expect + + def test_fails_when_exchange_criteria_do_not_match(self): + """ExchangeExpect raises when exchange criteria fail.""" + expect = ExchangeExpect([_exchange("Hello", 200)]) + + try: + expect.that_for_any(status_code=500) + except AssertionError as error: + assert "Expectation failed" in str(error) + else: + raise AssertionError("Expected ExchangeExpect to fail.") + + +class TestExchangeSelect: + """Tests for ExchangeSelect.""" + + def test_filters_exchanges_and_preserves_exchange_select_type(self): + """ExchangeSelect filters Exchange items and returns ExchangeSelect children.""" + exchanges = [_exchange("OK", 200), _exchange("Created", 201)] + + selected = ExchangeSelect(exchanges).where(status_code=200) + + assert isinstance(selected, ExchangeSelect) + assert [exchange.request.text for exchange in selected.get()] == ["OK"] + + def test_expect_returns_exchange_expect(self): + """ExchangeSelect.expect() returns ExchangeExpect.""" + selected = ExchangeSelect([_exchange("OK", 200)]) + + expect = selected.expect() + + assert isinstance(expect, ExchangeExpect) + expect.that_for_one(status_code=200) + diff --git a/dev/microsoft-agents-testing/tests/utils/test_pred.py b/dev/microsoft-agents-testing/tests/utils/test_pred.py new file mode 100644 index 000000000..159807181 --- /dev/null +++ b/dev/microsoft-agents-testing/tests/utils/test_pred.py @@ -0,0 +1,348 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +"""Tests for predicate helper utilities.""" + +from typing import cast + +import pytest +from microsoft_agents.activity import Activity, Attachment as SdkAttachment +from pydantic import BaseModel + +from microsoft_agents.testing.core.fluent import Expect, Select +from microsoft_agents.testing.utils import contains +from microsoft_agents.testing.utils.contains import Contains + + +class Attachment(BaseModel): + """Attachment-like model used by predicate helper tests.""" + + content_type: str + content: dict | None = None + + +class ActivityLike(BaseModel): + """Activity-like model with nested data and attachments.""" + + name: str + channel_data: dict | None = None + attachments: list[Attachment] | None = None + + +def is_hero_card(value): + """Return true when a traversed value is a hero card content type.""" + return value == "application/vnd.microsoft.card.hero" + + +class TestContains: + """Tests for the contains() predicate helper.""" + + def test_matches_direct_model_property(self): + """contains() can match a property on a Pydantic model.""" + activity = ActivityLike(name="message") + + assert contains(lambda value: value == "message")(activity) + + def test_matches_nested_dict_property(self): + """contains() can match a property on a nested dictionary.""" + activity = ActivityLike( + name="message", + channel_data={"tenant": {"id": "tenant-1"}}, + ) + + assert contains(lambda value: value == "tenant-1")(activity) + + def test_matches_nested_list_item_property(self): + """contains() can match a property inside an item in a nested list.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert contains(is_hero_card)(activity) + + def test_matches_nested_list_item_object(self): + """contains() can match a nested list item object.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert contains( + lambda value: isinstance(value, Attachment) + and value.content_type == "application/vnd.microsoft.card.hero" + )(activity) + + def test_matches_nested_item_with_dict_filter(self): + """contains() can match a nested object using dict criteria.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert contains({"content_type": "application/vnd.microsoft.card.hero"})( + activity + ) + + def test_matches_nested_item_with_kwargs(self): + """contains() can match a nested object using keyword criteria.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert contains(content_type="application/vnd.microsoft.card.hero")(activity) + + def test_matches_sdk_model_field_name_when_model_uses_alias(self): + """contains() supports Python field names for SDK models with aliases.""" + activity = Activity( + type="message", + attachments=[ + SdkAttachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert contains(content_type="application/vnd.microsoft.card.hero")(activity) + + def test_expect_property_contains_matches_sdk_alias_dump(self): + """contains() supports SDK field names after Expect dumps model aliases.""" + activity = Activity( + type="message", + attachments=[ + SdkAttachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + Expect([activity]).that_for_any( + attachments=contains(content_type="application/vnd.microsoft.card.hero") + ) + + def test_contains_class_accepts_dict_filter(self): + """Contains can be constructed with the same filter type as contains().""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert Contains({"content_type": "application/vnd.microsoft.card.hero"})( + activity + ) + + def test_contains_class_accepts_kwargs(self): + """Contains can be constructed with keyword criteria.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert Contains(content_type="application/vnd.microsoft.card.hero")(activity) + + def test_empty_dict_filter_can_be_combined_with_kwargs(self): + """An empty dict is valid when keyword criteria provide the filter.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert contains( + {}, content_type="application/vnd.microsoft.card.hero" + )(activity) + + def test_requires_filter_or_keyword_criteria(self): + """contains() rejects the unfiltered case instead of matching everything.""" + with pytest.raises(ValueError, match="criteria"): + contains() + + with pytest.raises(ValueError, match="criteria"): + Contains() + + with pytest.raises(ValueError, match="criteria"): + contains({}) + + with pytest.raises(ValueError, match="criteria"): + Contains({}) + + def test_rejects_explicit_none_filter(self): + """contains() does not accept None as a filter.""" + with pytest.raises(ValueError, match="cannot be None"): + contains(None) + + with pytest.raises(ValueError, match="cannot be None"): + Contains(None) + + with pytest.raises(ValueError, match="cannot be None"): + contains(None, content_type="application/vnd.microsoft.card.hero") + + def test_rejects_invalid_filter_type(self): + """contains() requires a callable or dict filter.""" + with pytest.raises(ValueError, match="dictionary or callable"): + contains("application/vnd.microsoft.card.hero") + + def test_kwargs_override_dict_filter_values(self): + """contains() merges kwargs into dict criteria like Select and Expect.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert contains( + {"content_type": "application/vnd.microsoft.card.thumbnail"}, + content_type="application/vnd.microsoft.card.hero", + )(activity) + + def test_depth_limits_search_depth(self): + """contains().depth() stops descending when depth is too shallow.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + + assert not contains(is_hero_card).depth(1)(activity) + assert not contains(content_type="application/vnd.microsoft.card.hero").depth( + 1 + )(activity) + + def test_depth_returns_new_contains_instance(self): + """contains().depth() returns a new predicate without mutating the original.""" + activity = ActivityLike( + name="message", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + predicate = contains(is_hero_card) + shallow_predicate = predicate.depth(1) + + assert predicate is not shallow_predicate + assert predicate(activity) + assert not shallow_predicate(activity) + + def test_depth_raises_for_negative_value(self): + """contains().depth() raises ValueError for negative depth.""" + with pytest.raises(ValueError, match="non-negative"): + contains(is_hero_card).depth(-1) + + def test_returns_false_for_non_container_source(self): + """contains() returns false instead of raising for scalar source values.""" + assert not contains(lambda value: value == "expected")( + "not a traversable source" + ) + + def test_select_where_filters_by_nested_list_item_property(self): + """contains() works as a root callable for Select.where().""" + activities = [ + ActivityLike( + name="hero", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ), + ActivityLike( + name="thumbnail", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.thumbnail"), + ], + ), + ActivityLike(name="empty", attachments=[]), + ] + + selected = Select(activities).where(contains(is_hero_card)).get() + + assert [cast(ActivityLike, activity).name for activity in selected] == ["hero"] + + def test_select_where_filters_by_nested_list_item_kwargs(self): + """contains() supports Select.where() with keyword criteria.""" + activities = [ + ActivityLike( + name="hero", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ), + ActivityLike( + name="thumbnail", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.thumbnail"), + ], + ), + ActivityLike(name="empty", attachments=[]), + ] + + selected = Select(activities).where( + contains(content_type="application/vnd.microsoft.card.hero") + ).get() + + assert [cast(ActivityLike, activity).name for activity in selected] == ["hero"] + + def test_expect_that_accepts_contains_as_root_callable(self): + """contains() works as a root callable for Expect.that().""" + activities = [ + ActivityLike( + name="hero", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + ] + + Expect(activities).that(contains(is_hero_card)) + + def test_expect_that_accepts_contains_as_property_callable(self): + """contains() works as a property callable for Expect.that().""" + activities = [ + ActivityLike( + name="hero", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + ] + + Expect(activities).that(attachments=contains(is_hero_card)) + + def test_expect_that_accepts_contains_with_kwargs(self): + """contains() supports Expect.that() with keyword criteria.""" + activities = [ + ActivityLike( + name="hero", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.hero"), + ], + ) + ] + + Expect(activities).that( + contains(content_type="application/vnd.microsoft.card.hero") + ) + + def test_expect_that_fails_when_nested_property_not_found(self): + """Expect.that() fails when contains() does not find a matching value.""" + activities = [ + ActivityLike( + name="thumbnail", + attachments=[ + Attachment(content_type="application/vnd.microsoft.card.thumbnail"), + ], + ) + ] + + with pytest.raises(AssertionError): + Expect(activities).that(contains(is_hero_card))