Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 30 additions & 11 deletions dev/microsoft-agents-testing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,14 +96,18 @@ client.expect().that_for_any(text="~Hello") # assert
```

Every method has an `ex_` variant (`ex_send`, `ex_invoke`, etc.) that returns
the raw `Exchange` objects instead of just the response activities.
the raw `Exchange` objects instead of just the response activities. The fluent
shortcuts are typed by collection: `expect()`/`select()` return
`ActivityExpect`/`ActivitySelect`, while `ex_expect()`/`ex_select()` return
`ExchangeExpect`/`ExchangeSelect`.

## Expect & Select

Fluent API for asserting on and filtering response collections. `Expect`
raises `AssertionError` with diagnostic context — it shows what was expected,
what was received, and which items were checked. Prefix a value with `~` for
substring matching, or pass a lambda for custom logic. The variable named `x` has a special meaning and is passed in dynamically during evaluation.
substring matching, or pass a lambda for custom logic. Lambda parameters named
`x`, `actual`, or `value` receive the resolved value during evaluation.

```python
client.expect().that_for_any(text="~hello") # any reply contains "hello"
Expand All @@ -112,6 +116,19 @@ client.expect().that_for_exactly(2, type="message") # exactly 2 messages
client.expect().that_for_any(text=lambda x: len(x) > 10) # lambda predicate
```

Use `contains` for nested model, dict, and iterable values. It requires a
callable, dictionary filter, or keyword criteria; `contains()` and
`contains({})` are invalid because an unfiltered predicate would match
everything.

```python
from microsoft_agents.testing.utils import contains

client.expect().that_for_any(
attachments=contains(content_type="application/vnd.microsoft.card.hero")
)
```

`Select` filters and slices before you assert or extract:

```python
Expand All @@ -125,15 +142,14 @@ Select(client.history()).where(type="message").expect().that(text="~hello")
Every request and response is recorded in a `Transcript`. When a test fails
you can print the conversation to see exactly what happened.

`ConversationTranscriptFormatter` gives a chat-style view;
`ActivityTranscriptFormatter` shows all activities with selectable fields.
Both support `DetailLevel` (`MINIMAL`, `STANDARD`, `DETAILED`, `FULL`) and
`TimeFormat` (`CLOCK`, `RELATIVE`, `ELAPSED`).
`ConversationTranscriptFormatter` gives a chat-style view,
`ActivityTranscriptFormatter` shows a flat JSON activity stream, and
`JsonTranscriptFormatter` shows exchange-grouped JSON.

```python
from microsoft_agents.testing import ConversationTranscriptFormatter, DetailLevel
from microsoft_agents.testing import ConversationTranscriptFormatter

ConversationTranscriptFormatter(detail=DetailLevel.FULL).print(client.transcript)
print(ConversationTranscriptFormatter().format(client.transcript))
```

```
Expand Down Expand Up @@ -181,9 +197,12 @@ class TestEcho: ...

| Document | Contents |
|----------|----------|
| [MOTIVATION.md](MOTIVATION.md) | Before/after code comparison |
| [API.md](API.md) | Public API reference |
| [SAMPLES.md](SAMPLES.md) | Guide to the runnable samples |
| [MOTIVATION.md](docs/MOTIVATION.md) | Before/after code comparison |
| [API.md](docs/API.md) | Public API reference |
| [ASSERTIONS.md](docs/ASSERTIONS.md) | Fluent `Expect`, `Select`, predicates, lambdas, and assertion internals |
| [CLI.md](docs/CLI.md) | `agt` command guide |
| [UTILITIES.md](docs/UTILITIES.md) | `contains`, `poll`, `send`, and `ex_send` helper guide |
| [SAMPLES.md](docs/SAMPLES.md) | Guide to the runnable samples |

## License

Expand Down
31 changes: 25 additions & 6 deletions dev/microsoft-agents-testing/docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ from microsoft_agents.testing import (
AiohttpScenario, ExternalScenario, Scenario, AgentEnvironment,
AgentClient,
ScenarioConfig, ClientConfig, ActivityTemplate,
Expect, Select,
Expect, Select, ActivityExpect, ActivitySelect,
ExchangeExpect, ExchangeSelect,
Transcript, Exchange,
ConversationTranscriptFormatter, ActivityTranscriptFormatter,
JsonTranscriptFormatter,
Expand Down Expand Up @@ -166,15 +167,23 @@ await client.send(activity, wait=0.5)

| Method | Returns | Description |
|--------|---------|-------------|
| `expect(history=False)` | `Expect` | Assert on response activities |
| `select(history=False)` | `Select` | Filter response activities |
| `ex_expect(history=False)` | `Expect` | Assert on exchanges |
| `ex_select(history=False)` | `Select` | Filter exchanges |
| `expect(history=False)` | `ActivityExpect` | Assert on response activities |
| `select(history=False)` | `ActivitySelect` | Filter response activities |
| `ex_expect(history=False)` | `ExchangeExpect` | Assert on exchanges |
| `ex_select(history=False)` | `ExchangeSelect` | Filter exchanges |

```python
from microsoft_agents.testing.utils import contains

# Assert any reply contains "hello" (case-sensitive substring)
client.expect().that_for_any(text="~hello")

# Search nested model, dict, and iterable values.
# A callable, dictionary filter, or keyword criteria is required.
client.expect().that_for_any(
attachments=contains(content_type="application/vnd.microsoft.card.hero")
)

# Filter then assert
client.select().where(type="message").expect().that(text="~world")
```
Expand Down Expand Up @@ -252,8 +261,13 @@ with diagnostic context on failure.

```python
Expect(items: Iterable[dict | BaseModel])
ActivityExpect(items: Iterable[Activity])
ExchangeExpect(items: Iterable[Exchange])
```

`AgentClient.expect()` returns `ActivityExpect`; `AgentClient.ex_expect()`
returns `ExchangeExpect`.

| Method | Passes when… |
|--------|-------------|
| `that(**kwargs)` | **All** items match |
Expand Down Expand Up @@ -304,8 +318,13 @@ Chainable filtering over a collection.

```python
Select(items: Iterable[dict | BaseModel])
ActivitySelect(items: Iterable[Activity])
ExchangeSelect(items: Iterable[Exchange])
```

`AgentClient.select()` returns `ActivitySelect`; `AgentClient.ex_select()`
returns `ExchangeSelect`.

| Method | Description |
|--------|-------------|
| `where(**kwargs)` | Keep items matching criteria |
Expand Down Expand Up @@ -607,7 +626,7 @@ agt scenario load --url http://localhost:3978/api/messages \
| Option | Default | Description |
|--------|---------|-------------|
| `--message / -m` | — | Text message to send |
| `--json-file / -j` | — | JSON activity file to send |
| `--json_file / -j` | — | JSON activity file to send |
| `--num / -n` | *(required)* | Number of concurrent requests |
| `--timeout / -t` | `5000` | Milliseconds per request before it is recorded as a timeout error |

Expand Down
Loading
Loading