test(model-onboarding): four payloads per path, Studio Web coded-copy agents

testcases/model-onboarding/README.md

@@ -1,9 +1,23 @@
 # model-onboarding testcase
 
 Exercises **one runtime-specified model** across the distinct `get_chat_model`
-code paths it is expected to support, plus optional file attachments. Rolls
-every `path × file` cell up into a single `success` boolean and asserts on both
-the output and the emitted traces.
+code paths it is expected to support. For **every path** it runs four
+capability payloads:
+
+- **`simple`** — a plain text call; asserts a non-empty completion.
+- **`tools`** — a full tool-calling **round trip**: bind a tool, let the model
+  request it, execute the tool, feed the result back, and assert the final
+  answer uses it.
+- **`files/<name>`** — one cell per selected file attachment, run through the
+  coded file-processing agent (Studio Web coded copy; real platform
+  attachment + Analyze Files internal tool).
+- **`is_tools`** — call-only Activity-tool check: bind the coded copy's real
+  Integration Service tools (Slack Send Message to Channel, Outlook 365 Send
+  Email) and assert the model produces a well-formed tool call. Nothing is
+  executed or sent. See [src/agents/README.md](src/agents/README.md).
+
+Every `path × payload` cell rolls up into a single `success` boolean, asserted
+alongside the emitted traces.
 
 Unlike `multimodal-invoke` (which hardcodes its model matrix), the model here is
 **input**. To onboard a model, edit `input.json` — no code change.
@@ -31,9 +45,11 @@ Unlike `multimodal-invoke` (which hardcodes its model matrix), the model here is
   (and misleading) failure.
 - **`agenthub_config`** — AgentHub config header value; must exist in the tenant
   behind your `BASE_URL`. Defaults to `agentsplayground`.
-- **`files`** — file attachments to test. Valid keys: `image`, `pdf`. Use `[]`
-  for a **text-only** model — an empty list runs a plain reachability check via
-  `ainvoke` instead of a multimodal call.
+- **`files`** — file attachments for the `files/*` payload. Valid keys:
+  `image`, `pdf`. Use `[]` for a **text-only** model — the `simple` and `tools`
+  payloads still run; only the per-file cells are skipped.
+- **`prompt`** — used by the `simple` and `files/*` payloads. The `tools`
+  payload uses its own fixed weather prompt.
 
 ## Prerequisites (external to the repo)
 

testcases/model-onboarding/input.json

@@ -4,6 +4,7 @@
         "model_name": "gpt-5.2-2025-12-11",
         "paths": ["azure_responses", "azure_chat_completions"],
         "agenthub_config": "agentsplayground",
-        "files": ["image", "pdf"]
+        "files": ["image", "pdf"],
+        "is_tools": {}
     }
 }

testcases/model-onboarding/src/agents/README.md

@@ -0,0 +1,49 @@
+# Coded agents for model-onboarding
+
+Each subfolder holds a **coded agent generated by Studio Web's "Clone as
+Coded Agent"** from its low-code twin, adapted for the onboarding testcase.
+Each agent's `agent.py` is the generated `main.py` with exactly one
+structural change — the module-level hardcoded
+`llm = get_chat_model('gpt-5.4', ...)` becomes `build_graph(llm, ...)` /
+`build_tools(...)` so the testcase injects the model built from `model_spec`
+(model + API flavor configurable). `utils.py` is the generated interpolation
+helper, shared verbatim.
+
+Provenance (alpha tenant, org `llm_gateway_automated_testing`):
+
+| agent | low-code twin | coded copy project |
+|---|---|---|
+| `file_processing` | FileProcessingAgent (solution `1823d2d6-…`) | `e7b8598c-…` |
+| `is_tools` | IsToolsAgent (solution `925fc5db-…`) | `0959f23b-…` |
+
+When a low-code twin changes, re-clone in Studio Web, download the generated
+`main.py` with `uip agent file list/get <projectId>`, and re-apply the
+`build_graph`/`build_tools` adaptation here.
+
+## `file_processing`
+
+Reads a single PDF or image and answers a task about it, via the production
+ReAct stack: `create_agent` + the *Analyze Files* internal tool
+(`create_internal_tool`). The testcase adapter uploads the test file as a
+real platform attachment (`sdk.attachments.upload_async`) and invokes the
+generated graph. Cells: `files/<name>`, one per selected file.
+
+## `is_tools`
+
+Tests that the model under test **produces a well-formed tool call** against
+the real Integration Service Activity tools — Slack *Send Message to
+Channel* and Outlook 365 *Send Email* — built via `create_integration_tool`
++ `AgentIntegrationToolResourceConfig` exactly as generated. Cell:
+`is_tools` (one per path), reporting the called tools, e.g.
+`✓ (Send Message to Channel, Send Email)`.
+
+**Call-only:** the tools are bound to the model but never executed — no IS
+connections are needed and nothing is actually sent, so the cell runs on
+every onboarding pass. Asserts ≥1 tool call with the right name and required
+args (`channel` + `messageToSend`; `message.toRecipients`).
+
+`model_spec.is_tools.prompt` overrides the communication request. For a full
+executed run (real Slack/email side effects), `build_graph(llm, connections)`
+remains — wire real connection ids and invoke it deliberately; the testcase
+never does. Note `create_integration_tool` requires UiPath auth at
+tool-creation time — locally unauthenticated runs fail legibly in the cell.

testcases/model-onboarding/src/agents/__init__.py

@@ -0,0 +1,30 @@
+"""Coded agents exercised by the model-onboarding testcase.
+
+Each agent is the coded (LangGraph) equivalent of a low-code UiPath agent,
+re-authored here so the testcase can run it against a configurable model +
+API flavor (sourced from ``model_spec`` in ``input.json``).
+
+Add a new coded agent by creating ``agents/<name>/agent.py`` exposing an async
+``run(model, prompt, files)`` coroutine and a ``NAME`` constant, then register
+it in ``AGENT_REGISTRY`` below. Every registered agent becomes a per-path cell
+in the onboarding grid.
+"""
+
+from typing import Awaitable, Callable
+
+from langchain_core.language_models import BaseChatModel
+
+from uipath_langchain.agent.multimodal.types import FileInfo
+
+from .file_processing.agent import NAME as FILE_PROCESSING_NAME
+from .file_processing.agent import run as run_file_processing
+
+# An agent entry: given a built model, the prompt, and the selected files,
+# returns a result string ("✓" or "✗ ...").
+AgentRunner = Callable[[BaseChatModel, str, list[FileInfo]], Awaitable[str]]
+
+AGENT_REGISTRY: dict[str, AgentRunner] = {
+    FILE_PROCESSING_NAME: run_file_processing,
+}
+
+__all__ = ["AGENT_REGISTRY", "AgentRunner"]

testcases/model-onboarding/src/agents/file_processing/__init__.py

@@ -0,0 +1,2 @@
+"""File-processing coded agent (coded equivalent of the low-code
+FileProcessingAgent built in Studio Web / Agent Builder)."""

testcases/model-onboarding/src/agents/file_processing/agent.py

@@ -0,0 +1,161 @@
+"""File-processing coded agent — Studio Web "Coded copy of FileProcessingAgent".
+
+Source of truth: the coded agent generated by Studio Web's "Clone as Coded
+Agent" from the low-code FileProcessingAgent (alpha, solution 1823d2d6-…,
+project e7b8598c-… — download via ``uip agent file get``); this module is
+that file with exactly one structural change: the module-level hardcoded
+``llm`` is replaced by ``build_graph(llm)`` so the onboarding testcase can
+inject the model built from ``model_spec`` (model + API flavor configurable).
+
+The ``run`` adapter at the bottom uploads the test file as a real platform
+attachment and invokes the generated graph — exercising the production ReAct
+agent (``create_agent``) and the real *Analyze Files* internal tool.
+"""
+
+from datetime import (
+    datetime,
+    timezone
+)
+from langchain_core.messages import (
+    HumanMessage,
+    SystemMessage
+)
+from pydantic import (
+    BaseModel,
+    Field
+)
+from pydantic import (
+    ConfigDict
+)
+from typing import (
+    Sequence
+)
+from uipath.agent.models.agent import (
+    AgentInternalToolResourceConfig
+)
+from uipath.agent.react import (
+    AGENT_SYSTEM_PROMPT_TEMPLATE
+)
+from uipath.platform.attachments import (
+    Attachment
+)
+from uipath_langchain.agent.react import (
+    create_agent
+)
+from uipath_langchain.agent.tools.internal_tools import (
+    create_internal_tool
+)
+
+from agents.utils import (
+    interpolate_legacy_message
+)
+
+# Required alias for job attachment detection at runtime
+__Job_attachment = Attachment
+
+NAME = "file_processing"
+
+
+# Input/Output Models
+class AgentInput(BaseModel):
+    model_config = ConfigDict(extra='allow')
+    prompt: str = Field(..., description="The task or question to answer about the attached file.")
+    fileIn: Attachment
+
+
+class AgentOutput(BaseModel):
+    model_config = ConfigDict(extra='allow')
+    content: str | None = Field(None, description="The agent's analysis of the attached file.")
+
+
+# Agent Messages Function
+def create_messages(state: AgentInput) -> Sequence[SystemMessage | HumanMessage]:
+    # Extract values safely from state
+    fileIn = getattr(state, 'fileIn', '')
+    prompt = getattr(state, 'prompt', '')
+
+    # Apply system prompt template
+    current_date = datetime.now(timezone.utc).strftime('%Y-%m-%d')
+    system_prompt_content = """You are a file-processing assistant. You are given a single file (PDF or image) and a task. Use the Analyze Files tool to read the file's contents, then answer the task concisely based only on what the file contains. If the file cannot be read, say so plainly."""
+    system_prompt_content = interpolate_legacy_message(system_prompt_content, state.model_dump())
+    enhanced_system_prompt = (
+        AGENT_SYSTEM_PROMPT_TEMPLATE
+        .replace('{{systemPrompt}}', system_prompt_content)
+        .replace('{{currentDate}}', current_date)
+        .replace('{{agentName}}', 'Mr Assistant')
+    )
+
+    return [
+        SystemMessage(content=enhanced_system_prompt),
+        HumanMessage(content=interpolate_legacy_message("""{{prompt}}
+
+{{fileIn}}""", state.model_dump())),
+    ]
+
+
+def build_graph(llm):
+    """Build the generated agent graph for an injected model.
+
+    Mirrors the Studio Web coded copy's module body; ``llm`` replaces the
+    hardcoded ``get_chat_model(model='gpt-5.4', ...)`` so the testcase can
+    exercise any model/path from ``model_spec``.
+    """
+    # Context Grounding Tool: analyze_files
+    analyze_files_config = AgentInternalToolResourceConfig(
+        name='Analyze Files',
+        description='Analyze one or more files with an LLM to extract, synthesize, or answer queries about their content.',
+        type='Internal',
+        input_schema={'type': 'object', 'properties': {'attachments': {'type': 'array', 'items': {'$ref': '#/definitions/job-attachment'}, 'description': 'Array of files, documents, images, or other attachments to process'}, 'analysisTask': {'type': 'string', 'description': 'The task, question, or instruction for processing the files'}}, 'required': ['attachments', 'analysisTask'], 'definitions': {'job-attachment': {'type': 'object', 'properties': {'ID': {'type': 'string', 'description': 'Orchestrator attachment key'}, 'FullName': {'type': 'string', 'description': 'File name'}, 'MimeType': {'type': 'string', 'description': 'MIME type, e.g. "application/pdf", "image/png"'}, 'Metadata': {'type': 'object', 'description': 'Dictionary<string, string> of metadata', 'additionalProperties': {'type': 'string'}}}, 'required': ['ID'], 'x-uipath-resource-kind': 'JobAttachment'}}},
+        output_schema={'type': 'object', 'properties': {'analysis': {'type': 'string', 'description': 'Analysis result of the attachments'}}, 'required': ['analysis']},
+        properties={'requireConversationalConfirmation': False, 'toolType': 'analyze-attachments'},
+        arguments={},
+        argument_properties={}
+    )
+    analyze_files_tool = create_internal_tool(analyze_files_config, llm)
+
+    # Collect all tools
+    tools = []
+    tools.append(analyze_files_tool)
+
+    # Create agent graph
+    return create_agent(model=llm, messages=create_messages, tools=tools, input_schema=AgentInput, output_schema=AgentOutput)
+
+
+async def _upload_attachment(file_info) -> Attachment:
+    """Fetch the test file and register it as a platform attachment."""
+    import httpx
+    from uipath.platform import UiPath
+    from uipath_langchain._utils import get_httpx_client_kwargs
+
+    async with httpx.AsyncClient(**get_httpx_client_kwargs()) as client:
+        response = await client.get(file_info.url)
+        response.raise_for_status()
+        content = response.content
+
+    sdk = UiPath()
+    attachment_id = await sdk.attachments.upload_async(
+        name=file_info.name, content=content
+    )
+    return Attachment(
+        id=attachment_id,
+        full_name=file_info.name,
+        mime_type=file_info.mime_type,
+    )
+
+
+async def run(model, prompt: str, files) -> str:
+    """Testcase adapter: run the generated graph over one attached file.
+
+    Returns ``"✓"`` or ``"✗ ..."`` per the onboarding cell contract.
+    """
+    if not files:
+        return "✗ file_processing requires a file"
+    attachment = await _upload_attachment(files[0])
+    graph = build_graph(model)
+    if hasattr(graph, "compile"):  # create_agent returns an uncompiled StateGraph
+        graph = graph.compile()
+    result = await graph.ainvoke(AgentInput(prompt=prompt, fileIn=attachment))
+    content = (result or {}).get("content") if isinstance(result, dict) else getattr(result, "content", None)
+    if content and str(content).strip():
+        return "✓"
+    return "✗ empty response"

testcases/model-onboarding/src/agents/is_tools/__init__.py

@@ -0,0 +1,2 @@
+"""IS-tools coded agent: exercises Integration Service activity tools
+across the BYO LLM vendor connector flavors."""