test(model-onboarding): simple, tool-call, and file payloads per path#1007
Closed
denispetre wants to merge 6 commits into
Closed
test(model-onboarding): simple, tool-call, and file payloads per path#1007denispetre wants to merge 6 commits into
denispetre wants to merge 6 commits into
Conversation
…path Previously each path only ran file-processing cells. Now every path runs three capability payloads: - simple: plain ainvoke, assert non-empty completion - tools: full tool-calling round trip — bind get_weather, force the call, execute the tool, feed the ToolMessage back, and assert the final answer uses the result - files/<name>: one cell per selected file attachment (unchanged behavior) files can now be empty without losing coverage: simple + tools still run. Round-trip logic verified with fake models (happy path, no-call, and ignored-result all assert correctly). README updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vertex path relied on autodetection to reach the Google class, so the GENERATE_CONTENT api_flavor was only covered by default, not by intent. Pin vendor_type=VERTEXAI + api_flavor=GENERATE_CONTENT so the flavor is exercised deterministically. Verified the path forwards both args to the factory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ucture Introduces testcases/model-onboarding/src/agents/ as a container for coded (LangGraph) agents run by the onboarding matrix, so more agents can be added as sibling subfolders and registered in AGENT_REGISTRY. First agent: file_processing — reads a PDF/image and answers a task. It is the coded equivalent of a low-code UiPath agent (FileProcessingAgent) built and validated in Studio Web/Agent Builder; that low-code source of truth is kept alongside under file_processing/lowcode/ (agent.json + Analyze Files tool). No automated low-code->coded eject exists, so agent.py is a faithful re-implementation with the system prompt kept in sync. The file payload in main.py now routes through the coded agent. The agent receives the model built from model_spec, so model + API flavor stay configurable and the run.sh / input.json / model_spec Action contract is unchanged. Verified: graph compiles, input.json validates, and `uipath init` loads the graph (imports the agents subpackage) successfully. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…avor
Adds agents/is_tools: for each configured flavor the model under test binds
an Integration Service activity tool, is forced to call it, the activity
executes through a real IS connection, and the result feeds back into a
final answer (full round trip). Cells appear as is_tools/<flavor>.
Six flavors, built from live `uip is resources describe` output (paths,
query/path/multipart routing, required fields all verified): azure_openai,
openai, openai_v1, bedrock_converse, vertex, anthropic. Bedrock's IS
connector exposes only converse. Multipart activities (openai, bedrock,
vertex) use json_body_section="body"; azure carries modelId in query,
vertex modelName in path.
Config via model_spec.is_tools in input.json: flavors, per-flavor connection
id overrides, per-flavor vendor model overrides. Defaults target the
llm_gateway_automated_testing alpha tenant (connections verified Enabled via
ping). Live smoke: anthropic flavor invoked end to end through the real
gateway ("OK", end_turn); openai_v1 default model needs a Foundry deployment
name — documented as configurable.
Verified: graph compiles, input.json validates, all 6 builders produce
coherent shapes, mocked round trips pass for all flavors, failure modes
(no tool call / unknown flavor / IS exception) are legible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Added a fourth payload: Per flavor, the model under test binds an Integration Service activity tool, is forced to call it, the activity executes through a real IS connection, and the result feeds a final answer (full round trip). Cells:
All shapes built from live Cost note: cells run per path × per flavor (each ≈2 model calls + 1 IS call) — trim |
…ents
Both test agents are now the coded agents generated by Studio Web's "Clone
as Coded Agent" from their low-code twins, downloaded via `uip agent file
get` and adapted with exactly one structural change: the module-level
hardcoded llm becomes build_graph(llm, ...) so the testcase injects the
model built from model_spec (model + API flavor stay configurable). The
pristine generated sources are kept under each agent's coded-copy/ dir and
utils.py is the generated interpolation helper, verbatim.
file_processing: production ReAct stack (create_agent + Analyze Files
internal tool); the adapter uploads the test file as a real platform
attachment and invokes the generated graph.
is_tools: replaces the hand-rolled LLM-gateway flavor registry with the
generated Slack Send Message to Channel + Outlook 365 Send Email Activity
tools (create_integration_tool + AgentIntegrationToolResourceConfig).
IsToolsSpec becomes {connections: {slack, outlook}, prompt}; the cell is
skipped when no connections are configured because a run performs real
communication actions. Connection ids are injected into the generated tool
configs (typed attribute path); create_agent returns an uncompiled
StateGraph so adapters compile before invoking.
Verified: graph compiles, input.json validates, both build_graphs construct
(is_tools under stub auth env; create_integration_tool builds UiPath() at
tool-creation time), guards are legible, and `uipath init` loads the graph.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per review: the onboarding test only needs the model to produce the tool call, not to execute it. run(model, prompt) now binds the generated Slack + Outlook Activity tools, sends the exact production messages (create_messages), and asserts >=1 well-formed tool call back: known bound tool name (the factory underscores display names -> Send_Message_to_Channel, Send_Email) and required args present (channel+messageToSend / message.toRecipients). Tools are never executed, so no IS connections are needed, nothing is sent, and the cell now runs on every onboarding pass. IsToolsSpec drops connections (prompt only); build_graph(llm, connections) remains for deliberate full executed runs. Verified all outcomes with fake models: both-tools ✓ with names, missing-args ✗, unknown-tool ✗, no-call ✗; graph compiles and input.json validates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Contributor
Author
|
Superseded by the clean-diff PR: same final state (four payloads incl. call-only is_tools, Studio Web coded-copy agents) without the intermediate churn. See the new PR referenced above. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The model-onboarding testcase now exercises three capability payloads for every code path, not just file processing:
simpleainvoke; asserts a non-empty completiontoolsget_weather, force the call, execute the tool, feed theToolMessageback, assert the final answer uses the resultfiles/<name>So
azure_responses(for example) now producessimple,tools,files/image,files/pdfcells, all rolled into the singlesuccessboolean.Notable
filesmay now be empty without losing coverage —simple+toolsstill run.toolspayload is a full round trip (executes the tool and checks the final answer used the result), per request — stricter than a request-only check.input.jsonschema is unchanged; existing specs just gain the two new cell types.Verification
Graph compiles and
input.jsonvalidates against the updated schema. Round-trip logic verified with fake models covering happy path, no-tool-call, and ignored-result cases (all assert correctly). Live LLM runs happen in CI / on-demand dispatch (needs tenant auth).