feat(mcp): leadbay_getting_started — guided first-run walkthrough - #175
Draft
ArtyETH06 wants to merge 18 commits into
Draft
feat(mcp): leadbay_getting_started — guided first-run walkthrough#175ArtyETH06 wants to merge 18 commits into
ArtyETH06 wants to merge 18 commits into
Conversation
A brand-new user learns Leadbay by DOING, not by reading. Three gates, each
presenting exactly ONE option, each click running a real call:
1. "Pull today's leads" -> leadbay_pull_leads (no args)
2. "Enrich top leads" -> leadbay_enrich_titles (no titles = free)
3. "Run this every morning" -> no Leadbay tool; the host's scheduler
Ships as both a prompt (auto-emits a Claude Code skill) and a read-only
composite tool returning the step manifest, so natural phrasing ("I'm new")
reaches the tour on hosts that don't surface MCP prompts.
Two hard rules encoded in the manifest, prompt, and tests:
- Gate 2 spends NOTHING. Omitting `titles` returns mode:"discover", the free
preview. Passing titles/confirm/email/phone launches a PAID reveal, and
spending a 90-second-old account's quota to demo a feature is the worst
outcome available here.
- Gate 3 has calls:null because Leadbay exposes no scheduling API. The gate's
literal recurring wording ("every morning") is what lets the host's existing
SCHEDULED_TASK_PARAGRAPH flow fire on its own terms — server.ts is untouched,
so the daily-check-in evals are unaffected.
A warming lens (computing_wishlist/computing_scores) is explained and re-pulled
rather than reported as "no leads" — pull_leads already returns a server-built
two-option warm-up payload for that state, rendered verbatim.
The only edits to existing test files are two one-line registration appends
(the prompts/list toEqual array and TOOLS_WITH_ROUTING), matching the precedent
in 6ddbcf8 and the instruction in CLAUDE.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Leadbay has NO CRM integration — no push, export, sync, or third-party OAuth exists, and report_friction's canonical missing_capability example is literally "Why can't I export to HubSpot?". But the AGENT usually does have one: users run a HubSpot/Salesforce/Pipedrive/Attio connector in the same host. So the new gate delegates rather than pretending. The walkthrough is now four gates: 1. "Pull today's leads" -> leadbay_pull_leads 2. "Enrich top leads" -> leadbay_enrich_titles (no titles = free) 3. "Add these to my CRM" -> no Leadbay tool; the agent's OWN connector 4. "Run this every morning" -> no Leadbay tool; the host's scheduler Gate 3 reuses the detection mechanism the connected-outreach-tool table in leadbay_prospecting_overview already established: read the host's installed-connector / installed-MCP inventory, else infer from conversation, else ask. It names the CAPABILITY, never a third-party tool name — a backticked `hubspot_create_company` would be the first such reference in the repo and would rot silently when the connector renames its tools. Four honesty guards, each with a test: - Never claim a CRM record was created unless the connector confirmed it. - Never write an email or phone into the CRM. Gate 2 is the FREE title preview, so no contact detail was ever revealed; writing one is fabricated PII. - With no connector: say so in one honest line, name the user's CRM, and offer leadbay_report_friction with category:'missing_capability' — the real route. - Don't hunt for a leadbay_* CRM tool; none exists. Modelled on the gate-4 scheduler delegation shipped in the previous commit — same shape, same "only the host can create one" discipline. Co-Authored-By: Claude <noreply@anthropic.com>
The tutorial now starts by proving the connection works, instead of opening with prose and jumping straight to leads. Five gates: 1. "Check my account" -> leadbay_account_status 2. "Pull today's leads" -> leadbay_pull_leads 3. "Enrich top leads" -> leadbay_enrich_titles (no titles = free) 4. "Add these to my CRM" -> no Leadbay tool; the agent's OWN connector 5. "Run this every morning" -> no Leadbay tool; the host's scheduler Gate 1 sits on top of two PINNED regressions, so its branches encode both and the tests assert them: - WORKFLOWS #30 — a brand-new org has no billing plan, so quota_status 401s. leadbay_account_status swallows that into `quota_error`. The gate must say NOTHING about quota and must NEVER suggest logging in again: the token is valid, the same response just read the account. This is the product#3761 401-hallucination bug, and a first-run tutorial is the worst possible place to reintroduce it. - WORKFLOWS #31 — account-status.ts:166 withholds the lens unless the trigger text mentions it. "Walk me through Leadbay" doesn't, so there is nothing to report; the gate must not volunteer it nor call another tool to find it. The lens appears naturally at gate 2. The underdeliver eval fixtures quota_status as a 401 on purpose, so the run proves the silence rather than assuming it. Co-Authored-By: Claude <noreply@anthropic.com>
…it fires
Live test showed the agent running the tools straight through without ever
firing a choice widget — the user watched a demo instead of taking a tutorial.
Root cause: the gates only DESCRIBED the widget in prose (gate_label /
gate_description as loose strings), leaving the agent to assemble the widget
call itself. Meanwhile leadbay_pull_leads ships a ready-made `next_steps`
{question, options[]} object, and the shared routing snippet says a next_steps
payload is the source of truth and must be mapped VERBATIM. The tour was on
the weak path.
Each step now carries that same payload shape:
next_steps: { question, options: [{label, description, kind}] } // exactly 1
explain: "what to TELL the user before firing the widget"
So the agent renders data instead of interpreting prose, and each gate is two
beats — explain, then ask, then WAIT for the click. The explain beat is the
tutorial half: gate 2 teaches what a lens is, gate 3 teaches what enrichment
is and that the preview costs nothing.
The prompt gains an explicit "NEVER run a step's tool without firing its
widget first and receiving the click" rule (with the one sane exception: the
user's own message already said to run everything), plus three failure modes
for exactly what went wrong.
The new prompt↔manifest widget-text audit caught a real bug on its first run:
markdown line-wrapping had split the widget strings across newlines, so the
shipped prompt and the manifest disagreed on the question text.
Co-Authored-By: Claude <noreply@anthropic.com>
…e clicks The buttons disappear when the tour ends. A user who was only ever shown buttons learned to click a tutorial and nothing about using Leadbay tomorrow. The manifest now carries `keep_going`: a what-you-want -> what-you-say cheat-sheet the agent renders as a small table at the finish. | Today's fresh leads | "Show me today's leads" | | Who to follow up with | "What should I follow up on" | | The story on one company | "Research <Company>" | | An email to a contact | "Draft outreach for <Contact>" | | Change who you target | "Narrow the audience to <sector>"| | Switch target audience | "Show me my lenses" | Every phrase is lifted VERBATIM from that tool's own routing.triggers, and a test enforces it: each `say` is checked against the trigger blocks in the generated tool descriptions, so a phrase that doesn't actually route fails the build. Teaching a phrase that silently does nothing is worse than teaching none, and "sounds about right" is exactly how that happens. Verified the guard bites: swapping in a plausible "Fetch my newest prospects please" fails with the offending phrase named. Skipped when the user abandons the tour early — they're already off doing what they wanted, and a tutorial summary would just interrupt. Co-Authored-By: Claude <noreply@anthropic.com>
…yllabus
The tour front-loaded text: STEP 0 asked for 2-3 sentences PLUS a preview of
all five upcoming clicks, and then gate 1 added its own explain beat on top.
The first button ended up buried under paragraphs nobody reads.
The opening is now, in one message:
1. one sentence on what Leadbay is
2. one short line naming the step — "Let's start with your account status."
3. gate 1's widget, fired immediately
and then it stops. No five-step preview, no lens explanation yet — each gate
already explains itself when its own turn comes, so saying it up front is
duplication that costs the user the thing they actually wanted: seeing it work.
Gate 1's widget text is shortened to match ("Check my Leadbay account status.")
and the gate no longer stacks a second explanation on top of the opening lines.
Two new failure modes cover the regression: opening with a wall of text, and
ending the first message without firing the widget at all.
The prompt<->manifest drift audit paid for itself again — markdown line-wrap
had split the new widget description across a newline, so prompt and manifest
disagreed on the exact string.
Co-Authored-By: Claude <noreply@anthropic.com>
…mprovised tour Live failure in Claude Desktop chat: "Walk me through Leadbay please" produced the agent's OWN product overview — a mental-model essay plus a lens table and a four-option "Where do you want to start?" widget. Nothing from leadbay_getting_started ran. The prompt was not missing. It was listed in PROMPT_CATALOG_BULLETS at ~char 10,786 of a 24.5k-char instruction block, one bullet among fifteen. A listing tells the agent the prompt EXISTS; nothing told it to CHOOSE it, so it wrote something itself. Adds a FIRST RUN routing line to the server instructions, placed before the generic start-here flow (now ~char 6,887, ahead of the catalog). It names the phrasings verbatim, says invoke `leadbay_getting_started` via `prompts/get`, and explicitly forbids improvising an overview — with the reason attached, so the prohibition doesn't read as arbitrary and get ignored the moment the agent thinks its own summary would be nicer. Deliberately NOT fixed by narrowing leadbay_prospecting_overview's very broad short_description: that regenerates its shipped snapshot and widens the blast radius. The routing line is the smaller, more direct change. New audit test pins the phrasings, the invoke verb, the anti-improvisation rule, and — load-bearing — that the line lands BEFORE the catalog listing. Co-Authored-By: Claude <noreply@anthropic.com>
Live in Claude Desktop, gate 1 came out as prose — "Let's start with your
account status — say the word and I'll check it" with a bold "-> Check my
account" line. No widget, no button.
Root cause was a contradiction the walkthrough shipped with: the prompt said
"exactly ONE option", while the shared next-steps routing snippet it INCLUDES
says "2-4 mutually-exclusive options". The model followed the shared rule and
fell back to prose. pull-leads.ts:123-127 already documented this exact failure
("a single option would make the model emit an invalid widget call, or silently
drop to prose"); the walkthrough was written against the other belief.
Every gate now carries two options: the forward action, then `I'm done for
now`. That satisfies the host contract so the widget renders, while keeping
exactly one way FORWARD — the tutorial still never asks a first-run user to
choose between paths. The exit must end the tour, never route elsewhere, or it
reintroduces the choice the rule exists to remove.
The rule is renamed ONE-FORWARD-OPTION and states the reason inline, so the
next person doesn't "fix" it back to one option.
Opening lines rewritten to lead with what the user gets ("a fresh batch of
companies worth selling to every day") and to promise something concrete
("five quick steps, and you'll have real leads by the end").
Tests now assert the two-option shape, exactly one exit, exit-ends-the-tour,
and forward-option-first.
Co-Authored-By: Claude <noreply@anthropic.com>
The user clicks a button labelled "check my account status" and got back "you're connected as X, an admin on Y". That under-delivers on the button they pressed. It was also wrong against the tool's own contract: leadbay_account_status's rendering_hint says "Report user + org, AND quota whenever readable — include quota even on a plain 'what account am I on?'", silent ONLY when quota is null, quota_error is set, or the org is unlimited. My gate told the agent to report user/org/plan and never mentioned quota at all, so it under-reported on every account where quota reads fine. Gate 1 now includes the canonical `rendering/quota-windows` snippet — the same one leadbay_account_status uses — so the tour renders what the web app renders: Daily / Weekly / Monthly, each with a ▰▱ gauge, % used, $ spent against cap, resets countdown, and the per-resource breakdown. Never raw "credits". The silence gate is preserved and widened to all three cases it actually covers (null / quota_error / unlimited_credits), not just the 401 — so WORKFLOWS #30 still holds and a plan-less org sees nothing about quota rather than an error. Two failure modes added: answering with a bare greeting when quota IS readable, and rendering credits or raw resource_type strings instead of the gauges. Co-Authored-By: Claude <noreply@anthropic.com>
The tour explained mechanics but never made the case. "Leadbay keeps a lens —
a description of who you sell to" tells a new user what the feature is; it does
not tell them why they should care, which is the actual question a first-run
user is asking.
Opening is now a short paragraph instead of two lines: what Leadbay is, how the
lens works, and what the five steps will leave them holding. Still bounded — it
must NOT walk the five steps one at a time, which is the wall-of-text version
that buried the first button two commits ago.
Every gate gains a concrete payoff, in working-life terms rather than feature
terms:
leads -> replaces the hour spent digging through directories; and the lens
sharpens from what you like, contact or skip
enrich -> ask for the operations director by title instead of pitching
whoever answers the switchboard
CRM -> no copy-pasting between tabs; a lead found here doesn't quietly
die in a chat window
daily -> prospecting is the first thing that slips on a busy week; this
removes the part that depends on remembering
Tests assert every gate carries a WHY IT'S USEFUL beat and that the opening
still teaches the lens without regressing to a syllabus, so a later trim can't
quietly strip the reasoning back out.
Co-Authored-By: Claude <noreply@anthropic.com>
Gate 1 rendered the quota gauges but left a first-run user staring at percentages and dollar figures they've never seen, with no way to tell whether they're good, bad, or something to worry about. It now follows the render with one or two plain lines: what the numbers count (the AI work Leadbay does for them — researching companies, qualifying leads — not something they spend by clicking around) and why it matters to them (it paces how many fresh leads arrive; heavy use now means a bigger batch queued next time, and it's where a smaller-than-expected batch would show its reason). Bounded deliberately: a sentence or two, no walking through every resource row, and no pricing pitch — the tool's own guidance already covers wait-vs-top-up when a window is actually exhausted. The explanation is skipped whenever the silence gate fires (quota null, quota_error, or unlimited_credits). Describing a gauge that isn't on screen is worse than saying nothing, and a test pins that. Co-Authored-By: Claude <noreply@anthropic.com>
The tour stopped at the free title preview and never revealed a contact, so a
first-run user finished the walkthrough without ever seeing the thing that
makes Leadbay useful: a name and an email to actually reach.
Gate 3 now runs in two beats:
Beat 1 — free. leadbay_enrich_titles with no titles/confirm/email/phone
returns mode:"discover", the available job titles. "Nothing spent
yet."
Beat 2 — paid, on consent. Ask them to pick 2-3 leads, state the cost BEFORE
they choose, then call again with the chosen titles, confirm:true
and email:true. Poll leadbay_bulk_enrich_status to completion and
report only the contacts that actually resolved.
The consent guarantee is the ordering, and it is explicit: the gate click
bought the free look, not the reveal. Silence is not consent, and neither is
"they clicked the gate earlier". Declining is a normal outcome — keep the
preview and move on.
It then says what it cost in one line (one credit per contact revealed), which
is the moment gate 1's quota numbers stop being abstract: they just watched
them move.
Gate 4 updated to match — it may now pass through real emails/phones, but ONLY
the ones the enrichment returned; if the user declined there are none, and
inventing one is fabrication.
The underdeliver eval keeps NO launch fixture on purpose: in that scenario the
user is never asked and never confirms, so a launch would hit an undeclared
endpoint and fail the run — the consent guarantee, enforced.
Co-Authored-By: Claude <noreply@anthropic.com>
The tour assumes an installed, signed-in connector — gate 1 is what proves it.
A user whose connector isn't installed, who can't sign in, or whose Leadbay
tools aren't appearing is upstream of every gate, and the walkthrough had
nothing for them but five buttons that would fail.
docs.leadbay.app/doc/leadbay-mcp/quickstart covers exactly that gap: connector
install, browser sign-in, the first query, and what to do when the tools don't
show up. The manifest now carries it as `docs_url`, with `docs_note` bounding
where it may appear:
1. BEFORE the tour, when the problem is SETUP rather than usage — point at
the page instead of guessing at install steps.
2. At the CLOSING, one plain link beside the keep_going cheat-sheet, for what
the five gates didn't cover (another machine, a teammate, signing back in).
Never between gates. A link mid-tour is an invitation to leave the thing the
user is in the middle of doing, which is why the audit pins the count at
exactly two rather than merely asserting the URL is present.
Both surfaces carry it — the prompt's routing branch + closing, and the tool
description — and the audit checks the prompt against the manifest's URL, so
the two can't drift the way the gate labels could.
Verified the count guard bites: injecting a third mention at gate 2 fails with
"expected the setup link twice, found 3".
Co-Authored-By: Claude <noreply@anthropic.com>
…send it to The tour ended at "here are some leads and here are some job titles." A first-run user never saw the thing that makes Leadbay worth opening: an email they could actually send. Two new beats, in the order that makes each one motivate the next. 3. "Draft the first email" -> leadbay_prepare_outreach (leadId ONLY = free) 4. "Find who to email" -> leadbay_enrich_titles, scoped to that ONE lead Gate 3 takes the top-scoring lead from gate 2 and drafts the opener through message_compose_v1. `recommended_contact` comes back with a null email at this point — that is not a gap to apologise for, it IS the hook: they now have an email written and nobody to send it to. Gate 4 answers exactly that question, which turns the paid reveal from an abstract "see who to contact" into the obvious next move, at one contact for one credit instead of two or three. Consent and honesty guards, each with a test: - Gate 3 spends NOTHING. `enrich:true` would launch a paid reveal off the back of a DRAFT click, so it is in forbidden_args with the reason attached. - The draft is addressed to the job TITLE. No contact name exists before gate 4 and inventing one is the single thing that would make the whole draft untrustworthy. - Gate 4 keeps the two-beat consent shape, now scoped to the drafted lead. The IRON LAW is NARROWED, deliberately: it said "never draft or send outreach", which the new gate would contradict outright. It now reads never SEND — drafting is the point and nothing leaves the chat, while sending, offering to send, and leadbay_report_outreach all stay forbidden. Gate labels are short because AskUserQuestion caps them at ~5 words; the audit caught "Draft an email to the top lead" at seven. Sentences live in `description`, where they belong. Also corrects drift the two-option and paid-reveal commits left behind: WORKFLOWS #50/#51 and the eval scenario still specified "EXACTLY ONE option, no 'Skip' sibling" — the normative eval contract asserting the opposite of shipped behaviour — and index.ts still called it "the three-gate script". Co-Authored-By: Claude <noreply@anthropic.com>
…rough Picking "I'm done for now" ended the tour in silence. That is the one moment a 1:1 offer is welcome rather than pushy: they have seen enough to know what Leadbay is, and stopped right before the setup work a call actually helps with — tuning the lens to their market, wiring the CRM push into their own stack, automating the daily run. The manifest gains `calendly_url` + `exit_offer`, and the prompt an "IF THEY TAKE THE EXIT" section. One short line and the link, then stop. Bounded on purpose, because the failure mode here is turning goodwill into a pitch: - ONE sentence and the link. They just said they were done. - Never re-open the walkthrough, never re-fire the declined gate, never argue for finishing the tour. - EXIT CLICK ONLY. If they left by TYPING a different request, skip it entirely and serve what they asked — a booking link on top of their real question is precisely the interruption they were avoiding. The URL is a PLACEHOLDER (calendly.com/leadbay-zoe/PLACEHOLDER) and must be swapped for Zoe's real link before this ships, or a new user gets a 404 at the friendliest moment in the tour. It is greppable by that literal word, and the audit pins the prompt's copy against the manifest so the two can't drift. Co-Authored-By: Claude <noreply@anthropic.com>
Replaces the placeholder that would have handed a new user a 404 at the friendliest moment in the tour. Verified the link resolves (HTTP 200). The `?month=2026-08` query param is stripped on purpose. It only pins which month Calendly's picker opens on, and this URL ships inside the product — a link carrying month=2026-08 opens on a stale calendar for everyone who clicks it after August. The bare URL opens on the current month, always. Co-Authored-By: Claude <noreply@anthropic.com>
…aimed the end
Live in Claude Desktop: stopping at gate 2 produced "No problem — we'll stop
there", the keep_going cheat-sheet and the docs link — and nothing about Zoe.
The offer never appeared.
Root cause was structural, not wording. The prompt shipped TWO sections that
both described the end of the tour: "IF THEY TAKE THE EXIT" and "CLOSING — HAND
THEM THE PHRASES". The agent reached the end, ran CLOSING, felt finished, and
stopped. The exit section it never applied. Worse, CLOSING's own skip-rule only
named the typed-off-script case, so an exit CLICK fell straight through to
"render the cheat-sheet" with nothing pointing back at the offer.
Merged into ONE section that owns how the tour stops, stating so explicitly
("this is the ONLY place that says what to do when the walkthrough stops"),
with three mutually-exclusive endings the agent picks between:
A finished all six gates -> cheat-sheet + setup link
B picked "I'm done for now" -> stop line + cheat-sheet + setup link
+ the 1:1 offer, REQUIRED and LAST
C typed something off-script -> none of it; serve what they asked
Beat 3 of ending B is called out as the one that gets dropped, with the reason
attached, because "an agent that renders the cheat-sheet feels finished" is
exactly what happened and a rule without its reason gets optimised away again.
The manifest's exit_offer now carries the same three-beat ordering, so both
surfaces describe one close rather than two.
Tests pin the structure a future edit would undo: the three endings exist and
are named where the agent decides, the offer is marked required-and-last, and
ending C explicitly gets no cheat-sheet, no link and no offer.
Co-Authored-By: Claude <noreply@anthropic.com>
`pnpm test:gate` failed at startup with "Could not resolve vitest.eval.config.ts". Not a failing eval — the runner could not START, and had not been able to since #71. That commit landed the live framework's parts (live-session-runner, live-mcp-server, mission-match-judge, eval-collector) and deleted the old fixture-based entry points, but never shipped replacements. The helpers were orphans: nothing imported them but each other, and all four package.json eval scripts pointed at files that do not exist on main. Adds the two missing pieces: - packages/mcp/vitest.eval.config.ts — serial (live sessions must not race), 15-min test timeout, includes only *.eval.ts so it can never overlap the unit config's *.test.ts. - packages/mcp/test/eval/scenarios.eval.ts — ONE runner over every *.scenario.ts, rather than the per-prompt boilerplate that rotted last time. Mechanical invariants first (required/forbidden calls, required_order as a subsequence), then the LLM judge, then the four floors. Three latent bugs surfaced the moment something finally imported the helpers, each found by running it rather than reading it: 1. mission-match-judge imports @leadbay/promptforge, whose exports point at dist/ — but promptforge's `build` script runs the prompt GENERATOR, and the actual tsc lives in a `compile` script nothing calls. dist/ has never existed. Aliased to source in the eval config. 2. callClaudeCLI built a shell command with JSON.stringify(prompt) — JSON escaping handed to /bin/sh, which applies its own. Died with "Unterminated quoted string" on the first real judge call, three times over. Now execFileSync with the prompt on stdin: no shell, no escaping. 3. The judge's 90s timeout could not read a 20-turn transcript. Raised, and made overridable via EVAL_JUDGE_TIMEOUT_MS. The getting-started scenarios were also single-turn, which cannot work for a GATED walkthrough: every gate stops and waits for a click, so the tour never advanced past gate 1 and the harness reported "required call never fired" — measuring the feature working correctly and calling it a failure. Both are now multi-turn, one turn per click, ending on the exit. Live proof, this branch: getting-started-completes-five-gates — MM 5 / IA 5 / NF 5 / TSF 5 19 assistant turns, 6 user turns, 217.8s, all 20 criteria passed That run exercises the whole walkthrough against the real Leadbay API, including the exit close that shipped broken earlier today. `pnpm -r test` is untouched (1351 passing): without EVAL=1 the eval file skips, so it can never fire in CI. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A brand-new user learns Leadbay by doing, not by reading. Five gates, each presenting exactly one option.
Check my accountleadbay_account_statusPull today's leadsleadbay_pull_leadsEnrich top leadsleadbay_enrich_titles(notitles→ freemode:"discover")Add these to my CRMRun this every morningShips as both a prompt (auto-emits a Claude Code skill) and a read-only composite tool returning the step manifest, so natural phrasing ("I'm new") reaches the tour on hosts that don't surface MCP prompts as slash commands.
Gate 1 sits on two pinned regressions
The tutorial opens by proving the connection works. That makes it the worst possible place to reintroduce either locked
account_statusbehaviour, so the manifest encodes both and the tests assert them:quota_status401s and lands inquota_error. The gate says nothing about quota and never suggests logging in again: the token is valid, the same response just read the account. (product#3761's 401-hallucination bug.) The underdeliver eval fixtures that 401 on purpose, so the run proves the silence instead of assuming it.account-status.ts:166withholds the lens unless the trigger text mentions it. "Walk me through Leadbay" doesn't, so there's nothing to report and no other tool to reach for. The lens appears naturally at gate 2.Leverage the host, don't fake the feature
Gates 4 and 5 delegate capabilities Leadbay does not have and the host usually does, returning
calls: nullso no agent can infer aleadbay_*tool that would do it.CRM (gate 4). Leadbay has no CRM integration — no push, export, sync, or third-party OAuth. (
report_friction's canonicalmissing_capabilityexample is literally "Why can't I export to HubSpot?".) But the agent often has a HubSpot/Salesforce/Pipedrive/Attio connector in the same host, so the gate tells it to check its own tool set and use that. Detection reuses the mechanism the connected-outreach-tool table inleadbay_prospecting_overviewalready established: host's installed-connector inventory → the conversation → ask.It names the capability, never a third-party tool name. A backticked
hubspot_create_companywould be the first such reference in this repo and would rot silently when that connector renames its tools.Scheduling (gate 5). Same shape. The gate's literal recurring wording ("every morning") is what lets the existing
SCHEDULED_TASK_PARAGRAPHflow fire on its own terms —server.tsis untouched, so the daily-check-in evals are unaffected.Honesty guards
Each has a test:
leadbay_report_frictionwithcategory: "missing_capability". Not instructions for a connector they don't have.titles/confirm/email/phonelaunches a paid reveal; spending a 90-second-old account's quota to demo a feature is the worst outcome this feature could produce.pull_leadsalready returns a server-built two-option warm-up payload forcomputing_wishlist/computing_scores, rendered verbatim.Gates
pnpm prompts:build·pnpm -r build·pnpm -r typecheck·pnpm -r test— all green: 1,318 tests, 0 failures (core 632, mcp 623, components 47, promptforge 16).pnpm prompts:checkclean (no template drift). Description well under the 17,000-char cap.Verified end-to-end over real MCP stdio JSON-RPC:
prompts/listreturns 13 (was 12) and includes the new prompt;prompts/getserves the full body with no unsubstituted placeholders. That confirms the two-place registration is complete —leadbay_extend_my_lensandleadbay_followup_check_inare both missing theirCATALOGentry today and so never appear inprompts/list.Eval coverage
The two scenarios are authored to the README shape and, like
pull-leads-orderandscan-portfolio-signals, go live when the scenario-execution glue lands — so there is no live judge run to report yet. No MM/IA/NF/TSF numbers; I won't paste a score I didn't observe. The deterministic red/green proof is in the unit mirrors:getting-started.test.ts(14 tests) — one option per gate, gate 1's two regressions, the four forbidden spend args,calls:null+ honesty guards on both delegated gates, the warming-lens branch, zero HTTP requestsgetting-started-walkthrough.test.ts(15 tests) — prompt↔manifest gate-label agreement, plus "names no third-party tool name", which is the drift a future edit would introduce silentlyI confirmed these bite: dropping
confirmfromforbidden_argsfails the spend-gate assertion.WORKFLOWS.md rows #50 (walkthrough) and #51 (over-claim guard) carry the contracts.
Not covered
Single-option widget rendering is unverified on a live host.
pull-leads.ts:123-127carries a code comment asserting a lone option makes the model emit an invalid widget call or silently drop to prose, and the shared routing snippet documents a 2–4 option contract. Built as specified with two mitigations — the prose fallback is stated in the tour, and the escape hatch is typing rather than a button — butAskUserQuestion(Claude Code) andask_user_input_v0(Claude chat) have different schemas and fail silently, so this needs eyes on both.The CRM push is untested against a real connector. No CRM MCP server was installed in this environment, so only the no-connector path was exercised. The with-connector branch is prompt guidance, verified by assertion, not by a live HubSpot write.
Also out of scope: storing a deep-link back to the CRM record (
crm-record-link.mdalready does this for imports) and makingleadbay_prospecting_overviewcross-route back to the tour — that regenerates its shipped snapshot, so it's a follow-up.Note for review
The only changes to existing test files are two one-line registration appends — the
prompts/listtoEqualarray andTOOLS_WITH_ROUTING. Both are the established way to register a prompt/routed tool here (precedent:6ddbcf85,dedb95e2; CLAUDE.md instructs the latter). No existing test logic was modified.Closes https://github.com/leadbay/product/issues/3952