Console-first model setup: /keys + /model become the complete, reliable path

CLAUDE.md

@@ -51,7 +51,7 @@ Four loops communicating **only through NDJSON/JSON files** in the watched repo'
 
 **Model boundary:** all model calls go through `critic/agent.py` — one non-interactive [pi](https://pi.dev) turn (`pi -p --no-session --no-tools …`, persona via `--system-prompt`). Judgment turns get no tools; verification turns also get no tools — the model instead writes a self-contained repro SCRIPT, which the harness (not the model) then executes in a throwaway staging directory (`critic/verify.py`, sharing `critic/probe.py`'s `run_script`), so repros never touch the watched repo. This replaced an earlier tool-enabled (`read,bash`) verification turn: the NVIDIA/pi backend frequently emitted its tool calls as literal, never-executed text, which made verification land "inconclusive" and withheld true findings — a script the harness executes itself has no such failure mode. `COUNCIL_MODEL=provider/model` overrides pi's default; `PI_BIN` overrides the executable. Set `CRITIC_CMD=<executable>` to stub the model in tests: it runs as `$CRITIC_CMD <prompt-file> <resolved-model>`, stdout is the reply. Personas live in `critic/persona.md` and `reflector/persona.md`.
 
-`critic/agent.py` also auto-loads `~/.codecouncil/env` (outside any watched repo — a credential placed there can never be committed regardless of which repo CodeCouncil is pointed at) to top up the subprocess environment, and always attaches `critic/pi_extensions/nvidia_provider.mjs` via `pi -e`. If `NVIDIA_API_KEY` resolves (real env or that file) and `COUNCIL_MODEL` is unset, the default becomes NVIDIA-hosted Nemotron (`nvidia-nim/nvidia/nemotron-3-super-120b-a12b`) — zero pi login required. Model `id`s in that extension must be NVIDIA's full catalog string (e.g. `nvidia/nemotron-3-super-120b-a12b`); pi's `openai-completions` provider sends `model.id` verbatim as the request's `model` field, so a shorter id 404s.
+`critic/agent.py` also auto-loads `~/.codecouncil/env` (outside any watched repo — a credential placed there can never be committed regardless of which repo CodeCouncil is pointed at) to top up the subprocess environment, and always attaches `critic/pi_extensions/nvidia_provider.mjs` via `pi -e`. If `COUNCIL_MODEL` is unset, the first configured key picks the default model (`core.config.KEY_DEFAULT_MODELS`, ordered: free NVIDIA-hosted Nemotron first, Anthropic last for decorrelation) — zero pi login required. Model `id`s in that extension must be NVIDIA's full catalog string (e.g. `nvidia/nemotron-3-super-120b-a12b`); pi's `openai-completions` provider sends `model.id` verbatim as the request's `model` field, so a shorter id 404s.
 
 **Measurement:** two independent signals of self-improvement — acceptance rate per heuristics version (`reflector/report.py`, mirrored exactly by `ui/server/council.ts` so the dashboard can't diverge from the real metric) and frozen eval cases (`evals/cases/*.json` + harvested `evals/cases-harvested/*.json`) replayed against every heuristics version. The signals are deliberately kept separate: the rewrite gate uses eval scores, rollback uses in-the-wild acceptance — never mix them.
 

README.md

@@ -66,8 +66,9 @@ on your PATH, wires up [pi](https://pi.dev) (the model runtime) if npm is
 available, and scaffolds `~/.codecouncil/env` for your key. Then:
 
 ```sh
-echo 'NVIDIA_API_KEY=nvapi-...' >> ~/.codecouncil/env   # free — see "Model providers"
-codecouncil /path/to/repo-you-code-in                   # hooks + all three loops (defaults to `.`)
+codecouncil /path/to/repo-you-code-in   # hooks + all three loops (defaults to `.`)
+# then type /keys in the running council — guided key setup with hidden input;
+# a model is picked automatically (free NVIDIA key: see "Model providers")
 ```
 
 > **Completely free path**, spelled out step by step — key signup on
@@ -137,8 +138,8 @@ slash commands work in place, Claude Code-style:
 
 | Command | What it does |
 |---|---|
-| `/keys` | Guided API-key setup (hidden input, saved to `~/.codecouncil/env`) |
-| `/model <p/m>` | Set + persist the primary model (restarts just the critic) |
+| `/keys` | Guided API-key setup (hidden input, saved to `~/.codecouncil/env`) — then offers that provider's model if you're not already on it |
+| `/model [p/m]` | Show (bare) or set + persist the primary model — set warns on a missing key or malformed id, and beats any launch `--model`/env (restarts just the critic) |
 | `/prober <p/m\|off>` | Council mode on/off (restarts just the critic) |
 | `/status` | Daemons, beats, last verdict, heuristics version, keys |
 | `/config` | Resolved configuration and where each value came from |
@@ -173,9 +174,10 @@ login — NVIDIA hosts Nemotron and other open models with a free API key:
 2. Open any model page and click **Get API Key** — it starts with
    `nvapi-`. (NVIDIA's own docs:
    [docs.api.nvidia.com](https://docs.api.nvidia.com/nim/reference/getting-started).)
-3. `echo 'NVIDIA_API_KEY=nvapi-...' >> ~/.codecouncil/env` — with that key
-   present and no model configured, CodeCouncil defaults to NVIDIA-hosted
-   Nemotron automatically.
+3. Run `codecouncil` and type `/keys` — guided, hidden input (for scripts,
+   the one-liner still works: `echo 'NVIDIA_API_KEY=nvapi-...' >> ~/.codecouncil/env`).
+   With a key present and no model configured, CodeCouncil picks that
+   provider's default model automatically.
 
 | Provider | Key in `~/.codecouncil/env` | Example `/model` value |
 |---|---|---|
@@ -186,6 +188,11 @@ login — NVIDIA hosts Nemotron and other open models with a free API key:
 | Google | `GEMINI_API_KEY` | `google/gemini-3-flash-preview` |
 | Groq | `GROQ_API_KEY` | `groq/openai/gpt-oss-120b` |
 
+With a key configured and no model set, CodeCouncil picks that provider's
+table entry automatically (first configured key wins, free NVIDIA first,
+Anthropic last) — `/keys` alone is a working setup; `/model` is only needed
+to switch.
+
 The `nvidia-nim/…` and `openrouter/…` IDs above are the exact strings from
 our [bake-off](docs/benchmarks/); for other providers, any model ID from
 [pi's provider list](https://pi.dev/docs) works as `provider/model-id`.

codecouncil/console.py

@@ -20,7 +20,7 @@
 HELP = """\
 commands (while the council runs):
   /keys              set up a model API key (guided, hidden input)
-  /model <p/m>       set + persist the primary model (restarts the critic)
+  /model [p/m]       show or set + persist the primary model (set restarts the critic)
   /prober <p/m|off>  set + persist the council prober (restarts the critic)
   /status            daemons, beats, last verdict, heuristics version, keys
   /config            show resolved configuration and where it came from
@@ -45,11 +45,18 @@ class Console:
     console never has to know how subprocesses are managed."""
 
     def __init__(self, repo: Path, restart_critic: Callable[[], None],
-                 stop: Callable[[], None], say: Callable[[str], None]):
+                 stop: Callable[[], None], say: Callable[[str], None],
+                 settings_info: Callable[[], dict] | None = None,
+                 on_override: Callable[[str], None] | None = None):
         self.repo = repo
         self.restart_critic = restart_critic
         self.stop = stop
         self.say = say
+        # settings_info: launcher closure -> {model, model_source, prober,
+        # prober_source, env}; on_override(knob): tells the launcher a knob was
+        # set here, so config.json outranks the launch flag/env from now on.
+        self.settings_info = settings_info
+        self.on_override = on_override or (lambda _knob: None)
 
     def handle(self, line: str) -> None:
         parsed = parse_command(line)
@@ -100,23 +107,83 @@ def _cmd_keys(self, _arg: str) -> None:
         cfg.update_env_key(name, value)
         self.say(f"{name} saved to {cfg.env_path()} (0600). Takes effect on the "
                  "next model call — no restart needed.")
+        self._offer_model_for_key(name)
 
     def _cmd_model(self, arg: str) -> None:
         if not arg:
-            self.say("usage: /model <provider/model>  (e.g. nvidia-nim/nvidia/nemotron-3-super-120b-a12b)")
+            self._model_info()
             return
+        for w in cfg.check_model(arg, self._env()):
+            self.say(f"warning: {w}")
         cfg.save_config({"model": arg})
+        self.on_override("model")
         self.say(f"primary model → {arg} (persisted). Restarting the critic…")
         self.restart_critic()
 
     def _cmd_prober(self, arg: str) -> None:
         if not arg:
             self.say("usage: /prober <provider/model> | /prober off")
             return
+        if arg.lower() != "off":
+            for w in cfg.check_model(arg, self._env()):
+                self.say(f"warning: {w}")
         cfg.save_config({"prober": None if arg.lower() == "off" else arg})
+        self.on_override("prober")
         self.say(f"prober → {arg} (persisted). Restarting the critic…")
         self.restart_critic()
 
+    def _env(self) -> dict:
+        """Key material for validation: settings_info's env when injected
+        (the launcher passes agent.local_env(), which includes
+        ~/.codecouncil/env), else read it directly."""
+        if self.settings_info:
+            return self.settings_info().get("env", {})
+        from critic.agent import local_env
+        return local_env()
+
+    def _model_info(self) -> None:
+        """Bare /model: current resolved model, which layer set it, and
+        copy-pasteable examples for the keys actually configured."""
+        info = self.settings_info() if self.settings_info else {}
+        env = self._env()
+        model, src = info.get("model"), info.get("model_source")
+        if model:
+            self.say(f"model: {model}  (source: {src})")
+        else:
+            self.say("model: pi default (nothing configured)")
+        have = [(k, d) for k, d in cfg.KEY_DEFAULT_MODELS if env.get(k)]
+        if have:
+            self.say("examples for your configured keys:")
+            for k, d in have:
+                self.say(f"  /model {d}   ({k} ✓)")
+        else:
+            self.say("no API keys configured — run /keys first")
+        self.say("usage: /model <provider/model-id>")
+
+    def _offer_model_for_key(self, key_name: str) -> None:
+        """After saving a key, close the loop on the model: if the resolved
+        model already runs on this key, say so; otherwise offer this key's
+        default so /keys alone always ends in a working, intentional setup."""
+        default = dict(cfg.KEY_DEFAULT_MODELS).get(key_name)
+        if not default or not self.settings_info:
+            return
+        info = self.settings_info()   # post-save: env file already updated
+        current = info.get("model")
+        if not current:
+            return
+        provider = current.split("/", 1)[0]
+        if cfg.PROVIDER_KEYS.get(provider) == key_name:
+            self.say(f"critic model: {current} (source: {info.get('model_source')})")
+            return
+        ans = input(f"switch primary model to {default}? [y/N]: ").strip().lower()
+        if ans in ("y", "yes"):
+            cfg.save_config({"model": default})
+            self.on_override("model")
+            self.say(f"primary model → {default} (persisted). Restarting the critic…")
+            self.restart_critic()
+        else:
+            self.say(f"keeping {current} — `/model {default}` switches later.")
+
     def _cmd_status(self, _arg: str) -> None:
         cc = self.repo / ".codecouncil"
         state = self._json(cc / "state.json")

codecouncil/main.py

@@ -41,9 +41,10 @@ def preflight(model: str | None, prober: str | None = None) -> list[str]:
     env = agent.local_env()  # includes ~/.codecouncil/env
     has_key = any(env.get(v) for v in KEY_VARS)
     if not model and not env.get("COUNCIL_MODEL") and not has_key:
-        warns.append("no model configured: pass --model, set COUNCIL_MODEL, or put an "
-                     "API key in ~/.codecouncil/env. pi will fall back to its own default, "
-                     "which may not be authenticated.")
+        warns.append("no model configured and no API key found: type /keys in this "
+                     "console once the council starts (guided, hidden input), or pass "
+                     "--model / set COUNCIL_MODEL / add a key to ~/.codecouncil/env. "
+                     "pi will fall back to its own default, which may not be authenticated.")
     # Council mode (Task 4): the prober is a second, independent model call
     # (critic/main.py's resolve_prober precedence: --prober flag > this same
     # COUNCIL_PROBER env fallback > None). openrouter/* providers need
@@ -88,13 +89,22 @@ def _pump(name: str, proc: subprocess.Popen) -> None:
             print(f"{_tag(name)} {text}", flush=True)
 
 
-def resolve_settings(args) -> tuple[str | None, str | None]:
-    """flag > env var > ~/.codecouncil/config.json — one rule for both knobs."""
+def resolve_settings(args, console_set: frozenset | set = frozenset()
+                     ) -> tuple[str | None, str | None]:
+    """flag > env var > ~/.codecouncil/config.json — one rule for both knobs.
+    A knob named in console_set was just set via /model or /prober: the console
+    persisted it to config.json, so the launch-time flag and any exported env
+    var must stop outranking it — that knob resolves from the config file only."""
     from core import config as cfg
-    env = os.environ
-    model = cfg.resolve(args.model, "COUNCIL_MODEL", "model", dict(env))
-    prober = cfg.resolve(args.prober, "COUNCIL_PROBER", "prober", dict(env))
-    return model, prober
+    env = dict(os.environ)
+
+    def one(knob: str, flag: str | None, env_name: str, key: str) -> str | None:
+        if knob in console_set:
+            return cfg.resolve(None, env_name, key, {})
+        return cfg.resolve(flag, env_name, key, env)
+
+    return (one("model", args.model, "COUNCIL_MODEL", "model"),
+            one("prober", args.prober, "COUNCIL_PROBER", "prober"))
 
 
 def main(argv: list[str] | None = None) -> int:
@@ -112,6 +122,8 @@ def main(argv: list[str] | None = None) -> int:
         print(f"error: {repo} is not a directory", file=sys.stderr)
         return 2
 
+    console_set: set[str] = set()  # knobs reconfigured via /model | /prober
+
     model, prober = resolve_settings(args)
     for w in preflight(model, prober):
         print(f"{_tag('critic')} warning: {w}", flush=True)
@@ -125,7 +137,7 @@ def main(argv: list[str] | None = None) -> int:
 
     def launch(name: str) -> None:
         # settings re-resolve on every (re)launch so /model and /prober apply
-        m, p = resolve_settings(args)
+        m, p = resolve_settings(args, console_set)
         env = os.environ.copy()
         if m:
             env["COUNCIL_MODEL"] = m
@@ -171,12 +183,38 @@ def restart_critic() -> None:
                 old.kill()
         launch("critic")
 
+    def settings_info() -> dict:
+        """Resolved model/prober + which layer won — for /model and /keys.
+        Adds the auto-default layer below config: with no explicit model, the
+        critic falls to the first configured key's default (critic/agent.py's
+        _resolve_model), and the console should show that truthfully."""
+        from core import config as cfg
+        env_file = agent.local_env()   # includes ~/.codecouncil/env keys
+        env = dict(os.environ)
+
+        def one(knob, flag, env_name, key):
+            if knob in console_set:
+                return cfg.resolve_with_source(None, env_name, key, {})
+            return cfg.resolve_with_source(flag, env_name, key, env)
+
+        m, msrc = one("model", args.model, "COUNCIL_MODEL", "model")
+        if m is None:
+            for k, d in cfg.KEY_DEFAULT_MODELS:
+                if env_file.get(k):
+                    m, msrc = d, f"auto:{k}"
+                    break
+        p, psrc = one("prober", args.prober, "COUNCIL_PROBER", "prober")
+        return {"model": m, "model_source": msrc,
+                "prober": p, "prober_source": psrc, "env": env_file}
+
     console_note = ""
     if sys.stdin.isatty():
         from .console import Console
         console = Console(repo=repo, restart_critic=restart_critic,
                           stop=stopping.set,
-                          say=lambda m: print(f"{_tag('critic')} {m}", flush=True))
+                          say=lambda m: print(f"{_tag('critic')} {m}", flush=True),
+                          settings_info=settings_info,
+                          on_override=console_set.add)
 
         def _read_stdin() -> None:
             for line in sys.stdin:

core/config.py

@@ -26,6 +26,53 @@
     "GROQ_API_KEY": "Groq-hosted open models (gsk_...)",
 }
 
+# provider prefix (first path segment of a "provider/model-id" value) -> the
+# API key that provider needs. /model uses this to warn at set time instead
+# of letting a missing key surface as per-beat critic failures.
+PROVIDER_KEYS = {
+    "nvidia-nim": "NVIDIA_API_KEY",
+    "openrouter": "OPENROUTER_API_KEY",
+    "openai": "OPENAI_API_KEY",
+    "anthropic": "ANTHROPIC_API_KEY",
+    "google": "GEMINI_API_KEY",
+    "groq": "GROQ_API_KEY",
+}
+
+# When no model is configured anywhere, the first key present picks a default
+# so a single /keys entry always yields a working council. Ordered: free
+# NVIDIA first; Anthropic last (a critic from the coding agent's own family
+# shares its blind spots — README "Model providers" caveat).
+KEY_DEFAULT_MODELS = (
+    ("NVIDIA_API_KEY", "nvidia-nim/nvidia/nemotron-3-super-120b-a12b"),
+    ("OPENROUTER_API_KEY", "openrouter/openai/gpt-5-mini"),
+    ("OPENAI_API_KEY", "openai/gpt-5-mini"),
+    ("GROQ_API_KEY", "groq/openai/gpt-oss-120b"),
+    ("GEMINI_API_KEY", "google/gemini-3-flash-preview"),
+    ("ANTHROPIC_API_KEY", "anthropic/claude-haiku-4-5"),
+)
+
+# providers whose model ids nest a vendor path (openrouter/openai/gpt-5-mini,
+# nvidia-nim/nvidia/nemotron-…) — a single segment after the prefix 404s
+_NESTED_ID_PROVIDERS = ("openrouter", "nvidia-nim")
+
+
+def check_model(model: str, env: dict[str, str]) -> list[str]:
+    """Warnings (never errors) for a /model value. Pure — no I/O."""
+    if "/" not in model:
+        return [f"'{model}' doesn't look like provider/model-id — e.g. openai/gpt-5-mini"]
+    provider, rest = model.split("/", 1)
+    warns = []
+    key = PROVIDER_KEYS.get(provider)
+    if key and not env.get(key):
+        warns.append(f"{provider}/… needs {key}, which isn't set — run /keys first")
+    if key is None:
+        warns.append(f"unknown provider '{provider}' — if pi doesn't support it, "
+                     "every critic beat will fail (see pi.dev/docs for providers)")
+    if provider in _NESTED_ID_PROVIDERS and "/" not in rest:
+        warns.append(f"{provider} model ids are nested — expected the full path, "
+                     f"e.g. {dict(KEY_DEFAULT_MODELS)[PROVIDER_KEYS[provider]]}")
+    return warns
+
 
 def config_path(base: Path | None = None) -> Path:
     return (base or CONFIG_DIR) / "config.json"
@@ -86,7 +133,22 @@ def update_env_key(name: str, value: str, base: Path | None = None) -> None:
     os.chmod(p, 0o600)
 
 
+def resolve_with_source(flag: str | None, env_name: str, config_key: str,
+                        env: dict[str, str], base: Path | None = None
+                        ) -> tuple[str | None, str]:
+    """resolve() plus WHERE the value came from: 'flag' | 'env:<NAME>' |
+    'config' | 'default' — so /model and /status can show the layer that won."""
+    if flag:
+        return flag, "flag"
+    if env.get(env_name):
+        return env[env_name], f"env:{env_name}"
+    v = load_config(base).get(config_key)
+    if v:
+        return v, "config"
+    return None, "default"
+
+
 def resolve(flag: str | None, env_name: str, config_key: str,
             env: dict[str, str], base: Path | None = None) -> str | None:
     """The one precedence rule: flag > env var > config file > None."""
-    return flag or env.get(env_name) or load_config(base).get(config_key) or None
+    return resolve_with_source(flag, env_name, config_key, env, base)[0]