Skip to content

**How can reasoning effort be explicitly configured for GPT-5.5 with the Codex harness?** #8

Description

@XueqingLin

I am running EdgeBench with the official Codex configuration:

model:
  model: gpt-5.5

defaults:
  agent: codex

I would like to explicitly specify the model’s reasoning effort, such as low, medium, or high, to ensure that evaluation results are reproducible and comparable.

Findings from the source code

After reviewing the current implementation, I found the following:

  1. sforge/harness/agent/codex.py installs @openai/codex@0.130.0.

  2. When OPENAI_BASE_URL is set, the harness generates ~/.codex/config.toml containing:

model_provider = "sforge-proxy"
model_verbosity = "medium"
model = "${CODEX_MODEL:-gpt-5.5}"

However, this configuration does not explicitly set model_reasoning_effort. model_verbosity appears to control response verbosity rather than reasoning effort.

  1. The Codex execution command is:
codex exec --dangerously-bypass-approvals-and-sandbox "$(cat {prompt_file})"

No reasoning-effort option is passed through the command line.

  1. SForgeConfig exposes agent_model and agent_extra_env, but I could not find a dedicated reasoning-effort field.

  2. sforge/visualizer/parsers/codex_output.py parses reasoning effort from the Codex session header and stores it as reasoning_effort. The example in that parser contains:

reasoning effort: high

This only demonstrates that the effective effort is observable in Codex output. It does not confirm which reasoning effort was used for the official GPT-5.5 leaderboard runs, because the example uses a different Codex version and model.

Questions

  1. Is there currently a supported way to configure reasoning effort for GPT-5.5 with the Codex harness, through the experiment YAML, an environment variable, or another configuration option?

  2. Which reasoning effort was used for the official GPT-5.5 + Codex leaderboard results: low, medium, high, or a model/CLI default?

  3. If the harness relies on the Codex CLI default, could changes in the default behavior across Codex CLI versions affect reproducibility and leaderboard comparability?

  4. Would it be possible to expose an explicit option such as:

model:
  model: gpt-5.5
  reasoning_effort: high

and map it to either:

model_reasoning_effort = "high"

or:

codex exec -c model_reasoning_effort=high ...

Explicitly recording the effective reasoning effort in the experiment metadata would also help reproduce and compare evaluation results.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions