I am running EdgeBench with the official Codex configuration:
model:
model: gpt-5.5
defaults:
agent: codex
I would like to explicitly specify the model’s reasoning effort, such as low, medium, or high, to ensure that evaluation results are reproducible and comparable.
Findings from the source code
After reviewing the current implementation, I found the following:
-
sforge/harness/agent/codex.py installs @openai/codex@0.130.0.
-
When OPENAI_BASE_URL is set, the harness generates ~/.codex/config.toml containing:
model_provider = "sforge-proxy"
model_verbosity = "medium"
model = "${CODEX_MODEL:-gpt-5.5}"
However, this configuration does not explicitly set model_reasoning_effort. model_verbosity appears to control response verbosity rather than reasoning effort.
- The Codex execution command is:
codex exec --dangerously-bypass-approvals-and-sandbox "$(cat {prompt_file})"
No reasoning-effort option is passed through the command line.
-
SForgeConfig exposes agent_model and agent_extra_env, but I could not find a dedicated reasoning-effort field.
-
sforge/visualizer/parsers/codex_output.py parses reasoning effort from the Codex session header and stores it as reasoning_effort. The example in that parser contains:
This only demonstrates that the effective effort is observable in Codex output. It does not confirm which reasoning effort was used for the official GPT-5.5 leaderboard runs, because the example uses a different Codex version and model.
Questions
-
Is there currently a supported way to configure reasoning effort for GPT-5.5 with the Codex harness, through the experiment YAML, an environment variable, or another configuration option?
-
Which reasoning effort was used for the official GPT-5.5 + Codex leaderboard results: low, medium, high, or a model/CLI default?
-
If the harness relies on the Codex CLI default, could changes in the default behavior across Codex CLI versions affect reproducibility and leaderboard comparability?
-
Would it be possible to expose an explicit option such as:
model:
model: gpt-5.5
reasoning_effort: high
and map it to either:
model_reasoning_effort = "high"
or:
codex exec -c model_reasoning_effort=high ...
Explicitly recording the effective reasoning effort in the experiment metadata would also help reproduce and compare evaluation results.
I am running EdgeBench with the official Codex configuration:
I would like to explicitly specify the model’s reasoning effort, such as
low,medium, orhigh, to ensure that evaluation results are reproducible and comparable.Findings from the source code
After reviewing the current implementation, I found the following:
sforge/harness/agent/codex.pyinstalls@openai/codex@0.130.0.When
OPENAI_BASE_URLis set, the harness generates~/.codex/config.tomlcontaining:However, this configuration does not explicitly set
model_reasoning_effort.model_verbosityappears to control response verbosity rather than reasoning effort.No reasoning-effort option is passed through the command line.
SForgeConfigexposesagent_modelandagent_extra_env, but I could not find a dedicated reasoning-effort field.sforge/visualizer/parsers/codex_output.pyparsesreasoning effortfrom the Codex session header and stores it asreasoning_effort. The example in that parser contains:This only demonstrates that the effective effort is observable in Codex output. It does not confirm which reasoning effort was used for the official GPT-5.5 leaderboard runs, because the example uses a different Codex version and model.
Questions
Is there currently a supported way to configure reasoning effort for GPT-5.5 with the Codex harness, through the experiment YAML, an environment variable, or another configuration option?
Which reasoning effort was used for the official GPT-5.5 + Codex leaderboard results:
low,medium,high, or a model/CLI default?If the harness relies on the Codex CLI default, could changes in the default behavior across Codex CLI versions affect reproducibility and leaderboard comparability?
Would it be possible to expose an explicit option such as:
and map it to either:
or:
codex exec -c model_reasoning_effort=high ...Explicitly recording the effective reasoning effort in the experiment metadata would also help reproduce and compare evaluation results.