Skip to content

DE-Arch evaluator always scores English tasks 0 (extract_actual_score only reads the Chinese key) #7

Description

@co-tao

Summary
evaluation_suite_arch/evaluate.py extracts the final score by looking only for the Chinese key 总得分. The English judging prompt asks the judge to emit "Total Score". As a result every English DE-Arch task is recorded as 0, regardless of the judge's actual verdict.

Location
dacomp-de/evaluation_suite_arch/evaluate.py, extract_actual_score():

def extract_actual_score(self, evaluation_result):
if "总得分" in evaluation_result:
return evaluation_result["总得分"]
elif "parse_error" not in evaluation_result:
total_score = 0
for value in evaluation_result.values():
if isinstance(value, dict) and "总得分" in value:
total_score += value["总得分"]
return total_score
return 0
The English prompt template (utils/eval_prompt.py) specifies a different key:

eval_prompt (English) → "Total Score": int (lines 65, 83)
eval_prompt_zh (Chinese) → "总得分": int (lines 153, 171)
So English responses fall through to return 0.

Reproduction
Run evaluate.py on any English DE-Arch task (--project-path

containing dacomp-de-arch-001.yaml).
Inspect the saved _evaluation.json.
Observed on our run:

raw_response ends with: "Total Score": 12
actual_score recorded as: 0
Chinese tasks in the same batch scored normally, which is the tell: 4 tasks, both English ones recorded 0, both Chinese ones non-zero.

Impact
With the released code, the English DE-Arch column cannot produce a non-zero score for any model.

Suggested fix
Accept both key names:

TOTAL_KEYS = ("总得分", "Total Score", "total_score")
for k in TOTAL_KEYS:
if k in evaluation_result:
return evaluation_result[k]
(Same for the nested-sum branch.)

Note: extract_max_score_from_rubric() in the same file already handles both 总分 and Total Score, so the bilingual intent is clearly there — this looks like an oversight in one function rather than a design decision.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions