Summary
evaluation_suite_arch/evaluate.py extracts the final score by looking only for the Chinese key 总得分. The English judging prompt asks the judge to emit "Total Score". As a result every English DE-Arch task is recorded as 0, regardless of the judge's actual verdict.
Location
dacomp-de/evaluation_suite_arch/evaluate.py, extract_actual_score():
def extract_actual_score(self, evaluation_result):
if "总得分" in evaluation_result:
return evaluation_result["总得分"]
elif "parse_error" not in evaluation_result:
total_score = 0
for value in evaluation_result.values():
if isinstance(value, dict) and "总得分" in value:
total_score += value["总得分"]
return total_score
return 0
The English prompt template (utils/eval_prompt.py) specifies a different key:
eval_prompt (English) → "Total Score": int (lines 65, 83)
eval_prompt_zh (Chinese) → "总得分": int (lines 153, 171)
So English responses fall through to return 0.
Reproduction
Run evaluate.py on any English DE-Arch task (--project-path
containing dacomp-de-arch-001.yaml).
Inspect the saved _evaluation.json.
Observed on our run:
raw_response ends with: "Total Score": 12
actual_score recorded as: 0
Chinese tasks in the same batch scored normally, which is the tell: 4 tasks, both English ones recorded 0, both Chinese ones non-zero.
Impact
With the released code, the English DE-Arch column cannot produce a non-zero score for any model.
Suggested fix
Accept both key names:
TOTAL_KEYS = ("总得分", "Total Score", "total_score")
for k in TOTAL_KEYS:
if k in evaluation_result:
return evaluation_result[k]
(Same for the nested-sum branch.)
Note: extract_max_score_from_rubric() in the same file already handles both 总分 and Total Score, so the bilingual intent is clearly there — this looks like an oversight in one function rather than a design decision.
Summary
evaluation_suite_arch/evaluate.py extracts the final score by looking only for the Chinese key 总得分. The English judging prompt asks the judge to emit "Total Score". As a result every English DE-Arch task is recorded as 0, regardless of the judge's actual verdict.
Location
dacomp-de/evaluation_suite_arch/evaluate.py, extract_actual_score():
def extract_actual_score(self, evaluation_result):
if "总得分" in evaluation_result:
return evaluation_result["总得分"]
elif "parse_error" not in evaluation_result:
total_score = 0
for value in evaluation_result.values():
if isinstance(value, dict) and "总得分" in value:
total_score += value["总得分"]
return total_score
return 0
The English prompt template (utils/eval_prompt.py) specifies a different key:
eval_prompt (English) → "Total Score": int (lines 65, 83)
eval_prompt_zh (Chinese) → "总得分": int (lines 153, 171)
So English responses fall through to return 0.
Reproduction
containing dacomp-de-arch-001.yaml).Run evaluate.py on any English DE-Arch task (--project-path
Inspect the saved _evaluation.json.
Observed on our run:
raw_response ends with: "Total Score": 12
actual_score recorded as: 0
Chinese tasks in the same batch scored normally, which is the tell: 4 tasks, both English ones recorded 0, both Chinese ones non-zero.
Impact
With the released code, the English DE-Arch column cannot produce a non-zero score for any model.
Suggested fix
Accept both key names:
TOTAL_KEYS = ("总得分", "Total Score", "total_score")
for k in TOTAL_KEYS:
if k in evaluation_result:
return evaluation_result[k]
(Same for the nested-sum branch.)
Note: extract_max_score_from_rubric() in the same file already handles both 总分 and Total Score, so the bilingual intent is clearly there — this looks like an oversight in one function rather than a design decision.