Model Validation Infrastructure, Script, and Workflow

[cwitkowitz]

Implements a model validation framework executed manually or as a daily GitHub Action. The main script headlessly drives the pyharp gradio endpoints (/controls, /process) across two tiers: local pyharp/examples/ apps, and every Hugging Face Space under teamup-tech.

Every model gets a default case with inputs synthesized from its own /controls spec; config.yml adds per-model and shared cases, declarative expect rules on outputs (audio/MIDI/label properties), and custom validators.

See model_validation/README.md for more details.

One current limitation is the lack of precision in tracking ZeroGPU quota usage. Since quota usage is dynamic and can be refunded, it might be worthwhile to retry ZeroGPU models before outright skipping the remaining ones after a single ZeroGPU quota exceeded error. Another workaround could be to set the number of workers to 1 to avoid eating up extra quota due to initial time reservations.

[vansh-c2002]

Here, if a text_box's default value happens to be "", "" or "test" would resolve to "test." Since the intent is default values everywhere, this may be an edge case we're overlooking.

elif ctype == "text_box":
args.append(spec.get("value") if spec.get("value") is not None else "test")
OR
elif ctype == "text_box":
value = spec.get("value")
args.append(value if value is not None else "test")

[cwitkowitz]

Is there any situation where a text box would have value=None? We don't set a default in pyharp, but does it show up as None or "" if unspecified?

[vansh-c2002]

I just checked with

import gradio as gr
tb = gr.Textbox()
print(tb.value)

This returned None. But gr.Textbox("") will return "", so I'm assuming we get the latter.

I looked through and i think the space has models deployed where either is the expected, so it may be good to test for both cases there?

MMT, MuScriptor and STAGE have '' defaults
And there are several others that have None as a default.

[2cylu2]

I ran the validation script locally against both local example servers (--local-examples) and remote Spaces:
--local-examples --load-only passes all 4 models.
Running full inference (--local-examples) fails for midi_synthesizer and ui_tester with:
AppError: The upstream Gradio app has raised an exception but has not enabled verbose error reporting. To enable, set show_error=True in launch().

It might be helpful to enable show_error=True either in pyharp's build_endpoint() or when launching the local example servers in harness.py. That way, validate_models.py can capture the actual traceback in report.json instead of the generic AppError, which should make debugging failures easier.

One thought on the ZeroGPU reservation issue mentioned in the PR description: instead of reducing --workers to 1 globally (which could slow down CPU model validation), we could run only the ZeroGPU models one at a time while keeping CPU models running in parallel. Since this script runs automatically, taking a little longer seems like a reasonable tradeoff if it helps avoid unnecessary quota usage.

One model (generalizable-accompaniment-generation) failed because it requires audio between 3 and 30 seconds long.
Would it make sense to increase the default synthesized audio duration to 3 or 5 seconds in assets.py, or should models with duration requirements always specify custom test files in config.yml under test_data/?