Add CVAT audio transcription pipeline support#3972
Open
zhiltsov-max wants to merge 53 commits into
Open
Conversation
Break the 3k-line src/handlers/job_creation.py into a package with one
file per task builder:
- builders/vision/{base,basic,boxes_from_points,skeletons_from_boxes}.py
- builders/audio/transcription.py (stub, raises NotImplementedError)
- factory.py (create_task), exceptions.py, utils.py
__init__.py exports only create_task.
Move DM_DATASET_FORMAT_MAPPING / DM_GT_DATASET_FORMAT_MAPPING to a new
src/core/tasks/cvat_formats.py and repoint job_export.py and the webhook
test's mock-patch paths to the new module locations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ckage roots
- Move run.py and debug.py into src/entrypoints/; launch them via
'python -m src.entrypoints.{run,debug}' in bin/start_*.sh. Fix debug.py
repo_root to Path(__file__).parents[2] after the move.
- Move the FastAPI app out of src/__init__.py into a new
src/apps/exchange_oracle.py; src/__init__.py is now empty.
- Repoint consumers: tests/conftest.py and the uvicorn app strings now use
src.apps.exchange_oracle:app, and src/crons/_cron_job.py imports Config
from src.core.config instead of the package root.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the audio_transcription task builder end to end: parse the input regions/GT TSVs, select ground-truth honeypots and bundle DS regions, mix them into assignments, cut and concatenate per-assignment clips via ffmpeg, upload clips + input copies + region/assignment meta to the oracle bucket, and create one CVAT task per assignment. - core/tasks/audio_transcription: task spec, domain meta types, and meta (de)serialization - utils/audio: ffmpeg cut_and_concat / probe_duration helpers - debug: route the audio builder through the dev cloud-storage host patch - optionally dump per-region DS/GT cuts when DEBUG is enabled Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match the CVAT server version used by the local deployment; the older 2.37.0 client rejected newer server responses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Bound region bundling: cap the join gap at min_duration and the merged span at max_duration, so bundles no longer bridge long silences over sparse input (kept assignment durations near the target). - Store assignment clips under a clips/ subdir on the oracle bucket; reference the full key in task creation (CVAT requires it). - Upload copies of the input regions/GT TSVs alongside the meta. - Optionally dump per-region DS/GT cuts into debug dirs when DEBUG is set. - Add a configurable random_seed to the task spec, forwarded to honeypot selection and assignment mixing. - Forward the annotation user_guide from the manifest to the CVAT project. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TranscriptionTaskValidation.target_metric was a plain str; make it the TargetMetrics enum (accuracy/wer/cer) and stop stringifying it in parse_audio_manifest. Applied identically in the exchange and recording oracles. Also correct the stale `# noqa: TC001` to `TCH001` on the runtime-needed pydantic imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Score audio-transcription assignments in the recording oracle (WER/CER, no CVAT calls, no honeypot rotation): download per-assignment TSVs + GT + assignment mapping, align annotator vs GT per presented honeypot via the audio_qe library, and accept/reject against the manifest target score. Supporting shared-core changes (kept identical across both oracles): - TaskResultsLayout with assignment_annotation_filename(); reused by the EO validator and RO checker. - parse_gt_tsv() extracted into meta.py; reused by the EO builder and RO checker (drops the RO-local _GtRow in favour of InputGtRegion). - NormalizerPreset extended to basic + 15 Tier-1 languages. - Rename LowAccuracyError -> LowQualityError (RO). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The oracle apps run from src/ and are never installed as a package, but
pyproject declared packages = [{include = "<name>_oracle"}] pointing at a
non-existent top-level dir, so `poetry install` failed/warned with
'does not contain any element'. Mark both projects non-package so install
resolves dependencies only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the exchange oracle layout in the recording oracle: relocate the
root run.py / debug.py to src/entrypoints/ and launch them via
'python -m src.entrypoints.{run,debug}' in the start scripts.
Recommend in-project virtualenvs (poetry config virtualenvs.in-project
true) in both READMEs so editors/debuggers pick up the .venv interpreter
(needed for the RO's local audio-qe path dependency), and ignore /.venv
in the exchange oracle .gitignore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restructure the escrow_recorded/export flow into a completion/ package (handlers + final_results + task_exporters/), mirroring validation/. Add a task-dispatched TaskExporter family: the image exporter wraps the existing datumaro GT merge; the audio exporter appends the ground-truth rows from gt.tsv to the exchange oracle's annotations.tsv (GT copied into the output), sorted per file+time with reassigned ids and a 'source' column marking each row as 'annotation' or 'gt', packed into the standard resulting_annotations.zip. Share the merged-annotations filename via TaskResultsLayout.ANNOTATIONS_FILENAME across both oracles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…imedelta in the impl
The audio transcription task builder imports numpy (seeded RNG for honeypot/assignment mixing) but numpy was only pulled in transitively. Declare it explicitly so the import can't break on a dependency reshuffle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…erance Validation-side improvements for audio transcription, shared across the exchange and recording oracles: - GT is organized into densely-annotated spans via a `span_id` column; honeypot cues are joined only within a span, never across gaps. - Add `min_gt_span_duration` detail: spans shorter than it are excluded. - Add `boundary_tolerance` detail (default 200ms), forwarded to the audio validator so annotator/GT interval boundary drift within the join pause no longer drops a honeypot. - Recording-oracle audio checker: resolve each job to its clip via the EO-recorded task->clip mapping; extend the honeypot inclusion window by half the adjacent pause on each side. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile lockfiles on top of the lib-bump base (cvat-sdk 2.69, audio-qe, numpy) after rebasing the audio work onto the library-bump + blockchain-node-drop branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Exchange oracle: unit test for audio-transcription task creation, asserting that input regions/GT are consumed as expected and each assignment mixes DS work with GT honeypots. Recording oracle: validation tests (pass + fail) driving the full process_intermediate_results flow against a golden fixture captured from the real EO builder. A perfect submission scores WER 0.0 (success); an empty one scores 1.0 and is rejected (failure). Supporting changes: - tests/utils/audio_transcription.py: shared helpers, kept byte-identical across both oracles. - tests/assets/utils/gen_audio_validation_fixture.py: maintenance script that runs the real EO builder and vendors its output as the RO golden fixture; runnable in the test container and importable without the test env for docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue tracking
Context behind the change
and stored in the DB
follow the same job-type based approach now
src/How has this been tested?
Release plan
Potential risks; What to monitor; Rollback plan