diff --git a/README.md b/README.md index f6c638c..4d44382 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,62 @@ -# MPATE-UE-1113 — Music Technology Coursework +# MPATE-UE-1113 Final Project — Does AI-Generated Cantopop Respect Cantonese Tone? +by Candy Xie -My personal work for NYU's *Music Technology* course (Spring 2026). Python-based audio analysis, music information retrieval, and clustering — built around `librosa`, `music21`, `numpy`, and `scikit-learn`. +A tone-tune mapping audit comparing **human-sung Cantopop** against **Suno-generated Cantopop** to test whether AI music generation respects Cantonese's six-tone system. + +## Question + +Cantonese is a tone language: pitch direction between adjacent syllables has to roughly match the lexical tonal direction, or the lyric becomes unintelligible. Wong & Diehl (2002) showed human Cantopop respects this rule **75–92%** of the time. **Does Suno?** + +## Method + +1. **Citation tones** — pick one 7–14 syllable snippet per track (no 變調 sandhi, no English loanwords, no proper nouns). Hand-annotate each character's Cantonese tone (1–6) and reduce to a 3-level target (High / Mid / Low) following Wong & Diehl's tone-ending mapping. +2. **F0 extraction** — auto-segment the snippet into syllables (voiced-region detection on Parselmouth's pitch contour), then use [Parselmouth](https://parselmouth.readthedocs.io/) (Python wrapper for Praat) to compute the median F0 per syllable. +3. **Direction comparison** — for every adjacent syllable pair, compare *expected* tonal direction (up/flat/down) against *actual* F0 direction. Count violations. +4. **Validate** — confirm the human songs land in Wong & Diehl's 75–92% band; read off Suno's match rate against the same baseline. + +## Dataset + +Three human/Suno pairs matched by tempo. Audio in `assignments/06-final-project/audio/`. + +| Tempo | Human (track, artist) | Suno (generated, V5.5) | +| -------- | ---------------------------------- | ----------------------------------- | +| Mid | 隔離 — Jace Chan (2023) | 玻璃 | +| Ballad | 高山低谷 — Phil Lam 林奕匡 (2014) | 雨窗一封 | +| Uptempo | 紅日 — Hacken Lee 李克勤 (1992) | 旺角快車 *(mis-rendered, see note)* | + +> **Note on the uptempo Suno track.** The track was prompted as 旺角快車 but Suno hallucinated lyrics from a different prompt (雨窗一封). The audit scores what Suno actually sang — itself an instructive failure mode. + +## Files + +- `assignments/06-final-project/cantonese_tone_audit.py` — core framework: tone mapping, F0 extraction, violation counting +- `assignments/06-final-project/audit_driver.py` — runs the audit across all 6 tracks, emits the comparison figure +- `assignments/06-final-project/audio/` — 3 human + 3 Suno mp3s +- `assignments/06-final-project/figures/human_vs_suno.png` — bar chart of match rate vs Wong & Diehl baseline + +## Run + +```bash +pip install praat-parselmouth librosa numpy pandas matplotlib +cd assignments/06-final-project +python audit_driver.py +``` + +Outputs: +- `figures/human_vs_suno.png` — bar chart of match rate per tempo bucket vs the Wong & Diehl baseline band +- per-snippet expected-direction tables and pair-by-pair violation reports to stdout +- top Suno violations grouped by track (b-roll material for the project video) + +The CJK glyphs in matplotlib labels rely on a system CJK font (PingFang / Heiti / Hiragino — all preinstalled on macOS). On Linux, install `fonts-noto-cjk` and add `'Noto Sans CJK TC'` to `matplotlib.rcParams['font.family']` near the top of `audit_driver.py`. + +## Findings + +See `figures/human_vs_suno.png` and the violation tables in the audit's stdout. Discussion and concrete failure pairs are walked through in the accompanying video. + +--- + +# MPATE-UE-1113 Coursework + +My personal work for NYU's *Music, Mind and Artificial Intelligence* course (Spring 2026). Python-based audio analysis, music information retrieval, and clustering — built around `librosa`, `music21`, `numpy`, and `scikit-learn`. ## Contents @@ -8,12 +64,13 @@ My personal work for NYU's *Music Technology* course (Spring 2026). Python-based - `01-python-basics` · `02-hello` · `03-leap-year` — warm-ups - `04-sine-wave` · `05-sine-square-saw` · `06-play-audio` — synthesis & playback - `07-librosa-tempo` — tempo / beat tracking with librosa -- **`assignments/`** — Five graded assignments +- **`assignments/`** — Graded assignments - `01-written-1+2` — written responses - `02-librosa` — audio feature extraction - `03-clustering` — clustering songs by audio features - `04-playlisting` — playlist generation - - `05-project-2` — final project + - `05-project-2` — project 2 + - `06-final-project` — final project (Cantonese tone audit, see top of README) - **`demos/`** — Side explorations (drum clustering, music21 analysis) - **`notes/`** — Personal study notes (sine-wave fundamentals) - **`misc/`** — Tempo-extraction tutorial I wrote up while reviewing for an assignment diff --git a/assignments/06-final-project/audio/gou-san-suno.mp3 b/assignments/06-final-project/audio/gou-san-suno.mp3 new file mode 100644 index 0000000..4f23fc5 Binary files /dev/null and b/assignments/06-final-project/audio/gou-san-suno.mp3 differ diff --git a/assignments/06-final-project/audio/gou-san.mp3 b/assignments/06-final-project/audio/gou-san.mp3 new file mode 100644 index 0000000..f32e31f Binary files /dev/null and b/assignments/06-final-project/audio/gou-san.mp3 differ diff --git a/assignments/06-final-project/audio/hong-jat-suno.mp3 b/assignments/06-final-project/audio/hong-jat-suno.mp3 new file mode 100644 index 0000000..9ee0dd0 Binary files /dev/null and b/assignments/06-final-project/audio/hong-jat-suno.mp3 differ diff --git a/assignments/06-final-project/audio/hong-jat.mp3 b/assignments/06-final-project/audio/hong-jat.mp3 new file mode 100644 index 0000000..5a8ad12 Binary files /dev/null and b/assignments/06-final-project/audio/hong-jat.mp3 differ diff --git a/assignments/06-final-project/audio/jace_gelei-suno.mp3 b/assignments/06-final-project/audio/jace_gelei-suno.mp3 new file mode 100644 index 0000000..6214474 Binary files /dev/null and b/assignments/06-final-project/audio/jace_gelei-suno.mp3 differ diff --git a/assignments/06-final-project/audio/jace_gelei.mp3 b/assignments/06-final-project/audio/jace_gelei.mp3 new file mode 100644 index 0000000..fad05a6 Binary files /dev/null and b/assignments/06-final-project/audio/jace_gelei.mp3 differ diff --git a/assignments/06-final-project/audit_driver.py b/assignments/06-final-project/audit_driver.py new file mode 100644 index 0000000..fa03851 --- /dev/null +++ b/assignments/06-final-project/audit_driver.py @@ -0,0 +1,355 @@ +""" +audit_driver.py + +End-to-end driver for the Cantopop tone audit project. +Run cell-by-cell in VS Code (the `# %%` markers create interactive cells) +or as a script: python3 audit_driver.py + +PIPELINE: + 1. Define dataset (3 human + 3 Suno snippets, matched by tempo). + 2. Manually annotate syllable timings per audio file (Audacity). + 3. Run F0 extraction + tone-tune audit per snippet. + 4. Aggregate human vs Suno violation rates by tempo bucket. + 5. Plot results for the video. +""" + +# %% [markdown] +# # Cantonese Tone Audit: Suno vs. Human Composers +# +# **Thesis.** Human Cantopop composers align melodic direction with +# Cantonese tonal transitions at a measurable rate (~75-92% per Wong & +# Diehl 2002, Lo 2013). I tested whether Suno respects the same constraint +# in its own original Cantopop output. +# +# **Method.** 3 human Cantopop songs paired with 3 Suno-generated tracks +# matched by tempo (mid / ballad / uptempo). Suno wrote its own lyrics +# from style prompts; I picked one 7-10 syllable snippet from each. +# Same audit framework applied to all 6 snippets. + +# %% imports +import numpy as np +import matplotlib +import matplotlib.pyplot as plt +import pandas as pd + +# Use a CJK-capable font so 中文 labels render instead of empty boxes. +# macOS ships with PingFang and Heiti; fall through to anything that works. +matplotlib.rcParams['font.family'] = [ + 'PingFang HK', 'PingFang TC', 'PingFang SC', + 'Heiti TC', 'Heiti SC', 'Hiragino Sans GB', + 'Arial Unicode MS', 'sans-serif', +] +matplotlib.rcParams['axes.unicode_minus'] = False +from cantonese_tone_audit import ( + analyze_snippet, + extract_syllable_f0, + auto_segment_syllables, + print_report, + expected_direction, + TONE_LEVEL, +) + +# %% [markdown] +# ## 1. Dataset (verified) +# +# Each snippet: +# - 7-10 syllables +# - No 變調 (tone change), no English loanwords, no proper nouns +# - All citation tones cross-checked + +# %% snippet definitions + +# ---- HUMAN: 隔離 (Jace Chan, 2023) — MID-TEMPO ---- +# Line: 想約他 私訊他 別無視我 (10 syllables) +H_GE_LEI = { + 'song': '隔離', + 'artist': 'Jace Chan', + 'source': 'human', + 'tempo': 'mid', + 'lyrics': ['想','約','他','私','訊','他','別','無','視','我'], + 'jyutping': ['soeng2','joek3','taa1','si1','seon3','taa1', + 'bit6','mou4','si6','ngo5'], + 'tones': [2, 3, 1, 1, 3, 1, 6, 4, 6, 5], + 'snippet_range': (25, 30), + 'syllable_times': [], + 'audio_path': 'audio/jace_gelei.mp3', + 'pitch_floor': 100, 'pitch_ceiling': 500, +} + +# ---- SUNO MID: 玻璃 chorus "留低好嗎 留低一晚" (8 syllables) ---- +# (audio file is named jace_gelei-suno.mp3 but contains 玻璃 generation) +# Caveat: 嗎 (maa3) is a Mandarin-loaned sentence particle. Native +# Cantonese question marker would be 啊/咩. Including this as-is. +S_SUNO_MID = { + 'song': '玻璃 (Suno)', + 'artist': 'Suno V5.5', + 'source': 'suno', + 'tempo': 'mid', + 'lyrics': ['留','低','好','嗎','留','低','一','晚'], + 'jyutping': ['lau4','dai1','hou2','maa3','lau4','dai1','jat1','maan5'], + 'tones': [4, 1, 2, 3, 4, 1, 1, 5], + 'snippet_range': (62, 68), # 1:02-1:08 + 'syllable_times': [], + 'audio_path': 'audio/jace_gelei-suno.mp3', + 'pitch_floor': 70, 'pitch_ceiling': 500, +} + +# ---- HUMAN: 高山低谷 (Phil Lam 林奕匡, 2014) — BALLAD ---- +# Snippet from verse: "你快樂過生活 我拼命去生存" (12 syllables) +# (The other iconic line "我在高山 我在低谷" is the chorus of this same song.) +H_GOU_SAAN = { + 'song': '高山低谷', + 'artist': 'Phil Lam (林奕匡)', + 'source': 'human', + 'tempo': 'ballad', + 'lyrics': ['你','快','樂','過','生','活','我','拼','命','去','生','存'], + 'jyutping': ['nei5','faai3','lok6','gwo3','sang1','wut6', + 'ngo5','ping3','ming6','heoi3','sang1','cyun4'], + 'tones': [5, 3, 6, 3, 1, 6, 5, 3, 6, 3, 1, 4], + 'snippet_range': (138, 144), # 2:18-2:24 + 'syllable_times': [], + 'audio_path': 'audio/gou-san.mp3', + 'pitch_floor': 70, 'pitch_ceiling': 500, # Phil Lam = male vocal +} + +# ---- SUNO BALLAD: 雨窗一封 pre-chorus +# "如果我肯早啲講 會唔會唔同結果" (14 syllables) ---- +# Caveat: 會 has wui5/wui6 ambiguity. Committing to wui5 (T5/M) as the +# modern Cantonese auxiliary reading. +S_SUNO_BALLAD = { + 'song': '雨窗一封 (Suno)', + 'artist': 'Suno V5.5', + 'source': 'suno', + 'tempo': 'ballad', + 'lyrics': ['如','果','我','肯','早','啲','講', + '會','唔','會','唔','同','結','果'], + 'jyutping': ['jyu4','gwo2','ngo5','hang2','zou2','di1','gong2', + 'wui5','m4','wui5','m4','tung4','git3','gwo2'], + 'tones': [4, 2, 5, 2, 2, 1, 2, 5, 4, 5, 4, 4, 3, 2], + 'snippet_range': (71, 77), # 1:11-1:17 + 'syllable_times': [], + 'audio_path': 'audio/gou-san-suno.mp3', + 'pitch_floor': 70, 'pitch_ceiling': 500, +} + +# ---- HUMAN: 紅日 (Hacken Lee 李克勤, 1992) — UPTEMPO ---- +# Line: 我願能一生永遠陪伴你 (10 syllables) +H_HUNG_JAT = { + 'song': '紅日', + 'artist': 'Hacken Lee (李克勤)', + 'source': 'human', + 'tempo': 'uptempo', + 'lyrics': ['我','願','能','一','生','永','遠','陪','伴','你'], + 'jyutping': ['ngo5','jyun6','nang4','jat1','sang1', + 'wing5','jyun5','pui4','bun6','nei5'], + 'tones': [5, 6, 4, 1, 1, 5, 5, 4, 6, 5], + 'snippet_range': (48, 52), + 'syllable_times': [], + 'audio_path': 'audio/hong-jat.mp3', + 'pitch_floor': 70, 'pitch_ceiling': 500, +} + +# ---- SUNO UPTEMPO: track titled 旺角快車 in Suno, but the audio +# at 1:11-1:14 actually sings lyrics from a DIFFERENT prompt (雨窗一封): +# "喺原地 等一個唔會返嚟的你" (12 syllables). +# This is itself an AI failure mode: Suno hallucinated lyrics that +# weren't in the 旺角快車 prompt. We audit what's actually sung. +# Tone caveats: 的 (dik1) is Mandarin-loaned, 會 wui5/wui6 ambiguity. +S_SUNO_UPTEMPO = { + 'song': '旺角快車 (Suno, mis-rendered)', + 'artist': 'Suno V5.5', + 'source': 'suno', + 'tempo': 'uptempo', + 'lyrics': ['喺','原','地','等','一','個','唔', + '會','返','嚟','的','你'], + 'jyutping': ['hai2','jyun4','dei6','dang2','jat1','go3','m4', + 'wui5','faan1','lai4','dik1','nei5'], + 'tones': [2, 4, 6, 2, 1, 3, 4, 5, 1, 4, 1, 5], + 'snippet_range': (71, 74), # 1:11-1:14 + 'syllable_times': [], + 'audio_path': 'audio/hong-jat-suno.mp3', + 'pitch_floor': 70, 'pitch_ceiling': 500, +} + +SNIPPETS = [H_GE_LEI, S_SUNO_MID, + H_GOU_SAAN, S_SUNO_BALLAD, + H_HUNG_JAT, S_SUNO_UPTEMPO] + +# %% [markdown] +# ## 2. Sanity check: expected directions (no audio yet) + +# %% expected directions +def expected_directions_table(snippet): + rows = [] + t = snippet['tones'] + lyr = snippet['lyrics'] + for i in range(len(t) - 1): + rows.append({ + 'pair': f"{lyr[i]}->{lyr[i+1]}", + 'tones': f"T{t[i]}->T{t[i+1]}", + 'levels': f"{TONE_LEVEL[t[i]]}->{TONE_LEVEL[t[i+1]]}", + 'expected': expected_direction(t[i], t[i+1]), + }) + return pd.DataFrame(rows) + +for s in SNIPPETS: + print(f"\n--- {s['song']} ({s['tempo']}, {s['source']}) ---") + print(f" syllables: {len(s['tones'])}, transitions: {len(s['tones'])-1}") + print(expected_directions_table(s).to_string(index=False)) + +# %% [markdown] +# ## 3. Find the snippet's time range in each song +# +# **You only need 2 numbers per song.** Open each mp3 in QuickTime +# Player (or any audio player). Scrub until you hear the snippet line. +# Note: +# - **start_sec** = time of the FIRST syllable's onset +# - **end_sec** = time AFTER the LAST syllable ends +# +# Set `snippet_range = (start_sec, end_sec)` in each dict above. +# The code subdivides into syllables automatically using onset detection. +# +# Total effort: ~5 min for 6 songs. +# +# **Tip — scrubbing in QuickTime:** drag the time slider; the time +# display shows mm:ss. Convert to seconds: e.g. 1:23 -> 83 sec. + +# %% [markdown] +# ## 4. Auto-segment + F0 extraction + audit + +# %% run audit +def audit_one(snippet, segment_method='onset', verbose=True): + if snippet.get('snippet_range') is None and not snippet['syllable_times']: + print(f"[skip] {snippet['song']} ({snippet['source']}) " + f"— snippet_range not set") + return None + + # Auto-segment if syllable_times empty + if not snippet['syllable_times']: + start, end = snippet['snippet_range'] + snippet['syllable_times'] = auto_segment_syllables( + snippet['audio_path'], start, end, + n_syllables=len(snippet['tones']), + method=segment_method, + pitch_floor=snippet['pitch_floor'], + pitch_ceiling=snippet['pitch_ceiling'], + ) + if verbose: + print(f" auto-segmented {snippet['song']} ({segment_method}):") + for ch, (a, b) in zip(snippet['lyrics'], snippet['syllable_times']): + print(f" {ch}: {a:.2f}-{b:.2f}s ({(b-a)*1000:.0f}ms)") + + f0 = extract_syllable_f0( + snippet['audio_path'], + snippet['syllable_times'], + pitch_floor=snippet['pitch_floor'], + pitch_ceiling=snippet['pitch_ceiling'], + ) + return analyze_snippet( + tones=snippet['tones'], + f0_values=f0, + lyrics=snippet['lyrics'], + ) + +# Lock segmentation to 'voiced' — most principled for tone work since +# F0 medians are computed on actually-voiced regions of each syllable. +SEGMENT_METHOD = 'voiced' + +results = [] +for s in SNIPPETS: + s['syllable_times'] = [] # reset; auto-segment each run + r = audit_one(s, segment_method=SEGMENT_METHOD, verbose=False) + if r is None: + continue + print_report(r, title=f"{s['song']} — {s['source'].upper()} ({s['tempo']})") + results.append({ + 'song': s['song'], + 'tempo': s['tempo'], + 'source': s['source'], + 'match_rate': r['match_rate'], + 'violation_rate': r['violation_rate'], + 'violations': r['violations'], + 'valid_pairs': r['valid_pairs'], + }) + +results_df = pd.DataFrame(results) +if not results_df.empty: + print("\n=== AGGREGATE ===") + print(results_df.to_string(index=False)) + +# %% [markdown] +# ## 5. Visualization + +# %% bar chart +def plot_audit(df, save_path='figures/human_vs_suno.png'): + import os + os.makedirs(os.path.dirname(save_path), exist_ok=True) + + tempos = ['mid', 'ballad', 'uptempo'] + x = np.arange(len(tempos)) + width = 0.35 + + human = [df[(df['tempo']==t) & (df['source']=='human')]['match_rate'].mean() + for t in tempos] + suno = [df[(df['tempo']==t) & (df['source']=='suno')]['match_rate'].mean() + for t in tempos] + human_labels = [df[(df['tempo']==t) & (df['source']=='human')]['song'].iloc[0] + if len(df[(df['tempo']==t) & (df['source']=='human')]) else '' + for t in tempos] + suno_labels = [df[(df['tempo']==t) & (df['source']=='suno')]['song'].iloc[0] + if len(df[(df['tempo']==t) & (df['source']=='suno')]) else '' + for t in tempos] + + fig, ax = plt.subplots(figsize=(9, 5.5)) + bars_h = ax.bar(x - width/2, human, width, label='Human', color='#2E86AB') + bars_s = ax.bar(x + width/2, suno, width, label='Suno', color='#E63946') + + ax.axhspan(0.75, 0.92, alpha=0.12, color='gray', + label='Wong & Diehl 2002 baseline (75-92%)') + + ax.set_ylabel('Tone-tune match rate', fontsize=12) + ax.set_xticks(x) + ax.set_xticklabels([t.upper() for t in tempos], fontsize=11) + ax.set_ylim(0, 1.05) + ax.set_title('Cantonese tone-tune match rate by tempo: Human vs Suno', + fontsize=13) + ax.legend(loc='lower right', fontsize=10) + ax.grid(axis='y', alpha=0.3) + + for i, (h, s, hl, sl) in enumerate(zip(human, suno, human_labels, suno_labels)): + if not np.isnan(h): + ax.text(i - width/2, h + 0.02, f'{h:.0%}\n{hl}', + ha='center', fontsize=9) + if not np.isnan(s): + ax.text(i + width/2, s + 0.02, f'{s:.0%}\n{sl}', + ha='center', fontsize=9) + + plt.tight_layout() + plt.savefig(save_path, dpi=150) + plt.show() + return fig + +if not results_df.empty: + plot_audit(results_df) + +# %% [markdown] +# ## 6. Violation examples (for video b-roll) + +# %% violations detail +def violation_examples(snippet, max_n=3): + r = audit_one(snippet) + if r is None: + return [] + return [p for p in r['pairs'] if p['violation'] is True][:max_n] + +for s in SNIPPETS: + if s['source'] != 'suno': + continue + bad = violation_examples(s) + if not bad: + continue + print(f"\n--- {s['song']} (Suno) — top violations ---") + for b in bad: + print(f" {b.get('lyrics', b['idx'])}: " + f"expected {b['expected']}, got {b['actual']} " + f"(tones T{b['tones'][0]}-T{b['tones'][1]})") diff --git a/assignments/06-final-project/cantonese_tone_audit.py b/assignments/06-final-project/cantonese_tone_audit.py new file mode 100644 index 0000000..9cd7898 --- /dev/null +++ b/assignments/06-final-project/cantonese_tone_audit.py @@ -0,0 +1,293 @@ +""" +cantonese_tone_audit.py + +Tone-tune mapping audit framework for Cantopop. +Compares expected melodic direction (from citation tones) against +actual melodic direction (from F0 extraction) per adjacent syllable pair. + +Framework: Wong & Diehl (2002) ordinal mapping, 3-level reduction by tone ending. +Validated baseline: ~75-92% match in human Cantopop corpora. + +Author: Candy Xie +Class: MPATE-UE 1113, Final Project, Spring 2026 +""" + +import numpy as np +import parselmouth + + +def load_sound(audio_path): + """ + Load .wav or .mp3 into a parselmouth.Sound. + mp3 goes through librosa to avoid ffmpeg dependency. + """ + if str(audio_path).lower().endswith('.wav'): + return parselmouth.Sound(str(audio_path)) + import librosa + y, sr = librosa.load(str(audio_path), sr=None, mono=True) + return parselmouth.Sound(y, sampling_frequency=sr) + + +# ============================================================ +# 1. TONE TO PITCH LEVEL MAPPING (Wong & Diehl 2002) +# ============================================================ +# Three-level reduction by tone ENDING (target pitch): +# T1 陰平 55 -> H +# T2 陰上 25 -> H (rising target reaches high) +# T3 陰去 33 -> M +# T4 陽平 21 -> L +# T5 陽上 23 -> M (rising target reaches mid) +# T6 陽去 22 -> L +# Entering tones (T7/T8/T9 in 9-tone) collapse onto T1/T3/T6. + +TONE_LEVEL = { + 1: 'H', + 2: 'H', + 3: 'M', + 4: 'L', + 5: 'M', + 6: 'L', + 7: 'H', + 8: 'M', + 9: 'L', +} + +LEVEL_RANK = {'L': 0, 'M': 1, 'H': 2} + + +# ============================================================ +# 2. EXPECTED VS ACTUAL DIRECTION +# ============================================================ + +def expected_direction(tone_a, tone_b): + """Expected melodic direction A -> B from citation tones.""" + a = LEVEL_RANK[TONE_LEVEL[tone_a]] + b = LEVEL_RANK[TONE_LEVEL[tone_b]] + if b > a: + return 'up' + elif b < a: + return 'down' + else: + return 'same' + + +def actual_direction(f0_a, f0_b, threshold_semitones=1.0): + """ + Direction from median F0 of A to B. + 1-semitone threshold filters micro-fluctuations. + """ + if f0_a is None or f0_b is None or f0_a <= 0 or f0_b <= 0: + return None + semitones = 12 * np.log2(f0_b / f0_a) + if semitones > threshold_semitones: + return 'up' + elif semitones < -threshold_semitones: + return 'down' + else: + return 'same' + + +def is_violation(expected, actual, strict=False): + """ + Wong & Diehl 2002 violation rule: + - Hard violation: melody opposite to tone (up vs down). + - Soft cases (level mismatch with same): default ignore. + """ + if actual is None: + return None + if expected == 'up' and actual == 'down': + return True + if expected == 'down' and actual == 'up': + return True + if strict: + if expected == 'same' and actual != 'same': + return True + if expected != 'same' and actual == 'same': + return True + return False + + +# ============================================================ +# 3. F0 EXTRACTION (Parselmouth) +# ============================================================ + +def extract_syllable_f0(audio_path, syllable_times, + pitch_floor=75, pitch_ceiling=600, + time_step=0.01): + """ + Median F0 (Hz) per manually-annotated syllable. + Tune pitch_floor/ceiling per singer: + female: floor=100, ceiling=500 + male: floor=70, ceiling=350 + """ + snd = load_sound(audio_path) + pitch = snd.to_pitch(time_step=time_step, + pitch_floor=pitch_floor, + pitch_ceiling=pitch_ceiling) + + f0_per_syllable = [] + for start, end in syllable_times: + values = [] + t = start + while t < end: + f = pitch.get_value_at_time(t) + if not np.isnan(f) and f > 0: + values.append(f) + t += time_step + f0_per_syllable.append(float(np.median(values)) if values else None) + return f0_per_syllable + + +# ============================================================ +# 3b. AUTO-SEGMENT SYLLABLES (no manual annotation needed) +# ============================================================ + +def auto_segment_syllables(audio_path, snippet_start, snippet_end, + n_syllables, method='onset', + pitch_floor=70, pitch_ceiling=500): + """ + Subdivide a snippet's time range into n_syllables (start, end) tuples. + + method='onset': librosa onset detection within the snippet range, + pick n-1 strongest onsets as boundaries. + method='voiced': parselmouth voicing — split at unvoiced gaps. + method='even': evenly spaced (fallback for legato vocals). + + Returns: list of (start_sec, end_sec) tuples, len = n_syllables. + """ + import librosa + import numpy as np + + if method == 'even': + edges = np.linspace(snippet_start, snippet_end, n_syllables + 1) + return [(edges[i], edges[i+1]) for i in range(n_syllables)] + + if method == 'onset': + y, sr = librosa.load(str(audio_path), sr=None, mono=True, + offset=snippet_start, + duration=snippet_end - snippet_start) + onset_times = librosa.onset.onset_detect( + y=y, sr=sr, units='time', backtrack=True + ) + onset_times = np.array(onset_times) + snippet_start + # need n-1 internal boundaries + needed = n_syllables - 1 + if len(onset_times) >= needed: + # pick most evenly distributed n-1 onsets + idx = np.linspace(0, len(onset_times) - 1, needed).astype(int) + internal = onset_times[idx] + else: + # fall back to even spacing + return auto_segment_syllables(audio_path, snippet_start, + snippet_end, n_syllables, + method='even') + edges = np.concatenate([[snippet_start], internal, [snippet_end]]) + return [(float(edges[i]), float(edges[i+1])) + for i in range(n_syllables)] + + if method == 'voiced': + snd = load_sound(audio_path) + snd_part = snd.extract_part(from_time=snippet_start, + to_time=snippet_end, + preserve_times=True) + pitch = snd_part.to_pitch(time_step=0.01, + pitch_floor=pitch_floor, + pitch_ceiling=pitch_ceiling) + # voiced/unvoiced timeline + times = np.arange(snippet_start, snippet_end, 0.01) + voiced = [] + for t in times: + f = pitch.get_value_at_time(t) + voiced.append(not np.isnan(f) and f > 0) + # find transitions unvoiced -> voiced (syllable starts) + starts = [snippet_start] + for i in range(1, len(voiced)): + if voiced[i] and not voiced[i-1]: + starts.append(times[i]) + starts = sorted(set(starts))[:n_syllables] + if len(starts) < n_syllables: + return auto_segment_syllables(audio_path, snippet_start, + snippet_end, n_syllables, + method='onset') + starts = starts + [snippet_end] + return [(float(starts[i]), float(starts[i+1])) + for i in range(n_syllables)] + + raise ValueError(f"unknown method: {method}") + + +# ============================================================ +# 4. SNIPPET ANALYSIS +# ============================================================ + +def analyze_snippet(tones, f0_values, lyrics=None, + threshold_semitones=1.0, strict=False): + """ + Compute violation rate for one snippet. + + tones: list of citation tones (1-9), one per syllable + f0_values: list of median F0 per syllable, None if unvoiced + lyrics: optional list of characters or jyutping + """ + assert len(tones) == len(f0_values), \ + f"Tone count {len(tones)} != F0 count {len(f0_values)}" + + pairs = [] + violations = 0 + valid_pairs = 0 + + for i in range(len(tones) - 1): + exp = expected_direction(tones[i], tones[i + 1]) + act = actual_direction(f0_values[i], f0_values[i + 1], + threshold_semitones) + viol = is_violation(exp, act, strict=strict) + + pair = { + 'idx': (i, i + 1), + 'tones': (tones[i], tones[i + 1]), + 'levels': (TONE_LEVEL[tones[i]], TONE_LEVEL[tones[i + 1]]), + 'expected': exp, + 'actual': act, + 'violation': viol, + } + if lyrics is not None: + pair['lyrics'] = (lyrics[i], lyrics[i + 1]) + pairs.append(pair) + + if viol is not None: + valid_pairs += 1 + if viol: + violations += 1 + + rate = violations / valid_pairs if valid_pairs else 0.0 + return { + 'violation_rate': rate, + 'match_rate': 1 - rate, + 'violations': violations, + 'valid_pairs': valid_pairs, + 'total_pairs': len(tones) - 1, + 'pairs': pairs, + } + + +# ============================================================ +# 5. PRETTY PRINT +# ============================================================ + +def print_report(result, title="Snippet"): + print(f"\n=== {title} ===") + print(f"Match rate: {result['match_rate']:.1%}") + print(f"Violation rate: {result['violation_rate']:.1%}") + print(f"Valid pairs: {result['valid_pairs']} / {result['total_pairs']}") + print(f"\n{'Pair':<6} {'Tones':<8} {'Levels':<10} " + f"{'Expected':<10} {'Actual':<10} {'Violation'}") + print("-" * 60) + for p in result['pairs']: + viol_str = '!' if p['violation'] else ('-' if p['violation'] is False + else 'skip') + print(f"{p['idx'][0]}-{p['idx'][1]:<4} " + f"T{p['tones'][0]}-T{p['tones'][1]:<5} " + f"{p['levels'][0]}-{p['levels'][1]:<8} " + f"{p['expected']:<10} " + f"{str(p['actual']):<10} " + f"{viol_str}") diff --git a/assignments/06-final-project/figures/human_vs_suno.png b/assignments/06-final-project/figures/human_vs_suno.png new file mode 100644 index 0000000..a7e1de0 Binary files /dev/null and b/assignments/06-final-project/figures/human_vs_suno.png differ