The Listening Coach — Build Plan
The plan for the real thing: a model that hears what Nina hears and maps it to her recovery steps. Distilled from the deep research in research/listening-coach.md (all numbers sourced there). Companion to PRD F15.
The honest feasibility scoreboard
| Capability | Verdict | Why |
|---|---|---|
| Breathiness | ✅ strongest | Best acoustic correlates (CPPS, GNE), best dataset transfer |
| Overall severity | ✅ | Aggregate measures work; ~0.87–0.91 correlation with clinicians published |
| Onset quality (hard/breathy attack) | ✅ | A DSP problem (attack-time proxies), not an ML problem |
| Register breaks | ✅ | F0 discontinuity + spectral-tilt jump detection — DSP |
| Roughness | 🟡 | Humans only agree at α ≈ 0.67–0.79 — that's the ceiling |
| Strain / pressed voice | 🟡 hardest | Hardest for humans too (α ≈ 0.59); ship as 3-level ordinal, never a score |
| Vocal fatigue | 🟡 | Only as within-speaker change over a session/series (~85% with speaker embeddings) |
| Resonance placement (nasal vs back vs mask) | 🔴 R&D | Nasal-vs-oral tractable (~0.68 F1 on Zoom-quality audio); "forward vs back" has zero public labels — Nina-defined from her data only; the moonshot module |
| "Diagnose like Nina" end-to-end | 🔴 as one model · ✅ as two stages | Perceptual heads → an auditable rules/retrieval layer over her documented quality→exercise mappings that she can edit |
The core insight from the research: don't predict "the average clinician" — predict Nina. Training the model to match one named expert ear removes inter-rater noise from the target (listener-dependent modeling is established: LDNet/MBNet). PVQD's 19 raters and Nina train jointly; Nina is the inference identity.
Architecture
student audio (≥1.5s segments, Zoom-quality)
├─ WavLM-Large embeddings (weighted-sum over layers) ← the trunk
├─ + classic acoustics (eGeMAPS, jitter/shimmer/HNR/CPPS) ← same features as our in-browser demos
├─ + channel covariates (codec/SNR/session stats) ← Zoom variation is confound #1
→ multi-task per-dimension heads (breathy/rough/strain/severity)
→ rater-embedding bias (PVQD raters + Nina)
→ DSP sidecars: onset quality, register breaks (no ML)
→ recommendation layer: quality profile → Nina's exercise mappings (rules she can read + edit)
Serving: real-time biofeedback stays in-browser (already shipped); the perceptual model runs server-side after a take; distill a small student model later for offline.
Data plan
- Bootstrap: PVQD (free, 296 recordings, per-dimension CAPE-V + GRBAS ratings from 19 experts — the only real per-dimension target set). Reproduce the published recipe (~0.85+ aggregate correlation) before touching her data. SVD/AVFAD for pretraining; VocalSet for singing techniques (its breathy/belt/fry labels map onto her vocabulary). Bridge2AI's open release is derived-features-only — can't train the SSL model; raw-audio path is realistically closed to us.
- Her recordings: transcript mining as a retrieval engine, not a label source. WhisperX + diarization with Nina's voice enrolled (she's in every recording — a real advantage) → find every moment she gives feedback ("that was breathy", "you're pushing", "more mask") → LLM-normalize to a fixed taxonomy → surface the preceding student audio as candidate clips. Three reasons this can't label alone: feedback follows (and overlaps) the phrase; her language is prescriptive, not descriptive; she only comments when something's wrong (no clean negatives).
- Her labels: a custom VAS rater page (waveform + per-dimension 100-point sliders + "unsure" + blind duplicate injection — existing tools don't do this well; ~2 days to build, likely a lab page). Seed set: 300 stratified clips, 15% blind duplicates to measure her self-consistency (that number becomes the model's ceiling). Then an active-learning loop: the model asks for the ~100 most informative clips per week of her time.
- Consent first, always: per-student opt-in ledger before any lesson audio enters the pipeline; two-party-consent states in mind; originals archived lossless.
Timeline & cost
- First 30 days (starts when the Drive folder lands): consent audit → WhisperX+diarization QA on 10 sessions → student-segment extraction with per-session channel stats → transcript mining → VAS rater page → Nina's 300-clip seed → frozen-WavLM baseline with speaker-disjoint validation, reported per-dimension against her own ceiling.
- 6 months: wk6 PVQD-pretrained multi-task head (~0.85 aggregate) · wk10 fine-tuned on ~800 Nina clips, breathiness CCC ≥ 0.70 · wk16 active-learning to ~2,000 clips, strain 3-class F1 ≥ 0.65 · wk20 nasality + onset/register DSP modules · wk24 recommendation layer + distilled student model.
- Compute: under ~$500 total (A100 rentals at ~$1–1.5/hr; frozen-trunk runs are minutes). Nina's labeling time is the scarce resource — the whole design optimizes for it.
The validation that earns the claim
Hold out 120+ clips the model never saw. Nina rates them cold; 20% re-rated weeks later (her intra-rater ICC = the ceiling). Two outside SLPs/coaches rate the same set. The shippable claim is not a correlation number — it's "the model agrees with Nina as well as another expert does." Pass bar: model-Nina ICC ≥ 0.9× Nina-Nina ICC on breathiness + severity. Plus top-1/top-3 agreement of exercise recommendations against her real session notes.
What this unlocks in the product
Phase 2's baseline-relative biofeedback grows into: post-take quality profiles on practice uploads → trend lines a provider can act on (RTM-grade "therapeutic data") → the triage nurse gaining ears ("hum for me… I hear a bit of breathiness — let's start there") — each step shipping only when it beats the validation bar, and always as wellness feedback under the scope-of-practice line.