GitCorner

Code

ossartcraftfilmcraft🌐
f67d38b35cc5
README.mdRendered Markdown

filmcraft-audio-dsp

Real-time audio DSP for FilmCraft. Layer L1, no dependencies, compiles for wasm32-unknown-unknown, no unsafe. Clean-room: written from the public standards (ITU-R BS.1770-4, EBU R128 / Tech 3341 / Tech 3342) and textbook DSP (RBJ cookbook biquads, Householder FDN, phase vocoder, spectral subtraction with the statistical late-reverberation model).

Stability

This crate is internal to the FilmCraft workspace (layer L1, see docs/architecture.md) and is not published. Its public items exist for the other workspace crates (engine, render, export, tts, ui-egui) only and may change or be removed without notice; there is no supported use from outside the workspace and no deprecation policy. Items no workspace crate uses are pub(crate) or removed (the effect implementation modules, fft, oversample, Smoothed, db_to_gain); remix::test_music is behind the test-util feature for dependent crates' tests.

Conventions

  • Planar f32 audio: &mut [&mut [f32]], one slice per channel, equal lengths.
  • Effects are built (allocating) with (sample_rate, channels); process, set_param and reset never allocate.
  • Parameters change between blocks and are smoothed per sample; output does not depend on how the stream is cut into blocks (tested for every effect with 0/1/random-size blocks).
  • Recursive state is flushed below ~1e-30, so no denormals without CPU FTZ flags.

Loudness (loudness)

LoudnessMeter::new(sample_rate, channels) → process(&[&[f32]]) / process_interleaved, then momentary(), short_term(), integrated(), loudness_range(), sample_peak(), true_peak() / true_peak_dbtp(), summary().

  • K-weighting derived for any sample rate by bilinear transform (matches the 48 kHz BS.1770 table).
  • Gated integrated loudness (−70 LUFS / −10 LU) and LRA (−70 / −20 LU, 10th–95th percentile) use a 0.01 LU histogram with exact energy sums: fixed memory for programmes of any length.
  • True peak: 4× (2× at 88.2/96 kHz) polyphase Kaiser-sinc interpolation.
  • Oracle-tested against ffmpeg's ebur128 filter (tests/loudness_oracle.rs, skipped without ffmpeg): M/S within 0.0005 LU, I within 0.007 LU, LRA within 0.04 LU and true peak within 0.045 dB (0.053 dB of the analytic value on an fs/4 tone) on sine, pink-noise, speech-like and gated stereo signals at 44.1/48/96 kHz.
  • normalize_gain_db(measured, target) and normalize_gain_db_peak_limited(...) for Normalize / Essential Sound loudness auto-match.

Effects (effects)

All implement AudioEffect (id, params, set_param, param, reset, process, latency). effects() lists EffectInfo { id, name, category, params, create }; each ParamSpec has id, name, range, default, Unit, log-scale hint and choice labels so UIs can be generated.

id effect
parametric_eq 8 bands, each peaking / shelf / LP / HP / notch / band-pass, RBJ biquads in TDF-II
simple_eq low shelf, mid peak, high shelf
compressor peak/RMS detector, soft knee, attack/release, make-up; stereo-linked
gate downward expander / gate with range and hold
limiter 5 ms look-ahead, true-peak-aware, ceiling guaranteed (latency reported)
delay feedback delay, ms or tempo-synced divisions, mix
reverb 8×8 Householder FDN, pre-delay, RT60 decay, HF damping, size, mix
amplify, channel_volume, balance gain, per-channel gain, balance / constant-power pan
invert, swap_channels, fill_left, fill_right polarity and routing
dehum 50/60 Hz notch comb with up to 10 harmonics
denoise STFT spectral gating, minimum-statistics noise floor (latency = FFT size)
pitch_shifter ±12 semitones, phase vocoder with peak shifting + phase locking (latency = FFT size)
deesser high-pass sidechain at frequency (2–12 kHz) detects sibilance relative to the broadband level (threshold); a dynamic high shelf cuts the band by up to reduction dB (1 ms / 60 ms, stereo-linked); exact identity when idle
dereverb STFT late-reverb suppression: λ_r(t) = e^{−2Δ·T_d}·λ_x(t − T_d) (Δ = 3 ln10 / rt60, T_d ≈ 50 ms), over-subtraction 1 + amount, gain floor down to −18 dB, smoothed gains (latency = FFT size)
speech_enhance Enhance Speech DSP chain: 80 Hz HPF, de-mud cut (250 / 350 Hz −3 dB), presence (2.5 / 4 kHz +4 dB) and air (+2 dB @ 10 kHz) boosts by tone, downward expander below −50 dBFS (3:1, ≤ 18 dB), 3:1 soft-knee compressor above −24 dBFS, +3 dB make-up, mix
stereo_width mid/side width 0–200 % (100 % = identity, 0 % = mono)
graphic_eq_10/20/30 constant-Q peaking bands at octave / half-octave / third-octave ISO centres (Q from the bandwidth), ±24 dB, master gain
parametric_eq_full Premiere's Parametric Equalizer: master gain, high-pass and low-pass (12–48 dB/oct Butterworth cascades), low shelf, 5 peaking bands, high shelf
notch_filter 6 constant-bandwidth cuts (FilterType::Cut = 1 + (g − 1)·BP: the width stays f/Q however deep), Narrow / Very Narrow / Super Narrow
scientific_filter Bessel / Butterworth / Chebyshev I / elliptic, LP / HP / BP / BS, order 1–12 (design: analog prototype → frequency transform at pre-warped edges → bilinear → biquads; elliptic via Landen-transformation Jacobi functions after Orfanidis)
fft_filter 8-point gain curve (log-frequency, linear or smoothstep), applied zero-phase by √Hann STFT at 75 % overlap (latency = FFT size)
dynamics_rack Dynamics: auto gate (hold) → expander → compressor (auto make-up) → limiter (soft clip), stereo-linked; transfer_db gives the static curve
multiband_compressor 4 bands with LR4 crossovers and all-pass phase compensation (bands sum to an all-pass: flat magnitude), per-band threshold/ratio/attack/release/gain/solo/bypass, output limiter, channel link
tube_compressor RMS detector, 10 dB soft knee, program-dependent release, gentle tanh(x/2)·2 saturation
chorus_flanger chorus (3 voices, 15–35 ms) or flanger (0.3–5 ms, feedback), transience emphasis, mix
flanger initial/final delay sweep, stereo phasing, ±feedback, inverted / special / sinusoidal modes
phaser 2–12 swept first-order all-passes, depth (octaves below the upper frequency), feedback, stereo phase difference
analog_delay Tape / Tape+Tube / Analog: low-passed, saturating feedback (bounded up to 200 %), trash, stereo spread
multitap_delay 4 independent feedback taps, levels, mix
convolution_reverb uniformly partitioned overlap-save convolution (256-sample partitions, latency 256) with 7 impulses generated here (noise with frequency-dependent exponential decay + early reflections, unit energy); room size (resampling), LF/HF damping, pre-delay, width
surround_reverb early reflections + FDN late field, centre input, low/high cut, width, dry/wet (stereo rendering)
click_remover LPC residual outlier detection (k·σ, σ from the median), AR forward/backward interpolation of ≤ 64-sample regions (latency 640)
channel_mixer, stereo_expander, mute 2×2 matrix with inversion; mid/side expand + centre pan; ramped mute
distortion, guitar_suite waveshaper (soft/hard/tube/foldback, asymmetry, tone); compressor → distortion → amp/cabinet voicing → filter
mastering, vocal_enhancer EQ → reverb → exciter → widener → loudness maximiser (transparent at defaults); fixed Male/Female/Music voicings
binauralizer ±angle virtual speakers through a spherical-head model (Woodworth ITD + Brown–Duda head shadow)
ambisonics_panner rotates a ±30° source pair (pan/tilt/roll) and re-pans to stereo; identity at 0/0/0
loudness_meter pass-through BS.1770 meter (LoudnessMeterFx::meter)

AudioEffect::response_db(freq) (EQs and filters) and transfer_db(band, input_db) (dynamics) expose the analytic curves for the graphical effect editors. Modulation effects start their LFO at the beginning of a chain, so a random-access render may differ in LFO phase from continuous playback.

effects::premiere_tests checks every effect: neutral settings are (delayed) identity, output is deterministic and latency constant (impulse lands at latency()), plus a level / frequency-response test on generated tones, noise or impulses and a realtime-factor floor.

Essential Sound builds on these (Repair: denoise, high-pass via parametric_eq, dehum, deesser, dereverb; Clarity: compressor, simple_eq, speech_enhance; Creative: reverb, stereo_width).

Enhance Speech decision. No machine-learning model is bundled: we found no openly licensed speech-enhancement model that is also practical to ship here (pure Rust, wasm, asset rules). DeepFilterNet (MIT/Apache-2.0, with a Rust implementation on tract) is the candidate for a future optional integration behind a trait outside this dependency-free crate, once its weights' licence and size are vetted as an asset. Until then speech_enhance is the DSP chain above.

Measured on synthetic signals (effects::essential::tests, 48 kHz):

Effect Signal Result
deesser 150 Hz harmonic voice + 5–10 kHz noise bursts sibilance −5.1 dB (8 dB max) / −9.8 dB (16 dB max) during bursts; voice band ±0.01 dB; non-sibilant passages ±0.001 dB
dereverb (100 %) harmonic syllables (100 ms / 400 ms gaps) + Polack-model late reverb, RT60 0.8 s tail-to-burst energy −5.0 → −10.1 dB (5.0 dB better); burst energy −1.7 dB
speech_enhance voice + fricative noise syllables over −60 dBFS noise presence/mud tilt +5.5 dB; noise between syllables −8.7 dB; overall level +1.0 dB
Dialogue chain denoise + parametric_eq (2 HP bands) + dehum + deesser + dereverb + compressor + simple_eq + speech_enhance + reverb, 60 s stereo 1.36 s = 44× realtime on one core (release, Apple Silicon; cargo test --release -p filmcraft-audio-dsp dialogue_chain_realtime -- --ignored --nocapture)

Ducking analysis (ducking)

Pure functions for Essential Sound auto-ducking:

  • envelope_db(channels, sample_rate, hop_s, window_s): centred-window RMS level per hop (dBFS, channel-averaged power, floor −120).
  • activity(env_db, hop_s, threshold_db, min_on_s, bridge_s): regions above the threshold; gaps shorter than bridge_s are merged first, then regions shorter than min_on_s are dropped.
  • sensitivity_threshold_db(s): Sensitivity 0–10 → −20 − 4·s dBFS.
  • duck_keyframes(regions, clip_start, clip_end, base_db, reduce_db, fade_s): (time, dB) volume keyframes — ramp down over fade_s before each region, hold, ramp up after; overlapping ramps merge; clamped to the clip (a ramp cut by an edge gets the ramp's value at the edge); strictly increasing times.

Audio synchronisation (sync)

find_offset(a, b, sample_rate, &SyncOptions) returns the lag L with b[n] ≈ g·a[n + L], a sub-sample estimate, and two trust figures (confidence = peak / RMS of the correlation, distinct = peak / next peak > 50 ms away; reliable() = ≥ 8 and ≥ 1.5).

  • Coarse: DC removal + normalisation, Blackman-windowed-sinc decimation to ≤ 8 kHz (further for long recordings, so the transform stays ≤ 4 M points), GCC-PHAT-β (R = A·B*/|A·B*|^0.75, 40 Hz – 0.4·rate band) with both signals packed into one complex FFT.
  • Refine: the same correlation at the full rate on the loudest common window (≤ 2¹⁷ samples), ±2 coarse steps around the coarse peak, parabolic interpolation. The highest peak there, of either sign, sets the lag and inverted.
  • Tested within one sample on speech-like multi-microphone signals (gains −20…+10 dB, SNR down to 3 dB, coloured microphones, fractional delays: worst 0.4 samples integer / 0.11 samples sub-sample), a 0 dB SNR room with a strong echo, inverted polarity, unrelated signals (unreliable) and max_lag. Two 10-minute recordings: 2.4 s (release, one core).
  • A band-limited, echoey scratch track against the clean recording keeps its polarity, either way round, and lands within 4 samples (its filters delay it by about 3).

Time stretch (stretch)

Wsola::new(sample_rate) time-stretches without changing the pitch (Speed/Duration ▸ Maintain Audio Pitch): 30 ms raised-cosine grains overlapped by half (15 ms hops). Each grain is read near its nominal input position, moved by at most 8 ms to where its first hop best matches the previous grain's natural continuation (normalised cross-correlation of a mono guide, coarse on every other sample and offset, then refined), so the crossfaded halves are in phase. align picks the position and overlap makes one output hop; the caller keeps the chain of positions and reads the input (the renderer keeps it per reader of a clip, see filmcraft-render's stretch). The renderer builds the guide only from the source channels the clip plays and reads grains only from the clip's own source span (in to out point; outside it reads silence), so a handle or an unplayed channel never sounds or steers the alignment. Tested on tones from 110 Hz to 1 kHz at 50 %–300 %: pitch within 2 %, level within 0.03 RMS, no steps larger than the tone's own slope; and on noise, where a read cut into blocks, or interleaved with another reader of the same clip, gives the same samples as one continuous read.

Why not the shuttle's PitchShifter (ui-egui's shuttle runs it ahead of a StreamResampler to keep pitch at J/L speeds)? That pair is a stream: it has latency (the FFT size) and state carried from one block to the next, so it only gives the right samples when it is fed every sample in order from where it started. Clip audio is read by random access: playback, export, scrubbing, waveforms and loudness matching each ask for any span of the timeline. A seek would have to rebuild the stream and pre-roll its latency, and every reader of a clip would need a stream of its own. Wsola keeps no state: grain k's position follows only from grain k - 1's, and the renderer keeps the last positions of each chain, keyed by where the chain's last read ended, so a read in order continues its own chain exactly (however the reads are cut, and whatever other readers of the clip do in between), and a read anywhere else starts a new chain a few grains early (see Known limits for what that means for the samples). A phase vocoder also smears transients over its FFT window, which matters for dialogue and music cut at speed; WSOLA copies the input's own waveform, so a transient that is heard keeps its shape (see Known limits for which ones are not heard, or are heard twice).

Known limits

  • Effects assume ≤ the channel count they were built with; extra channels pass through (channel-generic gain effects process all of them).
  • DeNoise adapts continuously; a perfectly stationary tone is (by design) treated as noise after ~1.5 s. There is no explicit "learn noise print" mode yet.
  • Filter type / choice changes switch immediately (continuous parameters are smoothed).
  • Time stretch (Maintain Audio Pitch): a read that starts a new chain (a seek, or a clip whose chain was dropped from the renderer's cache) chooses its grains from its own start, so the same span can come out slightly different (grain positions within the 8 ms tolerance) from one that continued a chain from earlier. Pitch and level are the same; the samples are not, so two reads of a span through different histories, such as an export that starts where playback stopped, are not bit-identical. Making them so needs every chain to be canonical (computed from the clip's start, or reset at fixed grains with a seam), which is not done.
  • Time stretch, faster than 1×: grains are read every speed × 15 ms of input but each plays only 30 ms, so above 2× part of the input between grains is never heard (a third of it at 3×). A click or consonant shorter than the gap can vanish.
  • Time stretch, slower than 1×: grains overlap more in the input than in the output, so a short transient can be heard more than once, a few milliseconds apart (a flam rather than a click).
  • Time stretch at a clip's edges: grains read silence outside the clip's source span, so the first and last hop (≤ 15 ms) fade against silence instead of continuing into the handle.
  • Time stretch: a single read whose source span exceeds 2^24 samples (a very fast clip read in one long request) is not stretched; the renderer resamples it instead (varispeed, so the pitch changes for that read).