videoskilletnotes Open the app ↗
On this page

Optimizations

Why the render path looks the way it does. ARCHITECTURE.md draws the path itself, and the FAQ is the one-page version of what a pass is.

One frame is 477,750 f32 samples (910 × 525) through up to twenty-four compute dispatches, sixty times a second, and six of those passes are FIR filters 33 to 55 taps wide. That budget is the reason for everything below. Every built-in preset lands 3.3–5.4 ms against a 3.3 ms always-on floor, and the headroom exists because the expensive things are gated, tiled, tiered, or not dispatched at all.

How to take a measurement is DEVELOPMENT.md › Measuring performance. This page is what the measurements decided. Every millisecond here was taken on the dev box — a WX 3200 under Firefox Nightly / Linux — best-of over interleaved runs, in August 2026, except where a section says otherwise. One machine on one day: read them for their ratios and their signs rather than their third digit, and re-derive before building on one.

What a frame costs#

In order of what actually moves the number:

The keyer, the synth and the strobe sit behind uniform branches, and the branches hold: the whole feature set against its parent revision lands 4.52 ms both sides at stock, with no separable difference. Engaged, the chroma keyer costs ~0.07 ms, synthOver ~0.01 ms for a full videoSynth per pixel, and the strobe nothing at all — a uniform multiply in decode, ON and OFF both 4.55 ms.

The rule: ablate before you optimize#

Delete the thing and measure the frame without it. That number is the upper bound on any optimization of it, and it is usually smaller than it looks.

The rule exists because three plausible optimizations were built here and measured exactly flat: the filter bank moved from a storage buffer to a uniform, a Chebyshev recurrence replacing the heterodyne phasor walk in under_down and channel, and a one-shot bake of crt_face's grain field. The FIR passes are not ALU-bound on this hardware, so arithmetic saved inside them rides in idle slots — which is not knowable from reading the shader, hence ADR 0007 rather than a comment.

The grain bake has since been re-measured and landed: per-pass timestamps put the hashing at 0.115 ms of a 2.29 ms stock frame, a tenth of a millisecond whole-frame wall clock could not separate from its own bimodal noise. The verdict stands where it was drawn, inside the FIR loops; crt_face is a per-pixel ALU pass, and there the arithmetic was the cost.

What per-pass timestamps found#

scripts/gpuprof times each pass on the GPU's own counter, headless. Its first pass over the stock frame, GPU ms:

changepassbeforeafterframe
RGB→YUV folded into encodeComposite (source A)encodeYuv0.23—2.29 → 1.94
encodeComposite0.220.16
phosphor grain baked once (grain_bake.wgsl)crtFace0.570.451.94 → 1.82
feedback camera gathered only while patched incompose0.1240.0711.82 → 1.77
halation tiered to 8 taps below crtHalation 0.2crtFace0.450.331.77 → 1.66

Confirmed in the app with perf.mjs: stock went 4.17 → 2.88 ms/frame best-of. That is more than the profiler's 0.63 ms of GPU time, which says the 7.6 MB buffer round trip cost the browser's queue more than it cost the card. The first three are bit-exact; the fourth is the bloom bargain, below. Every one is of the form do less — a pass not dispatched, a buffer not written, a gather not run, a fetch not made.

A second pass went after the looks rather than the floor, ranking the hot passes over a sweep of every preset. Each fix is bit-exact except where it says otherwise.

changelookpassbeforeafter
B's three encoders read its texel; encode_yuv.wgsl deleteddirtyMix + PiPframe2.802.32
colour-under tile staged behind the Y/C delay (was 55 storage taps, a deviate each)colourLate + chroma noisechannel2.990.62
FM fold's decay tabled per workgroupfmFoldchannel1.431.23
flywheel and HV sag walked in two wavesfullCollapsesync0.310.14
gun cutoff + gamma applied where decode writes the screen (not exact)lightThatStayscrtFace1.750.74
misconvergedcrtFace1.430.67
nightMonitorcrtFace1.390.59
saturation applied to each gather's result, not each tapstockcrtFace0.360.32

The gamma one is the trade. crt_face was running the beam transfer per tap — three pow a tap across spot, bloom and halation, a millisecond a frame on a look with all of them up. decode now applies cutoff and gamma as it writes outTex, sRGB-encoded while they are active so the byte keeps fine steps where gamma pushes the light (an rgba16float store was tried first and halved the sampler's rate, giving back most of the win). Stock is bit-exact; gamma looks change at hard edges, where a tap between two pixels now interpolates light rather than drive — lightThatStays mean 0.13/255, misconverged mean 0.43 concentrated in the SVM notch. Invisible at 1×, and a millisecond. Confirmed in Firefox with perf.mjs: stock 2.88 → 2.50 ms/frame, lightThatStays 4.18 → 2.57, colourLate 3.03 → 2.56, fullCollapse 4.18 → 2.82.

Three arms were reverted, and all three are worth knowing.

Skipping dispatches is the largest optimization here#

Every optional pass carries a when() predicate over the controls, and renderFrame skips the whole compute pass — encoder, bind group and dispatch — when it returns false. An idle feature costs nothing rather than costing a uniform multiply by zero.

What that is worth, measured against the parent revision with two dev servers alternating: the chroma keyer, the video synth and the blanking strobe together land 4.52 ms on both sides at stock — no separable difference at all. The true-waveform B chain (encodeChromaB → encodeCompositeB → mixB) totals ~0.9 ms engaged and dispatches nothing idle. The intercarrier-buzz readback is gated on the buzz being audible — which now takes a switch as well as a level, since the sound is off until asked for — so a listener who never touches it pays neither the copy nor the map.

Gating inside a shader gets the same treatment when a pass is already running. crt_face's bloom and halation used to share one gate and one loop body, so either control being up ran both radii — a look with bloom up and halation down paid a sixteen-tap gather over a 15-pixel disk for a result immediately multiplied by zero. Splitting them into a loop each, gated separately, is worth 0.12 ms of a 4.87 ms frame. compose had the same shape with the feedback camera: the seven-tap lens gather, the vignette and the sensor curve ran for every pixel and were mixed in at fbMix 0 — gated, 0.124 → 0.071 ms at stock.

The largest single one was a pass that only existed to hand a buffer to the next pass. encode_yuv wrote a vec4 per sample (7.6 MB a frame) that encode_composite read one dispatch later; the texel it came from is a quarter the bytes and already what the FIR's staging loop wanted, so encodeComposite now reads the picture itself — 0.35 ms of a 2.29 ms stock frame for the pass and the read it fed, bit-exact. Source B's three consumers (encodeChromaB, encodeCompositeB, mixB) read B's texel the same way, so encode_yuv.wgsl is gone altogether.

The cost of this design is that a gate can be forgotten. feedgates.spec.ts exists because a per-source fault whose gate does not know about it dispatches no pass, so its slider does nothing until some unrelated fault on the same input comes up — a failure that looks like a shader bug and is a table entry.

The raster is the data structure#

The signal is one flat array<f32>, allocated once, never read back to the CPU. Sample s of line row is index row * 910 + s, and that is the whole address scheme: no textures, no strides, no per-row objects.

Sampling at exactly 4 × F_SC is what makes the rest cheap. The subcarrier lands on a four-phase lattice — (0,1) (1,0) (0,−1) (−1,0) — so carrier(n, frame) is n & 3 and two selects, with no trig and no accumulated phase error. 910 samples per line is 227.5 cycles, which gives the 180° line alternation for free, and 525 lines gives the frame alternation, both out of n mod 4 rather than out of bookkeeping.

The decoder's demodulator takes that further. At every tap, exactly one of the U and V multiplies is against a zero, so demodAt throws half the arithmetic away before doing it: taps sum into four buckets by k mod 4, and the lattice is applied once at the end as a rotation of those buckets onto (U, V) instead of per tap. The loop is unrolled by four into a vec4f accumulator; the odd tail uses select rather than a dynamic vector index, which would spill the accumulator to scratch for the sake of up to three iterations.

Workgroup memory does four different jobs here#

Nine shaders stage something in var<workgroup> before they barrier, and it is worth knowing which of four things each one is doing, because the reason decides what may change.

Halo tiling, so a FIR reads storage once per sample rather than once per tap. Six passes filter the waveform (encode_composite, encode_chroma_b, chroma_extract, under_down, channel, decode), and each stages its own 64-sample span plus a 32-sample halo per side, barriers once, and filters out of shared memory. The kernels in the bank run 33 to 55 taps, comfortably inside the 65 a 32-sample halo allows.

The 64-thread width was measured rather than assumed. Staging costs (TILE_WG + 64) / TILE_WG loads per output, so wider workgroups re-stage less — but 64 and 128 land within noise of each other and 256 is ~8% slower. The halo traffic is not the bottleneck, so the width stays where scheduling granularity likes it.

This is where the row-uniform invariant comes from. decode stages a contiguous span of one raster row, so a horizontal offset applies to the whole workgroup or it reads outside the halo. Per-pixel horizontal scaling — H size, linearity, pincushion — needs the staging restructured first, and no amount of care in the shader gets around it.

Sharing a deviate that is expensive to draw. Snow is one Gaussian deviate per sample, and the 1-2-1 band-limit over it reads each neighbour's deviate — every thread's neighbour is some other thread's centre, so generating per-thread draws each deviate three times. gauss() is Box-Muller, two hashes plus a log and a cos, which makes that the most expensive redundancy in the pass. channel and feed both stage TILE_WG + 2 of them instead. The quadrature arm for channel's Rician envelope detector is staged only while rfSnow is on.

Hoisting a per-workgroup constant out of a per-sample loop. fb_composite's resonance is a network — one set of component values, not a different one per sample — so its 33 taps and its normalizer are functions of two controls alone. Building them inside the tap loop cost 33 cos and 33 exp at every one of the raster's 477,750 samples to arrive at the same 34 numbers each time. One thread designs the filter and everyone else reads plain coefficients: 66 transcendentals per workgroup rather than per sample, worth 3.22 → 3.06 ms/frame. The summation order is untouched, so this is not an approximation — pixdiff reads max 0 over 200 frames of a live sub-unity loop, which is the strictest check available on that pass because a one-bit error would compound every lap.

Hoisting into workgroup memory sets one trap, and it is a correctness trap rather than a performance one: the predicate sits ahead of the bounds return, since every invocation in the workgroup has to reach the barrier and 910 samples do not divide by 64.

channel tables the FM fold's 31-tap decay the same way.

crt_face's disk taps are the same move made at compile time: golden-angle direction × radius plus the beam gaussian weight, tabulated as constants. Computing them in the tap loop cost a cos/sin/sqrt/exp apiece — about 64 transcendentals per pixel, and the single most expensive thing in that pass.

Sharing an overlapping history window. line_analyze's chroma AGC lag walks an RC window up to 97 lines deep, and consecutive rows walk almost the same one — so measuring per consumer re-gated the same burst up to 97 times over. A workgroup of 64 rows needs 64 + span measurements between them, which is two or three gates per thread instead of ninety-seven, and the staged count follows the span rather than the ceiling. The RC weights stage the same way for the same reason: they are a function of k and the time constant, identical for every row in the frame. The gate and the summation order are unchanged, so pixdiff reads max 0 here too.

line_analyze is also the one place here where the win is latency on an idle GPU rather than throughput: 525 threads in 9 workgroups is a narrow dispatch, so the redundancy was never competing for occupancy.

The random fields underneath all of this are hashed from the global sample index rather than carried as state, which is what lets overlapping halos, Y/C-delayed reads and the staged tile all agree on the same deviate for the same sample without communicating.

Filters are designed on the CPU, and rebuilt as rarely as possible#

signal/filters.ts designs every kernel in the path from real frequency specs — windowed-sinc, Blackman, unity DC gain — and uploads them as taps in one buffer with a fixed 64-tap stride per section. Nothing in the signal path is an ad-hoc blur, which is what makes the physical-units convention hold end to end.

Designing a bank costs a CPU pass over five kernels, so filtersDirty decides when. Only five control keys touch it (encChromaMHz, demodMHz, chromaTail, lumaMHz, lumaPeak), and each of the three systems that can move a control per frame has a rule about them:

Uniforms are generated, packed once, and copied on the GPU timeline#

PARAM_DEFS in src/core/gpu/prelude.ts is the single source of truth for the Params struct, and field order there is the GPU memory layout. It generates the WGSL struct and a typed Record that packParams consumes, so a field added without being supplied in uniformValues() is a TypeScript error rather than a uniform that silently reads zero.

Packing writes into one reused ArrayBuffer scratch and issues one writeBuffer per frame.

Dub generations are where that pays off. Each extra generation is an independent playback pass needing its own gen seed and its own time-base walk, and the frame stages all of them up front into genParamsBuf, then moves them into the live buffer between generations with copyBufferToBuffer — on the GPU timeline, inside the same command encoder, rather than re-encoding or waiting. Only one u32 differs between slots, patched at GEN_OFFSET. The frame's own params sit in slot 0 and get copied back before the receiver runs, because the loop otherwise leaves every pass below reading gen = gens - 1.

The per-source feeds reuse the same trick sideways. packFeed spreads the program-bus pack and overrides only the fields a feed names, so each source's deck-and-cable faults are a second Params buffer bound to the same shader rather than a second set of fields — the mechanism is written once in feed.wgsl and costs no PARAM_DEFS growth. The trap that creates is in ARCHITECTURE.md, and it is worth reading before adding a fault: every field a feed does not override still holds the program bus's value.

State that lives in buffers between frames#

Anything the signal path has to remember stays on the GPU.

timingBuf carries the sync flywheel's persistent scalars just above the per-line offsets — vertical phase, PLL state, AGC gain, two second-order gain servos with their velocities, and the separator's lock age. They are named constants in the prelude rather than literal indices at the eight sites that touch them, because the raster is a constant here and a literal index is the one thing in this buffer that would not move if LINES did.

persistBufs ping-pong on frame parity. decode's lateral scatter reads neighbouring pixels, so a single buffer would hand it values the same dispatch is part way through overwriting.

Tiering, because tap count is the only lever#

crt_face's bloom and halation gather is ~0.30 ms of a 4.90 ms frame, 6%, measured by deleting both loops outright. Cost is linear in tap count at ~0.0094 ms/tap and does not care about radius — dropping eight taps saves 0.083 ms whether they sit on the 3.5-pixel bloom disk or the 15-pixel halo one, measured as separate arms and indistinguishable.

So there is no locality win hiding in this gather and no superlinearity to exploit. Both spreads tier the tap count instead of restructuring the sampling, and each tiers on whatever decides visibility for it:

All three thresholds are hard steps, so sweeping the control through one pops the result by that difference. That is the trade, written down beside the step that makes it.

crt_face's grain is the other kind of saving: the mottle is fixed to the glass, so sixteen hashes a pixel were reproducing the same field every frame. grain_bake.wgsl writes it once at engine construction into an r32float texture and the pass reads a texel — bit-exact, 0.115 ms.

The one serial pass#

sync.wgsl runs two 525-iteration loops, the PLL flywheel and the HV sag, each on a single lane. They have to be serial — each line's value depends on the previous line's — and sync_measure already scanned the waveform in parallel, so these lanes only run the recurrences over those measurements. The two are independent of each other, so a workgroup of 128 puts them in different waves and the scheduler runs them side by side: 0.31 → 0.14 ms on a look with the sag up, stock unchanged.

It is latency on one thread rather than GPU throughput, and it measures fine at 60 fps, but it is the one pass in the app that cannot scale. A third per-line recurrence should be a parallel prefix scan rather than another loop here.

The latency is now known, and it is the lane's own. Staging the measurements into workgroup memory and prefetching the next line both measured within 5% of the plain loop (0.17 ms at stock), so the cost is a single lane issuing a dependent recurrence at roughly 300 ns a line, not the memory it reads. Shortening the serial path is the only lever.

The one readback#

buzz_tap leaves 525 measurement pairs on the GPU per frame and the sound detector needs them on the CPU. That is the app's only steady-state GPU→CPU readback, and the reason it can exist without costing the render loop anything is that it never blocks.

mapAsync resolves a frame or two after submit, so a buffer copied into cannot be mapped, read and reused within one frame — hence a pool of three. When every staging buffer is still in flight the frame is skipped, which is the whole trick: the audio side is a ring with a rate servo and glides over a missing frame, whereas a stall here would show up in the picture. The flush runs after the submit that carries the copy and is never awaited, and the drive is re-read on arrival rather than captured at copy time, so letting go of the slider stops the sound next frame instead of playing out what was in flight.

A frame or two of latency is deliberate. Buzz is a rasp; nobody can hear 30 ms of sync error on one, and paying for tighter would mean blocking.

Getting pictures in#

Input staging is the one place where the cost lands on the rAF loop rather than on the GPU, and two playing clips cost more live frame rate than the heaviest preset does.

Off-thread decode and scale. createImageBitmap takes the crop rectangle and the target size together, so the decode, the resize and B's centred 4:3 crop all leave the main thread in one call. Source pictures are capped at a 1536 px long edge with aspect preserved. resizeQuality is deliberately 'low' — it matches what drawImage always did, and the chain resamples to a 754-wide raster and then damages it thoroughly, so nothing downstream could tell a better filter apart.

Don't decode a frame that isn't there. The pump dedups on currentTime, so a 30 fps clip under a faster loop is decoded once per source frame rather than once per render.

Zero copy where the browser allows it. Where the device has importExternalTexture (Chrome, feature-detected), blit_ext.wgsl samples a slot's fresh video frame straight off the browser's decoder into the slot texture, replacing the bitmap path's per-frame CPU decode/resize/upload. The import happens inside the frame that submits it, because an external texture expires with the task that imported it — the pump only parks the elements. Firefox has no such API and stays on the bitmap path unchanged.

Pacing is a separate problem from throughput#

The batch number is the GPU saturated. What a user sees is the rAF loop, paced by the display, and the display steps it in jumps rather than sliding — 48 → 24 on the dev panel.

frameLock trades rate the display was stepping anyway for a cadence that holds still: it renders every Nth refresh and submits nothing in between. A held re-present was tried and is worse — it made Firefox's scheduler slow rAF delivery itself.

The auto position (gpu/framelock.ts) decides from the loop's own cadence, and its constants encode platform lessons that deserve tests rather than rediscovery. rAF delivers catch-up callbacks milliseconds apart after a stall, so any floor trusting the fastest interval reads every normal frame after one as a miss; a percentile floor fails the other way, reading a window where most frames miss as steady-slow. What survived is judging each window on the spread of its own intervals: p75 against p25 × 1.5. A loop keeping any rate shows p75 ≈ p25, and a loop wavering between vsync steps shows p75 near double p25, because a skipped vsync doubles the interval. The spread is the stutter the eye objects to, and it needs no absolute refresh estimate, so 48, 60 and 144 Hz panels all work. A window that is slow but steady deliberately does not engage — the lock has nothing to trade there. Divisor changes get a grace window, because Firefox re-paces rAF between vsync and a ~60 Hz software tick depending on whether refreshes present, and the handover would otherwise score phantom misses against the mode that just started. Failed probes back off to a minute.

The main thread#

Does React cost this app frame rate? No, and that is measured. On the built app with every stage unfolded — 230 sliders, 4,434 DOM nodes — a ten-second idle profile attributes 0 ms to React. Holding a slider down at 60 pointer moves a second, which is the one thing in the app that asks React for work at anything like frame rate, holds 59 fps at 82% idle with React's share about 0.45 ms of 16.6. scripts/cpuprof.mjs --scenario=drag is that measurement; re-run it rather than citing this paragraph. Two things will tell you otherwise, and both have: a dev build measures React's development machinery (43% of the thread in jsxDEV and friends), and frame rate is the last thing to move, because the loop is vsync-capped.

Everything that has cost frame rate here has been other work on the same thread — a render-loop probe spinning at 8 kHz, a per-frame object copy, and a 21,420-read preset scan inside a render body. The design that keeps React out of the way is ARCHITECTURE.md › The React layer; two of its numbers came from here, that notifying per frame is a full panel render per frame (19 ms with every row mounted) and that a decorative pulse in a miniature measured 7 ms of style recalc per 3 s.

Anything else per-frame on this thread competes with the render loop too, so the per-frame path avoids allocating and the drag paths coalesce their writes. glide writes into controls in place rather than returning a fresh two-hundred-key object every frame. emit() walks its listener set live rather than over a defensive copy, which is safe regardless because Set iteration skips an entry deleted before it is reached. And storage and history writes are debounced out of the drag — a synchronous localStorage write per frame of a drag is paid on the thread feeding the GPU.

What a CPU profile found#

Sampled under Chrome's CDP profiler on the production build, macOS, August 2026. This one names its box and browser because the largest thing in it is a browser difference that would have read as zero on Firefox.

whatbeforeafter
main thread per frame (task)4.26 ms0.83 ms
of which the drain probe3.6 ms0.02 ms
LineState.update at rest86 µs23 µs
renderFrame's own time96 µs19 µs
idle, all 230 rows mounted72%94%

Frame rate said none of this. Every arm held 60 fps on both browsers before and after. Read TaskDuration per frame.

The last two are bit-exact by construction and checked as such: LineState's two implementations were run side by side over 240 frames against a seeded rand and compared float by float, and the built app was compared against the parent build over 60 stepped frames, hashing the canvas per frame. The hoists keep each expression's grouping (a * b * c hoisted as a, left as x * a * c rather than folded into a * c), because float multiplication does not associate.

The drag is comfortable because of the probe fix rather than beside it: the same drag on the parent build ran 6.4 ms/frame and gave up frame rate as the box got busier — 59.4, 54.1, 51.4 fps over three rounds, where the patched arm held 59.3, 58.1, 57.9. The drag was always affordable; there was 3.6 ms of probe in front of it, and the first thing headroom buys is tolerance of a busy machine.

A second pass went after the scenarios rather than the resting frame. Both things it found were the same shape, and it is the shape to look for here: a large multiple of 252.

whatwhenbeforeafter
matchPreset, per call in situany panel render612 µs24 µs
packFeed, per frameboth feeds engaged200 µs10 µs

A Controls object cannot be fast. V8 keeps a literal's hidden class, but 252 properties is far past the limit for either cloning one or building one a key at a time, so every {...controls} in this codebase lands in dictionary mode and reads about three times slower than DEFAULT_CONTROLS does — 9.9 µs against 3.1 for one pass over the keys. Nothing changes that short of making the board an array, which the other 250 sites that read it by name would not survive. So 9.9 µs a pass is a settled floor, and the lever is how many passes a hot path makes rather than how fast one is. Both fixes are that lever.

matchPreset was 85 presets × 252 keys, and a board matching nothing — anyone who has touched a knob — scanned all of them: 21,420 reads, in a render body, at 612 µs a call, ten times a second with a clip playing and 21 during a drag. Asking instead which keys the board holds off stock answers all 85 in one pass, because a preset matches exactly when it moves the same keys and agrees on them. packFeed was a 222-field copy into dictionary mode and then 234 names read back out, twice a frame; patchParams writes the seventeen overrides onto the packed block instead.

The residue is measured and left alone: stateUrl is rebuilt in a render body at about 48 µs a render, and the frame encodes ~20 compute passes one beginComputePass at a time, an upper bound around 40 µs. Two things the pass looked for and did not find: it does not degrade (four minutes on vhs held 60.0 fps at 831-864 µs/frame with no trend, heap flat), and the layered control machinery is cheap — eight routings, the paperclip and the stab gate at once cost 756 µs/frame, with applyMod not appearing in the profile at all.

Devices are created freely and never destroyed#

The expensive resource is not the GPUDevice. Four created and four held open, all presenting, cost a tab nothing — while one device.destroy() on a device that has been presenting ends the tab's rendering step, and the next document loaded in that tab inherits the damage.

So the app abandons devices instead of destroying them, and hands the live one to the next engine. HMR is the cheap path: a hot update recreates the engine, but the engine hands its device on alive and the successor adopts it from a stash that lives on globalThis — precisely so that editing gpu/context.ts itself does not throw it away with the module. An editing session costs one device however many saves it takes.

The runs, the three arms, and everything this forbids are in ADR 0004. Read it before touching anything that creates, releases or tears down a device.

What is left#

The backlog lives in IDEAS.md rather than here, and two of its entries are shaped by this page. Interlace is the raster restructure, with its own section there — including what field-rate timing would do to dropoutComp's 1H delay, which is not obvious from the timing alone. Intra-line geometry — hSize, hLin, pincushion, and the stair-stepping of a round()ed bend — sits under the deflection follow-ons, blocked on decode's staging for the reason above.

What this page owns is the other direction: what has already been measured and is not worth revisiting without new hardware or a new browser build — the two FIR-side micro-optimizations at the top, 256-wide FIR workgroups, asynchronous pipeline creation, and rearranging sync.wgsl's memory.

One constraint rather than an idea, since nothing is asking for it yet: a third per-line recurrence wants a parallel prefix scan, not a third loop in the one serial pass.

Further reading#