Video: decoding a stream, capturing one, and putting the frames on the screen

Design record, 2026-09-08, reviewed 2026-10-02 against master at 65e323a (react-x11 2.30.0, ntk ^8.17.8, @windowkit/appkit ^0.19.0, @windowkit/win32 ^0.0.6, @windowkit/wayland ^3.1.1, node-x11 4.2.3 in the lockfile). §0.2 is what has been built since — track 1 of §12 and the Cocoa half of track 2. §0.1 is what moved in the month between the two dates and what it changed here; §6 is the element, §7 the seam it rides, §8 the two consumers it is for, §12 what can be built now and in what order, and §13 the spikes every number came from — two of them new, and one of them overturning the conclusion the first version of this record drew for macOS. The mechanism it extends is sprites, which shipped where element-owned layer contents was designed; the constraint it runs into on X11 is the one protocol efficiency measures everywhere else.

---

0. TL;DR

• No X11 extension decodes video. The one that did — XvMC — is dead, and the things that replaced it (VA-API, VDPAU, NVDEC) are not extensions at all but C libraries that borrow a Display*. On X11 the decoder is always ours, in our process, and X's only job is moving finished frames to the screen without a copy per pixel. Nothing about that changed, and nothing will.
• On Cocoa both halves are OS services. VideoToolbox decodes in hardware into CVPixelBuffers that are already IOSurfaces, and AVPlayerLayer is a CALayer that shows an AVPlayer, which takes a URL and does the rest. Measured: 1.87ms per 1080p H.264 frame, hardware, every frame IOSurface-backed (§3.3).
• The trap that decided the first version of this record is not a trap on screen. A CALayer whose contents is a YUV IOSurface rasterizes nothing through render(in:) — that measurement stands — and shows the picture on screen, through the render server, for every two-plane and packed layout tried: 420v, 420f and 2vuy all render; only the three-plane y420 does not (§3.4, probed today through the window server's own capture). So the frame sink on macOS needs no AV layer class and no BGRA conversion: a plain layer and an NV12 surface, which costs 0.40ms of CPU per 1080p frame for the copy and the commit (§3.4.1). AVPlayerLayer is for sources — a URL, with audio, seeking and HLS — not a workaround.
• The seam this needs shipped, under another name. #499's presentedSurface() was never built; what landed is sprites (#819/#821/#828): a node offers parts of its drawing, the surface presenter lifts each onto a layer under promotion's rules — the same z-order test, the same clip boxes, the same hole, the same "declining is always safe" — and today every part's content is a raster the presenter paints by calling the part's own paint. Video is one addition to that vocabulary: a part whose content is a source rather than a paint (§7). Everything else about lifting a video is already written and tested.
• So a cross-backend <video> is one element with two sources. frames, a sink the application feeds decoded frames into, on every backend; and src, a URL the platform plays itself, on the backends that have a platform player — AVFoundation now, Media Foundation when the Windows half lands — refused with a typed error where there is none (§6). Presented two ways everywhere: lifted onto a layer when the frame says it can be, drawn into the window's bitmap when it cannot.
• Two consumers, one element. A <video> in a form is the element. A <video> inside @react-x11/components' <Html> is the element too, mounted beside the document at the box's rect the way <Html> already mounts a core <textinput> for an <input> (§8) — not a second video implementation inside the HTML renderer, and not a new seam.
• Windows has the pieces and none of the design. The win32 backend is a DirectComposition visual tree per window with <glarea> on a visual of its own and a pane host that already attaches a surface from a handle — which is precisely what IMFMediaEngine's windowless swapchain hands out. §4 is the plan and the list of things to probe on a Windows machine before writing any of it.
• The X11 numbers that set the budget still stand. A 1080p frame through core PutImage costs 33ms with a round trip (§2.5); SHM halves it and is already in ntk; Xv is now in node-x11 and in the lockfile (4.2.3), untouched by anything here; XQuartz still answers "no adaptors present", so every Xv path still needs the drawn fallback by construction.

0.1 Review, 2026-10-02: what moved

the record said, 2026-09-08	the tree says, 2026-10-02	what it changes here
a video node is an element-owned layer on #499's presentedSurface(), which should answer a layer	#499 is closed, nothing of it built; sprites shipped instead (src/nodes/sprites.js, src/cocoa/sprites.js), a per-frame accessor in the shape of opaqueRect() whose parts are rasters painted by the presenter	§7: the seam is a sprite part with contents, not a new accessor
a CALayer draws no YUV IOSurface; use AVSampleBufferDisplayLayer or pay 2.1× the decode for BGRA	on screen it draws every two-plane and packed layout; the blank was render(in:), a CPU path (§3.4)	the Cocoa frame sink is a plain layer and two bridge verbs, not an AV class; AV verbs are for src only
the implementing PR should first verify that the render server scans out a YUV surface	verified: 420v, 420f, 2vuy render; y420 and f420 (three planes) do not; colour attachments change the matching	NV12 is the sink's native format on macOS; I420 is interleaved on the way in
the bridge should grow createVideoLayer, enqueueSampleBuffer, flushVideoLayer and a synchroniser (against @windowkit/appkit ^0.7.0)	0.19.0: createSurfaceIOSurface, surfaceFromIOSurfaceID, setLayerContentsIOSurface, surfaceIsInUse, a surface-released event, snapshotWindow through CGWindowListCreateImage; still no CoreVideo, CoreMedia, VideoToolbox, AVKit or Metal linked	§9: the verb list, in two releases
whether snapshotWindow composites sublayers is open, and decides what the tests can assert	it is the window server's composite, and today's capture holds an AVPlayerLayer, an AVSampleBufferDisplayLayer and the YUV tiles	pixel tests on Cocoa go through snapshotWindow; no render(in:)
two backends	four: win32 (src/win32/, DirectComposition, <glarea> on a visual, a pane host over CreateSurfaceFromHandle) and Wayland (src/wayland/, dma-buf only, its own GLES rasterizer, no subsurfaces bound)	§4 and §5, new
Xv is a lockfile refresh away	the lockfile has node-x11 4.2.3 (PutImage, ShmPutImage, QueryImageAttributes, StopVideo, both notify selectors); ntk still has no Xv wrapper and the renderer calls none of it	the Xv rung is gated on nothing upstream; it is still work
camera capture is a separate element, later	unchanged — and the on-screen result makes both AVCaptureVideoPreviewLayer and a raw 2vuy surface viable layer contents, where the first version doubted both	§3.5's conclusion softens; the element is still later
the ecosystem has nothing to play a video with	@react-x11/components has <MediaPlayer> (mpv or VLC embedded over <foreign>, X11 only, transport over mpv's JSON IPC) and <Html> lays a <video> out as its poster, or a 300×150 box; neither plays a frame on macOS	§8: what each becomes

0.2 Built, 2026-10-02

• Core (§12 track 1): VideoFrames (src/videoframes.js, createVideoFrames, useVideoFrames), <video> (src/nodes/video.js) drawn on every backend through a Surface of its own, objectFit for <video> and <image> (src/nodes/fit.js), poster, HTML's sizing, onLoadedMetadata, the typed refusal of src (NoMediaPlaybackError) and useSupports('mediaPlayback'), false everywhere until a player lands. The JavaScript conversion (frameToRGBA) applies the matrix and the range and no transfer function.
• The contents sprite kind (§7) in src/cocoa/sprites.js, and the lift in src/cocoa/video.js: a ring of up to four video surfaces, each push written and the layer repointed in the push. Measured on the real bridge, threaded: sixty pushes, one window frame — the lift's.
• @windowkit/appkit release A (§9), with the shapes the work settled on: createVideoSurface(width, height, { format, colorSpace, range }), writeVideoSurface(target, format, planes, { strides, … }) — whose target may also be a 2D surface, which is the drawn presentation's whole native half — videoSurfaceIsInUse, releaseVideoSurface and videoFormats(). The surface holds the IOSurface alone, not the CVPixelBuffer it was made through: a pixel buffer keeps a use count on its surface, and IOSurfaceIsInUse would answer true for one nothing shows. videoSurfacePlanes was not needed: the copy is 0.07ms at 1080p.
• A colour finding the record did not have. With CoreVideo's tags alone, Core Animation shows a surface tagged BT.709 throughout with the exact 709 curve — while AVPlayerLayer, VideoToolbox's pixel transfer, Core Image, and Core Animation itself for any other tagging use Apple's 1.961 gamma: a video-range Y′=47 was sRGB 51 on such a layer and 40 under AVPlayerLayer playing the same clip. So §3.4's "colour attachments are read" was half the story: the bridge names each video surface's colour space as the one CoreVideo makes of its tags, which puts a lifted frame on the player's curve exactly, and every drawn frame goes through VideoToolbox, about 1ms at 1080p. A frame lifted, drawn, and played by AVPlayerLayer agree to within two levels on screen (the bridge's test/video-surface.js and test/player.js).

Also: the presenter bench's video scenario and its gate rule (§9) — a lifted sink holds to no window frame per push, a rule read as skipped over a bridge without video surfaces. And <video src> on macOS (§12 track 3): @windowkit/appkit's createPlayer, playerSet, playerSeek, playerCopyFrame and releasePlayer with the player-* events, and src/cocoa/player.js, the player as a sprite part's contents — its AVPlayerLayer lifted into the part, its frame copied into the element's surface while it plays and something is over it. HTML's props, events and imperative half; useSupports('mediaPlayback') true where the bridge plays. Not built yet: the Windows and Wayland rungs.

1. The asymmetry, stated once

Every other feature in docs/architecture/ has the same shape on every backend: a rule, N implementations, one set of semantics. Video does not, and pretending otherwise would produce an API that is a lie on half the backends.

On macOS, "play this video" is a supported product: hand AVPlayer a URL, attach AVPlayerLayer, and you have hardware decode, HLS, A/V sync, seeking, HDR and colour management, none of it ours. On Windows the same product is Media Foundation's IMFMediaEngine, whose event vocabulary is HTML5's media element verbatim. On X11 and Wayland every one of those is a component someone assembles — a demuxer, a decoder, a clock, a colour conversion, a presentation path — and the display server contributes only the last one, badly on X11 unless you leave the protocol for a native library, and not at all on Wayland, which is a buffer handoff by design.

The design consequence is that the element is shared and the source is not. A <video> whose only prop is a URL would work beautifully on Cocoa and be a lie on X11, where nothing in this project can open a URL and produce frames. A <video> whose only prop is a frame sink is honest everywhere but throws away the whole of AVFoundation. §6 is the resolution: both, with the frame sink as the primitive and the URL as the convenience of the backends that have a player, asked for with useSupports and refused with a typed error elsewhere.

2. X11

2.1 Xv, which does not decode

Xv is the extension people mean when they say "the video extension", and the first thing to be clear about is that it is a presentation extension. It converts colour, it scales, and it puts the result in a drawable. It does not decode anything.

The model is ports on adaptors. QueryAdaptors on a window answers the adaptors a screen has; each has ports, a type mask (ImageMask is the one that matters — an adaptor that accepts images from client memory rather than from a capture source), and the visuals it can draw into. You GrabPort one, ask ListImageFormats for the FOURCCs it takes (YV12, I420, UYVY, YUY2…), ask QueryImageAttributes for the plane pitches and offsets at your size, and then push a frame per PutImage — or ShmPutImage, which is the same request with a shared-memory segment in place of the payload.

Two things it gives you that are worth the trouble:

• The colour conversion is the server's. You push planar YUV at 1.5 bytes per pixel; the driver produces RGB. For 1080p that is 3.1MB per frame instead of 7.9MB, and no CPU of ours touches a pixel.
• The scale is the server's too, with QueryBestSize to ask what it would prefer, and the drawable rectangle chosen per frame — so a resized video costs nothing extra on our side.

And the caveats, which are why it is not simply the answer:

• Overlay adaptors and compositing do not mix. A hardware overlay writes past the compositor; under a compositing WM you want a textured adaptor, which every modern driver provides, but the distinction is not visible in the protocol and the failure is a black or misplaced rectangle.
• There is no clock. Xv presents when you ask. Pacing, A/V sync and vsync are entirely yours — which is exactly the seam the frame pacer already owns for everything else.
• It may not exist. Measured on this machine: XQuartz advertises X-Video Extension version 2.2 and answers no adaptors present. The extension is there and useless. Any Xv path needs a fallback by construction, not as a nicety.
• Where it draws. A textured adaptor's PutImage lands in the drawable it names. Named at our window it is overwritten by the next present of ntk's backing pixmap, and named at a child window of its own it stacks above everything drawn, which is the z-order flaw <glarea> and <MediaPlayer> both live with. The honest target is the backing pixmap itself, at the fitted rect, as a step of the paint pass — then the frame composites in paint order and the present carries it. Whether ntk will name that pixmap to a caller is a small seam to ask for, and the question the Xv rung starts with.

2.2 The decode extensions, and what replaced them

XvMC is the one X extension that ever did decode: it took MPEG-2 macroblocks, IDCT and motion compensation onto the GPU. It is effectively dead — a handful of drivers, MPEG-2 and some MPEG-4 only — and it is unreachable from a protocol client anyway, because the client API is a vendor libXvMC*.so the application dlopens, not wire requests. It is mentioned here so that the next person who finds it in a list of extensions does not spend an afternoon on it.

VA-API, VDPAU, NVDEC are what people actually use, and none of them is an X extension. They are C libraries that use X only to obtain a display and, optionally, to present: vaPutSurface(Drawable), VdpPresentationQueueTargetCreateX11. Underneath they go to DRI and the kernel. A pure-JS protocol client cannot speak them at all; reaching them means a native addon, at which point the addon may as well own decode and presentation together.

The conclusion is the TL;DR's: on X11 we decode, always. In practice that means ffmpeg through a native addon, a WASM decoder, or a subprocess pipe — and that is a dependency decision (§10), not a rendering one.

2.3 The transports

Given frames in our memory, three ways to get them onto the screen, in ascending order of how little they copy.

MIT-SHM. ShmPutImage moves a frame through shared memory instead of the socket. ntk already has this — ShmUploader in node_modules/ntk/lib/shm-upload.js, reached through the app's shm getter and used by putImageData and by Image uploads: a pool of segments rounded up to a 64KB quantum, the first upload of a size going out as core PutImage while a segment warms behind it, and each segment returned on ShmCompletion. Its own measurements put it at roughly 2× on uploads, and it is available on XQuartz here. This is the floor of any X11 video path, it is already built, and the frame sink's drawn presentation gets it for free by writing into an ntk Surface (§6.3).

DRI3 + Present. The modern zero-copy present: PixmapFromBuffer imports a dma-buf fd as a Pixmap, PresentPixmap flips it with target_msc/divisor/remainder for vsync, and PresentCompleteNotify/PresentIdleNotify hand back both a frame clock and buffer recycling. node-x11 ships dri3.js and present.js, and x11-dri is already an optional dependency. The blocker is the direction: the DRI3 receive side — importing a buffer somebody else allocated — is not wired, and that is precisely the direction a hardware decoder needs. This is the right long-term path and the largest piece of work in this document.

GLX / direct GL. Upload Y and CbCr as textures and convert in a fragment shader. This works today, through <glarea>, with no new protocol at all: see gl.md and the direct-GL notes in glx.md. It is the fastest thing to prototype and it costs a <glarea>'s constraints — the node owns a real child window or layer, and its pixels cannot be read back through the X path. It is also, as it happens, the shape of the Wayland answer (§5), where the backend's whole 2D context is a GLES rasterizer.

2.4 What node-x11 has

Until 2026-09-07 lib/ext/xv.js implemented the half of Xv that asks questions and none of the half that answers with pixels. That gap is closed — sidorares/node-x11#294 was fixed by #295 — and, since this record was first written, installed: the lockfile carries 4.2.3, so npm ci now gets the image path. The refresh the first version asked for has happened.

Xv minor opcode	4.1.0	4.2.0+
0–4 version, adaptors, encodings, port grab	yes	yes
5–8 PutVideo/PutStill/GetVideo/GetStill	no	no
9 StopVideo	no	yes
10 SelectVideoNotify / 11 SelectPortNotify	no	yes
12–16 best size, port attributes, image formats	yes	yes
17 QueryImageAttributes	no	yes
18 PutImage	no	yes
19 ShmPutImage	no	yes

5–8 remain absent on purpose — they are the capture-card half of the extension, as dead as XvMC, and §3.5's answer to "where do camera frames come from" is not Xv.

Two things about the implementation that bound what this document can assume:

• PutImage refuses a frame the connection cannot carry, rather than sending it and having the server drop the connection over a bad request length. The 40-byte header plus a padded payload has to fit max_request_length, which without BIG-REQUESTS is 256KB. A 1080p I420 frame is 3.1MB, so PutImage alone cannot carry an HD frame on a connection without BIG-REQUESTS, and will say so instead of failing mysteriously.
• ShmPutImage sidesteps that entirely, because its 52 bytes carry a SHMSEG and no payload. So for HD it is not an optimisation over PutImage; it is the path.

What is still missing is ours: ntk has no Xv wrapper (its only xv substrings are Xvfb and glxVisual), and nothing in src/ requires the extension. The rung is gated on nobody but us (§12).

2.5 Measured: what the naive path costs

The question that decides whether "decode in-process and PutImage the result" is even worth prototyping. A 1920×1080 frame at depth 24, pushed to an offscreen pixmap with a GetInputFocus round trip after each so the server has demonstrably consumed it, on XQuartz over a unix socket, M1 Pro:

path	median	min	p95
core PutImage, 7.9 MiB BGRA, + sync	33.19ms	25.22ms	48.26ms

That is 238 MiB/s and a 30.1fps ceiling with zero time left for layout, paint, or the rest of the application — for one video, at 1080p, on a local socket. It is a bound rather than a Linux number (XQuartz is not a DRI-backed server), but it is the right order of magnitude for the shape of the problem, and it prices the alternatives:

• 420v instead of BGRA is 3.1MB instead of 7.9MB — 2.6× less to push, before SHM.
• SHM removes the socket copy; ntk's own figure is ~2× on uploads.
• Xv does both and moves the colour conversion and the scale off our CPU — which is why it was worth the upstream request even though the extension is absent on XQuartz.

What follows for the element: on X11 the sink wants BGRA frames, and says so (§6.3's preferredFormats), because a decoder that can emit BGRA does the conversion in C for nothing, and one that cannot leaves it to a JavaScript loop over two million pixels. At 720p — the size a video in a form actually is — a BGRA frame is 3.7MB, which by the two numbers above the SHM path puts on the screen in the order of ten milliseconds on XQuartz — derived, not measured, and owed a measurement by the bench scenario in §12. That is the honest X11 budget until the Xv rung, and a document that said otherwise would be read by someone building a 4K player on a PutImage.

3. Cocoa

3.1 Decode

VTDecompressionSession is the primitive: build a CMVideoFormatDescription from the stream's parameter sets, feed it CMSampleBuffers, get CVPixelBuffers. Ask for IOSurface backing with kCVPixelBufferIOSurfacePropertiesKey and every output buffer is a surface the render server can already read.

Measured on this machine (macOS 15.2 then, M1 Pro), VTIsHardwareDecodeSupported:

codec	hardware
H.264	yes
HEVC	yes
ProRes 422	yes
AV1	no
VP9	no

AV1 is the one to plan around: hardware AV1 decode arrives with M3-class silicon, so an AV1 stream on this machine is a software path or nothing.

Above VideoToolbox, AVFoundation supplies the parts we would otherwise write: AVAsset + AVAssetReader for files frame by frame, AVPlayer / AVPlayerItem for URLs including HLS, AVPlayerItemVideoOutput.copyPixelBuffer(forItemTime:) to pull frames in step with a display link, AVCaptureVideoPreviewLayer for a camera, and ScreenCaptureKit for the screen.

3.2 The presentation ladder

Every rung is a CALayer, which is why all of them compose with layer promotion and with the sprite order unchanged.

1. AVPlayerLayer — set player, set videoGravity, done. No pixel code and no JS in the frame path at all. Everything about the source is AVFoundation's. The rung for src.
2. AVSampleBufferDisplayLayer — enqueue CMSampleBuffers from our own demuxer or socket; it decodes and schedules them itself, and AVSampleBufferRenderSynchronizer with a CMTimebase is how it is synchronised against audio. The rung for "a compressed stream that is not a file or a URL" — which no consumer in §8 has yet.
3. A plain CALayer whose contents is the decoded IOSurface — the rung this repo is already standing on, since setLayerContentsIOSurface is how <glarea>, the <Frame> pane host and the window swapchain all present. The rung for frames, now that §3.4 says it takes YUV.
4. CAMetalLayer + CVMetalTextureCache — zero-copy MTLTexture from the pixel buffer, full shader control, our own colour conversion and scaling. Nothing here needs it.
5. CAOpenGLLayer — legacy, and <glarea>'s neighbourhood.

3.3 Measured

The 2026-09-08 spike encodes a synthetic sequence with VideoToolbox, decodes it back, and times each step. 1920×1080 H.264, 60 frames, hardware acceleration confirmed on the session, every output buffer IOSurface-backed:

step	median
decode a 1080p H.264 frame, default output	1.868ms
decode the same frame, output forced to BGRA	3.917ms
layer.contents = ioSurface	0.04µs

At 1280×720 the same measurement is 1.143ms for H.264 and 0.932ms for HEVC. The flip is below the resolution of a microsecond timer, because the assignment itself is a pointer swap and the transaction commit is where CA's cost lives.

AVSampleBufferDisplayLayer accepted all 30 compressed sample buffers it was offered headless, with status rendering and no error.

3.4 The YUV surface: blank through render(in:), drawn on screen

A hardware decoder's natural output is 420v — bi-planar 8-bit video-range YCbCr, two planes, 3.1MB for 1080p. Our swapchain surfaces are BGRA. The obvious move is to hand the decoder's surface straight to the existing setLayerContentsIOSurface verb.

The first version of this record measured that through CALayer.render(in:) and got an empty box: 0 of 4096 pixels for a decoded 420v frame, 0 of 4096 for a captured packed 2vuy one, against BGRA and CGImage controls at 4096 of 4096 through the same probe. It read that as "a CALayer takes any YUV IOSurface and draws none of them", recommended AV layer classes for everything, and — to its credit — ended with "whether the render server scans out a YUV surface on screen is not settled by it, and that is the very first thing the implementing PR should check".

Checked today (scripts/spikes/video/yuvscan.swift, §13): ten tiles in a borderless window, each a CALayer on screen, the window captured through the window server (screencapture -l, the same source as the bridge's snapshotWindow), and the centre pixel of each tile read back. macOS 15.8.1, M1 Pro:

layer contents	on screen, window-server capture	render(in:), 2026-09-08
BGRA IOSurface — control	blue, as filled	4096 / 4096 px
420v IOSurface, 2 planes, video range	green, as filled	0 / 4096 px
420f IOSurface, 2 planes, full range	green, as filled	—
2vuy IOSurface, 1 packed plane	green, as filled	0 / 4096 px
y420 IOSurface, 3 planes, video range	nothing: the tile is transparent	—
f420 IOSurface, 3 planes, full range	nothing: the tile is transparent	—
AVSampleBufferDisplayLayer, 30 H.264 frames	red, as encoded	—
AVPlayerLayer, a .mov written by the probe	magenta, as encoded	—
CGImage — control	orange, as drawn	4096 / 4096 px

Three conclusions, and the first two overturn what was written here before:

• The render server draws YUV layer contents. render(in:) is a CPU path that does not, which is a fact about testing, not about the screen (§13 says what to test through instead). The "free" path is free.
• Two planes or packed, not three. Every bi-planar and packed layout tried renders; the three-plane planar ones — what ffmpeg calls yuv420p, the default output of nearly every decoder — draw nothing: the layer is transparent and the window shows through. So the sink's native format on macOS is NV12, and an I420 frame is interleaved into one on the way in: a loop the bridge's write verb does over 1.5 bytes per pixel, in the same pass as the copy it has to make anyway.
• Colour attachments are read, and change the picture. A second 420v tile with a BT.601 matrix and an sRGB colour space attached rendered a visibly lighter green than the bare one. The sink has to say what its bytes mean (§6.3: colorSpace, range), and the bridge has to attach it to the surface — the Generic-RGB default has bitten this backend before (#820), and a video that is subtly the wrong colour is the kind of defect nobody files.

The cost of the three ways out stays on record because two of them are still what src and a compressed stream use, and because the number is what proves the sink should not convert: forcing BGRA out of the decoder costs 2.1× the decode (3.917ms against 1.868ms) plus 8.3MB per surface against 3.1MB; AVSampleBufferDisplayLayer takes YCbCr natively and brings A/V sync; Metal is its own render loop.

3.4.1 Measured: what the frame sink costs through a plain layer

The question §6.3 rests on: when decoded bytes arrive from a decoder — over a pipe, from an addon, from a WASM module — what does it cost to show them through rung 3? scripts/spikes/video/sinkcost.swift stands a memcpy into a locked IOSurface in for the read, points a layer at it inside a transaction, and paces the loop to 60Hz so the render server has a real frame per iteration. 1920×1080, a ring of three surfaces, 240 frames, process CPU from getrusage, two runs each:

format	bytes per frame	CPU per frame	note
NV12 (420v)	3.11 MB	0.40 – 0.43 ms	the copy and the commit; nothing converts
BGRA	8.29 MB	0.69 – 0.72 ms	the same, over 2.7× the bytes

The wall time per iteration is the 60Hz pacing, not a cost. So a 1080p60 sink lifted onto a layer is about 25ms of CPU per second in the process that feeds it, before the decoder; the decoder is the whole budget, and the presentation is noise beside it. That is the number that says the sink needs no JS frame per video frame and no pacer claim: a push() on a lifted sink writes the surface and repoints the layer right there, and react-x11's own frame clock is not involved until the next frame re-decides the lift (§7).

3.5 Capture: the same stack, one step shorter

A webcam is the same set of API calls with the decode removed. AVCaptureSession delivers CMSampleBuffers carrying IOSurface-backed CVPixelBuffers — the same objects a VTDecompressionSession produces — and AVCaptureVideoPreviewLayer is a CALayer, so rung 1 of §3.2 serves a camera exactly as it serves a file. And since §3.4, so does rung 3: a camera's native 2vuy frame is a layer content the screen shows.

Three things separate capture from playback, and only one of them is work.

The permission half is already built. camera and microphone are first-class kinds in permissions.md — usePermission('camera'), permissionStatus, requestPermission, openPrivacySettings — translated in src/cocoa/permissions.js onto the bridge's authorizationStatus/requestAuthorization verbs, which are AVCaptureDevice's underneath (the one place the bridge links AVFoundation today). A capture element does not need to touch TCC at all.

Discovery needs no authorization. AVCaptureDevice.DiscoverySession enumerates cameras and microphones, names included, while the status is still notDetermined — no prompt, no indicator light. A device picker can be built and rendered before consent; only the frames are gated.

The conversion is not free, and this is the finding. A camera's native delivery is YUV — 2vuy at the 720p preset here. Asking the output for BGRA instead is one line and is the conventional capture configuration. Measured, 1280×720, 60 frames per configuration, camera live:

output configuration	delivered	process CPU	delivery interval	our callback
native, no settings	2vuy	4.16ms/frame	33.5ms	0.054ms
videoSettings BGRA	BGRA	11.12ms/frame	33.0ms	0.095ms

+6.96ms per frame, 2.67×, charged to our own process rather than to the camera daemon, on another thread — so it burns a core rather than dropping frames, which is precisely the kind of cost the frame pacer cannot see. The absolute figures amortize one-time session setup over 60 frames; the delta is the trustworthy part.

So capture reaches the same conclusion as playback by the same route: do not convert. Hand the session to AVCaptureVideoPreviewLayer, or hand its native surfaces to a plain layer; both are on the screen.

What the tests cannot do. A running AVCaptureVideoPreviewLayer with a live connection rasterized 0/4096 through layer.render(in:) — which, after §3.4, is unsurprising rather than alarming. Capture pixel tests go through the window server's capture like every other layer's, or through an AVCaptureVideoDataOutput tap.

One packaging finding, because it will cost someone an afternoon. A correctly signed .app carrying NSCameraUsageDescription and a CFBundleIdentifier, executed directly from a shell, answered requestAccess with granted = false while authorizationStatus stayed notDetermined — a refusal that never prompted. Launched through LaunchServices (open), so that the app is its own responsible process, the same binary prompted and was granted. This is the same shape as the notification rung's trap in packaging.md, and the same rule applies: a falsy grant with the status still notDetermined is not a denial, and code that reads it as one tells the user they declined something they were never asked.

4. Windows

Written on a Mac against the bridge's source at v0.0.6 and windows.md. Nothing below has been run; §4.3 is what to run first.

4.1 What the backend has

A window on win32 is a DirectComposition visual tree: one target, one root visual whose content is a virtual surface in B8G8R8A8, painted one BeginDraw per damage rect and committed once per frame (windows.md). Under the root visual hang the children an element can own today, each on a visual of its own:

• <glarea>, a composition swapchain filled through a WGL context and WGL_NV_DX_interop2, added at the bottom of the children (windows-gl.md). A child HWND was built, measured invisible in every variant, and abandoned (windows-embedding.md).
• A <Frame> pane, attached from a surface handle another process presents into: DCompositionCreateSurfaceHandle on the pane's side, CreateSurfaceFromHandle and a visual on the host's (paneAttach, paneSetRect with its clip).
• Layers, a premultiplied virtual surface on a visual at the top of the children, which is what <glarea>'s drawn children paint on (layerCreate, layerBeginDraw, layerScroll).

What it does not have is a presenter that decides, per frame, whether an element's content may sit on a visual of its own: the Windows retained tier and promotion are a Phase 4 probe in windows.md, not code, and the sprite seam is asked on macOS only. So on win32 today a visual is where <glarea>'s is — above the 2D surface, under the layers, and under whatever the 2D surface paints after it is not. There is no Media Foundation, no YUV, no DXGI_FORMAT_NV12 anywhere in the bridge, and windowPixels reads the window back through PrintWindow.

4.2 The plan

src is IMFMediaEngine, the platform's player, which has two ways to show a picture and the backend wants both, in this order:

1. Frame-server mode first. TransferVideoFrame copies the current frame into a D3D11 texture on request; the backend draws that texture into the window's surface at the fitted rect, in paint order, like an <image> whose bytes changed. A copy per frame — but a GPU copy, and correct under every tooltip, popup and fade, with no new z-order rule. This is the drawn presentation of §7, and on Windows it is the one that ships first because nothing there can yet decide the other.
2. Windowless swapchain mode second. EnableWindowlessSwapchainMode, GetVideoSwapchainHandle, and the handle goes to the exact call the pane host already makes — CreateSurfaceFromHandle onto a visual at the element's rect. Zero copies, the engine presents on its own clock, and the overlay planes can scan it out where the GPU has them. It needs what <glarea> lacks: a per-frame answer to "is anything drawn over this", which is the Windows half of promotion. Until then it is <glarea>'s stacking, and a video element should not inherit that quietly.

The engine's notifications — LOADEDMETADATA, TIMEUPDATE, PLAYING, PAUSE, ENDED, ERROR — are the element's events already (§6.1); the bridge forwards them as it forwards everything else, in batches, off the engine's work queue onto Node's thread.

frames is a texture. A decoded frame arrives as bytes; the bridge puts them in a D3D11_USAGE_DYNAMIC texture and the drawn presentation composites it. BGRA is the floor. NV12 is the better transport — DXGI_FORMAT_NV12 textures are routine, and a flip-model composition swapchain in NV12 on a visual of its own is what windows.md's rendering section already names for video — but whether the adapter will create one is a CheckFormatSupport question per machine, and the fallback is a ID3D11VideoProcessor blit to BGRA. Both are probes, not plans.

4.3 What to probe first, on the Windows machine

In the order that decides the most for the least:

1. IMFMediaEngine from a bridge thread: create it from MFMediaEngineClassFactory on the UI thread with a DXGI device manager over the bridge's D3D11 device, load a file URL, confirm LOADEDMETADATA and the frame size, and TransferVideoFrame into a texture the 2D surface can draw. That is the whole drawn path.
2. EnableWindowlessSwapchainMode + GetVideoSwapchainHandle + CreateSurfaceFromHandle on a visual: does the picture appear, does paneSetRect's clip apply to it, and does it respect the visual's place among the children?
3. windowPixels over 1 and 2: does PrintWindow with full-content rendering see a swapchain visual's frame? It decides what the tests can assert without a capture tool.
4. CheckFormatSupport(DXGI_FORMAT_NV12) with DISPLAY and TEXTURE2D, and a composition swapchain in NV12: on the GPU in the machine and on WARP, since CI is WARP.
5. The cost of the frame-server copy at 1080p60: a GPU copy is cheap, and the number is still owed before it is called cheap here.
6. Media Foundation's threading against the bridge's UI thread: the engine wants a multithreaded apartment and a work queue; the UI thread is an STA for OLE. Confirm the notify callback can post to the bridge's event batch without a wait on the UI thread.

5. Wayland

The Wayland backend presents dma-bufs and nothing else: every frame is drawn by its own GLES rasterizer into a GPU target and handed over through zwp_linux_dmabuf_v1; wl_shm is bound for screen capture only; and wl_subcompositor is never bound, by design — a subsurface would be a second swapchain, a second frame clock and a second import per frame (wayland-backend.md). Its <glarea> is a rectangle of the window's own target, not a surface of its own.

That makes its video story the shortest one here and, for the frame sink, the cheapest: the backend's 2D context is already a GL rasterizer, so decoded planes go up as textures and the colour conversion is a sampler in a fragment shader — the GLX transport of §2.3, native to the backend rather than bolted on, and with no conversion in JavaScript for any format. The sink's drawn presentation on Wayland takes NV12, I420 and BGRA alike and prefers whichever the decoder emits. Nothing lifts: there is no layer to lift onto and no compositor-side scaler asked for, and wp_viewporter is used for the window's fractional scale only.

src has no answer on Wayland for the same reason it has none on X11 — no display-server player exists; GStreamer and PipeWire are libraries, and the application that wants them brings them (§8.3) — so useSupports('mediaPlayback') is false there and <video src> refuses the way it does on X11. The zero-copy future is the same shape as X11's DRI3 one: a decoder that exports dma-buf fds (VA-API does) and a subsurface that imports them, both of which the backend has deliberately not built, and neither of which a form with a video in it needs.

6. The element

6.1 <video>

// a file or URL the platform plays itself — AVFoundation today,
// Media Foundation when the Windows half lands
<video
  src="file:///Users/me/clip.mp4"
  autoPlay
  muted
  loop
  style={{
    width: 320,
    height: 180,
    objectFit: 'contain',
    backgroundColor: '#000',
  }}
  onLoadedMetadata={({ width, height, duration }) => setDuration(duration)}
  onTimeUpdate={({ currentTime }) => setAt(currentTime)}
/>;

// frames the application decodes — every backend
const frames = useVideoFrames({ width: 1280, height: 720, format: 'NV12' });
useEffect(() => {
  const decoder = startDecoder(url, frames.preferredFormats);
  decoder.on('frame', (planes, time) => frames.push(planes, { time }));
  return () => decoder.stop();
}, [frames, url]);
<video frames={frames} style={{ width: '100%', aspectRatio: 16 / 9 }} />;

A <video> is a drawn node like <image> — it joins yoga, it is in the paint order, it is hit-tested like anything — not a drawn: false child window like <glarea> or <foreign>. That is the whole point of §7: a video in a form sits under the form's popups and tooltips and inside its scroll panes, and a child window can do neither.

prop
src	A path or URL, played by the platform. Where useSupports('mediaPlayback') is false the element reports one typed onError on a microtask and shows its poster — never a silent black box, the rule <foreign> set in #531.
frames	A VideoFrames sink (§6.3). Mutually exclusive with src, and passing both throws, the way <image>'s src/picture/drawable do.
poster	An ImageSource, shown until the first frame and whenever there is nothing to show.
autoPlay, loop, muted, volume, playbackRate	HTML's names and HTML's defaults: nothing plays until asked. Meaningful with src; a sink plays what it is pushed.
paused	Controlled when present, the shape <MediaPlayer paused> already has. Left out, the element is uncontrolled and the handle drives it.
onLoadedMetadata({ width, height, duration })	The stream's size is also the element's natural size, kept to its ratio when the style names one axis; before it is known, HTML's 300×150.
onPlay, onPause, onEnded, onTimeUpdate, onError	onTimeUpdate is coalesced to a few a second, as HTML's is, so a progress bar costs a render per tick and not per frame.
ref → { play(), pause(), seek(seconds), currentTime, duration }	The imperative half, for the same reason HTMLMediaElement has one: a seek is an event, not a state.
role, aria-label	video by default — a role the accessibility model gains with the element, AT-SPI's ROLE_VIDEO beside img's IMAGE — named by aria-label; the frames themselves are not described.

Style, in style like everything else: width/height as any element; objectFit — 'contain' by default, 'cover', 'fill' — decides where the picture goes in the box, and backgroundColor is what shows in the bars. objectFit is a new paint property and applies to <image> as well in the same change, since an image stretched to its box has wanted it for as long as there has been an <image>; borderRadius on a <video> is legal and costs the drawn presentation on macOS (§7), which the docs say in one sentence next to the prop.

6.2 What useSupports('mediaPlayback') answers

Whether <video src> plays on this display: true on Cocoa once the player verbs ship, true on win32 once §4's first probe becomes code, false on X11 and Wayland for good, false on every backend until then. It is a property of the display's pipeline, which is what useSupports answers; the desktop-service shape ({ available, backend, features }) is for things that come and go while the process runs, and a platform player does not. frames needs no capability: it works everywhere, at a cost that differs.

6.3 VideoFrames, the sink

createVideoFrames(app, {
  width, height,
  format: 'NV12' | 'I420' | 'BGRA',
  colorSpace?: 'bt709' | 'bt601' | 'bt2020', // default bt709
  range?: 'video' | 'full',                   // default video
}): VideoFrames

interface VideoFrames {
  readonly width: number; readonly height: number; readonly format: VideoFormat;
  /** this backend's order of preference, for a decoder that can choose */
  readonly preferredFormats: readonly VideoFormat[];
  push(planes: Buffer | readonly Buffer[], opts?: { strides?: number[]; time?: number }): void;
  close(): void;
}

useVideoFrames(options): VideoFrames // stable across renders; new when size, format or colour changes; nothing to close

Three rules, each of which is a cost somewhere:

• The newest frame is the frame. A push while the last one is not yet shown replaces it; nothing queues. A sink is a mirror of a decoder's output, not a transport with back-pressure, and the decoder's own clock is the pacing — exactly as a <canvas> animation's is.

• Every format is accepted everywhere; the cheap one differs. A decoder that can choose reads preferredFormats; one that cannot still works, with the conversion charged where it happens:

  backend	lifted	drawn	preferredFormats
  Cocoa	NV12 as is; I420 interleaved on the write (bridge); BGRA as is	BGRA; NV12/I420 converted in the bridge by VideoToolbox, colour-matched to sRGB, about 1ms at 1080p	NV12, I420, BGRA
  X11	—	BGRA into an ntk Surface (SHM where the server has it); NV12/I420 converted in JS, which is not free at 1080p	BGRA
  win32	later: an NV12 swapchain on a visual (§4.2)	BGRA into a texture; NV12/I420 through the video processor, or in JS until then	BGRA, then NV12
  Wayland	—	NV12, I420 and BGRA as textures; YUV sampled in the shader, no conversion anywhere	NV12, I420, BGRA

• push is not a frame. On a lifted sink it writes a surface from the ring and repoints the layer in a transaction of its own, and react-x11's frame clock does not run — 0.4ms of CPU at 1080p (§3.4.1) and no claim for the pacer to price. On a drawn sink it claims the element's content rect, bounded, and the next frame composites the new bytes like an <image> whose src changed — priced like any claim, and the first thing a profile of an X11 player shows.

Where the bytes come from is the application's business (§10), and the sink's design keeps it that way: an ffmpeg child process writing rawvideo to a pipe, a WASM decoder, a native addon over libmpv or VA-API all end in the same push.

7. The seam: a sprite whose content is a source

Sprites (extending.md, src/nodes/sprites.js, src/cocoa/sprites.js) are how an element hands parts of its drawing to the surface presenter, which lifts each onto a CALayer above the window's bitmap under promotion's rules: asked every frame after layout and before the damage is taken; refused when anything painted after the element reaches into the part, when the element or an ancestor fades, when a rounded clipping ancestor does not hold all of it, when it sits inside a <glarea>, or when a side exceeds 8192 device pixels; cut by a box layer with masksToBounds to every square clipping ancestor; a hole in the element's own paint from the frame spritesLifted says so; the bitmap under it claimed on the way onto a layer and off it. Today a part's content is always a raster: the presenter calls the part's paint into a surface of its own and hands that surface to the layer, and paints again when the version changes.

A video is a part whose content the presenter does not paint:

sprites() {
  if (!this.source || !this.frameRect) return null;
  return [{ key: 'video', rect: this.frameRect, clip: this.contentBox(), contents: this.source }];
}

• contents is a VideoFrames sink, or the player a <video src> made. A part with contents has no paint and no version, and for now no animations — nothing in §8 wants a video that fades on the render server, and a fade on the element itself is a refusal, as it is for every sprite.
• Lifted, the layer's content is the source's: for a sink, the ring surface push last wrote, repointed by push itself between frames (§6.3); for a player, the AVPlayerLayer the bridge made, added under the clip box at the part's place in the one order every layer above the bitmap shares. The element paints its bars and leaves the hole.
• Declined, the element draws the source's current frame itself: source.draw(ctx, rect), a drawImage of the sink's BGRA surface, or of the frame the bridge pulls from the player on request (AVPlayerItemVideoOutput, one verb, lazily made, §9). Every reason a sprite is declined is a reason a video is drawn for a while — a tooltip over it, a dialog's dim, a rounded card that does not hold it — and since declining is decided per frame, the cost lasts exactly as long as the reason.
• Everything else is promotion's, unchanged: the z-order test asked at the part's extent, the clip boxes, the claim of the bitmap under it, the one order with the promoted nodes. A contents part makes SpriteLayers skip its raster and ask the source for a layer or a surface, and that is the whole change on the presenter's side.

Why not a child window or layer of the element's own, the way <glarea> has one: because a <glarea>'s layer sits at a fixed zPosition above everything the presenter draws, is never clipped by an ancestor, and cannot be drawn over — <MediaPlayer>'s own documentation explains that its transport bar cannot be a sibling <box> for exactly this reason. A video in a form is content, and content lives in the paint order. The sprite seam is the one that lets a thing be on a layer when it can and in the bitmap when it must, and that is the property this element needs most.

Why not Node._promoted, promotion's own hole: because promotion lifts a plain <box> for an animation it can express, and re-rasters the node's children into the layer itself. A video's content is nobody's raster.

And why <video> is the first implementer and not only a consumer: the element's one part is itself, which is the simplest possible sprite — and it proves the contents kind for any registered element that wants to show a live source inside its own drawing, a chart with a camera feed in its corner being the obvious one. The X11, Wayland and win32 presenters never ask, so the same element draws itself there as it always would.

8. The two consumers

8.1 A <video> in a form

The element of §6, with nothing between the application and it. On macOS, <video src> with the player verbs: no JS in the frame path, audio and seeking from AVFoundation, lifted under the form's own popups and tooltips and drawn for the frames a dialog dims it. On X11, useSupports says no to src, and the application decodes into frames — the ecosystem's job to make that one line, below.

8.2 A <video> inside <Html>

@react-x11/components' <Html> is one registered element, <htmlview>, that parses, lays out and paints a document into the window's own context. Its <video> today is its poster as an image box, or a 300×150 frame with nothing in it; its <audio> is nothing at all; and it has no hook by which a host supplies a renderer for a tag. It lifts a CSS animation onto a layer through the sprite seam (#620), and it mounts real core widgets beside the document for form controls — a <textinput> at an <input>'s rect, cut to the box's clip and faded by its opacity, reported through onControls — on the rule its source states: a drawn control is a picture of a control.

A drawn video is a picture of a video, and the same rule gives the same answer. <Html> reports its media boxes the way it reports its controls — rect, clip, opacity, and the attributes src, poster, autoplay, loop, muted — and its React half mounts a core <video> beside the document at each, absolutely positioned and clipped exactly as a widget is. Everything the element knows applies: it lifts on macOS when its part is clear, it draws when it is not, it scrolls with the pane because its abs does, and src refuses where useSupports says so, which <Html> turns into the poster it shows today. onResource grows a kind: 'video' so a host that has a decoder can answer a URL with a frames sink instead, and a host that has none declines and gets the poster.

What <Html> decides for itself, and should say in its docs: a video box that something later in the document paints over — an overlay of controls, a caption positioned across it — is declined and stays a poster, the same paintedAfter rule its sprites already apply, because a sibling mounted above the document would hide the overlay. HTML's controls attribute is not drawn by <Html>: the transport bar an application wants is its own React, over or under the box, the position <MediaPlayer>'s docs already take. And <audio> is not this element: a sound with no box is a player with no presentation, which is a different feature and a smaller one.

8.3 <MediaPlayer>, and where a decoder comes from

<MediaPlayer> embeds mpv or VLC over <foreign>, drives it over mpv's JSON IPC, and is X11-only because --wid is an X mechanism; on macOS it reports unavailable. It keeps doing that: an application that already has mpv and an X display has a player with audio, subtitles and every codec, and nothing here replaces it. What changes is its ceiling. Its window stacks above everything drawn, so it cannot be scrolled under a header, dimmed by a dialog or overlaid by a transport bar — and a backend of it that decodes into a <video frames> sink instead of a window has none of those limits, on every backend, at the cost of the audio path and the sink's bytes. The shape of that backend is the ecosystem's decision, not this record's; the two candidates are an ffmpeg child process writing rawvideo to a pipe (pure JS, silent, every backend), and a native addon over libmpv's software render API (audio, seeking, subtitles, every backend, one prebuilt binary). Either one makes <MediaPlayer> the thing a React developer reaches for and <video> the primitive it stands on, which is the right order.

8.4 Camera, later

The same frames shape with the bytes from a capture session — on macOS the 2vuy surfaces §3.5 measured, as layer contents or as bytes — behind a createCameraFrames({ deviceId }) that owns device selection and the session's lifetime. The permission half needs nothing new. A separate piece of work, after this, on the same element.

9. Where the code goes

piece	where
the element	src/nodes/video.js beside image.js, the kind in src/nodes/kinds.js and src/Reconciler.js, VideoProps in src/types/elements.d.ts, a line in test/types/api.tsx; objectFit in src/styles.js for <video> and <image> both
the sink	src/videoframes.js — pure: formats, the ring, the conversion policy, preferredFormats — with a presentation per backend behind it: ntk Surface on X11, src/cocoa/video.js, src/win32/video.js, src/wayland/video.js
the sprite kind	the contents field in src/nodes/sprites.js's contract and src/node.d.ts; src/cocoa/sprites.js skips the raster for it; src/cocoa/promotion.js unchanged
the capability	mediaPlayback in src/appcontext.js's FEATURES
the player, Cocoa	src/cocoa/player.js over the verbs below, in the shape of src/cocoa/glarea.js
the player, win32	src/win32/player.js over IMFMediaEngine, after §4.3
bridge verbs, @windowkit/appkit	release A, the sink: createVideoSurface({ width, height, format, colorSpace, range }) → { handle, iosurfaceId } with the colour attachments set; writeVideoSurface(handle, planes), which interleaves I420 into NV12; an optional videoSurfacePlanes(handle) answering Buffers over the locked planes so a pipe read lands in the surface with no copy at all. Reuse, unchanged: setLayerContentsIOSurface, surfaceIsInUse, surface-released, releaseSurface. Release B, the player: createPlayer({ url }), playerSet(id, { rate, volume, muted, loop }), playerSeek(id, seconds), playerLayer(id) answering a layer handle addSublayer takes, playerCopyFrame(id) answering a BGRA surface for the drawn presentation, releasePlayer(id); events player-metadata, player-time, player-rate, player-ended, player-error. Release C, capture, later
bridge verbs, @windowkit/win32	after §4.3: mediaCreate, mediaSet, mediaSeek, mediaFrame (frame-server), mediaRelease and the engine's events; later mediaAttach onto a visual, over the pane host's CreateSurfaceFromHandle
Xv	a thin wrapper in ntk over node-x11's requests, port and format negotiation, ShmPutImage as the HD path, a clean decline where QueryAdaptors answers none
docs	a <video> section in elements.md; the ladder in macos.md and windows.md; the contents part in extending.md; this file is the record they point at
tests	test/video.test.js — pixels, fit, the poster, the refusal — on the mock and the in-process X server; test/cocoa-video.test.js over test/helpers/cocoa-bridge.js for which verbs a lift and a decline send; a real-bridge check through snapshotWindow; a sink scenario in scripts/bench/protocol.js and a rule in presenters-gate.json that a lifted sink paints no window frame per pushed frame
spikes	scripts/spikes/video/ — yuvscan.swift and sinkcost.swift, this PR

10. What is not here, and why

• A decoder. react-x11 will not ship one. On Cocoa and Windows src does not need one; everywhere, frames takes what the application decodes. Bundling ffmpeg would be the largest dependency in the project by an order of magnitude, and would land on every user of a toolkit most of whom want a button. §8.3 is where a decoder belongs.
• Audio for frames, and therefore A/V sync. A sink is silent frames, on every backend, and says so. src has audio wherever it has a player. A frames source that also carries audio is the libmpv shape of §8.3, and its sync is the decoder's.
• Seeking, buffering, HLS, DRM. All the platform player's where there is one and entirely absent otherwise. Anything the element exposes that only works with src is documented as src-only, in the way macos.md documents the rest of that backend's asymmetries.
• A compressed-stream source. AVSampleBufferDisplayLayer and its synchroniser are the right rung for one, no consumer has asked, and the verb count would double. When one does, it is a third source prop on the same element and the same sprite.
• <audio>. A player with no box is not this element.
• Fullscreen, picture-in-picture, the lock screen's now-playing. Window state is useWindowState()'s; the rest is desktop integration with its own ladder.
• Reading a lifted video's pixels through the X path. Like <glarea>, a lifted video is not in the window's bitmap. On macOS snapshotWindow sees it (§3.4); on X11 there is nothing lifted to miss.

11. Decisions the implementing PRs have to make, and the suggested answer

question	suggested answer
the seam	a sprite part with contents (§7), not a new accessor and not a child window; <video> offers itself as its one part
the element's primitive	frames, a sink, on every backend; src is the convenience of the backends with a player
src where there is no player	one typed onError, the poster stays, useSupports('mediaPlayback') to ask first — never a black box
the sink's native format	NV12 on macOS (two planes render, three do not), BGRA on X11 and win32 until their fast paths exist, anything on Wayland; published as preferredFormats
pixel format on Cocoa	YUV through a plain layer — not BGRA, measured at 2.1× the decode on playback and 2.67× the process CPU on capture for the same picture — and not an AV layer class for a sink, which §3.4 made unnecessary
colour	the sink names colorSpace and range; the bridge attaches them to the surface; the defaults are BT.709 video range, and a test pins that the bare and the attached surface differ, since §3.4 saw them differ
bridge verbs, Cocoa	two for the sink, six and five events for the player, in two releases (§9); the sink first, because it ships the whole element on macOS without AV
the frame clock	a lifted sink's push is not a react-x11 frame and is not priced (§6.3); a drawn sink's is, exactly like any other claim; a player's layer costs the frame nothing
the hole	the element's, through spritesLifted, as for every sprite; the bars are painted always, the picture only when drawn
rounded video corners	legal, and the drawn presentation; say so beside the prop rather than mask a layer that cannot be
X11 v1	BGRA frames into an ntk Surface with SHM; Xv second, into the backing pixmap if ntk will name it; DRI3 import as the real answer
Xv fallback	mandatory: QueryAdaptors answering none is every Mac, and the drawn path is the decline
win32 v1	IMFMediaEngine in frame-server mode, drawn in paint order; the swapchain visual waits for the Windows half of promotion
tests	the fake bridge for what reaches the natives; snapshotWindow for pixels on Cocoa, never render(in:); on X11 that the element paints the same with and without SHM; on every backend that a declined video and a lifted one show the same pixels
<Html>	mounts a core <video> beside the document, as it mounts a <textinput>; declines a box something paints over; grows onResource by kind: 'video'
camera	a frames source with device selection and a session, later, on the same element

12. What can be built now, and in what order

Five tracks. The first ships on every backend with no bridge release and no upstream; the next two are two @windowkit/appkit releases; the fourth waits for a Windows machine; the fifth is the ecosystem's.

1. Core, now. VideoFrames, <video frames> drawn through an ntk Surface on X11 and the shared 2D dialect on Cocoa and win32, NV12 and I420 converted to BGRA where a backend cannot take them, objectFit for <video> and <image>, poster, the typed refusal of src, useSupports('mediaPlayback') answering false everywhere, the contents sprite kind in src/cocoa/sprites.js behind a feature test for the verbs that do not exist yet. Pixel tests on the mock and the in-process server; a 720p sink scenario in the protocol bench, which is what makes the SHM path's effect visible. The whole element is usable from this step: an application with ffmpeg on its PATH plays a file on X11 and on macOS in forty lines, silently, at the drawn cost.
2. @windowkit/appkit release A — the sink's two verbs (§9). Then the Cocoa presenter lifts a <video frames> and a 1080p60 feed costs the process 25ms of CPU a second (§3.4.1) and react-x11 no frames at all. The acceptance test is the gate rule: a lifted sink paints no window frame per pushed frame, over the fake bridge; and a real-bridge snapshotWindow that shows the pushed colour.
3. @windowkit/appkit release B — the player. <video src> on macOS with audio, seeking and HLS from AVFoundation; mediaPlayback true there; the drawn presentation through playerCopyFrame so a dialog's dim or a rounded card still shows the picture. The first <video> a form developer meets is this one.
4. Windows, on the Windows machine. §4.3's probes, then src through IMFMediaEngine in frame-server mode and frames through a dynamic texture, both drawn; the swapchain visual and the NV12 swapchain when the backend has the Windows half of promotion to decide them.
5. @react-x11/components. <Html> reports media boxes and mounts <video>s; onResource learns video; <MediaPlayer> keeps mpv and VLC and grows a frames backend — ffmpeg first, since it is pure JS and silent is still a video, libmpv when someone wants audio on macOS.

After those, in order of value: the Xv rung on X11 (into the backing pixmap, if ntk will name it), the camera source, DRI3 import on X11 and a dma-buf subsurface on Wayland for the zero-copy decoders, and a compressed-stream source over AVSampleBufferDisplayLayer when a consumer appears.

13. The spikes

Every number above came from a program that uses the platform directly — no bridge, no renderer, no React — so that what it measures is the platform and not our use of it.

scripts/spikes/video/yuvscan.swift (2026-10-02) answers §3.4's on-screen question. It builds eight CALayers in a borderless window: BGRA and CGImage controls, 420v, 420f, 2vuy, y420 and f420 surfaces made through CVPixelBufferCreate with IOSurface backing and filled with one YCbCr colour per plane, an AVSampleBufferDisplayLayer fed thirty H.264 frames a VTCompressionSession encodes from solid red, and an AVPlayerLayer playing a .mov an AVAssetWriter writes from solid magenta. Two seconds in it captures the window through screencapture -l <windowNumber> — the window server's composite, the same source as CGWindowListCreateImage and therefore as the bridge's snapshotWindow, and the only capture a Swift program on this SDK is still allowed — and prints the centre pixel of each tile. Build and run with swiftc -O -swift-version 5 -o yuvscan yuvscan.swift && ./yuvscan; a capture that comes back empty is the Screen Recording permission, not the layer. The controls are the part that makes the YUV rows meaningful.

scripts/spikes/video/sinkcost.swift (2026-10-02) answers §3.4.1. It rotates three 1920×1080 surfaces, copies a prepared frame into each plane of the next one, repoints a layer at it inside a transaction, and paces to 60Hz; getrusage around 240 iterations gives the process CPU per frame, for NV12 and BGRA in turn, twice each. The memcpy stands in for the read from a decoder; a verb that answers Buffers over the locked planes removes even that.

The 2026-09-08 spikes behind §2.5, §3.1–3.3 and §3.5 are described there and were not committed; their shapes: encode synthetic frames with a VTCompressionSession to get a real stream with parameter sets, decode them back with and without BGRA forced, time each frame, check CVPixelBufferGetIOSurface, assign the surface to a layer two thousand times for the flip cost, probe rasterization with render(in:) against controls; push a 1920×1080 depth-24 image to an offscreen pixmap with core PutImage and a GetInputFocus round trip after each; run an AVCaptureSession at 1280×720 for 60 frames twice, native and BGRA, with getrusage around each run, from a bundle launched through LaunchServices so that the authorization request prompts, keeping nothing of what the camera saw.

14. Verdict

The Cocoa half got smaller in the month since this was first written, and for a good reason: the thing it was designed around turned out not to be there. A plain layer shows a YUV surface on screen, the seam that lifts part of an element onto a layer has shipped and been tested on something else, and the bridge can already address an IOSurface by id. What is left for a frame sink on macOS is two verbs and a field on a sprite; what is left for a URL is the player verbs, which are the first AVFoundation the bridge will link for anything but a permission prompt.

The X11 half is exactly as large as it was, and this record's main service there is still to say so before someone spends a week on it: the transport that exists in the protocol is capped near 30fps at 1080p, the one that would fix it is now installed and still absent on every Mac, and the one that is right is the largest piece of work here. A sink the application feeds, drawn through the surface every retained-surface element already draws through, is honest, is small, and is the version of this that ships on four backends at once.

Windows has every piece a player needs except a decision about stacking, which is promotion's to make there as it was on macOS, and a drawn presentation that needs no such decision. Wayland is a rasterizer, so its sink is a shader.

The thing not to build is still the API that hides the difference — and the thing not to build twice is the element: a video inside a document is the same <video> as one in a form, mounted where the document says it goes.