Skip to content

Portrait: live blur, f-stop control, editable depth photos, 24MP HDR stills - #275

Open
darioalessandro wants to merge 10 commits into
masterfrom
portrait-apple-parity
Open

darioalessandro wants to merge 10 commits into
masterfrom
portrait-apple-parity

Conversation

@darioalessandro

Copy link
Copy Markdown
Member

Portrait shipped in #271 as a first pass: a sharp preview, a fixed f/4.5 blur baked into a 12MP still, and no way to change it afterwards. This PR takes it most of the way to Apple's Camera. 16.1.0 never went out, so the Portrait wire was free to change and is redone here rather than patched.

What you get

  • The blur shows while you frame. In Portrait the camera streams depth alongside the video and blurs each preview frame before it goes to the remote (and the Watch). Outside Portrait the preview path is unchanged.
  • An f-stop ruler on the remote. f/1.4 to f/16, Apple's Depth Control range, with full stops marked. It is a rig setting: every camera gets the same aperture, and a camera that joins or swaps lenses is corrected to match.
  • Editable by default. The camera saves the capture file as it came off the sensor, with depth and the person, hair, skin, teeth and glasses mattes inside, so Photos can change the blur later. A Baked option in the rig tray renders the blur into the photo instead. Never both files.
  • Better Baked stills. 24MP instead of 12, the mattes feed the blur, and HDR is kept: the blur is rendered in HDR and saved with a fresh gain map, where before the re-encode silently dropped it.

How it behaves

  • Editable stills come out at the sensor's 4:3. Cropping to another ratio would re-encode the file and throw the depth away, so the aspect setting only applies to Baked.
  • Switching Photo to Portrait still keeps the same camera format; turning depth on pauses the preview for about half a second (measured, it is the camera reconfiguring).
  • Portrait only appears when every camera in the rig can do it.

Why not Apple's Cinematic renderer

WWDC26 suggests turning on Cinematic capture for a free live blur. I measured it on an iPhone 18 Pro before building anything: stills from a Cinematic session are video-sized (1080p or 4K), carry no depth and no HDR gain map, it only runs on the single Wide camera, and zoom is capped at 3x. The plain photo session with a depth output kept 24MP, depth and the gain map with video at 29.7 fps, so that is what this builds on.

Wire

  • CommandAction.SetPortrait = 36, carrying portrait_aperture and portrait_output (0 / Unknown = keep).
  • PortraitOutput { Unknown, Baked, Editable }.
  • CameraState.portrait: PortraitState { aperture, min_aperture, max_aperture, output }. Its presence is the capability; supports_portrait is gone.

What was checked

  • The full unit suite passes on the simulator and the app builds for Mac Catalyst.
  • New tests cover the wire round trip, the camera answering and refusing SetPortrait, the rig fan-out at 1, 2 and 4 cameras, the ruler range intersection, the Editable/Baked/crop decision, the 24MP pick, the HDR encode, and the preview blur's focus and radius maths on synthetic depth.
  • The design choices above come from a hardware probe on an iPhone 18 Pro (iOS 27.2).

Not verified yet

The new hardware tests compile but have not run on a phone. Before this ships, someone needs to see on a real device:

  • how the live blur looks, whether the depth map lines up with the frame, and whether the preview holds 30 fps in Portrait
  • that Photos offers Depth Control on an Editable still
  • a Baked still keeping HDR, and 24MP with depth through the app
  • the landscape layout with Manual exposure and Portrait both on (three vertical rulers plus zoom)
  • the new strings in 15 languages, which no native speaker has read

Known gaps, for follow-ups: a preview frame dropped for back-pressure is blurred before it is dropped; a Portrait shot whose result lands after switching to Photo gets a plain crop; front-camera Portrait, Portrait Lighting and the subject badge are not in this PR.

🤖 Generated with Claude Code

darioalessandro and others added 10 commits November 11, 2026 23:33
CommandAction.SetPortraitAperture = 36 carries the f-number in
CommandParameters.portrait_aperture. CameraState.portrait reports the
applied aperture and the range the camera accepts, present exactly when
supports_portrait is set. A command with no f-number is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ture

PortraitRenderer.render(_:aperture:aspectRatio:) is the still blur that
lived in CaptureEngine, now at a given f-number. Its range comes from
CIDepthBlurEffect's inputAperture attributes (f/1.4...f/16 if absent),
default f/4.5. The engine stores the clamped aperture, reports it in
PortraitState and uses it for every Portrait still.

PortraitPreviewBlur is a pass-through stub for the live preview blur.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CameraControlling.setPortraitAperture forwards through CameraRig to the
engine. The coordinator applies it in every phase but taking a picture
(busy) and not-a-camera, answering once with the state tagged
.setportraitaperture. Docs list it with the other control commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he capability

CameraState.portrait replaces supports_portrait: a camera that can take
Portrait stills reports the block, one that can't omits it. SetPortrait
(action 36) carries portrait_aperture (0 = keep) and portrait_output
(Unknown = keep). PortraitOutput is Baked or Editable; the camera opens
Editable at f/4.5 and reports both. The aperture range is f/1.4...f/16,
narrowed to what CIDepthBlurEffect accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ig settings

Portrait's aperture and output are rig settings, like the aspect: one
value each, sent with SetPortrait to every camera that reports a Portrait
block and differs, re-sent to a camera that joins or comes back from a
lens swap with something else, and counted in flight like every control
command. The rig opens at what the first camera reports, and its ruler
spans the range every camera accepts; a narrower camera narrows the rig.
The screen reads the cameras' reports, never the tap.

In Portrait mode the director shows an f-stop ruler (the exposure
rulers' column pill, configured: log2, 1/3-stop clicks, full stops tall)
and the rig tray gets an Editable / Baked picker with a one-line
explanation. The mode labels and the new strings are in all 15 locales.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In Portrait the camera attaches an AVCaptureDepthDataOutput next to the
photo depth delivery and keeps the newest disparity map. On the preview
lane each frame is blurred with CIMaskedVariableBlur: the subject's
disparity (median of the map's middle third) stays sharp, the radius
grows away from it and with a wider aperture. Output keeps the input's
pixel format and size. Outside Portrait the path is one atomic load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Editable (the default) delivers the capture file untouched, with depth,
the portrait matte and the hair/skin/teeth/glasses mattes embedded, so
Photos offers Depth Control. One pure decision, PortraitRenderer.delivery,
picks bake, pass-through or plain crop.

Portrait now asks for every offered semantic matte (on after depth, off
before it) and embeds them per shot. depthBlurEffectFilter(forImageData:)
loads disparity, matte, hair and glasses on its own; a test pins the
inputs. A Baked still from a HEIC with a gain map runs the blur on the
expanded HDR image too and Core Image writes a fresh gain map, keeping the
capture's metadata and orientation (iOS 18+, HEIC; SDR path otherwise).

The Portrait photo ceiling rises to 24MP, always picked from the format's
own supportedMaxPhotoDimensions. The hardware test now covers both outputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant