My app ran at 60 FPS in testing. Clean UI. No Instruments warnings.
Then we shipped. Within 3 days: “The app stutters after a minute” “My phone is burning up” “It crashes when I get a call”
Everything was “correct.” Tests passed. Architecture approved.
So what broke?
Lessons from shipping real-time camera features to production — and fixing what we missed.
Scope & Constraints
This article focuses on: - real-time camera pipelines (30–60 FPS) - iPhone hardware (A9 → A17 class devices) — principles apply broadly - GPU-accelerated processing using Metal
Offline processing, photo-only workflows, and Core ML–heavy pipelines follow very different trade-offs.
The Story
The first time I shipped a real-time Metal-based camera pipeline, everything looked great in testing.
60 FPS on my iPhone 6s. Clean UI. Instruments showed no obvious warnings.
Then we shipped.
Within three days, support tickets started rolling in: - “The app stutters after a minute” (iPhone 5s, iOS 10) - “My phone gets really hot” (iPhone 6, multiple iOS versions) - “It crashes when I get a call” (more devices than I’d like to admit)
The code was correct. Unit tests passed. The architecture review was approved.
But we missed something fundamental: we optimized for our test devices, not the long tail.
The fix wasn’t one thing. It was a series of uncomfortable realizations about memory, heat, and camera hardware I thought I understood.
What follows are the lessons from fixing it — the production version, not the demo. The version that has to work on an older iPhone, under sustained load, in a warm environment.
1️⃣ The Mental Model: Memory Flow, Not Images
The intuitive mental model for camera data goes like this:
- The camera captures a “frame”
- We get an “image”
- We process it
- We display it
This mental model quietly burns a large portion of your frame budget.
The actual pipeline
AVCaptureOutput
→ CMSampleBuffer (timing + metadata)
→ CVPixelBuffer (a memory region, often IOSurface-backed)
→ [optional] CVMetalTextureCache (zero-copy GPU mapping)
→ Metal compute
→ Presentation / EncodingThe key insight: every arrow can be close to zero-copy if structured correctly — or cost multiple milliseconds per frame if it isn’t.
For years, I made reasonable-sounding choices: - using UIImage for convenience - converting everything to BGRA “to simplify things” - copying buffers defensively
Each decision was easy to justify. Together, they made 60 FPS unsustainable on real devices.
Understanding this mental model changes what you optimize first. Next: where most performance is actually won or lost — at the capture layer.
2️⃣ Camera Output: Where Performance Is Decided
The capture configuration is where most performance wins — or losses — are locked in.
let output = AVCaptureVideoDataOutput()
output.videoSettings = [
kCVPixelBufferPixelFormatTypeKey as String:
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
]
output.alwaysDiscardsLateVideoFrames = true
output.setSampleBufferDelegate(self, queue: captureQueue)Why this matters:
- 420f matches the camera’s native format
- avoids implicit color-space conversions
- reduces memory bandwidth pressure
If you start with BGRA for convenience, you’re already paying a tax you may never recover from.
About alwaysDiscardsLateVideoFrames:
This is a design decision, not a performance trick. Setting it to true means: “I’d rather skip frames than build up latency.” For real-time processing, this is almost always correct. For recording, it might not be.
There’s no universal right answer. But there is a right answer for your use case, and you need to know which one you’re building.
3️⃣ CMSampleBuffer vs CVPixelBuffer: Know What You’re Touching
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
return
}
process(pixelBuffer)
}CMSampleBuffer provides: - timing - attachments - lifecycle context
CVPixelBuffer is the actual image memory.
You schedule work using the sample buffer. You do heavy processing on the pixel buffer.
Mixing those responsibilities is a common source of subtle bugs.
A bug I’ve hit: We were storing sample buffers in an array to implement a “rewind 5 seconds” feature. Worked fine at 30 FPS. At 60 FPS, we’d accumulate 300 buffers in 5 seconds, memory pressure would spike, and the app would jettison everything.
The fix was to extract only what we needed (the pixel buffer + presentation timestamp) and let the sample buffer go.
4️⃣ CVPixelBuffer: Memory, Not an Image
CVPixelBuffer is not an image object. It’s a lockable region of memory, often backed by an IOSurface.
CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
defer {
CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)
}Common production mistakes: - locking on the main thread - holding locks longer than necessary - copying buffer contents “just in case”
Each unnecessary copy costs: - latency - battery - thermal headroom
Usually all three.
The uncomfortable truth: If you’re copying pixel buffers, something earlier in your architecture probably needs fixing. In my experience, copying is rarely the right solution — it’s usually a sign that the underlying ownership model needs work.
5️⃣ Zero-Copy: Bridging Pixel Buffers to Metal
This is where real-time pipelines either scale — or quietly fall apart.
var textureCache: CVMetalTextureCache!
CVMetalTextureCacheCreate(nil, nil, device, nil, &textureCache)// Per frame - creating textures from 420f format
// You need TWO textures: one for Y plane, one for UV plane
var yTexture: CVMetalTexture?
CVMetalTextureCacheCreateTextureFromImage(
kCFAllocatorDefault,
textureCache,
pixelBuffer,
nil,
.r8Unorm, // Y plane is single channel
width,
height,
0, // plane index 0
&yTexture
)
var uvTexture: CVMetalTexture?
CVMetalTextureCacheCreateTextureFromImage(
kCFAllocatorDefault,
textureCache,
pixelBuffer,
nil,
.rg8Unorm, // UV plane is two channels
width / 2,
height / 2,
1, // plane index 1
&uvTexture
)
guard let yTex = CVMetalTextureGetTexture(yTexture),
let uvTex = CVMetalTextureGetTexture(uvTexture) else {
return
}
// Your compute shader now has access to both planes
// If you need BGRA, do the YUV→RGB conversion in the shaderWhen done correctly: - no memory copy - shared backing store - predictable latency
Important: Color space handling CVPixelBuffers from the camera often have color space attachments (e.g., BT.709, P3). If you’re displaying processed frames, preserve these attachments or you’ll get color shifts that are subtle in testing but obvious in production. Use CVBufferGetAttachment() to read and CVBufferSetAttachment() to preserve them.
The alternative (that you’ll find in production code):
// DON'T DO THIS
let bytes = CVPixelBufferGetBaseAddress(pixelBuffer)
let texture = device.makeTexture(descriptor: descriptor)
texture.replace(region: region, mipmapLevel: 0, withBytes: bytes, bytesPerRow: bytesPerRow)If you’re converting to BGRA and copying, this copies ~8MB per frame at 1080p — that’s nearly 500MB/second at 60 FPS. Even with YUV 420f (~3MB per frame), copying is still 180MB/second. You’ll thermal throttle quickly.
This is the difference between something that works in isolation and something that survives production traffic.
Sound familiar? These patterns show up everywhere once you know what to look for.
6️⃣ Metal: Throughput Over Graphics
For many camera workloads, rendering isn’t the hard part. Throughput is.
Compute shaders offer: - predictable cost - high parallelism - explicit control
guard let commandBuffer = commandQueue.makeCommandBuffer() else {
return
}
guard let encoder = commandBuffer.makeComputeCommandEncoder() else {
return
}
encoder.setComputePipelineState(pipelineState)
encoder.setTexture(yTexture, index: 0)
encoder.setTexture(uvTexture, index: 1)
encoder.setTexture(outputTexture, index: 2)
let threadgroupSize = MTLSize(width: 16, height: 16, depth: 1)
let threadgroupCount = MTLSize(
width: (width + threadgroupSize.width - 1) / threadgroupSize.width,
height: (height + threadgroupSize.height - 1) / threadgroupSize.height,
depth: 1
)
encoder.dispatchThreadgroups(threadgroupCount, threadsPerThreadgroup: threadgroupSize)
encoder.endEncoding()
commandBuffer.commit()Compute shader example:
kernel void processFrame(
texture2d<float, access::read> yTexture [[texture(0)]],
texture2d<float, access::read> uvTexture [[texture(1)]],
texture2d<float, access::write> output [[texture(2)]],
uint2 gid [[thread_position_in_grid]]
) {
if (gid.x >= output.get_width() || gid.y >= output.get_height()) {
return;
}
float y = yTexture.read(gid).r;
uint2 uvCoord = uint2(gid.x / 2, gid.y / 2);
float2 uv = uvTexture.read(uvCoord).rg;
// YUV to RGB conversion (BT.709)
float r = y + 1.5748 * (uv.y - 0.5);
float g = y - 0.1873 * (uv.x - 0.5) - 0.4681 * (uv.y - 0.5);
float b = y + 1.8556 * (uv.x - 0.5);
output.write(float4(r, g, b, 1.0), gid);
}This kind of work — color conversion, normalization, basic filtering — runs fastest on the GPU because it’s embarrassingly parallel.
7️⃣ Threading: Where Things Actually Break
Here’s the setup that works reliably:
// Capture queue: should be serial to maintain frame order
private let captureQueue = DispatchQueue(
label: "com.yourapp.capture",
qos: .userInteractive
)
// Processing queue: can be serial or concurrent depending on your needs
private let processingQueue = DispatchQueue(
label: "com.yourapp.processing",
qos: .userInitiated,
attributes: .concurrent
)
output.setSampleBufferDelegate(self, queue: captureQueue)func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
return
}
processingQueue.async { [weak self] in
guard let self = self else { return }
let result = self.process(pixelBuffer)
DispatchQueue.main.async { [weak self] in
self?.displayResults(result)
}
}
}Rules I follow: - capture queue ≠ processing queue - processing queue ≠ main thread - UI receives results, never raw buffers
A failure mode I’ve debugged: The processing queue would dispatch back to main with the pixel buffer to update a preview. At 30 FPS, the main thread could keep up. At 60 FPS, the main run loop was spending 80% of its time updating Metal layer contents, and UI became completely unresponsive.
The fix was to render on the processing queue and only signal the main thread that a new frame was ready.
A note on async/await: You might wonder why this uses GCD instead of Swift’s modern concurrency. The reason is performance: Task creation has measurable overhead (typically 100-200μs depending on the workload and system state). At 60 FPS, you have a 16.67ms budget per frame — every microsecond counts on the critical path. Additionally, AVFoundation’s camera APIs are fundamentally callback-based with no native async/await variant. You need precise control over queue QoS and serial vs concurrent behavior that async/await abstracts away.
Modern concurrency excels at structured concurrency, cancelation, and actor isolation — but real-time camera pipelines have different constraints. For exposing results to SwiftUI, you can bridge this GCD-based pipeline to an AsyncStream, giving you the best of both worlds: GCD’s performance on the critical path, and async/await’s ergonomics at the presentation layer. This hybrid approach is increasingly common in performance-critical iOS codebases.
About Sendable and strict concurrency checking: If you’re targeting Swift 6, you’ll need to mark your buffer processing closures and types as @Sendable and ensure CVPixelBuffer handling doesn’t trigger data race warnings. Since CVPixelBuffer is a Core Foundation type with manual memory management, wrap it in a @unchecked Sendable container with proper synchronization. The good news: this forces you to be explicit about ownership, which usually surfaces bugs early.
8️⃣ Production Failure Modes
Thermal Throttling: The Silent Performance Killer
What happens:
Around 85-92°C internal temp (varies by device), iOS starts throttling the GPU clock. On iPhone 6, this can drop GPU performance by 40%. Your carefully tuned 60 FPS pipeline suddenly runs at 35 FPS.
The insidious part: it doesn’t crash. It doesn’t error. It just gets slower.
Effective mitigation:
NotificationCenter.default.addObserver(
forName: ProcessInfo.thermalStateDidChangeNotification,
object: nil,
queue: .main
) { [weak self] _ in
self?.adjustForThermalState()
}
func adjustForThermalState() {
switch ProcessInfo.processInfo.thermalState {
case .nominal, .fair:
targetFrameRate = 60
case .serious:
targetFrameRate = 30
case .critical:
targetFrameRate = 15
captureSession.beginConfiguration()
captureSession.sessionPreset = .medium
captureSession.commitConfiguration()
@unknown default:
break
}
}This isn’t about performance anymore. It’s about graceful degradation.
Lifecycle Management: When Everything Breaks
Phone calls. Control Center. Picture-in-Picture. FaceTime. Notification Center. Screen recording.
Any of these can interrupt your capture session. The delegate methods fire in surprising orders. Sometimes you get sessionWasInterrupted but not sessionInterruptionEnded. Sometimes you get neither and the session just stops.
The only pattern that’s worked reliably:
private var captureSessionRunningObserver: NSKeyValueObservation?
func setupSessionMonitoring() {
captureSessionRunningObserver = captureSession.observe(
\.isRunning,
options: [.new]
) { [weak self] session, change in
guard let self = self else { return }
if change.newValue == false {
self.handleSessionStopped()
} else {
self.handleSessionStarted()
}
}
}
deinit {
captureSessionRunningObserver?.invalidate()
}KVO on isRunning catches everything. Notifications are too unreliable.
GPU Starvation and Latency Spikes
The scenario: You’re submitting one Metal command buffer per frame. Under normal conditions, fine. Under load (user switches apps, notification arrives, background process starts), the GPU scheduler falls behind. Command buffers queue up. Latency spikes from 16ms to 200ms.
Users perceive this as “frozen” even though frames are still being processed.
Effective mitigation:
private let gpuSemaphore = DispatchSemaphore(value: 3) // Max 3 in-flight frames
func submitFrame() {
guard let commandBuffer = commandQueue.makeCommandBuffer() else {
return
}
gpuSemaphore.wait()
commandBuffer.addCompletedHandler { [weak self] _ in
self?.gpuSemaphore.signal()
}
guard let encoder = commandBuffer.makeComputeCommandEncoder() else {
gpuSemaphore.signal()
return
}
// ... encode operations ...
encoder.endEncoding()
commandBuffer.commit()
}This creates backpressure. If the GPU can’t keep up, we drop frames at the source instead of building up latency.
Memory Leaks via Retain Cycles
While AVCaptureVideoDataOutput holds its delegate weakly, retain cycles still happen frequently through callback captures. The most common pattern: capturing self strongly in blocks dispatched from the delegate callback.
// WRONG - Common retain cycle
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
processingQueue.async {
// 'self' is captured strongly here if you use any properties/methods
self.process(pixelBuffer) // Cycle if processingQueue is owned by self
}
}
// CORRECT - Break the cycle
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
processingQueue.async { [weak self] in
guard let self = self else { return }
self.process(pixelBuffer)
}
}
// Always clean up in deinit
deinit {
captureSession.stopRunning()
output.setSampleBufferDelegate(nil, queue: nil)
captureSessionRunningObserver?.invalidate()
}I’ve debugged this exact issue multiple times. It’s easy to miss in the moment, even when you know better.
9️⃣ When NOT to Use This Pipeline
Senior engineers know when not to optimize.
Use Core Image instead if: - Built-in filters do what you need (they’re hardware-accelerated and well-tested) - 10-15ms latency is acceptable - You’re not running at 60 FPS
Use Accelerate instead if: - You’re doing CPU-side processing (vDSP operations, BLAS) - You need precise numerical results (GPU floating point has different rounding)
Use Vision framework instead if: - You need face detection, barcode scanning, object tracking - Latency doesn’t need to be <10ms - You want Apple’s ML models
Don’t optimize at all if: - Current performance is acceptable - You haven’t profiled to find the bottleneck - You’re guessing about what’s slow
The best optimization is often the one you don’t ship.
🔍 How to Actually Debug This
When things go wrong (and they will), here’s the Instruments workflow:
- Time Profiler — Find which thread is blocking
- Allocations — Find memory growth (often retained pixel buffers)
- Metal System Trace — Find GPU stalls and overdraw
- Thermal State — Find when throttling begins
- Leaks — Find retain cycles (though CVPixelBuffer leaks don’t always show up here)
Red flags to look for: - Allocations growing linearly with time → buffer retention - Main thread showing heavy CPU → queue architecture problem - GPU idle while CPU busy → upload bottleneck (copying instead of zero-copy) - GPU busy while frames drop → thermal throttling or incorrect semaphore limit
Closing — What Actually Matters
Real-time computer vision on iOS isn’t about clever shaders or obscure APIs.
It’s about:
- memory ownership
- scheduling discipline
- thermal and power trade-offs
- knowing when not to optimize
Once you internalize the pipeline, you stop fighting the platform — and start working with it.
This same architecture underpins camera apps, live streaming, video editors, and real-time analysis tools — just with different constraints.
What’s next?
I’m debating between these for Edition #12:
- Building a production face detection pipeline (Vision + Metal integration)
- Multi-camera capture patterns for iOS 17+
- How to instrument and monitor real-time pipelines in production
Which would be most useful to you? Reply and let me know what you’re working on.
P.S. If you’re debugging camera issues right now, the #1 thing to check: are you creating new pixel buffers, or are you reusing the camera’s buffer pool? That one change fixed 80% of the thermal issues I’ve seen.
About the Author
Bruno Valente Pimentel is a Principal iOS Architect & Consultant with 25+ years in software development and 15+ years specializing in the Apple ecosystem. He has delivered 30+ iOS apps (7 featured by Apple, including App of the Month and App of the Year projects) and mentored 30+ developers, with ~75% advancing to senior/lead roles.
Starting in the iPhone 3G/3GS era, Bruno has witnessed and mastered every evolution of iOS development—from OpenGL ES to Metal, from manual memory management to ARC, from UIKit to SwiftUI. He specializes in iOS architecture, AR/VR (ARKit, RealityKit), accessibility, AI/ML (Core ML, Vision, on-device model optimization), and real-time computer vision pipelines. He has worked with global brands including Financial Times, Bosch, NATO, and MTV, and recently provided technical advisory on AI-powered computer vision for aerospace clients.
As a WWDC attendee (5+ editions) and former CocoaHeads Rio de Janeiro chapter leader (8+ years), Bruno combines deep technical expertise with a commitment to knowledge-sharing and community building.
☕ Enjoying Swift in Practice? It's free and reader-supported. If an edition saved you some debugging time, you can buy me a coffee: https://buymeacoffee.com/swiftinpractice
Originally published as a Swift in Practice edition on LinkedIn. Comments and discussion happen there.