Introduction
In the Edition #4 - Understanding HLS: A Practical Guide for Swift Developers of Swift in Practice, we explored HLS fundamentals—how video files become master playlists, get segmented, and adapt quality dynamically during playback. We covered the protocol's architecture, playlist structures, and how AVFoundation handles adaptive streaming behind the scenes.
Since publishing that piece, the most frequent question I've received is: "When do we actually need Metal?"
It’s a fair question — and one that deserves more nuance than most articles offer. Here’s what I’ve observed working with teams building video experiences: there’s a persistent confusion about where Metal fits in the stack. People see “video” and “performance” and immediately think they need low-level GPU access.
But here’s the reality most of us miss:
Metal handles GPU compute and graphics. HLS is your streaming protocol. AVFoundation orchestrates the playback.
They’re not competing technologies — they work at completely different layers:
- HLS's domain: Managing the protocol, handling segmentation, adapting bitrates, serving playlists
- AVFoundation's domain: Decoding video, mixing audio, coordinating playback, leveraging hardware acceleration when available
- Metal's domain: Direct GPU programming for frame-level computation and custom rendering only when necessary
When I worked on a streaming client a few years ago, we introduced Metal too early — and playback got slower. Profiling revealed the bottleneck wasn’t GPU at all, but the network layer. That experience stuck with me.
So the real question isn’t “Should we use Metal?” It’s “What problem are we trying to solve that AVFoundation can’t already handle?”
The Architectural Misconception
Over the years, I’ve seen teams introduce Metal prematurely into video streaming architectures — driven by performance anxiety rather than profiling data. This decision invariably increases complexity, extends debugging cycles, and often degrades performance through unnecessary context switching between frameworks.
In reality, AVFoundation has been optimized for HLS playback across a decade of iOS releases. Its hardware utilization is near-optimal. When Metal remains transparent to your architecture, you’re benefiting from both correctness and performance.
When Metal Becomes Architecturally Necessary
Metal becomes relevant only when your requirements transcend standard playback into the realm of frame-level transformation:
- Real-time video effects — Applying filters, LUTs, or visual effects during playback.
- Custom rendering pipelines — Multi-camera compositions, overlays, or custom frame blending.
- GPU-accelerated encoding — On-device encoding pipelines that require GPU offloading.
- Advanced computer vision — Frame analysis for ML inference, temporal or spatial transformations.
These represent genuine architectural requirements where AVFoundation’s abstraction becomes a bottleneck, not a feature.
Metal Fundamentals: Framework Architecture
Understanding Metal requires grasping its architecture as a low-level abstraction over GPU resources:
MTLDevice — Represents the GPU. Immutable, thread-safe, your factory for all GPU resources.
MTLCommandQueue — Serializes GPU work; executes in order.
MTLCommandBuffer — GPU work packet; encoders populate and commit these.
MTLRenderPipelineState — Immutable GPU rendering configuration (shader bindings, attachment formats, rasterization state).
MTLTexture — GPU-resident memory. For video, typically BGRA or YCbCr, often created via CVMetalTextureCacheto minimize copies. Shaders — Compute kernels written in Metal Shading Language (MSL). Fragment shaders process pixels; compute shaders handle general-purpose GPU work.
Production Architecture: Real-Time Video Processing
Let’s translate this architecture into code you can actually use in production. The following implementation applies real-time effects to HLS streams while maintaining synchronization and resource efficiency.
import MetalKit
import AVFoundation
final class VideoFilterProcessor: NSObject {
private let metalDevice: MTLDevice
private let commandQueue: MTLCommandQueue
private let renderPipelineState: MTLRenderPipelineState
private let textureCache: CVMetalTextureCache?
private let processingQueue = DispatchQueue(label: "com.swiftinpractice.metal")
private var isProcessing = false
init?() {
guard let device = MTLCreateSystemDefaultDevice(),
let queue = device.makeCommandQueue(),
let library = device.makeDefaultLibrary(),
let vertexFunction = library.makeFunction(name: "vertexShader"),
let fragmentFunction = library.makeFunction(name: "colorGradingShader")
else { return nil }
metalDevice = device
commandQueue = queue
var cache: CVMetalTextureCache?
CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, device, nil, &cache)
textureCache = cache
let pipelineDescriptor = MTLRenderPipelineDescriptor()
pipelineDescriptor.vertexFunction = vertexFunction
pipelineDescriptor.fragmentFunction = fragmentFunction
pipelineDescriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
do {
renderPipelineState = try device.makeRenderPipelineState(descriptor: pipelineDescriptor)
} catch {
print("⚠️ Render pipeline creation failed: \(error)")
return nil
}
super.init()
}
func processFrame(_ pixelBuffer: CVPixelBuffer, completion: @escaping (CVPixelBuffer?) -> Void) {
guard !isProcessing else { completion(nil); return }
processingQueue.async { [weak self] in
guard let self = self else { return }
self.isProcessing = true
defer { self.isProcessing = false }
guard let sourceTexture = self.textureFromPixelBuffer(pixelBuffer),
let commandBuffer = self.commandQueue.makeCommandBuffer()
else { return completion(nil) }
let width = CVPixelBufferGetWidth(pixelBuffer)
let height = CVPixelBufferGetHeight(pixelBuffer)
let descriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .bgra8Unorm, width: width, height: height, mipmapped: false)
descriptor.usage = [.renderTarget, .shaderRead]
guard let outputTexture = self.metalDevice.makeTexture(descriptor: descriptor) else { return completion(nil) }
let renderPass = MTLRenderPassDescriptor()
renderPass.colorAttachments[0].texture = outputTexture
renderPass.colorAttachments[0].loadAction = .clear
renderPass.colorAttachments[0].clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 1)
renderPass.colorAttachments[0].storeAction = .store
guard let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPass) else { return completion(nil) }
encoder.setRenderPipelineState(self.renderPipelineState)
encoder.setFragmentTexture(sourceTexture, index: 0)
encoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
encoder.endEncoding()
commandBuffer.addCompletedHandler { _ in
completion(self.pixelBufferFromTexture(outputTexture))
}
commandBuffer.commit()
}
}
private func textureFromPixelBuffer(_ pixelBuffer: CVPixelBuffer) -> MTLTexture? {
guard let textureCache = textureCache else { return nil }
var temp: CVMetalTexture?
let width = CVPixelBufferGetWidth(pixelBuffer)
let height = CVPixelBufferGetHeight(pixelBuffer)
CVMetalTextureCacheCreateTextureFromImage(nil, textureCache, pixelBuffer, nil, .bgra8Unorm, width, height, 0, &temp)
return temp.flatMap { CVMetalTextureGetTexture($0) }
}
private func pixelBufferFromTexture(_ texture: MTLTexture) -> CVPixelBuffer? {
var pixelBuffer: CVPixelBuffer?
CVPixelBufferCreate(nil, texture.width, texture.height, kCVPixelFormatType_32BGRA, nil, &pixelBuffer)
guard let buffer = pixelBuffer, let base = CVPixelBufferGetBaseAddress(buffer) else { return nil }
CVPixelBufferLockBaseAddress(buffer, .writeOnly)
texture.getBytes(base, bytesPerRow: CVPixelBufferGetBytesPerRow(buffer),
from: MTLRegionMake2D(0, 0, texture.width, texture.height), mipmapLevel: 0)
CVPixelBufferUnlockBaseAddress(buffer, .writeOnly)
return buffer
}
}Production Metal Shader
#include <metal_stdlib>
using namespace metal;
vertex float4 vertexShader(uint vid [[vertex_id]]) {
static const float2 coords[3] = {
float2(-1.0, -1.0),
float2( 3.0, -1.0),
float2(-1.0, 3.0)
};
return float4(coords[vid], 0.0, 1.0);
}
fragment float4 colorGradingShader(float4 pos [[position]], texture2d<float> src [[texture(0)]]) {
constexpr sampler s(mag_filter::linear, min_filter::linear, address::clamp_to_edge);
float2 uv = pos.xy / float2(src.get_width(), src.get_height());
float4 color = src.sample(s, uv);
float3 adjusted = clamp(color.rgb * 1.1, 0.0, 1.0);
return float4(adjusted, color.a);
}📊 Performance Considerations & Profiling
Metal’s power demands discipline.
- 🚧 GPU Pipeline Stalls: texture.getBytes() is synchronous — prefer async readback via MTLBlitCommandEncoder.
- 🧹 Memory Pressure: Flush CVMetalTextureCache regularly.
- ⚙️ Command Buffer Overhead: Batch GPU work where possible.
- 🕰️ Synchronization: Use pull-based timing with AVPlayerItemVideoOutput.copyPixelBuffer(forItemTime:) instead of CADisplayLink loops.
Profile with: - Metal System Trace — visualize GPU load - Xcode GPU Frame Debugger — inspect shaders - Memory Viewer — track texture allocations
The Strategic Decision
Use Metal when: - You’re building creative tools requiring real-time frame effects - You need per-frame ML or vision analysis - Profiling shows pixel operations as the bottleneck - The complexity investment brings measurable product value
Avoid Metal when: - You’re doing standard HLS playback - Network or I/O dominates your performance profile - You lack GPU programming expertise or time to maintain shaders
In short:
Architectural clarity beats premature optimization.
💬 Join the Conversation
This decision framework — choosing the right tool over the most powerful one — separates shipping products from engineering complexity. If you’re navigating similar architectural decisions in production apps, I’d love to hear your perspective.
💡 Have you encountered real production scenarios where Metal became necessary? What did your profiling data show? How did your team make the transition?
About Swift in Practice
Swift in Practice is a weekly newsletter exploring production-grade architectures, Apple frameworks, and the real decisions behind high-performance iOS apps.
📩 Subscribe for deep dives into AVFoundation, Metal, Core ML, and beyond.
👋 Author
Bruno Valente Pimentel — Senior iOS Architect & Consultant 25+ years building at the intersection of code, creativity, and communication.
Currently open to collaborations and senior roles focused on video infrastructure, streaming systems, and Apple Intelligence-driven media tools.
☕ Enjoying Swift in Practice? It's free and reader-supported. If an edition saved you some debugging time, you can buy me a coffee: https://buymeacoffee.com/swiftinpractice
Originally published as a Swift in Practice edition on LinkedIn. Comments and discussion happen there.