dockkit

DockKit Framework for integrating with motorized camera stands and gimbals that physically track subjects by rotating the iPhone. DockKit handles motor control, subject detection, and framing so camera apps get 360-degree pan and 90-degree tilt tracking with no additional code. Apps can override system tracking to supply custom observations, control motors directly, or adjust framing. iOS 17+, Swift 6.3. Contents Setup Discovering Accessories System Tracking Custom Tracking Framing and Region of Interest Motor Control Animations Tracking State and Subject Selection Accessory Events Battery Monitoring Common Mistakes Review Checklist References Setup Import DockKit: import DockKit DockKit requires a physical DockKit-compatible accessory and a real device. The Simulator cannot connect to dock hardware. No special entitlements or Info.plist keys are required. The framework communicates with paired accessories automatically through the DockKit system daemon. The app must use AVFoundation camera APIs. DockKit hooks into the camera pipeline to analyze frames for system tracking. Discovering Accessories Use DockAccessoryManager.shared to observe dock connections: import DockKit func observeAccessories ( ) async throws { for await stateChange in try DockAccessoryManager . shared . accessoryStateChanges { switch stateChange . state { case . docked : guard let accessory = stateChange . accessory else { continue } // Accessory is connected and ready configureAccessory ( accessory ) case . undocked : // iPhone removed from dock handleUndocked ( ) @unknown default : break } } } accessoryStateChanges is an AsyncSequence that emits DockAccessory.StateChange values. Each change includes: state -- .docked or .undocked accessory -- the DockAccessory instance (present when docked) trackingButtonEnabled -- whether the physical tracking button is active Accessory Identity Each DockAccessory has an identifier with: name -- human-readable accessory name category -- .trackingStand (currently the only category) uuid -- unique identifier for this accessory Hardware details are available via firmwareVersion and hardwareModel . System Tracking System tracking is DockKit's default mode. When enabled, the system analyzes camera frames through built-in ML inference, detects faces and bodies, and drives the motors to keep subjects in frame. Any app using AVFoundation camera APIs benefits automatically. Enable or Disable // Enable system tracking (default) try await DockAccessoryManager . shared . setSystemTrackingEnabled ( true ) // Disable system tracking for custom control try await DockAccessoryManager . shared . setSystemTrackingEnabled ( false ) Check current state: let isEnabled = DockAccessoryManager . shared . isSystemTrackingEnabled System tracking state does not persist across app termination, reboots, or background/foreground transitions. Set it explicitly whenever the app needs a specific value. Tap to Select Subject Allow users to select a specific subject by tapping: // Select the subject at a screen coordinate try await accessory . selectSubject ( at : CGPoint ( x : 0.5 , y : 0.5 ) ) // Select specific subjects by identifier try await accessory . selectSubjects ( [ subjectUUID ] ) // Clear selection (return to automatic selection) try await accessory . selectSubjects ( [ ] ) Custom Tracking Disable system tracking and provide your own observations when using custom ML models or the Vision framework. Providing Observations Construct DockAccessory.Observation values from your inference output and pass them to the accessory at 10-30 fps: import DockKit import AVFoundation func processFrame ( _ sampleBuffer : CMSampleBuffer , accessory : DockAccessory , device : AVCaptureDevice ) async throws { let cameraInfo = DockAccessory . CameraInformation ( captureDevice : device . deviceType , cameraPosition : device . position , orientation : . corrected , cameraIntrinsics : nil , referenceDimensions : nil ) // Create observation from your detection output let observation = DockAccessory . Observation ( identifier : 0 , type : . humanFace , rect : detectedFaceRect , // CGRect in normalized coordinates faceYawAngle : nil ) try await accessory . track ( [ observation ] , cameraInformation : cameraInfo ) } Observation Types Type Use .humanFace Preserves system multi-person tracking and framing optimizations .humanBody Full body tracking .object Arbitrary objects (pets, hands, barcodes, etc.) The rect uses normalized coordinates with a lower-left origin (same coordinate system as Vision framework -- no conversion needed). Camera Information DockAccessory.CameraInformation describes the active camera. Set orientation to .corrected when coordinates are already relative to the bottom-left corner of the screen. Optional cameraIntrinsics and referenceDimensions improve tracking accuracy. Track variants also accept [AVMetadataObject] instead of observations, and an optional CVPixelBuffer for enhanced tracking. Framing and Region of Interest Framing Modes Control how the system frames tracked subjects: try await accessory . setFramingMode ( . center ) Mode Behavior .automatic System decides optimal framing .center Keep subject centered (default) .left Frame subject in left third .right Frame subject in right third Read the current mode: let currentMode = accessory . framingMode Use .left or .right when graphic overlays occupy part of the frame. Region of Interest Constrain tracking to a specific area of the video frame: // Normalized coordinates, origin at upper-left let squareRegion = CGRect ( x : 0.25 , y : 0.0 , width : 0.5 , height : 1.0 ) try await accessory . setRegionOfInterest ( squareRegion ) Read the current region: let currentROI = accessory . regionOfInterest Use region of interest when cropping to a non-standard aspect ratio (e.g., square video for conferencing) so subjects stay within the visible area. Motor Control Disable system tracking before controlling motors directly. Angular Velocity Set continuous rotation speed in radians per second: import Spatial // Pan right at 0.2 rad/s, tilt down at 0.1 rad/s let velocity = Vector3D ( x : 0.1 , y : 0.2 , z : 0.0 ) try await accessory . setAngularVelocity ( velocity ) // Stop all motion try await accessory . setAngularVelocity ( Vector3D ( ) ) Axes: x -- pitch (tilt). Positive tilts down on iOS. y -- yaw (pan). Positive pans right. z -- roll (if supported by hardware). Set Orientation Move to a specific position over a duration: let target = Vector3D ( x : 0.0 , y : 0.5 , z : 0.0 ) // Yaw 0.5 rad let progress = try accessory . setOrientation ( target , duration : . seconds ( 2 ) , relative : false ) Also accepts Rotation3D for quaternion-based orientation. Set relative: true to move relative to the current position. The returned Progress object tracks completion. Motion State Monitor the accessory's current position and velocity: for await state in accessory . motionStates { let positions = state . angularPositions // Vector3D let velocities = state . angularVelocities // Vector3D let time = state . timestamp if let error = state . error { // Motor error occurred } } Setting Limits Restrict range of motion and maximum speed per axis: let yawLimit = try DockAccessory . Limits . Limit ( positionRange : - 1.0 ..< 1.0 , // radians maximumSpeed : 0.5 // rad/s ) let limits = DockAccessory . Limits ( yaw : yawLimit , pitch : nil , roll : nil ) try accessory . setLimits ( limits ) Animations Built-in character animations that move the dock expressively: // Disable system tracking before animating try await DockAccessoryManager . shared . setSystemTrackingEnabled ( false ) let progress = try await accessory . animate ( motion : . kapow ) // Wait for completion while ! progress . isFinished && ! progress . isCancelled { try await Task . sleep ( for : . milliseconds ( 100 ) ) } // Restore system tracking try await DockAccessoryManager . shared . setSystemTrackingEnabled ( true ) Animation Effect .yes Nodding motion .no Shaking motion .wakeup Startup-style motion .kapow Dramatic pendulum swing Animations start from the accessory's current position and execute asynchronously. Always restore tracking state after completion. Tracking State and Subject Selection iOS 18+ exposes ML-derived tracking signals through trackingStates . Each TrackingState contains a timestamp and a list of TrackedSubjectType values ( .person or .object ). Persons include speakingConfidence , lookingAtCameraConfidence , and saliencyRank (1 = most important, lower is more salient). for await state in accessory . trackingStates { for subject in state . trackedSubjects { switch subject { case . person ( let person ) : let speaking = person . speakingConfidence // 0.0 - 1.0 let saliency = person . saliencyRank case . object ( let object ) : let saliency = object . saliencyRank } } } Use selectSubjects(_:) to lock tracking onto specific subjects by UUID. Pass an empty array to return to automatic selection. See references/dockkit-patterns.md for speaker-tracking and saliency filtering recipes. Accessory Events Physical buttons on the dock trigger events: for await event in accessory . accessoryEvents { switch event { case . cameraShutter : // Start or stop recording / take photo break case . cameraFlip : // Switch front/back camera break case . cameraZoom ( factor : let factor ) : // factor > 0 means zoom in, < 0 means zoom out break case . button ( id : let id , pressed : let pressed ) : // Custom button with identifier break @unknown default : break } } For first-party apps (Camera, FaceTime), shutter/flip/zoom events work automatically. Third-party apps receive these events and implement behavior through AVFoundation. Battery Monitoring Monitor the dock's battery status (iOS 18+). A dock can report multiple batteries, each identified by name : for await battery in accessory . batteryStates { let level = battery . batteryLevel // 0.0 - 1.0 let charging = battery . chargeState // .charging, .notCharging, .notChargeable let low = battery . lowBattery } Common Mistakes DON'T: Control motors without disabling system tracking // WRONG -- system tracking fights manual commands try await accessory . setAngularVelocity ( velocity ) // CORRECT -- disable system tracking first try await DockAccessoryManager . shared . setSystemTrackingEnabled ( false ) try await accessory . setAngularVelocity ( velocity ) DON'T: Assume tracking state persists across lifecycle events // WRONG -- state may have reset after backgrounding func applicationDidBecomeActive ( ) { // Assume custom tracking is still active } // CORRECT -- re-set tracking state on foreground func applicationDidBecomeActive ( ) { Task { try await DockAccessoryManager . shared . setSystemTrackingEnabled ( false ) } } DON'T: Call track() outside the recommended rate // WRONG -- calling once per second is too slow try await accessory . track ( observations , cameraInformation : cameraInfo ) // (called at 1 fps) // CORRECT -- call at 10-30 fps // Hook into AVCaptureVideoDataOutputSampleBufferDelegate for per-frame calls DON'T: Forget to restore tracking after animations // WRONG -- tracking stays disabled after animation try await DockAccessoryManager . shared . setSystemTrackingEnabled ( false ) let progress = try await accessory . animate ( motion : . kapow ) // CORRECT -- restore tracking when animation completes try await DockAccessoryManager . shared . setSystemTrackingEnabled ( false ) let progress = try await accessory . animate ( motion : . kapow ) while ! progress . isFinished && ! progress . isCancelled { try await Task . sleep ( for : . milliseconds ( 100 ) ) } try await DockAccessoryManager . shared . setSystemTrackingEnabled ( true ) DON'T: Use DockKit in Simulator DockKit requires a physical DockKit-compatible accessory. Guard initialization and provide fallback behavior when no accessory is available. Review Checklist import DockKit present where needed Subscribed to accessoryStateChanges to detect dock/undock events Handled both .docked and .undocked states System tracking disabled before custom tracking or motor control System tracking restored after animations complete Custom observations supplied at 10-30 fps Observation rect uses normalized coordinates (lower-left origin) Camera information matches the active capture device @unknown default handled in all switch statements over DockKit enums Motion limits set if restricting accessory range of motion Tracking state re-applied after app returns to foreground Accessory events handled for physical button integration Battery state monitored when showing dock status to user No DockKit code paths executed in Simulator builds References Extended patterns (Vision integration, service architecture, custom animations): references/dockkit-patterns.md DockKit framework DockAccessoryManager DockAccessory Controlling a DockKit accessory using your camera app Track custom objects in a frame Modify rotation and positioning programmatically Integrate with motorized iPhone stands using DockKit -- WWDC23 What's new in DockKit -- WWDC24