ARKit/FaceKit notes A developer reverse engineered Apple's internal FaceKit face-tracking API, revealing that ARKit's face tracking is built on two Espresso ML models, CaraNet and IDFitting, plus an identity tensor. FaceKit tracks up to four faces at once, returning 51 blendshape values, gaze points, eye angles, 66 landmarks, and a 1200-vertex face mesh, and can run on CPU, GPU, or the Neural Engine. The researcher noted no AI/LLMs were used in the work. ARKit face tracking is shrouded in a ton of myth and mystique. I reverse engineered the internal face tracking API, and this is how it actually works. This is based on reverse engineering the API of Apple's face detection framework and testing it with my own code. Some of it is educated guesswork based on names and strings. I did not reverse engineer the details of how ARKit/FaceKit works, but given what I've seen, I'm quite confident in the overall big picture. No AI/LLMs were used during this research. Yes, really. ARKit is just a high-level API that Apple provides. The actual face tracking component is called FaceKit, and it has a lower level and arguably simpler API that also provides a lot more insight into how it actualy works. FaceKit is an internal API that does two primary things: - User identity tracking/faceprint generation - Facial pose and blendshape tracking To do this, it uses two Espresso ML models, called CaraNet and IDFitting. CaraNet is the actual face tracking we're interested in. IDFitting is basically "Face ID lite": it lets FaceKit track who each detected face is, to tell multiple users apart. There is also an identity model tensor tensor.bin , which is also part of the user identity tracking. Oh yeah, FaceKit can track up to 4 faces at once . To use FaceKit, you need to do trivial face detection on your own using any CV library , then pass both the full frame and one rect/roll for each detected face. FaceKit will crop/rotate the faces from the frame and feed them into the ML models and postprocessing, and give you back the processed data. You also provide the camera intrinsics basically the camera lens projection and extrinsics camera position and location in world space . This doesn't need to be real data, you can just make one up based on a given FOV. It's just to give the face position in real world units, it does not affect the face tracking. FaceKit can run on CPU, GPU, or ANE Neural Engine . This is what you get from FaceKit, per tracked face: - Detection confidence - User identity: - A UUID assigned to the user, to tell users apart there is some mechanism to save a database of faceprints so that users can be persistently recognized later - The identity coefficients faceprint . This is not a photo, it's some abstract thing. - Face animation data THIS IS WHAT WE WANT : - The 51 blend shape values. Yes, 51. Tongue is separate. - The gaze point the point in space the user is looking at - L/R eye angles - Postprocessed values for blink left/right - The TongueOut value - Face pose info rotation, translation - Face geometry data: - Landmark positions 66 landmarks - Eye positions ? - Face mesh 1200 vertices - Time-smoothed versions of the above - eyeBlink L - eyeBlink R - eyeSquint L - eyeSquint R - eyeLookDown L - eyeLookDown R - eyeLookIn L - eyeLookIn R - eyeWide L - eyeWide R - eyeLookOut L - eyeLookOut R - eyeLookUp L - eyeLookUp R - browDown L - browDown R - browInnerUp - browOuterUp L - browOuterUp R - jawOpen - mouthClose - jawLeft - jawRight - jawForward - mouthUpperUp L - mouthUpperUp R - mouthLowerDown L - mouthLowerDown R - mouthRollUpper - mouthRollLower - mouthSmile L - mouthSmile R - mouthDimple L - mouthDimple R - mouthStretch L - mouthStretch R - mouthFrown L - mouthFrown R - mouthPress L - mouthPress R - mouthPucker - mouthFunnel - mouthLeft - mouthRight - mouthShrugLower - mouthShrugUpper - noseSneer L - noseSneer R - cheekPuff - cheekSquint L - cheekSquint R - RightEyeOuterCorner - RightEyeInnerCorner - RightEyeLowerOuter - RightEyeLowerInner - RightEyeUpperOuter - RightEyeUpperInner - RightEyePupil - LeftEyeOuterCorner - LeftEyeInnerCorner - LeftEyeLowerOuter - LeftEyeLowerInner - LeftEyeUpperOuter - LeftEyeUpperInner - LeftEyePupil - RightBrowOuter - RightBrowMiddle - RightBrowInner - LeftBrowOuter - LeftBrowMiddle - LeftBrowInner - MouthRightCorner - MouthRightUp1 - MouthRightUp2 - MouthRightPhiltrum - MouthCenterPhiltrum - MouthLeftPhiltrum - MouthLeftUp2 - MouthLeftUp1 - MouthLeftCorner - MouthLeftDown1 - MouthLeftDown2 - MouthCenterLower - MouthRightDown2 - MouthRightDown1 - MouthInnerUp - MouthInnerDown - MouthInnerUpRight - MouthInnerUpLeft - MouthInnerDownRight - MouthInnerDownLeft - NoseRidgeRoot - NoseRidge1 - NoseRidge2 - NoseRidgeTip - NoseBaseLeft - NoseBaseCenterLeft - NoseBaseCenter - NoseBaseCenterRight - NoseBaseRight - NoseAlaLeft1 - NoseAlaRight1 - NoseAlaLeft2 - NoseAlaRight2 - CheekRight0 - CheekRight1 - CheekRight2 - CheekRight3 - CheekRight4 - CheekRight5 - ChinCenter - CheekLeft0 - CheekLeft1 - CheekLeft2 - CheekLeft3 - CheekLeft4 - CheekLeft5 { // These two angle and rect are an input from face detection AngleInfoRoll = "-1.742425"; Rect = { Height = "0.3914024531841278"; Width = "0.2935518324375153"; X = "0.4109302163124084"; Y = "0.1959651857614517"; }; confidence = "0.999623"; "confidence level" = 950; debug = { "faceprint snapshot coefficients" = {length = 512, bytes = ... snip ... }; "identity snapshot coefficients" = {length = 256, bytes = ... snip ... }; "recognition framework version" = "Vision:9.5"; "rgb crop" = "