ARKit face tracking is shrouded in a ton of myth and mystique.
I reverse engineered the internal face tracking API, and this is how it actually works.
This is based on reverse engineering the API of Apple's face detection framework and testing it with my own code. Some of it is educated guesswork based on names and strings. I did not reverse engineer the details of how ARKit/FaceKit works, but given what I've seen, I'm quite confident in the overall big picture.
No AI/LLMs were used during this research. Yes, really.
ARKit is just a high-level API that Apple provides. The actual face tracking component is called FaceKit, and it has a lower level (and arguably simpler) API that also provides a lot more insight into how it actualy works.
FaceKit is an internal API that does two primary things:
- User identity tracking/faceprint generation
- Facial pose and blendshape tracking
To do this, it uses two Espresso ML models, called CaraNet and IDFitting. CaraNet is the actual face tracking we're interested in. IDFitting is basically "Face ID lite": it lets FaceKit track who each detected face is, to tell multiple users apart.
There is also an identity model tensor (tensor.bin), which is also part of the user identity tracking.
Oh yeah, FaceKit can track up to 4 faces at once.
To use FaceKit, you need to do trivial face detection on your own (using any CV library), then pass both the full frame and one rect/roll for each detected face. FaceKit will crop/rotate the faces from the frame and feed them into the ML models and postprocessing, and give you back the processed data.
You also provide the camera intrinsics (basically the camera lens projection) and extrinsics (camera position and location in world space). This doesn't need to be real data, you can just make one up based on a given FOV. It's just to give the face position in real world units, it does not affect the face tracking.
FaceKit can run on CPU, GPU, or ANE (Neural Engine).
This is what you get from FaceKit, per tracked face:
-
Detection confidence
-
User identity:
- A UUID assigned to the user, to tell users apart (there is some mechanism to save a database of faceprints so that users can be persistently recognized later)
- The identity coefficients (faceprint). This is not a photo, it's some abstract thing.
-
Face animation data (THIS IS WHAT WE WANT ):
- The 51 blend shape values. Yes, 51. Tongue is separate.
- The gaze point (the point in space the user is looking at)
- L/R eye angles
- Postprocessed values for blink left/right
- The TongueOut value
-
Face pose info (rotation, translation)
-
Face geometry data:
- Landmark positions (66 landmarks)
- Eye positions (?)
- Face mesh (1200 vertices)
-
Time-smoothed versions of the above
-
eyeBlink_L
-
eyeBlink_R
-
eyeSquint_L
-
eyeSquint_R
-
eyeLookDown_L
-
eyeLookDown_R
-
eyeLookIn_L
-
eyeLookIn_R
-
eyeWide_L
-
eyeWide_R
-
eyeLookOut_L
-
eyeLookOut_R
-
eyeLookUp_L
-
eyeLookUp_R
-
browDown_L
-
browDown_R
-
browInnerUp
-
browOuterUp_L
-
browOuterUp_R
-
jawOpen
-
mouthClose
-
jawLeft
-
jawRight
-
jawForward
-
mouthUpperUp_L
-
mouthUpperUp_R
-
mouthLowerDown_L
-
mouthLowerDown_R
-
mouthRollUpper
-
mouthRollLower
-
mouthSmile_L
-
mouthSmile_R
-
mouthDimple_L
-
mouthDimple_R
-
mouthStretch_L
-
mouthStretch_R
-
mouthFrown_L
-
mouthFrown_R
-
mouthPress_L
-
mouthPress_R
-
mouthPucker
-
mouthFunnel
-
mouthLeft
-
mouthRight
-
mouthShrugLower
-
mouthShrugUpper
-
noseSneer_L
-
noseSneer_R
-
cheekPuff
-
cheekSquint_L
-
cheekSquint_R
-
RightEyeOuterCorner
-
RightEyeInnerCorner
-
RightEyeLowerOuter
-
RightEyeLowerInner
-
RightEyeUpperOuter
-
RightEyeUpperInner
-
RightEyePupil
-
LeftEyeOuterCorner
-
LeftEyeInnerCorner
-
LeftEyeLowerOuter
-
LeftEyeLowerInner
-
LeftEyeUpperOuter
-
LeftEyeUpperInner
-
LeftEyePupil
-
RightBrowOuter
-
RightBrowMiddle
-
RightBrowInner
-
LeftBrowOuter
-
LeftBrowMiddle
-
LeftBrowInner
-
MouthRightCorner
-
MouthRightUp1
-
MouthRightUp2
-
MouthRightPhiltrum
-
MouthCenterPhiltrum
-
MouthLeftPhiltrum
-
MouthLeftUp2
-
MouthLeftUp1
-
MouthLeftCorner
-
MouthLeftDown1
-
MouthLeftDown2
-
MouthCenterLower
-
MouthRightDown2
-
MouthRightDown1
-
MouthInnerUp
-
MouthInnerDown
-
MouthInnerUpRight
-
MouthInnerUpLeft
-
MouthInnerDownRight
-
MouthInnerDownLeft
-
NoseRidgeRoot
-
NoseRidge1
-
NoseRidge2
-
NoseRidgeTip
-
NoseBaseLeft
-
NoseBaseCenterLeft
-
NoseBaseCenter
-
NoseBaseCenterRight
-
NoseBaseRight
-
NoseAlaLeft1
-
NoseAlaRight1
-
NoseAlaLeft2
-
NoseAlaRight2
-
CheekRight0
-
CheekRight1
-
CheekRight2
-
CheekRight3
-
CheekRight4
-
CheekRight5
-
ChinCenter
-
CheekLeft0
-
CheekLeft1
-
CheekLeft2
-
CheekLeft3
-
CheekLeft4
-
CheekLeft5
{
// These two (angle and rect) are an input from face detection
AngleInfoRoll = "-1.742425";
Rect = {
Height = "0.3914024531841278";
Width = "0.2935518324375153";
X = "0.4109302163124084";
Y = "0.1959651857614517";
};
confidence = "0.999623";
"confidence_level" = 950;
debug = {
"faceprint_snapshot_coefficients" = {length = 512, bytes = [... snip ...] };
"identity_snapshot_coefficients" = {length = 256, bytes = [... snip ...]};
"recognition_framework_version" = "Vision:9.5";
"rgb_crop" = "<CVPixelBuffer 0x75753c640 width=128 height=128 bytesPerRow=128 pixelFormat=L008 [...snip...]";
};
"exceeded_default_thresholds_mask" = 0;
"face_id" = "[ snip UUID ]";
"failure_type" = 0;
"identity_coefficients" = {length = 2004, bytes = [snip] };
"raw_data" = {
animation = {
blendshapes = {length = 204, bytes = 0x00000000 00000000 cfc2f63c a0b2f63c ... 1727cd3d 7387b73d };
gaze = (
"40.1123",
"-77.92969",
"960.9375"
);
"left_eye_pitch" = "0.1119128";
"left_eye_yaw" = "0.009138897";
"right_eye_pitch" = "0.111593";
"right_eye_yaw" = "0.07646886";
"studio_blink_left" = 0;
"studio_blink_right" = 0;
"tongue_out" = "2.873686e-07";
"virtual_blink_left" = 0;
"virtual_blink_right" = 0;
};
pose = {
rotation = (
(
"0.9879566",
"0.009564473",
"-0.1544346"
),
(
"-0.05812307",
"-0.902053",
"-0.4276938"
),
(
"-0.1433989",
"0.4315192",
"-0.8906334"
)
);
translation = (
"36.36377",
"-40.08634",
"408.1648"
);
};
};
// These are given as input
"rgb_camera" = {
extrinsics = {
rotation = (
(
1,
0,
0
),
(
0,
1,
0
),
(
0,
0,
1
)
);
translation = (
0,
0,
0
);
};
intrinsics = (
(
"400.0000000000001",
0,
"319.5"
),
(
0,
"400.0000000000001",
"239.5"
),
(
0,
0,
1
)
);
};
"smooth_data" = {
animation = {
blendshapes = {length = 204, bytes = 0x00000000 00000000 b9f51e3d 2bf31e3d ... 742cba3d 523aae3d };
gaze = (
"41.39601",
"-74.61397",
"958.7535"
);
"left_eye_pitch" = "0.1086617";
"left_eye_yaw" = "0.01053559";
"right_eye_pitch" = "0.1083396";
"right_eye_yaw" = "0.07801434";
"studio_blink_left" = 0;
"studio_blink_right" = 0;
"tongue_out" = "2.271057e-07";
"virtual_blink_left" = 0;
"virtual_blink_right" = 0;
};
geometry = {
landmarks = {length = 528, bytes = 0x4fac9943 eb032443 c093a643 c9302043 ... 5628c543 e3ad8643 };
"left_eye" = (
"31.56276",
"27.20679",
"25.45219"
);
"right_eye" = (
"-31.56293",
"27.20665",
"25.45242"
);
vertices = {length = 14640, bytes = 0x7bb3303d 72b6d3c1 eef36342 194a773d ... 7da041c2 28232a42 };
};
pose = {
rotation = (
(
"0.9888245",
"0.01015947",
"-0.1487369"
),
(
"-0.0548001",
"-0.9030608",
"-0.4260025"
),
(
"-0.1386464",
"0.4293926",
"-0.8924119"
)
);
translation = (
"36.20768",
"-39.92029",
"407.9984"
);
};
};
timestamp = {
epoch = 0;
flags = 3;
timescale = 30000;
value = 65624071935;
};
};
Comes in 2 variants: caranet and caranet_plus. The inputs to caranet are:
network_inputs_image_2_0_bridge: fp16 [1, 3, 128, 128]: This is a 128x128RGB crop of the face to be trackednetwork_inputs_var_0_bridge: fp16 [1, 10, 1, 1]: Not sure what this is (variance?)
The outputs are:
bs_latent_output: fp16 [1, 35, 1, 1]: The blend shapes. Yes, there's only 35!angles_output: [1, 3, 1, 1]: Face angle (pitch/yaw/roll)translation_output: [1, 3, 1, 1]: Face translation (x/y/z)gaze_output: [1, 3, 1, 1]: The gaze point (x/y/z)failure_logit_output: [1, 1, 1, 1]: Face tracking failure indicatortongue_logits_output: [1, 2, 1, 1]: Some kind of raw tongue values
These are postprocessed to produce the final values returned from FaceKit. I don't know exactly how much postprocessing is done.
caranet_plus adds these outputs:
kas_outputbasic_ens_output
I don't know what these two extra ones do.
caranet is available on all iOS and macOS devices. caranet_plus is only available on some/all iOS devices, not on macOS.
caranet_plus is enabled by an obfuscated flag and returns the output data using obfuscated key names, so it is probably "not ready for production use yet" and instead some kind of unreleased future feature.
The blend shape feature works identically in both versions
CaraNet has around 1.4M parameters (fp16).
The ID fitting model comes in 2 variants: id_fitting_multiframe_rgb_only and id_fitting_multiframe_rgbd. macOS only ships with the RGB model, while iOS devices with a Face ID camera include the RGB+Depth model.
Inputs:
network_inputs_reshape_1_0_bridge: fp16 [1, 9, 1, 1]: Not surenetwork_inputs_reshape_0_bridge: fp16 [1, 51, 1, 1]: Not surenetwork_inputs_latents_input_0_bridge: fp16 [1, 64, 1, 1]: Likely feedback from the previous framenetwork_inputs_image_2_0_bridge: fp16 [1, 3, 128, 128]: 128x128 RGB image
Outputs:
latents_output: fp16 [1, 64, 1, 1]: Should beidentity_snapshot_coefficientsin the raw data (raw data dump has fp16 resolution despite being fp32).neutral_pca_subset_coeffs_output: fp16, [1, 30, 1, 1]: Probably meansprincipal component analysis . This should be the data that is used to keep track of who a user is, by matching against previous outputs to decide whether it's still close enough (the same person) or a new identity.
The ID fitting model is only run periodically (once per second or so), to detect user identity. It can be completely disabled, which does not affect blend shape tracking.
2026-11-03 22:22:42.579546+0900 0x268324f Default 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] new track for user (0xb9ec70d38 uuid=unset) at (311,227), bbox size (260,260)
2026-11-03 22:22:42.657094+0900 0x268324f Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] using strict error thresholds until tracking is stable (user (0xb9ec70d38 uuid=unset))
2026-11-03 22:22:42.848307+0900 0x268324f Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] using strict error thresholds until tracking is stable (user (0xb9ec70d38 uuid=unset))
2026-11-03 22:22:42.927304+0900 0x268324f Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] using strict error thresholds until tracking is stable (user (0xb9ec70d38 uuid=unset))
2026-11-03 22:22:42.927349+0900 0x268324f Default 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] request faceprint for user (0xb9ec70d38 uuid=unset) at (332,224), bbox size (229,229), pose=0.925327 -0.0387898 -0.377181 5.36671
2026-11-03 22:22:43.007121+0900 0x268324f Default 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] Unknown user, assigned new UUID (0xb9ec70d38 uuid=BFF70D36-503E-4031-A352-8AF80B42BC28). New DB size 1. Closest mismatch n/a, d=inf
2026-11-03 22:22:43.010290+0900 0x268324f Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] (0xb9ec70d38 uuid=BFF70D36-503E-4031-A352-8AF80B42BC28): No cluster representation. Adding first cluster.
2026-11-03 22:22:43.024179+0900 0x2683246 Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] Using RGB
2026-11-03 22:22:43.025286+0900 0x2683246 Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] fit was successful
2026-11-03 22:22:43.828586+0900 0x268324f Default 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] request faceprint for user (0xb9ec70d38 uuid=BFF70D36-503E-4031-A352-8AF80B42BC28) at (331,221), bbox size (227,227), pose=0.924486 -0.0442936 -0.378635 3.13637
2026-11-03 22:22:43.885019+0900 0x268324f Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] (0xb9ec70d38 uuid=BFF70D36-503E-4031-A352-8AF80B42BC28): Cluster index 0 match -> adding point. Total points in cluster: 2
2026-11-03 22:22:43.886284+0900 0x2683246 Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] Using RGB
2026-11-03 22:22:43.887148+0900 0x2683246 Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] fit was successful
2026-11-03 22:22:45.331658+0900 0x268324f Default 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] request faceprint for user (0xb9ec70d38 uuid=BFF70D36-503E-4031-A352-8AF80B42BC28) at (330,223), bbox size (229,229), pose=0.918111 -0.0529584 -0.392768 2.0347
2026-11-03 22:22:45.385879+0900 0x268324f Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] (0xb9ec70d38 uuid=BFF70D36-503E-4031-A352-8AF80B42BC28): Cluster index 0 match -> adding point. Total points in cluster: 3
2026-11-03 22:22:45.387347+0900 0x2683246 Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] Using RGB
2026-11-03 22:22:45.388311+0900 0x2683246 Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] fit was successful
2026-11-03 22:22:46.905673+0900 0x268324f Default 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] request faceprint for user (0xb9ec70d38 uuid=BFF70D36-503E-4031-A352-8AF80B42BC28) at (327,224), bbox size (227,227), pose=0.892131 -0.0465612 -0.449371 -0.758928
2026-11-03 22:22:46.959915+0900 0x268324f Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] (0xb9ec70d38 uuid=BFF70D36-503E-4031-A352-8AF80B42BC28): Cluster index 0 match -> adding point. Total points in cluster: 4
2026-11-03 22:22:46.961332+0900 0x2683246 Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] Using RGB
2026-11-03 22:22:46.962296+0900 0x2683246 Debug 0x0 97987 0 FKTracker: (AppleCVA) [com.apple.AppleCVA:FaceKit] fit was successful
The CaraNet model outputs 35 blend shape values. These correspond to ARKit blend shapes, except that 8 of them have a signed value and are expanded into pairs of unsigned blend shapes:
- eyeBlink_L, eyeWide_L
- eyeBlink_R, eyeWide_R
- browDown_L, browOuterUp_L
- browDown_R, browOuterUp_R
- jawLeft, jawRight
- mouthSmile_L, mouthFrown_L
- mouthSmile_R, mouthFrown_R
- mouthLeft, mouthRight
If you are consuming ARKit blend shape data, that means that you can subtract the above pairs of blend shapes safely. Each pair can be uniquely represented as a -1..+1 value, they will never both be nonzero at the same time.
That accounts for 43. The remaining 8 are the eye angle blend shapes, which are (presumably) computed from the gaze point output (the eye pitch/yaw values returned by the FaceKit API are presumably also computed this way):
- eyeLookDown_L
- eyeLookDown_R
- eyeLookIn_L
- eyeLookIn_R
- eyeLookOut_L
- eyeLookOut_R
- eyeLookUp_L
- eyeLookUp_R
That's how you get the 51 FaceKit blend shapes from the raw ML model output. Add tongue (which has some kind of postprocessing), and that's the famous 52 "ARKit" blend shapes.
Note that this isn't just a straight mapping, there is additional postprocessing/filtering going on (in code) on top of this (you can tell because the blend shape output floats are not quantized to fp16 precision).
The face mesh data and landmark data are computed entirely from the blend shape data. As far as I can tell, it works like this:
- Blendshapes are applied to the base face quad mesh (as you would in any 3D shape key flow)
- The landmark positions are computed by weighing 1-3 vertices from the mesh for each landmark
So, this part of the data is all synthetic. This is the opposite to MediaPipe, which computes facial landmarks first and then people munge that data into ARKit-compatible blend shapes. That is probably one reason why MediaPipe is so much worse at this: VTubers want blend shape data, and ARKit/FaceKit computes it directly from the RGB face image, while MediaPipe does not and then people have to map that data to blend shapes.
No. ARKit blend shapes are computed entirely from an RGB image. The depth data from the Face ID/TrueDepth camera (structured light depth capture) is not used for iPhone ARKit blend shape tracking. IR image data (flood illuminated) is not used at all in FaceKit. Period.
FaceKit also works on macOS using any old webcam, using a private API.
The depth data is ONLY USED FOR THESE TWO THINGS:
- To improve the performance of the user identity detection (like Face ID) by switching to the RGBD variant of the IDFitting model.
- To "refine" the pose output from RGB tracking using depth data. This improves the face position/angle data. I'm pretty sure this has nothing to do with blend shapes, only the face position (especially distance to camera, which is otherwise guessed from the size of the face).
Because the selfie camera on iPhones is pretty good in low light. That's all. It has nothing to do with the depth/IR/Face ID stuff.
If you use a webcam with worse low light performance, then it'll be worse. If you use a fancy camera with better low light performance, then it'll be better.
Because nobody has developed a webcam tracker app that's as good (and lightweight) as Apple's FaceKit.
As FaceKit is part of iOS/macOS, it cannot be (legally) used on other OSes.
Only the API functions, to understand how to use it. I haven't reverse engineered/decompiled the actual processing part of FaceKit. The ML models are plain text networks stored in /System/Library/PrivateFrameworks/AppleCVA.framework/Versions/Current/Resources/resources_facekit and the input/output tensor shapes come from that. The blendshape latent output to final blendshape mappings (unsigned and signed) are in blendshape_indices.binary (the format is easy to guess by inspection, it's just two concatenated arrays mapping 1->1 and 1->2 indices)).