Gaze Detection and Gaze Tracking Explained

Roboflow published a guide explaining gaze detection and gaze tracking, defining gaze detection as determining where a person is looking and gaze tracking as continuously following gaze over time to analyze visual attention. The guide details the components of a gaze tracking system, including visual input capture, and demonstrates how to build a gaze tracking pipeline using Roboflow Workflows.

Gaze detection determines where a person is looking, while gaze tracking continuously follows their gaze over time to understand visual attention and behavior. This guide explains how gaze tracking with computer vision works, and shows how to build a complete gaze tracking pipeline using Roboflow Workflows. Computers can now understand where a person is looking using computer vision https://blog.roboflow.com/intro-to-computer-vision/ and AI. From eye-tracking systems used in human-computer interaction to cameras that estimate where a driver is looking, gaze tracking and gaze detection are becoming increasingly useful across industries. These technologies analyze a person's eyes, face, and head position to estimate their visual attention. They can help determine whether someone is looking at a screen, a product, a road, or a specific object in their environment. In this guide, you'll learn what gaze detection and gaze tracking are, how these systems work, where they are used, and how to build a gaze tracking pipeline using Roboflow tools. What Is Gaze Detection? Gaze detection is the process of determining the direction or target of a person's visual attention. A gaze detection system typically analyzes visual information such as eye position, pupil location, iris orientation, head pose, facial landmarks, and eye movement. By combining these visual cues, a gaze detection system estimates where a person is looking. Gaze detection does not necessarily require extremely precise measurements. In some applications, it may only need to classify gaze into broad categories, such as whether a person is looking at the screen or looking away. What Is Gaze Tracking? Gaze tracking goes a step further then gaze detection by continuously estimating a person's gaze over time. Instead of simply determining where someone is looking at a particular moment, a gaze tracking system can record the movement and trajectory of their gaze. In simple terms, gaze detection answers "Where is the person looking?" while gaze tracking answers "Where has the person been looking, and how has their gaze moved over time?" A gaze-tracking system should capture information such as: - Where a person looks first - How long they look at a particular area - Where they look next - How their gaze moves between objects - How frequently they return to a particular area This creates a gaze trajectory that can be analyzed to understand visual attention and behavior. Gaze tracking is therefore particularly useful when the goal is to study how visual attention changes over time. How Does a Gaze Tracking System Work? A gaze tracking system combines computer vision, deep learning, and temporal analysis to determine where a person is looking over time. While a gaze estimation model predicts the direction of a person's gaze, it is only one component of the overall system. Given below are the main components of a gaze tracking system and how they work together to estimate and track gaze over time. 1. Capture Visual Input The system starts with visual input from a camera or a pre-recorded video. This can come from a standard RGB webcam, smartphone camera, laptop camera, or a specialized eye-tracking camera. The system can process a live video stream in real time or analyze a previously recorded video. In both cases, the input consists of a sequence of frames, with each frame providing the visual information needed to analyze the person's face and eyes. The quality of this input can significantly affect the accuracy of the gaze-tracking system. Factors such as lighting, camera resolution, frame rate, head position, motion blur, and the distance between the person and the camera can influence how reliably the system detects facial and eye features. The Roboflow InferencePipeline https://docs.roboflow.com/reference/inference/inference-python/inference-pipeline?ref=blog.roboflow.com simplifies video processing by providing a direct, in-process interface for running computer vision models on video streams. It supports both prerecorded video files and live video streams, allowing you to add custom inference logic and control how frames and predictions are processed within your application. 2. Locate the Face Before estimating gaze, the system needs to determine where the person's face is located in each frame. A face detection model identifies the face and returns a bounding box around it, defining the region that contains the facial features needed for subsequent gaze estimation. Models such as RF-DETR https://blog.roboflow.com/rf-detr/ and YOLO can be fine-tuned https://blog.roboflow.com/train-yolov8-obb-model/ training-a-yolo26-oriented-bounding-box-obb-model-for-solar-panel-detection-in-aerial-imagery specifically for face detection. During prototyping, SAM 3 can also be used to quickly experiment with face detection without training a dedicated model. These models are available as blocks in Roboflow Workflows, https://roboflow.com/workflows/build?ref=blog.roboflow.com allowing you to build face detection pipelines by connecting pre-built models and processing blocks without managing the underlying deployment infrastructure. Workflows provides a visual, low-code approach to building and deploying computer vision pipelines. For production applications, a fine-tuned face detection model can provide more consistent detections and faster inference because it is optimized specifically for the task. However, training a custom model requires collecting and annotating a face dataset, training the model, and evaluating its performance. Roboflow Train https://roboflow.com/train?ref=blog.roboflow.com can simplify this process with AI-assisted annotation and model training without requiring you to manage the underlying infrastructure or write training code. 3. Prepare the Face Once the face has been located, the system crops it from the original frame using the detected bounding box. The cropped face is then resized and normalized to match the input requirements of the gaze estimation model. Additional preprocessing may also be applied to improve the quality and consistency of the face region. For example, the face can be aligned to a consistent orientation, while padding can be added to ensure important facial features are not cut off. Roboflow Workflows provides blocks such as Detection Offset https://docs.roboflow.com/workflows/blocks/blocks/transformations/detection-offset?ref=blog.roboflow.com for adjusting bounding boxes without writing custom code. The resulting face crop is then passed to the gaze estimation model, which analyzes the facial and eye features to predict the person's gaze direction. 4. Estimate the Gaze Direction The gaze estimation model analyzes the prepared face and predicts the direction in which the person is looking. Deep learning models learn the relationship between facial appearance, eye orientation, and head pose from training data to estimate gaze direction. Models such as L2CS-Net https://inference-models.roboflow.com/models/l2cs/?ref=blog.roboflow.com typically represent gaze direction using angular values, such as yaw and pitch, which describe the horizontal and vertical orientation of the person's gaze. These angles indicate the direction of the gaze, but do not directly identify the specific object, screen location, or point the person is looking at. L2CS-Net can be used through Roboflow's inference-models library. https://inference-models.roboflow.com/models/l2cs/?ref=blog.roboflow.com usage-examples Gaze estimation is only one component of a complete gaze tracking system. The model itself does not detect faces, track people across frames, or determine a specific point of regard. These tasks require additional components that work together with the gaze estimation model. 5. Convert Gaze Direction into a Point of Regard Gaze estimation tells us the direction in which a person is looking, but not necessarily the specific location they are looking at. For example, knowing that a person is looking 15 degrees to the right does not tell us which button, object, or area of a screen they are focusing on. To estimate a point of regard, the system combines the predicted gaze direction with information about the camera and the environment. In screen-based applications, this typically involves calibration or a learned mapping that relates the estimated gaze to positions on the screen. The resulting gaze estimate can then be represented as screen coordinates, such as x, y , indicating the approximate location where the person is looking. In other applications, the gaze direction can be combined with scene geometry or object detections to determine which object or region falls along the person's line of sight. 6. Track the Gaze Over Time Gaze estimation produces a prediction for each individual frame, while gaze tracking analyzes these predictions across multiple frames to understand how a person's gaze changes over time. To do this, the system first tracks detected faces across consecutive frames and associates each gaze prediction with the correct person. Object tracking algorithms such as OC-SORT https://blog.roboflow.com/how-to-use-the-oc-sort-tracker/ and BoT-SORT https://docs.roboflow.com/workflows/blocks/blocks/video-processing/bo-tsort-tracker?ref=blog.roboflow.com can be used for this purpose. These are also available as blocks in Roboflow Workflows, allowing you to add object tracking without implementing the tracking algorithms yourself. The tracker assigns a consistent ID to each detected face, allowing the system to maintain the same identity as a person moves between frames. Each gaze prediction can then be associated with the corresponding tracking ID and analyzed as a continuous sequence. Because gaze predictions can fluctuate between frames, temporal filtering or smoothing can be applied to reduce noise and sudden changes. This produces a more stable gaze trajectory and makes it easier to identify patterns in visual attention, such as when a person maintains their gaze on an area, shifts their attention, or looks away. By combining face tracking, gaze estimation, and temporal smoothing, the system can transform individual frame-level predictions into a continuous representation of gaze over time. Building a Gaze Tracking Pipeline with Roboflow Workflows and Inference The gaze tracking pipeline we will build uses Roboflow Workflows for face detection, tracking, and detection stabilization, along with Roboflow Inference to run the L2CS-Net gaze estimation model. You can also try the face detection workflow built in this guide here https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiQ0VHY3k5MUw5c05hNkJvYng3QzgiLCJ3b3Jrc3BhY2VJZCI6ImNlOWpQdXZRcFFoVXRiYkFXQWZ2UTdDZ3diTDIiLCJ1c2VySWQiOiJjZTlqUHV2UXBRaFV0YmJBV0FmdlE3Q2d3YkwyIiwiaWF0IjoxNzg4ODg1ODIyfQ.F07IxdkcgoMnvzm5YVCEeb651DyDimyXeAMw6OekvBk?ref=blog.roboflow.com . The pipeline processes a video frame by frame, maintains a consistent identity for each detected face, estimates gaze direction, smooths the predictions over time, and produces a final video with the estimated gaze direction visualized. Step 1: Detect Faces The first step in a gaze tracking pipeline is face detection. You can build a face detection workflow using Roboflow Workflows, which provides a range of pre-built blocks, including detection models, bounding box visualization, and label visualization blocks. These blocks make it easy to build and deploy computer vision pipelines without having to implement each component from scratch. To build a face detection workflow, you can either create it manually in Roboflow or use Roboflow Agent. To create one manually, log in to Roboflow, https://app.roboflow.com/?ref=blog.roboflow.com navigate to Workflows in the left sidebar, and select Create Workflow. This opens the Workflow Editor, where you can add the necessary blocks to build your pipeline. Alternatively, with Roboflow Agent available after you login https://app.roboflow.com/?ref=blog.roboflow.com , you can simply describe the workflow you want to create, and the agent will build it for you. As shown below, I asked the Agent to build a face detection workflow. Roboflow Agent then built a workflow that detects faces in both video streams and images. It also provided a UI where I could drag and drop images or videos to test the workflow. The generated workflow is shown below. It uses SAM 3 https://blog.roboflow.com/sam3/ to detect faces and Bounding Box Visualization and Label Visualization blocks to visualize and label the detected faces. The Agent automatically connected the appropriate blocks and configured their parameters so the workflow can perform face detection. If you want to build the workflow manually, you can click the + button in the upper-left corner of the Workflow Editor, search for the required workflow blocks, and add them to the workflow. You can then manually connect the blocks to create the processing workflow, as shown above. You can click on the SAM 3 block to view the configuration generated by the Agent. In this workflow, the detection class is set to "human face", instructing SAM 3 to detect human faces. If you decide to build the workflow manually, you can use the same configuration settings. The Agent also set the confidence threshold to 0.6 and enabled Non-Maximum Suppression NMS . https://blog.roboflow.com/non-max-merging/ NMS is a post-processing technique that removes overlapping bounding boxes that refer to the same object, keeping the detection with the highest confidence score. The NMS threshold was set to 0.6. This threshold determines how much overlap is allowed between two bounding boxes before one of them is suppressed. A lower threshold results in more aggressive removal of overlapping detections, while a higher threshold allows more overlapping boxes to remain. Make sure to save the workflow to preserve your changes. You can also test the workflow by running it on an image or video. When run on an image, the workflow outputs the visualized image along with the face detection predictions. When run on a video, the workflow processes the video frame by frame and generates predictions for each frame. The processed frames are then combined to produce the final output video, as shown below: Step 2: Track Faces Across Video Stream The face detections produced by SAM 3 are frame-independent, meaning that each frame is processed independently. As a result, the same face may receive a different detection identifier in subsequent frames. To associate the same face across consecutive frames and maintain its identity over time, the detections need to be passed through an object tracking algorithm. Roboflow provides several tracking blocks in Workflows, including BoT-SORT, https://docs.roboflow.com/workflows/blocks/blocks/video-processing/bo-tsort-tracker?ref=blog.roboflow.com OC-SORT, https://docs.roboflow.com/workflows/blocks/blocks/video-processing/ocsort-tracker?ref=blog.roboflow.com and SORT. https://docs.roboflow.com/workflows/blocks/blocks/video-processing/sort-tracker?ref=blog.roboflow.com You can ask Roboflow Agent to add a tracker to your workflow and use it to track detected faces across consecutive frames. The agent should then add the tracker to your workflow and connect it to the output of SAM 3, as shown below. You can configure the OC-SORT tracker by clicking on the block, which opens its configuration panel. For example, you can set the Minimum IoU Threshold to 0.2 to allow more tolerance for bounding box changes, Minimum Consecutive Frames to 2 to confirm detections quickly, and the Lost Track Buffer to 90 to retain tracks when a face temporarily disappears. You can also configure the bounding box visualization block to assign colors based on the track ID. This ensures that the same tracked face is displayed using a consistent color across frames, making it easier to visually distinguish between different people. Similarly, you can configure the label visualization block to use the track ID when assigning colors and display the tracker ID alongside each detected face. This allows you to see which detections belong to the same tracked person across consecutive frames. You can then run the workflow on your video. The resulting video output will contain bounding boxes and labels, with each face assigned a unique color and tracker ID that remain consistent throughout the video. This consistency is made possible by OC-Sort tracker, which maintains the identity of each detected face across frames. Step 3: Stabilize the Detections Roboflow Workflows also provides a Detections Stabilizer block https://docs.roboflow.com/workflows/blocks/blocks/video-processing/detections-stabilizer?ref=blog.roboflow.com , which helps reduce jitter and inconsistencies in detection bounding boxes across consecutive video frames. This produces more stable detections as objects move through the video. You can prompt the Roboflow Agent to add the Detections Stabilizer block to your workflow. Once added, the Detections Stabilizer should be connected to the OC-SORT Tracker as shown below. Configure the Detections Stabilizer to take the tracked detections produced by the OC-SORT Tracker as its input. The stabilized detections are the final face detection outputs. They contain faces that have been consistently detected, tracked across video frames, and stabilized to reduce bounding-box jitter. You can now run the workflow on your test video. https://www.pexels.com/video/people-brainstorming-7534730/?ref=blog.roboflow.com While reviewing the output, you can seek through the video to inspect the JSON output for each frame and view the corresponding stabilized predictions. Make sure to publish your changes, to make the workflow callable via API. Step 4: Store the Face Detection Now we can run the workflow using Python. You can find the deployment script under the Deploy button in the Workflow Editor. Roboflow provides example scripts for running the workflow on a webcam, RTSP stream, or video file, as well as options for running inference through the Cloud API or locally. We can use this deployment script as a starting point, but for gaze tracking, we need to modify it so that we can access the stabilized and tracked face detections produced by the workflow. We will use these detections to identify each face and determine the region of the frame that should be used for gaze estimation. Start by configuring the Roboflow client and setting up the WebRTC stream as shown in the script below. In the script, we use a video file as the input and request two outputs from the workflow: the visualized workflow frame and the stabilized face predictions, which contain the tracker IDs. python import os from inference sdk import InferenceHTTPClient from inference sdk.webrtc import VideoFileSource, StreamConfig, VideoMetadata client = InferenceHTTPClient.init api url="https://serverless.roboflow.com", api key=os.getenv "ROBOFLOW API KEY" WebRTC stream configuration config = StreamConfig stream output= , data output= "output image", Frame with bounding boxes visualized "stabilized predictions" Stabilized face predictions with tracker IDs , requested plan="webrtc-gpu-medium", requested region="us" INPUT VIDEO = "input.mp4" source = VideoFileSource INPUT VIDEO, realtime processing=False Start WebRTC session session = client.webrtc.stream source=source, workflow="human-face-detection-1788782690880", workspace="dikshants-blog-workspace", image input="image", config=config Make sure you have the inference-sdk installed too: pip install -U inference-sdk For each video frame, the workflow when run using session.run command returns the visualized frame along with the stabilized and tracked face detections. We will later draw the estimated gaze direction onto these visualized frames, so we first need to save them to disk. Add the following code to the above script to create a folder for storing the workflow output frames. If the folder already exists, it is removed and recreated to ensure that the output contains only frames from the current run. python import shutil Bounding box visualized frames output folder WORKFLOW OUTPUT FRAMES FOLDER = "workflow output frames" if os.path.exists WORKFLOW OUTPUT FRAMES FOLDER : shutil.rmtree WORKFLOW OUTPUT FRAMES FOLDER os.makedirs WORKFLOW OUTPUT FRAMES FOLDER, exist ok=True Next, add the following helper function to your script. It saves each workflow output frame as a JPEG image, with the frame number included in the filename so that the images remain in order and can later be matched with the corresponding face predictions. python import cv2 Save bounding box visualized frames def save workflow frame workflow output image, frame number : frame path = os.path.join WORKFLOW OUTPUT FRAMES FOLDER, f"frame {frame number:06d}.jpg" cv2.imwrite frame path, workflow output image Since we also need to store the face detections returned for each frame, add the following code to your script. It registers a callback that is triggered whenever the workflow returns data for a frame. The callback extracts the workflow output image and stabilized face predictions, then saves the image and stores the corresponding face detections in a dictionary. python import base64 import numpy as np Storage for face prediction coordinates face predictions by frame = {} Workflow data callback @session.on data def on prediction data: dict, metadata: VideoMetadata : if not data: return frame number = int metadata.frame id Get Roboflow output image workflow image data = data.get "output image" Get stabilized face predictions stabilized predictions = data.get "stabilized predictions", {} frame predictions = stabilized predictions.get "predictions", Decode workflow output image from Base64 workflow output image = None if workflow image data: workflow image value = workflow image data.get "value" if workflow image value: image bytes = base64.b64decode workflow image value image array = np.frombuffer image bytes, dtype=np.uint8 workflow output image = cv2.imdecode image array, cv2.IMREAD COLOR Save workflow output frame if workflow output image is not None: save workflow frame workflow output image, frame number Extract required face bounding boxes frame faces = for prediction in frame predictions: x = float prediction "x" y = float prediction "y" width = float prediction "width" height = float prediction "height" frame faces.append { "tracker id": prediction.get "tracker id" , "x": x, "y": y, "width": width, "height": height } Store bounding boxes for the current frame face predictions by frame str frame number = frame faces Progress counter for workflow frames processed if frame number % 10 == 0: print f"Frame {frame number}: Workflow response received" Inside the callback, we extract the workflow output image and stabilized face predictions. The output image is decoded from Base64 and saved to disk, while the bounding box coordinates and tracker ID are extracted for each detected face. The detections are then stored in a dictionary indexed by frame number. This allows us to retrieve the location and identity of each tracked face later when performing gaze estimation. Next, start the workflow by calling session.run . This processes the input video frame by frame and triggers the on prediction callback as each frame is processed. The callback saves the visualized workflow frame and stores the corresponding stabilized face detections. After the workflow finishes processing the video, save the collected face detections to a JSON file. This creates a persistent record of the face bounding boxes and tracker IDs for every processed frame. We can use this data in the next step to crop the tracked faces and perform gaze estimation on each face. Add the following code to the end of the same script, then run the entire script: python Run workflow session.run import json Save face predictions for entire video to JSON FACE PREDICTIONS JSON = "face predictions.json" with open FACE PREDICTIONS JSON, "w", encoding="utf-8" as f: json.dump face predictions by frame, f, indent=2 print "Roboflow processing completed." After the script finishes processing the video, you will have two outputs: 1. A folder containing the visualized workflow frames, which can later be annotated with gaze information. 2. A JSON file containing the stabilized face detections, including the bounding box coordinates and tracker ID for each face in every processed frame. These outputs provide everything needed for the next step, where we crop the tracked faces and perform gaze estimation. Step 5: Estimate Gaze Direction With the stabilized and tracked face detections stored from the previous step, we can now perform gaze estimation for each detected face. For this, we will use Roboflow's inference models https://inference-models.roboflow.com/?ref=blog.roboflow.com package with the L2CS-Net gaze estimation model. First, install the package for cpu only devices : pip install "inference models onnx-cpu " Load the L2CS-Net model using AutoModel. The rn50 model uses a ResNet-50 backbone and predicts the person's gaze direction as yaw and pitch angles. You can add the following code to a new script or continue building on the script from the previous step and run it as a complete pipeline. python import os from inference models import AutoModel Load L2CS-Net gaze model gaze model = AutoModel.from pretrained "l2cs-net/rn50", api key=os.getenv "ROBOFLOW API KEY" In the previous step, we stored the stabilized face detections for each frame in a JSON file. Load this file by adding the following code to the script so that we can retrieve the face bounding boxes and tracker IDs while processing each frame. python import json Stored face predictions JSON file from earlier FACE PREDICTIONS JSON = "face predictions.json" Load face predictions with open FACE PREDICTIONS JSON, "r", encoding="utf-8" as f: face predictions by frame = json.load f The workflow output frames were also saved with their frame numbers in the filenames. We can retrieve and sort these frames by frame number to ensure they are processed in the same order as the original video. Add the following code to the script to do this. python Extract frame number from filename def get frame number filename : return int os.path.splitext filename 0 .split " " 1 Workflow output frames folder defined earlier WORKFLOW OUTPUT FRAMES FOLDER = "workflow output frames" frame files = sorted f for f in os.listdir WORKFLOW OUTPUT FRAMES FOLDER if f.lower .endswith ".jpg" , key=get frame number We will draw the estimated gaze direction onto these workflow output frames. To store them, we create a separate folder to store the gaze-visualized frames. Add the following code snippet to the script: python import shutil Gaze visualized frames output folder GAZE VISUALIZED FRAMES FOLDER = "gaze visualized frames" if os.path.exists GAZE VISUALIZED FRAMES FOLDER : shutil.rmtree GAZE VISUALIZED FRAMES FOLDER os.makedirs GAZE VISUALIZED FRAMES FOLDER, exist ok=True Gaze predictions can vary slightly between consecutive frames, even when a person's actual gaze remains relatively stable. To reduce this frame-to-frame jitter, we can apply exponential smoothing to both the gaze angles and the face center coordinates. Add the following code to your script to perform gaze and face-center smoothing: Gaze tracking configuration GAZE SMOOTHING ALPHA = 0.18 FACE CENTER SMOOTHING ALPHA = 0.30 Smoothing function for gaze and face center values def smooth value previous, current, alpha : if previous is None: return current return alpha current + 1.0 - alpha previous The GAZE SMOOTHING ALPHA controls how quickly the smoothed gaze follows new predictions. A lower value produces stronger smoothing, while a higher value responds more quickly to changes. Similarly, FACE CENTER SMOOTHING ALPHA smooths the position of the face center used as the starting point for the gaze arrow. We can also customize the appearance of the gaze visualization by controlling the length and thickness of the arrow used to represent the estimated gaze direction. Add the following variables to your script to configure these properties: Gaze arrow visualization configuration ARROW LENGTH SCALE = 0.5 ARROW THICKNESS = 8 ARROW TIP LENGTH = 0.30 For each frame, we want to retrieve the corresponding stabilized face detections from the JSON file, crop each detected face using its bounding box, and pass the crop to L2CS-Net for gaze estimation. The model returns the estimated gaze as yaw and pitch angles. We then smooth these values using the previous prediction associated with the face's tracker id , allowing the pipeline to maintain a separate gaze history for each person when multiple faces are present. Finally, we convert the smoothed gaze angles into a direction vector and visualize it as an arrow originating from the center of the tracked face. To do these, add the following code snippet to the script: python import cv2 import numpy as np Tracker state gaze state = {} face center state = {} Progress counter for gaze frames written gaze frames written = 0 Process each frame for gaze estimation and visualization for filename in frame files: frame number = get frame number filename frame path = os.path.join WORKFLOW OUTPUT FRAMES FOLDER, filename img = cv2.imread frame path if img is None: continue Get face predictions for the current frame frame faces = face predictions by frame.get str frame number , Process each detected face for face in frame faces: tracker id = face.get "tracker id" x = float face "x" y = float face "y" width = float face "width" height = float face "height" Calculate bounding box coordinates x1 = int x - width / 2 y1 = int y - height / 2 x2 = int x + width / 2 y2 = int y + height / 2 Clamp bounding box to image boundaries x1 = max 0, min x1, img.shape 1 - 1 y1 = max 0, min y1, img.shape 0 - 1 x2 = max 0, min x2, img.shape 1 y2 = max 0, min y2, img.shape 0 Skip invalid bounding boxes if x2 <= x1 or y2 <= y1: continue Crop face in memory face image = img y1:y2, x1:x2 if face image.size == 0: continue Run L2CS-Net gaze estimation try: gaze result = gaze model.infer face image except Exception as e: print f"Gaze inference failed on frame {frame number}, " f"tracker {tracker id}: {e}" continue Get gaze angles try: yaw = float gaze result.yaw pitch = float gaze result.pitch except Exception as e: print f"Could not read gaze result on frame {frame number}, " f"tracker {tracker id}: {e}" continue Initialize state for a newly detected face if tracker id not in gaze state: gaze state tracker id = {"yaw": yaw,"pitch": pitch} if tracker id not in face center state: face center state tracker id = {"x": x,"y": y} Smooth gaze angles previous yaw = gaze state tracker id "yaw" previous pitch = gaze state tracker id "pitch" smooth yaw = smooth value previous yaw, yaw, GAZE SMOOTHING ALPHA smooth pitch = smooth value previous pitch, pitch, GAZE SMOOTHING ALPHA Save smoothed gaze state gaze state tracker id = {"yaw": smooth yaw, "pitch": smooth pitch} Smooth face center previous center x = face center state tracker id "x" previous center y = face center state tracker id "y" smooth center x = smooth value previous center x, x, FACE CENTER SMOOTHING ALPHA smooth center y = smooth value previous center y, y, FACE CENTER SMOOTHING ALPHA Save smoothed face center face center state tracker id = {"x": smooth center x, "y": smooth center y} Calculate gaze arrow length arrow length = max width, height ARROW LENGTH SCALE Convert gaze angles to a direction vector dx = -np.sin smooth yaw arrow length dy = -np.sin smooth pitch arrow length Arrow start coordinates start x = int smooth center x start y = int smooth center y Arrow end coordinates end x = int start x + dx end y = int start y + dy Draw gaze direction cv2.arrowedLine img, start x, start y , end x, end y , 0, 255, 0 , ARROW THICKNESS, cv2.LINE AA, tipLength=ARROW TIP LENGTH Save gaze-visualized frame output path = os.path.join GAZE VISUALIZED FRAMES FOLDER, f"frame {frame number:06d}.jpg" cv2.imwrite output path, img gaze frames written += 1 Progress counter if frame number % 10 == 0: print f"Frame {frame number}: " f"Gaze visualization completed" At the end of this process, the gaze visualized frames folder contains the workflow frames with the estimated gaze direction drawn for each tracked face. Because the gaze state is maintained separately for each tracker id, the smoothing is applied independently to each person in the video. These processed frames can then be assembled back into a video to produce the final gaze-tracking visualization. Step 6: Generate the Gaze Tracking Video The gaze estimation and visualization process generates a separate image for each video frame. In the final step, we can combine these processed frames back into a video while preserving the original frame rate and resolution. To do this, add the following code snippet to the script, where the output video path is defined. The code retrieves the gaze-visualized frames and sorts them by frame number to ensure they are written to the output video in the correct order. GAZE VIDEO PATH = "output.mp4" Get gaze-visualized frames gaze frame files = sorted f for f in os.listdir GAZE VISUALIZED FRAMES FOLDER if f.lower .endswith ".jpg" , key=get frame number Check if gaze-visualized frames were generated if not gaze frame files: raise RuntimeError "No gaze visualization frames were generated." Next, add the code snippet below. It determines the video's dimensions from the first processed frame and retrieves the original video's FPS so that the generated video plays back at the same speed as the input video. Input video used from earlier INPUT VIDEO = "input.mp4" Get video dimensions from the first gaze-visualized frame first frame = cv2.imread os.path.join GAZE VISUALIZED FRAMES FOLDER, gaze frame files 0 frame height, frame width = first frame.shape :2 Get original video FPS cap = cv2.VideoCapture INPUT VIDEO fps = cap.get cv2.CAP PROP FPS cap.release Now, add the code snippet below to the script. It creates a video writer and writes each gaze-visualized frame to the output video. Once all frames have been written, the video writer is released to finalize the output file. Create video writer for the final gaze-tracking video video writer = cv2.VideoWriter GAZE VIDEO PATH, cv2.VideoWriter fourcc "mp4v" , fps, frame width, frame height Write gaze-visualized frames to the final video for filename in gaze frame files: frame path = os.path.join GAZE VISUALIZED FRAMES FOLDER, filename frame = cv2.imread frame path if frame is None: continue video writer.write frame Finalize video writing video writer.release print f"Gaze tracking video saved to: {GAZE VIDEO PATH}" Now, run the script. The resulting output.mp4 contains the original video with the detected faces, tracking information, and estimated gaze directions visualized over time. In this way, you can create a complete gaze tracking pipeline that combines face detection, multi-object tracking, detection stabilization, and gaze estimation. Key Challenges in Gaze Tracking - Maintaining reliable processing speed: A real-time gaze tracking system often needs to run a face detection model, tracking pipeline, and gaze estimation model together for every frame. Running multiple models can significantly increase computational requirements and latency, making it challenging to maintain a consistent frame rate while still producing accurate gaze predictions. - Eyes moving out of the camera frame: When a person's eyes are partially or completely outside the video frame, gaze estimation becomes much more difficult. Models such as L2CS-Net rely on the eyes, to estimate gaze direction, so their predictions can become unreliable or fail entirely when the eyes are not visible. - Rapidly changing gaze: Quickly shifting the gaze from one direction to another can be difficult to track accurately. The system may lag behind the actual movement or produce unstable predictions when the gaze changes rapidly between consecutive frames. - Handling false gaze estimations: Gaze estimation models can occasionally produce incorrect predictions because of poor lighting, unusual head poses, occlusions, motion blur, or inaccurate face detections, which may throw off the entire tracking. - Maintaining stable tracking across frames: Gaze tracking is a temporal problem, so the system needs to maintain a consistent identity for each detected face across frames. Lost detections and frames, changing bounding boxes, or incorrect tracker assignments can cause the resulting gaze trajectory to become inconsistent. Major Applications of Gaze Tracking Gaze tracking can be applied in a variety of scenarios where understanding a person's visual attention and eye movements is useful. Some of the major applications include: - Human-Computer Interaction: Gaze tracking enables users to interact with computers and digital interfaces using their eye movements. It can be used for selecting, navigating, scrolling, and controlling on-screen elements. - Accessibility and Assistive Technology: Gaze tracking allows people with limited mobility to control computers and communication devices using their eyes. It can support virtual keyboards, communication systems, and hands-free device control. - Gaming and Virtual Reality: Gaze tracking enables games and VR applications to respond to where players are looking. It can support gaze-based controls, immersive interactions, adaptive interfaces, and foveated rendering. - User Experience and Usability Research: Researchers use gaze tracking to analyze how users visually navigate websites, applications, and interfaces. Metrics such as fixations and gaze paths can reveal which elements attract attention and how users interact with a design. - Medical and Psychological Research: Gaze tracking is widely used in research on visual perception, attention, reading behavior, and eye movement. It can provide quantitative data about how people process and respond to visual information. - Automotive and Aviation: Gaze tracking can help analyze how drivers, pilots, and operators visually monitor complex environments. It can provide insights into attention allocation and interaction with instruments, displays, and controls. Conclusion A complete gaze tracking pipeline can combine face detection, multi-object tracking, detection stabilization, and gaze estimation to produce a reliable gaze trajectory from video. With Roboflow Workflows, https://roboflow.com/workflows/build?ref=blog.roboflow.com you can build much of this pipeline visually using pre-built computer vision models and processing blocks, without having to implement each component from scratch. Roboflow Agent can further accelerate development by helping you create and configure workflows using natural language. You can then use Roboflow's inference models such as L2CS-Net to add specialized capabilities like gaze estimation to your application . Roboflow provides the tools you need to move from a computer vision idea to a working system faster. Start building your own computer vision pipeline with Roboflow today. https://app.roboflow.com/?ref=blog.roboflow.com Cite this Post Use the following entry to cite this post in your research: Dikshant Shah /author/dikshant/ . Sep 8, 2026 . Gaze Detection and Gaze Tracking Explained. Roboflow Blog: https://blog.roboflow.com/gaze-detection/