# Gaze Detection and Gaze Tracking Explained

> Source: <https://blog.roboflow.com/gaze-detection/>
> Published: 2026-09-08 21:36:35+00:00

*Gaze detection determines where a person is looking, while gaze tracking continuously follows their gaze over time to understand visual attention and behavior. This guide explains how gaze tracking with computer vision works, and shows how to build a complete gaze tracking pipeline using Roboflow Workflows.*

Computers can now understand where a person is looking using [computer vision](https://blog.roboflow.com/intro-to-computer-vision/) and AI. From eye-tracking systems used in human-computer interaction to cameras that estimate where a driver is looking, gaze tracking and gaze detection are becoming increasingly useful across industries.

These technologies analyze a person's eyes, face, and head position to estimate their visual attention. They can help determine whether someone is looking at a screen, a product, a road, or a specific object in their environment.

In this guide, you'll learn what gaze detection and gaze tracking are, how these systems work, where they are used, and how to build a gaze tracking pipeline using Roboflow tools.

## What Is Gaze Detection?

Gaze detection is the process of determining the direction or target of a person's visual attention. A gaze detection system typically analyzes visual information such as eye position, pupil location, iris orientation, head pose, facial landmarks, and eye movement. By combining these visual cues, a gaze detection system estimates where a person is looking.

Gaze detection does not necessarily require extremely precise measurements. In some applications, it may only need to classify gaze into broad categories, such as whether a person is looking at the screen or looking away.

## What Is Gaze Tracking?

Gaze tracking goes a step further then gaze detection by continuously estimating a person's gaze over time. Instead of simply determining where someone is looking at a particular moment, a gaze tracking system can record the movement and trajectory of their gaze.

In simple terms, gaze detection answers "Where is the person looking?" while gaze tracking answers "Where has the person been looking, and how has their gaze moved over time?"

A gaze-tracking system should capture information such as:

- Where a person looks first
- How long they look at a particular area
- Where they look next
- How their gaze moves between objects
- How frequently they return to a particular area

This creates a gaze trajectory that can be analyzed to understand visual attention and behavior. Gaze tracking is therefore particularly useful when the goal is to study how visual attention changes over time.

## How Does a Gaze Tracking System Work?

A gaze tracking system combines computer vision, deep learning, and temporal analysis to determine where a person is looking over time. While a gaze estimation model predicts the direction of a person's gaze, it is only one component of the overall system.

Given below are the main components of a gaze tracking system and how they work together to estimate and track gaze over time.

### 1. Capture Visual Input

The system starts with visual input from a camera or a pre-recorded video. This can come from a standard RGB webcam, smartphone camera, laptop camera, or a specialized eye-tracking camera. The system can process a live video stream in real time or analyze a previously recorded video. In both cases, the input consists of a sequence of frames, with each frame providing the visual information needed to analyze the person's face and eyes.

The quality of this input can significantly affect the accuracy of the gaze-tracking system. Factors such as lighting, camera resolution, frame rate, head position, motion blur, and the distance between the person and the camera can influence how reliably the system detects facial and eye features.

The Roboflow [InferencePipeline](https://docs.roboflow.com/reference/inference/inference-python/inference-pipeline?ref=blog.roboflow.com) simplifies video processing by providing a direct, in-process interface for running computer vision models on video streams. It supports both prerecorded video files and live video streams, allowing you to add custom inference logic and control how frames and predictions are processed within your application.

### 2. Locate the Face

Before estimating gaze, the system needs to determine where the person's face is located in each frame. A face detection model identifies the face and returns a bounding box around it, defining the region that contains the facial features needed for subsequent gaze estimation.

Models such as [RF-DETR](https://blog.roboflow.com/rf-detr/) and YOLO can be [fine-tuned](https://blog.roboflow.com/train-yolov8-obb-model/#training-a-yolo26-oriented-bounding-box-obb-model-for-solar-panel-detection-in-aerial-imagery) specifically for face detection. During prototyping, SAM 3 can also be used to quickly experiment with face detection without training a dedicated model. These models are available as blocks in [Roboflow Workflows,](https://roboflow.com/workflows/build?ref=blog.roboflow.com) allowing you to build face detection pipelines by connecting pre-built models and processing blocks without managing the underlying deployment infrastructure. Workflows provides a visual, low-code approach to building and deploying computer vision pipelines.

For production applications, a fine-tuned face detection model can provide more consistent detections and faster inference because it is optimized specifically for the task. However, training a custom model requires collecting and annotating a face dataset, training the model, and evaluating its performance. [Roboflow Train](https://roboflow.com/train?ref=blog.roboflow.com) can simplify this process with AI-assisted annotation and model training without requiring you to manage the underlying infrastructure or write training code.

### 3. Prepare the Face

Once the face has been located, the system crops it from the original frame using the detected bounding box. The cropped face is then resized and normalized to match the input requirements of the gaze estimation model.

Additional preprocessing may also be applied to improve the quality and consistency of the face region. For example, the face can be aligned to a consistent orientation, while padding can be added to ensure important facial features are not cut off.

Roboflow Workflows provides blocks such as [Detection Offset](https://docs.roboflow.com/workflows/blocks/blocks/transformations/detection-offset?ref=blog.roboflow.com) for adjusting bounding boxes without writing custom code. The resulting face crop is then passed to the gaze estimation model, which analyzes the facial and eye features to predict the person's gaze direction.

### 4. Estimate the Gaze Direction

The gaze estimation model analyzes the prepared face and predicts the direction in which the person is looking. Deep learning models learn the relationship between facial appearance, eye orientation, and head pose from training data to estimate gaze direction.

Models such as [L2CS-Net](https://inference-models.roboflow.com/models/l2cs/?ref=blog.roboflow.com) typically represent gaze direction using angular values, such as yaw and pitch, which describe the horizontal and vertical orientation of the person's gaze. These angles indicate the direction of the gaze, but do not directly identify the specific object, screen location, or point the person is looking at. L2CS-Net can be used through [Roboflow's inference-models library.](https://inference-models.roboflow.com/models/l2cs/?ref=blog.roboflow.com#usage-examples)

Gaze estimation is only one component of a complete gaze tracking system. The model itself does not detect faces, track people across frames, or determine a specific point of regard. These tasks require additional components that work together with the gaze estimation model.

### 5. Convert Gaze Direction into a Point of Regard

Gaze estimation tells us the direction in which a person is looking, but not necessarily the specific location they are looking at. For example, knowing that a person is looking 15 degrees to the right does not tell us which button, object, or area of a screen they are focusing on.

To estimate a point of regard, the system combines the predicted gaze direction with information about the camera and the environment. In screen-based applications, this typically involves calibration or a learned mapping that relates the estimated gaze to positions on the screen.

The resulting gaze estimate can then be represented as screen coordinates, such as (x, y), indicating the approximate location where the person is looking. In other applications, the gaze direction can be combined with scene geometry or object detections to determine which object or region falls along the person's line of sight.

### 6. Track the Gaze Over Time

Gaze estimation produces a prediction for each individual frame, while gaze tracking analyzes these predictions across multiple frames to understand how a person's gaze changes over time. To do this, the system first tracks detected faces across consecutive frames and associates each gaze prediction with the correct person.

Object tracking algorithms such as [OC-SORT](https://blog.roboflow.com/how-to-use-the-oc-sort-tracker/) and [BoT-SORT](https://docs.roboflow.com/workflows/blocks/blocks/video-processing/bo-tsort-tracker?ref=blog.roboflow.com) can be used for this purpose. These are also available as blocks in Roboflow Workflows, allowing you to add object tracking without implementing the tracking algorithms yourself. The tracker assigns a consistent ID to each detected face, allowing the system to maintain the same identity as a person moves between frames. Each gaze prediction can then be associated with the corresponding tracking ID and analyzed as a continuous sequence.

Because gaze predictions can fluctuate between frames, temporal filtering or smoothing can be applied to reduce noise and sudden changes. This produces a more stable gaze trajectory and makes it easier to identify patterns in visual attention, such as when a person maintains their gaze on an area, shifts their attention, or looks away. By combining face tracking, gaze estimation, and temporal smoothing, the system can transform individual frame-level predictions into a continuous representation of gaze over time.

## Building a Gaze Tracking Pipeline with Roboflow Workflows and Inference

The gaze tracking pipeline we will build uses Roboflow Workflows for face detection, tracking, and detection stabilization, along with Roboflow Inference to run the L2CS-Net gaze estimation model. You can also try the face detection workflow built in this guide [here](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiQ0VHY3k5MUw5c05hNkJvYng3QzgiLCJ3b3Jrc3BhY2VJZCI6ImNlOWpQdXZRcFFoVXRiYkFXQWZ2UTdDZ3diTDIiLCJ1c2VySWQiOiJjZTlqUHV2UXBRaFV0YmJBV0FmdlE3Q2d3YkwyIiwiaWF0IjoxNzg4ODg1ODIyfQ.F07IxdkcgoMnvzm5YVCEeb651DyDimyXeAMw6OekvBk?ref=blog.roboflow.com).

The pipeline processes a video frame by frame, maintains a consistent identity for each detected face, estimates gaze direction, smooths the predictions over time, and produces a final video with the estimated gaze direction visualized.

### Step 1: Detect Faces

The first step in a gaze tracking pipeline is face detection. You can build a face detection workflow using Roboflow Workflows, which provides a range of pre-built blocks, including detection models, bounding box visualization, and label visualization blocks. These blocks make it easy to build and deploy computer vision pipelines without having to implement each component from scratch.

To build a face detection workflow, you can either create it manually in Roboflow or use Roboflow Agent. To create one manually, [log in to Roboflow,](https://app.roboflow.com/?ref=blog.roboflow.com) navigate to **Workflows** in the left sidebar, and select Create Workflow. This opens the Workflow Editor, where you can add the necessary blocks to build your pipeline.

Alternatively, with Roboflow Agent [(available after you login)](https://app.roboflow.com/?ref=blog.roboflow.com), you can simply describe the workflow you want to create, and the agent will build it for you. As shown below, I asked the Agent to build a face detection workflow.

Roboflow Agent then built a workflow that detects faces in both video streams and images. It also provided a UI where I could drag and drop images or videos to test the workflow.

The generated workflow is shown below. It uses [SAM 3](https://blog.roboflow.com/sam3/) to detect faces and Bounding Box Visualization and Label Visualization blocks to visualize and label the detected faces.

The Agent automatically connected the appropriate blocks and configured their parameters so the workflow can perform face detection.

If you want to build the workflow manually, you can click the + button in the upper-left corner of the Workflow Editor, search for the required workflow blocks, and add them to the workflow. You can then manually connect the blocks to create the processing workflow, as shown above.

You can click on the SAM 3 block to view the configuration generated by the Agent. In this workflow, the detection class is set to "human face", instructing SAM 3 to detect human faces. If you decide to build the workflow manually, you can use the same configuration settings.

The Agent also set the confidence threshold to 0.6 and enabled [Non-Maximum Suppression (NMS).](https://blog.roboflow.com/non-max-merging/) NMS is a post-processing technique that removes overlapping bounding boxes that refer to the same object, keeping the detection with the highest confidence score.

The NMS threshold was set to 0.6. This threshold determines how much overlap is allowed between two bounding boxes before one of them is suppressed. A lower threshold results in more aggressive removal of overlapping detections, while a higher threshold allows more overlapping boxes to remain.

Make sure to save the workflow to preserve your changes. You can also test the workflow by running it on an image or video. When run on an image, the workflow outputs the visualized image along with the face detection predictions.

When run on a video, the workflow processes the video frame by frame and generates predictions for each frame. The processed frames are then combined to produce the final output video, as shown below:

### Step 2: Track Faces Across Video Stream

The face detections produced by SAM 3 are frame-independent, meaning that each frame is processed independently. As a result, the same face may receive a different detection identifier in subsequent frames. To associate the same face across consecutive frames and maintain its identity over time, the detections need to be passed through an object tracking algorithm.

Roboflow provides several tracking blocks in Workflows, including [BoT-SORT,](https://docs.roboflow.com/workflows/blocks/blocks/video-processing/bo-tsort-tracker?ref=blog.roboflow.com) [OC-SORT,](https://docs.roboflow.com/workflows/blocks/blocks/video-processing/ocsort-tracker?ref=blog.roboflow.com) and [SORT.](https://docs.roboflow.com/workflows/blocks/blocks/video-processing/sort-tracker?ref=blog.roboflow.com) You can ask Roboflow Agent to add a tracker to your workflow and use it to track detected faces across consecutive frames.

The agent should then add the tracker to your workflow and connect it to the output of SAM 3, as shown below.

You can configure the OC-SORT tracker by clicking on the block, which opens its configuration panel. For example, you can set the Minimum IoU Threshold to 0.2 to allow more tolerance for bounding box changes, Minimum Consecutive Frames to 2 to confirm detections quickly, and the Lost Track Buffer to 90 to retain tracks when a face temporarily disappears.

You can also configure the bounding box visualization block to assign colors based on the track ID. This ensures that the same tracked face is displayed using a consistent color across frames, making it easier to visually distinguish between different people.

Similarly, you can configure the label visualization block to use the track ID when assigning colors and display the tracker ID alongside each detected face. This allows you to see which detections belong to the same tracked person across consecutive frames.

You can then run the workflow on your video. The resulting video output will contain bounding boxes and labels, with each face assigned a unique color and tracker ID that remain consistent throughout the video. This consistency is made possible by OC-Sort tracker, which maintains the identity of each detected face across frames.

### Step 3: Stabilize the Detections

Roboflow Workflows also provides a [Detections Stabilizer block](https://docs.roboflow.com/workflows/blocks/blocks/video-processing/detections-stabilizer?ref=blog.roboflow.com), which helps reduce jitter and inconsistencies in detection bounding boxes across consecutive video frames. This produces more stable detections as objects move through the video.

You can prompt the Roboflow Agent to add the Detections Stabilizer block to your workflow.

Once added, the Detections Stabilizer should be connected to the OC-SORT Tracker as shown below.

Configure the Detections Stabilizer to take the tracked detections produced by the OC-SORT Tracker as its input.

The stabilized detections are the final face detection outputs. They contain faces that have been consistently detected, tracked across video frames, and stabilized to reduce bounding-box jitter.

You can now run the workflow on your [test video.](https://www.pexels.com/video/people-brainstorming-7534730/?ref=blog.roboflow.com) While reviewing the output, you can seek through the video to inspect the JSON output for each frame and view the corresponding stabilized predictions.

Make sure to publish your changes, to make the workflow callable via API.

### Step 4: Store the Face Detection

Now we can run the workflow using Python. You can find the deployment script under the **Deploy** button in the Workflow Editor. 

Roboflow provides example scripts for running the workflow on a webcam, RTSP stream, or video file, as well as options for running inference through the Cloud API or locally.

We can use this deployment script as a starting point, but for gaze tracking, we need to modify it so that we can access the stabilized and tracked face detections produced by the workflow. We will use these detections to identify each face and determine the region of the frame that should be used for gaze estimation.

Start by configuring the Roboflow client and setting up the WebRTC stream as shown in the script below. In the script, we use a video file as the input and request two outputs from the workflow: the visualized workflow frame and the stabilized face predictions, which contain the tracker IDs.

``` python
import os
from inference_sdk import InferenceHTTPClient
from inference_sdk.webrtc import VideoFileSource, StreamConfig, VideoMetadata

client = InferenceHTTPClient.init(
    api_url="https://serverless.roboflow.com", 
    api_key=os.getenv("ROBOFLOW_API_KEY")
)

# WebRTC stream configuration
config = StreamConfig(
    stream_output=[],
    data_output=[
        "output_image",              # Frame with bounding boxes visualized
        "stabilized_predictions"     # Stabilized face predictions with tracker IDs
    ],
    requested_plan="webrtc-gpu-medium",
    requested_region="us"
)

INPUT_VIDEO = "input.mp4"
source = VideoFileSource(INPUT_VIDEO, realtime_processing=False)

# Start WebRTC session
session = client.webrtc.stream(
    source=source,
    workflow="human-face-detection-1788782690880",
    workspace="dikshants-blog-workspace",
    image_input="image",
    config=config
)
```

Make sure you have the inference-sdk installed too:

```
pip install -U inference-sdk
```

For each video frame, the workflow when run using **session.run()** command returns the visualized frame along with the stabilized and tracked face detections. We will later draw the estimated gaze direction onto these visualized frames, so we first need to save them to disk.

Add the following code to the above script to create a folder for storing the workflow output frames. If the folder already exists, it is removed and recreated to ensure that the output contains only frames from the current run.

``` python
import shutil

# Bounding box visualized frames output folder
WORKFLOW_OUTPUT_FRAMES_FOLDER = "workflow_output_frames"

if os.path.exists(WORKFLOW_OUTPUT_FRAMES_FOLDER):
    shutil.rmtree(WORKFLOW_OUTPUT_FRAMES_FOLDER)

os.makedirs(WORKFLOW_OUTPUT_FRAMES_FOLDER, exist_ok=True)
```

Next, add the following helper function to your script. It saves each workflow output frame as a JPEG image, with the frame number included in the filename so that the images remain in order and can later be matched with the corresponding face predictions.

``` python
import cv2

# Save bounding box visualized frames
def save_workflow_frame(workflow_output_image, frame_number):
    frame_path = os.path.join(
        WORKFLOW_OUTPUT_FRAMES_FOLDER,
        f"frame_{frame_number:06d}.jpg"
    )
    cv2.imwrite(frame_path, workflow_output_image)
```

Since we also need to store the face detections returned for each frame, add the following code to your script. It registers a callback that is triggered whenever the workflow returns data for a frame. The callback extracts the workflow output image and stabilized face predictions, then saves the image and stores the corresponding face detections in a dictionary.

``` python
import base64
import numpy as np

# Storage for face prediction coordinates
face_predictions_by_frame = {}

# Workflow data callback
@session.on_data()
def on_prediction(data: dict, metadata: VideoMetadata):

    if not data:
        return
    
    frame_number = int(metadata.frame_id)

    # Get Roboflow output image
    workflow_image_data = data.get("output_image")

    # Get stabilized face predictions
    stabilized_predictions = data.get("stabilized_predictions", {})
    frame_predictions = stabilized_predictions.get("predictions", [])

    # Decode workflow output image from Base64
    workflow_output_image = None

    if workflow_image_data:
        workflow_image_value = workflow_image_data.get("value")

        if workflow_image_value:
            image_bytes = base64.b64decode(workflow_image_value)
            image_array = np.frombuffer(image_bytes, dtype=np.uint8)
            workflow_output_image = cv2.imdecode(
                image_array,
                cv2.IMREAD_COLOR
            )

    # Save workflow output frame
    if workflow_output_image is not None:
        save_workflow_frame(workflow_output_image, frame_number)
        
    # Extract required face bounding boxes
    frame_faces = []

    for prediction in frame_predictions:
        x = float(prediction["x"])
        y = float(prediction["y"])
        width = float(prediction["width"])
        height = float(prediction["height"])

        frame_faces.append({
            "tracker_id": prediction.get("tracker_id"),
            "x": x,
            "y": y,
            "width": width,
            "height": height
        })

    # Store bounding boxes for the current frame
    face_predictions_by_frame[str(frame_number)] = frame_faces

    # Progress counter for workflow frames processed
    if frame_number % 10 == 0:
        print(f"Frame {frame_number}: Workflow response received")
```

Inside the callback, we extract the workflow output image and stabilized face predictions. The output image is decoded from Base64 and saved to disk, while the bounding box coordinates and tracker ID are extracted for each detected face. The detections are then stored in a dictionary indexed by frame number. This allows us to retrieve the location and identity of each tracked face later when performing gaze estimation.

Next, start the workflow by calling **session.run().** This processes the input video frame by frame and triggers the **on_prediction()** callback as each frame is processed. The callback saves the visualized workflow frame and stores the corresponding stabilized face detections.

After the workflow finishes processing the video, save the collected face detections to a JSON file. This creates a persistent record of the face bounding boxes and tracker IDs for every processed frame. We can use this data in the next step to crop the tracked faces and perform gaze estimation on each face.

Add the following code to the end of the same script, then run the entire script:

``` python
# Run workflow
session.run()

import json

# Save face predictions for entire video to JSON
FACE_PREDICTIONS_JSON = "face_predictions.json"

with open(FACE_PREDICTIONS_JSON, "w", encoding="utf-8") as f:
    json.dump(face_predictions_by_frame, f, indent=2)

print("Roboflow processing completed.")
```

After the script finishes processing the video, you will have two outputs:

1. A folder containing the visualized workflow frames, which can later be annotated with gaze information.
2. A JSON file containing the stabilized face detections, including the bounding box coordinates and tracker ID for each face in every processed frame.

These outputs provide everything needed for the next step, where we crop the tracked faces and perform gaze estimation.

### Step 5: Estimate Gaze Direction

With the stabilized and tracked face detections stored from the previous step, we can now perform gaze estimation for each detected face. For this, we will use Roboflow's [inference_models](https://inference-models.roboflow.com/?ref=blog.roboflow.com) package with the L2CS-Net gaze estimation model.

First, install the package (for cpu only devices):

```
pip install "inference_models[onnx-cpu]"
```

Load the L2CS-Net model using AutoModel. The rn50 model uses a ResNet-50 backbone and predicts the person's gaze direction as **yaw** and **pitch** angles. You can add the following code to a new script or continue building on the script from the previous step and run it as a complete pipeline.

``` python
import os
from inference_models import AutoModel

# Load L2CS-Net gaze model
gaze_model = AutoModel.from_pretrained(
    "l2cs-net/rn50",
    api_key=os.getenv("ROBOFLOW_API_KEY")
)
```

In the previous step, we stored the stabilized face detections for each frame in a JSON file. Load this file by adding the following code to the script so that we can retrieve the face bounding boxes and tracker IDs while processing each frame.

``` python
import json

# Stored face predictions JSON file from earlier
FACE_PREDICTIONS_JSON = "face_predictions.json"

# Load face predictions
with open(FACE_PREDICTIONS_JSON, "r", encoding="utf-8") as f:
    face_predictions_by_frame = json.load(f)
```

The workflow output frames were also saved with their frame numbers in the filenames. We can retrieve and sort these frames by frame number to ensure they are processed in the same order as the original video. Add the following code to the script to do this.

``` python
# Extract frame number from filename
def get_frame_number(filename):
    return int(os.path.splitext(filename)[0].split("_")[1])

# Workflow output frames folder defined earlier
WORKFLOW_OUTPUT_FRAMES_FOLDER = "workflow_output_frames"
frame_files = sorted(
    [
        f
        for f in os.listdir(WORKFLOW_OUTPUT_FRAMES_FOLDER)
        if f.lower().endswith(".jpg")
    ],
    key=get_frame_number
)
```

We will draw the estimated gaze direction onto these workflow output frames. To store them, we create a separate folder to store the gaze-visualized frames. Add the following code snippet to the script:

``` python
import shutil

# Gaze visualized frames output folder
GAZE_VISUALIZED_FRAMES_FOLDER = "gaze_visualized_frames"

if os.path.exists(GAZE_VISUALIZED_FRAMES_FOLDER):
    shutil.rmtree(GAZE_VISUALIZED_FRAMES_FOLDER)

os.makedirs(GAZE_VISUALIZED_FRAMES_FOLDER, exist_ok=True)
```

Gaze predictions can vary slightly between consecutive frames, even when a person's actual gaze remains relatively stable. To reduce this frame-to-frame jitter, we can apply exponential smoothing to both the gaze angles and the face center coordinates.

Add the following code to your script to perform gaze and face-center smoothing:

```
# Gaze tracking configuration
GAZE_SMOOTHING_ALPHA = 0.18
FACE_CENTER_SMOOTHING_ALPHA = 0.30

# Smoothing function for gaze and face center values
def smooth_value(previous, current, alpha):
    if previous is None:
        return current

    return alpha * current + (1.0 - alpha) * previous
```

The GAZE_SMOOTHING_ALPHA controls how quickly the smoothed gaze follows new predictions. A lower value produces stronger smoothing, while a higher value responds more quickly to changes. Similarly, FACE_CENTER_SMOOTHING_ALPHA smooths the position of the face center used as the starting point for the gaze arrow.

We can also customize the appearance of the gaze visualization by controlling the length and thickness of the arrow used to represent the estimated gaze direction. Add the following variables to your script to configure these properties:

```
# Gaze arrow visualization configuration
ARROW_LENGTH_SCALE = 0.5
ARROW_THICKNESS = 8
ARROW_TIP_LENGTH = 0.30
```

For each frame, we want to retrieve the corresponding stabilized face detections from the JSON file, crop each detected face using its bounding box, and pass the crop to L2CS-Net for gaze estimation.

The model returns the estimated gaze as yaw and pitch angles. We then smooth these values using the previous prediction associated with the face's **tracker_id**, allowing the pipeline to maintain a separate gaze history for each person when multiple faces are present.

Finally, we convert the smoothed gaze angles into a direction vector and visualize it as an arrow originating from the center of the tracked face.

To do these, add the following code snippet to the script:

``` python
import cv2
import numpy as np

# Tracker state
gaze_state = {}
face_center_state = {}

# Progress counter for gaze frames written
gaze_frames_written = 0

# Process each frame for gaze estimation and visualization
for filename in frame_files:
    frame_number = get_frame_number(filename)
    frame_path = os.path.join(WORKFLOW_OUTPUT_FRAMES_FOLDER, filename)

    img = cv2.imread(frame_path)
    if img is None:
        continue

    # Get face predictions for the current frame
    frame_faces = face_predictions_by_frame.get(str(frame_number), [])

    # Process each detected face
    for face in frame_faces:
        tracker_id = face.get("tracker_id")

        x = float(face["x"])
        y = float(face["y"])
        width = float(face["width"])
        height = float(face["height"])

        # Calculate bounding box coordinates
        x1 = int(x - width / 2)
        y1 = int(y - height / 2)
        x2 = int(x + width / 2)
        y2 = int(y + height / 2)

        # Clamp bounding box to image boundaries
        x1 = max(0, min(x1, img.shape[1] - 1))
        y1 = max(0, min(y1, img.shape[0] - 1))
        x2 = max(0, min(x2, img.shape[1]))
        y2 = max(0, min(y2, img.shape[0]))

        # Skip invalid bounding boxes
        if x2 <= x1 or y2 <= y1:
            continue

        # Crop face in memory
        face_image = img[y1:y2, x1:x2]

        if face_image.size == 0:
            continue

        # Run L2CS-Net gaze estimation
        try:
            gaze_result = gaze_model.infer(face_image)
        except Exception as e:
            print(
                f"Gaze inference failed on frame {frame_number}, "
                f"tracker {tracker_id}: {e}"
            )
            continue

        # Get gaze angles
        try:
            yaw = float(gaze_result.yaw)
            pitch = float(gaze_result.pitch)
        except Exception as e:
            print(
                f"Could not read gaze result on frame {frame_number}, "
                f"tracker {tracker_id}: {e}"
            )
            continue

        # Initialize state for a newly detected face
        if tracker_id not in gaze_state:
            gaze_state[tracker_id] = {"yaw": yaw,"pitch": pitch}

        if tracker_id not in face_center_state:
            face_center_state[tracker_id] = {"x": x,"y": y}

        # Smooth gaze angles
        previous_yaw = gaze_state[tracker_id]["yaw"]
        previous_pitch = gaze_state[tracker_id]["pitch"]
        smooth_yaw = smooth_value(previous_yaw, yaw, GAZE_SMOOTHING_ALPHA)
        smooth_pitch = smooth_value(previous_pitch, pitch, GAZE_SMOOTHING_ALPHA)

        # Save smoothed gaze state
        gaze_state[tracker_id] = {"yaw": smooth_yaw, "pitch": smooth_pitch}

        # Smooth face center
        previous_center_x = face_center_state[tracker_id]["x"]
        previous_center_y = face_center_state[tracker_id]["y"]

        smooth_center_x = smooth_value(previous_center_x, x, FACE_CENTER_SMOOTHING_ALPHA)
        smooth_center_y = smooth_value(previous_center_y, y, FACE_CENTER_SMOOTHING_ALPHA)

        # Save smoothed face center
        face_center_state[tracker_id] = {"x": smooth_center_x, "y": smooth_center_y}

        # Calculate gaze arrow length
        arrow_length = max(width, height) * ARROW_LENGTH_SCALE

        # Convert gaze angles to a direction vector
        dx = -np.sin(smooth_yaw) * arrow_length
        dy = -np.sin(smooth_pitch) * arrow_length

        # Arrow start coordinates
        start_x = int(smooth_center_x)
        start_y = int(smooth_center_y)

        # Arrow end coordinates
        end_x = int(start_x + dx)
        end_y = int(start_y + dy)

        # Draw gaze direction
        cv2.arrowedLine(
            img,
            (start_x, start_y),
            (end_x, end_y),
            (0, 255, 0),
            ARROW_THICKNESS,
            cv2.LINE_AA,
            tipLength=ARROW_TIP_LENGTH
        )

    # Save gaze-visualized frame
    output_path = os.path.join(
        GAZE_VISUALIZED_FRAMES_FOLDER,
        f"frame_{frame_number:06d}.jpg"
    )

    cv2.imwrite(output_path, img)
    gaze_frames_written += 1

    # Progress counter
    if frame_number % 10 == 0:
        print(
            f"Frame {frame_number}: "
            f"Gaze visualization completed"
        )
```

At the end of this process, the **gaze_visualized_frames** folder contains the workflow frames with the estimated gaze direction drawn for each tracked face. Because the gaze state is maintained separately for each **tracker_id,** the smoothing is applied independently to each person in the video. 

These processed frames can then be assembled back into a video to produce the final gaze-tracking visualization.

### Step 6: Generate the Gaze Tracking Video

The gaze estimation and visualization process generates a separate image for each video frame. In the final step, we can combine these processed frames back into a video while preserving the original frame rate and resolution.

To do this, add the following code snippet to the script, where the output video path is defined. The code retrieves the gaze-visualized frames and sorts them by frame number to ensure they are written to the output video in the correct order.

```
GAZE_VIDEO_PATH = "output.mp4"

# Get gaze-visualized frames
gaze_frame_files = sorted(
    [
        f
        for f in os.listdir(GAZE_VISUALIZED_FRAMES_FOLDER)
        if f.lower().endswith(".jpg")
    ],
    key=get_frame_number
)

# Check if gaze-visualized frames were generated
if not gaze_frame_files:
    raise RuntimeError("No gaze visualization frames were generated.")
```

Next, add the code snippet below. It determines the video's dimensions from the first processed frame and retrieves the original video's FPS so that the generated video plays back at the same speed as the input video.

```
# Input video used from earlier
INPUT_VIDEO = "input.mp4"

# Get video dimensions from the first gaze-visualized frame
first_frame = cv2.imread(
    os.path.join(
        GAZE_VISUALIZED_FRAMES_FOLDER,
        gaze_frame_files[0]
    )
)

frame_height, frame_width = first_frame.shape[:2]

# Get original video FPS
cap = cv2.VideoCapture(INPUT_VIDEO)
fps = cap.get(cv2.CAP_PROP_FPS)
cap.release()
```

Now, add the code snippet below to the script. It creates a video writer and writes each gaze-visualized frame to the output video. Once all frames have been written, the video writer is released to finalize the output file.

```
# Create video writer for the final gaze-tracking video
video_writer = cv2.VideoWriter(
    GAZE_VIDEO_PATH,
    cv2.VideoWriter_fourcc(*"mp4v"),
    fps,
    (frame_width, frame_height)
)

# Write gaze-visualized frames to the final video
for filename in gaze_frame_files:
    frame_path = os.path.join(
        GAZE_VISUALIZED_FRAMES_FOLDER,
        filename
    )

    frame = cv2.imread(frame_path)

    if frame is None:
        continue

    video_writer.write(frame)

# Finalize video writing
video_writer.release()

print(f"Gaze tracking video saved to: {GAZE_VIDEO_PATH}")
```

Now, run the script. The resulting **output.mp4** contains the original video with the detected faces, tracking information, and estimated gaze directions visualized over time.

In this way, you can create a complete gaze tracking pipeline that combines face detection, multi-object tracking, detection stabilization, and gaze estimation.

## Key Challenges in Gaze Tracking

- **Maintaining reliable processing speed:** A real-time gaze tracking system often needs to run a face detection model, tracking pipeline, and gaze estimation model together for every frame. Running multiple models can significantly increase computational requirements and latency, making it challenging to maintain a consistent frame rate while still producing accurate gaze predictions.
- **Eyes moving out of the camera frame:** When a person's eyes are partially or completely outside the video frame, gaze estimation becomes much more difficult. Models such as L2CS-Net rely on the eyes, to estimate gaze direction, so their predictions can become unreliable or fail entirely when the eyes are not visible.
- **Rapidly changing gaze:** Quickly shifting the gaze from one direction to another can be difficult to track accurately. The system may lag behind the actual movement or produce unstable predictions when the gaze changes rapidly between consecutive frames.
- **Handling false gaze estimations:** Gaze estimation models can occasionally produce incorrect predictions because of poor lighting, unusual head poses, occlusions, motion blur, or inaccurate face detections, which may throw off the entire tracking.
- **Maintaining stable tracking across frames:** Gaze tracking is a temporal problem, so the system needs to maintain a consistent identity for each detected face across frames. Lost detections and frames, changing bounding boxes, or incorrect tracker assignments can cause the resulting gaze trajectory to become inconsistent.

## Major Applications of Gaze Tracking

Gaze tracking can be applied in a variety of scenarios where understanding a person's visual attention and eye movements is useful. Some of the major applications include:

- **Human-Computer Interaction:** Gaze tracking enables users to interact with computers and digital interfaces using their eye movements. It can be used for selecting, navigating, scrolling, and controlling on-screen elements.
- **Accessibility and Assistive Technology:** Gaze tracking allows people with limited mobility to control computers and communication devices using their eyes. It can support virtual keyboards, communication systems, and hands-free device control.
- **Gaming and Virtual Reality:** Gaze tracking enables games and VR applications to respond to where players are looking. It can support gaze-based controls, immersive interactions, adaptive interfaces, and foveated rendering.
- **User Experience and Usability Research:** Researchers use gaze tracking to analyze how users visually navigate websites, applications, and interfaces. Metrics such as fixations and gaze paths can reveal which elements attract attention and how users interact with a design.
- **Medical and Psychological Research:** Gaze tracking is widely used in research on visual perception, attention, reading behavior, and eye movement. It can provide quantitative data about how people process and respond to visual information.
- **Automotive and Aviation:** Gaze tracking can help analyze how drivers, pilots, and operators visually monitor complex environments. It can provide insights into attention allocation and interaction with instruments, displays, and controls.

## Conclusion

A complete gaze tracking pipeline can combine face detection, multi-object tracking, detection stabilization, and gaze estimation to produce a reliable gaze trajectory from video.

With [Roboflow Workflows,](https://roboflow.com/workflows/build?ref=blog.roboflow.com) you can build much of this pipeline visually using pre-built computer vision models and processing blocks, without having to implement each component from scratch. Roboflow Agent can further accelerate development by helping you create and configure workflows using natural language. 

You can then use Roboflow's inference models such as L2CS-Net to add specialized capabilities like gaze estimation to your application**.**

Roboflow provides the tools you need to move from a computer vision idea to a working system faster. Start building your own computer vision pipeline with Roboflow [today.](https://app.roboflow.com/?ref=blog.roboflow.com)

### **Cite this Post**

Use the following entry to cite this post in your research:

[Dikshant Shah](/author/dikshant/). (Sep 8, 2026).
      Gaze Detection and Gaze Tracking Explained. Roboflow Blog: https://blog.roboflow.com/gaze-detection/