NVIDIA Eye Contact NIM leverages state-of-the-art AI models to dynamically redirect a user’s eye position toward the camera to simulate natural eye contact and enhance remote digital engagement. NVIDIA Eye Contact NIM models are built on the NVIDIA software platform, incorporating CUDA, TensorRT, and Triton to offer out-of-the-box GPU acceleration.
Architecture# #
NVIDIA Eye Contact operates on a region of interest around the eyes, also known as the eye patch. The eye patch is extracted from a video frame using the NVIDIA face tracking pipeline, which computes the 2D facial landmarks and the 6DOF head pose from the video frame. This head pose is then fed into the eye contact network.
The eye contact network has a disentangled encoder-decoder architecture. The encoder estimates the gaze angle from the input eye patch along with a set of features, also known as embeddings. Based on these embeddings, the decoder performs redirection of the gaze in the input patch to make the face look forward.
The final stage of the pipeline involves blending the eye patch back into the original video frame using an inverse transformation.
For more details about the model, read [Improve Human Connection in Video Conferences with NVIDIA Maxine Eye Contact](https://developer.nvidia.com/blog/improve-human-connection-in-video-conferences-with-nvidia-maxine-eye-contact/) in the NVIDIA Developer Technical Blog.
## Input Modes[#](#input-modes)
NVIDIA Eye Contact NIM supports two modes of operation:
- Streaming mode (recommended) In this mode, the input files are streamed to the NIM in chunks. As the chunks arrive, NIM runs inference incrementally and streams the output back to the client in chunks, even before the whole input file is received by the NIM. The server automatically detects streamable videos and uses the streaming inference path internally, so no extra flag is required to enable this behavior on the server. For best performance, use streamable videos as input.
- Transactional mode In this mode, whole input files are uploaded to the NIM before the inference begins. The inference is run on the input files to obtain the output file, which is then sent back to the client. The server automatically falls back to this mode when the input video is not streamable.
For more details, refer to the [Input Modes](basic-inference.html#input-modes) section of Basic Inference.
## Try It Out[#](#try-it-out)
Try the NVIDIA Eye Contact NIM at [build.nvidia.com/nvidia/eyecontact](https://build.nvidia.com/nvidia/eyecontact).
To experience the NVIDIA Eye Contact NIM API without having to host your own servers, use the Try API feature, which uses the NVIDIA Cloud Function backend.