DIY AI Physical Therapist: Real-Time Pose Correction with React Native and MediaPipe

Have you ever tried doing physical therapy exercises at home, only to wonder if your form is actually helping or just making things worse? 🤕 Traditional home rehabilitation often lacks the "watchful eye" of a professional. However, with the rise of on-device computer vision and React Native development, we can now build powerful, low-latency movement correction tools that run directly on a smartphone.

In this tutorial, we are diving deep into MediaPipe pose estimation and mobile AI integration to build a "Smart Rehab Coach." We’ll explore how to capture real-time landmarks, calculate joint angles, and provide instant feedback—all while maintaining 60 FPS performance. By leveraging TensorFlow Lite and on-device processing, we ensure user privacy while delivering a seamless experience. 🚀

To achieve real-time feedback, we need a pipeline that minimizes the "bridge" overhead in React Native. We use a frame processor to pipe camera data directly into the MediaPipe inference engine.

graph TD
  A[Camera Stream] -->|Native Frame| B(MediaPipe Pose Landmarker)
  B -->|33 3D Landmarks| C{Angle Calculation Engine}
  C -->|Compare| D[Reference Library]
  D -->|Feedback| E[React Native UI Overlay]
  E -->|Visual Cues| F[User]

  subgraph "On-Device Processing"
  B
  C
  D
  end

Before we start coding, ensure your environment is ready:

The heart of our application is the MediaPipe Pose Landmarker. Unlike cloud-based solutions, this runs locally on the device's NPU/GPU.

import { PoseLandmarker, FilesetResolver } from "@mediapipe/tasks-vision";

async function createPoseLandmarker() {
  const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
  );

  const poseLandmarker = await PoseLandmarker.createFromOptions(vision, {
    baseOptions: {
      modelAssetPath: `pose_landmarker_lite.task`,
      delegate: "GPU" // Critical for real-time performance ⚡
    },
    runningMode: "VIDEO",
    numPoses: 1
  });

  return poseLandmarker;
}

To tell a user their arm isn't straight enough, we need to calculate the angle between three points (e.g., Shoulder, Elbow, Wrist). We use the Law of Cosines or the atan2 function.

// Helper to calculate angle between three landmarks
const calculateAngle = (a: Landmark, b: Landmark, c: Landmark): number => {
  const radians = Math.atan2(c.y - b.y, c.x - b.x) - 
                  Math.atan2(a.y - b.y, a.x - b.x);
  let angle = Math.abs((radians * 180.0) / Math.PI);

  if (angle > 180.0) {
    angle = 360 - angle;
  }
  return angle;
};

// Implementation for a Bicep Curl or Overhead Press
const checkForm = (landmarks: Landmark[]) => {
  const shoulder = landmarks[11];
  const elbow = landmarks[13];
  const wrist = landmarks[15];

  const elbowAngle = calculateAngle(shoulder, elbow, wrist);

  if (elbowAngle > 160) return "Fully Extended";
  if (elbowAngle < 45) return "Good Contraction";
  return "Keep Moving!";
};

In React Native, we use react-native-vision-camera with a frame processor. This allows us to run our logic on every single frame captured by the lens.

function RehabCamera() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet';
    const poses = detectPose(frame); // Native call to MediaPipe

    if (poses.length > 0) {
      const angle = calculateAngle(poses[0].landmarks[11], ...);

      // Update UI via Shared Values
      if (angle < 90) {
        feedbackText.value = "Lower your hips!";
      }
    }
  }, []);

  return (
    <Camera
      device={device}
      isActive={true}
      frameProcessor={frameProcessor}
    />
  );
}

Building a production-ready vision app involves more than just landmark detection. You need to handle jitter (using a Kalman Filter), varying light conditions, and different body types.

For those looking to scale this into an enterprise-grade solution, check out the specialized patterns on WellAlly Tech Blog. They cover advanced topics like:

Integrating these patterns ensures that your app doesn't just "detect" poses, but actually "understands" human movement at a clinical level.

On-device AI is transforming how we approach healthcare and fitness. By combining React Native for the UI and MediaPipe for the intelligence, we can create low-latency, private, and highly effective rehabilitation tools.

The future of physical therapy isn't just in the clinic—it's in the pocket of every patient. 📱💪

Are you building something with Pose Estimation? Drop a comment below or share your repo! I’d love to see how you’re handling landmark smoothing!