Combine camera perception and language understanding into a robot pipeline.
The robot receives a visual scene, extracts useful structured information, combines it with a user instruction, and produces a validated task plan for ROS 2 execution.
By the end of this tutorial, you will have:
You should have:
Version note: NVIDIA Jetson, JetPack, CUDA, TensorRT, Isaac ROS, and ROS 2 compatibility changes over time. Check the current NVIDIA support matrix and the documentation for your exact board before installing packages. Do not blindly mix commands from different JetPack/ROS 2 releases.
Start by confirming the device and installed software:
uname -a
cat /etc/os-release
Then update package metadata:
sudo apt update
Keep the base system consistent with the JetPack release supported by your target robotics stack.
Install the ROS 2 distribution supported by your Jetson/Isaac ROS combination.
After installation, source ROS 2:
source /opt/ros/<ros-distro>/setup.bash
Verify that ROS 2 is available:
ros2 --help
Add the source command to your shell configuration if appropriate:
echo "source /opt/ros/<ros-distro>/setup.bash" >> ~/.bashrc
source ~/.bashrc
mkdir -p ~/robot_ws/src
cd ~/robot_ws
colcon build
source install/setup.bash
A typical workspace becomes:
robot_ws/
├── src/
├── build/
├── install/
└── log/
For Python:
cd ~/robot_ws/src
ros2 pkg create --build-type ament_python robot_ai_demo
For C++:
ros2 pkg create --build-type ament_cmake robot_ai_demo_cpp
Choose the language that best matches the latency and integration requirements of your application.
A production robot should separate responsibilities.
Sensors
|
v
ROS 2 Drivers
|
v
Perception / Localization
|
v
Decision / Mission Logic
|
v
Safety Layer
|
v
Motor Controller
For a Flutter operator application:
Flutter
|
HTTPS / WebSocket
|
Robot Gateway
|
ROS 2
|
Jetson
|
Robot
The Flutter application should normally communicate with a controlled gateway instead of directly exposing the ROS graph to the public internet.
Create a small publisher and subscriber, then build the workspace:
cd ~/robot_ws
colcon build --symlink-install
source install/setup.bash
Run the publisher:
ros2 run robot_ai_demo publisher
In another terminal:
source ~/robot_ws/install/setup.bash
ros2 topic list
ros2 topic echo /robot_status
This simple test proves that your ROS 2 environment is functioning before you add cameras, AI models, or motor controllers.
For this tutorial, the main component is conceptually one of:
Keep this component independent from the UI. Publish structured ROS 2 messages instead of UI-specific data.
Example:
camera/image
|
v
object_detector
|
v
/objects
|
+----> decision_node
|
+----> telemetry_gateway
At minimum, log:
Useful ROS 2 commands include:
ros2 node list
ros2 topic list
ros2 topic info /robot_status
ros2 topic hz /robot_status
Never allow an AI model or remote UI to directly bypass safety logic.
A simple command path should be:
User/AI Intent
|
v
Command Validation
|
v
Robot State Check
|
v
Safety Rules
|
v
ROS 2 Command
Examples of safety rules:
For Flutter projects, expose a small API such as:
GET /api/robot/status
GET /api/robot/telemetry
POST /api/robot/command
WS /ws/robot
Example WebSocket payload:
{
"type": "command",
"command": "stop",
"sequence": 1024
}
Flutter can then maintain:
ConnectionState
RobotState
TelemetryState
MissionState
AlertState
Use BLoC, Riverpod, or another state-management approach to keep network events separate from presentation.
Test one layer at a time.
ros2 topic list
ros2 topic echo /robot_status
Measure:
Test:
Verify:
Do not optimize before measuring.
Record a baseline and then investigate:
For NVIDIA-accelerated applications, investigate TensorRT, DeepStream, and Isaac ROS where they match the workload.
Record:
Jetson model:
JetPack:
CUDA:
TensorRT:
ROS 2:
Isaac ROS:
Python:
Model:
Camera:
LiDAR:
For serious deployments, containerize the application and keep configuration separate from application code.
source /opt/ros/<ros-distro>/setup.bash
source ~/robot_ws/install/setup.bash
ros2 pkg list | grep robot
Check:
ros2 topic list
ros2 topic info /your_topic
ros2 topic hz /your_topic
Then verify that the sensor publisher is actually running.
Profile the complete pipeline. Do not assume the neural network is the only bottleneck. Camera conversion, memory copies, preprocessing, ROS serialization, and postprocessing can all contribute significant latency.
Implement:
Before deploying a robot, verify:
NVIDIA Jetson is most useful when it is treated as an edge-computing platform inside a larger robotics architecture rather than simply as a small Linux computer. ROS 2 provides the communication and modularity layer, while NVIDIA acceleration can handle demanding perception workloads.
For Flutter-based robotics applications, a gateway between Flutter and ROS 2 creates a clean separation: the mobile application focuses on user experience, while Jetson and ROS 2 remain responsible for robot-side computation.