Apex Teleoperation FAQ and Quick Reference

Updated: August 6, 2026
Products: Marvin Pro, Gento Skye, and Gento Luna
Controllers: Orin, Tianzhun, and Lingjing Thor
End effectors: DM/OmniGripper, ZY gripper, and Wuji dexterous hand
Purpose: customer troubleshooting, technical-support training, and on-site quick reference

1. Before Troubleshooting

Q1: Why must the complete machine configuration be recorded first?

Pro and Gento use different control chains, while Ubuntu 22.04/Humble and Ubuntu 24.04/Jazzy require different packages. Record at least:

Product line: Marvin Pro / Gento
Robot: Pro / Skye / Luna
Arms: M6 696 / M6 Lite / M3 / Gento configuration
Controller: Orin / Tianzhun / Lingjing Thor
OS and ROS: Ubuntu 22.04 + Humble / Ubuntu 24.04 + Jazzy
End effector: DM / ZY / Wuji / none
Software: kernelmind-apex, kernelmind-apex-tool, Apex-Teleop, SDK, headset APK
Network: controller IP, robot IP, headset Host IP, ROS_DOMAIN_ID

On the target machine:

dpkg --print-architecture
lsb_release -a
echo "$ROS_DISTRO"
dpkg-query -W -f='${Package} ${Version} ${Architecture}\n' \
  | grep -Ei 'kernelmind|apex|teleop|marvin|gento|ros-(humble|jazzy)'

Controller	Ubuntu	ROS 2	Architecture
Orin	22.04	Humble	arm64
Tianzhun	22.04	Humble	arm64
Lingjing Thor	24.04	Jazzy	arm64

Q2: What general checks should be run first?

source /etc/apex/apex_ros_env.sh 2>/dev/null || \
  source /opt/kernelmind/apex/install/setup.bash
systemctl --no-pager --failed
ros2 node list
ros2 topic list
ros2 service list

Check the five core services:

systemctl status apex-backend.service --no-pager
systemctl status apex-robot.service --no-pager
systemctl status apex-camera.service --no-pager
systemctl status apex-teleop.service --no-pager
systemctl status apex-tool.service --no-pager

Older releases may not have a separate apex-tool.service. Before testing motion, clear the workspace, reduce the motion range, and confirm that no second command source is active.

2. Startup, Frontend, and Services

Q3: What is the standard startup order?

Check network and machine configuration
-> start Backend
-> verify GMSL/Argus
-> Start Robot in ApexApp
-> Start Camera
-> Start Teleop
-> start Tool for the installed end effector
-> connect the headset and perform a small-motion test
-> start recording last

Start Backend with:

sudo systemctl start apex-backend.service

Q4: When should Backend be restarted?

Inspect it first:

systemctl status apex-backend.service --no-pager
curl -sS -o /dev/null -w 'HTTP %{http_code}\n' http://127.0.0.1:8080/docs
journalctl -u apex-backend.service -n 100 --no-pager

Restart it when the frontend cannot reach the local API, port 8080 is unresponsive because Backend is stuck, /etc/apex/apex.env was changed, or Backend itself is repeatedly failing:

sudo systemctl restart apex-backend.service

Camera timeouts, robot disconnects, and gripper faults should be handled in their own modules. Restarting Backend is not a universal repair.

Q5: Which service corresponds to each module?

Symptom	First service or component
Frontend API, module buttons, or port 8080	apex-backend.service
Robot connection, robot IP, or SDK state	apex-robot.service
Camera black screen, configuration, or video process	apex-camera.service
Headset input, IK/QP, or teleoperation mode	apex-teleop.service
DM/ZY/Wuji end-effector node	apex-tool.service or the manually launched node
Argus has no frames or image-buffer timeout	nvargus-daemon, then Camera

journalctl -u apex-robot.service -n 100 --no-pager
journalctl -u apex-camera.service -n 100 --no-pager
journalctl -u apex-teleop.service -n 100 --no-pager
journalctl -u apex-tool.service -n 100 --no-pager

Q6: Should Teleop mode be controller or dexhand?

grep -E 'APEX_TELEOP_MODE|APEX_TOOL_TYPE' /etc/apex/apex.env

• Standard controllers with DM/ZY grippers usually use APEX_TELEOP_MODE=controller.
• Gloves or dexterous hands use APEX_TELEOP_MODE=dexhand.
• Some releases automatically select glove mode for APEX_TOOL_TYPE=wuji or wujihand.
• End-effector values are generally dm, zy, wuji, or none.

After editing, restart Teleop and Tool:

sudoedit /etc/apex/apex.env
sudo systemctl restart apex-teleop.service
sudo systemctl restart apex-tool.service

Q7: What should be checked when the launch terminal shows red errors?

Find the earliest specific error before a summary such as process has died. Common signatures include missing shared libraries, missing files, symbol lookup errors, exit code 127, version mismatch, and robot connection loss.

ldd <executable-path-from-the-error> | grep 'not found'

Confirm Ubuntu, ROS distribution, arm64 architecture, and release compatibility before installing a dependency. Do not link unrelated .so versions together.

Q8: What if /etc/apex/apex_ros_env.sh does not exist?

ls -l /etc/apex
ls -l /opt/kernelmind/apex/install/setup.bash

For a temporary shell:

source /opt/ros/<humble-or-jazzy>/setup.bash
source /opt/kernelmind/apex/install/setup.bash

Do not mix Humble and Jazzy. If the Apex setup file is also absent, verify that kernelmind-apex is installed correctly.

Q9: What if the installed environment references a build-machine path?

A message such as not found: /home/marvin/test_apex_ws/install/local_setup.bash indicates a leaked build prefix. Locate it with:

grep -Rns '/home/marvin/test_apex_ws/install' /opt/kernelmind/apex/install 2>/dev/null

Back up the affected text environment file and relocate the prefix to /opt/kernelmind/apex/install. Do not create a fake build-workspace directory to hide the error.

3. Camera, Argus, H264, and WebRTC

Q10: Where are camera IDs and camera_sources configured?

/opt/kernelmind/apex/install/gmsl_quadcam/share/gmsl_quadcam/config

systemctl cat apex-camera.service
grep -Rns 'camera_sources' \
  /opt/kernelmind/apex/install/gmsl_quadcam/share/gmsl_quadcam/config

Common files are quad_csi_quickview.yaml and sh5_quad_csi_quickview.yaml.

Controller	Common CSI IDs
Lingjing Thor	0/1/2/3
Tianzhun	0/1/4/5
Orin DK	0/6/7/8

Actual IDs must be verified on the target hardware. Restart Camera after changing them.

Q11: How should unused camera inputs be configured?

Set missing cameras to none:

camera_sources:
  head_left: "csi://0"
  head_right: "csi://1"
  hand_left: "none"
  hand_right: "none"

Q12: How is an upside-down composite corrected?

flip_180: true

Restart Camera after changing it. This option rotates the whole composite by 180 degrees; per-camera rotation or mirroring may not be supported by the installed release.

Q13: How should image-buffer timeout or rendering black be handled?

Stop Camera, test each real sensor ID with the delivered Argus utility, restart Argus if wiring and configuration are correct, and then restart Camera:

sudo systemctl stop apex-camera.service
command -v argus_camera
argus_camera -d 0
sudo systemctl restart nvargus-daemon
sudo systemctl restart apex-camera.service

Q14: How can GMSL hardware and drivers be checked?

systemctl --type=service --all | grep -Ei 'gmsl|camera|argus'
dmesg | grep -Ei 'gmsl|csi|argus' | tail -n 100
v4l2-ctl --list-devices

If each camera works in the low-level test, inspect Apex sources, encoding dependencies, and service parameters. If it does not, inspect power, cables, the deserializer, connectors, and sensor IDs.

Q15: Why might /usb_cam_0/image_raw be absent?

That USB-camera topic does not apply to every GMSL release. Inspect actual image topics:

ros2 topic list | grep -Ei 'image|camera|quad|compressed'
ros2 topic info /quad_tile/compressed -v

GMSL versions may expose only /quad_tile/compressed. If no image topics exist, verify ROS Domain, Camera service, Argus frames, camera_sources, and the quad_csi_quickview dependencies.

Q16: How is a missing WebRTC video stream diagnosed on Windows or in a browser?

Open edge://webrtc-internals or chrome://webrtc-internals.

• No PeerConnection or createOffer: inspect frontend initialization, release, codec configuration, and playback conditions.
• PeerConnection with no video: inspect the video track, H264 profile, decoder, bitrate, packet loss, and signaling logs.
• No camera frames: repair GMSL/Argus first.

Q17: What if Thor reports libnppicc.so.13 => not found?

Verify arm64 and the Jetson repository before installing NPP:

dpkg --print-architecture
apt-cache policy libnpp-13-0 libnpp-dev-13-0
apt-cache show libnpp-13-0 | grep -E 'Architecture|Version'
sudo apt install libnpp-13-0 libnpp-dev-13-0
sudo ldconfig
ldd /opt/kernelmind/apex/install/gmsl_quadcam/lib/gmsl_quadcam/quad_csi_quickview \
  | grep 'not found'

Never use an ubuntu2404/x86_64 CUDA repository on an arm64 Thor.

4. Robot Connection and Teleoperation

Q18-Q21: Robot IP, disconnects, and version mismatch

robot_ip must equal the real robot-controller IP. The host interface must be on a reachable subnet but must not use the same address.

grep -Rns 'robot_ip' /opt/kernelmind/apex/install/*/share/*/config 2>/dev/null
ip -br address
ip route get <ROBOT_IP>
ping -c 4 <ROBOT_IP>

After changing the active YAML, restart Robot and Teleop. Persistent Robot connection lost is not normal even if it later reconnects; inspect packet loss, power and cabling, competing SDK clients, and SDK/firmware compatibility. A controller version of zero or Version mismatch requires a matched host, SDK, and controller firmware release.

Q22-Q24: The headset connects but one or both arms do not move

Find the last topic in the command chain that still has data:

source /etc/apex/apex_ros_env.sh
ros2 topic echo /tj/info/vr_connected --once
ros2 topic echo /tj/control/target_poseL --once
ros2 topic echo /tj/control/enableL --once
ros2 topic echo /tj/control/teleop/ik_request --once
ros2 topic echo /tj/control/ik_request --once
ros2 topic echo /tj/control/qp_controller/joint_cmd_A --once
ros2 topic echo /tj/control/input_mode --once
ros2 topic echo /tj/control/joint_cmd_A --once
ros2 topic echo /tj/info/robot_state --once

Pro uses /tj/control/ik_request/vr, while Gento uses /tj/control/teleop/ik_request. Releases may expose different state topics, so use the target ros2 topic list. A working gripper does not prove that arm IK/QP, Ready, or final commands are healthy. For a one-sided fault, compare L/R targets, enables, QP output, final commands, feedback, power, cabling, and SDK error codes.

Q25-Q27: Headset network and system prompts

For controller_udp connect failed, verify the Host IP, routes, firewall, matching APK, and the TCP 9010 session.

Port	Protocol	Purpose
9000/9001	UDP	Left/right controller input
9002/9003	UDP	Left/right end-effector feedback
9004	UDP	Body or auxiliary data
9010	TCP	Session and heartbeat
8888	UDP	Host discovery

ss -lntup | grep -E ':8888|:9000|:9001|:9002|:9003|:9004|:9010'

A DHCP offer only means that the server attempted an assignment. Check the headset's actual IP, configured Host IP, controller interface IP, TCP and UDP traffic, APK version, and permissions. Complete headset boundary, safety, privacy, camera, and storage prompts before starting Apex.

Q28-Q29: Arm Kp/Kd and oscillation

Common Pro parameter files are under:

/opt/kernelmind/apex/install/marvin_ros_control/share/marvin_ros_control/config

Locate left_kd and right_kd, back up the active file, preserve array lengths and floating-point formatting, change values only slightly, restart Robot then Teleop, and validate unloaded single-arm motion first. If oscillation begins, stop immediately and restore the backup. Arm gains, QP task weights, and gripper MIT gains are different settings.

Q30: Gento shakes or stops while leaning forward with arms extended

This is a known issue in affected releases. Avoid repeatedly reaching the problematic limit and update to a Teleop release that supports the newer Gento SDK, then validate leaning, extension, loaded holding, and recovery.

Q31: How should a soft stop be cleared?

Confirm safety, inspect the fault and stop reason, clear the fault, perform ServoReset/controller reset if required, restore the mode, and set Ready again. Query the actual target services before calling them:

ros2 service list | grep control
ros2 service type <service-name>

5. DM, ZY, and Wuji End Effectors

Q32-Q33: Duplicate Tool nodes and missing gripper commands

Never run a systemd Tool node and a manual node against the same device:

systemctl status apex-tool.service --no-pager
ros2 node list | grep -Ei 'gripper|tool|hand'
ros2 topic info /tj/control/gripperValueL -v

For missing /control/gripperValueL/R, verify ROS_DOMAIN_ID, APEX_TELEOP_MODE, APEX_TOOL_TYPE, publishers, and a compatible headset APK. All nodes and terminals must use the same Domain.

Q34: How is a DM gripper started, reset, and tested once?

source /etc/apex/apex_ros_env.sh
source /opt/kernelmind/apex_tool/install/setup.bash 2>/dev/null || true
ros2 launch dm_gripper_py dm_gripper.launch.py
ros2 service call /tj/control/reset_grippers std_srvs/srv/Trigger '{}'
ros2 topic pub --once /tj/control/gripperValueL std_msgs/msg/Float32 '{data: 0.2}'
ros2 topic pub --once /tj/control/gripperValueR std_msgs/msg/Float32 '{data: 0.2}'

Command direction and scaling depend on the delivered launch. Do not mix raw motor positions with normalized ROS values.

Q35-Q37: Missing vCAN, one-sided enable failure, or no elasticity

ip -details link show vcan0
ip -details link show vcan1
ros2 topic echo /tj/info/gripper_feedback_L --once
ros2 topic echo /tj/info/gripper_feedback_R --once
ros2 topic echo /tj/info/gripper_feedback_L_err --once
ros2 topic echo /tj/info/gripper_feedback_R_err --once

Creating an empty vCAN interface does not bridge data to physical hardware. Check Robot's end-effector bridge, power, terminal board, fuses, cables, IDs, CAN channels, initialization logs, and error codes. Reset the grippers before changing MIT gains.

Q38-Q40: DM gains, two-gripper interference, and small position error

Locate the active DM_gripper.py and controlMIT calls before modifying Kp/Kd:

find /opt/kernelmind -path '*dm_gripper_py*' -name 'DM_gripper.py' 2>/dev/null
grep -Rns 'controlMIT' /opt/kernelmind/apex*/install 2>/dev/null

When both grippers interfere, first eliminate duplicate nodes and command publishers, then inspect motor IDs, left/right channels, vCAN traffic, power drop, command rate, and smoothing. Small feedback error may be normal mechanical clearance or load; investigate growing error, failure to reach the target, oscillation, or error codes.

Q41: What if ZY reports executable 'zy_gripper_node' not found?

ros2 pkg prefix zy_gripper_py
ls /opt/kernelmind/apex*/install/zy_gripper_py/lib/zy_gripper_py
ls /opt/kernelmind/apex*/install/zy_gripper_py/share/zy_gripper_py/launch

Install the complete colcon package output, not only share/launch, or rebuild it in the correct ROS environment.

Q42-Q43: Wuji topics or unstable tracking

echo "$ROS_DOMAIN_ID"
grep -E 'ROS_DOMAIN_ID|APEX_TELEOP_MODE|APEX_TOOL_TYPE' /etc/apex/apex.env
ros2 topic list | grep -Ei 'hand|glove|footkey'

Wuji generally requires APEX_TOOL_TYPE=wuji and dexhand/glove mode. Expected topics include hand commands, hand states, and /control/footkey. If topics are healthy but motion is unstable, keep the controller/tracker visible to the headset and verify glove, pedal enable, side mapping, commands, and feedback.

6. ROS Domain, Topics, and Command Sources

Q44: Nodes run but cannot see each other's topics after an update

echo "$ROS_DOMAIN_ID"
echo "$ROS_LOCALHOST_ONLY"
grep -E 'ROS_DOMAIN_ID|ROS_LOCALHOST_ONLY' /etc/apex/apex.env

All participants must use the same Domain. Cross-machine DDS generally requires ROS_LOCALHOST_ONLY=0. Restart affected services and open a fresh terminal after changing the environment.

Q45: How can input, QP, and final-source failures be separated?

ros2 topic echo /tj/control/teleop/ik_request --once
ros2 topic echo /tj/control/ik_request --once
ros2 topic echo /tj/control/qp_controller/joint_cmd_A --once
ros2 topic echo /tj/control/input_mode --once
ros2 topic echo /tj/control/joint_cmd_A --once

Last healthy stage	Next area to inspect
No Teleop IK	Headset, enable, controller_udp
Teleop IK but no selected IK	IK source/mux
IK but no QP	QP, model, robot state, libraries
QP but no final command	input_mode and joint_cmd_mux
Final command but no motion	Ready, SDK, network, hardware

Q46: Is the Jazzy ROS_LOCALHOST_ONLY warning itself a fault?

Not necessarily, but a value of 1 still restricts cross-machine discovery. Check its actual effect rather than ignoring it because the message is only a warning.

7. Libraries, Releases, and Dependencies

Q47-Q49: Missing Pinocchio, eiquadprog, MarvinSDK, or incomplete packages

ldd /opt/kernelmind/apex/install/marvin_teleop/lib/marvin_teleop/teleop_manager | grep 'not found'
ldd /opt/kernelmind/apex/install/marvin_qp_controller/lib/marvin_qp_controller/qp_controller | grep 'not found'
find /usr/local /usr /opt -name '*MarvinSDK*.so*' 2>/dev/null
ldconfig -p | grep -i marvin
sudo dpkg --audit

On Jazzy, matching packages may include ros-jazzy-pinocchio and ros-jazzy-eiquadprog. A specific missing SONAME means ABI mismatch; install the build-time version or rebuild on the target instead of creating a false symlink. In dpkg -l, only ii means fully configured.

Q50: What if apt reports amd64 does not match system arm64?

Stop immediately and do not run apt autoremove. Disable the incorrect x86 CUDA source, clean apt, restore the correct Jetson arm64 repository, and simulate repair first:

sudo dpkg --configure -a
sudo apt clean
sudo apt update
sudo apt --fix-broken install -s

Do not proceed if the simulation removes JetPack, L4T, CUDA, TensorRT, or ROS core packages.

Q51: How are Apex Python dependency conflicts checked?

/opt/kernelmind/venv/bin/pip check

Repair only the Apex virtual environment, not system Python. A healthy result is No broken requirements found.

Q52: How is release compatibility confirmed?

Record kernelmind-apex, kernelmind-apex-tool, Apex-Teleop, SDK, controller firmware, headset APK, Ubuntu, ROS, and architecture. Simulate package installation first:

sudo apt-get -s install ./<package>.deb

8. Recording, Storage, and Rates

Q53-Q54: Damaged MCAP, missing metadata, or unavailable storage

Check recorder exit, disk disconnect/read-only/full state, stop/flush completion, and the actual recorder package:

mount | grep BAG_STORAGE
df -h | grep BAG_STORAGE
ros2 pkg prefix recording_playback_nodes_py
ros2 pkg prefix bag_recorder_nodes_py
grep -n 'BAG_STORAGE_ROOT' /etc/apex/apex.env

The common location is /media/<user>/BAG_STORAGE/recorded_bags. Perform a short recording and confirm both MCAP and metadata.yaml before a long test.

Q55: How should EEF, gripper, and joint rates be measured?

source /etc/apex/apex_ros_env.sh
ros2 topic hz /tj/info/eef_left
ros2 topic hz /tj/info/eef_right
ros2 topic hz /tj/info/gripper_feedback_L
ros2 topic hz /tj/info/gripper_feedback_R
ros2 topic hz /tj/info/joint_feedback
ros2 topic hz /tj/joint_states

Record warm-up, duration, motion load, publisher count, sample count, mean, standard deviation, P95/P99, maximum period, and dropouts. A historical Marvin Pro 120-second full-load test measured approximately 939 Hz for EEF, 198 Hz for grippers, 200 Hz for joint feedback, and 100 Hz for /tj/joint_states. These are references, not hard real-time guarantees, and must not be used as Skye/Luna acceptance values. See Marvin Pro ROS Topic List and Gento ROS 2 Interfaces for current design rates.

9. Licensing and Other Issues

Q56: What do UID directory not found and Not licensed for this machine mean?

The target registration material is missing and teleop_manager exits. Follow the authorized UID deployment process. Do not bypass licensing with empty or fabricated files.

Q57: Does an offline ToDesk or NoMachine session mean Apex failed?

No. Remote-desktop connectivity and Apex services are separate systems and must be diagnosed separately.

Q58: What does Address already in use mean?

Another process already listens on the frontend port:

ss -ltnp | grep ':<PORT>'
ps -ef | grep '[a]pp.py'

Use the existing service or stop the identified old process. Do not repeatedly launch duplicate frontends.

Q59: Why can Tool packages be absent while main Apex is installed?

Newer releases may deliver kernelmind-apex-tool as a separate package under /opt/kernelmind/apex_tool/install:

dpkg-query -W -f='${Package} ${Version} ${Architecture}\n' \
  | grep -Ei 'kernelmind-apex-tool|kernelmind-apex'
source /etc/apex/apex_ros_env.sh
source /opt/kernelmind/apex_tool/install/setup.bash
ros2 pkg prefix apex_tool
ros2 pkg prefix dm_gripper_py
ros2 pkg prefix zy_gripper_py

Source main Apex first and the Tool overlay second.

10. Symptom Quick Reference

Symptom	First check	Direction
ApexApp cannot manage modules	Backend/API and 8080/docs	Inspect and restart Backend if required
Camera buffer timeout	Low-level Argus test	Restart Argus, then Camera
Wrong/missing camera	camera_sources	Use actual CSI IDs; set absent inputs to none
Upside-down image	flip_180	Change YAML and restart Camera
Missing USB image topic	Actual image topics	GMSL may use /quad_tile/compressed
No WebRTC PeerConnection	Frontend and codec	Inspect browser WebRTC internals
Robot cannot connect	IP and subnet	Correct active YAML; restart Robot and Teleop
Gripper works but arms do not	IK/QP/Ready chain	Find the first topic without data
controller_udp connect failed	Host IP and ports	Verify APK, routes, firewall, TCP session
Topics disappear after update	ROS Domain	Unify ROS_DOMAIN_ID
Exit code 127	Shared libraries	Run ldd and install matching dependencies
amd64 package on arm64	Repository architecture	Stop and repair repositories safely
DM No such device	vCAN and bridge	Verify interface, Robot bridge, and physical CAN
Two grippers oscillate	Duplicate publishers	Keep one source, then inspect power and CAN
ZY executable missing	Incomplete install	Install full colcon output
MCAP damaged, no YAML	Recorder and storage	Verify mount, flush, and active package
UID directory missing	License registration	Deploy authorized UID material

11. Common Paths

Purpose	Path
Apex root	/opt/kernelmind/apex
Main install	/opt/kernelmind/apex/install
Tool overlay	/opt/kernelmind/apex_tool/install
Environment configuration	/etc/apex/apex.env
ROS environment script	/etc/apex/apex_ros_env.sh
Robot configuration	/opt/kernelmind/apex/install/marvin_ros_control/share/marvin_ros_control/config
New Robot configuration candidate	/opt/kernelmind/apex/install/marvin_ros2_control/share/marvin_ros2_control/config
Teleop configuration	/opt/kernelmind/apex/install/marvin_teleop/share/marvin_teleop/config
QP configuration	/opt/kernelmind/apex/install/marvin_qp_controller/share/marvin_qp_controller/config
Camera configuration	/opt/kernelmind/apex/install/gmsl_quadcam/share/gmsl_quadcam/config
Apex ROS logs	/var/log/apex/ros
User ROS logs	~/.ros/log
Recording directory	/media/<user>/BAG_STORAGE/recorded_bags
Gento IK playback	~/gento_replay_bags

12. Information to Return to Support

1. Robot, arms, controller, and end-effector configuration
2. Ubuntu, ROS, and CPU architecture
3. kernelmind-apex, Apex-Teleop, Tool, SDK, and APK versions
4. Operation immediately before the failure
5. Complete error output, especially the first and final errors
6. Service status and latest 100 journal lines
7. ROS_DOMAIN_ID and ROS_LOCALHOST_ONLY
8. Relevant topic info/echo output
9. Reproducibility and which restart temporarily recovers it
10. Modified configuration files with before/after values

Separate confirmed facts, verified experience, and unverified hypotheses. Do not label a terminal board, collision, network, or motor as the final cause without logs or a controlled swap test.