WebRTC – Teleoperation

WebRTC is is a free, open project that provides browsers and mobile applications with Real-Time Communications (RTC) capabilities via simple APIs.  More specifically, low latency and multiple video/audio/data streams are provided by WebRTC. Coupled with a web application, they enable teleoperation for mobile robots. We use EasyRTC and Node.js to create a simple signaling server and HTML/javascript teleoperation application. The robot’s embedded GUI displays the remote user image and is connected to the videoconference session using Qt/QWebEngine (Chromium based). The GUI also bridges ROS  messages through the WebRTC’s data channel and internal websockets. A Virtual camera stream corresponding to the current map (updated at 1 Hz) is created by RTAB-Map.  

Features

  • Manual Teleoperation. A virtual joystick can be used to control the robot.
  • Robot Status. Battery, speed, wireless network information is sent periodically (1 Hz).
  • Vital Sign Monitoring . BLE Medical Sensors information is displayed (SPO2, Thermometer, Glucose Sensor, Blood Pressure, Scale).
  • Semi-Autonomous Behaviors (selected by the operator by clicking buttons)
    • Audio Tracking.  The robot to rotates automatically in the direction of the person speaking.
    • Automatic Docking. When the robot detects the docking station, a trajectory to dock is automatically generated.
    • Face Tracking. The robot rotates automatically in the direction of the detected faces.
    • Automatic Navigation. Navigation goals can be set by clicking on the “interactive” map. Pre-determined locations can also be clicked.

Click on the image below to see the web-based interface.

To integrate vital sign monitoring with videoconferencing and robot telepresence control, a telecommunication framework must provide bidirectional multipoint communication of audio streams, multi video feed streaming (e.g., general view of the scene and a close-up view of a region of interest; a navigation camera and a videoconference camera on a mobile robot) and data (e.g., vital signs, commands and states of the robot) from the same telecommunication station. In spite of the high number of software videoconferencing systems available, none provides the complete spectrum from vital sign monitoring to multi video feed stream and robot control. We therefore designed a solution from the ground-up to be a flexible, cloud-based, real-time and versatile framework. It provides secure audio, video and data transfer over a high-speed Internet connection using TLS encryption. Having full control of the telecommunication framework is mandatory for robot telepresence projects since we can select the right codecs with minimum latency and can optimize data transmission according to the application’s needs and the target bandwidth. We also developed a ROS bridge to ease integration with existing software modules.