Software
This page provides instructions to prepare a new mini PC for running the real robot with our codebase.
Note
These setup steps are specific to configuring the mini PC for working with the real robot. If you only need to run in simulation on a dev machine, please see the Usage section of the codebase README. The full policy learning pipeline can be tested in simulation without a physical robot, including phone teleoperation, data collection, and policy inference.
Mini PC setup
This section describes how we configured a Beelink S12 Pro mini PC as an onboard computer for running controllers. These steps are tailored for this model but should apply to other mini PCs as well.
Ubuntu 22
To create a bootable USB drive for Ubuntu 22, follow these steps:
- Go the release page for Ubuntu 22
- Scroll down and download this file:
ubuntu-22.04.4-desktop-amd64.iso
- Download and install balenaEtcher
- Use balenaEtcher to flash the ISO image onto a USB drive
Note
We used Ubuntu 22.04.4 LTS. Newer versions should work as well but are untested and could require additional modifications.
Tip
For faster installation, use a USB drive that supports USB 3.0 or higher. The Ubuntu installation will take much longer with a USB 2.0 drive.
Install Ubuntu 22 on the mini PC:
- Power on the mini PC and press
<F7>
to enter the boot menu - Select the
UEFI USB
option to boot from the USB drive - Select Install Ubuntu when prompted
- Connect to a Wi-Fi network when prompted
- In the Updates and other Software screen, choose Minimal installation instead of Normal installation. Do not check the box for Install third-party software.
- In the Installation type screen, select Erase disk and install Ubuntu
- Continue with the installation procedure until completion
Tip
If the boot menu does not appear, restart the mini PC and repeatedly press <F7>
during the boot process.
Note
We use the "minimal installation" to minimize unnecessary software, as this mini PC will be used to run performance-critical real-time controllers.
Automatic power on
For convenience, we recommend configuring the mini PC to automatically power on when plugged in. Follow these steps to enable this in the BIOS:
- Reboot the mini PC and press
<Del>
to enter the BIOS setup - Navigate to
Chipset > PCH-IO Configuration > State After G3
- Change
S5 State
toS0 State
- Save changes and exit
- Turn off the mini PC and unplug the power adapter
- Plug the power adapter back in and verify that the mini PC powers on automatically
Headless setup
The mini PC will mainly be used in headless mode without a monitor via SSH and tmux, as it is inconvenient to keep a monitor connected to a moving mobile base.
To set up SSH and tmux:
sudo apt install openssh-server tmux
SSH allows you to connect to the mini PC remotely from your dev machine:
ssh <username>@<minipc-hostname>
Replace <username>
and <minipc-hostname>
as appropriate.
Tip
If you are on a network with no DNS server, try appending .local
to the hostname:
ssh <username>@<minipc-hostname>.local
We use tmux to create persistent sessions that remain intact even if the SSH connection drops. Here is a typical tmux workflow:
- SSH into the mini PC
- Run the command
tmux
to start a new tmux session - If a session already exists, run
tmux a
to reattach to it - Run tasks in the tmux session
- Create new windows to run tasks in parallel
Tip
If you have not used tmux before, please see the "Sessions" and "Windows" sections in the tmux cheat sheet.
Tip
By default, mouse scrolling is disabled in tmux. To enable it:
- Open
~/.tmux.conf
in a text editor - Add the line:
set -g mouse on
- Save and exit
- Reload the config file:
tmux source ~/.tmux.conf
CPU frequency scaling
To minimize latency when running real-time controllers, CPU frequency scaling should be disabled.
First, install cpufrequtils
:
sudo apt install cpufrequtils
Check the current CPU frequency policy:
cpufreq-info -p
Example output showing the powersave
governor is active:
400000 4000000 powersave
700000 3400000 powersave
To set the CPU to use the performance
governor (disable scaling), run:
sudo sh -c 'echo "GOVERNOR=performance" > /etc/default/cpufrequtils'
With this change, the default governor to will be set to performance
upon reboot.
Wi-Fi power management
For improved performance and latency when communicating with the mini PC over Wi-Fi, wireless interface power management should be disabled.
Check the current wireless settings:
iwconfig
If you see Power Management: on
, follow these steps to disable power management:
- Open
/etc/NetworkManager/conf.d/default-wifi-powersave-on.conf
in a text editor - Change
wifi.powersave
towifi.powersave = 2
Real-time kernel
To improve the real-time performance of controllers, we highly recommend installing the the PREEMPT_RT
real-time kernel.
Note
The real-time kernel is not compatible with Nvidia drivers.
Building the real-time kernel
Please follow the real-time kernel setup guide from the Franka documentation. Here’s a quick summary of the steps we used:
Note
If you prefer to directly install the same kernel image we built rather than building it yourself, you can download the files here:
To install these files, run:
sudo dpkg -i linux-image-6.8.2-rt11_6.8.2-1_amd64.deb linux-headers-6.8.2-rt11_6.8.2-1_amd64.deb
If this installs successfully, you can skip ahead to the section below on setting the real-time kernel as the default GRUB boot option.
Install the required dependencies:
sudo apt-get install build-essential bc curl ca-certificates gnupg2 libssl-dev lsb-release libelf-dev bison flex dwarves zstd libncurses-dev debhelper
Download the necessary source files:
mkdir linux-6.8.2-rt11
cd linux-6.8.2-rt11
curl -SLO https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.8.2.tar.xz
curl -SLO https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.8.2.tar.sign
curl -SLO https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.8/patch-6.8.2-rt11.patch.xz
curl -SLO https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.5/patch-6.8.2-rt11.patch.sign
Note
We recommend installing the exact PREEMPT_RT
version we used.
Other versions may have issues such as broken Wi-Fi or Ethernet connectivity.
Extract and patch the kernel:
xz -d *.xz
tar xf linux-*.tar
cd linux-*/
patch -p1 < ../patch-*.patch
cp -v /boot/config-$(uname -r) .config
make olddefconfig
make menuconfig
In the menu interface that opens, apply the following changes and save before exiting:
- Under General setup > Preemption Model, select Fully Preemptible Kernel (Real-Time)
- Under Cryptographic API > Certificates for signature checking > Additional X.509 keys for default system keyring, remove
"debian/canonical-certs.pem"
- Under Cryptographic API > Certificates for signature checking > X.509 certificates to be preloaded into the system blacklist keyring, remove
"debian/canonical-revoked-certs.pem"
- Under Device Drivers > Graphics support > Intel 8xx/9xx/G3x/G4x/HD Graphics, press
<N>
to exclude (this option addresses i915 latency spikes incyclictest
)
To build the kernel, run:
make -j$(nproc) bindeb-pkg
Note
Building the kernel in parallel may hide error messages. For clearer output, you can build sequentially:
make bindeb-pkg
However, note that the sequential build can take up to 4 hours.
Once the kernel has been built, install it with:
sudo dpkg -i linux-image-6.8.2-rt11_6.8.2-1_amd64.deb linux-headers-6.8.2-rt11_6.8.2-1_amd64.deb
Note
After installing this kernel, if you plug in a monitor after the mini PC has booted, it will not display anything. To use the monitor, plug it in before booting the mini PC. You can restore normal monitor behavior by building a kernel that skips the Intel graphics option above (but it will impact latency performance).
Setting real-time kernel as default boot option
To set the new real-time kernel as the default boot option, follow these steps:
- Open the
/etc/default/grub
file in a text editor - Change
GRUB_DEFAULT=0
toGRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.8.2-rt11"
- Run this command to update GRUB:
sudo update-grub
Note
If you are using a different kernel version, modify the text to match your kernel version.
You can check /boot/grub/grub.cfg
for the correct identifier of the real-time kernel menu entry.
Note
There are no spaces around the >
symbol in the GRUB_DEFAULT
entry.
Verification
Reboot the mini PC and perform the following verifications:
CPU frequency scaling
To verify that CPU frequency scaling is set to performance
, run this command:
cpufreq-info -p
You should see output like this, indicating that frequency scaling is disabled:
700000 3400000 performance
Wi-Fi power management
To check if Wi-Fi power management is disabled, run:
iwconfig
Verify that the output reads:
Power Management:off
You can also ping the mini PC from your dev machine and verify that the ping times are around 1-3 ms. With power management enabled, ping times are typically much higher (40-100 ms).
Real-time kernel
To confirm that the real-time kernel is in use, run:
uname -a
The output should include the PREEMPT_RT
label.
For example:
Linux iprl-minipc-01 6.8.2-rt11 #1 SMP PREEMPT_RT Sun Oct 27 18:09:44 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux
If the kernel name ends in -generic
, it means the system is still using the generic kernel, not the real-time one:
Linux iprl-minipc-01 6.8.0-48-generic #48~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 7 11:24:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Latency testing
Once the real-time kernel is confirmed to be in use, run a latency test with cyclictest
to measure system performance for 1 hour.
First, install the rt-tests
package:
sudo apt install rt-tests
Next, run the following command to measure the real-time latency performance of your system:
sudo cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0 --duration=1h
This test will capture latency statistics across all CPU cores for 1 hour. In our tests using the Beelink S12 Pro, the generic kernel showed significantly higher latencies:
6.8.2-rt11
- < 20 us (microseconds)6.8.0-48-generic
- 1000+ us
Note
We recommend unplugging unnecessary peripherals such as monitors during the test as they can cause latency spikes. We conducted our tests over an SSH connection inside a tmux session.
Arm setup
Follow the steps below to configure the mini PC for controlling your robotic arm.
Ethernet connection
To establish an Ethernet connection to the Kinova arm, run this command:
sudo nmcli con add type ethernet con-name Kinova ifname <ethernet-interface> ipv4.method manual ipv4.addresses 192.168.1.11/24
Replace <ethernet-interface>
with your actual Ethernet interface name, which you can find by running ip link
.
Then, verify that the Kinova
connection is listed when running:
nmcli con
Activate the connection:
sudo nmcli con up Kinova
Make sure the arm is powered on and test the connection by pinging the arm:
ping 192.168.1.10
Communication test
To evaluate network communication with the arm, run the following ping test for 100 seconds:
sudo ping 192.168.1.10 -i 0.001 -D -c 100000 -s 1200
Here is some example output:
100000 packets transmitted, 100000 received, 0% packet loss, time 99999ms
rtt min/avg/max/mdev = 0.143/0.252/0.495/0.059 ms
The result should show 0% packet loss and round-trip times (RTT) well under 1 ms.
CAN setup
Follow the setup instructions in Yihuai Gao's ARX5 SDK to set up the USB-to-CAN adapter for your ARX5 arm.
The only modification required is to replace all instances of can0
with can1
, as the can0
interface is already used by the mobile base.
Codebase setup
This section describes the steps required to set up the codebase on the mini PC for use with the real robot.
Real-time priority
The low-level controllers should run with real-time process priority to ensure optimal performance and minimal latency.
Create a file at /etc/security/limits.d/99-realtime.conf
with the following contents:
@realtime soft rtprio 99
@realtime hard rtprio 99
This configuration allows members of the realtime
group to set real-time priority for processes.
Next, create the realtime
group (if it doesn’t already exist) and add the current user to it:
sudo addgroup realtime
sudo usermod -aG realtime $USER
Note
These new permissions will take effect only after logging out and logging back in.
CANivore
To communicate with the drive system, we need to enable SocketCAN support for the CANivore USB-to-CAN adapter.
First, add the CTR Electronics package repository:
export YEAR=2024
sudo curl -s --compressed -o /usr/share/keyrings/ctr-pubkey.gpg "https://deb.ctr-electronics.com/ctr-pubkey.gpg"
sudo curl -s --compressed -o /etc/apt/sources.list.d/ctr${YEAR}.list "https://deb.ctr-electronics.com/ctr${YEAR}.list"
Then, install the canivore-usb
package:
sudo apt update
sudo apt install canivore-usb
Note
More details are available in the official CANivore documentation.
Gamepad
To enable gamepad input in headless sessions, add the current user to the input
group:
sudo usermod -aG input $USER
This grants permission to read gamepad input events without requiring root access.
Note
The new permission will take effect only after logging out and logging back in.
To confirm that the gamepad is working, install and run jstest
:
sudo apt install joystick
jstest /dev/input/js0
Here is some example jstest
output:
Driver version is 2.1.0.
Joystick (Logitech Gamepad F710) has 8 axes (X, Y, Z, Rx, Ry, Rz, Hat0X, Hat0Y)
and 11 buttons (BtnA, BtnB, BtnX, BtnY, BtnTL, BtnTR, BtnSelect, BtnStart, BtnMode, BtnThumbL, BtnThumbR).
Testing ... (interrupt to exit)
Axes: 0: 0 1: -2 2:-32767 3: 0 4: -2 5:-32767 6: 0 7: 0 Buttons: 0:off 1:off 2:off 3:off 4:off 5:off 6:off 7:off 8:off 9:off 10:off
Tip
If you have a graphical display connected, you can use jstest-gtk
instead of jstest
.
Logitech camera
If you will be using a Logitech camera (as a base or wrist camera), follow these steps to enable camera access in headless sessions.
Create a file at /etc/udev/rules.d/99-webcam.rules
with the following contents:
# Logitech C930e
SUBSYSTEMS=="usb", ATTRS{idVendor}=="046d", ATTRS{idProduct}=="0843", MODE="666"
Then, run the following commands to apply the new udev rule:
sudo udevadm control --reload-rules
sudo service udev restart
sudo udevadm trigger
Mamba environment
Please install Mamba following the official instructions. We use the Miniforge distribution.
Run the following commands to set up the Mamba environment:
git clone https://github.com/jimmyyhwu/tidybot2.git
cd ~/tidybot2
mamba create -n tidybot2 python=3.10.14
mamba activate tidybot2
pip install -r requirements.txt
Constants
There are a few constants specific to your hardware that need to be set in the codebase within the constants.py
file.
Encoder offsets
Each mobile base has a unique set of encoder offsets, which are used by the low-level controller to determine the absolute steer position of each caster.
Before setting the offsets, follow these steps to prepare the base:
- Double-check that all components inside the mobile base, including the camping battery and SLA battery, are fully secured
- Ensure that all cables are properly secured and kept away from the caster module gears
- Turn the mobile base onto its side, with the right side of the base facing up
The zero steer position is the configuration that the base assumes when driving forward, with all casters in the trailing position:
Note
In these photos, the robot's forward (+x) direction is towards the right side.
Use a straightedge to align the top wheels so that they are both in the zero position:
Repeat this process for the bottom two wheels as well.
Once all four casters are in the zero steer position, run the following Python code snippet to print out the ENCODER_MAGNET_OFFSETS
value you should use:
from base_controller import Vehicle
vehicle = Vehicle()
vehicle.get_encoder_offsets()
Update the ENCODER_MAGNET_OFFSETS
value in constants.py
with the printed value.
Tip
If you find later on that your base odometry is not accurate, you can improve performance by fine-tuning the encoder offsets in increments of 1/4096. Inaccurate odometry can cause the robot, when commanded to go straight, to take a curved path or rotate slightly.
Base dimensions
The constants h_x
and h_y
in constants.py
specify the distance from the mobile base center to each caster steer axis in the x
and y
directions.
If you are using one of our reference designs, you can just uncomment the appropriate line in constants.py
.
For the larger Kinova or Franka designs:
h_x, h_y = 0.190150 * np.array([1.0, 1.0, -1.0, -1.0]), 0.170150 * np.array([-1.0, 1.0, 1.0, -1.0]) # Kinova / Franka
For the smaller ARX5 design:
h_x, h_y = 0.140150 * np.array([1.0, 1.0, -1.0, -1.0]), 0.120150 * np.array([-1.0, 1.0, 1.0, -1.0]) # ARX5
If you are using a custom frame size, please modify these values to match your robot's measurements.
Logitech camera
If you are using a Logitech camera, you need to specify its serial number in constants.py
.
Retrieve the camera's serial number using either of the following commands:
lsusb -v -d 046d:0843 | grep iSerial
udevadm info -n video0 | grep ID_SERIAL_SHORT
Modify the BASE_CAMERA_SERIAL
value in constants.py
to match your camera's serial number.
Arm setup
If you are using the Kinova Gen3 arm with the codebase, follow the additional setup steps below:
Kortex API
To communicate with the arm from Python, please install the Kortex Python API, which can be downloaded from:
Run the following commands to install:
pip install kortex_api-2.6.0.post3-py3-none-any.whl
pip install protobuf==3.20.0 # kortex_api specifies protobuf==3.5.1 which is outdated
Wrist camera
If you want to read images from the Kinova wrist camera in Python, you need to install OpenCV with GStreamer support.
First, install GStreamer:
sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
If you are using the same mini PC as us, you can download our pre-built OpenCV wheel:
Otherwise, please expand the section below to build the wheel yourself:
Building OpenCV with GStreamer support
Install OpenCV build dependencies:
sudo apt install python3-dev python3-numpy libavcodec-dev libavformat-dev libswscale-dev libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libgtk-3-dev
Build OpenCV from source with GStreamer support:
mamba create -n opencv python=3.10.14
pip install scikit-build==0.17.6 numpy==1.26.4
git clone --recursive https://github.com/opencv/opencv-python.git
git checkout tags/80
export CMAKE_ARGS="-DWITH_GSTREAMER=ON"
pip wheel . --verbose
After building, you should end up with a Python wheel file called opencv_python-4.9.0.80-cp310-cp310-linux_x86_64.whl
.
Next, install the OpenCV Wheel:
pip install --force-reinstall --no-deps opencv_python-4.9.0.80-cp310-cp310-linux_x86_64.whl
To verify that the installed OpenCV has GStreamer support, run the following command:
python -c "import cv2 as cv; print(cv.getBuildInformation())" | grep GStreamer
You should see the following line in the output:
GStreamer: YES (1.20.3)
Torque offsets
Before running torque control on the Kinova arm, it is a good idea to zero the torque offsets. First, make sure the surrounding area is clear, as the arm may sweep across a large space while moving to the zero configuration. Next, run the following Python code snippet and follow the terminal prompts:
from kinova import TorqueControlledArm
arm = TorqueControlledArm()
arm.zero_torque_offsets()
arm.disconnect()
ARX5 controller
To control the ARX5 arm, we use Yihuai Gao's ARX5 SDK.
Start by cloning the arx5
branch of our codebase, which adds support for the ARX5 arm:
git clone https://github.com/jimmyyhwu/tidybot2.git
cd ~/tidybot2
git fetch origin arx5 # Branch for ARX5 arm
git checkout arx5
Next, install the arx5-sdk
package within the tidybot2
directory:
cd ~/tidybot2
git clone https://github.com/real-stanford/arx5-sdk.git
cd arx5-sdk
git checkout d046707 # Last tested commit
mamba env create -f conda_environments/py310_environment.yaml
pip install mujoco==3.2.4 scipy==1.12.0
mamba activate arx-py310
mkdir build && cd build
cmake .. && make -j
These modifications enable the ARX5 arm to be used following the same steps outlined in the Usage page.
Note
Be sure to activate the arx-py310
Mamba environment in the terminal session where you run arm_server.py
.