# IsaacLab Troubleshooting

## Common

- Sometimes, IsaacLab uses all the CPU cores and crash the machine. If that happens, you can limit the CPU cores in use by `taskset`. For example, if you want to use 0,1,2,3 cores, you can run:
    ```bash
    taskset -c 0,1,2,3 python ...  # your script
    ```

- If you meet `Failed to create change watch for ... No space left on device`, it doesn't really matter. You can simply ignore it, or change your system setting by running `echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf` or just simply restart the machine.

- If you meet the fail of import rclpy, try:
    ```bash
    conda install -c conda-forge libstdcxx-ng=12
    ```

### GPU memory problems with IsaacLab
If you interrupt the process, the GPU memory may not be released because **the forked process is not killed**. You need to manually kill it by running:
```bash
pkill -9 pt_main_thread
```


### No Space XXX
- If you meet `Failed to create change watch for ... No space left on device`, it doesn't really matter. You can simply ignore it, or change your system setting by running `echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf` or just simply restart the system.

### libGLU problem
- If you are running on a server in headless mode, and meet `Failed to open /.../iray/libneuray.so: libGLU.so.1: cannot open shared object file: No such file or directory`, try:
    ```bash
    sudo apt-get update
    sudo apt-get install libglu1-mesa
    ```

## Collect demonstrations
If you run collect_demo.py with IsaacLab handler:

- If you interrupt the process, the GPU memory may not be released due to the [IsaacLab's design problem](https://isaac-sim.github.io/IsaacLab/main/source/refs/troubleshooting.html#preventing-memory-leaks-in-the-simulator). You need to manually kill it by running:
    ```bash
    pkill -9 pt_main_thread
    ```

- If you find the collected demo only contains the first trajectory `demo_0000`, it is likely that the multiprocessing is not working. You can comment the "Option 2" in `collect_demo.py` and uncomment the "Option 1" and try again.

- If you are running on a server in headless mode, and meet `Failed to open /.../iray/libneuray.so: libGLU.so.1: cannot open shared object file: No such file or directory`, try:
    ```bash
    sudo apt-get update
    sudo apt-get install libglu1-mesa
    ```

- If you get the rendered image with severe aliasing artifacts, it could be very tricky bugs either in IsaacLab, IsaacSim or CUDA. We saw this bug in some machines, but couldn't reproduce it on other machines, and find no way to debug. If you unfortunatedly get into this bug, we recommand you to use other machines and try again.