Prepare Data#

If you are not a developer, you don’t have to follow this instruction. The required data will be downloaded automatically from HuggingFace when you run the code.

For the current stage, the HuggingFace dataset is private. So you need a token to access it. See tips for using HuggingFace here.

Data on HuggingFace#

See tips for using HuggingFace here.

Download the release-ready data by running:

git clone [email protected]:datasets/RoboVerseOrg/roboverse_data

Note

If you haven’t used Git LFS on your machine, run the following command to install it:

sudo apt install git-lfs
git lfs install

File structure#

roboverse_data
|-- robots
    |-- {robot}
        |-- usd
            |-- *.usd
        |-- urdf
            |-- *.urdf
        |-- mjcf
            |-- *.xml

|-- trajs
    |-- {benchmark}
        |-- {task}
            |-- v2
                |-- {robot}_v2.pkl

|-- assets
    |-- {benchmark}
        |-- {task}
            |-- COMMON
            |-- {object}
                |-- textures
                    |-- *.png
                |-- usd
                    |-- *.usd
                |-- urdf
                    |-- *.urdf
                |-- mjcf
                    |-- *.xml
|-- materials
    |-- arnold
        |-- {Category}
            |-- *.mdl
    |-- vMaterial_2
        |-- {Category}
            |-- *.mdl
    |-- ...

|-- scenes
    |-- {source} (e.g. arnold, physcene)
        |-- {scene}
            |-- textures
                |-- *.png
            |-- usd
                |-- *.usd
            |-- urdf
                |-- *.urdf
            |-- mjcf
                |-- *.xml

metasim
|-- data
    |-- quickstarts
        |-- ...
    |-- robots (deprecated)
        |-- {robot}
            |-- usd
                |-- *.usd
            |-- urdf
                |-- *.urdf
            |-- mjcf
                |-- *.xml

Explanation:

  • When assets are reused in multiple tasks, they can be stored in a COMMON folder to avoid redundancy.

Naming convention:

  • The {benchmark} is in lowercase

  • The {task} is in snake_case

  • The {robot} is in snake_case

  • The {object} is in snake_case

Data on Google Drive (Deprecated)#

  1. Please download the data from here and put it under ./data.

  2. Download the converted isaaclab data from here and put it under ./data_isaaclab.

  3. Then link ./data/source_data to ./data_isaaclab/source_data.

    cd data_isaaclab
ln -s ../data/source_data source_data

Sync data with Google Drive#

We recommend using rclone to sync the data with google drive.

Setup rclone#

Create a rclone config under the official guide, during which you need to create your own client id and token by following this.

The final rclone config should look like this:

[roboverse]
type = drive
client_id = <your_client_id>
client_secret = <your_client_secret>
scope = drive
token = <your_token>
team_drive =
root_folder_id = 1ORMP3__KIlXettN8eUCF3YQNybZQxzkw

[roboverse_isaaclab]
type = drive
client_id = <your_client_id>
client_secret = <your_client_secret>
scope = drive
token = <your_token>
team_drive =
root_folder_id = 1nF-5SU4nC6S_vgC_NL7E7Rt63Zl9sYqW

Download data#

rclone copy -P roboverse: data --exclude='demo/**'
rclone copy -P roboverse_isaaclab: data_isaaclab

You may want to speed up by setting --multi-thread-streams=16 (default is 4).

Upload data#

For example, if you are a developer who is migrating the rlbench task, you can upload the assets and source demos to google drive by running:

rclone copy -P data/assets/rlbench roboverse_isaaclab:assets/rlbench --exclude='.thumbs/**'
rclone copy -P data/source_data/rlbench roboverse:data/source_data --include='trajectory-unified.pkl'

You should replace -P with -n (means dry run) to check before uploading.