Benchmark Results
The following results are trained on 50 episodes and evaluated on another 50 episodes.
In-distribution Evaluation
| Task name |
RGB |
RGBD |
PointCloud |
Fusion |
| resnet18 |
resnet18 |
ViT |
MultiViT |
DP3 |
spUnet |
ResNet+DP3 |
| CloseBoxL0 |
0.81 |
0.91 |
0.89 |
0.80 |
0.82 |
0.92 |
1.00 |
| CloseBoxL1 |
0.40 |
0.58 |
0.40 |
0.42 |
0.73 |
0.88 |
0.94 |
| CloseBoxL2 |
0.42 |
0.30 |
0.30 |
0.32 |
0.82 |
0.62 |
0.95 |
| StackCubeL0 |
0.91 |
0.87 |
0.06 |
0.06 |
0.00 |
0.00 |
0.66 |
| StackCubeL1 |
0.01 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
| StackCubeL2 |
0.01 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
| PickAlphabetSoupL0 |
0.70 |
0.42 |
0.56 |
0.59 |
0.00 |
0.00 |
0.77 |
| PickAlphabetSoupL1 |
0.00 |
0.03 |
0.10 |
0.16 |
0.00 |
0.00 |
0.67 |
| PickAlphabetSoupL2 |
0.00 |
0.26 |
0.30 |
0.27 |
0.00 |
0.00 |
0.73 |
| PickButterL0 |
0.79 |
0.73 |
0.40 |
0.47 |
0.00 |
0.00 |
0.74 |
| PickButterL1 |
0.65 |
0.60 |
0.32 |
0.27 |
0.00 |
0.00 |
0.82 |
| PickButterL2 |
0.52 |
0.52 |
0.34 |
0.35 |
0.00 |
0.00 |
0.88 |
| PickOrangeJuiceL0 |
0.30 |
0.26 |
0.31 |
0.28 |
0.29 |
0.35 |
0.50 |
| PickOrangeJuiceL1 |
0.09 |
0.25 |
0.53 |
0.50 |
0.53 |
0.49 |
0.53 |
| PickOrangeJuiceL2 |
0.13 |
0.14 |
0.28 |
0.31 |
0.05 |
0.00 |
0.51 |
| PickTomatoSauceL0 |
0.25 |
0.26 |
0.40 |
0.43 |
0.00 |
0.00 |
0.81 |
| PickTomatoSauceL1 |
0.01 |
0.08 |
0.10 |
0.12 |
0.00 |
0.00 |
0.68 |
| PickTomatoSauceL2 |
0.01 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.81 |
Out-of-distribution Evaluation(Zero-shot)
| Task name |
RGB |
RGBD |
PointCloud |
Fusion |
| resnet18 |
resnet18 |
ViT |
MultiViT |
DP3 |
spUnet |
ResNet+DP3 |
| CloseBoxL0 |
0.52 |
0.72 |
0.68 |
0.80 |
0.60 |
0.94 |
0.97 |
| CloseBoxL1 |
0.20 |
0.50 |
0.36 |
0.34 |
0.77 |
0.88 |
0.95 |
| CloseBoxL2 |
0.32 |
0.38 |
0.40 |
0.32 |
0.38 |
0.42 |
0.95 |
| StackCubeL0 |
0.29 |
0.19 |
0.00 |
0.02 |
0.00 |
0.00 |
0.10 |
| StackCubeL1 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
| StackCubeL2 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
| PickAlphabetSoupL0 |
0.66 |
0.50 |
0.58 |
0.63 |
0.00 |
0.00 |
0.89 |
| PickAlphabetSoupL1 |
0.01 |
0.00 |
0.03 |
0.04 |
0.00 |
0.00 |
0.74 |
| PickAlphabetSoupL2 |
0.00 |
0.22 |
0.30 |
0.32 |
0.00 |
0.00 |
0.75 |
| PickButterL0 |
0.80 |
0.74 |
0.78 |
0.81 |
0.00 |
0.00 |
0.87 |
| PickButterL1 |
0.68 |
0.62 |
0.70 |
0.64 |
0.00 |
0.00 |
0.91 |
| PickButterL2 |
0.67 |
0.50 |
0.56 |
0.62 |
0.00 |
0.00 |
0.86 |
| PickOrangeJuiceL0 |
0.27 |
0.20 |
0.27 |
0.30 |
0.14 |
0.20 |
0.49 |
| PickOrangeJuiceL1 |
0.10 |
0.19 |
0.42 |
0.48 |
0.00 |
0.00 |
0.58 |
| PickOrangeJuiceL2 |
0.15 |
0.15 |
0.23 |
0.20 |
0.07 |
0.03 |
0.53 |
| PickTomatoSauceL0 |
0.23 |
0.25 |
0.34 |
0.31 |
0.00 |
0.00 |
0.77 |
| PickTomatoSauceL1 |
0.01 |
0.07 |
0.06 |
0.12 |
0.00 |
0.00 |
0.65 |
| PickTomatoSauceL2 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.74 |
