Pollen-Vision: Unified interface for Zero-Shot vision models in robotics

Hugging Face Blog · 2 years ago

Pollen Robotics released pollen-vision, an open-source library that combines zero-shot vision models to enable robots to detect and locate objects in 3D space without requiring training. The library integrates OWL-VIT for object detection, Mobile SAM for segmentation, and RAM for image tagging, with OWL-VIT processing a single object prompt in approximately 75 milliseconds on an RTX 3070 GPU. The system allows their Reachy humanoid robot to estimate object positions in 3D coordinates and grasp unknown items in unstructured environments.