SUNSPOT Dataset

We introduce a new dataset, SUN-Spot, for localizing objects using spatial referring expressions (REs). SUN-Spot is the only RE dataset which uses RGB-D images. It also contains a greater average number of spatial prepositions and more cluttered scenes than previous RE datasets. Using a simple baseline, we show that including a depth channel in RE models can improve performance on both generation and comprehension.

Paper

Preview of Pages 1 and 2 Download the paper here.

Citation

C. Mauceri, M. Palmer, and C. Heckman, “SUN-Spot: An RGB-D Dataset With Spatial Referring Expressions,” in International Conference on Computer Vision Workshop on Closing the Loop Between Vision and Language , 2019.

@inproceedings{Mauceri2019,
author = {Mauceri, Cecilia and Palmer, Martha and Heckman, Christoffer},
booktitle = {International Conference on Computer Vision Workshop on Closing the Loop Between Vision and Language},
title = {{SUN-Spot: An RGB-D Dataset With Spatial Referring Expressions}},
year = {2019}
}

Downloads

Examples

To provide a taste of the images and annotations in SUN-Spot, here are 10 randomly selected objects from the dataset with all of their referring expressions.

pitcher

pitcher

  • the item sitting to the left of the sink with the water in it. used to pour water into cups and store water. sitting in front of the window.
  • the pitcher is on the counter , to the left of the sink , in front of the mirror.
  • the pitcher is to the left of the sink in front of the window
alarm

alarm

  • the white alarm is on the nightstand and close to the bed.
candle

candle

  • the candlestick is on the left under the mirror.
cdplayer

cdplayer

  • cd player is next to the stacked books
bag

bag

  • the bag is on a table in front of a mirror and near two paintings
  • the bag sits at the left end of the table that is below the mirror.
  • this bag is on the table farthest to the right. it's red and white.
  • the red bag on the table
circuitbreaker

circuitbreaker

  • the circuit breaker is on a wall to the right of the red window shades.
plate

plate

  • plate on the table , under and to the right of the clock on the wall
  • the plate is on a table centered between two maroon chairs with another white plate to the right.
tissue

tissue

  • the tissue is sitting on top of the toilet.
folders

folders

  • the yellow folders are underneath the white papers on top of the counter.
  • the folders in the tray on the right side next to the other tray
lunch_box

lunch_box

  • the lunch box is front of the black box and to the right of the cylinder on the right hand corner of the desktop.