Close

interaction

A project log for holoplayer one: an interactive midair 3D display

a mid-air light field and depth camera tracking Touch the hologram!

alexalex 11/21/2017 at 13:170 Comments

So now we've got floating 2D images.  That's some sci-fi stuff right there.  How can we touch them?

We need some way to track finger motion in midair.  We'd worked with Leap Motions on a previous product, so they were a good choice for a first experiment.  We set one up near an aerial display, and we were able to use it to track hand movement near the image.  Enough for a quick demo.

It wasn't great for an actual product, though -- Leaps need a fair amount of processing power (like a modern macbook), which ruled out small, cheap computers driving the display.  The cameras would freak out if they say an object in their field of view, or heaven forbid got a glimpse of retroreflective material.  And finally, Leaps are utter magic 90% of time time, when they work, and are super frustrating the 10% of the time that they don't work.  How would you feel if your mouse just didn't work 10% of the time?  I would feel bad, too.

We had a serendipitous stroke of luck when we found a device called an airbar (https://air.bar/)

Airbar is a thin little bar designed to sit underneath your laptop screen, plug into your laptop's usb port, and turn a non-touchscreen laptop screen into a touchscreen.  They work with a really clever technique where they have a 1D array of IR emitters and sensors and by scanning through all their emitters, they can piece together an image of what's in the plane of the sensor.  As an added bonus, they don't actually need to be up against a screen to work -- they effectively turn a 2D plane of air into a touchscreen, and they look like a mouse to a computer.  They were perfect for this!

We got super stoked about this idea, and we started rushing around making midair interface demos.  Basic stuff like translate-rotate-scale viewers for 3D models.  Midair painting, games, and more.  It was super exciting!

Except for one nagging problem:  when we interact with cell phones and tablet screens, we're using a 2D planar interface to interact with a 2D planar screen.  We were doing the same thing here, except our screen was blurrier, dimmer, bulkier and harder to interact with.  Why did it need to float?

Why did it need to float?

Discussions