You-Eye UX

Gesture control for 3D objects on 2D screens, using the front-facing camera.

Year: 2021
Role: Interaction Designer, Prototyper
Tags: HCIGesture RecognitionUIUX design

Can the front-facing camera make 3D objects on a 2D screen feel touchable?

Every laptop and phone ships with a front-facing camera, yet 3D objects on screen are still dragged with a mouse. You-Eye UX prototypes the alternative: hand gestures read by that camera, mapped to rotate, scale, and move the object you are looking at.

Can the front-facing camera make 3D objects on a 2D screen feel touchable?

Gesture-controlled 3D object seen through the screen.

The starting question: can we capitalize on the front-facing camera and gesture recognition for more intuitive interaction with 3D objects seen through 2D screens?

Concept framing: bridging hand space and screen space.

We studied where hands beat the mouse and where they lose. Coarse spatial moves are faster in the air. Precision still belongs to the cursor.

Research into gesture vocabularies and camera-based hand tracking.

The interaction flow keeps the gesture set small: pinch, drag, spread. A small vocabulary holds up against the camera's recognition error rate.

Interaction flow mapping gestures to 3D manipulation.

Wireframes for the gesture-driven 3D viewer.

The working prototype runs SparkAR hand tracking wired into Unity. Gestures in front of the laptop move the object on screen.

Live prototype: rotating a 3D object with a hand gesture in front of the screen.

Interface concept showing gesture-controlled 3D viewing.

Interface concept: gesture states and object feedback.

Credits

Team: Joseph Wu, Aishwarya Sreenivas, Vishal Vaidhyanathan
Context: Harvard GSD ADV9672: Proseminar in Mediums, Fall 2021
Professor: Allen Sayegh
Tools: SparkAR, Unity, C#