Project Overview

Tangibles lack widespread adoption in standard workplaces. Rather, WIMP or  "Window, Icon, Menu, Point Device" interfaces still dominate. Why can't we have a more playful and enriched way to interact with data? Lego-Links was an attempt to offer a versatile, user-customizable tangible user interface (TUI) experience in the workplace in the context of a standard desktop setup. Based on a camera setup and computer vision techniques, Lego-based physical tokens could be built and recognized for a variety of use-cases. This included serving as shortcuts, places to store digital, or as a portal for data transfer. 

Approach

I went through many stages and iterations with prototypes.  I relied on computer vision techniques to recognize constructed tokens. Starting with a standard web camera, I experimented with MatLab to determine an algorithm to extract visual data. With some help from a colleague, I then ported the strategy over to a WPF/C# application using OpenCV for better interactivity. Challenges I faced on a recognition front were perspective and lens distortion as well as reflections that would result in loss of colour data. 

I then experimented with two orientations. I first began with a standard web camera, pointed down at a 'token' built from Lego on the desk (horizontal-setup). Later I experimented with placing tokens on the top-edge of the computer monitor (vertical setup). This transition in orientation required exchanging a standard camera with a depth-sensing one. 

With both these setups, I experimented with various use-cases. This included using the tokens as "physical shortcuts" to store data assets, launch applications with user-customized settings, change the language settings of the computer (I was in Austria, so it was handy to switch between a German and English keyboard), change one's availability status on communication platforms like Skype, and open different files on the computer. Other concepts included blocking certain website (like Social Media websites at work),  controlling a music player, and using a token as a flag to automatically log events in one's Outlook calendar. In this way, one would not have to take time in one's day to log work hours. 

Results

Discussion

There were a number of insights I gained from prototyping, particularly with regards to the setup orientation. For instance,  the horizontal setup invited hands-on manipulation and placement of tokens, but had a large physical foot-print on the table top. In contrast, the vertical setup had a smaller physical footprint and greater visibility to others in the workplace. However, the tricky positioning of vertical tokens in combination with their fragility discouraged frequent hands-on interaction.

In general, I found that such an interface can have the following benefits: 

• Duality of Purpose in the Physical and Digital World
  A 'busy' token can have the dual purpose of informing others about their desire not to be disturbed in a shared physical space, and also serves to change one's status on digital platforms
• Reduce Digital Overhead
  The calendar token can eliminate the need to specifically take time to log one's activities. Instead, the user can simply place a token in front of the camera and later remove when he or she starts and stops an activity. 
• Offline Reminders
  Tokens are not only useful to affect digital outcomes, but can also be useful in situations where the computer is off. In this case, they can serve as persistent physical reminders of certain tasks.
• Intuitive Physical Exchange
  In co-located spaces, passing over a physical token may be more intuitive than digital means of data exchange, such as setting up a common network drive or writing an email with an attachment.

Course

Completed as a Semester 2 project in Interactive Media at the University of Applied Sciences Upper Austria.

THemes

Tangible User Interfaces (TUI), Data  Tokens, Data Storage, Data Transfer

Skills & Technologies

Computer Vision, UI/UX Design, 
Depth Sensing Cameras (Kinect and RealSense, MATLAB, WPF/C#