A Tango data viewer application showing the image from the fisheye camera on the left and the IR range image on the right.

A Tango data viewer application showing the image from the fisheye camera on the left and the IR range image on the right.

A Tango data viewer application showing the image from the fisheye camera on the left and the IR range image on the right.

Improving our understanding of the world in which we live is a core piece of what makes us human. We seek to explore, map, and measure the places around us, and to have deep knowledge of our environment. Over time, technologies to help us with these processes have improved. Where we once had the compass and the map, we now have mobile computing devices, GPS, and large data collection efforts that put the world at our fingertips.

Today, you can determine your location to within a few meters, see image previews of your route before you ever get in your car, and view 3D reconstructions of entire cities. On top of this foundation, has emerged an entire ecosystem of applications that use this data. You can find the closest restaurant, get realtime traffic information for your commute, or view art in museums halfway across the world. In short, the way in which we interact with our world, our environment, and each other is constantly changing.

Project Tango, an effort by the Advanced Technologies and Products (ATAP) group at Google, seeks to take the next step in this mapping evolution.  Instead of depending on the infrastructure, expertise, and tools of others to provide maps of the world, Tango empowers users to build their own understanding, all with a phone. Imagine knowing your location to within centimeters. Imagine building 3D maps of the world in parallel with other users. Imagine being able to track not just the top down location of a device, but also its full 3D position and orientation. The technology is ambitious, the potential applications are powerful, and hiDOF is proud to be a core part of the effort to take the next leap in mapping and localization technology.

On-the-fly occupancy map construction.

Building an occupancy map on the fly by combining visual inertial odometry and depth data in an octree.

The desire to use a camera to map the world is nothing new. Many decades of research have been thrown at this problem and, slowly but surely, we’ve made progress. Many of the techniques used in Project Tango have roots in mapping challenges of the past, Google’s Street View cars are one example that comes to mind. The basic idea is to recognize various landmarks in the world (the corner of a certain table, a picture on the wall, etc.) across different images, and to use that information to back out the structure of an environment. This information can be further improved by using accelerometers, gyroscopes, and depth sensors to augment landmark tracking, which improves the quality of the map that you build.

When the ATAP group first approached us at hiDOF, the challenge they faced was to select and transfer appropriate technologies from this vast research space into a robust, efficient product ready to be shipped on a mobile phone. This is an incredibly daunting task. Though there has been research in the space, most Simultaneous Localization and Mapping (SLAM) software today works only on high powered computers, or even massive clusters of machines. Project Tango, in contrast, requires running a significant amount of mapping infrastructure on the phone itself to provide realtime feedback to users. This introduces two distinct challenges. You must carefully manage both your power and CPU budgets.

The sparse mapping Android application after a walk around hiDOF headquarters.  The image window shows the positions of the current features being tracked by hardware, and the plot shows a top-down view of the path.

The sparse mapping Android application after a walk around hiDOF headquarters. The image window shows the positions of the current features being tracked by hardware, and the plot shows a top-down view of the path.

Over the last year, hiDOF has applied its knowledge and expertise in the SLAM and technology transfer spaces to Project Tango. We worked closely with the ATAP team and partners around the world to make progress towards a handheld, mapping and tracking device. The ultimate goal is to generate realistic, dense maps of the world. Our area of focus, however, has been to provide reliable estimates of the pose of a phone (position and orientation), relative to its environment. Specifically, we have worked with ATAP to build a system capable of providing drift free, realtime, 6 degree of freedom, localization in typical indoor environments.

There’s still a long way to go, but we believe that this technology has the potential for big impact and creative applications. We’ve already seen Project Tango used to guide the blind, enable immersive gaming experiences tied to the real world, and provide a compact sensing package for a robot. The unique hardware of the device, including a wide angle camera, inertial measurement unit, and depth sensor, opens up many applications on its own, but the combination of this sensor suite with SLAM technology makes for truly endless possibilities.

At hiDOF, we’re proud to be a part of this effort, and to work on core components and algorithms of the Project Tango infrastructure. As an organization, we love a challenge, and look forward to continuing to test, improve, and innovate in this domain.

For more technical details about hiDOF’s work on Project Tango, check out our case study.