Something I put together while in Portland. Try it online here. 

I added links to some webpage as screensaver apps for OSX and Windows. The resolution is perfect for my MacBookPro at 1440x900, might look a little funny on a larger resolution.

Just playing around with Processing in the evenings. I made a simple sketch that allows you to upload an image and it will draw it in a sketchy way. Here is the original image and the generated image.

You can give it a try by clicking this link.

A while back I started working on a little game called Flight Island. The goal of the game is to fly through a series of glowing red waypoints to get to the finish line with the best time and the most waypoints touched. However instead of using a mouse and keyboard or even a joystick this game can be controlled by a smartphone. 

The game was created using Unity3D and communicates via OSC to the phone over WiFi. The game also has a custom made socket by my co-worker Ermal that allows Unity3D to take in gesture based input as well as live video feedback from other computer vision based input methods. 

Lately I've been playing around the Processing.js and javascript to make my site a bit more interactive for mobile users and just more fun in general. I thought it would be pretty awesome to get sensor data through a webpage and turns out it's actually pretty easy.

In the Pre-Release Processing 2.0 alpha there is a Javascript exporter in addition to the existing Android exporter. This makes it stupid simple to get your processing sketches to run on the web in Javascript instead of Java. 


One major advantage in using Processing.js is that you can use javascript to do a bunch of fancy things like grab accelerometer and GPS data. In fact there is a GPS example that comes with the new pre-release of Processing. All you have to do is switch Processing into the right mode by clicking on Standard in the top right corner and choosing Javascript, then choose File->Examples->HTML5->GeoLocation.

You can see a live demo of this sketch by clicking here. 

The one thing I didn't find in the Processing.js examples is how to work with accelerometer, gyro and other sensor data. So after a little searching online I found a pretty awesome site called www.IsThisAnEarthQuake.com

The site uses javascript to get the sensor data and draws a seismograph on screen. Rad right? Ok so that's cool and all but I want my Processing sketches to work with sensor data so now it's time to merge what we learned from the GPS demo with the IsThisAnEarthquake.com example.

In order to find the Processing sketch you need to add these lines to your HTML file. 

The acceleromter.js file looks like this. Very simple. I'm no javascript expert so there may be a better or cleaner way to do this.

The accel.pde sketch looks like this. Again pretty simple just to show how it works.

The sketch looks like this. The background color will change based on the mode. 

From here it was pretty straight-forward to convert my Interhacktive banner to take in the accelerometer data to shake the squares. Now if you are viewing my site on an an Apple laptop or any laptop with accelerometers built in...or even a mobile or tablet it will shake with the sensor data. 

I definitely had fun playing with this little used piece of new tech. Hope this helps people get started and I can't wait to see what people come up with. 

I was given a 24 hour challenge to create a fruit ninja clone. This is how far I got. Fruit Ninja Clone in 24 Hours

I made it using Unity3d, it also worked with a variety of depth cams and webcams doing hand tracking but for the sake of ease, it's shown with simple mouse input on the web. 

Yes I know there are a lot of things wrong with it, and of course there are only oranges at the moment...but the general idea is there. 

Sadly I think properly texture mapping the orange took me the most time. I suppose I'll add more fruits, more sounds and call it good. Amusing.

Augmented Reality has made some pretty advanced leaps since it's early ARToolKit days, but what really lacks in my opinions is the ability to venture away from markers and the usual trackable objects. In this SmartAR demo video the 3d character literally jumps off the table onto the floor and walks away from the original AR tracker. This is pretty unique since most AR tracking relies on having the marker in full view or at least partially viewable to the camera. 

Of course there are markerless AR solutions out there but very few seem to track as fast or as accurate as marker based methods. 

I would like to think that it's just a matter of time before we will have 3d characters not only jump off the table and away from their markers but climbing up onto the couch or interacting with the dog in real-time. 

This is a dirt simple concept, put a camera on one side of a wall and project what it sees on the other. However it makes me wonder if the world will ever be motion tracked or wired up with enough sensors and connected enough to tell when a car, bike or skateboarder is about to fly through the intersection just as you accelerate, predict the accident and simple not allow you to accelerate. I can imagine vital information such as a moving object being augmented onto the inside of a car's window so this style of transparent wall could be built into a vehicle's display system. 

I've seen a lot of projection mappings and unique interactions, but I must say this water based projection looks really hot and is the first of it's kind...I think. It's not terribly complicated or difficult to do, but it is certainly well executed. Way to go Wieden+Kennedy.

This keynote presentation was create in Unity3D so it can easily be exported to Windows, OSX, iOS, Android, Xbox 360, Wii, PS3 and the web...of course the Kinect side of things will only work on OSX and Windows. 

There are a few versions of this presentation since I realize not everyone has a Kinect.

First here is the web version.

OSX Version - DOES NOT require a Kinect

Windows Version - DOES NOT require a Kinect

OSX Version - DOES require a Kinect

Windows Version - DOES require a Kinect

You can also use TouchOSC for iPhone, iPad and Android with this template to control the slides and mouse cursor on screen. 

Kinect drivers can be downloaded from ZigFu.com for the Mac and PC. Instructions on how to get it to work on a mac are in their blog. 

If you have any questions please feel free to email me any time. 

Back in the day depth cameras were really expensive so many researchers and hackers simply focused on what they could accomplish using a standard color webcam. With the Kinect being all the rage these days many people who were using basic webcams for computer vision to generate a 3d models in the past are now finding that their old projects are getting new attention and may finally make it out of the research lab and into the product market.

So what were people trying before the Kinect? Many approaches tried to match up groups of pixels in previous frames to the current frame to track where the camera had moved in relation to the scene as well as checking for occlusions, how fast the groups of pixels moved and other really clever tricks. There are many different algorithms that have a different takes on this same concept...this is also similar to how Augmented Reality trackers work. Other people projected a pattern of visible light on an object and would use a webcam to measure how much the pattern changed when shown on an object, this is called the Structured Light approach.

If you are unaware, the Kinect uses a Structured Light approach, however it does so with an infrared laser to create the pattern and an infrared webcam to detect it. In addition it has a standard color webcam which can be aligned to the depth image detected by the infrared camera.

Of course there are a number of non-real-time 3d model generating applications and projects such as PhotoSynth by Microsoft and PhotoFly by Autodesk. There is also this crazy concept of crowd sourced or cloud based 3d model generation using tons of images from say flickr or any large database of images...also not a real-time thing, but worth noting.

Here are a few different methods people have tried for reconstructing a 3d object using computer vision.

Some people use markers like a checkerboard pattern to give the camera a good clean known reference to track. I kinda like this approach because of it's simplicity and how clean of a model you can get out of it.

Pro-FORMA is an older project that I had long forgotten about until my coworker Michael Chu send me a link to this video this week. This is a bit more impressive since it is kinda roughly being moved around a table with little regard to how precisely it is being rotated. Also it's ignoring the background and hand! 

This one is from Kyle Mcdonald, a heavy contributor to the OpenFrameworks community. I really like this Structured Light project because even though it doesn't create the most clean 3d models, it was hacked together using OpenFrameworks plus full instructions and source code are available online...and it came out long before the Kinect.

Now we get into the fancy stuff...this is the real deal and probably one of the best 3d reconstructions from a single color webcam with no assistance from structured light. There is a technical paper describing this approach in detail if you are interested in learning more.

Nicholas Burrus created RGBDemo and is now with his 6th version it can not only scan objects, but the entire environment as well as segment out some of those objects. Source code and binaries are available on his site. This is a spinoff of PCL (Point Cloud Library).

Another take on Nicholas Burrus' RGBDemo using an electronically controlled lazy susan to rotate box with an object in it has made it's way around the web with a little help from Hack-A-Day. This helps the stitching of RGBDemo because of the precise consistent control over what it sees. The little green pieces of tape appear to be there just to give the background a little texture so the algorithm has a good spot to match up against the previous frame.

The RGBDivision which is part of the University of Washington and previously an Intel Research Lab created this application to help people generate 3d models and environments. The application gives an indicator of how well the current frame matches up to the previous frame as well as a previous of the last good frame if you stray away.

The big Kinect related news out of Siggraph 2011 was Kinect Fusion. Not only does it reconstruct a single object, it stitches the entire environment and appears to be able to separate object and allow for multi-touch style interaction on any surface curved or not. They also added a pretty impressive real-time physics simulation or metablob like particles splashing onto a shirt and being shaken off in real-time...in my opinion they are just showing off at this point. Light years ahead of the rest of the stuff out there.

Alright so what's actually made it to market so far? Well there are a few companies scanning faces out there but for the mass market, general consumers the only thing I've seen so far is from the Kinect Fun Labs for Xbox 360.

Kinect Fun Labs allows you to generate a 3d model by taking an RGB+D image of the front and an RGB+D image of the back and stitching the two together into a 3d model. Not exactly perfect but it seems to work for simple objects. 


I have a feeling that real-time model generation will definitely explode over the next few years. 

This goal of the game is simple to throw pies at Rupert Murdoch, Fox News and Bill O'Reilly. Tracking is done by the Kinect using a OpenNI wrapper for Unity3d. The velocity of the throw determines how fast and at what angle the pie is launched.


To start the game simply pose like the character until it locks onto you and you'll see yourself controlling the character. If you need more pies there is a whole table of them right behind you, simply take a few steps back, reach down to the table full of pies with your right hand and it will pick one up. Then step up to the line and THROW THAT PIE! You can toss it overhand, softball style, underhand or even like a frisbee. 

Unfortunately you'll need a windows 7 system to run this app...and you have to install the OpenNI drivers as well. 
Here are links to those drivers. First install this one, then this one, and finally this one..here is the key you need to get the last one to work. 0KOIk2JeIBYClPWVnMoRKn5cdY4=


Below is the zip file containing the executable. Have fun! If anyone is interested in source, just let me know. 

Every so often this blog will be updated with new projects, news, tutorials and source code. For older posts check out www.ProjectAllusion.com