Computational Cameras @ SIGGRAPH

Refocussing a single light-field photograph

Last time I blogged about the interesting talk given by Will Wright at this year’s SIGGRAPH conference. Will’s talk was a spell-binding overview of current visualization and entertainment trends, and the role that human psychology and aesthetics plays within it. However, SIGGRAPH also has a more facts-and-numbers side – there actually is an academic conference hiding between all the glitz and glamour. (The one with technical papers.)

Now when I say “conference” I immediately conjure up images of badly dressed nerds in exotic locations, getting all excited about mind-numbingly boring topics, and then getting drunk and trying to hit on the only attractive girl among the several hundred conference-goers. But in SIGGRAPH’s case, at least, the topics are not all that mind-numbingly boring. Because in addition to the academic quality enforced by the reviewers, there is also a strong emphasis on originality, and a “what can this be used for, and does it look cool?” criteria.

There were several very interesting sessions, some of which focused on

• Realistic simulation of fluid, fire, and computer animation physics
• Human-computer interaction (Holographic teleconferencing with eye tracking, 3D with focused-ultrasound force-feedback)
• Smart image editing (Resizing images without visibly cropping or distorting the content, and automatic black-and-white to colour conversion)
• Making deformable 3D computer meshes, and then doing crazy things to them – like squashing a bunny and ripping a cow in two
• 3D visualization of scientific data – complete with 3D glasses! (watching the Sun’s erupting surface, or travelling through an Egyptian mummy)

For this blog post, however, I want to talk about the “computational cameras” session. These talks might give you a glimpse into the future of photography…

Invertible motion blur

Deblurring images of a moving vehicle

I always thought it would be quite possible to deblur an image if you knew how your subject moved. For example of you photograped a car driving past while using a slow shutter speed. Turns out it’s not that easy, since you actually lose data you cannot recover. Unless you’re clever, and take more than one photograph with different amounts of blur, and combine them.

Dark flash photography

UV + IR Flash. "Smile"!

We all know (right?) that using a flash spoils the mood and the lighting, and temporarily blinds the poor people looking at the camera. But at poorly lit venues using natural light only gives horribly noisy images. Solution? Take one photograph with a barely visible ultraviolet+infra-red flash, and paint in the colours using a noisy natural-light photograph. The result looks almost natural!

4D lens for universal depth-of-field

A new type of optimized plenoptic camera lens

These researchers used clever frequency-domain mathematics to physically design and build a new lens (by cutting up an expensive Canon L lens and shuffling around the pieces). This Frankenstein’s creation makes a blurry photo (duh), but with some clever processing you can get an in-focus image, at any (or all) focal depths of your choice! The guys (and girl) also showed that “traditional” designs which did the same thing were suboptimal, since you only need a 3D subspace of the 4D image “light field” data. This lens focuses all its energy into this 3D sub-domain, thereby getting better performance than what was previously done. Does this sound like abstract gobbledygook? It actually works!

Bokodes

Bokode: Visual Tags for Camera-based Interaction from a Distance

Getting its name from “Bokeh” + “code”, this is a really brilliant concept, which made me think: “now why couldn’t I think of something as elegant as this!?” Well done, MIT!

The idea is to place a tiny special reflector / LED light in an object. When you now take an out of focus photograph of it, you’ll get a sharp (in-focus) image of a barcode or data matrix superimposed on blurry background scene! The tiny little object will appear large in your view-finder, even over moderate distances. Say hello to ultra-small bar codes!

The authors then went on to show some really cool applications for this technology – e.g. you can use bokodes to accurately track an object in 3D using a single camera, you could use them to replace RFID tags (simpler and cheaper), use bokodes in an interactive classroom to identify audience members, or to get electronic business cards as you drive by in your car.

In conclusion

Innovation in photography – even fundamental innovation – is far from dead. I can’t wait to see what they’ll dream up for next year!

Will Wright @ SIGGRAPH

Will Wright

I am writing this post from the floor of SIGGRAPH 2009, this year being held in New Orleans, USA.

Yesterday I had the privilege of attending a keynote talk by Will Wright, the designer behind several landmark games such as SimCity, The Sims, and Spore. He presented an entertaining and panoramic talk entitled “Playing with Perception” which covered topics involving human psychology, visual processing, game design and the interrelatedness of multimedia technology. During his hour behind the microphone he swept through a staggering 270 slides, thereby wildly breaking the golden rule of  “one slide per minute”, but coming from him, it worked. Respect.

I am guessing here, but from his talk I deduced that he is also a hobby photographer, and he continued to dedicate several of his many slides to this topic. Here are a few of the points which caught my attention:

• New technology fails to have its intented impact where there’s a misunderstanding, or ignorance of, human psychology.
• Several decades ago everyone expected 3D imaging and displays to be widespread by the turn of the century. However, 2D photographic effects (like shallow depth-of-field) are used to great effect in movies and photography, where it focusses our attention where the director intended. This lack in 3D makes the experience poorer rather than richer. (For a live concert broadcast experience, however, 3D works well, since here we demand to focus our attention freely in an immersive “I’m there” experience.)
• Visual art (painting) gradually evolved from crude rock painting to photorealism, before in recent times further developing into subjective and emotion-driven abstraction. The same may hold for animation and photography, at least in some contexts.
• Case in point tilt-shift photography is taking us further away from reality, and yet has immense emotional appeal. A possible reason is that the toy-like appearance of the photographed reality gives us the feeling of being able to manipulate and play with the scene (à la SimCity) – this pleases our senses.
• Being behind a camera lens gives you the ability to see the world in new ways, and trains your perception. After a while you see creative angles in everyday objects, even when your camera is not with you.
• With the addition of digital and CG tools, film makers have the ability to create (almost) any conceivable image on screen – the question is therefore no longer “what is possible to create?”, but purely “what should we create?”

Indeed, what should we create? It’s often amazing to see how extremely simple technologies can be enormously immersive. A clear favourite of mine is this music video, which doesn’t make use of any computer animation. Digital effects can greatly enrich our experience, but the art lies in knowing when, and how much, it is needed.

Looking at his professional success it is clear that Will deeply understands the aesthetics of interactive technology. And this is the exactly what I admire in SIGGRAPH  - it brings technical innovation and aesthetics together in a single venue – something rare indeed.

Here’s to innovation, beauty and perception!

PS: Since writing the above blog, I have seen the 3D CG session featuring mind expanding scientific visualisation from NASA Goddard’s space flight centre. Stereoscopic 3D showing the sun’s coronal mass ejections in the UV spectrum (photographed in stereo by the twin SOHO spacecraft), a visualization of deep-ocean currents, seasonal ice cap variations, and much more. This was followed by a biomedical visualization tour of CTs obtained from 3000 year old Egyptian mummies. And then there were the animated shorts of Pixar, followed by an excerpt from U2 – live in 3D.

3D is indeed amazing when used correctly – it is just taking long to mature. When it comes into its own it will blow your mind – wow!