Archive for the ‘Graphics’ Category

Chroma key with Firtree

Sunday, June 29th, 2008

The best way to test your dog food is to eat it yourself and so this afternoon I set out to write a little PyGTK app that would use the Firtree Python bindings to allow you to pull an alpha matte via chroma key. Chroma key, for the less SFX minded amongst you, is the process whereby the green backgrounds so beloved by special effects people everywhere are ‘cut out’ and replaced by… well, whatever you want your actors to be standing in front of.

Pulling a matte, as such a process is known, is a little tricksy and requires not a little tweaking. I had in mind an app which would let you load a high-res photo, pan and zoom around it lightning fast and let you see the results of tweaking all the matting parameters in real-time. Just the sort of thing Firtree is supposed to be good at. An afternoon’s hacking later and we have this little gem:

It is a recording of the chroma key app that, as of 10 minutes ago, resides in Firtree’s examples directory. You load your image, select something that is background coloured and fiddle with some sliders. Firtree’s pixel pushing powers allow it to reflect the result in real-time. I love it when software starts to become useful and not just Another Framework TM.

This is actually the second little app I’ve written which has used Firtree. I find writing little apps like this very good for shaking out bugs and exposing use cases I hadn’t though of. The first ‘non-trivial’ Firtree app was one I wrote at work. The app displays streams from a pair of computer vision cameras and allows you to monitor the stream, record it and overlay detected features in real time. It is suprisingly useful to be able to get on-demand feedack as to whether your funcky computer vision algorithm is working :).

Update: Should anyone care, you can also browse the source to the app via launchpad.

Firtree redux

Wednesday, June 11th, 2008

This evening, firtree managed to render this:

blobby checkerboards!

Why is this impressive? Well, it means that lots of things are working and it is slowly reaching the point where it might actually be useful. Read on for an overview of the system.

(more…)

Firtree: a teaser

Tuesday, June 10th, 2008

Further to my previous post: this evening I typed the following into my computer:

kernel vec4 testKernel(void)
{
    float squareSize = 10.0;
    vec2 dc = mod(destCoord(), squareSize*2.0);
    vec2 discriminant = step(squareSize, dc);
    float flag = discriminant.x + discriminant.y - 
                 2.0*discriminant.x*discriminant.y;
    return vec4((0.25 + 0.5*flag) * vec3(1,1,1), 1);
}

My personal project, firtree, compiled and JIT-ed it into a GPU shader. I was then presented with this:

A checkerboard

Then I typed in:

kernel vec4 testKernel(void)
{
    float dotPitch = 60.0;
    vec2 dc = mod(destCoord(), dotPitch) - 0.5*dotPitch;
    float discriminant = 1.0 - smoothstep(0.3*dotPitch-0.5,
           0.3*dotPitch+0.5, length(dc));
    discriminant = sqrt(discriminant);
    return vec4(discriminant,discriminant,0,1);
}

I got this:

A spotty pattern

You’re not excited are you? Well, anyone familiar with GPU image processing frameworks might be.

In all seriousness, firtree isn’t anywhere near ready to play with the big boys. Come to that, it barely crawls out of its pram. For a start, kernels can’t be chained together… yet. The infrastructure is there though…

More Blending

Tuesday, April 29th, 2008

Another motion capture movie for you. This time it is for a presentation so is a little less interesting but it is longer :).

More 3d rubbish

Tuesday, April 22nd, 2008

The resulting video from last night’s blending shows a strange cube-like UFO flying over a panoramic photo of Lecce in Italy. New things wot I learned in this video: motion blur via post-processing composite nodes and glossy reflections.

Camera mapping

Monday, April 21st, 2008

Last night I stumbled upon a cool technique called ‘camera mapping’ which lets you ’step inside’ a photograph without too much effort. I couldn’t resist having a go so after a brief search on flickr, a static photo is transformed into a video. Not very convincing perhaps but it was no more than an hour’s work beginning to end :)

Update: I made a slightly better version which uses a better model for the midground mountain, is in HD and adds a little specular to the water to make it look a little more ‘alive’.

Real time radiosity

Friday, March 16th, 2007

After lots of sweat and tears at work, the radiosity demo can now be made public. This is a video recorded of our real time radiosity engine working away…

Digg it and make me feel proud!

DirectX vs OpenGL

Sunday, October 29th, 2006

I’ve just been flipping through the comments on a story on OSNews about DirectX 10. Lots of the comments were postulating conspiracy theories about why games programmers use DirectX and not OpenGL.

I’ve used both APIs and let me tell you OpenGL just doesn’t compare to DirectX. The actually 3D API of both is pretty much on a par modulo object vs state-based models. Where DirectX wins is a) there is more than just 3D in the API even if Direct{Input,Sound,Show} and friends are pretty awful and b) far more importantly there is a bushel of useful utility functions.

To take a trivial example, DirectX has utilities to save any texture or rendering surface in its own DDS file format. DDS can save in formats that no other standard format can, e.g. S3-style texture compression, multiple MIP levels, floating-point buffer formats, etc.

It wouldn’t be rocket science to make a little library to save/load OpenGL textures from DDS files but there just isn’t an easily available one. Direct3D has loads of useful utilities like this which makes the life of someone on a short turnaround timescale just that bit more convenient.

The final thing about Direct3D which makes it useful is the flip side of one explicit design goal of OpenGL. Direct3D cases and, to an extent, leads the hardware making the successive incompatable versions of the API follow a similar model to the hardware of the day. Hence we’re on DirectX 9. OpenGL still appears tied to SGI two-decade old architecture. Hence we’re on OpenGL 2[1].

What can be done to fix this? Make a simple utility library to save/load textures would be good for one. Also a rapidly updated OO API on top of OpenGL that codifies all the myriad OpenGL frameworks within a single abstraction layer would be another. Just banging on about how Open{GL,AL} + SDL is a valid solution isn’t.

[1] Playing version number penis size is dangerous but in this case it does illustrate a valid point.

Apple rant

Monday, August 21st, 2006

OK so this is a slightly poorly titled post. It isn’t completely Apple’s fault that this weekend was such a pain :). Also this a hacker post mostly for my own therepy so it is probably terribly boring…

Update: Of course now I see that xine-lib CVS already has had some Darwin love applied… Grr.

This weekend I found a little time to try to move a little further in making XinePlayer a Universal application[1]. This isn’t as trivial as it might at first sound since, of course, XinePlayer uses xine as its media playback engine and xine a) doesn’t have a build chain that plays nice with Universal binaries — in fact it is autotools which doesn’t play nice generally, b) isn’t primarily developed on OSX and especially not Intel OSX and c) has lots of processor specific parts.

Carry on reading for the pain that was involved but, to cut a long story short, I can now build a standalone Universal XineKit which contains all the plugins one might expect and can be embedded in your apps. It can be embedded so easily in fact that I wrote a really quick Universal XineKit-based DVD player with which I successfully watched my brand new Matrix DVDs. Amazingly it even worked when run under Rosetta!

Before I talk about XineKit I should just note that the nasty hacky way I was displaying video with previous XinePlayer versions just didn’t work under Intel OS X. Consequently I took the plunge and made a generic video output system the first thing that will be implemented by my, to be GPLd, utility framework RWUtilityKit. Ultimately I’m going to roll all the stuff I wrote for handling global hotkeys, preferences dialogues, etc into it. The output system I wrote is now based on OpenGL and performs colourspace conversion in hardware. For some planar formats it even uses a little fragment program[2] which does this using the magic of modern GPUs.

The API (1, 2) for this video renderer is pretty easy to use. Just create as many video frames as you want. Fill in the buffers and call setCurrentFrame: on a RWVideoView plonked somewhere in your window. Under the covers it uses the new CoreVideo framework to provide synchonisation with refresh rate, etc.

The first hurdle was getting xine to compile on Intel OS X at all. Luckily there has been some OS X love shown to xine independant of any of my efforts but almost none of this was to Intel bits. Mostly I had to workaround bugs in Apple’s Intel gcc C compiler, especially when using its inline assembler. Once that was sorted there were a few little endian fixups and some Darwin specific fixes for byte swapping etc. The patch set ended up quite small but was an effort to obtain. Some of them are suitable submission to xine though (especially the trivial endian patches) so it might be easier compiling xine in the future.

The second hurdle was making libxine and all of its plugins Universal. I took the sledgehammer approach and decided to cross-compile a complete libxine for both x86 and PPC and then merge them together with lipo. The build script is nasty but it does result (on my MacBook at least) in a Universal build of xine.

Lastly I wanted to separate out XineKit from XinePlayer to make it useful as a standalone framework. Making that easy for the application writer was hindered by the nastiness in the OSX linker. Basically all libraries are linked into objects using the full path and the concept of a ’search path’ is somewhat different from that of the ELF loader. Consequently one has to work inside a small box relative to the final binary which loads the framework. Unfortunately there is no such box when one trys to link shared libraries with Frameworks. After a lot of effort I eventually gave up on getting the XineKit framework to play nice with an embedded libxine.dylib and decided to just use static linking. It bloats the binary a little but it does at least work.

[1] For the non-OSXers out there a Universal binary is what used to be called a fat binary :). It contains, in this case, both x86 and PPC object code so the same binary can be used on both versions of OSX.

[2] A fragment program is a small bit of assembler you can upload to the GPU which gets run once for each pixel you render. In this case it reads the planar video texture created by xine and does the requisite YUV to RGB magic. This might be the subject of a future rant where I explain exaxtly why DirectX is nicer to code for than OpenGL.

Radiosity 2

Tuesday, July 11th, 2006

Well we launched our radiosity algorithm on the world at 9am this morning. It is eight hours later and our poor little web server has spewed 6.23GB to the world. We’ve already had people a) not believing us and b) thinking we’re using a popular algorithm (PRT) which we aren’t. For anyone reading this (and suprisingly the biggest referrer to our site is my blog!) here are some FAQ answers.

Does it handle dynamic geometry?
Not yet but this isn’t forced by the algorithm, just our implementation.
Is it real?
In that we just hand the model to our precompute + renderer yes. No tricks, all the stuff you see in the video is real in that it wasn’t authored.
How long is the precompute?
It depends on the scene, for the scene in the movie for a HQ precompute one would get some pocket change out of half-an-hour.