This evening, firtree managed to render this:
Why is this impressive? Well, it means that lots of things are working and it is slowly reaching the point where it might actually be useful. Read on for an overview of the system.
Firtree lets you define a particular image processing operation, for example ‘change contrast to X’ or ’set lightness to Y’, via a so-called kernel written in a little domain-specific language which is an extension of GLSL. Kernels can be viewed as functions which are passed a co-ordinate for a pixel we wish to generate and return the colour (and alpha) for that pixel. Just like functions, they can also be passed parameters. Crucially, these parameters can include references to other kernels, called samplers which allow you to chain kernels together to form more complex operations.
Lets take the example above. What follows is written using the internal (read: ugly) API for Firtree. Eventually this will be abstracted a bit more. Firstly we define a kernel which will generate a checkerboard pattern:
const char* g_CheckerKernelSource =
" kernel vec4 checkerKernel(float squareSize, __color backColor,"
" __color foreColor)"
" {"
" vec2 dc = mod(destCoord(), squareSize*2.0);"
" vec2 discriminant = step(squareSize, dc);"
" float flag = discriminant.x + discriminant.y - "
" 2.0*discriminant.x*discriminant.y;"
" return mix(backColor, foreColor, flag);"
" }"
;
Kernel g_CheckerKernel(g_CheckerKernelSource);
KernelSamplerParameter g_CheckerSampler(g_CheckerKernel);We also define a sampler, g_CheckerSampler, which we will use later to chain this kernel. The kernel takes three parameters, the size (in pixels) of the checkerboard squares and a foreground and background colour.
Similarly, we’ll define our spotty kernel:
const char* g_SpotKernelSource =
" kernel vec4 spotKernel(float dotPitch, __color backColor,"
" __color dotColor)"
" {"
" vec2 dc = mod(destCoord(), dotPitch) - 0.5*dotPitch;"
" float discriminant = smoothstep(0.3*dotPitch-0.5,"
" 0.3*dotPitch+0.5, length(dc));"
" discriminant = sqrt(discriminant);"
" return mix(dotColor, backColor, discriminant);"
" }"
;
Kernel g_SpotKernel(g_SpotKernelSource);
KernelSamplerParameter g_SpotSampler(g_SpotKernel);Again the kernel takes three parameters; a foreground and background colour as before but this time it takes the ‘dot pitch’ (the separation between dots). Eventually Firtree will support plain ol’ image samplers but these test patterns will have to do for the moment.
Finally, we can define a kernel which implements the Porter Duff ‘over’ operator for alpha compositing:
const char* g_OverKernelSource =
" kernel vec4 compositeOver(sampler a, sampler b)"
" {"
" vec4 aCol = sample(a, samplerCoord(a));"
" vec4 bCol = sample(b, samplerCoord(b));"
" return aCol + bCol * (1.0 - aCol.a);"
" }"
;
Kernel g_OverKernel(g_OverKernelSource);
KernelSamplerParameter g_GlobalSampler(g_OverKernel);Notice how this kernel takes two samplers. The sample() function will return the value of a sampler at a particular location and the samplerCoord() function returns the location of the current pixel transformed into the sampler’s co-ordinate system.
All that is required now is to set the kernel parameters. Let’s first of all set the checkerboard square size and various colours:
static float squareColor[] = {0.75, 0.75, 0.75, 0.75};
static float backColor[] = {0.25, 0.25, 0.25, 0.25};
static float dotColor[] = {0.7, 0.0, 0.0, 0.7};
static float clearColor[] = {0.0, 0.0, 0.0, 0.0};
g_CheckerKernel.SetValueForKey(10.f, "squareSize");
g_CheckerKernel.SetValueForKey(backColor, 4, "backColor");
g_CheckerKernel.SetValueForKey(squareColor, 4, "foreColor");
g_SpotKernel.SetValueForKey(dotColor, 4, "dotColor");
g_SpotKernel.SetValueForKey(clearColor, 4, "backColor");Now let’s wire up our ‘over’ operator:
g_OverKernel.SetValueForKey(g_SpotSampler, "a"); g_OverKernel.SetValueForKey(g_CheckerSampler, "b");
And finally, to show that it is possible, let’s rotate the spot sampler’s co-ordinate system using an affine transformation matrix:
float angle = 0.2f;
float spotTransform[] = {
cos(angle), -sin(angle), -320.f,
sin(angle), cos(angle), -240.f,
};
g_SpotSampler.SetTransform(spotTransform);Now, firtree goes off and JIT-s us up a singe GPU shader which rolls all of the kernels into one operation. But wait! You might have noticed we don’t set the dot pitch. Well, we do that inside our render loop:
g_SpotKernel.SetValueForKey(
10.f * (1.0f + (float)sin(0.01f*epoch)) + 30.f,
"dotPitch");The epoch variable is monotonic in time in the program. Changing a parameter like this is perfectly valid. In fact non-sampler parameters can be changed with no performance cost as a JIT is only performed when the samplers are changed (and hence the pipeline has been restructured).
The end result on screen is the dotty pattern above except that in real life it animates in a motion sickness causing manner :).
So why on earth is this interesting? Well for a start it means that image processing operations can be coded up and executed blisteringly fast. For a high-speed video system, the ability to run a corner detector over an image at 200Hz is highly desirable. This particular example will happily run at 780Hz @ 1024×768 before becoming CPU limited. Less prosaically, a similar system is used on OS X to provide lots of the eye candy (e.g. the slightly blurry reflection on the Dock).