Understanding the ray tracing debate

Intel are currently working on technology that they think will completely revolutionise video game development: a ray traced graphics system. Now to understand the significance of this, it's necessary to understand a bit about current graphics technology.

Every scene in a video game starts out it's life as a large collection of objects that together are usually combined into a scene graph. These objects are usually expressed in terms of their component polygon meshes, textures, etc. After all the physics calculations, collision detection and other game related work has taken place on the CPU, the resultant information is plugged into a rendering engine which traverses the graph and sends the display information to the graphics card. The graphics card turns this 3D co-ordinate data into a 2D image and presents it into a viewport on the user's monitor. The ray tracing debate primarily revolves around this 3D to 2D rendering process.

Modern graphics cards are very fast at performing floating point calculations and matrix operations. Despite video game graphics only using three dimensions, most APIs use four dimensional homogenous co-ordinates and matrices, with each point specified as a floating point number. The usual x, y, z co-ordinates are supplemented by the mysterious w co-ordinate. You need this extra dimension to perform translations in addition to linear transformations (known as affine transformations) with matrix multiplication. A video game will define a special matrix that is pre-multiplied with each point of every polygon by the graphics card. This special matrix is called a projection matrix and defines how 3D objects will be "projected" into 2D space. Game developers usually want to make their games look photo-realistic, so often they will use a modified perspective matrix to orient and translate the scene around the camera and project the co-ordinates similar to how an eye would observe the objects in real life due to how light works. This process is known as rasterisation. After the graphics card has worked out the position of all the objects, it then iterates over every pixel visible on the screen and performs lighting and texture calculations to work out the colour of the visible pixel.

As you will undoubtedly have noticed, this is extremely non-physical. In real life, a camera has a sensor, a lens and a scene has lights and objects. Photons travel from the lights, get caught up in all sorts of quantum electrodynamic interactions with matter. Eventually, these photons make their way through the camera lens and hit part of the camera sensor causing a change in energy that can be detected and processed electronically.

This important difference means that a lot of the visual effects you take for granted in video games had to be painstakingly emulated by the designers and programmers of the game and it's engine. Shadows, reflection, diffusion, refraction and many other lighting effects aren't possible with raw rasterisation. Often game engines will try to pre-render things like permanent shadows on the floor caused by buildings. Beyond that, it's up to the game developer to write shaders that attempt to simulate lighting effects for the most important elements of the scene, neglecting the less important elements.

This is where ray tracing comes in. When you see photorealistic renders produced by computers, you'll often find that they were produced using a ray tracing algorithm. Ray tracing usually refers to the process of casting a ray from the each pixel of the camera and seeing what object it intersects. When this ray intersects an object, it can produce up to three new types of rays depending on the geometry of the object – a reflection, a refraction and a shadow. This process is repeated until the ray encounters a light source, and a colour for the pixel is generated using the information accumulated during the ray's travels. This algorithm is very simplistic, but modern algorithms can generate multiple rays per pixel, generate multiple rays per intersection, use statistical processes and perform fairly complex calculations at intersection. The core of the algorithm remains the same, however. Further extensions include Radiosity. This makes scenes look far more realistic, but it's far too complex to explain here.

Ray traced graphics will supposedly use a version of this algorithm instead of the rasterisation method outlined above. It's clear where the appeal of this system lies. The calculations are very similar to those required for the physics engine, which is good, because arguably lighting is just an application of physics. nVidia are already on their back foot and acting defensive, probably because such a change in the industry would require a complete redesign of their graphics cards, which are currently optimised for a rasterisation pipeline.

The problem with ray tracing, however, is that it is exceptionally slow. Despite all the recent advances in ray tracing and the hugely complex tasks involved in making rasterised images look decent, on current computers rasterisation has the edge if not only for the fact that it's easier to take shortcuts. However, as video games become increasingly more dynamic, require increasingly more polygons and computers become even more powerful, ray tracing will become more attractive. If you consider rasterisation without all the shortcuts afforded by the fact that users don't expect every little bit of the scene to be photorealistic, it's not much faster than well optimised and accelerated ray tracing. Global illumination, high polygon models, decent lighting effects and motion blur are all far easier to implement with ray tracing. Additionally, models can be represented mathematically when ray tracing, allowing far more realistic curves and shapes. I cringe whenever I see ten sided "wheels" in video games.

A second problem is that ray tracing isn't good for everything. Many of the desktop 3D effects available on Vista, OS X and Linux wouldn't make sense with ray tracing. These desktop effect systems pretty much define the perfect use case for rasterisation. However, by the time that ray tracing becomes viable, the chances are that floating point CPUs will easily be fast enough to perform these calculations without specific rasterisation pipelines.

So where is the future of game development going? Well for the meantime, high quality graphics are rasterised, and they'll stay that way for at least a few years. Though, as time goes on we're going to see more and more research going into ray tracing and perhaps preliminary support in game engines for ray tracing. I think that the computer graphics will never be 100% ray traced simply because certain tasks gain no advantage from ray tracing. The next generation of video game consoles (PS4, Xbox "720"?) may very well have ray tracing hardware. Certainly, if we continue to aspire to photorealism, ray tracing is the future.