Deliverables
Because this project also served as my MSc Individual Project at Imperial College London I have written a report and a presentation for it as well.
You can also see what this project does in less than 3 minutes from the following video:
Future work
Although the final deadline for this project has passed there are still a couple of things left to be done. I have split them into two main categories: implementation, improving and extending on the existing code / ideas and algorithm, introducing new ideas that should give better results.
- Implementation
The existing render manager can be improved by extending it to support:
- Multiple lights – at the moment only one light is supported. Multiple lights can be added by using a 3D texture, because every light generates a maximum of up to 10 textures.
- Multiple mesh types – for now only genmesh is supported. Although this is the most generic type of mesh used in Crystal Space, support for materials from other types of meshes has to be added as well.
- Rendering smoke and clouds – this will involve adding a volumetric rendering technique and apply the algorithm from the current render manager without any further modifications.
- Algorithm
A couple of ideas can be added to produce better renderings or make the application run faster.
- Different split ratio for each ray – at the moment the split ratio is computed globally and this causes problems when objects with different densities are present in the scene. One possible way of having a unique split function for each ray is by creating a split texture, storing the split ratios for each individual ray onto a corresponding pixel.
- Recomputing the split ratio in real-time – this is the current implementation bottleneck, because it is done on the CPU. If the split ratio will be computed for each ray, then no global information will be needed and the computation can be done faster on the GPU by adding a new image processing render pass.
- Compute the optimal number of layers – the number of layers is currently chosen by the user. However, a test scene can be created so that the maximum number of layers that keeps the application real-time will be automatically chosen.
Performance
As a final step for this project I tested the performance of the three algorithms I implemented throughout the summer: Opacity Shadow Maps (OSM), Deep Opacity Maps (DOM) and Bounding Opacity Maps (BOM).
I measured the performance in frames-per-second (FPS) using the average FPS provided by Fraps benchmarking tool for a period of 60 seconds. The GPU card (Nvidia GT 335M) had the most important contribution in the measures taken, because all three algorithms are GPU bound (they involve rendering to texture multiple times and no significantly computational task is done on the CPU).
The performance is shown in Figure 1
Figure 1 Plot generated using gnuplot. The variance between the number of layers and the FPS for the three algorithms implemented for this project. Even thought Bounding Opacity Maps have the worst performance they produce realistic renderings even when using the minimum number of layers tested because they follow the real-world light’s distribution (BOM).
Adding common lighting effects
- Blinn-Phong shading model
The methods presented so far give only information about shadows cast by translucent objects on themselves. In a Blinn-Phong shading model this would correspond to the diffuse lighting, but the specular and the ambient lighting have to be added as well. These two terms were used as described in the Blinn-Phong model (Figure 1), although for the specular component more advanced techniques such as Kajiya and Kay or Marschner could be used when rendering hair.
Figure 1 Picture from Crystal Space. Scene rendered without any specular or ambient lighting (a) and scene rendered using these two terms as described in the Blinn-Phong model (b).
- Shadow mapping
Opaque objects were introduced in the scene by using the depth map that computes the initial splitting positions. This map is actually a regular shadow map, so adding opaque objects to the scene was just a matter of applying a common shadow mapping algorithm without any additional computational costs (Figure 2).
Figure 2 Picture from Crystal Space. Opaque objects don’t cast shadows (a) and cast shadows using shadow mapping (b).
Hybrid split
As I mentioned in the previous post a hybrid split between the linear and the logarithmic split can be a good idea because when the linear splitting scheme falls short the logarithmic one can be used and the other way around.
Because when multiple layers contain the same information artifacts may occur, the criteria for choosing the ratio between linear and logarithmic splitting is so that it produces consecutive layers as different from each other as possible. Or put in another way each new layer should bring new information. In terms of computer vision this can be translated to having the mutual information between these images as small as possible.
Two techniques of measuring mutual information were tested: sum of absolute differences and cross-correlation coefficient.
The sum of absolute differences is pretty straight forward to compute and it involves adding the absolute value of the difference between each two corresponding pixels from the two images.
The cross-correlation coefficient represents the ratio between the covariance of two images and the product of their standard deviation, and can be computed using the following formula:
where Ī(•) is the mean of image I. Another useful property about correlation is that it has values on a scale ranging from [-1, 1] and it gives a linear indication of the similarity between images.
As expected from the findings in the previous post the mutual information is smaller when choosing a more linear split for sparse objects and a more logarithmic one for denser objects (Figure 1).
Figure 1 Plot generated using gnuplot. Linear splitting corresponds to a split ratio of 0 while logarithmic splitting maps to the value of 1.
Because, as we can see from Figure 1, the cross-correlation coefficient (shown in green) covers a wider range of values, giving better estimate for each density value, it was chosen as the default method of computing the mutual information. The cross-correlation probably performs better due to the influence of standard deviation, which is completely neglected for the sum of absolute differences (shown in red).
Splitting scheme
- Linear splitting
The most frequently used splitting scheme for choosing the opacity maps’ position is the linear one. It has been used as the primary splitting scheme in both opacity shadow maps (OSM) and deep opacity maps (DOM).
However, if we were to look at the light’s distribution on real-world translucent objects such as clouds, trees or hair we can observe that for dense objects the lighting caused by self-shadowing changes only at the very beginning of the object (Figure 1).
Figure 1 Real-world photographs of clouds (a) and bushes (b). It can be observed that for these objects the lighting only changes at the very beginning of the object.
In such cases a linear distribution would produce layers that contain almost the same information from a certain layer onwards (Figure 2). A distribution that would have more layers near the beginning of the object and fewer at the end would probably give better results.
Figure 2 Layers obtained using linear splitting on a scene with a dense model. The last four layers contain almost the same information.
- Logarithmic splitting
The logarithmic distribution has a slower increase rate and therefore produces a splitting that has a higher density of layers at the beginning of the object (Figure 3).
Figure 3 Comparison between linear and logarithmic distributions. Linear increase, blue, versus logarithmic increase, green (a), linear split (b) and logarithmic split (c).
Obtaining layers that have different shadow information prevents artifacts like the ones shown in Figure 4.
Figure 4 Difference in rendering when using linear splitting (a) and logarithmic splitting (b). Linear splitting (a) produces incorrect self-shadows because most of the layers contain the same information (Figure 2).
- Hybrid split
Although logarithmic splitting produces good results on dense objects, it doesn’t preform well on sparse objects because the lighting caused by self-shadowing changes throughout the entire length of the object (Figure 5).
Figure 5 Real-world photographs of clouds (a) and trees (b). It can be observed that for sparse objects the lighting changes throughout the entire length of the object.
The rendering artifacts that occur when logarithmic splitting is performed on sparse objects can be seen in Figure 6.
Figure 6 Difference in rendering when using logarithmic splitting (a) and linear splitting (b). Logarithmic splitting (a) produces artifacts: the willow is incorrectly lit near the top, because the layers don’t uniformly cover the whole length of the sparse object.
However, because the linear splitting scheme can be used when the logarithmic one fails and vice versa, using a hybrid split between the two of them based on the given scene should produce artifacts free renderings. More on this hybrid split in the next post.
Bounding Opacity Maps
Because Deep Opacity Maps (DOM) give information only about the starting position for the splitting points, two major issues appear:
- The position of the splitting points can’t be precisely determined for any type / shape of object because the end splitting position is not specified.
- The layers only follow the light’s distribution at the beginning of the object, where the start points are known via the depth map (Figure 1).
Figure 1 A translucent full sphere as seen in real-life (a), the distribution of layers when using DOM (b) and the way the light is distributed in real-life (c).
Furthermore, the example from Figure 1 is not a particular case, the light distribution following the shape of the object for other translucent real world objects, such as blonde hair or trees, as can be seen in Figure 2 and Figure 3.
Figure 2 Real-life lighting of blonde hair (a) and the corresponding layers and light distribution (b). It can be observed that the layers and the light distribution follow the shape of the object.
Figure 3 Real-life lighting of a tree (a) and the corresponding layers and light distribution (b). It can be observed that the layers and the light distribution follow the shape of the object.
By computing an extra depth map, in which depth information about the furthest away points is given, instead of the closest ones, the limitation of DOM regarding the lack of information for the end splitting points is solved, and more important the layers follow the light’s distribution in real-life.
This is achieved by Bounding Opacity Maps (BOM) where the layering follows the light distribution in real-life by interpolating the values from the two depth maps when choosing the splitting points.
The difference in splitting between DOM and BOM in Crystal Space can be seen in Figure 4 and the difference in rendering in Figure 5.
Figure 4 Difference in splitting between DOM (a) and BOM (b) when using 16 layers – first layer corresponds to light green and the last layer to black. It can be seen that because the end splitting points are not specified in DOM the layers don’t cover the whole length of the object (the final color is not black as in (b)).
Figure 5 Difference in rendering between DOM (a) and BOM (b) when using 16 layers. Because DOM don’t specify the end splitting points some grass strands (from the red circle), corresponding to the last layer, are given false shadow information.