Cel Shader

I've recently had to dabble in non-photorealistic rendering (NPR) techniques, specifically cel or toon shading and silhouette edge rendering, and thought I'd share a simple cel shader that I implemented.

Toon shading is a rendering style that gives everything a "cartoony" look. It can have a good deal of variation applied to it, but at its simplest it applies solid colors with distinct color boundaries to objects, akin to what you'd see in cartoons. The cartoon feel can be further enhanced by applying some type of black outlining around significant features of the object, i.e. some type of silhouette edge rendering.

Below is a composite screenshot taken from my demo showing the difference between an NPR-rendered and non-NPR-rendered teapot. The teapot on the left uses classic Phong shading, while the teapot on the right uses cel shading with silhouette edge rendering.


The cel shading is accomplished by first calculating classic Phong shading values, then mapping them to their final color based on the computed light intensity. My particular implementation uses an ambient layer, three diffuse layers, and one specular layer. To make the color transitions slightly smooth (instead of a hard boundary) we use the HLSL instinsic smoothstep which bleeds each subsequent layer into the previous one. There are other more efficient techniques for cel shading (with varying results) but this is what I chose to go with. The HLSL 4.0 pixel shader code is shown below.

// layerXStart: Percent of light intensity at which to start each diffuse layer. Subsequent layers
// should be greater than the previous ones. Valid range is [0,1].
static float layer1Start = 0.0;
static float layer2Start = 0.1;
static float layer3Start = 0.25;

// Range, at the beginning of each layer, to bleed into the previous one. Valid range is (0,1].
static float layerBleed = 0.025;

// layerXColor: Percent of diffuse material color for each diffuse layer. Valid range is [0,1].
static float layer1Color = 0.35;
static float layer2Color = 0.7;
static float layer3Color = 1.0;

// Specular modifiers have the same meanings as above.
static float specStart = 0.35;
static float specBleed = 0.2;
static float specColor = 0.7;

float4 PS_CelShade(PS_INPUT input) : SV_TARGET
{
   input.normal = normalize(input.normal);
   input.viewDir = normalize(input.viewDir);
   vector reflection = reflect(lightDir * -1.0, input.normal);
 
   float diffuseIntensity = saturate(dot(lightDir, input.normal));
   float specIntensity = pow(saturate(dot(input.viewDir, reflection)), potSpecPower) * ceil(diffuseIntensity);
 
   float toonIntensity = 0.0;
   toonIntensity += smoothstep(layer1Start, layer1Start + layerBleed, diffuseIntensity) * layer1Color;
   toonIntensity += smoothstep(layer2Start, layer2Start + layerBleed, diffuseIntensity) * (layer2Color - layer1Color);
   toonIntensity += smoothstep(layer3Start, layer3Start + layerBleed, diffuseIntensity) * (layer3Color - layer2Color);
 
   float4 outputColor = 0.0;
   outputColor += potKs * (lightKs * smoothstep(specStart, specStart + specBleed, specIntensity) * specColor);
   outputColor += potKd * (lightKd * toonIntensity);
   outputColor += potKa * lightKa;
   outputColor.a = 1.0;
 
   return outputColor;
}

Silhouette rendering is accomplished through a bit of rendering trickery known as the shell extension method. After the cel shading is performed, the object is rendered again with front-face culling turned on, i.e. the back-facing triangles will be visible instead of the front. Each vertex is then extented slightly along the vertex normal in the vertex shader, proportional to the distance from the camera (to create a uniform extension as viewed from the camera). The new vertex positions are then sent to the pixel shader, which just outputs black as the color, resulting in a black outline around the object wherever the back-facing pixels are visible.

Of course, this particular method of silhouette rendering only works if an object's vertices all use shared, averaged vertex normals. For example, this method wouldn't work on a cube whose vertices all used different vertex normals; it would leave clear gaps in the outline around the cube.

Here's the complete vertex and pixel shader code for silhouette edge rendering.

// This controls the thickness of the outline.
static float thickness = 0.0035;

// Vertex shader that uses the shell extension method to create a silhouette outline.
float4 VS_ShellExtension(VS_INPUT input) : SV_POSITION
{
   input.position.w = 1.0;
   input.normal.w = 0.0;
 
   // Put everything into world-space first.
   input.position = mul(input.position, world);
   input.normal = mul(input.normal, world);
 
   // Offset the vertex along the normal, proportional to the distance from the camera.
   float distance = min(length(input.position - cameraPos), 20.0);
   input.position += input.normal * distance * thickness;
   input.position = mul(input.position, viewProj);
 
   return input.position;
}

// Simply outputs black.
float4 PS_Black(float4 position : SV_POSITION) : SV_TARGET
{
   return float4(0.0, 0.0, 0.0, 1.0);
}

When calculating the distance from the camera you may have noticed the upper bound placed upon it. Without this, the outline would maintain a uniform thickness regardless of the size of the model, which is exactly what we want as long as the model is visibly large enough that the outlining doesn't dwarf the actual geometry. As the model gets farther and farther away from the camera, however, the outline begins to overcome the geometry until eventually all you see is a black blob. Placing an upper limit on the distance forces the outline to start scaling down with the model at that distance, thus avoiding the problem of the "black blob".

The source code for the demo is given below, along with an .exe-only version for those who just want to see it in action. Again, a DX10-capable GPU is required. Enjoy!

CelShader Source.zip (full source code/project)
CelShader Demo.zip (.exe only)