A stencil buffer is an extra buffer, in addition to the color buffer (pixel buffer) and depth buffer (z-buffering) found on modern graphics hardware. The buffer is per pixel, and works on integer values, usually with a depth of one byte per pixel. The depth buffer and stencil buffer often share the same area in the RAM of the graphics hardware.
In the simplest case, the stencil buffer is used to limit the area of rendering (stenciling). More advanced usage of the stencil buffer makes use of the strong connection between the depth buffer and the stencil buffer in the rendering pipeline. For example, stencil values can be automatically increased/decreased for every pixel that fails or passes the depth test.
The simple combination of depth test and stencil modifiers make a vast number of effects possible (such as shadows, outline drawing or highlighting of intersections between complex primitives) though they often require several rendering passes and, therefore, can put a heavy load on the graphics hardware.
The most typical application is still to add shadows to 3D applications. It is also used for planar reflections.
Other rendering techniques, such as portal rendering, use the stencil buffer in other ways; for example, it can be used to find the area of the screen obscured by a portal and re-render those pixels correctly.
The stencil buffer and its modifiers can be accessed in computer graphics APIs like OpenGL and Direct3D.