Kernel Marker

Kernel markers were a static kernel instrumentation support mechanism for Linux kernel source code, allowing special tools such as LTTng or SystemTap to trace information exposed by these probe points. Kernel markers were declared in the kernel code by one-liners of the form:

trace_mark(name, format_string, ...);

Where name is the marker's unique name, and format_string describes the remaining arguments' types.

A marker can be on or off depending on whether a probe is connected to it or not. Code which wants to hook into a trace point first calls:

int marker_probe_register(const char *name, const char *format_string, marker_probe_func *probe, void *pdata);

to register its probe callback with the marker point (pdata is a private data value that the code wants to pass to the probe). Later, the probe is turned on and off using:

int marker_arm(const char *name); int marker_disarm(const char *name);

Using markers has a negligible overhead thanks in part to Immediate Values, another support mechanism that embeds switches in the code that can be dynamically turned on and off, without using a memory reference and thus saving cache lines.

The initial motivation to create this static instrumentation infrastructure was the large performance overhead induced by the predating dynamic instrumentation mechanism Kprobe mechanism, which depends on breakpoints. Static instrumentation can also more easily survive source code changes because the markers are in the source code.

Kernel Markers consisted essentially of a C preprocessing macro which added, in the instrumented function, a branch over a function call. By doing so, neither the stack setup nor the function call are executed when instrumentation is not enabled. By identifying the branch executing stack setup and function call as unlikely (using the gcc built-in expect), a hint is given to the compiler to position the tracing instructions away from cache lines involved in standard kernel execution.

Two Kernel Markers drawbacks were identified which led to its replacement by Tracepoints:

• Type verification was limited to scalar types because the API is based on format strings. This could be problematic if pointers must be dereferenced by the tracer code.
• The Markers "hide" the instrumentation in the source code, keeping no global registry of the instrumentation. This makes namespace conventions and tracking of instrumentation modification difficult unless the whole kernel tree is monitored.

A patch-set implementing them was merged into version 2.6.24, which was released on January 24, 2008. To address issues regarding kernel markers, Mathieu Desnoyers, their original author, implemented a simpler and more type-safe version of static probe points named Tracepoints. A patch-set implementing Tracepoints was merged into version 2.6.28, which was released on December 25, 2008. Starting then, kernel markers were slowly removed from kernel sources and eventually fully removed in Linux kernel 2.6.32, which was released on December 3, 2009.