DTT Reference Manual

8 November 2005

This version:
http://lkdtt.sourceforge.net/reference_manual.html
Author:
Fernando Luis Vázquez Cao

Abstract

This manual pretends to offer a brief overview of the Dump Test Tools (DTT), covering both design and usage aspects. I would be grateful if you could review the code and provide some feedback.

Status of this document

Quick Table of Contents

Files

The DTT package is constituted by the following files:

DTT's main features

DTT's goal is to provide a light weight infrastructure to generate severe kernel failures (panics, hangs, exceptions, etc) in a controlled manner. These anomalies in the kernel are caused at so called crash points, which were implemented as hooks (IBM's "Kernel hooks" in particular). I will use "crash point" and "hook" interchangeably throughout this document.

Pursuing to minimize the intrusiveness of DTT, a mechanism was created so that hooks inserted in a kernel module can be registered from user space (thus saving some lines in the module's init and exit functions). Besides, hooks built into the kernel (built-in hooks) are automatically registered at DTT's initialization time.

Finally, it is possible to perform dynamic configuration of the hooks.

Details are offered in subsequent sections.

Kernel patches

Installation

To apply the kernel patches please follow the steps below:

# cd KERNEL_SRC_BASE
# zcat PATH_TO_PATCH/kernelhooks-v1.9-2.6.9-combined.patch.gz | patch -p1
# zcat PATH_TO_PATCH/dumptesttools-core.patch.gz | patch -p1
# zcat PATH_TO_PATCH/dumptesttools-crashpoints.patch.gz | patch -p1
Alternatively, the full patch can be used:
# zcat PATH_TO_PATCH/dtt-full.patch.gz | patch -p1

Once the desired patches have been applied we can proceed to configure the kernel as usual making sure that the options below are also enabled.

# make menuconfig
	Kernel hacking --->
		Kernel debugging [*]
			Kernel Hook support [*] or [M]
				Crash points [*] or [M]

Kernel build and system reboot:

# make
# make modules_install
# make install
# shutdown -r now

How-to use

In case DTT was compiled as a module modprobe it:

# modprobe dtt [iter_num={>0}] [debug={0|1}]

The module has the following options:

iter_num:
set the recursion level for the stack overflow test (default is 10) (XXX Yeah, I know, the naming of this option could not be any worse. I will rename it to recur_level. "iter" stands for iteration (no comments). I should sleep more.)

stack growth >= iter_num * sizeof(buf) (please see "recursive_loop" function at drivers/dtt/khdtt.c)

debug:
verbosity level of debug messages

Adding crash points (hooks)

After applying the aforementioned "dumptesttools-crashpoints.patch.gz" patch a basic set of hooks is installed in the kernel source. The procedure to install new hooks depends on whether it is to reside on a kernel module or not.

In any case the header file "include/linux/khdtt.h" is always needed (#include ).

To the kernel image

Definition

Even though the method is general, architecture-dependent definitions are conveniently moved to architecture-specific parts of the kernel's source tree.

Insertion

To add a new hook (crash point) insert a CPOINT macro at the desired location.

          Example: CPOINT(MEM_SWAPOUT)    (mm/vmscan.c)

            Argument 1: hook's name as defined previously

Registration

When the dtt module loads the built-in hooks are automatically registered.

Building the kernel

# make;make modules_install;make install

To a kernel module

Hooks inserted in kernel modules do not need to be defined at kernel compile-time and, further to this, can be registered from user space at any moment. The purpose of this is reducing the intrusiveness of the DTT patches. Of course you still can register the hook from the module's init function. Thus two registration methods are provided for kernel modules.

Definition

Insertion

Registration

A distinctive characteristic of DTT is that it possible to register hooks inserted in a kernel module both from the module itself and user space.

Registration removal

Building the kernel

          Example:

            make;make modules_install

User-space tools

Installation

# tar xzf dtt_tools.tar.gz
# cd dtt_tools
# make

How-to use

ttutils command [options]

    command = {help|ver|version|list|ls|add|set|reset}
help No options available

Display usage information.

Example:
          # ./ttutils
          Usage: ./ttutils {help|ver|version|list|ls|set|reset|add|rmv} [options]
ver(sion)No options available

Display version number of DTT.

Example:
          # ./ttutils version
          Version: 0.1
list|ls No options available

Show registered points (crash points)

Example:
          # ./ttutils ls
          id      crash type      crash point name        count	location
          1       none            INT_HARDWARE_ENTRY      0	kern
          2       dummy1          INT_TASKLET_ENTRY       63	kern
          3       none            FS_DEVRW                0	kern
          4       panic           MEM_SWAPOUT             7	kern
          5       none            TASKLET                 0	kern
          6       none            TIMERADD                0	kern
          201     none            test_cpoint1            0     mod
          100     none            test_cpoint2            0     dyn
set: ttutils set -p cpoint_name [-t trigger_type] [-c pass_num]

Change the characteristics of a registered hook (crash point).

-p cpoint_name

hook's name

-t hook_type:

Type of kernel failure to be generated. Currently available types:

  • none - just do nothing
  • panic - call panic()
  • bug - call BUG()
  • exception - generate an exception
  • loop - infinite loop
  • overflow - bloat kernel stack

If no -t option is specified the hook type remains unchanged.

Note: when a hook is registered its type is set to none by default.

-c pass_num:

Number of times the crash point is crossed before the failure associated with its type is triggered. The default value is 10.

Example:

          # ./ttutils set -p INT_TASKLET_ENTRY -t panic -c 100
reset: ttutils set -p cpoint_name

Disable a hook. Besides, the associated counter that keeps track of the number of times the crash point was traversed is also reset.

-p cpoint_name:

Hook's name.

-f:

Reset not only the hook's counter but also revert the hook's type to none.

Example:

          # ./ttutils set -p INT_TASKLET_ENTRY -c 150
          # ./ttutils ls
          id	crash type	crash point name	count	location
          1	none		INT_HARDWARE_ENTRY	0	kern
          2	panic		INT_TASKLET_ENTRY	145	kern

          [without -f option]
          # ./ttutils reset -p INT_TASKLET_ENTRY
          # ./ttutils ls
          id      crash type      crash point name        count   location
          1       none            INT_HARDWARE_ENTRY      0       kern
          2       panic		INT_TASKLET_ENTRY       0       kern

          [using -f]
          # ./ttutils reset -p INT_TASKLET_ENTRY -f
          # ./ttutils ls
          id      crash type      crash point name        count   location
          1       none            INT_HARDWARE_ENTRY      0       kern
          2       none            INT_TASKLET_ENTRY       0       kern
add: ttutils add -p cpoint_name -n id

Register a hook from a kernel module so that it can be actually used.

This is aimed at modules that do not register the hooks inserted in their own code.

Please note that registering does not imply activation. Activation is accomplished using "-set".

-p cpoint_name

Hook's name.

-n id:

Hook's ID.

Example:

          # ./ttutils ls
          id      crash type      crash point name        count   location
          1       none            INT_HARDWARE_ENTRY      0       kern
          2       none            INT_TASKLET_ENTRY       0       kern
          201     none            test_cpoint1            0       mod

          # ./ttutils add -p IDE_CORE_CP -n 50

          # ./ttutils ls
          id      crash type      crash point name        count   location
          1       none            INT_HARDWARE_ENTRY      0       kern
          2       none            INT_TASKLET_ENTRY       0       kern
          201     none            test_cpoint1            0       mod
          50      none            IDE_CORE_CP             0       dyn
rmv: ttutils rmv -p cpoint_name

Remove a hook registered with "-add".

-p cpoint_name

Hook's name.

Example:

          # ./ttutils set -p IDE_CORE_CP -t dummy1 -c 3

          # ./ttutils ls
          id      crash type      crash point name        count   location
          1       none            INT_HARDWARE_ENTRY      0       kern
          2       none            INT_TASKLET_ENTRY       0       kern
          201     none            test_cpoint1            0       mod
          50      dummy1          IDE_CORE_CP             3       dyn

          # eject

          # ./ttutils ls
          id      crash type      crash point name        count   location
          1       none            INT_HARDWARE_ENTRY      0       kern
          2       none            INT_TASKLET_ENTRY       0       kern
          201     none            test_cpoint1            0       mod
          50      dummy1          IDE_CORE_CP             2       dyn

          # ./ttutils rmv -p test_cpoint2

          # ./ttutils ls
          1       none            INT_HARDWARE_ENTRY      0       kern
          2       none            INT_TASKLET_ENTRY       0       kern
          201     none            test_cpoint1            0       mod

TODO