This manual pretends to offer a brief overview of the Dump Test Tools (DTT), covering both design and usage aspects. I would be grateful if you could review the code and provide some feedback.
The DTT package is constituted by the following files:
DTT's goal is to provide a light weight infrastructure to generate severe kernel failures (panics, hangs, exceptions, etc) in a controlled manner. These anomalies in the kernel are caused at so called crash points, which were implemented as hooks (IBM's "Kernel hooks" in particular). I will use "crash point" and "hook" interchangeably throughout this document.
Pursuing to minimize the intrusiveness of DTT, a mechanism was created so that hooks inserted in a kernel module can be registered from user space (thus saving some lines in the module's init and exit functions). Besides, hooks built into the kernel (built-in hooks) are automatically registered at DTT's initialization time.
Finally, it is possible to perform dynamic configuration of the hooks.
Details are offered in subsequent sections.
To apply the kernel patches please follow the steps below:
# cd KERNEL_SRC_BASE # zcat PATH_TO_PATCH/kernelhooks-v1.9-2.6.9-combined.patch.gz | patch -p1 # zcat PATH_TO_PATCH/dumptesttools-core.patch.gz | patch -p1 # zcat PATH_TO_PATCH/dumptesttools-crashpoints.patch.gz | patch -p1Alternatively, the full patch can be used:
# zcat PATH_TO_PATCH/dtt-full.patch.gz | patch -p1
Once the desired patches have been applied we can proceed to configure the kernel as usual making sure that the options below are also enabled.
# make menuconfig Kernel hacking ---> Kernel debugging [*] Kernel Hook support [*] or [M] Crash points [*] or [M]
Kernel build and system reboot:
# make # make modules_install # make install # shutdown -r now
In case DTT was compiled as a module modprobe it:
# modprobe dtt [iter_num={>0}] [debug={0|1}]
The module has the following options:
stack growth >= iter_num * sizeof(buf) (please see "recursive_loop" function at drivers/dtt/khdtt.c)
After applying the aforementioned "dumptesttools-crashpoints.patch.gz" patch a basic set of hooks is installed in the kernel source. The procedure to install new hooks depends on whether it is to reside on a kernel module or not.
In any case the header file "include/linux/khdtt.h" is always needed
（#include
Even though the method is general, architecture-dependent definitions are conveniently moved to architecture-specific parts of the kernel's source tree.
Insert a DECLARE_CPOINT definition in "include/linux/dtt_cpoints.h".
Example: DECLARE_CPOINT(MEM_SWAPOUT, 4)
Argument 1: hook's name
Argument 2: hook's ID. It must be an unused number in the
range 1~MAX_CPOINT_ID==300 (include/linux/khdtt.h)
Should be defined in "include/asm-*/dtt_cpoints.h".
Example: DECLARE_CPOINT(INT_HARDWARE_ENTRY,1)
Argument 1: hook's name
Argument 2: hook's id
To add a new hook (crash point) insert a CPOINT macro at the desired location.
Example: CPOINT(MEM_SWAPOUT) (mm/vmscan.c)
Argument 1: hook's name as defined previously
When the dtt module loads the built-in hooks are automatically registered.
# make;make modules_install;make install
Hooks inserted in kernel modules do not need to be defined at kernel compile-time and, further to this, can be registered from user space at any moment. The purpose of this is reducing the intrusiveness of the DTT patches. Of course you still can register the hook from the module's init function. Thus two registration methods are provided for kernel modules.
Insert a DECLARE_CPOINT macro in the file where the hook is to be attached (somewhere near the declaration of global variables should not be a bad place).
Example: DECLARE_CPOINT(SCSI_DISPATCH_CMD, 151) (drivers/scsi/scsi.c)
Argument 1: hook's name
Argument 2: hook's ID. It must be an unused number in the
range 1~MAX_CPOINT_ID==300 (include/linux/khdtt.h)
In case you are inserting the hook in an inline function use DECLARE_CPOINT_INLINE instead of DECLARE_CPOINT
Usage: DECLARE_CPOINT_INLINE(name, cpoint_id)
To add a new hook (crash point) insert a CPOINT macro at the desired location.
Example: CPOINT(SCSI_DISPATCH_CMD) (scsi_dispatch_cmd function)
Argument 1: hook's name as declared before.
In case you are inserting the hook in an inline function use CPOINT_INLINE instead of CPOINT.
Usage: CPOINT_INLINE(name)
A distinctive characteristic of DTT is that it possible to register hooks inserted in a kernel module both from the module itself and user space.
Insert the adequate cpoint_add macro somewhere in the module's init function.
Example: cpoint_add(SCSI_DISPATCH_CMD)
Argument 1: hook's name as declared before.
It is possible to register hooks from user space. For this purpose the ttutils application is provided. As one might expect root privileges are needed to run it.
Example: # ./ttutils add -p SCSI_DISPATCH_CMD -n 151
For more details please consult "User space tools" below.
Call cpoint_remove from the module's exit function.
Example: cpoint_remove(SCSI_DISPATCH_CMD)
Argument 1: hook's name.
Hooks registered from user-space have to be removed from user-space. Once again, we need ttutils for that.
Example:
# ./ttutils rmv -p SCSI_DISPATCH_CMD
For more details please consult "User space tools" below.
Example:
make;make modules_install
# tar xzf dtt_tools.tar.gz # cd dtt_tools # make
ttutils command [options]
command = {help|ver|version|list|ls|add|set|reset}
Display usage information.
Example:
# ./ttutils
Usage: ./ttutils {help|ver|version|list|ls|set|reset｜add|rmv} [options]
Display version number of DTT.
Example:
# ./ttutils version
Version: 0.1
Show registered points (crash points)
Example:
# ./ttutils ls
id crash type crash point name count location
1 none INT_HARDWARE_ENTRY 0 kern
2 dummy1 INT_TASKLET_ENTRY 63 kern
3 none FS_DEVRW 0 kern
4 panic MEM_SWAPOUT 7 kern
5 none TASKLET 0 kern
6 none TIMERADD 0 kern
201 none test_cpoint1 0 mod
100 none test_cpoint2 0 dyn
Change the characteristics of a registered hook (crash point).
hook's name
Type of kernel failure to be generated. Currently available types:
If no -t option is specified the hook type remains unchanged.
Note: when a hook is registered its type is set to none by default.
Number of times the crash point is crossed before the failure associated with its type is triggered. The default value is 10.
Example:
# ./ttutils set -p INT_TASKLET_ENTRY -t panic -c 100
Disable a hook. Besides, the associated counter that keeps track of the number of times the crash point was traversed is also reset.
Hook's name.
Reset not only the hook's counter but also revert the hook's type to none.
Example:
# ./ttutils set -p INT_TASKLET_ENTRY -c 150
# ./ttutils ls
id crash type crash point name count location
1 none INT_HARDWARE_ENTRY 0 kern
2 panic INT_TASKLET_ENTRY 145 kern
[without -f option]
# ./ttutils reset -p INT_TASKLET_ENTRY
# ./ttutils ls
id crash type crash point name count location
1 none INT_HARDWARE_ENTRY 0 kern
2 panic INT_TASKLET_ENTRY 0 kern
[using -f]
# ./ttutils reset -p INT_TASKLET_ENTRY -f
# ./ttutils ls
id crash type crash point name count location
1 none INT_HARDWARE_ENTRY 0 kern
2 none INT_TASKLET_ENTRY 0 kern
Register a hook from a kernel module so that it can be actually used.
This is aimed at modules that do not register the hooks inserted in their own code.
Please note that registering does not imply activation. Activation is accomplished using "-set".
Hook's name.
Hook's ID.
Example:
# ./ttutils ls
id crash type crash point name count location
1 none INT_HARDWARE_ENTRY 0 kern
2 none INT_TASKLET_ENTRY 0 kern
201 none test_cpoint1 0 mod
# ./ttutils add -p IDE_CORE_CP -n 50
# ./ttutils ls
id crash type crash point name count location
1 none INT_HARDWARE_ENTRY 0 kern
2 none INT_TASKLET_ENTRY 0 kern
201 none test_cpoint1 0 mod
50 none IDE_CORE_CP 0 dyn
Remove a hook registered with "-add".
Hook's name.
Example:
# ./ttutils set -p IDE_CORE_CP -t dummy1 -c 3
# ./ttutils ls
id crash type crash point name count location
1 none INT_HARDWARE_ENTRY 0 kern
2 none INT_TASKLET_ENTRY 0 kern
201 none test_cpoint1 0 mod
50 dummy1 IDE_CORE_CP 3 dyn
# eject
# ./ttutils ls
id crash type crash point name count location
1 none INT_HARDWARE_ENTRY 0 kern
2 none INT_TASKLET_ENTRY 0 kern
201 none test_cpoint1 0 mod
50 dummy1 IDE_CORE_CP 2 dyn
# ./ttutils rmv -p test_cpoint2
# ./ttutils ls
1 none INT_HARDWARE_ENTRY 0 kern
2 none INT_TASKLET_ENTRY 0 kern
201 none test_cpoint1 0 mod