Collecting a thread dump from a core file (or from a running process) on Linux

Resolving The Problem

Only one of the two procedures below should be executed at one time. If your server has cored, follow step A. If your server is hanging, follow step B.

A. Generating thread dump output from a core file generated by and ITDS binary:

1. First, gdb must be installed. Please see your vender's documentation on how to install the gdb rpm.

2. As root, run the following:
gdb <path_to_ITDS_binary> <path_to_core_file> | tee /tmp/gdb.out
(This command will leave you at the gdb command prompt.)

On 32-bit systems, the path to the binary will be:
/opt/ibm/ldap/<version>/sbin/32/<program>

On 64-bit systems, the path will be:
/opt/ibm/ldap/<version>/sbin/64/<program>

Note: you can use the 'file' command to determine which binary dropped the core file, eg:

# file /home/ldapdb2/idsslapd-ldapdb2/workdir/core.11617._2009-10-27-09_12_08
/home/ldapdb2/idsslapd-ldapdb2/workdir/core.11617._2009-10-27-09_12_08: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'ibmslapd'

3. At the gdb prompt type 'thread apply all where':
gdb> thread apply all where
and wait to be returned to the gdb prompt

4. Type 'quit' to exit
All the output printed will be printed to /tmp/gdb.out. Check the file to make sure it contains something, and make this file available to Support.

Here are examples for calling gdb for each version:

5.2: gdb /usr/ldap/bin/ibmslapd /usr/ldap/bin/core.2344 | tee gdb.out
6.0: gdb /opt/ibm/ldap/V6.0/sbin/32/ibmslapd /home/ldapdb2/idsslapd-ldapdb2/workdir/core.23445 | tee gdb.out
6.1: gdb /opt/ibm/ldap/V6.1/sbin/32/db2ldif /home/ldapdb2idsslapd-ldapdb2/workdir/core.2344 | tee gdb.out
6.1 Linux x86_64: gdb /opt/ibm/ldap/V6.1/sbin/64/ldif2db /home/ldapdb2idsslapd-ldapdb2/workdir/core.2343 | tee gdb.out

Note that in 6.*, the file /opt/ibm/ldap/V6.*/sbin/ibmslapd is only a wrapper script that calls the real binary '/opt/ibm/ldap/V6.*/sbin/<bit-width>/<program>'. If gdb is pointed at the wrapper script instead of the true binary, it will fail with a file format error like the one below:

"This GDB was configured as "i386-redhat-linux-gnu"..."/opt/ibm/ldap/V6.1/sbin/ibmslapd": not in executable format: File format not recognized"

It's important to remember that the program that dropped the core may not be ibmslapd; please make sure to provide the path to the correct binary.

B. Generating a thread dump from a running ibmslapd process:

This is generally used when ibmslapd appears to be hanging. Determining exactly what ibmslapd is doing when it appears to be hung can be very important for troubleshooting. When a server appears to hang, it can be truly hung, or can be waiting on other software, like DB2 for example.

Important Note:
When running pstack or gdb against a running process to capture thread dumps, you must run it at least 4 times consecutively, with a 10 second gap between each run (so we can see if the threads are changing at all).

1. pstack and/or gdb must be installed.

2. Get the process id (PID) for the running ibmslapd process:

ps -ef | egrep -w "PID|ibmslapd"

3. Run, as root and dump the process threads into a file - Use only one of the methods below:

3.i. If pstack command is available then prefer using this method:

pstack <PID_of_ibmslapd> | tee pstack1.out

Run the above command three more times (with 10 sec intervals in between) and capture the output into separate output files such as pstack2.out etc.

gdb <path_to_ibmslapd_binary> <pid_from_above> | tee gdb1.out
which puts you at the gdb prompt again.

gdb> thread apply all where

While the gdb debugger is attached to the running ibmslapd process, the server is unable to process any requests. You may have to hit enter several times (until you see the gdb prompt) to capture stack trace for all the threads.

Type 'quit' to exit gdb, and make the gdb1.out file available to Support.
Examples:
gdb /opt/ibm/ldap/V6.3/sbin/64/ibmslapd 2344 | tee gdb.out

Run the above method three more times (with 10 sec intervals in between) and capture the output into separate output files such as gdb2.out etc.