Troubleshooting
Problem
If your Java application experiences OutOfMemory(OOM) errors on an AIX® operating system, there are a number of diagnostic data files that are useful for diagnosing the problem. This mustgather document describes about the diagnostic data files to be collected and the procedure to collect the same.
Resolving The Problem
The following set of diagnostic data files need to be collected to diagnose an OutOfMemory error in the Java application:
1. Javacores (Multiple javacore files might be required to diagnose native memory issues)
2. Core dumps (At least two core dumps might be required to diagnose native memory issues)
3. Heapdumps
4. Snap traces for analyzing native OOM errors
5. Verbose GC logs (native_stderr.log)
6. Process logs (ps)
For these diagnostic data files to be created, check that the system is configured correctly as per setup document.
Please collect the diagnostic data files for an OutOfMemory error in the Java application as per the below steps:
1. Collect the javacore, heap dump and snap trace file generated automatically by Java runtime on an OOM occurrence.
NOTE: From Java 6.2.6 SR1 level onwards, collect the core dump produced automatically by Java runtime on an OOM occurence.
2. For Java heap exhaustion issues, collect the javacore, core dump, heap dump and snap trace files generated automatically by using -Xdump option mentioned in setup.
3. For native memory issues, generate at least three java core, core dump, heap dump and snap trace files by using the following command:
kill -3 [PID_of_problem_JVM]
When you observe the application starting to use excessive amounts of memory, start generating these above files at regular intervals.
The time interval between each generation of files depends on how quickly the application encounters native OOM error. For example, if the application takes 30 minutes to produce an native OOM error, issue the kill command at 10-minute intervals.
Please note that all the javacores, core dumps, heap dumps and snap traces can be found in the current path and the names of them will follow the below format:
Javacore : javacore.<time stamp>.<id>.txt
Core dump : core.<time stamp>.<id>.dmp
Snap trace : snap.<time stamp>.<id>.trc
Heap dump : heapdump.<time stamp>.<id>.phd
Run the JExtract tool against the generated core dump file using the following command:
<installpath>/jre/bin/jextract [coredump]
For 64bit JVM enabled with -Xcompressedrefs option, please use
<installpath>/jre/bin/jextract -J-Xcompressedrefs [coredump]
Save the resulting archive file, which has a name in the format of
core.<time stamp>.<id>.dmp.zip
4. Collect Verbose Garbage Collector data. This data is in the location specified during setup. Alternatively, the data is sent to the stderr output.
5. Collect process size monitoring data. The data is in the location specified during setup.
After collecting all the above diagnostic data files, you can submit them for help with diagnosing the problem.
Additional Information
When diagnosing the native OOM errors deeply, IBM service might ask you to collect additional information as mentioned below:
1. If the problem is suspected to be caused by a leak within the JVM, you can enable the JVM argument: -Xcheck:memory:callsite=<threshold>
This prints an allocation summary for every <threshold> allocations, by tracking calls from the JVM to the operating system memory allocation functions (malloc(), calloc(), free() and so on), along with callsite information showing which area of code requested the memory. Please use this information to track the memory allocated by an area of code that has not been freed, and to monitor how the size of that memory varies over time.
This option writes a summary of all allocations made or freed by the JVM, for every "N" allocations. "N" is typically a number in the range 100000 – 500000.The output of this option is written to stderr.
Please note the below important considerations for using the Xcheck option:
a) Only available on JDK release 5.0 and above
b) Performance can be badly affected if your threshold is set too low
c) Does not track allocations from outside the JVM
2. If the problem is suspected to be caused by Java Class Library (JCL) memory allocations, you can enable the following option:-Dcom.ibm.dbgmalloc=true
3. If operating system level debugging of native memory leaks is required, you can use the debugging extensions of the malloc subsystem of AIX®. For details about using AIX® debugging extensions, see the related information links.
Related Information
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21222490