Troubleshooting
Problem
Your application server terminated without warning or a system core is generated but you are not sure what data to collect. The following MustGather will assist you in collecting the critical data to troubleshoot issues with an IBM® WebSphere® Application Server crash on the Solaris platform.
Symptom
System core generated, Java™ Virtual Machine process disappears or terminates
Resolving The Problem
Your application server terminated without warning or a system core is generated but you are not sure what data to collect. The following MustGather will assist you in collecting the critical data to troubleshoot issues with an IBM® WebSphere® Application Server crash on the Solaris platform.
Data to Collect
Location of dumps are usually in the profile's root directory
- Core dump (if requested)
- solaris_crash_RESULTS.tar.gz (produced from the script solaris_crash.sh below)
- hs_err_pid*.log
- AppServer Logs
- SystemOut.log
- SystemErr.log
- native_stdout.log (verbose GC data by default is in there)
- native_stderr.log
- /var/adm/messages
- server.xml
- console.log
- messages.log
- jvm.options
For WebSphere Liberty, additionally these files:
PROCESSING THE CORE DUMP
Running the script below processes the core as well as collects the OS patch level information:
./solaris_crash.sh [CORE_FILE]
The file solaris_crash_RESULTS.tar.gz will be generated in the script's current directory.
[CORE_FILE] should contain the complete path of the core file.
Make sure to locate the system core, usually named core, in the profile's root directory (it usually will have the process ID appended to the end of the filename, like core.314159). Search the system for core files if not found in the profile root directory. The core dump does not need to be submitted unless specifically requested by IBM Support.
Locate the hs_err_pid:
The hs_err_pid*.log is very critical in identifying the cause of the crash. This should be in the same location as the core file. The location of the file (and a few lines from the file) are recorded in the native_stdout.log.
Submit the data to IBM Support.
NO CORE DUMP GENERATED
If you find no core dump, and none of the logs (such as native_stdout.log) indicate one was generated, check first if the nodeagent had terminated the JVM and restarted it, as you may not get a dump if that occurs.
Another reason for this, especially if a crash was registered in the error output, is that the hard ulimits for core and file are not set to unlimited.
If there were no dumps and no signs of a crash, but the process did terminate, check this document for further debugging.
WebSphere Application Server process exits without leaving a footprint in log files or no core dumps
http://www.IBM.com/support/docview.wss?uid=swg21144595
ADDITIONAL QUESTIONS
- What are hs_err_pid* files?
These text files are produced when a core is dumped by a process. It can be opened in a text editor and explains details about the current thread and what caused the crash. This is a critical file in debugging crashes.
This is generated in the current working directory, same as the core file. The file is named hs_err_pid[PID].log, where [PID] is the process ID.
The heading of this file is also sent to the native_stdout.log.
Sample output:
#
# An unexpected error has been detected by HotSpot
# Virtual Machine:
#
# SIGSEGV (0xb) at pc=0xff33d8e0, pid=25638, tid=1
#
- How can I run the script manually?
If you cannot download/run the script provided, you can run these steps manually:
env > env.out
ulimit -a > ulimit.out
uname -a > uname.out
showrev -p > showrev.out
pkginfo -l > pkginfo.out
ls -al [CORE_FILE] > ls.out
/usr/proc/bin/pstack [CORE_FILE] > pstack_crash.out
/usr/proc/bin/pmap [CORE_FILE] > pmap_crash.out
/usr/proc/bin/pldd [CORE_FILE] > pldd_crash.out
[CORE_FILE] should contain the complete path of the core file.
Related information
Submitting information to IBM support
Steps to getting support for WebSphere Application Server
MustGather: Read first for WebSphere Application Server
Troubleshooting guide for WebSphere Application Server
To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:
- IBM Support Assistant (ISA)
- Service Request (SR)
- FTP to the Enhanced Customer Data Repository (ECuRep)
Related Information
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Crash","Platform":[{"code":"PF027","label":"Solaris"}],"Version":"9.0.0.0;8.5.5;8.5;8.0;7.0","Edition":"Base;Express;Liberty;Network Deployment","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SS7JFU","label":"WebSphere Application Server - Express"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Hangs\/performance degradation","Platform":[{"code":"PF027","label":"Solaris"}],"Version":"8.5.5;8.0;7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]