PK95297: APPLICATION SERVER STARTUP FAILURE WITH CWRLS0030W MESSAGE WHEN THE HAMANAGER GLOBAL STATE REBUILD FAILED DUE TO OUTOFMEMORY

Fixes are available

APAR status

Closed as program error.

Error description

Application Server startup failure with CWRLS0030W message when
the HAManager global state rebuild failed due to OutOfMemory.

The application server fails to open for for e-business with the
following message continuously repeated in the systemout.log:

CWRLS0030W: Waiting for HAManager to activate recovery
processing for local WebSphere server.

This could be due to HAManager current coordinator server is
experiencing OutOfMemory conditions and HAManager global state
rebuild failed.

In a large scale cell its difficult to identify the server that
is experiencing the OutOfMemory condition and stop the server
that is experiencing the OutOfMemory.

The following message is logged in coordinator server

HMGR0113E: A communication error occurred while forwarding group
change information to the active coordinator. The exception is
com.ibm.wsspi.hamanager.datastack.DataStackException:
unrecoverable exception
at
com.ibm.ws.hamanager.coordinator.impl.DCSPluginImpl.sendMessage(
DCSPluginImpl.java:493)
at
com.ibm.ws.hamanager.coordinator.impl.DCSPluginImpl.sendMessage(
DCSPluginImpl.java:440)

........

Caused by: java.lang.OutOfMemoryError
at java.lang.Class.getDeclaredFieldImpl(Native Method)
at java.lang.Class.getDeclaredField(Class.java:501)

........

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server V6.0.2, V6.1, and V7.0 Network       *
*                  Deployment editions.                        *
****************************************************************
* PROBLEM DESCRIPTION: Improve HA Manager serviceability       *
*                      with out of memory scenarios.           *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
When a Java process experiences an out of memory error it is
often the case that the JVM is partially functional.  The JVM
may be able to allocate enough memory to respond to small
messages (e.g. heartbeat)but fail to be able to send or
receive larger messages.  This condition can cause the High
Availability (HA) Manager component to think the process is OK
when in fact the process is not functioning properly.  The
code changes will introduce several new serviceability related
messages to help inform the system administrator of a problem
and point them to the failing process.

In addition to the out of memory related changes, there is
also a code change to work around the issue where HMGR0031E
messages are logged due to duplicate host name entries in the
local hosts file. More information on this issue can be found
at the following URL:
http://www.ibm.com/support/docview.wss?uid=swg21407032

Problem conclusion

The code changes will enhance the following messages by
providing information on which process(s) in the core group are
active coordinators.

HMGR0206I: The Coordinator is an Active Coordinator for core
group {0}. The active coordinator set is {1}.

HMGR0207I: The Coordinator was previously an Active
Coordinator for core group {0} but has lost leadership. The
active coordinator set is {1}.

HMGR0228I: The Coordinator is not an Active Coordinator for
core group {0}. The active coordinator set is {1}.

In addition, two new messages will be introduced to help
detect and point the system administrator to the process
experiencing a problem.

HMGR0234W: The active coordinator detected that the state
rebuild operation did not complete within the expected time.
Waiting for state messages from the following process set: {0}.

HMGR0235W: The core group member {0} reported a problem. The
member reporting the problem logged the reason using message
{1}.

The HMGR0234W message will only be logged on the active
coordinator process.
The HMGR0235W message can be logged by any process in the core
group on behalf of the troubled JVM.

An example scenario of using these new messages could be as
follows.

1.) The administrator is restarting a WebSphere cluster and
notices that one (or more) of the application servers did
not start properly. Examining one of the logs shows the
server is logging CWRLS0030W messages every minute.

2.) From any SystemOut.log file the administrator can now
determine which process is the active coordinator by
looking for HMGR0206, HMGR0207, or HMGR0228 messages.

3.) The administrator then goes to the active coordinator
process and reviews the SystemOut.log file there.  A
HMGR0234W message is repeatedly logged indicating which
process failed to send state messages.

4.) For an OutOfMemory scenario any of the core group processes
may also have logged the HMGR0235W message indicating a
process in the core group needs attention.

5.) The administrator now has some indication of which core
group process might be having trouble and they can immediately
examine the log for that process and take the correct course
of action (for example, stop or terminate the failing process).


The fix for this APAR is currently targeted for inclusion in
fix packs 6.0.2.41, 6.1.0.31, and 7.0.0.9.  Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PK95297
Reported component name
WEBSPHERE APP S
Reported component ID
5724J0800
Reported release
61I
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2009-09-02
Closed date
2009-11-11
Last modified date
2010-02-16

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PM02780

Fix information

Fixed component name
WEBSPHERE APP S
Fixed component ID
5724J0800

Applicable component levels

R60A PSY
UP
R60H PSY
UP
R60I PSY
UP
R60P PSY
UP
R60S PSY
UP
R60W PSY
UP
R60Z PSY
UP
R61A PSY
UP
R61H PSY
UP
R61I PSY
UP
R61P PSY
UP
R61S PSY
UP
R61W PSY
UP
R61Z PSY
UP
R700 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
29 December 2021

Tips

PK95297: APPLICATION SERVER STARTUP FAILURE WITH CWRLS0030W MESSAGE WHEN THE HAMANAGER GLOBAL STATE REBUILD FAILED DUE TO OUTOFMEMORY

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R60A PSY

R60H PSY

R60I PSY

R60P PSY

R60S PSY

R60W PSY

R60Z PSY

R61A PSY

R61H PSY

R61I PSY

R61P PSY

R61S PSY

R61W PSY

R61Z PSY

R700 PSY

Document Information

Share your feedback

Need support?