PM39049: CWSID0029E MESSAGE WHEN WSAS SERVICE INTEGRATION BUS MESSAGING ENGINE FAILSOVER AFTER A NETWORK OUTAGE

Fixes are available

APAR status

Closed as program error.

Error description

In a Websphere Application Server Service Integration Bus
Cluster Bus Member, a network outage causes the messaging
engine to start on another server even though it is still
running - a so called split brain scenario.

When the network is restored, and the split brain scenario is
resolved, the messaging engine instructed to start on the
second server is then told to stop.

After this, the SystemOut log of the server where the original
messaging engine that was always running has the following
messages:

CWSIS1535E: The messaging engine's unique id does not match
that found in the data store. ME_UUID=xxxxxxxxxxxxxxxx,
INC_UUID=yyyyyyyyyyyyyyyy, ME_UUID(DB)=xxxxxxxxxxxxxxxx,
INC_UUID(DB)=zzzzzzzzzzzzzzzz

CWSID0029E: Messaging engine XYZ suffered a common mode error.
CWSID0016I: Messaging engine XYZ is in state Failed!.

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  Users of the default messaging provider     *
*                  for IBM WebSphere Application Server        *
****************************************************************
* PROBLEM DESCRIPTION: CWSIS1535E,CWSID0029E and CWSID0029E    *
*                      messages in the SystemOut log of a      *
*                      cluster bus member server following a   *
*                      network outage.                         *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
A split brain scenario refers to a case where the HAManager's
connectivity between servers is broken, resulting in two
isolated parts of a core group believing they are the only
servers running.  In this scenario, the HAManager tries to
ensure all services are running.
For the Service Integration Bus, the HAManager will instruct
any messaging engines that need to be running to start, even
though they may already be running in another server.  The
HAManager does not know they are running in another server
because of the split brain.
Normally, the new messaging engine incarnation (INC2)
instructed to start will not be able to because the original
incarnation (INC1) is still running and  holds a lock on the
database preventing any other incarnation starting.
If, however, the network outage also caused INC1's database
connection to break, INC1 loses its lock and INC2 is then
able to start successfully and update the table to indicate
that INC2 is now the owner.
After the network is restored, HAManager realizes that there
are two messaging engine incarnations running and instructs
INC2 to stop.
When INC2 releases its lock on the database, INC1, who has
been attempting to reobtain the lock, will be able to connect
and finds that the incarnation ID has changed.  At this
point, it incorrectly declares a common mode error.
The following would be the sequence of the events that can be
observed on both the messaging engine SystemOut logs.
++++++++++++++
INC1 SystemOut
++++++++++++++
CWSIS1594I: The messaging engine, ME_UUID=ME_ID,
INC_UUID=INC1_ID, has lost the lock on the data store.
CWSIS1538I: The messaging engine, ME_UUID=<ME_ID>,
INC_UUID=<INC1_ID>, is attempting to obtain an exclusive lock
on the data store
...
CWSIS1537I: The messaging engine, ME_UUID=ME_ID,
INC_UUID=<INC1_ID>, has acquired an exclusive lock on the data
store.
CWSIS1535E: The messaging engine's unique id does not match
that found in the data store. ME_UUID=<ME_ID>,
INC_UUID=<INC1_ID> ME_UUID(DB)=<ME_ID>,
INC_UUID(DB)=<INC2_ID>
CWSIS1546I: The messaging engine, ME_UUID=<ME_ID_1>,
INC_UUID=<INC1_ID>, has lost an existing lock or failed to
gain an initial lock on the data store.
CWSID0029E: Messaging engine <BUS> suffered a common mode
error.
CWSID0016I: Messaging engine <BUS> is in state Failed!.
+++++++++++++
ME2 SystemOut
+++++++++++++
CWSID0016I: Messaging engine <BUS> is in state Starting.  --->
this would happen around the time there is a network outage.
CWSIS1537I: The messaging engine, ME_UUID=<ME_ID>,
INC_UUID=<INC2_ID>, has acquired an exclusive lock on the data
store.
CWSID0016I: Messaging engine <BUS> is in state StoppingMember.
---> this would happen soon after the network is restored.

Problem conclusion

The code has been modified to ensure that the messaging engine
declares a local error when it finds the incarnation ID has
changed.  This causes the server where the messaging engine is
running to be restarted.  This allows HAManager to start the
messaging engine on any other available server.

The fix for this APAR is currently targeted for inclusion in
fix pack 6.1.0.41, 7.0.0.21 and 8.0.0.3.  Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PM39049
Reported component name
PLAT MSG COM
Reported component ID
620600101
Reported release
200
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-05-12
Closed date
2011-07-26
Last modified date
2011-07-26

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
PLAT MSG COM
Fixed component ID
620600101

Applicable component levels

R200 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
27 October 2021

Tips

PM39049: CWSID0029E MESSAGE WHEN WSAS SERVICE INTEGRATION BUS MESSAGING ENGINE FAILSOVER AFTER A NETWORK OUTAGE

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R200 PSY

Document Information

Share your feedback

Need support?