PH12773: ADD/REMOVE NEIGHBOR LOOP WITH MESSAGES ODCF8041I, ODCF8040I, CWOBB1009W

9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21

APAR status

• Closed as program error.

Error description

• Log shows:
  ODCF8041I: Detected process ..... stopped.
  ODCF8040I: Detected process ..... started.
  CWOBB1009W: Process ..... rejoined.
  
  The rapid leave join is caused by unhandled exceptions that
  propagate all the way back up to the main P2P job queue loop
  causing it to exit, that job queue loop rooted at
  com.ibm.son.mesh.Peer.run()is suppose to run for the entire
  lifetime of the process.
  
  In this case the unhandled exception was an
  ´java.lang.IllegalArgumentException´ encountered during a
  TCP
  read
  
  Additional symptoms:
  Data fills up the receive queue for up on the UDP port for
  endpoint OVERLAY_UDP_LISTENER_ADDRESS
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server WAS ND edition                       *
  ****************************************************************
  * PROBLEM DESCRIPTION: SystemOut log files contain rapidly     *
  *                      repeating                               *
  *                      ODCF8041I/ODCF8040I/CWOBB1009W          *
  *                      messages                                *
  *                      accompanied by slow memory growth.      *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  This issue can be identified by rapidly repeating
  ODCF8041I/ODCF8040I (Detected process {0} stopped/started) and
  CWOBB1009W (Process {0} rejoined messages) in the SystemOut.log
  files accompanied by slow memory growth in maps owned by the
  com.ibm.son.mesh.Peer object. This issue is caused by
  exceptions, most predominantly
  ´java.lang.IllegalArgumentException(s)´ encountered during a
  TCP
  read, that were not properly caught and handled which were
  allowed to propagate all the way back up to the main P2P/SON,
  the Intelligent Management communication layer, Peer job queue
  loop causing it to exit prematurely.
  

Problem conclusion

• The Intelligent Management communication layer (P2P/SON) has
  been hardened to catch and properly handle all known exceptions
  as well as catching any potentially unknown exception types,
  outputting an FFDC in that case, to ensure the main P2P/SON Peer
  job queue loop can never exit prematurely while the JVM is still
  running.
  
  The fix for this APAR is currently targeted for inclusion in fix
  packs 8.5.5.16 and 9.0.5.1.  Please refer to the Recommended
  Updates page for delivery information:
  http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBSPHERE FOR Z

• Fixed component ID

  5655I3500

Applicable component levels

• R850 PSY

     UP