Fixes are available
8.0.0.3: WebSphere Application Server V8.0 Fix Pack 3
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
APAR status
Closed as program error.
Error description
Under high load (usually a combination of high message throughput and large messages) clients connected remotely to a Messaging Engine find that the Messaging Engine becomes unresponsive. Javacores collected from the JVM hosting the messaging engine show large number of SIBFAPInboundThreadPool threads blocked waiting on PriorityQueue$ActivatableMonitor. The waiting threads have the following stack: at com/ibm/ws/sib/jfapchannel/impl/rldispatcher/Receive ListenerDispatcher.queueInvocationCommon(ReceiveListener Dispatcher.java:571(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/rldispatcher/Receive ListenerDispatcher.queueDataReceivedInvocation(Receive ListenerDispatcher.java:667(CompiledCode)) at com/ibm/ws/sib/jfapchannel/impl/InboundTransmissionParser. dispatchToConversationListenerMethod(InboundTransmission Parser.java:1157(CompiledCode)) The blocking thread has the following stack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:167(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/PriorityQueue$Activatable Monitor.waitOn(PriorityQueue.java:119(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/PriorityQueue.queueInternal (PriorityQueue.java:347(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/PriorityQueue.queue (PriorityQueue.java:337(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/Connection.send(Connection. java:677(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.send Internal(ConversationImpl.java:1157(Compiled Code)) .......................... at com/ibm/ws/sib/jfapchannel/impl/rldispatcher/DispatchTo AllNonEmptyDispatchableImpl$DispatchToAllDispatchQueue. enqueue(DispatchToAllNonEmptyDispatchableImpl.java:308 (Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/rldispatcher/Receive ListenerDispatcher.queueInvocationCommon(ReceiveListener Dispatcher.java:571(Compiled Code)) The rest of some of the stacks will indicate that either a stop or unlockall operation is being attempted. Once the hang occurs the application server must be restarted. A poorly performing network can also cause the hang.
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: Users of the default messaging provider * * for IBM WebSphere Application Server * **************************************************************** * PROBLEM DESCRIPTION: Under high load remote clients find * * the Messaging Engine hangs. * **************************************************************** * RECOMMENDATION: * **************************************************************** The Messaging Engine processes client requests either synchronously or asyncronsously depending on the request type. Synchronous requests, including stop and unlockall, must wait for the Channel Framework to complete the sending of any in flight asynchronous responses before being able to send the synchronous request's response. If the in flight asynchronous response takes a long time (for example when sending a large message to the client) then it is possible that the Channel Framework will run out of threads to call the Service Integration Bus. All the threads from the thread pool allocated to the Channel Framework (in the SIBFAPInboundThreadPool) will become used by the synchronous requests that are all blocked waiting for the asynchronous response to complete. Once all the threads are used then the Channel Framework may be unable to finish sending the asynchronous response (because there are no more threads to continue the send). The hang ensues.
Problem conclusion
This APAR introduces two new custom properties: com.ibm.ws.sib.jfapchannel.ALWAYS_DISPATCH_STOP (WSAS 7.0.x default of false, WSAS 8.0.x and above default of true) com.ibm.ws.sib.jfapchannel.ALWAYS_DISPATCH_UNLOCKALL (WSAS 7.0.x default of false, WSAS 8.0.x and above default of true) When these properties are set to true, stop and unlockall requests are performed asynchronously. This prevents the hang. These two new properties complement the existing com.ibm.ws.sib.jfapchannel.ALWAYS_DISPATCH_CLOSE property (introduced in PM19834). When all three properties are set to true together, all client requests are processed asynchronously. If it is found that one of these properties needs to be set it is recommended that all are set. The fix for this APAR is currently targeted for inclusion in fix pack 7.0.0.23 and 8.0.0.3. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Increase the size of the SIBFAPInboundThreadPool thread pool. This does not make the hang impossible but delays the occurrence of the hang potentially long enough for the asynchronous response to complete.
Comments
APAR Information
APAR number
PM48958
Reported component name
WAS SIB & SIBWS
Reported component ID
620800101
Reported release
300
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-09-29
Closed date
2011-12-08
Last modified date
2015-08-24
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
PM60682
Fix information
Fixed component name
WAS SIB & SIBWS
Fixed component ID
620800101
Applicable component levels
R300 PSY
UP
R800 PSY
UP
Document Information
Modified date:
28 October 2021