Fixes are available
7.0.0.5: WebSphere Application Server V7.0 Fix Pack 5 for AIX
7.0.0.5: WebSphere Application Server V7.0 Fix Pack 5 for IBM i
7.0.0.5: WebSphere Application Server V7.0 Fix Pack 5 for Windows
7.0.0.5: WebSphere Application Server V7.0 Fix Pack 5 for HP-UX
7.0.0.5: Java SDK 1.6 SR5 Cumulative Fix for WebSphere Application Server
7.0.0.5: WebSphere Application Server V7.0 Fix Pack 5 for Solaris
7.0.0.5: WebSphere Application Server V7.0 Fix Pack 5 for Linux
Java SDK 1.5 SR10 Cumulative Fix for WebSphere Application Server
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for IBM i
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for AIX
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Windows
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for HP-UX
7.0.0.7: Java SDK 1.6 SR6 Cumulative Fix for WebSphere Application Server
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Solaris
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Linux
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for IBM i
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Windows
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for AIX
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for HP-UX
7.0.0.9: Java SDK 1.6 SR7 Cumulative Fix for WebSphere Application Server
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Solaris
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Linux
6.1.0.31: Java SDK 1.5 SR11 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for IBM i
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Windows
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for HP-UX
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for AIX
7.0.0.11: Java SDK 1.6 SR7 Cumulative Fix for WebSphere Application Server
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Solaris
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Linux
6.1.0.33: Java SDK 1.5 SR12 FP1 Cumulative Fix for WebSphere
6.1.0.29: Java SDK 1.5 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for AIX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for HP-UX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for IBM i
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Linux
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Solaris
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Windows
7.0.0.13: Java SDK 1.6 SR8FP1 Cumulative Fix for WebSphere Application Server
6.1.0.35: Java SDK 1.5 SR12 FP2 Cumulative Fix for WebSphere
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
6.1.0.37: Java SDK 1.5 SR12 FP3 Cumulative Fix for WebSphere
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
6.1.0.39: Java SDK 1.5 SR12 FP4 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
6.1.0.41: Java SDK 1.5 SR12 FP5 Cumulative Fix for WebSphere Application Server
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
APAR status
Closed as program error.
Error description
Threads are hanging in the Web services com.ibm.ws.webservices.engine.transport.channel.OutboundConnecti onCache class. This results in slow running and hanging transactions. . Enabling com.ibm.ws.webservices.*=all tracing will show that there are these messages occurring in the trace log. . This is the message that is logged when a WebService gets a connection from the pool. [11/17/08 9:24:33:598 PST] 00000104 OutboundConne 1 Enter: WSWS3595I: Current pool size: 25. Connections-in-use size: 2. Configured pool size: 25 . This is logged when a WebService has to wait to get a connection from the pool. [11/17/08 9:24:33:577 PST] 00000090 OutboundConne 1 WSWS3597I: Current thread: Thread[WebContainer : 40,5,main] is going to wait 300000 milliseconds for notification. This latter is really the indicator for the problem. . These are the messages logged when a WebService returns a connection to the pool. The return of the connection is what triggers the notify so that waiting threads can get a connection from the pool. [11/17/08 9:24:33:298 PST] 0000007e OutboundConne 1 WSWS3533I: Returning an outbound connection object: com.ibm.ws.webservices.engine.transport.http.HttpOutboundChannel Connection@110a110a to the cache. [11/17/08 9:24:33:298 PST] 0000007e OutboundConne 1 WSWS3598I: Notify the next waiting thread. .------------------------------------------------------------- A thread dump will show a number of threads in this state 3XMTHREADINFO "WebContainer : 66" (TID:0x3AAB7A00, sys_thread_t:0x3A4F5350, state:CW, native ID:0x0019C08F) prio=5 4XESTACKTRACE at java/lang/Object.wait(Native Method) 4XESTACKTRACE at java/lang/Object.wait(Object.java:231(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/transport/channel/OutboundConnecti onCache.findGroupAndGetConnection(OutboundConnectionCache.java:5 02(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/transport/http/HTTPSender.invoke(H TTPSender.java:521(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa ndlerWrapper.java:228(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/handlers/jaxrpc/JAXRPCHandler.invo ke(JAXRPCHandler.java:152(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa ndlerWrapper.java:228(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa ndlerWrapper.java:228(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/WebServicesEngine.invoke(WebServic esEngine.java:336(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/client/Connection.invokeEngine(Con nection.java:929(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/client/Connection.invoke(Connectio n.java:722(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/client/Connection.invoke(Connectio n.java:673(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/client/Connection.invoke(Connectio n.java:495(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/client/Stub$Invoke.invoke(Stub.jav a:947(Compiled Code)) . This is a WebService request that is waiting to be notified that a connection is available.
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of WebSphere Application Server * * using web services. * **************************************************************** * PROBLEM DESCRIPTION: A high number of web serivces requests * * for new hosts connections can result * * in degraded throughput. * **************************************************************** * RECOMMENDATION: * **************************************************************** To improve performance, the web serivces client engine maintains connection objects for some hosts after the connection is closed (soft connections). When subsequent requests for the same host arrive, the client engine can use an existing soft connection object instead creating a new connection. As a result, the client connection pool contains both "in use" connections and soft connections. If the client engine requests a connection to a new host and the pool of connections is full, the requesting thread enters a wait state. Under normal circumstances, the waiting thread is quickly awakened. Problems occur if there is a burst of connection requests for new hosts. This high load can cause a large number of threads to enter the wait state. The large number of waiting threads can degrade the overall throughput of the system. (Note that you may not see any failures, but may observe a severe degradation in throughput.) Problems occur if there is a burst of connection requests for new hosts. This high load can cause a large number of threads to enter the wait state.
Problem conclusion
The solution is to aggressively free the "soft connections" when this particular kind of high load scenario is detected. There is now a JVM property that determines how many waiting connection requests are tolerated before agressively freeing soft connections. The new JVM property is "com.ibm.websphere.webservices.http.waitingThreadsThreshold" and the default value is 5. By default, once 5 threads are waiting for connections, the client engine will aggressively remove soft connections. Note that this change will not help in the situation where all of the connections are actually in use. That situation can only be improved by increasing the maximum connection limit (and probably the maximum threads). The fix for this APAR is currently targeted for inclusion in fixpack 6.0.2.35, 6.1.0.25 and 7.0.0.5. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PK75788
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
61A
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2008-11-17
Closed date
2009-02-19
Last modified date
2009-02-19
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
R60A PSY
UP
R60H PSY
UP
R60I PSY
UP
R60P PSY
UP
R60S PSY
UP
R60W PSY
UP
R60Z PSY
UP
R61A PSY
UP
R61H PSY
UP
R61I PSY
UP
R61P PSY
UP
R61S PSY
UP
R61W PSY
UP
R61Z PSY
UP
R700 PSY
UP
Document Information
Modified date:
29 December 2021