PM23482: SERVER SLOWDOWN CAUSED BY MANY THREADS STUCK WAITING FOR SOAP RESPONSE

Fixes are available

APAR status

Closed as program error.

Error description

A Portal server experienced dramatic slowdown.
All worker threads were in a wait.
39 of them had the following java stack:

Method
------
java/net/SocketInputStream.socketRead0
java/net/SocketInputStream.read
com/ibm/jsse2/a.a
com/ibm/jsse2/a.a
com/ibm/jsse2/jc.a
com/ibm/jsse2/jc.a
com/ibm/jsse2/e.read
java/io/BufferedInputStream.fill
java/io/BufferedInputStream.read
org/apache/soap/util/net/HTTPUtils.post
org/apache/soap/transport/http/SOAPHTTPConnection.send
org/apache/soap/rpc/Call.invoke
com/ibm/ws/management/connector/soap/SOAPConnectorClient$8.run
com/ibm/ws/security/util/AccessController.doPrivileged
com/ibm/ws/management/connector/soap/SOAPConnectorClient.
    invokeTemplate
com/ibm/ws/management/connector/soap/SOAPConnectorClient.invoke
com/ibm/ws/management/connector/soap/SOAPConnectorClient.invoke
    $Proxy130.invoke
com/ibm/ws/management/AdminClientImpl.invoke
com/ibm/ws/security/token/WSCredentialTokenMapper$
    GetOpaqueTokenFromMBeanAction.run
com/ibm/ws/security/auth/ContextManagerImpl.runAs
com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem
com/ibm/ws/security/token/WSCredentialTokenMapper.
    getOpaqueTokenFromMBean
...

Soap request timeout is in the range 3 to 10 minutes by default.
.
Other customers could experience the request timeout with
AbendEC3 04130007 on the servant region.
( SEC3 RSN04130007)

Local fix

If security attribute propagation is not being used, then
this particular slowdown can be avoided by disabling it.
In the adminconsole follow this path:
Secure administration, applications, and infrastructure > z/OS
security options > Web security - General settings > single
sign-on (SSO)

Uncheck the box called
"Web inbound security attribute propagation"

save and sync the configuration, then restart the server.

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server V6.1.0                               *
****************************************************************
* PROBLEM DESCRIPTION: A Portal server experienced dramatic    *
*                      slowdown.                               *
*                      All worker threads were in a wait.      *
*                      39 of them had the following java       *
*                      stack:                                  *
*                                                              *
*                      Method                                  *
*                      ------                                  *
*                      java/net/SocketInputStream.socketRead0  *
*                      java/net/SocketInputStream.read         *
*                      com/ibm/jsse2/a.a                       *
*                      com/ibm/jsse2/a.a                       *
*                      com/ibm/jsse2/jc.a                      *
*                      com/ibm/jsse2/jc.a                      *
*                      com/ibm/jsse2/e.read                    *
*                      java/io/BufferedInputStream.fill        *
*                      java/io/BufferedInputStream.read        *
*                      org/apache/soap/util/net/HTTPUtils.post *
*                      org/apache/soap/transport/http/SOAPHTTP *
*                      Connection.send                         *
*                      org/apache/soap/rpc/Call.invoke         *
*                      com/ibm/ws/management/connector/soap/SO *
*                      APConnectorClient$8.run                 *
*                      com/ibm/ws/security/util/AccessControll *
*                      er.doPrivileged                         *
*                      com/ibm/ws/management/connector/soap/SO *
*                      APConnectorClient.                      *
*                      invokeTemplate                          *
*                      com/ibm/ws/management/connector/soap/SO *
*                      APConnectorClient.invoke                *
*                      com/ibm/ws/management/connector/soap/SO *
*                      APConnectorClient.invoke                *
*                      $Proxy130.invoke                        *
*                      com/ibm/ws/management/AdminClientImpl.i *
*                      nvoke                                   *
*                      com/ibm/ws/security/token/WSCredentialT *
*                      okenMapper$                             *
*                      GetOpaqueTokenFromMBeanAction.run       *
*                      com/ibm/ws/security/auth/ContextManager *
*                      Impl.runAs                              *
*                      com/ibm/ws/security/auth/ContextManager *
*                      Impl.runAsSystem                        *
*                      com/ibm/ws/security/token/WSCredentialT *
*                      okenMapper.                             *
*                      getOpaqueTokenFromMBean                 *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
WebSphere Application Server was defaulting to the global SOAP
timeout. When Single Sign-On is enabled, and an inbound
request needs to be authenticated, WebSphere Application
Server will make an attempt to retrieve the subject from the
originating server. This connection had no way to timeout (or
be disabled) during this callback.

Problem conclusion

WebSphere Application Server will now employ two new custom
properties to handle this connection:

com.ibm.websphere.security.disableGetTokenFromMBean: this
property will disable the SOAP call.

com.ibm.websphere.security.tokenFromMBeanSoapTimeout: this
property will allow a configurable SOAP timeout. The range is
the same as the global SOAP timeout.

Both properties can be set as security custom properties by
navigating to Security - Global Security - custom
properties and clicking New.

APAR PM23482 requires changes to documentation.

NOTE: Periodically, we refresh the documentation on our
Web site, so the changes might have been made before you
read this text. To access the latest on-line
documentation, go to the product library page at:

http://www.ibm.com/software/webservers/appserv/library

The following change to the WebSphere Application Server
Version 6.1 Information Center will be made available in
April, 2011.

The following descriptions of two new security custom
properties will be added to the "Security custom
properties" topic:

com.ibm.websphere.security.disableGetTokenFromMBean

Use this property to disables the outbound SOAP call to
retrieve the subject from the originating server when
Single Sign-On is enabled.

Typically, when Single Sign-On is enabled, and an
inbound request needs to be authenticated, the receiving
server attempts to retrieve the authentication from the
originating server. The connection between the sending and
receiving servers never times out during this callback
process.

When this property is set to true, the receiving server
does not attempt to authenticate the inbound request.

Default false

com.ibm.websphere.security.tokenFromMBeanSoapTimeout

Use this property to specify the amount of time the
receiving server waits for an outbound SOAP call to
retrieve the proper authentication from the originating
server when Single Sign-On is enabled.

There is no default value for this property. If no
value is specified, the global SOAP timeout value is
used as the timeout value for the SOAP connection.

APAR PM23482 is currently targeted for inclusion in Service
Level (Fix Pack) 6.1.0.37 of WebSphere Application Server V6.1.

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PM23482
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
610
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2010-09-29
Closed date
2011-01-20
Last modified date
2011-05-02

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PM28382

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R610 PSY UK65883
UP11/04/06 P F104

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
25 October 2021

Tips

PM23482: SERVER SLOWDOWN CAUSED BY MANY THREADS STUCK WAITING FOR SOAP RESPONSE

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R610 PSY UK65883

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?