PK64775: WEBSPHERE APPLICATION SERVER RUNS OUTOFMEMORY IN NATIVE HEAP

Fixes are available

APAR status

Closed as program error.

Error description

WebSphere Application server, the control OR servant region
runs outofmemory in native LE heap.

You will also see the below residual errors in the control
region joblog

 JVMSHRC021E An unknown error code has been returned
 JVMSHRC017E Error code: -167
 JVMSHRC171E z/OS cannot create cache of requested size: Please
check your z/OS system BPXPRMxx settings
 JVMDUMP006I Processing Dump Event "systhrow", detail
"java/lang/OutOfMemoryError" - Please Wait.
 JVMDUMP006I Processing Dump Event "systhrow", detail
"java/lang/OutOfMemoryError" - Please Wait.
..
..
UTE430: can't allocate buffer
UTE437: Unable to load formatStrings for j9mm
UTE430: can't allocate buffer
..
..
Trace: 2008/04/14 23:36:32.765 01 t=9C6988 c=UNK key=S2
(13007002)
   ThreadId: 0000003c
   FunctionName:com.ibm.ws.dcs.vri.transportAdapter.rmmImpl.
ptpDiscovery.DiscoveryServerMgr
   SourceId:com.ibm.ws.dcs.vri.transportAdapter.rmmImpl.
ptpDiscovery.DiscoveryServerMgr
   Category: WARNING
   ExtendedMessage: BBOO0221W: DCSV0002W: DCS Stack
DefaultCoreGroup at Member WS\NA1A\WSIS5A: An unexpected
internal programming state has occurred. Internal details:
"onRcvEvent got exception"{Method=DiscoveryRcv.onEvent(),
DiscoveryState=null} . The exception
 is java.lang.OutOfMemoryError: Failed to fork OS thread
  at java.lang.Thread.startImpl(Native Method)
  at java.lang.Thread.start(Thread.java:979)

In this case, when checked the heap we found that the 70k
elements were growing.

Allocated element sizes and frequencies.

ElementSize SizeInHex ElementCnt VirtualMeg

      70664  00011408       4207      283.5 ****
  268439568  10001010          1      256.0
   36909608  02333228          1       35.2
     131080  00020008        273       34.1
    8388616  00800008          3       24.0

When browsed the storage addreses, they were all filled
with zeros like below.

4EA9A9D0   4E7B2000   00011408   00000000   00000000   | N{
...........
4EA9A9E0.:4EA9AFFF.--All bytes contain X'00'

In an other case, we saw the servant go native OOM with the
below symptoms.

LOCAL SUBPOOL USAGE SUMMARY
Allocated element sizes and frequencies.
ElementSize SizeInHex ElementCnt VirtualMeg
      69720  00011058      11009      732.0  ***TOO MANY**
  536875024  20001010          1      512.0
     131080  00020008        305       38.1

Eyecathers for addresses with 69720 sizes.

1FC466E8                         1AC62000   00011058   |
.F....
1FC466F0.:1FC46FFF.--All bytes contain X'00'
1FC47000   C2C2D6D4   C5E3C100   00000002   80000001   |
BBOMETA.......


 43946798                         438F6000   00011058   |
..-..
439467A0.:43946FFF.--All bytes contain X'00'
43947000   C2C2D6D4   C5E3C100   00000002   80000000   |
BBOMETA.......

From the dump, there was only 2MB left in the servant. The TCB
from the CEEDUMP in the servant region joblog showed
bytebufferpoolmgmt call stack.

Callstack of the TCB that triggered the CEEDUMP
Java stack:
-----------
Method
------
sun/misc/Unsafe.allocateMemory
java/nio/DirectByteBuffer.<init>
java/nio/ByteBuffer.allocateDirect
com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateB
com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateC
com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateD
com/ibm/ws390/channel/xmem/XMemReadRequestContext.read
com/ibm/ws/http/channel/impl/HttpServiceContextImpl.fillABuffer
com/ibm/ws/http/channel/impl/HttpServiceContextImpl.readSingleBl
com/ibm/ws/http/channel/impl/HttpServiceContextImpl.readBodyBuff
com/ibm/ws/http/channel/inbound/impl/HttpInboundServiceContextIm
com/ibm/ws/webcontainer/channel/WCCByteBufferInputStream.bufferI
com/ibm/ws/webcontainer/channel/WCCByteBufferInputStream.read
com/ibm/ws/webcontainer/srt/http/HttpInputStream.read
org/apache/commons/fileupload/MultipartStream$ItemInputStream.ma
org/apache/commons/fileupload/MultipartStream$ItemInputStream.re
com/xyz/service/fileupload/server/util/MultipartHttpServletReque
com/xyz/service/fileupload/server/util/MultipartHttpServletReque
com/xyz/service/fileupload/server/util/RequestEventBuilder.getPr
com/xyz/service/fileupload/server/util/RequestEventBuilder.build
com/xyz/service/fileupload/server/servlet/FileServiceServlet.exe
com/xyz/service/fileupload/server/servlet/FileServiceServlet.doP
javax/servlet/http/HttpServlet.service
javax/servlet/http/HttpServlet.service
com/ibm/ws/webcontainer/servlet/ServletWrapper.service
com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest
com/ibm/ws/wswebcontainer/servlet/ServletWrapper.handleRequest
com/ibm/ws/webcontainer/webapp/WebApp.handleRequest
com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest
com/ibm/ws/webcontainer/WebContainer.handleRequest
com/ibm/ws/wswebcontainer/WebContainer.handleRequest
com/ibm/ws/webcontainer/channel/WCChannelLink.ready
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscr
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewIn
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready
com/ibm/ws390/channel/xmem/XMemConnLink.ready
com/ibm/ws390/xmem/XMemSRBridgeImpl.httpinvoke
com/ibm/ws390/xmem/XMemSRCppUtilities.httpinvoke
com/ibm/ws390/orb/ServerRegionBridge.httpinvoke
com/ibm/ws390/orb/ORBEJSBridge.httpinvoke
sun/reflect/NativeMethodAccessorImpl.invoke0
sun/reflect/NativeMethodAccessorImpl.invoke
sun/reflect/DelegatingMethodAccessorImpl.invoke
java/lang/reflect/Method.invoke
com/ibm/ws390/orb/parameters/HTTPInvoke.HTTPInvokeParmSetter
com/ibm/ws390/orb/CommonBridge.nativeRunApplicationThread
com/ibm/ws390/orb/CommonBridge.runApplicationThread
com/ibm/ws/util/ThreadPool$ZOSWorker.run

Local fix

Follow this procedure to activate the new direct ByteBuffer
manager in your cell.  Note that the cell will need to be
recycled after updating the process definitions.

 1. In the administrative console, click Application servers
    > server > Java and Process Management > Process
    Definition.

 2. Select Control, and then click Java Virtual Machine.

 3. Enter the following the following value in the Generic
    JVM arguments field.

 -Dcom.ibm.ws.buffermgmt.impl.WsByteBufferPoolManagerImpl=
   com.ibm.ws.buffermgmt.impl.ZOSWsByteBufferPoolManagerImpl

 4. Repeat steps 2 and 3 selecting Servant instead of Control
    region.

 5. Repeat steps 2 and 3 selecting Adjunct if one exists for
    the server.

 6. Repeat steps 2 and 3 for the deployment manager and node
    agents.

 7. The new direct ByteBuffer manager will be activated in each
    WebSphere server after it has been recycled.

Problem summary

****************************************************************
* USERS AFFECTED:  All users of WebSphere Application Server   *
*                  V6.1 for z/OS                               *
****************************************************************
* PROBLEM DESCRIPTION: Memory management support previously    *
*                      delivered in PK36998 is not enabled by  *
*                      default.                                *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
PK36998 is a previously delivered APAR which addressed an out
of memory condition.  It was originally thought that this
problem was limited to very large HTTP message workloads.
However, this problem is now being seen in much more common
workloads.

Problem conclusion

The memory management support delivered in PK36998 will now be
enabled by default. This message will now be seen in the server
joblog during initialization indicating the support is active:

BBOO0222I: ZOSWsByteBufferPoolManagerImpl initialized with
config parms

APAR PK64775 requires changes to documentation.

NOTE: Periodically, we refresh the documentation on our
Web site, so the changes might have been made before you
read this text. To access the latest on-line
documentation, go to the product library page at:

http://www.ibm.com/software/webservers/appserv/library

The following changes to the z/OS version of the WebSphere
Application Server Version 6.1.x Information Center will be
made available in September, 2008.

Step 9 in the topic "Fine tuning the LE heap" states that
if your Language Environment (LE) storage is being exhausted
before the Java Virtual Machine (JVM) runs a garbage
collection cycle, you can use a JVM generic argument to
indicate that storage for individual direct byte buffers
should be released as soon as the buffer is no longer needed.
Starting with Service Level 6.1.0.21, storage for individual
direct byte buffers will automatically be released as soon
as the buffer is no longer needed. Therefore, Step 9 will
be deleted because this step will no longer need to be
performed.

APAR PK64775 is currently targeted for inclusion in Service
Level (Fix Pack) 6.1.0.21 of WebSphere Application Server V6.1
for z/OS.

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PK64775
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
610
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt
Submitted date
2008-04-21
Closed date
2008-08-11
Last modified date
2009-02-06

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PK75954

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R610 PSY UK41575
UP08/11/26 P F811

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 December 2021

Tips

PK64775: WEBSPHERE APPLICATION SERVER RUNS OUTOFMEMORY IN NATIVE HEAP

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R610 PSY UK41575

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?