PM73621: JAVA/LANG/OUTOFMEMORYERROR DUE TO HUGE ALLOCATION REQUEST OF AROUND 2GB OF CHAR[]

Fixes are available

APAR status

Closed as program error.

Error description

A load test of a Portal front-end application is executed.  The
application in the Portal front-end invokes EJBs over RMI/IIOP
hosted by a clustered backend WebSphere for z/OS 7.0.0.21
Application server. During the test, a back-end server
controller region experiences java/lang/OutOfMemoryError due to
huge allocation request of around 2GB of char[].
 -
In addition, several variations of java.rmi.MarshalException:
CORBA MARSHAL exceptions are reported by the application code
running in the Portal front-end servant regions.

The back-end WAS servers ASR1/ASR2 are running in 31 bit mode.
===== BACKEND =====
In the back-end controller SYSOUT we see this allocation:

<af type="tenured" id="1" timestamp="Jun 21 20:39:55 2012"
   intervalms="0.000">
  <minimum requested_bytes="2038957072"/>
  <time exclusiveaccessms="0.038" meanexclusiveaccessms="0.038"
   threads="0" lastthreadtid="0x00000008090AE700" />
  <refs soft="23882" weak="29930" phantom="31"
. .
</af>

When this occurs there is a headpdump, javacore and snap.

JVMDUMP006I Processing dump event "allocation", detail
"2038957072 bytes, type char[] - please wait.
JVMDUMP032I JVM requested Heap dump using
'/SYS/T/Websenv/home/Y/heapdump.20120940.703887.33333.0001.phd'
in response to an event.

The javacore for the controller shows call stack of the thread
hitting OutOfMemoryError:

"WebSphere:ORB.thread.pool t=00bc1cf0" J9VMThread:
   0x0000000809079700, j9thread_t:0x00000008090785A0,
   java/lang/Thread: 0x0000000837279C50, state:R, prio=5
   (native thread ID:0x1D00F500, native priority: 0x5,
   native policy:UNKNOWN)
Java callstack:
at com/ibm/rmi/iiop/CDRReader.readBytesForString(CDRReader.
   java:2296(Compiled Code))at com/ibm/rmi/iiop/CDRReader.
   readStringOrIndirection(CDRReader.java(Compiled Code))
at com/ibm/rmi/iiop/CDRReader.read_codebase_URL(CDRReader.
   java:2890)
at com/ibm/rmi/iiop/CDRReader.fast_read_value(CDRReader.
   java:1910(Compiled Code))
at com/ibm/rmi/io/IIOPInputStream.readValueType(IIOPInputStream.
   java:2242(Compiled Code))
at com/ibm/rmi/io/IIOPInputStream.readValueArray(
   IIOPInputStream.java:1252(Compiled Code))
at com/ibm/rmi/io/IIOPInputStream.inputArray(IIOPInputStream.
   java:1187(Compiled Code))
at com/ibm/rmi/io/IIOPInputStream.continueSimpleReadObject(
   IIOPInputStream.java:585(Compiled Code))
at com/ibm/rmi/io/IIOPInputStream.simpleReadObjectLoop(
   IIOPInputStream.java:537(Compiled Code))
at com/ibm/rmi/io/IIOPInputStream.simpleReadObject(
   IIOPInputStream.java:475(Compiled Code))
at com/ibm/rmi/io/ValueHandlerImpl.readValue(ValueHandlerImpl.
   java:210(Compiled Code))
at com/ibm/rmi/iiop/CDRReader.read_value
   (CDRReader.java:2028(Compiled Code))
at com/bc/comput/vdd/_VDDsomethingZZ_Stub.
   getTransactions(_VDDsomethingZZ_Stub.java:529
   (Compiled Code))
at com/bc/tunerset/vddsomethingZZportlet/util/
   vddDestiny.getRangeTransactions(vddDestiny.java:102)
at com/bc/tunerset/vddsomethingZZportlet/actions/
   vddSumDo.loadRangeTransactions(vddSumDo.java:138)
at com/bc/tunerset/vddsomethingZZportlet/actions/vddSumDo.
   doExecute(vddSumDo.java:113(Compiled Code))
at com/bc/tunerset/vddsomethingZZportlet/actions/BeginTo.
   execute(BeginTo.java:64(Compiled Code))
at com/ibm/portal/struts/portlet/WpRequestProcessor.
   processActionPerform(WpRequestProcessor.java:400
   (Compiled Code))
at org/apache/struts/action/RequestProcessor.process(
   RequestProcessor.java:274(Compiled Code))

The java.lang.OutOfMemory error occurs in method
com/ibm/rmi/iiop/CDRReader.readBytesForString while the ORB is
demarshalling the supplied parameters.  The data stream being
parsed by the ORB appears to have been corrupted, and a very
huge value is being interpreted as the current parameter's
length.

The ORB trace  from the front-end portal server that originally
marshalled this parameter shows the length being entered into
the message was a 'reasonable' value. A TCPIP Packet Trace on
this front-end portal server clearly shows that the length
value in this IIOP message has been changed to a very huge
number.  This corruption must have happened between the time
the message was built and handed to TCPIP for transmission to
the back end server, and the time TCPIP actually sent the
packets to the back-end server.

In every case, the IIOP message being sent was 300 kilobytes or
larger. This kind of damage was never noticed for messages with
length less than TCPIP's default send buffer size.  (Look in
TCPIP's TCPCONFIG file to find the setting for TCPSENDBFRSIZE.
Analysis of the corruption suggests that perhaps the "send"
buffer containing the corrupted message was concurrently being
used for a "receive" operation as well.

=====FRONT END=====
java.lang.NullPointerException
 at java.util.Date.getMillisOf(Date.java:950)
 at java.util.Date.compareTo(Date.java:970)
 ...

java.lang.NullPointerException
  at com.user.name.type.sort.stuff.compare(
    stuff.java:74)
 at org.apache.commons.collections.comparators.ComparatorChain.
    compare(ComparatorChain.java:310)
 at java.util.Arrays.mergeSort(Arrays.java:2874)
 ...

java.rmi.MarshalException: CORBA MARSHAL 0x4942f89a No; nested
exception is:
 org.omg.CORBA.MARSHAL: Unable to read value from underlying
 bridge : null  vmcid: IBM  minor code: 89A  completed: No
 at com.ibm.CORBA.iiop.UtilDelegateImpl.mapSystemException(
    UtilDelegateImpl.java:277)
 at javax.rmi.CORBA.Util.mapSystemException(Util.java:84)
 ...
Caused by: org.omg.CORBA.MARSHAL: Unable to read value from
underlying bridge : null  vmcid: IBM  minor code: 89A
completed: No
 at com.ibm.rmi.iiop.CDRReader.read_value(CDRReader.java:2046)
 ...
Caused by: java.lang.NegativeArraySizeException
 at com.ibm.rmi.iiop.CDRReader.readBytesForString(
    CDRReader.java:2296)
 at com.ibm.rmi.iiop.CDRReader.readRepIdOrIndirection(
    CDRReader.java:1019)
-or-
Caused by: com.ibm.rmi.io.FastIndirectionException:   vmcid: 0x0
 minor code: 0 completed: Maybe
-or-
Caused by: org.omg.CORBA.MARSHAL: No available data: Read beyond
end of input stream  vmcid: OMG  minor code: 8 completed: Maybe
 at com.ibm.rmi.iiop.CDRReader.availableData(CDRReader.java:905)
-or-
Caused by: org.omg.CORBA.MARSHAL: RepID indirection @ ######
vmcid: IBM  minor code: 8A3 completed: Maybe
-or-
Caused by: org.omg.CORBA.MARSHAL: No available data: Read beyond
end of input stream  vmcid: OMG  minor code: 8 completed: Maybe
 at com.ibm.rmi.iiop.CDRReader.availableData(CDRReader.java:905)
-or-
Caused by: java.lang.NegativeArraySizeException
 at com.ibm.rmi.iiop.CDRReader.readBytesForString(
    CDRReader.java:2296)
 ...
-or-
Caused by: java.io.IOException: The RepId is not of an RMI value
type (Expected RepId = IDL:omg.org/CORBA/WStringValue:1.0;
Received RepId =
RMI:org.omg.customRMI.java.util.Date:AC117E28FE36587A:686A81014B
597419)
 at com.ibm.rmi.util.RepositoryId.useFullValueDescription(
    RepositoryId.java:734)
 ...
-or-
Caused by: java.lang.StringIndexOutOfBoundsException: String
index out of range: -1
 at java.lang.String.substring(String.java:1072)
 ...
-or-
Caused by: java.lang.ArrayIndexOutOfBoundsException
 at com.ibm.rmi.iiop.CDRInputStream.read_octet_array(
    CDRInputStream.java:408)
-or-
Caused by: org.omg.CORBA.MARSHAL: Undefined type information
value in value_tag  vmcid: IBM  minor code: 899  completed: No
 at com.ibm.rmi.iiop.CDRReader.fast_read_value(
    CDRReader.java:1930)
 ...
-or-
Caused by: org.omg.CORBA.COMM_FAILURE:   vmcid: 0xc9c26000
minor code: A3B completed: Maybe
 at java.lang.J9VMInternals.newInstanceImpl(Native Method)
 ...
-or-
Caused by: org.omg.CORBA.MARSHAL: Requested stream format not
supported  vmcid: OMG  minor code: 7 completed: Maybe
 at
com.ibm.rmi.io.IIOPInputStream.setFormatVersion(IIOPInputStream.
java:312)

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server V7.0, V8.0 and V8.5                  *
****************************************************************
* PROBLEM DESCRIPTION: An outbound IIOP request sent from an   *
*                      application running in a WebSphere      *
*                      Application Server for z/OS Servant     *
*                      becomes corrupted while being sent      *
*                      over TCP/IP.                            *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
In certain situations, the storage used for sending an outbound
IIOP request can be released before TCP/IP has a chance to read
all of the data.  This means that TCP/IP may send a portion of
the outbound request, the storage may then be reused and
overwritten, and when TCP/IP reads the rest of the request the
storage no longer contains the request data.
The external symptom of this problem can vary depending on how
the target server handles reading the corrupt data.  In
general, the target server will be unable to parse the inbound
IIOP message and will respond to the client with an error
message.

Problem conclusion

Code was fixed to prevent the early release of storage.

APAR PM73621 is currently targeted for inclusion in
WebSphere Application Server Fix Packs 7.0.0.29, 8.0.0.6, and
8.5.0.2.

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PM73621
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2012-09-25
Closed date
2012-11-21
Last modified date
2013-07-03

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R700 PSY UK94926
UP13/06/20 P F306

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
29 October 2021

Tips

PM73621: JAVA/LANG/OUTOFMEMORYERROR DUE TO HUGE ALLOCATION REQUEST OF AROUND 2GB OF CHAR[]

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R700 PSY UK94926

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?