PK79872: DEFAULT MESSAGING PROVIDER BECAME UNRESPONSIVE FOR MANY SECONDS UNDER HEAVY LOAD, USING A FILESTORE AS A MESSAGE STORE

Fixes are available

APAR status

Closed as program error.

Error description

In a publish/subscribe scenario under heavy load, a spike in
message production occurred which could not be processed
immediately by the subscribing applications. This caused the
depth of the publication point to increase to tens of
thousands of messages.

The customer's application then became unresponsive.
Transaction timeouts were reported in the application servers
hosting applications, and the depth of the publication point
remained high. CPU utilization in the environment was limited
to a small subset of a multiple CPUs available in the hardware.

The system did not recover automatically within a time
acceptable to the customer, and a manual restart of the
environment was required to cause the messages to be consumed.

Multiple javacores were taken of the messaging engine, and
these consistently showed a thread called "CheckpointHelper"
with the following stack:
com.ibm.ws.objectManager.AbstractSingleFileObjectStore.
         mergeFreeSpace(AbstractSingleFileObjectStore.java:963)
com.ibm.ws.objectManager.AbstractSingleFileObjectStore.
     freeAllocatedSpace(AbstractSingleFileObjectStore.java:925)
com.ibm.ws.objectManager.AbstractSingleFileObjectStore.
        updateDirectory(AbstractSingleFileObjectStore.java:821)
com.ibm.ws.objectManager.SingleFileObjectStore.
                          flush(SingleFileObjectStore.java:552)
com.ibm.ws.objectManager.ObjectManagerState.
                 performCheckpoint(ObjectManagerState.java:893)
com.ibm.ws.objectManager.ObjectManagerState.
                         access$500(ObjectManagerState.java:52)
com.ibm.ws.objectManager.ObjectManagerState$
             CheckpointHelper.run(ObjectManagerState.java:3052)
at java.lang.Thread.run(Thread.java:595)

Local fix

While the work performed by the checkpoint is necessary, the
algorithm is altered so that the amount of work scales better
with the number of consumes performed within one checkpoint

Problem summary

****************************************************************
* USERS AFFECTED:  Users of the default messaging provider for *
*                  WebSphere Application Server V6.1 and V7.0  *
*                  using a filestore as a message store.       *
****************************************************************
* PROBLEM DESCRIPTION: Processing performed asynchronously     *
*                      by the filestore of a messaging         *
*                      engine is inefficient when a large      *
*                      number of messages are consumed from    *
*                      a queue or topic with a high depth.     *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Multiple javacores taken from a system experiencing this issue
will show a thread called "CheckpointHelper" in the following
method: com.ibm.ws.objectManager.AbstractSingleFileObjectStore.
freeAllocatedSpace()

The processing being performed within the freeAllocatedSpace
method occurs each time the filestore performs an activity
called a checkpoint.
Checkpoints are a necessary function of the filestore, and
occur frequently on a system under load.
Checkpoints occur asynchronously and often can occur without
halting the production or consumption of messages.

In the problem scenario, the algorithm being performed within
the freeAllocatedSpace method becomes inefficient, and can
take over a minute to complete during each checkpoint.
Because of this inefficiency, the asynchronous checkpoints
are unable to keep up with the workload of messaging being
performed, and become a bottleneck. This causes long delays to
applications, resulting in transaction timeouts.

The combination of factors that cause to the algorithm
to become inefficient are as follows:
- A period of time where message producers send significantly
  more messages than can be consumed, causing a build-up of
  messages on the destination that cannot be consumed
  immediately. Internally this means the consume of these
  messages happens in different checkpoint to the send of the
  message, and the resulting build-up of messages causes
  fragmentation of the filestore.
- A publish/subscribe scenario where each message sent
  (published) is delivered to multiple consumers (subscribers).
  This is because the cost in the algorithm scales
  inefficiently with the number of messages consumed between
  checkpoints.

Problem conclusion

The fix for this APAR changes the algorithm within the
freeAllocatedSpace() method so that the processing cost of
checkpoints scales more efficiently with the number of messages
consumed between checkpoints.

The fix for this APAR is currently targeted for inclusion in
fixpack 6.1.0.25 and 7.0.0.5.  Please refer to the Recommended
Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

```
ZE Fix Error, see PK92941. 2009/08/27
```

Comments

APAR Information

APAR number
PK79872
Reported component name
PLAT MSG COM
Reported component ID
620600101
Reported release
200
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2009-02-03
Closed date
2009-02-24
Last modified date
2009-08-27

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
PLAT MSG COM
Fixed component ID
620600101

Applicable component levels

R200 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 December 2021

Tips

PK79872: DEFAULT MESSAGING PROVIDER BECAME UNRESPONSIVE FOR MANY SECONDS UNDER HEAVY LOAD, USING A FILESTORE AS A MESSAGE STORE

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R200 PSY

Document Information

Share your feedback

Need support?