IBM Support

PI74364: AEXZ ABEND LOOP IN CICS AFTER REACHING T8 THREADLIMIT

A fix is available

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • The problem appears to be caused by the changing of the
    THREADLIMIT (due to the limit being reduced).
    
    Task XXXXX wants a system thread but has to suspend on THR_POOL
    because there are currently no free T8s.
    
    When task YYYYY finishes with TCB T8QNX it resumes task XXXXX,
    which was waiting for a T8 slot and also resumes task ZZZZZ
    which is waiting in SJTH for a thread.
    
    ZZZZZ gets going first and allocates itself T8QNX and tries to
    do a CHANGE_MODE. Task XXXXX then attempts to delete that TCB
    because it requires a new T8 TCB to use as the system thread.
    
    Task XXXXX gets stuck on the JVMSERV lock which allows ZZZZZ to
    get running on T8QNX. When ZZZZZ finally does a CHANGE_MODE off
    of the T8 task XXXXX is able to complete the deletion.
    This then causes the follow on problems (like AEXZ loop).
    
    The dump shows that if the dispatcher chooses to delete a T8 at
    exactly the same time as SJ domain allocates it to an
    application task then the reported problem can occur. CICS needs
    to prevent this situation happening.
    

Local fix

  • not available
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED: All CICS users.                              *
    ****************************************************************
    * PROBLEM DESCRIPTION: Reducing a JVM server thread limit      *
    *                      causes tasks to abend AEXZ after        *
    *                      a CHANGE_MODE fails with                *
    *                      response=DSAT_EXCEPTION                 *
    *                      and reason=DSAT_TCB_FAILED.             *
    ****************************************************************
    * RECOMMENDATION: .                                            *
    ****************************************************************
    CICS is running a JVM server. Work is running inside the JVM,
    The thread limit has been reached, and more tasks are queueing
    to enter the JVM. These waiting tasks are suspended, waiting
    for a JVMTHRED resource.
    The JVM thread limit is then reduced.
    .
    CICS deletes any T8 TCBs not currently allocated to tasks.
    All application threads are allocated to tasks so cannot be
    deleted. If a system thread T8 TCB exists it will be deleted.
    .
    An Inquire is performed against the JVMSERVER, and CICS must
    create a new system thread T8 TCB to complete this.
    CICS is already at the MAXTHRDTCBS limit imposed by the
    dispatcher, and all existing T8 TCBs are allocated to tasks.
    This request must suspend on THR_POOL to wait for a T8 to become
    available.
    .
    One of the user tasks completes its work in the JVM and
    disassociates itself from the T8 TCB it was using. The TCB is
    not placed on the dispatcher free chain of T8 TCBs, and is
    passed directly to the task waiting on THR_POOL. That task
    will delete this T8 and create a new one to use as a system
    thread. SJ domain does not know that the TCB will be deleted.
    A race condition occurs, It is possible that before the
    delete can take effect, the TCB is allocated to one of the
    waiters on JVMTHRED.
    .
    It is expected that JVMTHRED waiter being allocated to this
    TCB would fail to associate to it, but due to defects in
    how SJ and DS domain cooperate in TCB allocation it may not
    fail. This allows a TCB to be allocated to a task even though
    that TCB had already been selected for deletion.
    .
    If the JVMTHRED waiting gets running on that TCB, but has to
    switch to a different TCB (for example to the CICS QR TCB)
    then the delete may take effect and shut down that TCB while
    the task is executing elsewhere.
    This will cause a change_mode back to the T8 TCB to return a
    DSAT_EXCEPTION response, reason code DSAT_TCB_FAILED.
    The task might then begin to terminate with abendAEXZ.
    In CICS TS V5.2, a CICS region hang ensues.
    .
    Additional keywords: 0C4 S0C4 abends0C4 TCB_FAILED SOS loop
    BPXM023I JVMDUMP039I gpf DFHSJ0005 DFHSM0133 SM0133
    

Problem conclusion

  • CICS TS V5.1, V5.2, and V5.3 are modified in DFHDSDS4 to ensure
    that attempts to associate to an open TCB will fail if the
    dispatcher free open TCB chain (free_open_basespace_ds_tcbs)
    is empty.
    .
    CICS TS V5.1 and V5.2 have been modified to perform
    DELETE_OPEN_TCB instead of DELETE_TCB when removing a T8
    open TCB. This change is already present in CICS TS V5.3.
    .
    CICS TS V5,2 has been modified to retain a reusable system
    thread T8 TCB. Once created, this system thread can be
    reused for subsequent system requests. This reduces the amount
    of time spent deleting and creating T8 TCBs for system requests.
    This change is already present in CICS TS V5.3.
    .
    In CICS TS V5.2 and V5.3, the CICS dump formatter (DFHPD690 and
    DFHPD700) is updated in the SJ formatted report to list the
    system T8 TCB, if one is allocated.
    .
    CICS TS V5.2 and V5.3 are modified in the NOTIFY_DELETE_TCB
    function to resolve the race between deleting and reusing
    an open TCB (This change not required in V5.1).
    .
    The CICS Transaction Server for z/OS Version 5 Release 2
    (SC34-7295-00) and Version 5 Release 3 (SC34-7430-00)
    Trace Entries manuals are modified. In trace entry SJ 0C0C,
    the data4 item is removed (it was duplicate of data2).
    The data5 item becomes data4.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PI74364

  • Reported component name

    CICS TS Z/OS V5

  • Reported component ID

    5655Y0400

  • Reported release

    900

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    YesHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-01-05

  • Closed date

    2017-02-14

  • Last modified date

    2017-03-02

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    UI44641 UI44642 UI44643 UI44644 UI44645 UI44646 UI44647

Modules/Macros

  • DFHAMSJ  DFHAPJVM DFHAXIS2 DFHCDJNI DFHDSATT DFHDSATX DFHDSATY
    DFHDSDS4 DFHDSIT  DFHEIQSY DFHMNXM  DFHSJBD  DFHSJDM  DFHSJDS
    DFHSJDST DFHSJDUF DFHSJIN  DFHSJIS  DFHSJIT  DFHSJJS  DFHSJJST
    DFHSJL   DFHSJNT  DFHSJRE  DFHSJRL  DFHSJRM  DFHSJSC  DFHSJSM
    DFHSJST  DFHSJTH  DFHSJTRI DFHSJXM  DFHSODS  DFHSTP   DFJ@H356
    DFJ@H360 DFJ@H427 DFJ@H467 DFJ@H468 DFJDTCOE DFJOUTRE DFJWLPPL
    

Publications Referenced
SC34729500SC34743000   

Fix information

  • Fixed component name

    CICS TS Z/OS V5

  • Fixed component ID

    5655Y0400

Applicable component levels

  • R000 PSY UI44645

       UP17/02/20 P F702 ¢

  • R003 PSY UI44647

       UP17/02/21 P F702 ¢

  • R00D PSY UI44646

       UP17/02/20 P F702 ¢

  • R800 PSY UI44641

       UP17/02/20 P F702 ¢

  • R900 PSY UI44642

       UP17/02/20 P F702 ¢

  • R903 PSY UI44644

       UP17/02/20 P F702 ¢

  • R90D PSY UI44643

       UP17/02/20 P F702 ¢

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGMGV","label":"CICS Transaction Server"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.2","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.2","Edition":"","Line of Business":{"code":"","label":""}}]

Document Information

Modified date:
02 March 2017