A fix is available
APAR status
Closed as program error.
Error description
The problem appears to be caused by the changing of the THREADLIMIT (due to the limit being reduced). Task XXXXX wants a system thread but has to suspend on THR_POOL because there are currently no free T8s. When task YYYYY finishes with TCB T8QNX it resumes task XXXXX, which was waiting for a T8 slot and also resumes task ZZZZZ which is waiting in SJTH for a thread. ZZZZZ gets going first and allocates itself T8QNX and tries to do a CHANGE_MODE. Task XXXXX then attempts to delete that TCB because it requires a new T8 TCB to use as the system thread. Task XXXXX gets stuck on the JVMSERV lock which allows ZZZZZ to get running on T8QNX. When ZZZZZ finally does a CHANGE_MODE off of the T8 task XXXXX is able to complete the deletion. This then causes the follow on problems (like AEXZ loop). The dump shows that if the dispatcher chooses to delete a T8 at exactly the same time as SJ domain allocates it to an application task then the reported problem can occur. CICS needs to prevent this situation happening.
Local fix
not available
Problem summary
**************************************************************** * USERS AFFECTED: All CICS users. * **************************************************************** * PROBLEM DESCRIPTION: Reducing a JVM server thread limit * * causes tasks to abend AEXZ after * * a CHANGE_MODE fails with * * response=DSAT_EXCEPTION * * and reason=DSAT_TCB_FAILED. * **************************************************************** * RECOMMENDATION: . * **************************************************************** CICS is running a JVM server. Work is running inside the JVM, The thread limit has been reached, and more tasks are queueing to enter the JVM. These waiting tasks are suspended, waiting for a JVMTHRED resource. The JVM thread limit is then reduced. . CICS deletes any T8 TCBs not currently allocated to tasks. All application threads are allocated to tasks so cannot be deleted. If a system thread T8 TCB exists it will be deleted. . An Inquire is performed against the JVMSERVER, and CICS must create a new system thread T8 TCB to complete this. CICS is already at the MAXTHRDTCBS limit imposed by the dispatcher, and all existing T8 TCBs are allocated to tasks. This request must suspend on THR_POOL to wait for a T8 to become available. . One of the user tasks completes its work in the JVM and disassociates itself from the T8 TCB it was using. The TCB is not placed on the dispatcher free chain of T8 TCBs, and is passed directly to the task waiting on THR_POOL. That task will delete this T8 and create a new one to use as a system thread. SJ domain does not know that the TCB will be deleted. A race condition occurs, It is possible that before the delete can take effect, the TCB is allocated to one of the waiters on JVMTHRED. . It is expected that JVMTHRED waiter being allocated to this TCB would fail to associate to it, but due to defects in how SJ and DS domain cooperate in TCB allocation it may not fail. This allows a TCB to be allocated to a task even though that TCB had already been selected for deletion. . If the JVMTHRED waiting gets running on that TCB, but has to switch to a different TCB (for example to the CICS QR TCB) then the delete may take effect and shut down that TCB while the task is executing elsewhere. This will cause a change_mode back to the T8 TCB to return a DSAT_EXCEPTION response, reason code DSAT_TCB_FAILED. The task might then begin to terminate with abendAEXZ. In CICS TS V5.2, a CICS region hang ensues. . Additional keywords: 0C4 S0C4 abends0C4 TCB_FAILED SOS loop BPXM023I JVMDUMP039I gpf DFHSJ0005 DFHSM0133 SM0133
Problem conclusion
CICS TS V5.1, V5.2, and V5.3 are modified in DFHDSDS4 to ensure that attempts to associate to an open TCB will fail if the dispatcher free open TCB chain (free_open_basespace_ds_tcbs) is empty. . CICS TS V5.1 and V5.2 have been modified to perform DELETE_OPEN_TCB instead of DELETE_TCB when removing a T8 open TCB. This change is already present in CICS TS V5.3. . CICS TS V5,2 has been modified to retain a reusable system thread T8 TCB. Once created, this system thread can be reused for subsequent system requests. This reduces the amount of time spent deleting and creating T8 TCBs for system requests. This change is already present in CICS TS V5.3. . In CICS TS V5.2 and V5.3, the CICS dump formatter (DFHPD690 and DFHPD700) is updated in the SJ formatted report to list the system T8 TCB, if one is allocated. . CICS TS V5.2 and V5.3 are modified in the NOTIFY_DELETE_TCB function to resolve the race between deleting and reusing an open TCB (This change not required in V5.1). . The CICS Transaction Server for z/OS Version 5 Release 2 (SC34-7295-00) and Version 5 Release 3 (SC34-7430-00) Trace Entries manuals are modified. In trace entry SJ 0C0C, the data4 item is removed (it was duplicate of data2). The data5 item becomes data4.
Temporary fix
Comments
APAR Information
APAR number
PI74364
Reported component name
CICS TS Z/OS V5
Reported component ID
5655Y0400
Reported release
900
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2017-01-05
Closed date
2017-02-14
Last modified date
2017-03-02
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
UI44641 UI44642 UI44643 UI44644 UI44645 UI44646 UI44647
Modules/Macros
DFHAMSJ DFHAPJVM DFHAXIS2 DFHCDJNI DFHDSATT DFHDSATX DFHDSATY DFHDSDS4 DFHDSIT DFHEIQSY DFHMNXM DFHSJBD DFHSJDM DFHSJDS DFHSJDST DFHSJDUF DFHSJIN DFHSJIS DFHSJIT DFHSJJS DFHSJJST DFHSJL DFHSJNT DFHSJRE DFHSJRL DFHSJRM DFHSJSC DFHSJSM DFHSJST DFHSJTH DFHSJTRI DFHSJXM DFHSODS DFHSTP DFJ@H356 DFJ@H360 DFJ@H427 DFJ@H467 DFJ@H468 DFJDTCOE DFJOUTRE DFJWLPPL
SC34729500 | SC34743000 |
Fix information
Fixed component name
CICS TS Z/OS V5
Fixed component ID
5655Y0400
Applicable component levels
R000 PSY UI44645
UP17/02/20 P F702 ¢
R003 PSY UI44647
UP17/02/21 P F702 ¢
R00D PSY UI44646
UP17/02/20 P F702 ¢
R800 PSY UI44641
UP17/02/20 P F702 ¢
R900 PSY UI44642
UP17/02/20 P F702 ¢
R903 PSY UI44644
UP17/02/20 P F702 ¢
R90D PSY UI44643
UP17/02/20 P F702 ¢
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGMGV","label":"CICS Transaction Server"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.2","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.2","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
02 March 2017