PH22672: A CICS JVM REMAINS IN DISABLING STATE AFTER CANCEL OPERATIONS

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• A JVM hangs indefinitely in state "DISABELING" after performing
  JVM cancel operations (PHASEOUT/PURGE/FORCEPURGE/KILL).
  
  
  This problem is caused by a deadlock between the action to
  disable the JVM and a user program/task that is performing
  INQUIRE JVMSERVER commands.
  
  In summary, the action to disable the JVM server was initially
  held on sj_jvmserver_stats_collected_ecb because the monitoring
  task's current INQUIRE JVMSERVER was in the process of
  collecting statistics on the JVMSERVER. Collecting statistics
  includes making a call into the JVM itself to make some JVM
  calls.
  
  What's happened here is that the INQUIRE JVMSERVER call
  finished, causing sj_jvmserver_stats_collected_ecb to be posted.
  This makes the task attempting to disable the JVM server
  dispatchable. However, the monitoring task continues on QR and
  immediately makes another INQUIRE JVMSERVER call and gets
  dispatched into the JVM on a T8/thread.
  
  The disable task then gets to run on QR and obtains the JVMSRVLK
  and starts to disable the JVM server. Because no user tasks are
  running it proceeds to deleting any open TCBs. It deletes the T8
  the statistics gather task is running on. It then makes a call
  to delete the TP TCB which results in SJ domain telling the JVM
  to terminate. Because no abends have occurred this is a graceful
  shutdown to let all non-daemon threads terminate. This task gets
  stuck here.
  
  The statistics gathering task then resumes on QR because it was
  purged when its T8 was deleted. It drives runaway, records an
  abend in the JVM, then attempts to disable the JVM server itself
  but gets stuck trying to get the JVMSRVLK. If it got the lock it
  would be able to determine the JVM was already disabling and
  leave but it can't get that far. Regardless, because the T8 was
  deleted the thread the JVM thinks is still running will never
  terminate.
  

Local fix

• Not available
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All CICS users.                              *
  ****************************************************************
  * PROBLEM DESCRIPTION: A JVM server is stuck in DISABLING      *
  *                      state.                                  *
  ****************************************************************
  A task is running repeatedly issuing INQUIRE JVMSERVER commands.
  A request comes in to disable the JVM server.
  
  The task to disable the JVM server waits for any tasks
  collecting statistics to finish before it starts the disable
  processing.
  An INQUIRE JVMSERVER call completes which allows the disable
  task to start disabling the JVM server. However, the task then
  immediately issues another INQUIRE JVMSERVER call. The disable
  task proceeds to disable the JVM server whilst holding the JVM
  server lock.
  Meanwhile, the INQURE task gets dispatched into the JVM server
  on a thread (T8 TCB). The disable task deletes all open TCBs as
  part of the disable processing, including the T8 that the
  INQUIRE call has just been dispatched on.
  The disable task then continues and attempts to terminate the
  the JVM, here it waits for all non-daemon threads to terminate,
  this task gets stuck here.
  
  The INQUIRE task continues and enters recovery because the
  thread it was running on has been deleted. It drives runaway
  and records an AICA abend. It then attempts to disable the JVM
  server, as part of this it tries to obtain the JVM server lock.
  At this point the disable task still holds the JVM server lock
  so the INQUIRE task suspends on this lock.
  
  These two tasks are now deadlocked. The disable task is waiting
  for all non-daemon threads to terminate before it terminates the
  enclave. This will never happen as it believes the thread for
  the INQUIRE task is still running.
  The INQUIRE task is waiting on the JVM server lock which is held
  by the disable task.
  
  The JVM server is now stuck in a disabling state.
  

Problem conclusion

• The code has been changed so that if another request for
  statistics has come into the system, the disable task will wait
  for this request to finish before proceeding to disable the JVM
  server.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI70699 UI70700 UI70701 UI70702 UI70703

Modules/Macros

• DFHSJJS
  

Fix information

• Fixed component name

  CICS TS Z/OS V5

• Fixed component ID

  5655Y0400

Applicable component levels

• R000 PSY UI70702

     UP20/08/03 P F007

• R100 PSY UI70701

     UP20/08/03 P F007

• R200 PSY UI70703

     UP20/08/03 P F007

• R300 PSY UI70700

     UP20/08/03 P F007

• R900 PSY UI70699

     UP20/07/30 P F007

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.