PK96600: APACHE SERVER FORK BOMB DUE TO APR_PROC_MUTEX_LOCK FAILING

APAR status

• Closed as program error.

Error description

• If apr_proc_mutex_lock repeatedly fails, it is possible for the
  Apache server to continually attempt to fork more child
  processes until storage is exhausted.
  The error log may contain these messages:
  EDC5121I Invalid argument.: apr_proc_mutex_lock failed.
  Attempting to shutdown process gracefully.
  EDC5121I Invalid argument.: apr_proc_mutex_unlock failed.
  Attempting to shutdown process gracefully.
  And the syslog will show the forking with these messages:
  BPXP024I BPXAS INITIATOR STARTED ON BEHALF OF JOB xxxxxxxx
  RUNNING IN ASID nnnn
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  Users whose IBM HTTP Server parent process  *
  *                  virtual memory might become corrupted       *
  ****************************************************************
  * PROBLEM DESCRIPTION: If the SysV semaphore used for          *
  *                      serializing accept() calls is           *
  *                      corrupted, IBM HTTP Server can fork     *
  *                      until memory is exhausted.              *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  If the SysV semaphore in the IBM HTTP Server parent process
  used for serializing accept() calls (accept_mutex) is
  corrupted, the worker processes detect the error and exit
  without notifying the parent process that there is a problem.
  Then the parent process will fork more worker processes to try
  to maintain MinSpareThreads.  If the accept_mutex is still
  damaged, all new worker processes also exit quietly and the
  forking will continue.
  

Problem conclusion

• If an IBM HTTP Server worker process gets an error trying to
  lock or unlock the accept_mutex and a restart isn't in
  progress, it will check to see if this worker has ever served
  any HTTP requests.  If not, it now exits with a CHILD_SICK
  error code that tells the IBM HTTP Server parent process that
  there is a problem.  If all of the worker processes exit with
  CHILD_SICK, IBM HTTP Server will terminate.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WAS IHS ZOS

• Fixed component ID

  5655I3510

Applicable component levels