IC67387: TIVOLI STORAGE MANAGER SERVER RUNS OUT OF ARCHIVE LOG SPACE.

IBM Tivoli Storage Manager V6.1 Fix Pack 4 (6.1.4) Server Downloads

APAR status

• Closed as program error.

Error description

• A Tivoli Storage Manager server V6.1 or V6.2 may run out of
  space in the Archive Log. When the server runs out of log space,
  the activity log will cease to record messages and the server
  will appear to be unresponsive. This is more likely to happen on
  a server under heavy workload, such as simultaneous client
  backups and table reorg.
  
  Customer/L2 Diagnostics:
  The server activity log will stop recording messages.
  For example :
  
  ANR0293I Reorganization for table Archive.Objects started.
  ANR0403I Session xxx ended for node ABC (Linux86).
  ANR0136I Table updating statistics performed successfully
   for 5 of 5.
  ANR0000I Resuming table reorg after pause for Archive.Objects.
  <no more activity log entries>
  
  The server console will show the following errors :
  
  ANR0171I dbieval.c(824): Error detected on 22:12, database in
   evaluation mode.
  ANR0175E dbieval.c(917): Space exhausted for one or more of the
   following:  DB, LOG, or instance directory.
  ANR0162W Supplemental database diagnostic information:
   -1:57011:-968 ([IBM][CLI Driver][DB2/LINUXX8664] SQL0968C
   The file system is full. SQLSTATE=57011).
  
  The db2diag.log will show errors indicating that an archive of
  a log fails. For example :
  
  2010-03-18-03.11.42.063949-180 E15772851E808   LEVEL: Error (OS)
  PID     : 31170            TID  : 47757867870528PROC : db2sysc 0
  INSTANCE: tsminst1             NODE : 000
  EDUID   : 18                   EDUNAME: db2logmgr (TSMDB1) 0
  FUNCTION: DB2 UDB, oper system services, sqlowrite, probe:70
  MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
            DIA8312C Disk was full.
  CALLED  : OS, -, write                        OSERR: ENOSPC (28)
  DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes
  0x00002B6F7EBFCE18 : 4C00 0000 8004 0000                L.......
  DATA #2 : unsigned integer, 8 bytes
  131072
  DATA #3 : signed integer, 8 bytes
  53248
  DATA #4 : String, 105 bytes
  Search for ossError*Analysis probe point after this log entry
  for further self-diagnosis of this problem.
  
  2010-03-18-03.11.43.874946-180 E15774595E558       LEVEL: Error
  PID     : 31170            TID  : 47757867870528PROC : db2sysc 0
  INSTANCE: tsminst1             NODE : 000
  EDUID   : 18                   EDUNAME: db2logmgr (TSMDB1) 0
  FUNCTION: DB2 UDB, data protection services,
            sqlpgArchiveLogFile, probe:3160
  MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
            DIA8312C Disk was full.
  DATA #1 : <preformatted>
  Failed to archive log file S0001953.LOG to
   /tsm/archlog/archmeth1/tsminst1/TSMDB1/NODE0000/C0000003/
   from /tsm/log/NODE0000/.
  
  
  Starting with Tivoli Storage Manager, V6.1.2, the Tivoli Storage
  Manager automatic DB backup are enabled. This is documented
  in the following wiki article :
  
  http://www.ibm.com/developerworks/wikis/display/tivolistorageman
  ager/Tivoli+Storage+Manager+V6.1.2+automatic+DB+backup
  
  The automatic db backup feature triggers automatic db backups
  based on the active log consumption/utilization. The algorithm
  for automatic db backup does not take into account the archive
  log, only the active log. As such, the server may run out of
  archive log space.
  
  TSM Versions Affected:
  TSM Servers V6.1 and V6.2 on all platforms.
  
  Additional Keywords:
  zz61 archlog arch archivelog log diskfull
  

Local fix

• 1. Run regular database backups to clear archive log.
  2. Add more space to the archive log location.
  3. Disable table reorg using the "ALLOWREORGTABLE NO" server
     option. Restart server after adding option.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: See error description.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be fixed   *
  *                 in level  6.1.4.0 and 6.2.2.0. Note that     *
  *                 this is                                      *
  *                 subject to change at the discretion of IBM.  *
  ****************************************************************
  *
  

Problem conclusion

• This problem was fixed.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP