Repairing File Systems with fsck in AIX (LED 517 or 518)

Answer

This document covers the use of the fsck (file system check) command in Maintenance mode to repair inconsistencies in file systems. The procedure described is useful when file system corruption in the primary root file systems is suspected or, in many cases, to correct an IPL hang at LED value 517, 518, or LED value 555.

This document applies to the AIX OS as well as PowerVM (VIO) Server.

Recovery procedure

1. Boot your system into a limited function maintenance shell (Service, or Maintenance mode) from AIX bootable media to perform file system checks on your root file systems.

   Please refer to your system user's or installation and service guide for specific IPL procedures related to your type and model of hardware. You can also refer to the document titled "Booting in Service Mode," available at Support Search for IBM Power Servers.

2. With bootable media of the same version and level as the system, boot the system. If this is a VIOS LPAR, use the correct VIOS media. The bootable media can be any ONE of the following:
   A NIM SPOT at the same AIX OS level (or IOS level for PowerVM VIO servers) may be used.
   • Bootable CD-ROM
   • NON_AUTOINSTALL mksysb
   • Bootable Install Tape
3. Follow the screen prompts to the following menu:

      Welcome to Base Operating System 
      Installation and Maintenance 
   

4. Choose Start Maintenance Mode for System Recovery (Option 3).

   The next screen displays the Maintenance menu.

5. Choose Access a Root Volume Group (Option 1).

   The next screen displays a warning that indicates you will not be able to return to the Base OS menu without rebooting.

6. Choose 0 continue.

   The next screen displays information about all volume groups on the system.

7. Select the root volume group by number.

8. Choose Access this volume group and start a shell before mounting file systems (Option 2).

   If you get errors from the preceding option, do not continue with the rest of this procedure. Correct the problem causing the error. If you need assistance correcting the problem causing the error, contact one of the following:

   If no errors occur, proceed with the following steps.

   • local branch office
   • your point of sale
   • your AIX support center
9. Run the following commands to check and repair file systems.

   NOTE: The -y option gives fsck permission to repair file system corruption when necessary. This flag can be used to avoid having to manually answer multiple confirmation prompts, however, use of this flag can cause permanent, unnecessary data loss in some situations.

    fsck /dev/hd4 
    fsck /dev/hd2 
    fsck /dev/hd3 
    fsck /dev/hd9var 
    fsck /dev/hd1 
   

10. To format the default jfslog for the rootvg Journaled File System (JFS) file systems, run the following command:

     /usr/sbin/logform /dev/hd8 
    

    Answer yes when asked if you want to destroy the log.

11. If your system is hanging at LED 517 or 518 during a Normal mode boot, it is possible the /etc/filesystems file is corrupt or missing. To temporarily replace the disk-based /etc/filesystems file, run the following commands:

     mount /dev/hd4 /mnt
     mv /mnt/etc/filesystems /mnt/etc/filesystems.[MMDDYY]
    
     cp /etc/filesystems /mnt/etc/filesystems
     umount /mnt
    

    MMDDYY represents the current two-digit representation of the Month, Day and Year, respectively.

    Important Note: The above process will create a basic filesystems file for rootvg, but it will be missing the following important stanza that you will need to add manually:

    /proc:
            dev       = /proc
            vol       = "/proc"
            mount     = true
            check     = false
            free      = false
            vfs       = procfs
    

    Be sure to add this /proc stanza when copying other filesystems entries from your previous filesystems files, or if you decide to keep the newly created one using this process.

12. Type exit to exit from the shell. The file systems should automatically mount after you type exit. If you receive error messages, reboot into a limited function maintenance shell again to attempt to address the failure causes.

13. If you have user-created file systems in the rootvg volume group, run fsck on them now. Enter:

     fsck /dev/[LVname] 
    

    LVname is the name of your user-defined logical volume.

14. If you used the preceding procedure to temporarily replace the /etc/filesystems file, and you have user-created file systems in the rootvg volume group, you must also run the following command:

     imfs -l /dev/[LVname]
    

15. If you used the preceding procedure to temporarily replace the /etc/filesystems file, also run the following command:

     imfs [VGname]
    

    The preceding commands can be repeated for each user-defined volume group on the system.

16. If your system was hanging at LED 517 or 518 and you are unable to activate non-rootvg volume groups in Service mode, you can manually edit the /etc/filesystems file and add the appropriate entries.

    The file /etc/filesystems.MMDDYY saved in the preceding steps may be used as a reference if it is readable. However, the imfs method is preferred since it uses information stored in the logical volume control block to re-populate the /etc/filesystems file.

17. If your system has a mode select key, turn it to the Normal position.

18. Reboot the system into Normal mode using the following command:

     sync;sync;sync;reboot 
    

If you followed all of the preceding steps and the system still stops at an LED 517 or 518 during a reboot in Normal mode, you may want to consider reinstalling your system from a recent backup. Isolating the cause of the hang could be excessively time-consuming and may not be cost-effective in your operating environment. To isolate the possible cause of the hang, would require a debug boot of the system. Instructions for doing this are included in the document "Capturing Boot Debug", available at Support Search for IBM Power Servers. It is still possible, in the end, that isolation of the problem may indicate a restore or reinstall of AIX is necessary to correct it.

If you need assistance correcting the problem causing the halt, contact one of the following:

• local branch office
• your point of sale
• your AIX support center