Home AIX boot hangs with HMC 2700 LED code
Post
Cancel

AIX boot hangs with HMC 2700 LED code

We recently upgraded the firmare on our Power frame, which required shutting down some of our AIX LPAR’s. The firmware upgrade went well, as did starting up all the AIX LPAR’s, except for one. This particular LPAR booted to HMC LED code 2700 and hung there. I restarted the partition to the Open Firmware (OF) prompt, and tried booting again using verbose mode to see where the boot process was hanging.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
----------------
Attempting to configure device 'fscsi6'
 cfgmgr Time: 0        LEDS: 0x2700
Invoking /usr/lib/methods/cfgefscsi -1 -l fscsi6
Number of running methods: 4
----------------
Completed method for: fcs7, Elapsed time = 0
Return code = 0
***** stdout *****
fscsi7
*** no stderr ****
----------------
Time: 0         LEDS: 0x2700 for fscsi6
Number of running methods: 3
 cfgmgr exec(/bin/sh,-c,/usr/lib/methods/cfgefscsi -1 -l fscsi6){135238,102466}
----------------
Attempting to configure device 'fscsi7'
 cfgmgr Time: 0        LEDS: 0x2700
Invoking /usr/lib/methods/cfgefscsi -1 -l fscsi7
exec(/usr/lib/methods/cfgefscsi,-1,-l,fscsi6){135238,102466}
Number of running methods: 4
exec(/bin/sh,-c,/usr/lib/methods/cfgefscsi -1 -l fscsi7){122944,102466}
exec(/usr/lib/methods/cfgefscsi,-1,-l,fscsi7){122944,102466}
----------------
Completed method for: fscsi7, Elapsed time = 0
Return code = 0
*** no stdout ****
*** no stderr ****
----------------
Time: 0         LEDS: 0x2700 for fscsi6
Number of running methods: 3
 cfgmgr ----------------
Completed method for: fscsi5, Elapsed time = 0
Return code = 0
*** no stdout ****
*** no stderr ****
----------------
Time: 0         LEDS: 0x2700 for fscsi6
Number of running methods: 2
 cfgmgr

To verbose boot from OF, you can do the following from the HMC.

1
2
3
$ chsysstate -r lpar -m <frame> -o on -n -b of -f -p <lpar>
$ mkvterm -m <frame> -p <lpar>
0> boot -s verbose

From the output, we can see that the virtual fibre channel adapter scans for visible LUN’s presented through the adapter and runs cfgmgr against them. In this instance, the cfgmgr process is hanging, hence causing the HMC LED 2700 code. This particular LPAR has LUN’s presented to it from EMC storage through VIOS via NPIV, some of which had a state of NR (Not Ready).

Note that I’ve deliberately masked the Symmetrix and LUN ID’s.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Name: aix_lpar1
 
   Symmetrix ID       : XXXXXXXXXX
   Last updated at    : Mon Mar 18 16:20:54 2013
   Masking Views      : Yes
   FAST Policy        : Yes
 
   Devices (62):
 
    {
    ---------------------------------------------------------
    Sym                             Device               Cap
    Dev    Pdev Name                Config        Sts    (MB)
    ---------------------------------------------------------
    XXXX   N/A                      RDF1+TDEV      NR   16386
    XXXX   N/A                      RDF1+TDEV      RW   20481
    XXXX   N/A                      RDF1+TDEV      RW   51203
    XXXX   N/A                      RDF1+TDEV      RW    8633
    XXXX   N/A                      RDF1+TDEV      NR   65537
    XXXX   N/A                      RDF1+TDEV      RW   79873
    XXXX   N/A                      RDF1+TDEV      RW   79873
    XXXX   N/A                      RDF1+TDEV      NR   71681
    XXXX   N/A                      RDF1+TDEV      NR   61440
    XXXX   N/A                      RDF1+TDEV      NR  131074
    XXXX   N/A                      RDF1+TDEV      NR  153600
    XXXX   N/A                      RDF1+TDEV      NR  212993
    XXXX   N/A                      RDF1+TDEV      NR  311303
    XXXX   N/A                      RDF1+TDEV      NR   16385
    XXXX   N/A                      RDF1+TDEV      NR   16385
    XXXX   N/A                      RDF1+TDEV      NR   16385
    XXXX   N/A                      RDF1+TDEV      NR   16385
    XXXX   N/A                      RDF1+TDEV      NR   16385
    ...
    ...

The LUN’s masked to this host have a clone configured from another AIX LPAR, which at some point resulted in the target LUN’s (our HMC LED 2700 host) being placed in a NR state. Resyncing the target LUN’s from the source put them back into the correct state, and allowed the AIX LPAR to boot successfully. During investigation, a PMR was raised with IBM, which eventually came to the same conclusion that EMC LUN’s in an NR state will cause an AIX LPAR to hang during boot. The cfgmgr operation will eventually fail the LUNs, and the LPAR will boot, but in our instance, with so many LUN’s in an NR state, failures (timeouts) across all LUNs wouldn’t have happened for many hours.

Preventing this from occuring in the future would be to verify from the storage end that the LUNs are in a ready state prior to boot, or to place all those LUNs in a ‘Defined’ state in AIX before shutting down. There wasn’t much information when I looked for HMC LED 2700 hangs, so hopefully this helps someone with an AIX environment and EMC storage having the same symptoms.

This post is licensed under CC BY 4.0 by the author.

Manually fixing dirty JFS2 filesystems

Patching CVE-2014-6271 and CVE-2014-7169 on AIX via NIM (bash bug aka shellshock)