Post

Patching CVE-2014-6271 and CVE-2014-7169 on AIX via NIM (bash bug aka shellshock)

Posted Updated
By Kristian Milos 5 min read

Below I detail how I patched over 800 AIX LPAR’s that were exposed by CVE-2014-6271 and CVE-2014-7169, also known as shellshock, using the NIM server.

From everything that I’ve been reading on IBM’s Knowledge Centre, creating an LPP source containing only RPM’s isn’t possible. To patch my AIX environment, I decided to use the script resource available to the NIM master, along with the pre-existing NFS mounts that I had configured.

  1. Configure NFS share.

    1
2
3
4
5
6

     NIM:kristian# cat /etc/exports
 /export/nim/images -ro,anon=0
    
 NIM:kristian# showmount -e
 export list for NIM:
 /export/nim/images (everyone)

  2. Download patched bash RPM to NIM master.

    1
2
3

     NIM:kristian# ls -l /export/nim/images/bash_CVE-2014-6271-7169
 total 3448
 -rw-r-----    1 root  system     1765643 Sep 30 08:22 bash-4.2-17.aix5.1.ppc.rpm

  3. Script to patch bash /export/nim/patches/install_bash_CVE-2014-6271-7169

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

     #!/bin/ksh
 #
 # Script to install new version of bash to
 # patch CVE-2014-6271 and CVE-2014-7169
 #
 # Kristian Milos (29/09/14)
    
 # Get NIM master hostname
 NIM_MASTER_HOSTNAME=`grep NIM_MASTER_HOSTNAME /etc/niminfo | awk -F = '{ print $2 }'`
    
 # Create temporary mount location
 mkdir /install_bash_CVE-2014-6271-7169
    
 # NFS mount patch
 mount ${NIM_MASTER_HOSTNAME}:/export/nim/images/bash_CVE-2014-6271-7169 /install_bash_CVE-2014-6271-7169
    
 # Install patch
 rpm -Uvh /install_bash_CVE-2014-6271-7169/bash-4.2-17.aix5.1.ppc.rpm
    
 # Unmount NFS mount
 umount /install_bash_CVE-2014-6271-7169
    
 # Remove temporary mount location
 rm -r /install_bash_CVE-2014-6271-7169
    
 exit

  4. Define NIM resource

    Now that we have the location of the RPM on the NIM master, and the script that will be run on the NIM client to patch bash, we can now define a NIM script resource.

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

     NIM:kristian# nim -o define -t script \
 -a server=master \
 -a location=/export/nim/patches/install_bash_CVE-2014-6271-7169 \
 -a comments="bash fix for CVE-2014-6271 and CVE-2014-7169" bash_CVE-2014-6271-7169
    
 NIM:kristian# lsnim -l bash_CVE-2014-6271-7169
 bash_CVE-2014-6271-7169:
 class       = resources
 type        = script
 comments    = bash fix for CVE-2014-6271 and CVE-2014-7169
 Rstate      = ready for use
 prev_state  = unavailable for use
 location    = /export/nim/patches/install_bash_CVE-2014-6271-7169
 alloc_count = 0
 server      = master

  5. Define NIM machine group

    We will now create a NIM machine group that will contain all the NIM clients that we will update. I find the easiest way to do this is by listing out all the NIM client definitions in the format required for the group define command. An example is shown below.

    1
2
3
4
5
6
7

     NIM:kristian# for i in `lsnim -t standalone | awk '{ print $1 }'`; do echo "-a add_member=$i \\"; done
 -a add_member=aix1 \
 -a add_member=aix2 \
 -a add_member=aix3 \
 -a add_member=aix4 \
 -a add_member=aix5 \
 -a add_member=aix6 \

    Define the NIM group

    1
2
3
4
5
6
7

     NIM:kristian# nim -o define -t mac_group \
 -a add_member=aix1 \
 -a add_member=aix2 \
 -a add_member=aix3 \
 -a add_member=aix4 \
 -a add_member=aix5 \
 -a add_member=aix6 PROD_LPARS

  6. Validate NIM master to client communications

    The next thing I do is validate that the NIM master can actually talk to all the NIM clients in the machine group.

    1
2
3
4
5
6
7

     NIM:kristian# for srv in `lsnim -g PROD_LPARS | grep member | grep -v EXCLUDED | awk '{ print $3 }' | sed 's/;//' | sort`; do printf "%-20s" $srv; nim -o lslpp $srv >/dev/null 2>&1; [ "$?" == 0 ] && echo OK || echo "Problem"; done
 aix1            OK
 aix2            OK
 aix3            OK
 aix4            OK
 aix5            Problem
 aix6            OK

  7. Exclude uncontactable hosts

    For any NIM client that returns “Problem”, I exclude them from the NIM group operation

    1

     NIM:kristian# nim -o select -a exclude=aix5 PROD_LPARS

  8. Patch all NIM clients via NIM master

    We’re now in a position to execute the patch across all the NIM clients listed in the group definition.

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

     NIM:kristian# nim -o cust -a script=bash_CVE-2014-6271-7169 -a concurrent=10 PROD_LPARS
 +-----------------------------------------------------------------------------+
                     Concurrency Control
 +-----------------------------------------------------------------------------+
    
 Processing will begin with the first 5 machines from the group...
    
 +-----------------------------------------------------------------------------+
                     Initiating "cust" Operation
 +-----------------------------------------------------------------------------+
 Allocating resources ...
    
 Initiating the cust operation on machine 1 of 5: aix1 ...
    
 Initiating the cust operation on machine 2 of 5: aix2 ...
    
 Initiating the cust operation on machine 3 of 5: aix3 ...
    
 Initiating the cust operation on machine 4 of 5: aix4 ...
    
 Initiating the cust operation on machine 5 of 5: aix6 ...
    
 +-----------------------------------------------------------------------------+
                     "cust" Operation Summary
 +-----------------------------------------------------------------------------+
 Target                  Result
 ------                  ------
 aix1                    INITIATED
 aix2                    INITIATED
 aix3                    INITIATED
 aix4                    INITIATED
 aix6                    INITIATED
    
 Note: Use the lsnim command to monitor progress of "INITIATED"
 targets by viewing their NIM database definition.
    
 +-----------------------------------------------------------------------------+
                     Concurrency Control
 +-----------------------------------------------------------------------------+
 The first 8 machines have been processed.  As machines finish
 installing processing will resume with the remaining members
 of the group, one at a time.
    
 +-----------------------------------------------------------------------------+
                     Concurrency Control: "cust" Operation Summary
 +-----------------------------------------------------------------------------+
 Target                  Result
 ------                  ------
 aix1                    COMPLETE
 aix2                    COMPLETE
 aix3                    COMPLETE
 aix4                    COMPLETE
 aix6                    COMPLETE

  9. Validate installed version of bash

    Once the process has completed, you can validate the version of bash installed across all NIM clients by running the following command.

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

     NIM:kristian# for i in `lsnim -g PROD_LPARS | grep member | grep -v EXCLUDED | awk '{ print $3 }' | sed 's/;//' | sort`; do echo $i; nim -o lslpp -a lslpp_flags=-Lc -a filesets=bash $i | grep bash | awk -F : '{ print $2 }'; echo ""; done
 aix1
 bash-4.2-17
    
 aix2
 bash-4.2-17
    
 aix3
 bash-4.2-17
    
 aix4
 bash-4.2-17
    
 aix6
 bash-4.2-17

  10. Include previously excluded NIM members

    One final clean up task, is to ensure you include all NIM members back into the group if you excluded them previously.

    1

    NIM:kristian# nim -o select -a include_all=yes PROD_LPARS
aix
aix nim scripting shell shell-scripting
This post is licensed under CC BY 4.0 by the author.