Status, Gather & Array

Occasionally you may want to check on the health of your cluster, upload log files to Isilon support or execute command on all or a sub-set of nodes within your cluster.  The below covers these three common requirements.

Status

The isi staus command can be used to get an overview of the health of your Isilon cluster.  As with most commands, there are a number of options that can be applied to customise the output.

-q              [Return information on all nodes]
-D              [Displays more detailed job information]
-d              [Shows disk pool summary  -d diskpool can be used to display a single pool]
-n nodeID       [Returns detailed information from a single node]
-w              [Wide display, will not truncate at 80 characters]

Combinations of the above options can be used such as isi status -Dw to display more detailed information without truncating the output. isi stat appears to return exactly the same information as isi status - the choice is yours.

Gather

If you are experiencing a problem with your cluster, then you'll want to gather system logs and upload them to Isilon support.  Isilon have made this process very easy either via the web interface (help > diagnostics > gather info) or via the command line.

To generate and uplaod your system logs run isi_gather_info from a ssh session on your Isilon cluster.

zeus-1# isi_gather_info

Unlocking gather-status

Gather-status unlocked

This may take several minutes.  Please do not interrupt the script.

...................................................Information gathering completed....creating compressed package...........Packaging complete...

Package: /ifs/data/Isilon_Support/pkg/IsilonLogs-zeus-20121014-231033.tgz

Package uploaded.

Cleaning up temporary data... done.

Gather-status unlocked

The package is stored under /ifs/data/Isilon_Support  If the upload fails then you can manually ftp the package to ftp.isilon.com/incoming

There are a number of command line switches that can be used with isi_gather_info, I've included some of the more common ones below:

--local-only    [As the name suggest, run on the local node only]
-T              [Use temp path, useful if the node can’t mount /ifs]
--noupload      [Generate and tar / gzip the package, but don't upload]
-x              [Used to exclude modules from being collected]

Uploading the package to Isilon support can be slow if you are not based in the US or if you have a large number of quotas or have a large FSA directory.  Using the  -x ifsvar_modules fsa or -x ifsvar_modules quotas can be useful in this situation.

Any (or all) of the below modules can be excluded.

Array

With an Isilon cluster being made up of multiple nodes, you will occasionally want to run command on some or all nodes.  Isilon provides the isi_for_array command to meet this requirement.

zeus-1# isi_for_array -s "df -h /var/crash | grep crash"

zeus-1: /dev/mirror/var-crash    1.9G    6.4M    1.8G     0%    /var/crash

zeus-2: /dev/mirror/var-crash    1.9G    1.5M    1.8G     0%    /var/crash

zeus-3: /dev/mirror/var-crash    1.9G    1.9G    0k     100%    /var/crash

In the above example it is possible to see which node has a full /var/crash area.
isi_for_array has a number of options to modify the behaviour of the command, the below are some of the more common ones.

-s      [Execute in sequence on node 1 then 2 then 3, rather than in parallel]
-n 1    [Run on node 1]   

-n 1-3  [Run on nodes 1 to 3)   
-n 1,3  [Run on nodes 1 and 3]
-x 2    [Exclude node 2]

-X      [Exclude unavailable nodes]