Monday, 15 October 2012

Status, Gather & Array

Occasionally you may want to check on the health of your cluster, upload log files to Isilon support or execute command on all or a sub-set of nodes within your cluster.  The below covers these three common requirements.

Status

The isi staus command can be used to get an overview of the health of your Isilon cluster.  As with most commands, there are a number of options that can be applied to customise the output.

-q              [Return information on all nodes]
-D              [Displays more detailed job information]
-d              [Shows disk pool summary  -d diskpool can be used to display a single pool]
-n nodeID       [Returns detailed information from a single node]
-w              [Wide display, will not truncate at 80 characters]


Combinations of the above options can be used such as isi status -Dw to display more detailed information without truncating the output. isi stat appears to return exactly the same information as isi status - the choice is yours.

Gather

If you are experiencing a problem with your cluster, then you'll want to gather system logs and upload them to Isilon support.  Isilon have made this process very easy either via the web interface (help > diagnostics > gather info) or via the command line.

To generate and uplaod your system logs run isi_gather_info from a ssh session on your Isilon cluster.

zeus-1# isi_gather_info
Unlocking gather-status
Gather-status unlocked

This may take several minutes.  Please do not interrupt the script.
...................................................Information gathering completed....creating compressed package...........Packaging complete...
Package: /ifs/data/Isilon_Support/pkg/IsilonLogs-zeus-20121014-231033.tgz
Package uploaded.
Cleaning up temporary data... done.
Gather-status unlocked

The package is stored under /ifs/data/Isilon_Support  If the upload fails then you can manually ftp the package to ftp.isilon.com/incoming

There are a number of command line switches that can be used with isi_gather_info, I've included some of the more common ones below:

--local-only    [As the name suggest, run on the local node only]
-T              [Use temp path, useful if the node can’t mount /ifs]
--noupload      [Generate and tar / gzip the package, but don't upload]
-x              [Used to exclude modules from being collected]


Uploading the package to Isilon support can be slow if you are not based in the US or if you have a large number of quotas or have a large FSA directory.  Using the  -x ifsvar_modules fsa or -x ifsvar_modules quotas can be useful in this situation.

Any (or all) of the below modules can be excluded.

Array

With an Isilon cluster being made up of multiple nodes, you will occasionally want to run command on some or all nodes.  Isilon provides the isi_for_array command to meet this requirement.

zeus-1# isi_for_array -s "df -h /var/crash | grep crash"
zeus-1: /dev/mirror/var-crash    1.9G    6.4M    1.8G     0%    /var/crash
zeus-2: /dev/mirror/var-crash    1.9G    1.5M    1.8G     0%    /var/crash
zeus-3: /dev/mirror/var-crash    1.9G    1.9G    0k     100%    /var/crash

In the above example it is possible to see which node has a full /var/crash area.
isi_for_array has a number of options to modify the behaviour of the command, the below are some of the more common ones.

-s      [Execute in sequence on node 1 then 2 then 3, rather than in parallel]
-n 1    [Run on node 1]   
-n 1-3  [Run on nodes 1 to 3)   
-n 1,3  [Run on nodes 1 and 3]
-x 2    [Exclude node 2]
-X      [Exclude unavailable nodes]


Friday, 12 October 2012

Joining up the dots

OneFS

Isilon like most storage companies runs a custom LINUX OS, in this instance a derivative of FreeBSD which they call OneFS.  Unlike many other companies, Isilon don't hide their LINUX lineage or hide commands.  Chances are anything you can do from the command line on your favourite LINUX box, you can do on an Isilon node...that does not mean that you should!

Isilon maintains a number of supported product variants.  As you can see below, Isilon has five product lines that they're currently maintaining.

 

GA (General Availability)

6.5.5.x
6.5.4.x
6.0.x
5.5.x
5.0.x

Quite why they maintain both 6.5.4.x and 6.5.5.x I couldn't say.

As well as the general availability, Isilon also has early access builds (may be production-ready, but test well) and beta version of OneFS.

 

EA (Early Access)

6.5.5.x

 

Beta (Only for the brave)

7.0 Beta 3

[Note:  Previously major Isilon release were named after verities of chillis (or Chilli Peppers for our American cousins).  Alas the most recent release have had names such as Whirled Peas (word peace) and Mavericks...Mavericks is OK but doesn't have quite the same kick as Dorset Naga :-) ]

As a general rule of thumb, you always want to be within a couple of dot releases of the latest GA build.

Recently moved to a monthly dot release schedule, which can make keeping up with the latest GA quite a challenge - thankfully rolling upgrade between minor releases is possible.


Which Dot Am I?

Finding out which version of OneFS you are running is very straightforward.  If you launch the wweb interface then the version of OneFS is displayed.  Virtually every page within the Isilon web interface also displays the current version.



For those that like to use the command like the command line, the version is displayed after you ssh onto a node

Copyright (c) 2001-2010 Isilon Systems, Inc.  All rights reserved.
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.

Copyright (c) 2001-2010 Isilon Systems, Inc.  All rights reserved. Isilon and
Isilon Systems are registered trademarks of Isilon Systems, Inc.

Isilon OneFS v6.0.3.10
zeus-1# 



No Patches?

As mentioned above, Isilon has recently moved to a monthly minor release schedule, one of the reasons for this is to try and avoid the need to issue individual patch files.  Occasionally a patch or two may come out, but these can be easily installed via the command line.

Install Package
zeus-1# isi pkg install patch-71234.tar
List Packages
zeus-1# isi pkg list
Uninstall Package
zeus-1#  isi pkg delete patch-71234

Patches normally come with a readme file which will advise if the cluster needs to be restarted or a script executed once the patch has been installed.


In a later post, I'll comment on how to check the OS version installed across multiple nodes.

Until next time.

Thursday, 11 October 2012

Why IsI Blogging?

For the past six years or so I’ve been leading an effort to migrate my companies storage infrastructure to Isilon scale-out storage.

During that timeframe the number of Isilon customer in both the UK and globally has been steadily increasing, but there still remain relatively few technical articles on the day-to-day operational use of Isilon storage…hence this blog.

The Isilon web interface is surprisingly good when compared to other top-table storage vendors (well at least from OneFS 5.5 onwards) so most of my posts will relate to what can be done from the command line.

Until next time…