S.M.A.R.T : Using smartctl

Production systems running critical applications has a high requirement to be UP all the time . But there are times when the system suddenly crashes loosing critical data . If it is a disk failure , then we have to reinstall all applications and do the necessary configurations to make the system running again. We may also loose some amount of critical data , even if proper backup is in place. It is always good if we get some kind of alerting that the disk is going to fail in near future . In that case , we can have a scheduled downtime , intimating consumers about the scheduled down time . That will help us to come out of this catastrophe with a minimal impact. SMART monitoring tools is all about this. Let's now have a brief overview of SMART and also how to use SMART tools


What is S.M.A.R.T

S.M.A.R.T stands for Self monitoring , analysis and reporting technology. It is  the industry-standard reliability prediction indicator for both IDE/ATA and SCSI hard disk drives. The purpose of SMART is to monitor the reliability of the hard drive and predict drive failures, and to carry out different types of drive self-tests. S.M.A.R.T enabled hard drives maintains  a set of attributes, and sets threshold values beyond which attributes should not pass under normal operation . Each attribute has a raw value, whose meaning is entirely up to the drive manufacturer, and a normalized value, which ranges from 1 to 253. Depending on the manufacturer, a value of 100 or 200 will often be chosen as the "normal" value. smartmontools is a free software package that can monitor S.M.A.R.T. attributes and run hard drive self-tests.

S.M.A.R.T Monitoring tools

Smartmontools was originally derived from the Linux but runs on any modern Darwin (Mac OS X), Linux, FreeBSD, NetBSD, OpenBSD, Solaris or Windows hosts. It  supports ATA/ATAPI/SATA and
SCSI disks.The smartmontools package contains two utility programs (smartctl and smartd) to control and monitor S.M.A.R.T build storage systems.
smartd  is  a  daemon that monitors the S.M.A.R.T system built into the  hard drives.  The main configuration file for smartd is /etc/smartd.conf . smartd polls the S.M.A.R.T enabled hard drives every 30 mins and logs the the  S.M.A.R.T related errors  into  /var/log/messages .
smartctl is a  command line control and monitor utility for S.M.A.R.T enabled disk drives .The syntax of the command is :
                   smartctl   [options]   device
  Some of the options are:

  -i: Prints  the device model number, serial number, firmware version, and ATA Standard
       version/revision information.
 -a: Prints all SMART information about the disk.

 -d TYPE : Specifies the type of the device.The valid arguments can be ata, scsi, sat, etc. If this option
                   is  not used then smartctl will attempt to guess the device type from the device name.
  -s VALUE : Enables or disables SMART on device. Value can be on or off.
  -H :  Displays the SMART health status . If  the  device reports failing health status, this means either
          that the device has already failed, or that it is predicting its own failure within the next 24 hours.
          If this happens, use the ´-a´ option to get more information, and get your data off the disk and to
          someplace safe as soon as you can.
   -A : Prints only the vendor specific SMART Attributes.

smartctl with examples:

Printing the Device metadata information:

-bash-3.2$ sudo smartctl -i /dev/ad0
Password:
smartctl version 5.36 [x86_64-unknown-freebsd6.1] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     GB0500EAFYL
Serial Number:    WCASY7342065
Firmware Version: HPG1
User Capacity:    500,107,862,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 4a
Local Time is:    Thu Feb 11 01:26:16 2010 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


Printing the vendor specific SMART Attributes:

-bash-3.2$ sudo smartctl -A /dev/ad0
smartctl version 5.36 [x86_64-unknown-freebsd6.1] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   238   238   021    Pre-fail  Always       -       1058
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       24
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002f   200   200   051    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       1360
 10 Spin_Retry_Count        0x0033   100   253   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0033   100   253   051    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       23
184 Unknown_Attribute       0x0033   100   100   097    Pre-fail  Always       -       0
187 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
188 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
190 Unknown_Attribute       0x0022   078   068   045    Old_age   Always       -       22
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       22
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       24
194 Temperature_Celsius     0x0022   125   115   000    Old_age   Always       -       22
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

 Knowing the health of a disk drive:

 -bash-3.2$ sudo smartctl -H /dev/ad0
smartctl version 5.36 [x86_64-unknown-freebsd6.1] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


Smart Health check on a disk drive which is failing.


-bash-3.2$ sudo smartctl -H /dev/ad0
smartctl version 5.36 [x86_64-unknown-freebsd6.1] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
*Drive failure expected in less than 24 hours.* SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   001   001   036    Pre-fail  Always   FAILING_NOW 2059
196 Reallocated_Event_Count 0x0033   001   001   036    Pre-fail  Always   FAILING_NOW 2059
 
Querying a hard drive behind a RAID controller. 
 
We can use the -d option to reflect the correct device type . The valid arguments to this option are :  ata,scsi,cciss,N, and 3ware,N where N is the device number.  
In our case , the RAID controller is HP's smartarray RAID controllers, so the device type will be cciss .

e.g sudo /usr/sbin/smartctl -d cciss,0 -H /dev/cciss/c0d0
smartctl version 5.33 [x86_64-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen Home page is http://smartmontools.sourceforge.net/ SMART Health Status: OK
 

Comments

Popular posts from this blog

PSSH : Parallel SSH to execute commands on a number of hosts

How to add check_http as a service in Nagios Monitoring using NRPE

Configuring Nagios to monitor services using NRPE