Skip to content

Fixing FreeNAS Currently unreadable (pending) sectors error

8 October 2014

If you have redundancy, one or two bad sectors can be fixed even without any downtime.

The error message might look something like this:

10-06-2014 16:25:20 Daemon.Critical storage2 Oct  6 16:25:21 smartd[1560]: Device: /dev/ada2, 2 Currently unreadable (pending) sectors

You can see that the faulty device is /dev/ada2. Now, login to console and run long SMART test:

smartctl -t long /dev/ada2

It should tell you that the test is started and when it will finish. After it finishes, check test results:

smartctl -a /dev/ada2

The output will tell you two important things. In information section there is sector size:

Sector Size:      512 bytes logical/physical

Near the end there is SMART Self-test log which tells you whether it failed and what’s the faulty sector number (LBA_of_first_error column)

# 1  Extended offline    Completed: read failure       90%     25166         892134344

Now we have all the info we need to fix it. We will directly write to this sector to force it to reallocate. The important parameters are ‘of’ which is your faulty device name, ‘bs’ which is sector size and ‘seek’ which is sector number.

sysctl kern.geom.debugflags=16
dd if=/dev/zero of=/dev/ada2 bs=892134344 count=1 seek= conv=noerror,sync

Then check if the ‘Current_Pending_Sector’ in ‘SMART Attributes Data Structure’ table went to 0:

smartctl -a /dev/ada2

If not, repeat the long test and write to all sectors until they reallocate. Then run a scrub, replace ‘poolname’ with your poolname:

zpool scrub poolname

Finally check scrub status

zpool status -v poolname

That should do it.

Advertisements

From → Uncategorized

13 Comments
  1. Matt permalink

    Did it work? Is BS not blocksize? seek= the sector?

  2. johnrezzo permalink

    That dd command looks dangerously wrong, unless you really want to write a block as big as 850MiB. Once smartctl told you the LBA_of_first_error, and assuming your device has 512 byte sector blocks, you should then use something like:
    dd if=/dev/zero of=/dev/yourdevice bs=512 count=1 seek=892134344

    • best to check the bs value, using diskinfo -v /dev/ada(your disk number)

  3. Johnrezzo is correct. bs=sector. However, even though my sectors were 512, nothing was fixed after 40+ tries. It only worked once I went above what the sector number was and did 2048, so my bs=2048. Only then did the SMART get reset to 0. After scrubbing, everything was 100% healthy.
    so my shell command was: dd if=/dev/zero of=/dev/ada2 bs=2048 count=1 seek=27945554

    • BS= stripesize and hence 512 did not work. 2048 worked but best to find the info using command: diskinfo -v /dev/ada1

  4. Oooof. I really wish I had read the comments first before trying this…

    • noobfix permalink

      In reply to “Oooof” lol, One would think Freenas whould have a simple web ui point and click way of dealing with this common problem but No Freenas Failed us massively, when your one and only job is to handle storage drives how could you not handle something as simple as smart errors hum, anyway the safest simplest noob way to handle this if you care about your data is to download and burn a cd seatools for dos, run the long test on the drive then repair the bad sectors / blocks then boot freenas and do a scrub all done all is well till next time pain in the but it works good and is easy here is a link to the download seatools for dos http://www.seagate.com/support/downloads/item/seatools-dos-master/

      and yes it works on any drive brand and any file system for me anyway have fun

  5. Instead of increasing sector number, you can just increase count, since total bytes written = sector size * count. And if the part of the disk that is failing is more than a few bytes large, it might pay to set count=something higher than 1, maybe even, say, 10, to force that number of sectors to remap.

    In my case I also used /dev/random instead of /dev/zero.

    Due to not reading the comments first, I unfortunately (what was I thinking? Oh yeah, I was not) executed the original command listed in the blog with bs= and seek=0. Then waited some seconds before realizing it was wrong and trying to stop it with control-C which seemed nonresponsive. Basically I zeroed out my drive lol. This caused ZFS to take it completely offline and put my pool in a DEGRADED state. I’m currently dealing with that, considering just physically removing the drive at this point, chalking it up as a loss and replacing it with a spare I have and resilvering.

    Original Poster: PLEASE edit your post to fix the commands! Your advice will cause people to zero out their disks! lol

    Since drives (as I understand it) already have some capacity set aside for automatic sector remapping as the drive is used, it’s my opinion that if these actually start showing up in userland, things are already f***ed and it may pay to just remove the drive entirely. Seems a waste since the rest of the drive is still theoretically readwriteable, but whatever, capacity is cheap these days.

  6. Ah, I see what the problem here is. WordPress is filtering out anything in between less-than and greater-than signs, period. That’s lame. SO my command above that said bs= should have said bs=[some large number]. And I bet the OP had the same thing happen for the seek= parameter.

  7. Error in the information given. It should be;

    Original:
    sysctl kern.geom.debugflags=16
    dd if=/dev/zero of=/dev/ada2 bs=892134344 count=1 seek= conv=noerror,sync

    Corrected:
    sysctl kern.geom.debugflags=16
    dd if=/dev/zero of=/dev/ada2 bs=4096 count=1 seek=892134344 conv=noerror,sync

    Note: 4096 is the stripesize found by using command: diskinfo -v /dev/ada(your disk number)

    eg; for me my disk is 1 and hence ada1 is used for this example;

    —–
    [root@freenas ~]# diskinfo -v /dev/ada1
    /dev/ada1
    512 # sectorsize
    2000398934016 # mediasize in bytes (1.8T)
    3907029168 # mediasize in sectors
    4096 # stripesize
    0 # stripeoffset
    3876021 # Cylinders according to firmware.
    16 # Heads according to firmware.
    63 # Sectors according to firmware.
    WD-WCAVY6679527 # Disk ident.
    ———

    The seek is the sector eg shown below;

    So in my case the seek would equal = 887447520

    ——–
    No Errors Logged

    SMART Self-test log structure revision number 1
    Num Test_Description Status Remaining LifeTime(hours) LBA
    _of_first_error
    # 1 Extended offline Completed: read failure 80% 23265 887
    447520
    # 2 Short offline Completed without error 00% 18175 –
    # 3 Short offline Completed without error 00% 13959 –
    # 4 Short offline Completed without error 00% 12206 –
    # 5 Short offline Completed without error 00% 2308 –

    SMART Selective self-test log data structure revision number 1
    SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
    1 0 0 Not_testing
    2 0 0 Not_testing
    3 0 0 Not_testing
    4 0 0 Not_testing
    5 0 0 Not_testing
    Selective self-test flags (0x0):
    After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.

    [root@freenas ~]# c
    ———-

    hence the command i would use is;

    sysctl kern.geom.debugflags=16
    dd if=/dev/zero of=/dev/ada1 bs=4096 count=1 seek=887447520 conv=noerror,sync

    key bs= stripesize
    seek= sector

    I hope this helps because although the above text is useful it did have an error in it and took me some time to resolve after reading a number of other article.

    All the best if fixing your bad sectors.

  8. blackbear permalink

    “If you have redundancy”

    How does one determine that?

  9. hoxide permalink

    Reblogged this on Hoxide’s Blog and commented:
    very useful

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: