Freenas/ZFS and FreeNAS expansion

From MattWiki

Disclaimer: Use this info at your own risk, don't come to me if all your data disappears one day. But it works for me.

Background

Want to use 4 different sized drives together as one large storage tank with 1 drive fault tolerance?

Do you also want to be able to upgrade the smallest capacity drive with a larger one without having to move all of your data to a temporary location then re-build a new array and then copying all your data back?

I've figured out how you can do it.

I've figured out how to make FreeNAS using ZFS (nightly build) to use an array of 4 drives (theoretically start with one drive but I'm not going to talk about that here) and allow the array to grow by replacing the smallest drive in the array with a larger one similarly to how a Drobo works. I'm going to describe the process then I'll do a real world example.

Summary

This is the basic idea... I will be creating five raid5 sets inside one ZFS pool. I will be creating four partitions on each of the four drives and using these partitions as the storage containers for the raid arrays. This way I can control the sizes of each partition and get the best use out of the drive space within a raid5 array. ZFS allows you to expand the size of a raid array and the method I'm describing let's you take advantage of that cool feature.

What I will be doing is slicing up the four hard drives into as many equal sized parts as I can. From there I'll build five Raid5 arrays (one for each line of partitions in the phase 1 illustration) and create a zpool called tank0 containing all five raid5 arrays.

Details

As you can see below I will start with four drives:

Drive 0 - 80 Gig

Drive 1 - 120 Gig

Drive 2 - 200 Gig

Drive 3 - 200 Gig


The zpool will be five raid5 arrays and will give me about 329 Gigs of protected storage (54% use of the total space of the drives). Then I'll take the 80 Gig drive out of the picture and put in a 250 Gig and I'll end up with the following drives:

Drive 0 - 120 Gig

Drive 1 - 200 Gig

Drive 2 - 200 Gig

Drive 3 - 250 Gig


My zpool will increase in size to about 531 Gigs of protected storage (68% use of the total space of the drives). This will be done without doing any backing up, just swapping of drives. It is a four step process and each drive must be installed wiped repartitioned and resilvered (replaced) back into the zpool. I actually moved all the drives around one at a time. But I'm not sure if I had to bother. It might be possible and easier by just swapping out the smallest drive with the largest one and repartion all the drives in place one at a time... I'll test that in the future. As for right now I've completed it the long way and it works!

Configurations

See how the configurations change below from phase 1 to phase 2 to get an idea of what I'm about to do.

Phase 1

Phase 1
Drive 0 Drive 1 Drive 2 Drive 3 Partition Size
ad0 ad1 ad2 ad3
Drive Size 80 Gig 120 Gig 200 Gig 200 Gig
Actual usable 81918 123482 203884 203884
ad0s1 ad1s1 ad2s1 ad3s1 Raid5a 81726 x 3 245178
ad0s2 ad1s2 ad2s2 Raid5b 64 x 2 128
ad1s3 ad2s3 ad3s2 Raid5c 41628 x 2 83256
ad0s3 ad1s4 ad3s3 Raid5d 64 x 2 128
ad0s4 ad2s4 ad3s4 Raid5e 64 x 2 128
Unused Space 0 0 80402 80402
Total Space (sum of all drives) 613168 Megabytes = 613 Gigabytes
Total available within Zpool 328818 Megabytes = 329 Gigabytes
Percent of usable storage 54%

Phase 2

Phase 2
Drive 0 Drive 1 Drive 2 Drive 3 Partition Size
ad0 ad1 ad2 ad3
Drive Size 120 Gig 200 Gig 200 Gig 250 Gig
Actual usable 123482 203884 203884 250018
ad0s1 ad1s1 ad2s1 ad3s1 Raid5a 123290 x 3 369870
ad0s2 ad1s2 ad2s2 Raid5b 64 x 2 128
ad1s3 ad2s3 ad3s2 Raid5c 80466 x 2 160932
ad0s3 ad1s4 ad3s3 Raid5d 64 x 2 128
ad0s4 ad2s4 ad3s4 Raid5e 64 x 2 128
Unused Space 0 0 0 46134
Total Space (sum of all drives) 781268 Megabytes = 781 Gigabytes
Total available within Zpool 531186 Megabytes = 531 Gigabytes
Percent of usable storage 68%

Procedure

The procedure using a FreeNAS nightly build (newest experimental version):

Step 1

First thing you need to do is figure out the best use of your four drives space. Use the chart in Phase 1. I suggest using a spreadsheet to work out the best sizes for the partitions. Remember if you are doing this yourself, make sure you keep the partition sizes the same size or larger. You can't shrink them, ZFS only allows you to replace parts of a raid array with one the same size or larger.

Step 2

Using FreeNAS's GUI Mount the four disks and enable SMART monitoring Disks|Management| +

Preformatted file system set as ZFS storage pool device.

This allows FreeNAS to monitor the drive using S.M.A.R.T. and can be setup to email when a drive is showing symptoms of problems.

Step 3

Now I create raid5a1 on drive0 (ad0) using fdisk-linux (which I removed the original FreeNAS fdisk and renamed the fdisk-linux to fdisk on my FreeNAS server). Get the fdisk-linux package [here]

Step 4

Delete or rename the original /sbin/fdisk and copy the fdisk-linux to your /sbin folder. I renamed the fdisk-linux to fdisk and that is what I used for the rest of this document.

Step 5

From the command prompt:

Partition Drives

Partition Drive 0 (ad0) (delete all partitions on the drives first)

Beginning the Partitioning
freenas:/sbin# fdisk /dev/ad0
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 9964.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-9964, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-9964, default 9964): +81726M

Command (m for help): p

Disk /dev/ad0: 81.9 GB, 81964302336 bytes
255 heads, 63 sectors/track, 9964 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad0s1             1      9937  79818921   83  Linux

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (9938-9964, default 9938):
Using default value 9938
Last cylinder or +size or +sizeM or +sizeK (9938-9964, default 9964): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (9947-9964, default 9947):
Using default value 9947
Last cylinder or +size or +sizeM or +sizeK (9947-9964, default 9964): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First cylinder (9956-9964, default 9956):
Using default value 9956
Last cylinder or +size or +sizeM or +sizeK (9956-9964, default 9964):
Using default value 9964

Command (m for help): p

Disk /dev/ad0: 81.9 GB, 81964302336 bytes
255 heads, 63 sectors/track, 9964 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad0s1             1      9937  79818921   83  Linux
/dev/ad0s2          9938      9946     72292+  83  Linux
/dev/ad0s3          9947      9955     72292+  83  Linux
/dev/ad0s4          9956      9964     72292+  83  Linux

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad0: 81.9 GB, 81964302336 bytes
255 heads, 63 sectors/track, 9964 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad0s1             1      9937  79818921   a5  FreeBSD
/dev/ad0s2          9938      9946     72292+  a5  FreeBSD
/dev/ad0s3          9947      9955     72292+  a5  FreeBSD
/dev/ad0s4          9956      9964     72292+  a5  FreeBSD

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
Partition Drive 1 (ad1)
freenas:/sbin# fdisk /dev/ad1
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 15017.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-15017, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-15017, default 15017): +81726M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (9938-15017, default 9938):
Using default value 9938
Last cylinder or +size or +sizeM or +sizeK (9938-15017, default 15017): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (9947-15017, default 9947):
Using default value 9947
Last cylinder or +size or +sizeM or +sizeK (9947-15017, default 15017): +41628M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First cylinder (15009-15017, default 15009):
Using default value 15009
Last cylinder or +size or +sizeM or +sizeK (15009-15017, default 15017):
Using default value 15017

Command (m for help): p

Disk /dev/ad1: 123.5 GB, 123521334784 bytes
255 heads, 63 sectors/track, 15017 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad1s1             1      9937  79818921   83  Linux
/dev/ad1s2          9938      9946     72292+  83  Linux
/dev/ad1s3          9947     15008  40660515   83  Linux
/dev/ad1s4         15009     15017     72292+  83  Linux

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad1: 123.5 GB, 123521334784 bytes
255 heads, 63 sectors/track, 15017 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad1s1             1      9937  79818921   a5  FreeBSD
/dev/ad1s2          9938      9946     72292+  a5  FreeBSD
/dev/ad1s3          9947     15008  40660515   a5  FreeBSD
/dev/ad1s4         15009     15017     72292+  a5  FreeBSD

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
Now we Partition Drive 2 (ad2)
freenas:/sbin# fdisk /dev/ad2
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +81726M

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad2s1             1      9937  79818921   83  Linux

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (9938-24792, default 9938):
Using default value 9938
Last cylinder or +size or +sizeM or +sizeK (9938-24792, default 24792): +64M

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad2s1             1      9937  79818921   83  Linux
/dev/ad2s2          9938      9946     72292+  83  Linux

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (9947-24792, default 9947):
Using default value 9947
Last cylinder or +size or +sizeM or +sizeK (9947-24792, default 24792): +41628M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First cylinder (15009-24792, default 15009):
Using default value 15009
Last cylinder or +size or +sizeM or +sizeK (15009-24792, default 24792): +64M

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad2s1             1      9937  79818921   83  Linux
/dev/ad2s2          9938      9946     72292+  83  Linux
/dev/ad2s3          9947     15008  40660515   83  Linux
/dev/ad2s4         15009     15017     72292+  83  Linux

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad2s1             1      9937  79818921   a5  FreeBSD
/dev/ad2s2          9938      9946     72292+  a5  FreeBSD
/dev/ad2s3          9947     15008  40660515   a5  FreeBSD
/dev/ad2s4         15009     15017     72292+  a5  FreeBSD

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
Now we partition Drive3 (ad3)
freenas:/sbin# fdisk /dev/ad3
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +81726M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (9938-24792, default 9938):
Using default value 9938
Last cylinder or +size or +sizeM or +sizeK (9938-24792, default 24792): +41628M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15000-24792, default 15000):
Using default value 15000
Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First cylinder (15009-24792, default 15009):
Using default value 15009
Last cylinder or +size or +sizeM or +sizeK (15009-24792, default 24792): +64M

Command (m for help): p

Disk /dev/ad3: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad3s1             1      9937  79818921   83  Linux
/dev/ad3s2          9938     14999  40660515   83  Linux
/dev/ad3s3         15000     15008     72292+  83  Linux
/dev/ad3s4         15009     15017     72292+  83  Linux

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad3: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad3s1             1      9937  79818921   a5  FreeBSD
/dev/ad3s2          9938     14999  40660515   a5  FreeBSD
/dev/ad3s3         15000     15008     72292+  a5  FreeBSD
/dev/ad3s4         15009     15017     72292+  a5  FreeBSD

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
Now let’s create our zpool containing our 5 (Raid5a to Raid5e) raid5 arrays (of the partitions)
freenas:/sbin# zpool create -m /mnt/tank0 tank0 raidz1 /dev/ad0s1 /dev/ad1s1 /dev/ad2s1 /dev/ad3s1
freenas:/sbin# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    494M     87M    367M    19%    /
devfs          1.0K    1.0K      0B   100%    /dev
tank0          224G    128K    224G     0%    /mnt/tank0
freenas:/sbin# zpool status
  pool: tank0
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0

errors: No known data errors
Now we add the 2nd raid array to the same pool (tank0).
 We must force since we are adding a 3 device raid to a pool that already has a 4 device raid.
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s2 /dev/ad1s2 /dev/ad2s2
freenas:/sbin# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    494M     87M    367M    19%    /
devfs          1.0K    1.0K      0B   100%    /dev
tank0          224G      0B    224G     0%    /mnt/tank0
freenas:/sbin# zpool status
  pool: tank0
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0

errors: No known data errors
Now we add the 3rd raid array:
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad1s3 /dev/ad2s3 /dev/ad3s2
freenas:/sbin# df -H
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    518M     92M    385M    19%    /
devfs          1.0k    1.0k      0B   100%    /dev
tank0          322G      0B    322G     0%    /mnt/tank0
freenas:/sbin# zpool status
  pool: tank0
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0

errors: No known data errors
Now we add the 4th raid array:
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s3 /dev/ad1s4 /dev/ad3s3
freenas:/sbin# df -H
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    518M     92M    385M    19%    /
devfs          1.0k    1.0k      0B   100%    /dev
tank0          322G      0B    322G     0%    /mnt/tank0
freenas:/sbin# zpool status
  pool: tank0
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0

errors: No known data errors
Now we add the 5th and final raid to our pool (tank0)
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s4 /dev/ad2s4 /dev/ad3s4
freenas:/sbin# df -H
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    518M     92M    385M    19%    /
devfs          1.0k    1.0k      0B   100%    /dev
tank0          322G      0B    322G     0%    /mnt/tank0
freenas:/sbin# zpool status
  pool: tank0
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors
Before we can share it on the network with SMB (samba) we have to enable writing:
freenas:/sbin# ls -la /mnt
total 6
drwxr-xr-x   3 root  wheel  512 Apr 19 00:21 .
drwxr-xr-x  19 root  wheel  512 Apr 18 01:37 ..
drwxr-xr-x   2 root  wheel    2 Apr 19 00:21 tank0
freenas:/sbin# chmod 777 /mnt/tank0/

Now add the tank0 to samba and test the speed.

Testing

Over my 100Mb/s Ethernet connection.

Writing a 1,411,699,808 byte file to tank0 over the network took 152 seconds. Which is 74 Mb/s not too bad considering all the writing to the different raid5 arrays within the pool. ZFS must do a lot in memory… Ram usage was around 25% of my 1 Gig FreeNAS box and CPU varied from 25% to 50% from my AMD Sempron(tm) Processor 3000+, running at 1800 Mhz

Reading a 1,411,699,808 byte file from tank0 over the network took 160 seconds. Which is 70 Mb/s not too bad either. Ram usage was around 14% of my 1 Gig FreeNAS box and CPU was consistently at 25% from my AMD Sempron(tm) Processor 3000+, running at 1800 Mhz

It is surprising that reading from the zpool tank0 was a little slower then writing… Either way these speeds aren’t too shabby and will do well for streaming HD quality movies or doing archival backups of your data.

Swapping Drives On The Fly

Now on to swapping out the 80 Gig drive for a 250 Gig without backing up to another drive and then rebuilding the array and copying it all back.

First scrub the zpool to make sure it’s all good.

freenas:/sbin# zpool scrub tank0
freenas:/sbin# zpool status
  pool: tank0
 state: ONLINE
 scrub: scrub in progress, 12.48% done, 0h1m to go
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors

After a little while the scrub completes. This will take longer with more data in the pool.

freenas:/sbin# zpool status
  pool: tank0
 state: ONLINE
 scrub: scrub completed with 0 errors on Sun Apr 19 01:07:58 2009
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors
Let’s export the zpool.
freenas:/sbin# zpool export -f tank0

Shutdown the computer and swap out the 200 Gig drive3 with the new larger 250 Gig drive 3. We work our way down to drive0 in steps, swapping larger drives along the way.

With the new drive installed we have to import the pool to see it.
freenas:~# zpool import
  pool: tank0
    id: 1762523649679414016
 state: FAULTED
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://www.sun.com/msg/ZFS-8000-3C
config:

        tank0       UNAVAIL   insufficient replicas
          raidz1    DEGRADED
            ad1s1   ONLINE
            ad2s1   ONLINE
            ad2s1   UNAVAIL   cannot open
            ad3s1   ONLINE
          raidz1    UNAVAIL   corrupted data
            ad2s2   ONLINE
            ad2s2   UNAVAIL   cannot open
            ad3s2   ONLINE
          raidz2    DEGRADED
            ad2s3   UNAVAIL   cannot open
            ad3s3   ONLINE
            ad2s4   UNAVAIL   cannot open
            ad3s4   ONLINE

  pool: tank0
    id: 325956304794594593
 state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: http://www.sun.com/msg/ZFS-8000-2Q
config:

        tank0       DEGRADED
          raidz1    DEGRADED
            ad0s1   ONLINE
            ad1s1   ONLINE
            ad2s1   ONLINE
            ad3s1   UNAVAIL   cannot open
          raidz1    ONLINE
            ad0s2   ONLINE
            ad1s2   ONLINE
            ad2s2   ONLINE
          raidz1    DEGRADED
            ad1s3   ONLINE
            ad2s3   ONLINE
            ad3s2   UNAVAIL   cannot open
          raidz1    DEGRADED
            ad0s3   ONLINE
            ad1s4   ONLINE
            ad3s3   UNAVAIL   cannot open
          raidz1    DEGRADED
            ad0s4   ONLINE
            ad2s4   ONLINE
            ad3s4   UNAVAIL   cannot open

During testing I already used the name tank0 and built a similar raid array. ZFS is trying to get all the info it can. Since they both have the same name I must import the real one using it’s ID.

freenas:~# zpool import 325956304794594593
freenas:~# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: resilver completed with 0 errors on Sun Apr 19 01:25:45 2009
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       DEGRADED     0     0     0
          raidz1    DEGRADED     0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   UNAVAIL      0     0     0  cannot open
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    DEGRADED     0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   UNAVAIL      0     0     0  cannot open
          raidz1    DEGRADED     0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   UNAVAIL      0     0     0  cannot open
          raidz1    DEGRADED     0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   UNAVAIL      0     0     0  cannot open

errors: No known data errors
So far so good, now we partition Drive3 (the new 250 Gig drive) ad3 and replace the partitions into the raid arrays of our zpool.
freenas:~# fdisk /dev/ad3

The number of cylinders for this disk is set to 30401.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-30401, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-30401, default 30401): +123290M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14991-30401, default 14991):
Using default value 14991
Last cylinder or +size or +sizeM or +sizeK (14991-30401, default 30401): +80466M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (24775-30401, default 24775):
Using default value 24775
Last cylinder or +size or +sizeM or +sizeK (24775-30401, default 30401): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First cylinder (24784-30401, default 24784):
Using default value 24784
Last cylinder or +size or +sizeM or +sizeK (24784-30401, default 30401): +64M

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad3: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad3s1             1     14990 120407143+  a5  FreeBSD
/dev/ad3s2         14991     24774  78589980   a5  FreeBSD
/dev/ad3s3         24775     24783     72292+  a5  FreeBSD
/dev/ad3s4         24784     24792     72292+  a5  FreeBSD

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
Now we replace the partitions of old drive3 (200 Gig) with the new Drive3 (250 Gig).
freenas:~# zpool replace tank0 ad3s1
freenas:~# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: resilver in progress, 29.02% done, 0h0m to go
config:

        NAME             STATE     READ WRITE CKSUM
        tank0            DEGRADED     0     0     0
          raidz1         DEGRADED     0     0     0
            ad0s1        ONLINE       0     0     0
            ad1s1        ONLINE       0     0     0
            ad2s1        ONLINE       0     0     0
            replacing    DEGRADED     0     0     0
              ad3s1/old  UNAVAIL      0     0     0  cannot open
              ad3s1      ONLINE       0     0     0
          raidz1         ONLINE       0     0     0
            ad0s2        ONLINE       0     0     0
            ad1s2        ONLINE       0     0     0
            ad2s2        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            ad1s3        ONLINE       0     0     0
            ad2s3        ONLINE       0     0     0
            ad3s2        UNAVAIL      0     0     0  cannot open
          raidz1         DEGRADED     0     0     0
            ad0s3        ONLINE       0     0     0
            ad1s4        ONLINE       0     0     0
            ad3s3        UNAVAIL      0     0     0  cannot open
          raidz1         DEGRADED     0     0     0
            ad0s4        ONLINE       0     0     0
            ad2s4        ONLINE       0     0     0
            ad3s4        UNAVAIL      0     0     0  cannot open

errors: No known data errors
After it completes we replace the other 3 partitions:
freenas:~# zpool replace tank0 ad3s2
freenas:~# zpool replace tank0 ad3s3
freenas:~# zpool replace tank0 ad3s4

When they are complete we have a nice clean pool again:

freenas:~# zpool status
  pool: tank0
 state: ONLINE
 scrub: resilver completed with 0 errors on Sun Apr 19 01:46:31 2009
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors

Testing

Copy some more files on tank0 for testing… No problem. I’ve got a 500 Meg .ZIP file on tank0 and I’ll do a test on it to make sure the data is still perfect, just to be 100% sure.

Test found no errors. As expected.

Let’s scrub again then when it completes without error, we export and shutdown the computer. Then we swap out drive 2 (200 Gig) with the old drive3 200 Gig drive, probably not needed since they are the same size, but I’m going to do it for completeness.

freenas:~# zpool scrub tank0
freenas:~# zpool export –f tank0

Shutdown and swap out drives reboot with new drive2 (200 Gig):

freenas:~# zpool import
  pool: tank0
    id: 325956304794594593
 state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: http://www.sun.com/msg/ZFS-8000-2Q
config:

        tank0       DEGRADED
          raidz1    DEGRADED
            ad0s1   ONLINE
            ad1s1   ONLINE
            ad2s1   UNAVAIL   cannot open
            ad3s1   ONLINE
          raidz1    DEGRADED
            ad0s2   ONLINE
            ad1s2   ONLINE
            ad2s2   UNAVAIL   cannot open
          raidz1    DEGRADED
            ad1s3   ONLINE
            ad2s3   UNAVAIL   cannot open
            ad3s2   ONLINE
          raidz1    ONLINE
            ad0s3   ONLINE
            ad1s4   ONLINE
            ad3s3   ONLINE
          raidz1    DEGRADED
            ad0s4   ONLINE
            ad2s4   UNAVAIL   cannot open
            ad3s4   ONLINE

freenas:~# zpool import tank0
freenas:~# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: resilver completed with 0 errors on Sun Apr 19 02:05:57 2009
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       DEGRADED     0     0     0
          raidz1    DEGRADED     0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   UNAVAIL      0     0     0  cannot open
            ad3s1   ONLINE       0     0     0
          raidz1    DEGRADED     0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   UNAVAIL      0     0     0  cannot open
          raidz1    DEGRADED     0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   UNAVAIL      0     0     0  cannot open
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    DEGRADED     0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   UNAVAIL      0     0     0  cannot open
            ad3s4   ONLINE       0     0     0

errors: No known data errors

Now we repartition drive2 with our new sizes:

freenas:~# fdisk /dev/ad2

The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): d
Partition number (1-4): 4

Command (m for help): d
Partition number (1-4): 3

Command (m for help): d
Partition number (1-4): 2

Command (m for help): d
Selected partition 1

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +123290M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14991-24792, default 14991):
Using default value 14991
Last cylinder or +size or +sizeM or +sizeK (14991-24792, default 24792): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15000-24792, default 15000):
Using default value 15000
Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +80466M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First cylinder (24784-24792, default 24784):
Using default value 24784
Last cylinder or +size or +sizeM or +sizeK (24784-24792, default 24792):
Using default value 24792

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad2s1             1     14990 120407143+  83  Linux
/dev/ad2s2         14991     14999     72292+  83  Linux
/dev/ad2s3         15000     24783  78589980   83  Linux
/dev/ad2s4         24784     24792     72292+  83  Linux

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad2s1             1     14990 120407143+  a5  FreeBSD
/dev/ad2s2         14991     14999     72292+  a5  FreeBSD
/dev/ad2s3         15000     24783  78589980   a5  FreeBSD
/dev/ad2s4         24784     24792     72292+  a5  FreeBSD

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Now we replace drive2’s larger partitions in our pool:

freenas:~# zpool replace tank0 ad2s1
freenas:~# zpool replace tank0 ad2s2
freenas:~# zpool replace tank0 ad2s3
freenas:~# zpool replace tank0 ad2s4
freenas:~# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress, 27.04% done, 0h0m to go
config:

        NAME             STATE     READ WRITE CKSUM
        tank0            DEGRADED     0     0     0
          raidz1         DEGRADED     0     0     0
            ad0s1        ONLINE       0     0     0
            ad1s1        ONLINE       0     0     0
            replacing    DEGRADED     0     0     0
              ad2s1/old  UNAVAIL      0     0     0  cannot open
              ad2s1      ONLINE       0     0     0
            ad3s1        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            ad0s2        ONLINE       0     0     0
            ad1s2        ONLINE       0     0     0
            replacing    DEGRADED     0     0     0
              ad2s2/old  UNAVAIL      0     0     0  cannot open
              ad2s2      ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            ad1s3        ONLINE       0     0     0
            replacing    DEGRADED     0     0     0
              ad2s3/old  UNAVAIL      0     0     0  cannot open
              ad2s3      ONLINE       0     0     0
            ad3s2        ONLINE       0     0     0
          raidz1         ONLINE       0     0     0
            ad0s3        ONLINE       0     0     0
            ad1s4        ONLINE       0     0     0
            ad3s3        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            ad0s4        ONLINE       0     0     0
            replacing    DEGRADED     0     0     0
              ad2s4/old  UNAVAIL      0     0     0  cannot open
              ad2s4      ONLINE       0     0     0
            ad3s4        ONLINE       0     0     0

errors: No known data errors

After awhile resilvering should complete and we look clean again:

freenas:~# zpool status
  pool: tank0
 state: ONLINE
 scrub: resilver completed with 0 errors on Sun Apr 19 02:14:01 2009
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors

Some more quick tests, TOK as they should.

Size is still the same:

freenas:~# df -H
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    518M     92M    385M    19%    /
devfs          1.0k    1.0k      0B   100%    /dev
tank0          322G    3.4G    319G     1%    /mnt/tank0

It will grow after the last part of any one raid array is replaced with a larger one.

So all is good now we need to swap out the Drive1 (120 Gig) drive with the old Drive2 (200 Gig) drive. So we scrub and export:

freenas:~# zpool scrub tank0

Got to 97.7% then my system gave some errors (didn’t have time to read the screen) and then it rebooted.

I wonder if it’s FreeNAS’s ZFS problem or something else?

After reboot the pool tank0 came up ok by itself. Maybe it did complete then the reboot happened. I’m not 100% sure.

freenas:~# zpool status
  pool: tank0
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors

Going to do another scrub…

freenas:~# zpool scrub tank0

Completed without any problems this time. Weird!

Now to export and swap out the drives.

freenas:~# zpool export tank0

Shutdown and swap Drive1 (120 Gig) with old Drive2 (200 Gig).

Reboot and check the zpool

freenas:~# zpool status
no pools available
freenas:~# zpool import
  pool: tank0
    id: 325956304794594593
 state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: http://www.sun.com/msg/ZFS-8000-2Q
config:

        tank0       DEGRADED
          raidz1    DEGRADED
            ad0s1   ONLINE
            ad1s1   UNAVAIL   cannot open
            ad2s1   ONLINE
            ad3s1   ONLINE
          raidz1    DEGRADED
            ad0s2   ONLINE
            ad1s2   UNAVAIL   cannot open
            ad2s2   ONLINE
          raidz1    DEGRADED
            ad1s3   UNAVAIL   cannot open
            ad2s3   ONLINE
            ad3s2   ONLINE
          raidz1    DEGRADED
            ad0s3   ONLINE
            ad1s4   UNAVAIL   cannot open
            ad3s3   ONLINE
          raidz1    ONLINE
            ad0s4   ONLINE
            ad2s4   ONLINE
            ad3s4   ONLINE

Now we import our zpool

freenas:~# zpool import tank0
freenas:~# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: resilver completed with 0 errors on Sun Apr 19 09:19:02 2009
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       DEGRADED     0     0     0
          raidz1    DEGRADED     0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   UNAVAIL      0     0     0  cannot open
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    DEGRADED     0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   UNAVAIL      0     0     0  cannot open
            ad2s2   ONLINE       0     0     0
          raidz1    DEGRADED     0     0     0
            ad1s3   UNAVAIL      0     0     0  cannot open
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    DEGRADED     0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   UNAVAIL      0     0     0  cannot open
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors

As you can see ad1 partitions need to be replaced. But first we have to resize our new drive1 (200 Gig) partitions.

freenas:~# fdisk /dev/ad1

The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): d
Partition number (1-4): 4

Command (m for help): d
Partition number (1-4): 3

Command (m for help): d
Partition number (1-4): 2

Command (m for help): d
Selected partition 1

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +123290M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14991-24792, default 14991):
Using default value 14991
Last cylinder or +size or +sizeM or +sizeK (14991-24792, default 24792): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15000-24792, default 15000):
Using default value 15000
Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +80466M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First cylinder (24784-24792, default 24784):
Using default value 24784
Last cylinder or +size or +sizeM or +sizeK (24784-24792, default 24792):
Using default value 24792

Command (m for help): p

Disk /dev/ad1: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad1s1             1     14990 120407143+  83  Linux
/dev/ad1s2         14991     14999     72292+  83  Linux
/dev/ad1s3         15000     24783  78589980   83  Linux
/dev/ad1s4         24784     24792     72292+  83  Linux

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad1: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad1s1             1     14990 120407143+  a5  FreeBSD
/dev/ad1s2         14991     14999     72292+  a5  FreeBSD
/dev/ad1s3         15000     24783  78589980   a5  FreeBSD
/dev/ad1s4         24784     24792     72292+  a5  FreeBSD

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

New partitions have been created, we can now do the zpool replace thing.

freenas:~# zpool replace tank0 /dev/ad1s1
freenas:~# zpool replace tank0 /dev/ad1s2
freenas:~# zpool replace tank0 /dev/ad1s3
freenas:~# zpool replace tank0 /dev/ad1s4

I don’t wait for each partition to resilver. I type all four replace commands as soon as I can and I let it chug away.

freenas:~# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress, 63.24% done, 0h0m to go
config:

        NAME             STATE     READ WRITE CKSUM
        tank0            DEGRADED     0     0     0
          raidz1         DEGRADED     0     0     0
            ad0s1        ONLINE       0     0     0
            replacing    DEGRADED     0     0     0
              ad1s1/old  UNAVAIL      0     0     0  cannot open
              ad1s1      ONLINE       0     0     0
            ad2s1        ONLINE       0     0     0
            ad3s1        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            ad0s2        ONLINE       0     0     0
            replacing    DEGRADED     0     0     0
              ad1s2/old  UNAVAIL      0     0     0  cannot open
              ad1s2      ONLINE       0     0     0
            ad2s2        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            replacing    DEGRADED     0     0     0
              ad1s3/old  UNAVAIL      0     0     0  cannot open
              ad1s3      ONLINE       0     0     0
            ad2s3        ONLINE       0     0     0
            ad3s2        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            ad0s3        ONLINE       0     0     0
            replacing    DEGRADED     0     0     0
              ad1s4/old  UNAVAIL      0     0     0  cannot open
              ad1s4      ONLINE       0     0     0
            ad3s3        ONLINE       0     0     0
          raidz1         ONLINE       0     0     0
            ad0s4        ONLINE       0     0     0
            ad2s4        ONLINE       0     0     0
            ad3s4        ONLINE       0     0     0

errors: No known data errors

Once it finishes we should look like:

freenas:~# zpool status
  pool: tank0
 state: ONLINE
 scrub: resilver completed with 0 errors on Sun Apr 19 09:28:56 2009
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors

Still the same size:

freenas:~# df -H
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    518M     92M    385M    19%    /
devfs          1.0k    1.0k      0B   100%    /dev
tank0          322G    3.4G    319G     1%    /mnt/tank0

Our tank0 pool still hasn’t changed Size, it should after we swap the last (fourth) drives partitions.

So now we get ready to replace the last drive. Drive0 (80 Gig, ad0) with our old drive1 (120 Gig) drive.

freenas:~# zpool scrub tank0

Once it completes (check with #zpool status) we export it and shutdown the computer.

freenas:~# zpool export tank0

Ok to shutdown and swap drives now.

With the last drive replaced, Drive0 (80 Gig) is Drive0 (120 Gig).

We do it all one last time:

freenas:~# zpool status
no pools available
freenas:~# zpool import
  pool: tank0
    id: 325956304794594593
 state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: http://www.sun.com/msg/ZFS-8000-2Q
config:

        tank0       DEGRADED
          raidz1    DEGRADED
            ad0s1   UNAVAIL   cannot open
            ad1s1   ONLINE
            ad2s1   ONLINE
            ad3s1   ONLINE
          raidz1    DEGRADED
            ad0s2   UNAVAIL   cannot open
            ad1s2   ONLINE
            ad2s2   ONLINE
          raidz1    ONLINE
            ad1s3   ONLINE
            ad2s3   ONLINE
            ad3s2   ONLINE
          raidz1    DEGRADED
            ad0s3   UNAVAIL   cannot open
            ad1s4   ONLINE
            ad3s3   ONLINE
          raidz1    DEGRADED
            ad0s4   UNAVAIL   cannot open
            ad2s4   ONLINE
            ad3s4   ONLINE
freenas:~# zpool import tank0

Now repartition drive0

freenas:~# fdisk /dev/ad0

The number of cylinders for this disk is set to 15017.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): d
Partition number (1-4): 4

Command (m for help): d
Partition number (1-4): 3

Command (m for help): d
Partition number (1-4): 2

Command (m for help): d
Selected partition 1

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-15017, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-15017, default 15017): +123290M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14991-15017, default 14991):
Using default value 14991
Last cylinder or +size or +sizeM or +sizeK (14991-15017, default 15017): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15000-15017, default 15000):
Using default value 15000
Last cylinder or +size or +sizeM or +sizeK (15000-15017, default 15017): +64M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Selected partition 4
First cylinder (15009-15017, default 15009):
Using default value 15009
Last cylinder or +size or +sizeM or +sizeK (15009-15017, default 15017):
Using default value 15017

Command (m for help): p

Disk /dev/ad0: 123.5 GB, 123521334784 bytes
255 heads, 63 sectors/track, 15017 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot    Start       End    Blocks   Id  System
/dev/ad0s1             1     14990 120407143+  83  Linux
/dev/ad0s2         14991     14999     72292+  83  Linux
/dev/ad0s3         15000     15008     72292+  83  Linux
/dev/ad0s4         15009     15017     72292+  83  Linux

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Now we replace our new partitions into the pool:

freenas:~# zpool replace tank0 /dev/ad0s1
freenas:~# zpool replace tank0 /dev/ad0s2
freenas:~# zpool replace tank0 /dev/ad0s3
freenas:~# zpool replace tank0 /dev/ad0s4
freenas:~# zpool status
  pool: tank0
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress, 14.32% done, 0h1m to go
config:

        NAME             STATE     READ WRITE CKSUM
        tank0            DEGRADED     0     0     0
          raidz1         DEGRADED     0     0     0
            replacing    DEGRADED     0     0     0
              ad0s1/old  UNAVAIL      0     0     0  cannot open
              ad0s1      ONLINE       0     0     0
            ad1s1        ONLINE       0     0     0
            ad2s1        ONLINE       0     0     0
            ad3s1        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            replacing    DEGRADED     0     0     0
              ad0s2/old  UNAVAIL      0     0     0  cannot open
              ad0s2      ONLINE       0     0     0
            ad1s2        ONLINE       0     0     0
            ad2s2        ONLINE       0     0     0
          raidz1         ONLINE       0     0     0
            ad1s3        ONLINE       0     0     0
            ad2s3        ONLINE       0     0     0
            ad3s2        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            replacing    DEGRADED     0     0     0
              ad0s3/old  UNAVAIL      0     0     0  cannot open
              ad0s3      ONLINE       0     0     0
            ad1s4        ONLINE       0     0     0
            ad3s3        ONLINE       0     0     0
          raidz1         DEGRADED     0     0     0
            replacing    DEGRADED     0     0     0
              ad0s4/old  UNAVAIL      0     0     0  cannot open
              ad0s4      ONLINE       0     0     0
            ad2s4        ONLINE       0     0     0
            ad3s4        ONLINE       0     0     0

errors: No known data errors

Once it finishes, we should now have a nice clean pool:

freenas:~# zpool status
  pool: tank0
 state: ONLINE
 scrub: resilver completed with 0 errors on Sun Apr 19 09:52:42 2009
config:

        NAME        STATE     READ WRITE CKSUM
        tank0       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s1   ONLINE       0     0     0
            ad1s1   ONLINE       0     0     0
            ad2s1   ONLINE       0     0     0
            ad3s1   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s2   ONLINE       0     0     0
            ad1s2   ONLINE       0     0     0
            ad2s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad1s3   ONLINE       0     0     0
            ad2s3   ONLINE       0     0     0
            ad3s2   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s3   ONLINE       0     0     0
            ad1s4   ONLINE       0     0     0
            ad3s3   ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad0s4   ONLINE       0     0     0
            ad2s4   ONLINE       0     0     0
            ad3s4   ONLINE       0     0     0

errors: No known data errors

We now have more storage space and we didn’t have to find somewhere to store our old data temporarily, which will get harder to do as the storage pool grows larger and larger. We just expanded it in place.

Old Space:

freenas:~# df -H
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    518M     92M    385M    19%    /
devfs          1.0k    1.0k      0B   100%    /dev
tank0          322G    3.4G    319G     1%    /mnt/tank0

After one last Reboot:

freenas:~# df -H
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    518M     92M    385M    19%    /
devfs          1.0k    1.0k      0B   100%    /dev
tank0          520G    3.4G    517G     1%    /mnt/tank0

I also add all the new drives in my FreeNAS using the webgui under Disk Management and enable SMART on them so I can have SMART monitoring within FreeNAS email me if some errors on any of my drives start to show up.

The read and write times are still the same. I thought they might have gotten a little better since the new 250Gig drive is faster then the older 80Gig drive. But with all the raid work going on I guess so much is being done in RAM that drive read and write speeds aren't too heavily used. That's a good thing!

I hope this method is helpful for others as I will be using it all the time now. One thing to remember you can't swap out IDE drives to SATA drives. The reason is the SATA controllers will show up with a new device name and ZFS needs to replace the exact device name for the old one. So if you are going to do this and want to be future proof I suggest starting with four SATA drives.

Good luck, Glen Hewlett.

ps Here's a bunch more drive expansion examples (the drive sizes aren't perfect so to maximize space you might want to verify your drive sizes yourself)

Phase 1											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	80 Gig	120 Gig	200 Gig	200 Gig							
Actual usable	80043	120621	199148	199148						598960.00	
	1	1	1	1	Raid5a	79851	x 3	239553			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	40578	x 2	81156			
	1	1		1	Raid6d	64	x 2	128			
	1		1	1	Raid6e	64	x 2	128			
											
											
Unused Space	0	64	78591	78591			Available	321093		Percentage of useage	54%
Phase 2											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	250 Gig	120 Gig	200 Gig	200 Gig							
Actual usable	244196	120621	199148	199148						763113.00	
	1	1	1	1	Raid5a	79851	x 3	239553			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	40578	x 2	81156			
	1	1		1	Raid6d	64	x 2	128			
	1		1	1	Raid6e	78655	x 2	157310			
											
											
Unused Space	85562	64	0	0			Available	478275		Percentage of useage	63%
Phase 3											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	250 Gig	300 Gig	200 Gig	200 Gig							
Actual usable	244196	293034	199148	199148						935526.00	
	1	1	1	1	Raid5a	79851	x 3	239553			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	40578	x 2	81156			
	1	1		1	Raid6d	64	x 2	128			
	1		1	1	Raid6e	78655	x 2	157310			
											
											
Unused Space	85562	172477	0	0			Available	478275		Percentage of useage	51%
Phase 4											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	200 Gig	250 Gig	300 Gig	500 Gig							
Actual usable	199148	244196	293034	488392						1224770.00	
	1	1	1	1	Raid5a	153908	x 3	461724			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	90160	x 2	180320			
	1	1		1	Raid6d	64	x 2	128			
	1		1	1	Raid6e	45112	x 2	90224			
											
											
Unused Space	0	0	3790	199148			Available	732524		Percentage of useage	60%
Phase 5											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	250 Gig	300 Gig	500 Gig	500 Gig							
Actual usable	244196	293034	488392	488392						1514014.00	
	1	1	1	1	Raid5a	198956	x 3	596868			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	93950	x 2	187900			
	1	1		1	Raid6d	64	x 2	128			
	1		1	1	Raid6e	45112	x 2	90224			
											
											
Unused Space	0	0	150310	150310			Available	875248		Percentage of useage	58%
Phase 6											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	300 Gig	500 Gig	500 Gig	500 Gig							
Actual usable	293034	488392	488392	488392						1758210.00	
	1	1	1	1	Raid5a	247794	x 3	743382			
 	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	195422	x 2	390844			
	1	1		1	Raid6d	64	x 2	128			
	1		1	1	Raid6e	45112	x 2	90224			
											 
											
Unused Space	0	45048	0	0			Available	1224706		Percentage of useage	70%
Phase 7											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	500 Gig	500 Gig	500 Gig	500 Gig							
Actual usable	488392	488392	488392	488392						1953568.00	
	1	1	1	1	Raid5a	247794	x 3	743382			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	195422	x 2	390844			
	1	1		1	Raid6d	64	x 2	128			
	1		1	1	Raid6e	45112	x 2	90224			
											
											
Unused Space	195358	45048	0	0			Available	1224706		Percentage of useage	63%
Phase 8											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	500 Gig	500 Gig	500 Gig	750 Gig							
Actual usable	488392	488392	488392	732588						2197764.00	
	1	1	1	1	Raid5a	247794	x 3	743382			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	195422	x 2	390844			
	1	1		1	Raid6d	45112	x 2	90224			
	1		1	1	Raid6e	45112	x 2	90224			
											
											
Unused Space	150310	0	0	199148			Available	1314802		Percentage of useage	60%
Phase 9											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	500 Gig	500 Gig	750 Gig	1000 Gig							
Actual usable	488392	488392	732588	976784						2686156.00	
	1	1	1	1	Raid5a	247794	x 3	743382			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	195422	x 2	390844			
	1	1		1	Raid6d	45112	x 2	90224			
	1		1	1	Raid6e	195422	x 2	390844			
											
											
Unused Space	0	0	93886	293034			Available	1615422		Percentage of useage	60%
Phase 10											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	500 Gig	750 Gig	1000 Gig	1500 Gig							
Actual usable	488392	732588	976784	1465176						3662940.00	
	1	1	1	1	Raid5a	247794	x 3	743382			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	439618	x 2	879236			
	1	1		1	Raid6d	45112	x 2	90224			
	1		1	1	Raid6e	195422	x 2	390844			
											
											
Unused Space	0	0	93886	537230			Available	2103814		Percentage of useage	57%
Phase 10											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	750 Gig	1000 Gig	1500 Gig	2000 Gig							
Actual usable	732588	976784	1465176	1953568						5128116.00	
	1	1	1	1	Raid5a	491990	x 3	1475970			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	439618	x 2	879236			
	1	1		1	Raid6d	45112	x 2	90224			
	1		1	1	Raid6e	195422	x 2	390844			
											
											
Unused Space	0	0	338082	781426			Available	2836402		Percentage of useage	55%
Phase 11											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	1000 Gig	1500 Gig	2000 Gig	2000 Gig							
Actual usable	976784	1465176	1953568	1953568						6349096.00	
	1	1	1	1	Raid5a	736186	x 3	2208558			
	1	1	1		Raid5b	64	x 2	128			
		1	1	1	Raid5c	683814	x 2	1367628			
	1	1		1	Raid6d	45112	x 2	90224			
	1		1	1	Raid6e	195422	x 2	390844			
											 
											
Unused Space	0	0	338082	293034			Available	4057382		Percentage of useage	64%
Phase 12											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	1500 Gig	2000 Gig	2000 Gig	2000 Gig							
Actual usable	1465176	1953568	1953568	1953568						7325880.00	
	1	1	1	1	Raid5a	1029220	x 3	3087660			
	1	1	1		Raid5b	45112	x 2	90224			
		1	1	1	Raid5c	683814	x 2	1367628			
	1	1		1	Raid6d	45112	x 2	90224			
	1		1	1	Raid6e	195422	x 2	390844			
											
											
Unused Space	150310	150310	0	0			Available	5026580		Percentage of useage	69%
Phase 13											
	Drive 0	 Drive 1	Drive 2	Drive 3							
	ad0	ad1	ad2	ad3							
Drive Size	2000 Gig	2000 Gig	2000 Gig	2000 Gig							
Actual usable	1953568	1953568	1953568	1953568						7814272.00	
	1	1	1	1	Raid5a	1029220	x 3	3087660			
	1	1	1		Raid5b	45112	x 2	90224			
		1	1	1	Raid5c	683814	x 2	1367628			
	1	1		1	Raid6d	45112	x 2	90224			
	1		1	1	Raid6e	195422	x 2	390844			
											
											
Unused Space	638702	150310	0	0			Available	5026580		Percentage of useage	64%