Freenas/ZFS and FreeNAS expansion

Disclaimer: Use this info at your own risk, don't come to me if all your data disappears one day. But it works for me.

Background
Want to use 4 different sized drives together as one large storage tank with 1 drive fault tolerance?

Do you also want to be able to upgrade the smallest capacity drive with a larger one without having to move all of your data to a temporary location then re-build a new array and then copying all your data back?

I've figured out how you can do it.

I've figured out how to make FreeNAS using ZFS (nightly build) to use an array of 4 drives (theoretically start with one drive but I'm not going to talk about that here) and allow the array to grow by replacing the smallest drive in the array with a larger one similarly to how a Drobo works. I'm going to describe the process then I'll do a real world example.

Summary
This is the basic idea... I will be creating five raid5 sets inside one ZFS pool. I will be creating four partitions on each of the four drives and using these partitions as the storage containers for the raid arrays. This way I can control the sizes of each partition and get the best use out of the drive space within a raid5 array. ZFS allows you to expand the size of a raid array and the method I'm describing let's you take advantage of that cool feature.

What I will be doing is slicing up the four hard drives into as many equal sized parts as I can. From there I'll build five Raid5 arrays (one for each line of partitions in the phase 1 illustration) and create a zpool called tank0 containing all five raid5 arrays.

Details
As you can see below I will start with four drives:

Drive 0 - 80 Gig

Drive 1 - 120 Gig

Drive 2 - 200 Gig

Drive 3 - 200 Gig

The zpool will be five raid5 arrays and will give me about 329 Gigs of protected storage (54% use of the total space of the drives). Then I'll take the 80 Gig drive out of the picture and put in a 250 Gig and I'll end up with the following drives:

Drive 0 - 120 Gig

Drive 1 - 200 Gig

Drive 2 - 200 Gig

Drive 3 - 250 Gig

My zpool will increase in size to about 531 Gigs of protected storage (68% use of the total space of the drives). This will be done without doing any backing up, just swapping of drives. It is a four step process and each drive must be installed wiped repartitioned and resilvered (replaced) back into the zpool. I actually moved all the drives around one at a time. But I'm not sure if I had to bother. It might be possible and easier by just swapping out the smallest drive with the largest one and repartion all the drives in place one at a time... I'll test that in the future. As for right now I've completed it the long way and it works!

Configurations
See how the configurations change below from phase 1 to phase 2 to get an idea of what I'm about to do.

Procedure
The procedure using a FreeNAS nightly build (newest experimental version):

Step 1
First thing you need to do is figure out the best use of your four drives space. Use the chart in Phase 1. I suggest using a spreadsheet to work out the best sizes for the partitions. Remember if you are doing this yourself, make sure you keep the partition sizes the same size or larger. You can't shrink them, ZFS only allows you to replace parts of a raid array with one the same size or larger.

Step 2
Using FreeNAS's GUI Mount the four disks and enable SMART monitoring Disks|Management| +

Preformatted file system set as ZFS storage pool device.

This allows FreeNAS to monitor the drive using S.M.A.R.T. and can be setup to email when a drive is showing symptoms of problems.

Step 3
Now I create raid5a1 on drive0 (ad0) using fdisk-linux (which I removed the original FreeNAS fdisk and renamed the fdisk-linux to fdisk on my FreeNAS server). Get the fdisk-linux package [here]

Step 4
Delete or rename the original /sbin/fdisk and copy the fdisk-linux to your /sbin folder. I renamed the fdisk-linux to fdisk and that is what I used for the rest of this document.

Step 5
From the command prompt:

Partition Drives
Partition Drive 0 (ad0) (delete all partitions on the drives first)

Beginning the Partitioning
freenas:/sbin# fdisk /dev/ad0 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.

The number of cylinders for this disk is set to 9964. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-9964, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-9964, default 9964): +81726M

Command (m for help): p

Disk /dev/ad0: 81.9 GB, 81964302336 bytes 255 heads, 63 sectors/track, 9964 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad0s1            1      9937  79818921   83  Linux

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (9938-9964, default 9938): Using default value 9938 Last cylinder or +size or +sizeM or +sizeK (9938-9964, default 9964): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (9947-9964, default 9947): Using default value 9947 Last cylinder or +size or +sizeM or +sizeK (9947-9964, default 9964): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Selected partition 4 First cylinder (9956-9964, default 9956): Using default value 9956 Last cylinder or +size or +sizeM or +sizeK (9956-9964, default 9964): Using default value 9964

Command (m for help): p

Disk /dev/ad0: 81.9 GB, 81964302336 bytes 255 heads, 63 sectors/track, 9964 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad0s1            1      9937  79818921   83  Linux /dev/ad0s2         9938      9946     72292+  83  Linux /dev/ad0s3         9947      9955     72292+  83  Linux /dev/ad0s4         9956      9964     72292+  83  Linux

Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad0: 81.9 GB, 81964302336 bytes 255 heads, 63 sectors/track, 9964 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad0s1            1      9937  79818921   a5  FreeBSD /dev/ad0s2         9938      9946     72292+  a5  FreeBSD /dev/ad0s3         9947      9955     72292+  a5  FreeBSD /dev/ad0s4         9956      9964     72292+  a5  FreeBSD

Command (m for help): w The partition table has been altered!

Calling ioctl to re-read partition table. Syncing disks.

Partition Drive 1 (ad1)
freenas:/sbin# fdisk /dev/ad1 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.

The number of cylinders for this disk is set to 15017. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-15017, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-15017, default 15017): +81726M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (9938-15017, default 9938): Using default value 9938 Last cylinder or +size or +sizeM or +sizeK (9938-15017, default 15017): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (9947-15017, default 9947): Using default value 9947 Last cylinder or +size or +sizeM or +sizeK (9947-15017, default 15017): +41628M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Selected partition 4 First cylinder (15009-15017, default 15009): Using default value 15009 Last cylinder or +size or +sizeM or +sizeK (15009-15017, default 15017): Using default value 15017

Command (m for help): p

Disk /dev/ad1: 123.5 GB, 123521334784 bytes 255 heads, 63 sectors/track, 15017 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad1s1            1      9937  79818921   83  Linux /dev/ad1s2         9938      9946     72292+  83  Linux /dev/ad1s3         9947     15008  40660515   83  Linux /dev/ad1s4        15009     15017     72292+  83  Linux

Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad1: 123.5 GB, 123521334784 bytes 255 heads, 63 sectors/track, 15017 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad1s1            1      9937  79818921   a5  FreeBSD /dev/ad1s2         9938      9946     72292+  a5  FreeBSD /dev/ad1s3         9947     15008  40660515   a5  FreeBSD /dev/ad1s4        15009     15017     72292+  a5  FreeBSD

Command (m for help): w The partition table has been altered!

Calling ioctl to re-read partition table. Syncing disks.

Now we Partition Drive 2 (ad2)
freenas:/sbin# fdisk /dev/ad2 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.

The number of cylinders for this disk is set to 24792. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-24792, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +81726M

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad2s1            1      9937  79818921   83  Linux

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (9938-24792, default 9938): Using default value 9938 Last cylinder or +size or +sizeM or +sizeK (9938-24792, default 24792): +64M

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad2s1            1      9937  79818921   83  Linux /dev/ad2s2         9938      9946     72292+  83  Linux

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (9947-24792, default 9947): Using default value 9947 Last cylinder or +size or +sizeM or +sizeK (9947-24792, default 24792): +41628M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Selected partition 4 First cylinder (15009-24792, default 15009): Using default value 15009 Last cylinder or +size or +sizeM or +sizeK (15009-24792, default 24792): +64M

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad2s1            1      9937  79818921   83  Linux /dev/ad2s2         9938      9946     72292+  83  Linux /dev/ad2s3         9947     15008  40660515   83  Linux /dev/ad2s4        15009     15017     72292+  83  Linux

Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad2s1            1      9937  79818921   a5  FreeBSD /dev/ad2s2         9938      9946     72292+  a5  FreeBSD /dev/ad2s3         9947     15008  40660515   a5  FreeBSD /dev/ad2s4        15009     15017     72292+  a5  FreeBSD

Command (m for help): w The partition table has been altered!

Calling ioctl to re-read partition table. Syncing disks.

Now we partition Drive3 (ad3)
freenas:/sbin# fdisk /dev/ad3 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.

The number of cylinders for this disk is set to 24792. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-24792, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +81726M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (9938-24792, default 9938): Using default value 9938 Last cylinder or +size or +sizeM or +sizeK (9938-24792, default 24792): +41628M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (15000-24792, default 15000): Using default value 15000 Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Selected partition 4 First cylinder (15009-24792, default 15009): Using default value 15009 Last cylinder or +size or +sizeM or +sizeK (15009-24792, default 24792): +64M

Command (m for help): p

Disk /dev/ad3: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad3s1            1      9937  79818921   83  Linux /dev/ad3s2         9938     14999  40660515   83  Linux /dev/ad3s3        15000     15008     72292+  83  Linux /dev/ad3s4        15009     15017     72292+  83  Linux

Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad3: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad3s1            1      9937  79818921   a5  FreeBSD /dev/ad3s2         9938     14999  40660515   a5  FreeBSD /dev/ad3s3        15000     15008     72292+  a5  FreeBSD /dev/ad3s4        15009     15017     72292+  a5  FreeBSD

Command (m for help): w The partition table has been altered!

Calling ioctl to re-read partition table. Syncing disks.

Now let’s create our zpool containing our 5 (Raid5a to Raid5e) raid5 arrays (of the partitions)
freenas:/sbin# zpool create -m /mnt/tank0 tank0 raidz1 /dev/ad0s1 /dev/ad1s1 /dev/ad2s1 /dev/ad3s1 freenas:/sbin# df -h Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    494M     87M    367M    19%    / devfs         1.0K    1.0K      0B   100%    /dev tank0         224G    128K    224G     0%    /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0

errors: No known data errors

Now we add the 2nd raid array to the same pool (tank0).
We must force since we are adding a 3 device raid to a pool that already has a 4 device raid. freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s2 /dev/ad1s2 /dev/ad2s2 freenas:/sbin# df -h Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    494M     87M    367M    19%    / devfs         1.0K    1.0K      0B   100%    /dev tank0         224G      0B    224G     0%    /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0

errors: No known data errors

Now we add the 3rd raid array:
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad1s3 /dev/ad2s3 /dev/ad3s2 freenas:/sbin# df -H Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    518M     92M    385M    19%    / devfs         1.0k    1.0k      0B   100%    /dev tank0         322G      0B    322G     0%    /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0

errors: No known data errors

Now we add the 4th raid array:
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s3 /dev/ad1s4 /dev/ad3s3 freenas:/sbin# df -H Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    518M     92M    385M    19%    / devfs         1.0k    1.0k      0B   100%    /dev tank0         322G      0B    322G     0%    /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0

errors: No known data errors

Now we add the 5th and final raid to our pool (tank0)
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s4 /dev/ad2s4 /dev/ad3s4 freenas:/sbin# df -H Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    518M     92M    385M    19%    / devfs         1.0k    1.0k      0B   100%    /dev tank0         322G      0B    322G     0%    /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors

Before we can share it on the network with SMB (samba) we have to enable writing:
freenas:/sbin# ls -la /mnt total 6 drwxr-xr-x  3 root  wheel  512 Apr 19 00:21. drwxr-xr-x 19 root  wheel  512 Apr 18 01:37 .. drwxr-xr-x  2 root  wheel    2 Apr 19 00:21 tank0 freenas:/sbin# chmod 777 /mnt/tank0/ Now add the tank0 to samba and test the speed.

Testing
Over my 100Mb/s Ethernet connection.

Writing a 1,411,699,808 byte file to tank0 over the network took 152 seconds. Which is 74 Mb/s not too bad considering all the writing to the different raid5 arrays within the pool. ZFS must do a lot in memory… Ram usage was around 25% of my 1 Gig FreeNAS box and CPU varied from 25% to 50% from my AMD Sempron(tm) Processor 3000+, running at 1800 Mhz

Reading a 1,411,699,808 byte file from tank0 over the network took 160 seconds. Which is 70 Mb/s not too bad either. Ram usage was around 14% of my 1 Gig FreeNAS box and CPU was consistently at 25% from my AMD Sempron(tm) Processor 3000+, running at 1800 Mhz

It is surprising that reading from the zpool tank0 was a little slower then writing… Either way these speeds aren’t too shabby and will do well for streaming HD quality movies or doing archival backups of your data.

Swapping Drives On The Fly
Now on to swapping out the 80 Gig drive for a 250 Gig without backing up to another drive and then rebuilding the array and copying it all back.

First scrub the zpool to make sure it’s all good.
freenas:/sbin# zpool scrub tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: scrub in progress, 12.48% done, 0h1m to go config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors After a little while the scrub completes. This will take longer with more data in the pool. freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: scrub completed with 0 errors on Sun Apr 19 01:07:58 2009 config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors

Let’s export the zpool.
freenas:/sbin# zpool export -f tank0 Shutdown the computer and swap out the 200 Gig drive3 with the new larger 250 Gig drive 3. We work our way down to drive0 in steps, swapping larger drives along the way.

With the new drive installed we have to import the pool to see it.
freenas:~# zpool import pool: tank0 id: 1762523649679414016 state: FAULTED status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://www.sun.com/msg/ZFS-8000-3C config:

tank0      UNAVAIL   insufficient replicas raidz1   DEGRADED ad1s1  ONLINE ad2s1  ONLINE ad2s1  UNAVAIL   cannot open ad3s1  ONLINE raidz1   UNAVAIL   corrupted data ad2s2  ONLINE ad2s2  UNAVAIL   cannot open ad3s2  ONLINE raidz2   DEGRADED ad2s3  UNAVAIL   cannot open ad3s3  ONLINE ad2s4  UNAVAIL   cannot open ad3s4  ONLINE

pool: tank0 id: 325956304794594593 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config:

tank0      DEGRADED raidz1   DEGRADED ad0s1  ONLINE ad1s1  ONLINE ad2s1  ONLINE ad3s1  UNAVAIL   cannot open raidz1   ONLINE ad0s2  ONLINE ad1s2  ONLINE ad2s2  ONLINE raidz1   DEGRADED ad1s3  ONLINE ad2s3  ONLINE ad3s2  UNAVAIL   cannot open raidz1   DEGRADED ad0s3  ONLINE ad1s4  ONLINE ad3s3  UNAVAIL   cannot open raidz1   DEGRADED ad0s4  ONLINE ad2s4  ONLINE ad3s4  UNAVAIL   cannot open During testing I already used the name tank0 and built a similar raid array. ZFS is trying to get all the info it can. Since they both have the same name I must import the real one using it’s ID. freenas:~# zpool import 325956304794594593 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Sun Apr 19 01:25:45 2009 config:

NAME       STATE     READ WRITE CKSUM tank0      DEGRADED     0     0     0 raidz1   DEGRADED     0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  UNAVAIL      0     0     0  cannot open raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   DEGRADED     0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  UNAVAIL      0     0     0  cannot open raidz1   DEGRADED     0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  UNAVAIL      0     0     0  cannot open raidz1   DEGRADED     0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  UNAVAIL      0     0     0  cannot open

errors: No known data errors

So far so good, now we partition Drive3 (the new 250 Gig drive) ad3 and replace the partitions into the raid arrays of our zpool.
freenas:~# fdisk /dev/ad3

The number of cylinders for this disk is set to 30401. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-30401, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-30401, default 30401): +123290M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (14991-30401, default 14991): Using default value 14991 Last cylinder or +size or +sizeM or +sizeK (14991-30401, default 30401): +80466M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (24775-30401, default 24775): Using default value 24775 Last cylinder or +size or +sizeM or +sizeK (24775-30401, default 30401): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Selected partition 4 First cylinder (24784-30401, default 24784): Using default value 24784 Last cylinder or +size or +sizeM or +sizeK (24784-30401, default 30401): +64M

Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad3: 250.0 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad3s1            1     14990 120407143+  a5  FreeBSD /dev/ad3s2        14991     24774  78589980   a5  FreeBSD /dev/ad3s3        24775     24783     72292+  a5  FreeBSD /dev/ad3s4        24784     24792     72292+  a5  FreeBSD

Command (m for help): w The partition table has been altered!

Calling ioctl to re-read partition table. Syncing disks.

Now we replace the partitions of old drive3 (200 Gig) with the new Drive3 (250 Gig).
freenas:~# zpool replace tank0 ad3s1 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver in progress, 29.02% done, 0h0m to go config:

NAME            STATE     READ WRITE CKSUM tank0           DEGRADED     0     0     0 raidz1        DEGRADED     0     0     0 ad0s1       ONLINE       0     0     0 ad1s1       ONLINE       0     0     0 ad2s1       ONLINE       0     0     0 replacing   DEGRADED     0     0     0 ad3s1/old UNAVAIL      0     0     0  cannot open ad3s1     ONLINE       0     0     0 raidz1        ONLINE       0     0     0 ad0s2       ONLINE       0     0     0 ad1s2       ONLINE       0     0     0 ad2s2       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 ad1s3       ONLINE       0     0     0 ad2s3       ONLINE       0     0     0 ad3s2       UNAVAIL      0     0     0  cannot open raidz1        DEGRADED     0     0     0 ad0s3       ONLINE       0     0     0 ad1s4       ONLINE       0     0     0 ad3s3       UNAVAIL      0     0     0  cannot open raidz1        DEGRADED     0     0     0 ad0s4       ONLINE       0     0     0 ad2s4       ONLINE       0     0     0 ad3s4       UNAVAIL      0     0     0  cannot open

errors: No known data errors

After it completes we replace the other 3 partitions:
freenas:~# zpool replace tank0 ad3s2 freenas:~# zpool replace tank0 ad3s3 freenas:~# zpool replace tank0 ad3s4 When they are complete we have a nice clean pool again: freenas:~# zpool status pool: tank0 state: ONLINE scrub: resilver completed with 0 errors on Sun Apr 19 01:46:31 2009 config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors

Testing
Copy some more files on tank0 for testing… No problem. I’ve got a 500 Meg .ZIP file on tank0 and I’ll do a test on it to make sure the data is still perfect, just to be 100% sure.

Test found no errors. As expected.

Let’s scrub again then when it completes without error, we export and shutdown the computer. Then we swap out drive 2 (200 Gig) with the old drive3 200 Gig drive, probably not needed since they are the same size, but I’m going to do it for completeness. freenas:~# zpool scrub tank0 freenas:~# zpool export –f tank0 Shutdown and swap out drives reboot with new drive2 (200 Gig): freenas:~# zpool import pool: tank0 id: 325956304794594593 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config:

tank0      DEGRADED raidz1   DEGRADED ad0s1  ONLINE ad1s1  ONLINE ad2s1  UNAVAIL   cannot open ad3s1  ONLINE raidz1   DEGRADED ad0s2  ONLINE ad1s2  ONLINE ad2s2  UNAVAIL   cannot open raidz1   DEGRADED ad1s3  ONLINE ad2s3  UNAVAIL   cannot open ad3s2  ONLINE raidz1   ONLINE ad0s3  ONLINE ad1s4  ONLINE ad3s3  ONLINE raidz1   DEGRADED ad0s4  ONLINE ad2s4  UNAVAIL   cannot open ad3s4  ONLINE

freenas:~# zpool import tank0 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Sun Apr 19 02:05:57 2009 config:

NAME       STATE     READ WRITE CKSUM tank0      DEGRADED     0     0     0 raidz1   DEGRADED     0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  UNAVAIL      0     0     0  cannot open ad3s1  ONLINE       0     0     0 raidz1   DEGRADED     0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  UNAVAIL      0     0     0  cannot open raidz1   DEGRADED     0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  UNAVAIL      0     0     0  cannot open ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   DEGRADED     0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  UNAVAIL      0     0     0  cannot open ad3s4  ONLINE       0     0     0

errors: No known data errors Now we repartition drive2 with our new sizes: freenas:~# fdisk /dev/ad2

The number of cylinders for this disk is set to 24792. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): d Partition number (1-4): 4

Command (m for help): d Partition number (1-4): 3

Command (m for help): d Partition number (1-4): 2

Command (m for help): d Selected partition 1

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-24792, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +123290M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (14991-24792, default 14991): Using default value 14991 Last cylinder or +size or +sizeM or +sizeK (14991-24792, default 24792): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (15000-24792, default 15000): Using default value 15000 Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +80466M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Selected partition 4 First cylinder (24784-24792, default 24784): Using default value 24784 Last cylinder or +size or +sizeM or +sizeK (24784-24792, default 24792): Using default value 24792

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad2s1            1     14990 120407143+  83  Linux /dev/ad2s2        14991     14999     72292+  83  Linux /dev/ad2s3        15000     24783  78589980   83  Linux /dev/ad2s4        24784     24792     72292+  83  Linux

Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad2s1            1     14990 120407143+  a5  FreeBSD /dev/ad2s2        14991     14999     72292+  a5  FreeBSD /dev/ad2s3        15000     24783  78589980   a5  FreeBSD /dev/ad2s4        24784     24792     72292+  a5  FreeBSD

Command (m for help): w The partition table has been altered!

Calling ioctl to re-read partition table. Syncing disks. Now we replace drive2’s larger partitions in our pool: freenas:~# zpool replace tank0 ad2s1 freenas:~# zpool replace tank0 ad2s2 freenas:~# zpool replace tank0 ad2s3 freenas:~# zpool replace tank0 ad2s4 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress, 27.04% done, 0h0m to go config:

NAME            STATE     READ WRITE CKSUM tank0           DEGRADED     0     0     0 raidz1        DEGRADED     0     0     0 ad0s1       ONLINE       0     0     0 ad1s1       ONLINE       0     0     0 replacing   DEGRADED     0     0     0 ad2s1/old UNAVAIL      0     0     0  cannot open ad2s1     ONLINE       0     0     0 ad3s1       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 ad0s2       ONLINE       0     0     0 ad1s2       ONLINE       0     0     0 replacing   DEGRADED     0     0     0 ad2s2/old UNAVAIL      0     0     0  cannot open ad2s2     ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 ad1s3       ONLINE       0     0     0 replacing   DEGRADED     0     0     0 ad2s3/old UNAVAIL      0     0     0  cannot open ad2s3     ONLINE       0     0     0 ad3s2       ONLINE       0     0     0 raidz1        ONLINE       0     0     0 ad0s3       ONLINE       0     0     0 ad1s4       ONLINE       0     0     0 ad3s3       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 ad0s4       ONLINE       0     0     0 replacing   DEGRADED     0     0     0 ad2s4/old UNAVAIL      0     0     0  cannot open ad2s4     ONLINE       0     0     0 ad3s4       ONLINE       0     0     0

errors: No known data errors After awhile resilvering should complete and we look clean again: freenas:~# zpool status pool: tank0 state: ONLINE scrub: resilver completed with 0 errors on Sun Apr 19 02:14:01 2009 config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors Some more quick tests, TOK as they should.

Size is still the same: freenas:~# df -H Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    518M     92M    385M    19%    / devfs         1.0k    1.0k      0B   100%    /dev tank0         322G    3.4G    319G     1%    /mnt/tank0 It will grow after the last part of any one raid array is replaced with a larger one.

So all is good now we need to swap out the Drive1 (120 Gig) drive with the old Drive2 (200 Gig) drive. So we scrub and export: freenas:~# zpool scrub tank0 Got to 97.7% then my system gave some errors (didn’t have time to read the screen) and then it rebooted.

I wonder if it’s FreeNAS’s ZFS problem or something else?

After reboot the pool tank0 came up ok by itself. Maybe it did complete then the reboot happened. I’m not 100% sure. freenas:~# zpool status pool: tank0 state: ONLINE scrub: none requested config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors Going to do another scrub… freenas:~# zpool scrub tank0 Completed without any problems this time. Weird!

Now to export and swap out the drives. freenas:~# zpool export tank0 Shutdown and swap Drive1 (120 Gig) with old Drive2 (200 Gig).

Reboot and check the zpool freenas:~# zpool status no pools available freenas:~# zpool import pool: tank0 id: 325956304794594593 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config:

tank0      DEGRADED raidz1   DEGRADED ad0s1  ONLINE ad1s1  UNAVAIL   cannot open ad2s1  ONLINE ad3s1  ONLINE raidz1   DEGRADED ad0s2  ONLINE ad1s2  UNAVAIL   cannot open ad2s2  ONLINE raidz1   DEGRADED ad1s3  UNAVAIL   cannot open ad2s3  ONLINE ad3s2  ONLINE raidz1   DEGRADED ad0s3  ONLINE ad1s4  UNAVAIL   cannot open ad3s3  ONLINE raidz1   ONLINE ad0s4  ONLINE ad2s4  ONLINE ad3s4  ONLINE Now we import our zpool freenas:~# zpool import tank0 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Sun Apr 19 09:19:02 2009 config:

NAME       STATE     READ WRITE CKSUM tank0      DEGRADED     0     0     0 raidz1   DEGRADED     0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  UNAVAIL      0     0     0  cannot open ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   DEGRADED     0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  UNAVAIL      0     0     0  cannot open ad2s2  ONLINE       0     0     0 raidz1   DEGRADED     0     0     0 ad1s3  UNAVAIL      0     0     0  cannot open ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   DEGRADED     0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  UNAVAIL      0     0     0  cannot open ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors As you can see ad1 partitions need to be replaced. But first we have to resize our new drive1 (200 Gig) partitions. freenas:~# fdisk /dev/ad1

The number of cylinders for this disk is set to 24792. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): d Partition number (1-4): 4

Command (m for help): d Partition number (1-4): 3

Command (m for help): d Partition number (1-4): 2

Command (m for help): d Selected partition 1

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-24792, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +123290M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (14991-24792, default 14991): Using default value 14991 Last cylinder or +size or +sizeM or +sizeK (14991-24792, default 24792): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (15000-24792, default 15000): Using default value 15000 Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +80466M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Selected partition 4 First cylinder (24784-24792, default 24784): Using default value 24784 Last cylinder or +size or +sizeM or +sizeK (24784-24792, default 24792): Using default value 24792

Command (m for help): p

Disk /dev/ad1: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad1s1            1     14990 120407143+  83  Linux /dev/ad1s2        14991     14999     72292+  83  Linux /dev/ad1s3        15000     24783  78589980   83  Linux /dev/ad1s4        24784     24792     72292+  83  Linux

Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): p

Disk /dev/ad1: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad1s1            1     14990 120407143+  a5  FreeBSD /dev/ad1s2        14991     14999     72292+  a5  FreeBSD /dev/ad1s3        15000     24783  78589980   a5  FreeBSD /dev/ad1s4        24784     24792     72292+  a5  FreeBSD

Command (m for help): w The partition table has been altered!

Calling ioctl to re-read partition table. Syncing disks. New partitions have been created, we can now do the zpool replace thing. freenas:~# zpool replace tank0 /dev/ad1s1 freenas:~# zpool replace tank0 /dev/ad1s2 freenas:~# zpool replace tank0 /dev/ad1s3 freenas:~# zpool replace tank0 /dev/ad1s4 I don’t wait for each partition to resilver. I type all four replace commands as soon as I can and I let it chug away. freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress, 63.24% done, 0h0m to go config:

NAME            STATE     READ WRITE CKSUM tank0           DEGRADED     0     0     0 raidz1        DEGRADED     0     0     0 ad0s1       ONLINE       0     0     0 replacing   DEGRADED     0     0     0 ad1s1/old UNAVAIL      0     0     0  cannot open ad1s1     ONLINE       0     0     0 ad2s1       ONLINE       0     0     0 ad3s1       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 ad0s2       ONLINE       0     0     0 replacing   DEGRADED     0     0     0 ad1s2/old UNAVAIL      0     0     0  cannot open ad1s2     ONLINE       0     0     0 ad2s2       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 replacing   DEGRADED     0     0     0 ad1s3/old UNAVAIL      0     0     0  cannot open ad1s3     ONLINE       0     0     0 ad2s3       ONLINE       0     0     0 ad3s2       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 ad0s3       ONLINE       0     0     0 replacing   DEGRADED     0     0     0 ad1s4/old UNAVAIL      0     0     0  cannot open ad1s4     ONLINE       0     0     0 ad3s3       ONLINE       0     0     0 raidz1        ONLINE       0     0     0 ad0s4       ONLINE       0     0     0 ad2s4       ONLINE       0     0     0 ad3s4       ONLINE       0     0     0

errors: No known data errors Once it finishes we should look like: freenas:~# zpool status pool: tank0 state: ONLINE scrub: resilver completed with 0 errors on Sun Apr 19 09:28:56 2009 config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors Still the same size: freenas:~# df -H Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    518M     92M    385M    19%    / devfs         1.0k    1.0k      0B   100%    /dev tank0         322G    3.4G    319G     1%    /mnt/tank0 Our tank0 pool still hasn’t changed Size, it should after we swap the last (fourth) drives partitions.

So now we get ready to replace the last drive. Drive0 (80 Gig, ad0) with our old drive1 (120 Gig) drive. freenas:~# zpool scrub tank0 Once it completes (check with #zpool status) we export it and shutdown the computer. freenas:~# zpool export tank0 Ok to shutdown and swap drives now.

With the last drive replaced, Drive0 (80 Gig) is Drive0 (120 Gig).

We do it all one last time: freenas:~# zpool status no pools available freenas:~# zpool import pool: tank0 id: 325956304794594593 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config:

tank0      DEGRADED raidz1   DEGRADED ad0s1  UNAVAIL   cannot open ad1s1  ONLINE ad2s1  ONLINE ad3s1  ONLINE raidz1   DEGRADED ad0s2  UNAVAIL   cannot open ad1s2  ONLINE ad2s2  ONLINE raidz1   ONLINE ad1s3  ONLINE ad2s3  ONLINE ad3s2  ONLINE raidz1   DEGRADED ad0s3  UNAVAIL   cannot open ad1s4  ONLINE ad3s3  ONLINE raidz1   DEGRADED ad0s4  UNAVAIL   cannot open ad2s4  ONLINE ad3s4  ONLINE freenas:~# zpool import tank0 Now repartition drive0 freenas:~# fdisk /dev/ad0

The number of cylinders for this disk is set to 15017. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): d Partition number (1-4): 4

Command (m for help): d Partition number (1-4): 3

Command (m for help): d Partition number (1-4): 2

Command (m for help): d Selected partition 1

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-15017, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-15017, default 15017): +123290M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (14991-15017, default 14991): Using default value 14991 Last cylinder or +size or +sizeM or +sizeK (14991-15017, default 15017): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (15000-15017, default 15000): Using default value 15000 Last cylinder or +size or +sizeM or +sizeK (15000-15017, default 15017): +64M

Command (m for help): n Command action e  extended p  primary partition (1-4) p Selected partition 4 First cylinder (15009-15017, default 15009): Using default value 15009 Last cylinder or +size or +sizeM or +sizeK (15009-15017, default 15017): Using default value 15017

Command (m for help): p

Disk /dev/ad0: 123.5 GB, 123521334784 bytes 255 heads, 63 sectors/track, 15017 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot   Start       End    Blocks   Id  System /dev/ad0s1            1     14990 120407143+  83  Linux /dev/ad0s2        14991     14999     72292+  83  Linux /dev/ad0s3        15000     15008     72292+  83  Linux /dev/ad0s4        15009     15017     72292+  83  Linux

Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD)

Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD)

Command (m for help): w The partition table has been altered!

Calling ioctl to re-read partition table. Syncing disks. Now we replace our new partitions into the pool: freenas:~# zpool replace tank0 /dev/ad0s1 freenas:~# zpool replace tank0 /dev/ad0s2 freenas:~# zpool replace tank0 /dev/ad0s3 freenas:~# zpool replace tank0 /dev/ad0s4 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress, 14.32% done, 0h1m to go config:

NAME            STATE     READ WRITE CKSUM tank0           DEGRADED     0     0     0 raidz1        DEGRADED     0     0     0 replacing   DEGRADED     0     0     0 ad0s1/old UNAVAIL      0     0     0  cannot open ad0s1     ONLINE       0     0     0 ad1s1       ONLINE       0     0     0 ad2s1       ONLINE       0     0     0 ad3s1       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 replacing   DEGRADED     0     0     0 ad0s2/old UNAVAIL      0     0     0  cannot open ad0s2     ONLINE       0     0     0 ad1s2       ONLINE       0     0     0 ad2s2       ONLINE       0     0     0 raidz1        ONLINE       0     0     0 ad1s3       ONLINE       0     0     0 ad2s3       ONLINE       0     0     0 ad3s2       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 replacing   DEGRADED     0     0     0 ad0s3/old UNAVAIL      0     0     0  cannot open ad0s3     ONLINE       0     0     0 ad1s4       ONLINE       0     0     0 ad3s3       ONLINE       0     0     0 raidz1        DEGRADED     0     0     0 replacing   DEGRADED     0     0     0 ad0s4/old UNAVAIL      0     0     0  cannot open ad0s4     ONLINE       0     0     0 ad2s4       ONLINE       0     0     0 ad3s4       ONLINE       0     0     0

errors: No known data errors Once it finishes, we should now have a nice clean pool: freenas:~# zpool status pool: tank0 state: ONLINE scrub: resilver completed with 0 errors on Sun Apr 19 09:52:42 2009 config:

NAME       STATE     READ WRITE CKSUM tank0      ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s1  ONLINE       0     0     0 ad1s1  ONLINE       0     0     0 ad2s1  ONLINE       0     0     0 ad3s1  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s2  ONLINE       0     0     0 ad1s2  ONLINE       0     0     0 ad2s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad1s3  ONLINE       0     0     0 ad2s3  ONLINE       0     0     0 ad3s2  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s3  ONLINE       0     0     0 ad1s4  ONLINE       0     0     0 ad3s3  ONLINE       0     0     0 raidz1   ONLINE       0     0     0 ad0s4  ONLINE       0     0     0 ad2s4  ONLINE       0     0     0 ad3s4  ONLINE       0     0     0

errors: No known data errors We now have more storage space and we didn’t have to find somewhere to store our old data temporarily, which will get harder to do as the storage pool grows larger and larger. We just expanded it in place.

Old Space: freenas:~# df -H Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    518M     92M    385M    19%    / devfs         1.0k    1.0k      0B   100%    /dev tank0         322G    3.4G    319G     1%    /mnt/tank0 After one last Reboot: freenas:~# df -H Filesystem    Size    Used   Avail Capacity  Mounted on /dev/da0s1a    518M     92M    385M    19%    / devfs         1.0k    1.0k      0B   100%    /dev tank0         520G    3.4G    517G     1%    /mnt/tank0 I also add all the new drives in my FreeNAS using the webgui under Disk Management and enable SMART on them so I can have SMART monitoring within FreeNAS email me if some errors on any of my drives start to show up.

The read and write times are still the same. I thought they might have gotten a little better since the new 250Gig drive is faster then the older 80Gig drive. But with all the raid work going on I guess so much is being done in RAM that drive read and write speeds aren't too heavily used. That's a good thing!

I hope this method is helpful for others as I will be using it all the time now. One thing to remember you can't swap out IDE drives to SATA drives. The reason is the SATA controllers will show up with a new device name and ZFS needs to replace the exact device name for the old one. So if you are going to do this and want to be future proof I suggest starting with four SATA drives.

Good luck, Glen Hewlett.

ps Here's a bunch more drive expansion examples (the drive sizes aren't perfect so to maximize space you might want to verify your drive sizes yourself)

Phase 1 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	80 Gig	120 Gig	200 Gig	200 Gig Actual usable	80043	120621	199148	199148						598960.00 1	1	1	1	Raid5a	79851	x 3	239553 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	40578	x 2	81156 1	1		1	Raid6d	64	x 2	128 1		1	1	Raid6e	64	x 2	128 Unused Space	0	64	78591	78591			Available	321093		Percentage of useage	54% Phase 2 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	250 Gig	120 Gig	200 Gig	200 Gig Actual usable	244196	120621	199148	199148						763113.00 1	1	1	1	Raid5a	79851	x 3	239553 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	40578	x 2	81156 1	1		1	Raid6d	64	x 2	128 1		1	1	Raid6e	78655	x 2	157310 Unused Space	85562	64	0	0			Available	478275		Percentage of useage	63%

Phase 3 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	250 Gig	300 Gig	200 Gig	200 Gig Actual usable	244196	293034	199148	199148						935526.00 1	1	1	1	Raid5a	79851	x 3	239553 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	40578	x 2	81156 1	1		1	Raid6d	64	x 2	128 1		1	1	Raid6e	78655	x 2	157310 Unused Space	85562	172477	0	0			Available	478275		Percentage of useage	51%

Phase 4 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	200 Gig	250 Gig	300 Gig	500 Gig Actual usable	199148	244196	293034	488392						1224770.00 1	1	1	1	Raid5a	153908	x 3	461724 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	90160	x 2	180320 1	1		1	Raid6d	64	x 2	128 1		1	1	Raid6e	45112	x 2	90224 Unused Space	0	0	3790	199148			Available	732524		Percentage of useage	60%

Phase 5 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	250 Gig	300 Gig	500 Gig	500 Gig Actual usable	244196	293034	488392	488392						1514014.00 1	1	1	1	Raid5a	198956	x 3	596868 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	93950	x 2	187900 1	1		1	Raid6d	64	x 2	128 1		1	1	Raid6e	45112	x 2	90224 Unused Space	0	0	150310	150310			Available	875248		Percentage of useage	58%

Phase 6 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	300 Gig	500 Gig	500 Gig	500 Gig Actual usable	293034	488392	488392	488392						1758210.00 1	1	1	1	Raid5a	247794	x 3	743382 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	195422	x 2	390844 1	1		1	Raid6d	64	x 2	128 1		1	1	Raid6e	45112	x 2	90224 Unused Space	0	45048	0	0			Available	1224706		Percentage of useage	70%

Phase 7 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	500 Gig	500 Gig	500 Gig	500 Gig Actual usable	488392	488392	488392	488392						1953568.00 1	1	1	1	Raid5a	247794	x 3	743382 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	195422	x 2	390844 1	1		1	Raid6d	64	x 2	128 1		1	1	Raid6e	45112	x 2	90224 Unused Space	195358	45048	0	0			Available	1224706		Percentage of useage	63%

Phase 8 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	500 Gig	500 Gig	500 Gig	750 Gig Actual usable	488392	488392	488392	732588						2197764.00 1	1	1	1	Raid5a	247794	x 3	743382 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	195422	x 2	390844 1	1		1	Raid6d	45112	x 2	90224 1		1	1	Raid6e	45112	x 2	90224 Unused Space	150310	0	0	199148			Available	1314802		Percentage of useage	60%

Phase 9 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	500 Gig	500 Gig	750 Gig	1000 Gig Actual usable	488392	488392	732588	976784						2686156.00 1	1	1	1	Raid5a	247794	x 3	743382 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	195422	x 2	390844 1	1		1	Raid6d	45112	x 2	90224 1		1	1	Raid6e	195422	x 2	390844 Unused Space	0	0	93886	293034			Available	1615422		Percentage of useage	60%

Phase 10 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	500 Gig	750 Gig	1000 Gig	1500 Gig Actual usable	488392	732588	976784	1465176						3662940.00 1	1	1	1	Raid5a	247794	x 3	743382 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	439618	x 2	879236 1	1		1	Raid6d	45112	x 2	90224 1		1	1	Raid6e	195422	x 2	390844 Unused Space	0	0	93886	537230			Available	2103814		Percentage of useage	57%

Phase 10 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	750 Gig	1000 Gig	1500 Gig	2000 Gig Actual usable	732588	976784	1465176	1953568						5128116.00 1	1	1	1	Raid5a	491990	x 3	1475970 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	439618	x 2	879236 1	1		1	Raid6d	45112	x 2	90224 1		1	1	Raid6e	195422	x 2	390844 Unused Space	0	0	338082	781426			Available	2836402		Percentage of useage	55%

Phase 11 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	1000 Gig	1500 Gig	2000 Gig	2000 Gig Actual usable	976784	1465176	1953568	1953568						6349096.00 1	1	1	1	Raid5a	736186	x 3	2208558 1	1	1		Raid5b	64	x 2	128 1	1	1	Raid5c	683814	x 2	1367628 1	1		1	Raid6d	45112	x 2	90224 1		1	1	Raid6e	195422	x 2	390844 Unused Space	0	0	338082	293034			Available	4057382		Percentage of useage	64%

Phase 12 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	1500 Gig	2000 Gig	2000 Gig	2000 Gig Actual usable	1465176	1953568	1953568	1953568						7325880.00 1	1	1	1	Raid5a	1029220	x 3	3087660 1	1	1		Raid5b	45112	x 2	90224 1	1	1	Raid5c	683814	x 2	1367628 1	1		1	Raid6d	45112	x 2	90224 1		1	1	Raid6e	195422	x 2	390844 Unused Space	150310	150310	0	0			Available	5026580		Percentage of useage	69%

Phase 13 Drive 0	 Drive 1	Drive 2	Drive 3 ad0	ad1	ad2	ad3 Drive Size	2000 Gig	2000 Gig	2000 Gig	2000 Gig Actual usable	1953568	1953568	1953568	1953568						7814272.00 1	1	1	1	Raid5a	1029220	x 3	3087660 1	1	1		Raid5b	45112	x 2	90224 1	1	1	Raid5c	683814	x 2	1367628 1	1		1	Raid6d	45112	x 2	90224 1		1	1	Raid6e	195422	x 2	390844 Unused Space	638702	150310	0	0			Available	5026580		Percentage of useage	64%