Freenas/ZFS and FreeNAS expansion
(Redirected from Freenas/ZFS and FreeNAS expansion/)
Disclaimer: Use this info at your own risk, don't come to me if all your data disappears one day. But it works for me.
Background[edit | edit source]
Want to use 4 different sized drives together as one large storage tank with 1 drive fault tolerance?
Do you also want to be able to upgrade the smallest capacity drive with a larger one without having to move all of your data to a temporary location then re-build a new array and then copying all your data back?
I've figured out how you can do it.
I've figured out how to make FreeNAS using ZFS (nightly build) to use an array of 4 drives (theoretically start with one drive but I'm not going to talk about that here) and allow the array to grow by replacing the smallest drive in the array with a larger one similarly to how a Drobo works. I'm going to describe the process then I'll do a real world example.
Summary[edit | edit source]
This is the basic idea... I will be creating five raid5 sets inside one ZFS pool. I will be creating four partitions on each of the four drives and using these partitions as the storage containers for the raid arrays. This way I can control the sizes of each partition and get the best use out of the drive space within a raid5 array. ZFS allows you to expand the size of a raid array and the method I'm describing let's you take advantage of that cool feature.
What I will be doing is slicing up the four hard drives into as many equal sized parts as I can. From there I'll build five Raid5 arrays (one for each line of partitions in the phase 1 illustration) and create a zpool called tank0 containing all five raid5 arrays.
Details[edit | edit source]
As you can see below I will start with four drives:
Drive 0 - 80 Gig
Drive 1 - 120 Gig
Drive 2 - 200 Gig
Drive 3 - 200 Gig
The zpool will be five raid5 arrays and will give me about 329 Gigs of protected storage (54% use of the total space of the drives). Then I'll take the 80 Gig drive out of the picture and put in a 250 Gig and I'll end up with the following drives:
Drive 0 - 120 Gig
Drive 1 - 200 Gig
Drive 2 - 200 Gig
Drive 3 - 250 Gig
My zpool will increase in size to about 531 Gigs of protected storage (68% use of the total space of the drives). This will be done without doing any backing up, just swapping of drives. It is a four step process and each drive must be installed wiped repartitioned and resilvered (replaced) back into the zpool. I actually moved all the drives around one at a time. But I'm not sure if I had to bother. It might be possible and easier by just swapping out the smallest drive with the largest one and repartion all the drives in place one at a time... I'll test that in the future. As for right now I've completed it the long way and it works!
Configurations[edit | edit source]
See how the configurations change below from phase 1 to phase 2 to get an idea of what I'm about to do.
Phase 1[edit | edit source]
| Drive 0 | Drive 1 | Drive 2 | Drive 3 | Partition Size | ||||
| ad0 | ad1 | ad2 | ad3 | |||||
| Drive Size | 80 Gig | 120 Gig | 200 Gig | 200 Gig | ||||
| Actual usable | 81918 | 123482 | 203884 | 203884 | ||||
| ad0s1 | ad1s1 | ad2s1 | ad3s1 | Raid5a | 81726 | x 3 | 245178 | |
| ad0s2 | ad1s2 | ad2s2 | Raid5b | 64 | x 2 | 128 | ||
| ad1s3 | ad2s3 | ad3s2 | Raid5c | 41628 | x 2 | 83256 | ||
| ad0s3 | ad1s4 | ad3s3 | Raid5d | 64 | x 2 | 128 | ||
| ad0s4 | ad2s4 | ad3s4 | Raid5e | 64 | x 2 | 128 | ||
| Unused Space | 0 | 0 | 80402 | 80402 |
| Total Space (sum of all drives) | 613168 Megabytes = 613 Gigabytes |
| Total available within Zpool | 328818 Megabytes = 329 Gigabytes |
| Percent of usable storage | 54% |
Phase 2[edit | edit source]
| Drive 0 | Drive 1 | Drive 2 | Drive 3 | Partition Size | ||||
| ad0 | ad1 | ad2 | ad3 | |||||
| Drive Size | 120 Gig | 200 Gig | 200 Gig | 250 Gig | ||||
| Actual usable | 123482 | 203884 | 203884 | 250018 | ||||
| ad0s1 | ad1s1 | ad2s1 | ad3s1 | Raid5a | 123290 | x 3 | 369870 | |
| ad0s2 | ad1s2 | ad2s2 | Raid5b | 64 | x 2 | 128 | ||
| ad1s3 | ad2s3 | ad3s2 | Raid5c | 80466 | x 2 | 160932 | ||
| ad0s3 | ad1s4 | ad3s3 | Raid5d | 64 | x 2 | 128 | ||
| ad0s4 | ad2s4 | ad3s4 | Raid5e | 64 | x 2 | 128 | ||
| Unused Space | 0 | 0 | 0 | 46134 |
| Total Space (sum of all drives) | 781268 Megabytes = 781 Gigabytes |
| Total available within Zpool | 531186 Megabytes = 531 Gigabytes |
| Percent of usable storage | 68% |
Procedure[edit | edit source]
The procedure using a FreeNAS nightly build (newest experimental version):
Step 1[edit | edit source]
First thing you need to do is figure out the best use of your four drives space. Use the chart in Phase 1. I suggest using a spreadsheet to work out the best sizes for the partitions. Remember if you are doing this yourself, make sure you keep the partition sizes the same size or larger. You can't shrink them, ZFS only allows you to replace parts of a raid array with one the same size or larger.
Step 2[edit | edit source]
Using FreeNAS's GUI Mount the four disks and enable SMART monitoring Disks|Management| +
Preformatted file system set as ZFS storage pool device.
This allows FreeNAS to monitor the drive using S.M.A.R.T. and can be setup to email when a drive is showing symptoms of problems.
Step 3[edit | edit source]
Now I create raid5a1 on drive0 (ad0) using fdisk-linux (which I removed the original FreeNAS fdisk and renamed the fdisk-linux to fdisk on my FreeNAS server). Get the fdisk-linux package [here]
Step 4[edit | edit source]
Delete or rename the original /sbin/fdisk and copy the fdisk-linux to your /sbin folder. I renamed the fdisk-linux to fdisk and that is what I used for the rest of this document.
Step 5[edit | edit source]
From the command prompt:
Partition Drives[edit | edit source]
Partition Drive 0 (ad0) (delete all partitions on the drives first)
Beginning the Partitioning[edit | edit source]
freenas:/sbin# fdisk /dev/ad0
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 9964.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-9964, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-9964, default 9964): +81726M
Command (m for help): p
Disk /dev/ad0: 81.9 GB, 81964302336 bytes
255 heads, 63 sectors/track, 9964 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad0s1 1 9937 79818921 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (9938-9964, default 9938):
Using default value 9938
Last cylinder or +size or +sizeM or +sizeK (9938-9964, default 9964): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (9947-9964, default 9947):
Using default value 9947
Last cylinder or +size or +sizeM or +sizeK (9947-9964, default 9964): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (9956-9964, default 9956):
Using default value 9956
Last cylinder or +size or +sizeM or +sizeK (9956-9964, default 9964):
Using default value 9964
Command (m for help): p
Disk /dev/ad0: 81.9 GB, 81964302336 bytes
255 heads, 63 sectors/track, 9964 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad0s1 1 9937 79818921 83 Linux
/dev/ad0s2 9938 9946 72292+ 83 Linux
/dev/ad0s3 9947 9955 72292+ 83 Linux
/dev/ad0s4 9956 9964 72292+ 83 Linux
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)
Command (m for help): p
Disk /dev/ad0: 81.9 GB, 81964302336 bytes
255 heads, 63 sectors/track, 9964 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad0s1 1 9937 79818921 a5 FreeBSD
/dev/ad0s2 9938 9946 72292+ a5 FreeBSD
/dev/ad0s3 9947 9955 72292+ a5 FreeBSD
/dev/ad0s4 9956 9964 72292+ a5 FreeBSD
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Partition Drive 1 (ad1)[edit | edit source]
freenas:/sbin# fdisk /dev/ad1
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 15017.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-15017, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-15017, default 15017): +81726M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (9938-15017, default 9938):
Using default value 9938
Last cylinder or +size or +sizeM or +sizeK (9938-15017, default 15017): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (9947-15017, default 9947):
Using default value 9947
Last cylinder or +size or +sizeM or +sizeK (9947-15017, default 15017): +41628M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (15009-15017, default 15009):
Using default value 15009
Last cylinder or +size or +sizeM or +sizeK (15009-15017, default 15017):
Using default value 15017
Command (m for help): p
Disk /dev/ad1: 123.5 GB, 123521334784 bytes
255 heads, 63 sectors/track, 15017 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad1s1 1 9937 79818921 83 Linux
/dev/ad1s2 9938 9946 72292+ 83 Linux
/dev/ad1s3 9947 15008 40660515 83 Linux
/dev/ad1s4 15009 15017 72292+ 83 Linux
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)
Command (m for help): p
Disk /dev/ad1: 123.5 GB, 123521334784 bytes
255 heads, 63 sectors/track, 15017 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad1s1 1 9937 79818921 a5 FreeBSD
/dev/ad1s2 9938 9946 72292+ a5 FreeBSD
/dev/ad1s3 9947 15008 40660515 a5 FreeBSD
/dev/ad1s4 15009 15017 72292+ a5 FreeBSD
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Now we Partition Drive 2 (ad2)[edit | edit source]
freenas:/sbin# fdisk /dev/ad2
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +81726M
Command (m for help): p
Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad2s1 1 9937 79818921 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (9938-24792, default 9938):
Using default value 9938
Last cylinder or +size or +sizeM or +sizeK (9938-24792, default 24792): +64M
Command (m for help): p
Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad2s1 1 9937 79818921 83 Linux
/dev/ad2s2 9938 9946 72292+ 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (9947-24792, default 9947):
Using default value 9947
Last cylinder or +size or +sizeM or +sizeK (9947-24792, default 24792): +41628M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (15009-24792, default 15009):
Using default value 15009
Last cylinder or +size or +sizeM or +sizeK (15009-24792, default 24792): +64M
Command (m for help): p
Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad2s1 1 9937 79818921 83 Linux
/dev/ad2s2 9938 9946 72292+ 83 Linux
/dev/ad2s3 9947 15008 40660515 83 Linux
/dev/ad2s4 15009 15017 72292+ 83 Linux
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)
Command (m for help): p
Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad2s1 1 9937 79818921 a5 FreeBSD
/dev/ad2s2 9938 9946 72292+ a5 FreeBSD
/dev/ad2s3 9947 15008 40660515 a5 FreeBSD
/dev/ad2s4 15009 15017 72292+ a5 FreeBSD
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Now we partition Drive3 (ad3)[edit | edit source]
freenas:/sbin# fdisk /dev/ad3
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +81726M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (9938-24792, default 9938):
Using default value 9938
Last cylinder or +size or +sizeM or +sizeK (9938-24792, default 24792): +41628M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15000-24792, default 15000):
Using default value 15000
Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (15009-24792, default 15009):
Using default value 15009
Last cylinder or +size or +sizeM or +sizeK (15009-24792, default 24792): +64M
Command (m for help): p
Disk /dev/ad3: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad3s1 1 9937 79818921 83 Linux
/dev/ad3s2 9938 14999 40660515 83 Linux
/dev/ad3s3 15000 15008 72292+ 83 Linux
/dev/ad3s4 15009 15017 72292+ 83 Linux
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)
Command (m for help): p
Disk /dev/ad3: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad3s1 1 9937 79818921 a5 FreeBSD
/dev/ad3s2 9938 14999 40660515 a5 FreeBSD
/dev/ad3s3 15000 15008 72292+ a5 FreeBSD
/dev/ad3s4 15009 15017 72292+ a5 FreeBSD
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Now let’s create our zpool containing our 5 (Raid5a to Raid5e) raid5 arrays (of the partitions)[edit | edit source]
freenas:/sbin# zpool create -m /mnt/tank0 tank0 raidz1 /dev/ad0s1 /dev/ad1s1 /dev/ad2s1 /dev/ad3s1
freenas:/sbin# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 494M 87M 367M 19% /
devfs 1.0K 1.0K 0B 100% /dev
tank0 224G 128K 224G 0% /mnt/tank0
freenas:/sbin# zpool status
pool: tank0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
errors: No known data errors
Now we add the 2nd raid array to the same pool (tank0).[edit | edit source]
We must force since we are adding a 3 device raid to a pool that already has a 4 device raid.
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s2 /dev/ad1s2 /dev/ad2s2
freenas:/sbin# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 494M 87M 367M 19% /
devfs 1.0K 1.0K 0B 100% /dev
tank0 224G 0B 224G 0% /mnt/tank0
freenas:/sbin# zpool status
pool: tank0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
errors: No known data errors
Now we add the 3rd raid array:[edit | edit source]
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad1s3 /dev/ad2s3 /dev/ad3s2
freenas:/sbin# df -H
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 518M 92M 385M 19% /
devfs 1.0k 1.0k 0B 100% /dev
tank0 322G 0B 322G 0% /mnt/tank0
freenas:/sbin# zpool status
pool: tank0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
errors: No known data errors
Now we add the 4th raid array:[edit | edit source]
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s3 /dev/ad1s4 /dev/ad3s3
freenas:/sbin# df -H
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 518M 92M 385M 19% /
devfs 1.0k 1.0k 0B 100% /dev
tank0 322G 0B 322G 0% /mnt/tank0
freenas:/sbin# zpool status
pool: tank0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
errors: No known data errors
Now we add the 5th and final raid to our pool (tank0)[edit | edit source]
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s4 /dev/ad2s4 /dev/ad3s4
freenas:/sbin# df -H
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 518M 92M 385M 19% /
devfs 1.0k 1.0k 0B 100% /dev
tank0 322G 0B 322G 0% /mnt/tank0
freenas:/sbin# zpool status
pool: tank0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
[edit | edit source]
freenas:/sbin# ls -la /mnt total 6 drwxr-xr-x 3 root wheel 512 Apr 19 00:21 . drwxr-xr-x 19 root wheel 512 Apr 18 01:37 .. drwxr-xr-x 2 root wheel 2 Apr 19 00:21 tank0 freenas:/sbin# chmod 777 /mnt/tank0/
Now add the tank0 to samba and test the speed.
Testing[edit | edit source]
Over my 100Mb/s Ethernet connection.
Writing a 1,411,699,808 byte file to tank0 over the network took 152 seconds. Which is 74 Mb/s not too bad considering all the writing to the different raid5 arrays within the pool. ZFS must do a lot in memory… Ram usage was around 25% of my 1 Gig FreeNAS box and CPU varied from 25% to 50% from my AMD Sempron(tm) Processor 3000+, running at 1800 Mhz
Reading a 1,411,699,808 byte file from tank0 over the network took 160 seconds. Which is 70 Mb/s not too bad either. Ram usage was around 14% of my 1 Gig FreeNAS box and CPU was consistently at 25% from my AMD Sempron(tm) Processor 3000+, running at 1800 Mhz
It is surprising that reading from the zpool tank0 was a little slower then writing… Either way these speeds aren’t too shabby and will do well for streaming HD quality movies or doing archival backups of your data.
Swapping Drives On The Fly[edit | edit source]
Now on to swapping out the 80 Gig drive for a 250 Gig without backing up to another drive and then rebuilding the array and copying it all back.
First scrub the zpool to make sure it’s all good.[edit | edit source]
freenas:/sbin# zpool scrub tank0
freenas:/sbin# zpool status
pool: tank0
state: ONLINE
scrub: scrub in progress, 12.48% done, 0h1m to go
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
After a little while the scrub completes. This will take longer with more data in the pool.
freenas:/sbin# zpool status
pool: tank0
state: ONLINE
scrub: scrub completed with 0 errors on Sun Apr 19 01:07:58 2009
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
Let’s export the zpool.[edit | edit source]
freenas:/sbin# zpool export -f tank0
Shutdown the computer and swap out the 200 Gig drive3 with the new larger 250 Gig drive 3. We work our way down to drive0 in steps, swapping larger drives along the way.
With the new drive installed we have to import the pool to see it.[edit | edit source]
freenas:~# zpool import
pool: tank0
id: 1762523649679414016
state: FAULTED
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://www.sun.com/msg/ZFS-8000-3C
config:
tank0 UNAVAIL insufficient replicas
raidz1 DEGRADED
ad1s1 ONLINE
ad2s1 ONLINE
ad2s1 UNAVAIL cannot open
ad3s1 ONLINE
raidz1 UNAVAIL corrupted data
ad2s2 ONLINE
ad2s2 UNAVAIL cannot open
ad3s2 ONLINE
raidz2 DEGRADED
ad2s3 UNAVAIL cannot open
ad3s3 ONLINE
ad2s4 UNAVAIL cannot open
ad3s4 ONLINE
pool: tank0
id: 325956304794594593
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:
tank0 DEGRADED
raidz1 DEGRADED
ad0s1 ONLINE
ad1s1 ONLINE
ad2s1 ONLINE
ad3s1 UNAVAIL cannot open
raidz1 ONLINE
ad0s2 ONLINE
ad1s2 ONLINE
ad2s2 ONLINE
raidz1 DEGRADED
ad1s3 ONLINE
ad2s3 ONLINE
ad3s2 UNAVAIL cannot open
raidz1 DEGRADED
ad0s3 ONLINE
ad1s4 ONLINE
ad3s3 UNAVAIL cannot open
raidz1 DEGRADED
ad0s4 ONLINE
ad2s4 ONLINE
ad3s4 UNAVAIL cannot open
During testing I already used the name tank0 and built a similar raid array. ZFS is trying to get all the info it can. Since they both have the same name I must import the real one using it’s ID.
freenas:~# zpool import 325956304794594593
freenas:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Sun Apr 19 01:25:45 2009
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 UNAVAIL 0 0 0 cannot open
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 UNAVAIL 0 0 0 cannot open
raidz1 DEGRADED 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 UNAVAIL 0 0 0 cannot open
raidz1 DEGRADED 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 UNAVAIL 0 0 0 cannot open
errors: No known data errors
So far so good, now we partition Drive3 (the new 250 Gig drive) ad3 and replace the partitions into the raid arrays of our zpool.[edit | edit source]
freenas:~# fdisk /dev/ad3
The number of cylinders for this disk is set to 30401.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-30401, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-30401, default 30401): +123290M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14991-30401, default 14991):
Using default value 14991
Last cylinder or +size or +sizeM or +sizeK (14991-30401, default 30401): +80466M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (24775-30401, default 24775):
Using default value 24775
Last cylinder or +size or +sizeM or +sizeK (24775-30401, default 30401): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (24784-30401, default 24784):
Using default value 24784
Last cylinder or +size or +sizeM or +sizeK (24784-30401, default 30401): +64M
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)
Command (m for help): p
Disk /dev/ad3: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad3s1 1 14990 120407143+ a5 FreeBSD
/dev/ad3s2 14991 24774 78589980 a5 FreeBSD
/dev/ad3s3 24775 24783 72292+ a5 FreeBSD
/dev/ad3s4 24784 24792 72292+ a5 FreeBSD
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Now we replace the partitions of old drive3 (200 Gig) with the new Drive3 (250 Gig).[edit | edit source]
freenas:~# zpool replace tank0 ad3s1
freenas:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver in progress, 29.02% done, 0h0m to go
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad3s1/old UNAVAIL 0 0 0 cannot open
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 UNAVAIL 0 0 0 cannot open
raidz1 DEGRADED 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 UNAVAIL 0 0 0 cannot open
raidz1 DEGRADED 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 UNAVAIL 0 0 0 cannot open
errors: No known data errors
After it completes we replace the other 3 partitions:[edit | edit source]
freenas:~# zpool replace tank0 ad3s2 freenas:~# zpool replace tank0 ad3s3 freenas:~# zpool replace tank0 ad3s4
When they are complete we have a nice clean pool again:
freenas:~# zpool status
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Sun Apr 19 01:46:31 2009
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
Testing[edit | edit source]
Copy some more files on tank0 for testing… No problem. I’ve got a 500 Meg .ZIP file on tank0 and I’ll do a test on it to make sure the data is still perfect, just to be 100% sure.
Test found no errors. As expected.
Let’s scrub again then when it completes without error, we export and shutdown the computer. Then we swap out drive 2 (200 Gig) with the old drive3 200 Gig drive, probably not needed since they are the same size, but I’m going to do it for completeness.
freenas:~# zpool scrub tank0 freenas:~# zpool export –f tank0
Shutdown and swap out drives reboot with new drive2 (200 Gig):
freenas:~# zpool import
pool: tank0
id: 325956304794594593
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:
tank0 DEGRADED
raidz1 DEGRADED
ad0s1 ONLINE
ad1s1 ONLINE
ad2s1 UNAVAIL cannot open
ad3s1 ONLINE
raidz1 DEGRADED
ad0s2 ONLINE
ad1s2 ONLINE
ad2s2 UNAVAIL cannot open
raidz1 DEGRADED
ad1s3 ONLINE
ad2s3 UNAVAIL cannot open
ad3s2 ONLINE
raidz1 ONLINE
ad0s3 ONLINE
ad1s4 ONLINE
ad3s3 ONLINE
raidz1 DEGRADED
ad0s4 ONLINE
ad2s4 UNAVAIL cannot open
ad3s4 ONLINE
freenas:~# zpool import tank0
freenas:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Sun Apr 19 02:05:57 2009
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 UNAVAIL 0 0 0 cannot open
ad3s1 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 UNAVAIL 0 0 0 cannot open
raidz1 DEGRADED 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 UNAVAIL 0 0 0 cannot open
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 UNAVAIL 0 0 0 cannot open
ad3s4 ONLINE 0 0 0
errors: No known data errors
Now we repartition drive2 with our new sizes:
freenas:~# fdisk /dev/ad2
The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): d
Partition number (1-4): 4
Command (m for help): d
Partition number (1-4): 3
Command (m for help): d
Partition number (1-4): 2
Command (m for help): d
Selected partition 1
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +123290M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14991-24792, default 14991):
Using default value 14991
Last cylinder or +size or +sizeM or +sizeK (14991-24792, default 24792): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15000-24792, default 15000):
Using default value 15000
Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +80466M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (24784-24792, default 24784):
Using default value 24784
Last cylinder or +size or +sizeM or +sizeK (24784-24792, default 24792):
Using default value 24792
Command (m for help): p
Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad2s1 1 14990 120407143+ 83 Linux
/dev/ad2s2 14991 14999 72292+ 83 Linux
/dev/ad2s3 15000 24783 78589980 83 Linux
/dev/ad2s4 24784 24792 72292+ 83 Linux
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)
Command (m for help): p
Disk /dev/ad2: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad2s1 1 14990 120407143+ a5 FreeBSD
/dev/ad2s2 14991 14999 72292+ a5 FreeBSD
/dev/ad2s3 15000 24783 78589980 a5 FreeBSD
/dev/ad2s4 24784 24792 72292+ a5 FreeBSD
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Now we replace drive2’s larger partitions in our pool:
freenas:~# zpool replace tank0 ad2s1
freenas:~# zpool replace tank0 ad2s2
freenas:~# zpool replace tank0 ad2s3
freenas:~# zpool replace tank0 ad2s4
freenas:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 27.04% done, 0h0m to go
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad2s1/old UNAVAIL 0 0 0 cannot open
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad2s2/old UNAVAIL 0 0 0 cannot open
ad2s2 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad1s3 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad2s3/old UNAVAIL 0 0 0 cannot open
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad0s4 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad2s4/old UNAVAIL 0 0 0 cannot open
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
After awhile resilvering should complete and we look clean again:
freenas:~# zpool status
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Sun Apr 19 02:14:01 2009
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
Some more quick tests, TOK as they should.
Size is still the same:
freenas:~# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 322G 3.4G 319G 1% /mnt/tank0
It will grow after the last part of any one raid array is replaced with a larger one.
So all is good now we need to swap out the Drive1 (120 Gig) drive with the old Drive2 (200 Gig) drive. So we scrub and export:
freenas:~# zpool scrub tank0
Got to 97.7% then my system gave some errors (didn’t have time to read the screen) and then it rebooted.
I wonder if it’s FreeNAS’s ZFS problem or something else?
After reboot the pool tank0 came up ok by itself. Maybe it did complete then the reboot happened. I’m not 100% sure.
freenas:~# zpool status
pool: tank0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
Going to do another scrub…
freenas:~# zpool scrub tank0
Completed without any problems this time. Weird!
Now to export and swap out the drives.
freenas:~# zpool export tank0
Shutdown and swap Drive1 (120 Gig) with old Drive2 (200 Gig).
Reboot and check the zpool
freenas:~# zpool status
no pools available
freenas:~# zpool import
pool: tank0
id: 325956304794594593
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:
tank0 DEGRADED
raidz1 DEGRADED
ad0s1 ONLINE
ad1s1 UNAVAIL cannot open
ad2s1 ONLINE
ad3s1 ONLINE
raidz1 DEGRADED
ad0s2 ONLINE
ad1s2 UNAVAIL cannot open
ad2s2 ONLINE
raidz1 DEGRADED
ad1s3 UNAVAIL cannot open
ad2s3 ONLINE
ad3s2 ONLINE
raidz1 DEGRADED
ad0s3 ONLINE
ad1s4 UNAVAIL cannot open
ad3s3 ONLINE
raidz1 ONLINE
ad0s4 ONLINE
ad2s4 ONLINE
ad3s4 ONLINE
Now we import our zpool
freenas:~# zpool import tank0
freenas:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Sun Apr 19 09:19:02 2009
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 UNAVAIL 0 0 0 cannot open
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 UNAVAIL 0 0 0 cannot open
ad2s2 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad1s3 UNAVAIL 0 0 0 cannot open
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 UNAVAIL 0 0 0 cannot open
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
As you can see ad1 partitions need to be replaced. But first we have to resize our new drive1 (200 Gig) partitions.
freenas:~# fdisk /dev/ad1
The number of cylinders for this disk is set to 24792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): d
Partition number (1-4): 4
Command (m for help): d
Partition number (1-4): 3
Command (m for help): d
Partition number (1-4): 2
Command (m for help): d
Selected partition 1
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24792, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +123290M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14991-24792, default 14991):
Using default value 14991
Last cylinder or +size or +sizeM or +sizeK (14991-24792, default 24792): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15000-24792, default 15000):
Using default value 15000
Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +80466M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (24784-24792, default 24784):
Using default value 24784
Last cylinder or +size or +sizeM or +sizeK (24784-24792, default 24792):
Using default value 24792
Command (m for help): p
Disk /dev/ad1: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad1s1 1 14990 120407143+ 83 Linux
/dev/ad1s2 14991 14999 72292+ 83 Linux
/dev/ad1s3 15000 24783 78589980 83 Linux
/dev/ad1s4 24784 24792 72292+ 83 Linux
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)
Command (m for help): p
Disk /dev/ad1: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad1s1 1 14990 120407143+ a5 FreeBSD
/dev/ad1s2 14991 14999 72292+ a5 FreeBSD
/dev/ad1s3 15000 24783 78589980 a5 FreeBSD
/dev/ad1s4 24784 24792 72292+ a5 FreeBSD
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
New partitions have been created, we can now do the zpool replace thing.
freenas:~# zpool replace tank0 /dev/ad1s1 freenas:~# zpool replace tank0 /dev/ad1s2 freenas:~# zpool replace tank0 /dev/ad1s3 freenas:~# zpool replace tank0 /dev/ad1s4
I don’t wait for each partition to resilver. I type all four replace commands as soon as I can and I let it chug away.
freenas:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 63.24% done, 0h0m to go
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad0s1 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad1s1/old UNAVAIL 0 0 0 cannot open
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad0s2 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad1s2/old UNAVAIL 0 0 0 cannot open
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
replacing DEGRADED 0 0 0
ad1s3/old UNAVAIL 0 0 0 cannot open
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ad0s3 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad1s4/old UNAVAIL 0 0 0 cannot open
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
Once it finishes we should look like:
freenas:~# zpool status
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Sun Apr 19 09:28:56 2009
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
Still the same size:
freenas:~# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 322G 3.4G 319G 1% /mnt/tank0
Our tank0 pool still hasn’t changed Size, it should after we swap the last (fourth) drives partitions.
So now we get ready to replace the last drive. Drive0 (80 Gig, ad0) with our old drive1 (120 Gig) drive.
freenas:~# zpool scrub tank0
Once it completes (check with #zpool status) we export it and shutdown the computer.
freenas:~# zpool export tank0
Ok to shutdown and swap drives now.
With the last drive replaced, Drive0 (80 Gig) is Drive0 (120 Gig).
We do it all one last time:
freenas:~# zpool status
no pools available
freenas:~# zpool import
pool: tank0
id: 325956304794594593
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:
tank0 DEGRADED
raidz1 DEGRADED
ad0s1 UNAVAIL cannot open
ad1s1 ONLINE
ad2s1 ONLINE
ad3s1 ONLINE
raidz1 DEGRADED
ad0s2 UNAVAIL cannot open
ad1s2 ONLINE
ad2s2 ONLINE
raidz1 ONLINE
ad1s3 ONLINE
ad2s3 ONLINE
ad3s2 ONLINE
raidz1 DEGRADED
ad0s3 UNAVAIL cannot open
ad1s4 ONLINE
ad3s3 ONLINE
raidz1 DEGRADED
ad0s4 UNAVAIL cannot open
ad2s4 ONLINE
ad3s4 ONLINE
freenas:~# zpool import tank0
Now repartition drive0
freenas:~# fdisk /dev/ad0
The number of cylinders for this disk is set to 15017.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): d
Partition number (1-4): 4
Command (m for help): d
Partition number (1-4): 3
Command (m for help): d
Partition number (1-4): 2
Command (m for help): d
Selected partition 1
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-15017, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-15017, default 15017): +123290M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14991-15017, default 14991):
Using default value 14991
Last cylinder or +size or +sizeM or +sizeK (14991-15017, default 15017): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15000-15017, default 15000):
Using default value 15000
Last cylinder or +size or +sizeM or +sizeK (15000-15017, default 15017): +64M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (15009-15017, default 15009):
Using default value 15009
Last cylinder or +size or +sizeM or +sizeK (15009-15017, default 15017):
Using default value 15017
Command (m for help): p
Disk /dev/ad0: 123.5 GB, 123521334784 bytes
255 heads, 63 sectors/track, 15017 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/ad0s1 1 14990 120407143+ 83 Linux
/dev/ad0s2 14991 14999 72292+ 83 Linux
/dev/ad0s3 15000 15008 72292+ 83 Linux
/dev/ad0s4 15009 15017 72292+ 83 Linux
Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): a5
Changed system type of partition 2 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): a5
Changed system type of partition 3 to a5 (FreeBSD)
Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): a5
Changed system type of partition 4 to a5 (FreeBSD)
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Now we replace our new partitions into the pool:
freenas:~# zpool replace tank0 /dev/ad0s1
freenas:~# zpool replace tank0 /dev/ad0s2
freenas:~# zpool replace tank0 /dev/ad0s3
freenas:~# zpool replace tank0 /dev/ad0s4
freenas:~# zpool status
pool: tank0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 14.32% done, 0h1m to go
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
replacing DEGRADED 0 0 0
ad0s1/old UNAVAIL 0 0 0 cannot open
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
replacing DEGRADED 0 0 0
ad0s2/old UNAVAIL 0 0 0 cannot open
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
replacing DEGRADED 0 0 0
ad0s3/old UNAVAIL 0 0 0 cannot open
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
replacing DEGRADED 0 0 0
ad0s4/old UNAVAIL 0 0 0 cannot open
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
Once it finishes, we should now have a nice clean pool:
freenas:~# zpool status
pool: tank0
state: ONLINE
scrub: resilver completed with 0 errors on Sun Apr 19 09:52:42 2009
config:
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s1 ONLINE 0 0 0
ad1s1 ONLINE 0 0 0
ad2s1 ONLINE 0 0 0
ad3s1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s2 ONLINE 0 0 0
ad1s2 ONLINE 0 0 0
ad2s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1s3 ONLINE 0 0 0
ad2s3 ONLINE 0 0 0
ad3s2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s3 ONLINE 0 0 0
ad1s4 ONLINE 0 0 0
ad3s3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0s4 ONLINE 0 0 0
ad2s4 ONLINE 0 0 0
ad3s4 ONLINE 0 0 0
errors: No known data errors
We now have more storage space and we didn’t have to find somewhere to store our old data temporarily, which will get harder to do as the storage pool grows larger and larger. We just expanded it in place.
Old Space:
freenas:~# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 322G 3.4G 319G 1% /mnt/tank0
After one last Reboot:
freenas:~# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 520G 3.4G 517G 1% /mnt/tank0
I also add all the new drives in my FreeNAS using the webgui under Disk Management and enable SMART on them so I can have SMART monitoring within FreeNAS email me if some errors on any of my drives start to show up.
The read and write times are still the same. I thought they might have gotten a little better since the new 250Gig drive is faster then the older 80Gig drive. But with all the raid work going on I guess so much is being done in RAM that drive read and write speeds aren't too heavily used. That's a good thing!
I hope this method is helpful for others as I will be using it all the time now. One thing to remember you can't swap out IDE drives to SATA drives. The reason is the SATA controllers will show up with a new device name and ZFS needs to replace the exact device name for the old one. So if you are going to do this and want to be future proof I suggest starting with four SATA drives.
Good luck, Glen Hewlett.
ps Here's a bunch more drive expansion examples (the drive sizes aren't perfect so to maximize space you might want to verify your drive sizes yourself)
Phase 1 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 80 Gig 120 Gig 200 Gig 200 Gig Actual usable 80043 120621 199148 199148 598960.00 1 1 1 1 Raid5a 79851 x 3 239553 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 40578 x 2 81156 1 1 1 Raid6d 64 x 2 128 1 1 1 Raid6e 64 x 2 128 Unused Space 0 64 78591 78591 Available 321093 Percentage of useage 54%
Phase 2 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 250 Gig 120 Gig 200 Gig 200 Gig Actual usable 244196 120621 199148 199148 763113.00 1 1 1 1 Raid5a 79851 x 3 239553 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 40578 x 2 81156 1 1 1 Raid6d 64 x 2 128 1 1 1 Raid6e 78655 x 2 157310 Unused Space 85562 64 0 0 Available 478275 Percentage of useage 63%
Phase 3 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 250 Gig 300 Gig 200 Gig 200 Gig Actual usable 244196 293034 199148 199148 935526.00 1 1 1 1 Raid5a 79851 x 3 239553 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 40578 x 2 81156 1 1 1 Raid6d 64 x 2 128 1 1 1 Raid6e 78655 x 2 157310 Unused Space 85562 172477 0 0 Available 478275 Percentage of useage 51%
Phase 4 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 200 Gig 250 Gig 300 Gig 500 Gig Actual usable 199148 244196 293034 488392 1224770.00 1 1 1 1 Raid5a 153908 x 3 461724 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 90160 x 2 180320 1 1 1 Raid6d 64 x 2 128 1 1 1 Raid6e 45112 x 2 90224 Unused Space 0 0 3790 199148 Available 732524 Percentage of useage 60%
Phase 5 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 250 Gig 300 Gig 500 Gig 500 Gig Actual usable 244196 293034 488392 488392 1514014.00 1 1 1 1 Raid5a 198956 x 3 596868 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 93950 x 2 187900 1 1 1 Raid6d 64 x 2 128 1 1 1 Raid6e 45112 x 2 90224 Unused Space 0 0 150310 150310 Available 875248 Percentage of useage 58%
Phase 6 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 300 Gig 500 Gig 500 Gig 500 Gig Actual usable 293034 488392 488392 488392 1758210.00 1 1 1 1 Raid5a 247794 x 3 743382 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 195422 x 2 390844 1 1 1 Raid6d 64 x 2 128 1 1 1 Raid6e 45112 x 2 90224 Unused Space 0 45048 0 0 Available 1224706 Percentage of useage 70%
Phase 7 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 500 Gig 500 Gig 500 Gig 500 Gig Actual usable 488392 488392 488392 488392 1953568.00 1 1 1 1 Raid5a 247794 x 3 743382 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 195422 x 2 390844 1 1 1 Raid6d 64 x 2 128 1 1 1 Raid6e 45112 x 2 90224 Unused Space 195358 45048 0 0 Available 1224706 Percentage of useage 63%
Phase 8 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 500 Gig 500 Gig 500 Gig 750 Gig Actual usable 488392 488392 488392 732588 2197764.00 1 1 1 1 Raid5a 247794 x 3 743382 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 195422 x 2 390844 1 1 1 Raid6d 45112 x 2 90224 1 1 1 Raid6e 45112 x 2 90224 Unused Space 150310 0 0 199148 Available 1314802 Percentage of useage 60%
Phase 9 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 500 Gig 500 Gig 750 Gig 1000 Gig Actual usable 488392 488392 732588 976784 2686156.00 1 1 1 1 Raid5a 247794 x 3 743382 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 195422 x 2 390844 1 1 1 Raid6d 45112 x 2 90224 1 1 1 Raid6e 195422 x 2 390844 Unused Space 0 0 93886 293034 Available 1615422 Percentage of useage 60%
Phase 10 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 500 Gig 750 Gig 1000 Gig 1500 Gig Actual usable 488392 732588 976784 1465176 3662940.00 1 1 1 1 Raid5a 247794 x 3 743382 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 439618 x 2 879236 1 1 1 Raid6d 45112 x 2 90224 1 1 1 Raid6e 195422 x 2 390844 Unused Space 0 0 93886 537230 Available 2103814 Percentage of useage 57%
Phase 10 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 750 Gig 1000 Gig 1500 Gig 2000 Gig Actual usable 732588 976784 1465176 1953568 5128116.00 1 1 1 1 Raid5a 491990 x 3 1475970 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 439618 x 2 879236 1 1 1 Raid6d 45112 x 2 90224 1 1 1 Raid6e 195422 x 2 390844 Unused Space 0 0 338082 781426 Available 2836402 Percentage of useage 55%
Phase 11 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 1000 Gig 1500 Gig 2000 Gig 2000 Gig Actual usable 976784 1465176 1953568 1953568 6349096.00 1 1 1 1 Raid5a 736186 x 3 2208558 1 1 1 Raid5b 64 x 2 128 1 1 1 Raid5c 683814 x 2 1367628 1 1 1 Raid6d 45112 x 2 90224 1 1 1 Raid6e 195422 x 2 390844 Unused Space 0 0 338082 293034 Available 4057382 Percentage of useage 64%
Phase 12 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 1500 Gig 2000 Gig 2000 Gig 2000 Gig Actual usable 1465176 1953568 1953568 1953568 7325880.00 1 1 1 1 Raid5a 1029220 x 3 3087660 1 1 1 Raid5b 45112 x 2 90224 1 1 1 Raid5c 683814 x 2 1367628 1 1 1 Raid6d 45112 x 2 90224 1 1 1 Raid6e 195422 x 2 390844 Unused Space 150310 150310 0 0 Available 5026580 Percentage of useage 69%
Phase 13 Drive 0 Drive 1 Drive 2 Drive 3 ad0 ad1 ad2 ad3 Drive Size 2000 Gig 2000 Gig 2000 Gig 2000 Gig Actual usable 1953568 1953568 1953568 1953568 7814272.00 1 1 1 1 Raid5a 1029220 x 3 3087660 1 1 1 Raid5b 45112 x 2 90224 1 1 1 Raid5c 683814 x 2 1367628 1 1 1 Raid6d 45112 x 2 90224 1 1 1 Raid6e 195422 x 2 390844 Unused Space 638702 150310 0 0 Available 5026580 Percentage of useage 64%