Firstly I would like to admit that despite reading many posts about thoroughly testing any FreeNAS setup before putting into production, I found myself in a situation where my backup server ran out of storage space over the holiday season. Unable to purchase my usual iSCSI storage arrays I thought I would throw together a FreeNAS unit from some spare kit I had lying around:
Chenbro 12 disk chassis
LSI 9211-4i SAS/SATA RAID - 4-port card
Chenbro CK22803 - 16 Port SAS Expander
DELL Precision T3500 Motherboard
Intel XEON CPU
12Gb ECC RAM
12 x 2Tb Western Digital 7200rpm disks
Additional Intel Dual 1Gbps NIC
FreeNAS 9.3-STABLE booting from USB stick on internal USB port on motherboard.
2 x configured RAIDZ2 vdev’s with 6 disks each.
I have wanted to test freeNAS in our environment for a while now after hearing great reports about ZFS, however this isn't how I planned to test it!
My backup server is a Proxmox 3.4 host with 7Tb of internal storage. I planned to extend this with the two RAIDZ2 vdevs of my new FreeNAS box. I have connected the FreeNAS box to my backup server with one of the intel nics directly on a dedicated subnet (no switch).
The server runs one Windows 2012 guest which receives encrypted backup data from several offices around the UK and writes it to local disks. Pretty much all of the disk activity is writing to storage with the occasional request to restore the odd file or folder.
I struggled to get iSCSI working between the Proxmox and FreeNas box in the limited time I had so instead I created two NFS shares (one for each 7Tb volume) on FreeNAS and mounted those in proxmox. I then created a virtual disk on the NFS mount for my windows guest and formatted it GPT in Windows.
After a very quick test to check reading and writing to the new disks I started to transfer one of our site’s data over to the new volume (around 3.5Tb). Initially this looked very promising, seeing transfer rates of around 115MB/s but after a short while this would drop down to 3MB/s and sometimes 0MB/s for several minutes before regaining full speed then dropping again. The transfer took over 3 days to complete and the volume within Windows locked up a couple of times, forcing me to reboot the server.
Resuming nightly backups for that site of around 10Gb I expected the FreeNAS box to cope as this data was coming in over the internet, however transfer to disk was still very up and down and again locked up.
I understand that I really shouldn't have put this into production without proper testing, but you (I) sometimes do silly things in desperation.
I assumed my initial transfer problem was due to me sending a large amount of data which was being cached in RAM so it appeared to be quick initially until the RAM filled up. However when running my usual backup job I would have thought the FreeNAS would easily cope with the typical data load. Is the problem down to it being one large disk image file on the FreeNAS box, rather than individual files that my Windows OS is writing?
So, where do I turn to from here to get this box performing as I would hope? I have a couple of 120Gb SSDs on hand to perhaps create a ZIL, but I believe I would be better off adding more RAM to the MAX of 24Gb for this motherboard?
Are there any built in tools to test the performance of the disk writes and controller cards?
Any suggestions would be really appreciated.
Thanks
Gordon
Chenbro 12 disk chassis
LSI 9211-4i SAS/SATA RAID - 4-port card
Chenbro CK22803 - 16 Port SAS Expander
DELL Precision T3500 Motherboard
Intel XEON CPU
12Gb ECC RAM
12 x 2Tb Western Digital 7200rpm disks
Additional Intel Dual 1Gbps NIC
FreeNAS 9.3-STABLE booting from USB stick on internal USB port on motherboard.
2 x configured RAIDZ2 vdev’s with 6 disks each.
I have wanted to test freeNAS in our environment for a while now after hearing great reports about ZFS, however this isn't how I planned to test it!
My backup server is a Proxmox 3.4 host with 7Tb of internal storage. I planned to extend this with the two RAIDZ2 vdevs of my new FreeNAS box. I have connected the FreeNAS box to my backup server with one of the intel nics directly on a dedicated subnet (no switch).
The server runs one Windows 2012 guest which receives encrypted backup data from several offices around the UK and writes it to local disks. Pretty much all of the disk activity is writing to storage with the occasional request to restore the odd file or folder.
I struggled to get iSCSI working between the Proxmox and FreeNas box in the limited time I had so instead I created two NFS shares (one for each 7Tb volume) on FreeNAS and mounted those in proxmox. I then created a virtual disk on the NFS mount for my windows guest and formatted it GPT in Windows.
After a very quick test to check reading and writing to the new disks I started to transfer one of our site’s data over to the new volume (around 3.5Tb). Initially this looked very promising, seeing transfer rates of around 115MB/s but after a short while this would drop down to 3MB/s and sometimes 0MB/s for several minutes before regaining full speed then dropping again. The transfer took over 3 days to complete and the volume within Windows locked up a couple of times, forcing me to reboot the server.
Resuming nightly backups for that site of around 10Gb I expected the FreeNAS box to cope as this data was coming in over the internet, however transfer to disk was still very up and down and again locked up.
I understand that I really shouldn't have put this into production without proper testing, but you (I) sometimes do silly things in desperation.
I assumed my initial transfer problem was due to me sending a large amount of data which was being cached in RAM so it appeared to be quick initially until the RAM filled up. However when running my usual backup job I would have thought the FreeNAS would easily cope with the typical data load. Is the problem down to it being one large disk image file on the FreeNAS box, rather than individual files that my Windows OS is writing?
So, where do I turn to from here to get this box performing as I would hope? I have a couple of 120Gb SSDs on hand to perhaps create a ZIL, but I believe I would be better off adding more RAM to the MAX of 24Gb for this motherboard?
Are there any built in tools to test the performance of the disk writes and controller cards?
Any suggestions would be really appreciated.
Thanks
Gordon
