emoulton
Cadet
- Joined
- Apr 29, 2015
- Messages
- 4
Hello, I'm Eli, previously the builder of a terrible, (but very, very lucky), storage array. Nineteen 1TB SATA disks in a Windows RAID5 set. It runs on 64 bit Windows XP, uses SATA port multipliers, and miraculously has never dropped a drive in the 6 years it's been alive. This monstrosity must be replaced!
I'm quite new to FreeBSD and ZFS. After doing lot of reading on the subject of building and setting up a storage server using ZFS, I've decided to go with FreeNAS. Hardware wise, this build is one of the more ambitious ones that I've done, and my lack of experience with FreeBSD and ZFS has made a somewhat nerve wracking build all the more stressful. I would appreciate any and all comments and advice that anyone may have regarding this build. (Especially the configuration).
First, the hardware:
Chassis: Supermicro SC846XE26-R1K28B
Motherboard: Supermicro X10SRA-F-O
CPU: Intel Xeon E5-1620 v3 3.5GHz
Backplane: Supermicro SAS3-846EL2
RAM: 4x Samsung M386A4G40DM0-CPB 32GB (128GB total)
HBA: 2x LSI 9207-4i4e (Flashed to P16 firmware)
OS Drive: Intel SSDSC2BF120H501 120GB SSD
Storage Drives: 24x WD RE WD4001FYYG 4GB Nearline SAS
L2ARC Drives: 2x Intel SSDSC2BA400G401 400GB SSD
NIC: Emulex OCe11102-NT Dual Port 10GBaseT
Switch: Cisco SG500XG-8F8T 16 Port (8 copper and 8 SFP) 10GbE
The software and configuration:
I'm running FreeNAS-9.3-STABLE-201504152200. I've got all 24 disks setup as a single volume containing four 6 disk RAIDZ2 sets. The two 400GB SSDs are being used for L2ARC for the volume. My main concern about how I configured things has to do with a non-standard swap setup, and a manual setup of SAS multipathing.
I didn't want swap on the storage pool, so I created a swap file on the OS disk. The file is stored at /root/.swap0, and I configured it to be mounted using an rc.conf tunable in the web GUI. It survives reboots, but it would be good to know if this might become an issue later.
As for the multipath setup, FreeNAS initially set it up automatically, but it was in active/passive mode, and the backplane supports active/active. I deleted the multipath setup from the command line, and manually reconfigured it. From what I could find out, it looks like the multipath setup in FreeBSD writes some metadata to the disks that stores the configuration; I'm hoping it'll be persistent. It has survived reboots, but once again, I'd like to know if this could cause me problems later.
I configured the ZFS volume on the command line so that I could make sure that the drives that I wanted in a particular RAIDZ2 set were the right ones. As far as I can tell, this shouldn't be a problem as the configuration is stored on the drives.
This system will be mainly used to store compressed video. It'll pretty much be seeing only sequential reads and writes from two 10GbE workstations, and two or three 1GbE servers. I've read that enabling lz4 compression costs almost nothing, and might as well be enabled even if the files being stored aren't very compressible. Given the workload I've described, what would you recommend?
I haven't setup the 10GbE interfaces yet, though I'll probably be doing that today.
I'd like to finish up this post by mentioning how useful I've found these forums while doing all my research. The many excellent guides, and discussions I've seen here have been a great boon. Thanks for that everyone!
I'm quite new to FreeBSD and ZFS. After doing lot of reading on the subject of building and setting up a storage server using ZFS, I've decided to go with FreeNAS. Hardware wise, this build is one of the more ambitious ones that I've done, and my lack of experience with FreeBSD and ZFS has made a somewhat nerve wracking build all the more stressful. I would appreciate any and all comments and advice that anyone may have regarding this build. (Especially the configuration).
First, the hardware:
Chassis: Supermicro SC846XE26-R1K28B
Motherboard: Supermicro X10SRA-F-O
CPU: Intel Xeon E5-1620 v3 3.5GHz
Backplane: Supermicro SAS3-846EL2
RAM: 4x Samsung M386A4G40DM0-CPB 32GB (128GB total)
HBA: 2x LSI 9207-4i4e (Flashed to P16 firmware)
OS Drive: Intel SSDSC2BF120H501 120GB SSD
Storage Drives: 24x WD RE WD4001FYYG 4GB Nearline SAS
L2ARC Drives: 2x Intel SSDSC2BA400G401 400GB SSD
NIC: Emulex OCe11102-NT Dual Port 10GBaseT
Switch: Cisco SG500XG-8F8T 16 Port (8 copper and 8 SFP) 10GbE
The software and configuration:
I'm running FreeNAS-9.3-STABLE-201504152200. I've got all 24 disks setup as a single volume containing four 6 disk RAIDZ2 sets. The two 400GB SSDs are being used for L2ARC for the volume. My main concern about how I configured things has to do with a non-standard swap setup, and a manual setup of SAS multipathing.
I didn't want swap on the storage pool, so I created a swap file on the OS disk. The file is stored at /root/.swap0, and I configured it to be mounted using an rc.conf tunable in the web GUI. It survives reboots, but it would be good to know if this might become an issue later.
As for the multipath setup, FreeNAS initially set it up automatically, but it was in active/passive mode, and the backplane supports active/active. I deleted the multipath setup from the command line, and manually reconfigured it. From what I could find out, it looks like the multipath setup in FreeBSD writes some metadata to the disks that stores the configuration; I'm hoping it'll be persistent. It has survived reboots, but once again, I'd like to know if this could cause me problems later.
I configured the ZFS volume on the command line so that I could make sure that the drives that I wanted in a particular RAIDZ2 set were the right ones. As far as I can tell, this shouldn't be a problem as the configuration is stored on the drives.
This system will be mainly used to store compressed video. It'll pretty much be seeing only sequential reads and writes from two 10GbE workstations, and two or three 1GbE servers. I've read that enabling lz4 compression costs almost nothing, and might as well be enabled even if the files being stored aren't very compressible. Given the workload I've described, what would you recommend?
I haven't setup the 10GbE interfaces yet, though I'll probably be doing that today.
I'd like to finish up this post by mentioning how useful I've found these forums while doing all my research. The many excellent guides, and discussions I've seen here have been a great boon. Thanks for that everyone!