Xeon Gold 6144. My first NAS

[TFAiSO]

Thanks Chris. If I go with the 24 drive config to max out the 10Gb-line, what would the Vdev layout look like? If I understand you right, I need 4 vdevs with 6 drives each in a single zpool (if each drive has 170MB/s).

 

[Stux]

Are you more concerned about sequential performance, random performance or capacity?

 

[TFAiSO]

I'm not so interested in random performance as multiple users will do streaming, hence sequential.
Capacity would be nice to increase a bit. Raidz2 is a given for me, but very large sets might cause a way too long rebuild time if a disk gets bad.

 

[Stux]

4x 6-way Raidz2 is a nice compromise between IOPS and capacity. An alternative is 3x 8-way RAIDZ2. 25% less IOPS but 2 extra disks of capacity.

 

[TFAiSO]

Thanks Stux. Since the rest of the machine is quite powerful, it would feel strange to strangle the IOPS capability. I’ll start with 2x6 z2. I’ll put drive 13 as hot spare.

Will do SAS IT flashing tomorrow.

.iso FN11 created without issues. Ready to install the OS and begin sw config when the SSD boot drives arrive later in the week.

 

[TFAiSO]

Guys, thanks for hanging in here. The last piece I'm stuck with in regards to the physical installation are the boot drives. Those are normal Intel SATA SSD's.

I found two SATA3 RAID ports on the motherboard called: "I-SATA 0-3" and "I-SATA 4-7" (using the Intel PCH controller)
* What kind of cable do I need to connect two disks to a single such port?
* Can I draw power from the DOM-ports on the motherboard or do I have to use a Molex-to-SATA cable converter?

 

[Chris Moore]

There should be standard SATA ports on the system board that you would use for this.
It has been a little while, did your hardware list evolve? What system board did you end up getting?
The "DOM" ports, if I recall correctly, only give 5 volts, so I don't know if they can be used for this. I wouldn't expect so.

 

[TFAiSO]

Thanks for your reply Chris.
The MB is as indicated on the first page. :)
The HBA is a Lenovo/IBM M1215.
As per the manual the only SATA ports are the two I-Sata handling up to 8 HDD’s, and 2 DOM’s. In the product page, it says there are 10 ports but I can’t find them. I must be completely blind, have I missed something?

I’ll start off with 6x2 drives (two vdev’s).

 

[Chris Moore]

Ok, I looked on the Supermicro site to see what the connectors are. The two yellow ones are regular SATA ports. You can use those for the SSD boot drive, but they will not power the drive the way they would with a SATA DOM. You will need a power cable to the drives.
I looked back at the first post and I remember there was some discussion on it, but I don't recall which chassis you went with?
Still, you should be able to get power for the SSDs, it might just need an adapter cable. What do you see?

 

[TFAiSO]

Hi there,
Now all the hardware has been installed and I have managed to install FreeNAS. I made a few mistakes after mirroring, and wanted to do a fresh install.
I reinstalled on the ada1 and booted up. Now the system says that my boot environment is DEGRADED...
I thought I made a clean installation... Does this mean my installation or system boot failed, so that it booted up from the old disk? I see Plex is running which I had running on the old drive... Perhaps the system couldn't boot from the freshly installed drive and used the second (old) one? I saw when I tried to boot the fresh installation-drive, just now, that it started off fine with GRUB etc. then all chars got garbled and the system rebooted! Perhaps this is what happened?

 

[styno]

If you want to install directly to a mirror you have to select both drives instead of only one.
If you want to reinstall to a single drive and leave the previous one in your system, you have to make sure the correct drive is selected during boot.

 

[TFAiSO]

I think one of my boot devices have gone bad. After I updated the functional FreeNAS to latest version, I now get error: file i386-pc not found. I'm on grub rescue and I have no idea how this could happen... I will detach the drives and check them.
*EDIT* Before I detach anything, I will reinstall FreeNAS using both drives upon installation from the USB media.

 

[Stux]

Thing about a FreeNAS boot Mirror, is that all though the data is mirrored, only one drive is ever booted from.

Thus if that drive goes bad your boot might fail, until you switch your bios to boot from the other half of the mirror.

Using a raid card to Impmlement the boot Mirror ironically solves this. It’s ironic because normally raid cards aren’t recommended.

 

[Chris Moore]

Well, if you have the boot order set up correctly, it'll try the first drive and then try the second drive. As long as it doesn't get hung up for some reason, it will still boot.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Stux]

The "getting hung up" bit is fairly likely when dealing with a failed disk and BIOSes which don't handle the unexpected very well unfortunately.

 

[TFAiSO]

Hi folks,

Success!

I noticed I couldn't find any of the two boot drives in BIOS, unless I went to the last SAVE/RESET menu in bios. On the last page I found a "boot override" option and my UEFI drives (yes, I installed FreeNAS as UEFI) comes up there. I haven't restarted the system so I hope the override will be saved permanently...

I followed your advice to choose both drives upon installation. Now the system show green light and is not degraded. Sweet!

If anyone wonders... I had to reinstall everything as Plex screwed up my system. I followed a howto on how to make a manual installation, plex started as root, all directories with wrong permissions and I couldn't find Plex in the GUI. Instead of spending days trying to clean it all up, I reinstalled.
This time, I will read more and make the installation of plex "slower".

I will try to search to find all the Plex howto's again... *sigh* To get a plain working vanilla Plex installation is not easy!

 

[TFAiSO]

This is what the finished installation looks like.

 

[styno]

I haven't restarted the system so I hope the override will be saved permanently...

I highly doubt that... Imho, you'd be better off testing that right now...

 

[TFAiSO]

I highly doubt that... Imho, you'd be better off testing that right now...

Styno, you were right. Boot Override is not saved. When I booted up this morning, it didn't boot up. I solved it by creating a new boot option, changed from "DUAL" to "UEFI" in the bios settings and now it boots SSD disk 1 every time, as it should.

 

[TFAiSO]

I'm back. This time I am doing it slowly but surely. Instead of installing Plex as root, I have created a user called 'red'. I picked a very simple password (like 566) and I can't login. More complex passwords doesn't help me. Even after I restart the system, I can't login as that user. Anyone spots what could be wrong here ?