BUILD C226, i3, 10 drive build

[Richman]

The 11 disk rule has mathematical basis for the point at which the pool is so wide that small writes takes excessively long to perform.

I'm not clear on what cj's point is either, but I'm kinda tired and foggy.

I am not clear on it either as it doesn't make sense in my brain. If the data is so small that it ca't be split up across all disks and only uses a couple like three, then why would it take .......cj quote, "excessively long to perform"?

 

[indy]

All right, CPU (1220v3), MB (X10SL7-F) and RAM (2x KVR16LE11/8I) are ordered.
Unfortunately on backorder because the Supermicro MB and that specific RAM is kind of hard to come by, especially from the same vendor.
However I am not really in a hurry anyway.

As soon as that order ships, my next order will be:
Fractal Define R4 PCGH Edition
Seasonic 400W Platinum Fanless
Cyberpower DL450ELCD

Regarding the UPS:
Having read some unfortunate threads about ZFS and power failures I definitely want a UPS from the start.
It does not have to be anything special, just enough to give the server time to do a shutdown.
Good efficiency and low noise on AC power would be great though.
The Cyberpower is very cheap (~55€) and should at least be usb-hid compatible.
Do you guys see anything immediately wrong with it?
Manual: http://eu.cyberpowersystems.com/usermanual/download/DL450-650-850ELCD_EN.pdf

 

[cyberjock]

I am not clear on it either as it doesn't make sense in my brain. If the data is so small that it ca't be split up across all disks and only uses a couple like three, then why would it take .......cj quote, "excessively long to perform"?

You have increased overhead from writing small transactions individually.

 

[jyavenard]

All right, CPU (1220v3), MB (X10SL7-F) and RAM (2x KVR16LE11/8I) are ordered.
Unfortunately on backorder because the Supermicro MB and that specific RAM is kind of hard to come by, especially from the same vendor.

Why did you order that specific RAM? that's just generic server RAM; couldn't you have found more available model instead?

SM Kingston certified RAM for this model is: KVR16E11/8EF

 

[indy]

The RAM is low voltage and Intel certified, I am not expecting a huge delay.

 

[indy]

Still waiting on the hardware...

In the meantime here is another look at the hard drive configuration from a cost perspective:

(Note that I did not include the additional hardware required to run the server.)
Some observations:
- I thought the different setups would be much farther apart in terms of GB/€.
- My preferred setup of 8x 3TB comes in second place and has the best ratio under 1000€.

 

[jgreco]

And why didn't you include the cost of the server hardware? Since you're going to pay that as essentially a flat tax to have a NAS, that weighs in very heavily as a component of the cost-per-usable-TB computation.

 

[indy]

Because the hard drive configuration does not affect the other components in my case.
The problem is that I can not project my future storage needs.
My solution is to get a solid foundation for the server and buy the cheapest hard drive space on top of that.
If/when my storage needs outgrow that server I will (probably) only change the hard drive configuration.

 

[jgreco]

Experiment for you:

Try summing up your system with the cost of your server hardware.

Now sum it up again, taking that total and adding to it the cheapest cost that you think the future hard drive expansion might be had.

Now divide both numbers by usable TB's achieved.

The latter option is almost always significantly more expensive. The additional cost of doing it right, now, is almost always rewarded in the end, unless you are absolutely certain you only need X amount of space and X is the number you're building for today.

 

[indy]

Approaching this from the other side, I could buy 50TB of space now for a very good overall space/cost ratio.
I would probably never use a fraction of it and that high initial cost would never see a return of investment.
As always, its not easy to find a happy medium.
I agree though that my approach and the graphic I made is oversimplifying things.

On a sidenote: If only zfs had the ability to expand vdevs we would not even have this argument ;)

 

[indy]

Lets hope this does not happen too often on the production server ;)

 

[indy]

Progress!
X10SL7-F and ram (2x KVR16LE11/8KF) are underway.

Only thing missing are the disks, still a bit undecided there.
My plan is to order 8x 3TB Red.
Then benchmark 3 of them striped on the Intel controller and decide between an 8 disk raidz2 or 11 disk raidz3.

 

[jyavenard]

8 disks in raidz2 isn't an ideal combination as far as performance go. 6 or 10 is

 

[indy]

My best guess is 8 disks will outperform 6.
But I will benchmark a couple of configurations and then decide on a final setup.

My plans for next week or so ;)

Code:

bootable stick
	rufus 1.4.3
	freedos 1.1

boot dos

flash mb
	/c11/ami.bat
	reboot
	load default

flash ipmi
	/smt123/adu.bat
	reboot

boot efi

flash lsi
	Note SAS controller ID from MB sticker
	/9207/
	sas2flash.efi -uflash 2308ir.bak

	sas2flash.efi -o -e 7
	sas2flash.efi -f 9207-8.bin
	sas2flash.efi -o -sasadd <id from mb sticker>
	
	#FALLBACK# 	/ph16it/
	#FALLBACK# 	smt2308t.nsh
		
shutdown
reset bios

memtest86

label disks
attach disks

check parking
	/wd/
	wdidle3.exe /r
	wdidle3.exe /d

check tler
	/wd/
	tlerscan.bat
	tler10.bat

benchmark
	3x sas
	3x sata

 

[indy]

So far so good...
Installation went well and all firmware updates went well.
And after some trouble with dhcp and understanding how ipmi lan-port sharing works the I can even access it ;)
Is it normal though that the ipmi console stops working as the system shuts down?
Seems like using the dedicated port might be a better idea?

Next up: hard disks
Pretty sure I will be going with a 11 disk raidz3 configuration.

Reason being that as soon as a disk in a raidz2 setup fails I would have to order a new one.
Which would mean I would have a cold spare when the original disk gets repaired.
And I rather have an additional parity than a cold spare.

It should fit nicely with this adapter I hope:
http://www.newegg.com/Product/Product.aspx?Item=N82E16817988002

 

[Z300M]

Indy:

I can't find it spelled out in the X10SL7-F manual, but another user commented on the ""fact"/assumption? that the CPU fan connector is so far from the CPU: at the corner near the SAS and SATA connectors -- obviously referring to the FanA connector, whereas you apparently have it connected to one of the other Fan connectors. Perhaps they are interchangeable from the electronic standpoint, but it does seem logical that the one labeled with a letter rather than a number would be for the CPU and the numbered ones for the (multiple) case fans.

It should fit nicely with this adapter I hope:
http://www.newegg.com/Product/Product.aspx?Item=N82E16817988002

Apart from the fact that your choice is apparently out of stock, it seems to me that the iStarUSA BPU-350SATA (five 3.5" drives in the space of three 5.25" bays) makes better use of limited space:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816215001

 

[jyavenard]

Indy:

I can't find it spelled out in the X10SL7-F manual, but another user commented on the ""fact"/assumption? that the CPU fan connector is so far from the CPU: at the corner near the SAS and SATA connectors -- obviously referring to the FanA connector, whereas you apparently have it connected to one of the other Fan connectors. Perhaps they are interchangeable from the electronic standpoint, but it does seem logical that the one labeled with a letter rather than a number would be for the CPU and the numbered ones for the (multiple) case

You could plug your CPU fan on any of the pwm connectors. However, the BMC is controlling the speed of that fan according to the CPU temperature. If you plan on running the fans at full speed, then it doesn't matter. Otherwise use it only for the CPU fan.

The fanA connector, if you check a supermicro typical chassis is the fan on the left. They have a fan setting defined to have automatic speed for all fans except the fan connected to fanA.
That insure extension cards, if any are kept at maximum cool.

 

[indy]

@z300m
Now that you mention it, I cant find it in the manual either.
Pretty sure though that jayavenard is correct though, I remember reading that somewhere.

Regarding your hot-swap cage recommendation, I do not have 3x 5.25 bays :)

 

[tio]

Use this one then

http://www.newegg.com/Product/Product.aspx?Item=N82E16816215310
Or get two of these

http://www.newegg.com/Product/Product.aspx?Item=N82E16816215222

 

[indy]

Not a big fan of hot-swap cages personally because they lack vibration damping.
I rather have the hard disks mounted with rubber grommets and cables to make them a bit less noisy.