Extremely cheap HP Z400 build with Xeon and 24GB RAM - aprox 800$

[doman18]

Hello guys!
In my company we use Qnap TS-439. We want to switch it to more reliable and expandable machine with ZFS and Qnap will be as backup. We only interested in storing files and sharing them to Windows clients. Nothing more (no media center, surveillance or any similar things)
Because so far we are pretty happy from two used HP Z400 worksations (Intel Xeon W3520 2.67GHz,12GB RAM ECC,120SSD,Windows 10) bought from reliable reseller we want to use the same machine, but with more RAM, different chasis and maybe different CPU.

1$ = 3,6PLN

CPU: Intel Xeon W3520 or Intel Xeon E5620
- we are more forward to latter because it has AES NI which is usefull in case we would want use VPN (we have 2 OpenVPN severs)
RAM: 24 GB DDR3 ECC
- this is maximum for this mainboard, so as best practice says we can handle 24TBs of data without a hassle? Anything above will slow things right?
PSU: Delta 475W 80 bronze, ATX
- we want to stick with ATX standard, no rack or any fancy swapable psu's
SSD: 120GB Sandisk
- dont remember the model name
onboard SATA: 6xSATAII
- yea, its the only bottleneck, but i hope there are some PCI-E controllers compatible with FreeBSD which can extend that number.

This whole setup will cost us 1280PLN (about 355$)

Additionally ...

chasis: Ravcore Goliath - 300PLN (about 83$)
- We want to move to it all interiors from original HP Z400 chasis. AFAIK cheapest chasis with 7x3,5" bays and 4x5,25" bays. And if you put something like IcyBox IB-555SSK you get 12x3,5" bays! IMO crazy for this level of price!

disks: some WD REDs (4x2TB) in RAID10 - 1240PLN (about 350$)
- we want use RAID10 because its easier to expand than RAIDZ1 or 2. After about 5 years we gathered 1,6TB od data (documents and blueprints mostly) which only 0,5GB is most precious so we think 4TB of usefull space would be more than enough for some time.

What do you think? My only concern is about max drive capacity. AFAIK mainboard can recognise 3TB disks in GPT, but is it somewhat relevant to ZFS?

Sorry for any grammar mistakes, im from Poland.

 

[Chris Moore]

Anything above will slow things right?

You don't have to limit your storage to the 1TB to 1GB ratio.

PSU: Delta 475W 80 bronze, ATX

The capacity of the power supply is dictated by the number of drives. You don't have to be limited by the onboard SATA controller.

i hope there are some PCI-E controllers compatible with FreeBSD which can extend that number

You really need to take a little time and read the hardware guides that have been put together on the forum. Look here:
FreeNAS® Quick Hardware Guide
and
Hardware Recommendations Guide
You should use a SAS HBA to run the drives so you don't need to worry about any limitations of the SATA controller.

 

[MrToddsFriends]

PSU: Delta 475W 80 bronze, ATX
- we want to stick with ATX standard, no rack or any fancy swapable psu's

Don't forget to read the Proper Power Supply Sizing Guidance (the guidance by jgreco itself, not necessarily the whole 180+ postings thread following it).

 

[doman18]

You don't have to limit your storage to the 1TB to 1GB ratio.

Yea, i know that this reccomendations is only , well, reccomendation and is not mandatory but its just a best practice to keep performance on optimal level right?

You really need to take a little time and read the hardware guides that have been put together on the forum. Look here:
FreeNAS® Quick Hardware Guide
and
Hardware Recommendations Guide

Ive read them after making this topic and they clear out some things. Ive also seen some videos in youtube with hardware reccomendations.

You should use a SAS HBA to run the drives so you don't need to worry about any limitations of the SATA controller.

Ive never used SAS. AFAIK is quite compatible with regular SATA drives right? I'ts not about SATA speed which is enough i guess, but about the number of SATA ports and max single disk capacity.

One more thing. Even though SATA2 3Gb/s are enough for us, i was thinking about using as much of this bandwidth as i can. And i was thinking to remove the only bottleneck by bonding two or three NICs in load balancing mode? What do you think about that? There are about 22 workers in our network who use NAS. Is it worth to do it?

/edit
The other thing im thinking is to give up from SSD and buy something more usefull instead. Our workers ususally work on small files and i dont think L2ARC on SSD would improve things if we have 24GB RAM

 

[Chris Moore]

Yea, i know that this reccomendations is only , well, reccomendation and is not mandatory but its just a best practice to keep performance on optimal level right?

It will depend on your ARC hit ratio and that depends on the kind of work the NAS is actually doing.

The other thing im thinking is to give up from SSD and buy something more usefull instead. Our workers ususally work on small files and i dont think L2ARC on SSD would improve things if we have 24GB RAM

Most likely the SSD would actually slow things down. I would not suggest a L2ARC for the purpose you have described.

Ive never used SAS. AFAIK is quite compatible with regular SATA drives right?

Correct. You can use SATA drives on a SAS controller and if you use one like this:
http://www.ebay.com/itm/Dell-H310-6...-IT-Mode-for-ZFS-FreeNAS-unRAID-/162661388939
It just needs to be flashed to IT mode to work properly with FreeNAS, there are many brands that all use the same chipset. It does not need to be this exact card. This type of card gives you the ability to run 8 drives and I know that it will work with drives up to 8TB with no difficulty. If you use a SAS expander with it, you can connect as many (if I recall correctly) 128 drives to the one card.
A SAS expander like this one: http://www.ebay.com/itm/Intel-RES2S...er-Card-SAS-SATA-PCI-Express-x4-/141788425753
Would increase the number of drives you could run from the HBA above up to 16 and you can use multiple expander cards. This one can go in a card slot but does not need to, it only uses the slot for power. You can connect a MOLEX power input to the connector on the top of the card and not put it in a slot. One or the other, not both. The SAS expander works kind of like a network switch for the drives that are connected. Think of the HBA as the network card and the SAS expander like a switch and the drives are like the computers.

And i was thinking to remove the only bottleneck by bonding two or three NICs in load balancing mode? What do you think about that? There are about 22 workers in our network who use NAS. Is it worth to do it?

If you have a network switch that supports using the network ports in a LAG, that might be worth it, as you could have several clients accessing the server at the same time. It is usually not worth it in a home environment, but it could be of benefit with 22 workers. It will not increase the total speed, just give you a better ability to server more simultaneous requests.
That is also a reason for using more disks. Generally, the more disks you have, the more data can flow all at once. That SAS controller can supply more than 10GB of data to the system all at once, but the drives and the network interface will be the slow spots.
If you put 6 drives in a RAIDz2 array, you should be able to fully saturate a 1GB network link. If you have a network switch with a 10GB port, you might want to consider putting a 10GB network interface in the server, but then you will see that the drives are the slow point and the only solution is more drives.
Let us know if you have more questions. Someone on the forum will be able to answer.

 

[Chris Moore]

I understand that you may not be able to access hardware like this, but I would suggest getting something like this:
http://www.ebay.com/itm/SUPERMICRO-...DRI-F-BPN-SAS2-846EL1-24x-TRAYS-/382231880992

Then you just need to add a few components to it to get it where you need it to be and with 24 drive bays it will be enough storage for years to come.

 

[doman18]

Thanks for responses.

Ok, im not bother with L2ARC but ZIL is something i would gladly set for safety purposes. Also im strongly against installing freenas on usb drives. I would like to do it on internal device, more prone to accidental fall off. Can i use ssd both for freenas and ZIL? Or take two smaller devices.

As for LAG, soon we will change network infrastructure. We will use tp link T1600G-52TS or similar switch. As you see it has both LACP and SFP slots. So yea its a matter of choice

As for controller, i already seen that Dell and is most available device here, from list of reccomended hardware for freenas.
http://allegro.pl/raid-dell-perc-h310-sas-sata-przewody-fv-amso-i6808730388.html
Just dont know if it will work with Z400 mobo - didnt find info about that. And buy some SAS-SATA cables to connect Icy dock
http://allegro.pl/icy-dock-obudowa-5x3-5-sata-raid-backplane-i6963892107.html
http://allegro.pl/kabel-mini-sas-sff-8087-4x-sata-0-5m-i6901414568.html
One thing that bothers me that when i read hw recommendations and topics about controllers amost nobody asks about max single disk capacity. For me 4TB is minimum to consider any controller.

I understand that you may not be able to access hardware like this, but I would suggest getting something like this:
http://www.ebay.com/itm/SUPERMICRO-...DRI-F-BPN-SAS2-846EL1-24x-TRAYS-/382231880992

Then you just need to add a few components to it to get it where you need it to be and with 24 drive bays it will be enough storage for years to come.

As for rack servers 24bay is huge overkill. I was rather thinking about 8 or 12 bay devices like HP DL380 G7 or HP SE326M1 which are more than enough for us and are for about ~600-700pln (167$ - 200$). Rack servers has many advantages but the main disadvantage here is power consumption. AFAIK rack servers take 200-300W in idle. This is serious power flow. ATX is less hungry for that.

 

[Chris Moore]

ZIL is something i would gladly set for safety purposes.

I don't understand why you think you need ZIL. The most important thing you can have is a good UPS to keep your server from going down unexpectedly. That is actually true for all the computers and network gear that interconnects the computers.

AFAIK rack servers take 200-300W in idle.

I have two rack servers that have triple redundant power supplies that are rated at 900 watts. They draw 150 watts. The power draw is dictated by the components that are using the power, not by the power supply rating.

As for LAG, soon we will change network infrastructure. We will use tp link T1600G-52TS or similar switch. As you see it has both LACP and SFP slots. So yea its a matter of choice

As for networking, something like this would be a good choice:
http://www.ebay.com/itm/DELL-NETWORKING-N1524-24x-1GbE-4x-10GbE-SFP-fixed-ports/152470960221
This Dell switch has 4 - SFP+ ports (the + part is important) SFP+ supports 10GB where the older SFP port only support 1GB.
You could then put one of these network cards in the server:
http://www.ebay.com/itm/110-1107-30...NGLE-PORT-10GB-SFP-PCIE-ADAPTER-/282590499118
With a cable like this to connect the two:
http://www.ebay.com/itm/10Gtek-for-...AC-Cable-2-Meter-10GBase-CU-NEW-/401415315227
That would get you 10GB connectivity to the switch and allow many clients full speed access to the server.
It would make the physical disks the slow point in the system and adding additional vdevs to the pool would make that connectivity faster.
The individual disks being the slow point and the only way to make it faster is add more disks, you don't necessarily want bigger disks, but more of them. I am looking at building a new server using 24 disks at 1GB each, not for capacity but speed.

Just dont know if it will work with Z400 mobo - didnt find info about that.

The type of motherboard should not hinder the function of a SAS controller. If you have concerns about the Dell card, you can get the same functionality from several vendors that use the same chip. Here is another option:
http://www.ebay.com/itm/LSI-SAS-921...BA-Both-Brackets-Bundle-Options-/152585762071

 

[Chris Moore]

Can i use ssd both for freenas and ZIL? Or take two smaller devices.

FreeNAS does use ZIL, it is an inherent part of ZFS. The thing you appear to be talking about is a Separate ZFS Intent Log or “SLOG”.
Having a SLOG will improve write performance. It doesn't particularly improve 'safety' as far as I know. It only takes a vew gigabytes though but to really help, it needs to be very fast. It also needs to have a high endurance, so you would need an enterprise grade SSD drive and it is usually best to run them in a mirrored pair.

 

[doman18]

I don't understand why you think you need ZIL. The most important thing you can have is a good UPS to keep your server from going down unexpectedly. That is actually true for all the computers and network gear that interconnects the computers.

My company is on the 1 floor, below there is a bank. They have power generators connected to the whole building. So we can say we do have UPS - quite a big one :)

So you sayin that there is no point to us SSD in any way?

I have two rack servers that have triple redundant power supplies that are rated at 900 watts. They draw 150 watts. The power draw is dictated by the components that are using the power, not by the power supply rating.

Then what do you think about those? (We want to stick with LFF not SFF disks)
http://allegro.pl/serwer-hp-se326m1-2xl5520-16gb-2xpsu-12xlff-fv-wwa-i7001602429.html
http://allegro.pl/serwer-hp-dl180-g6-2xe5540-8gb-p410-256-12xlff-fv-i6993244688.html

Offcourse i would ask seller to estimate price for versions with only one CPU (weakest with AES NI i can get) and 32GB ram. I would also ask about dell controller to switch with this p410. But i just want to know first would those be better choice than modified Z400 i mentioned before?

 

[Chris Moore]

I would like to do it on internal device, more prone to accidental fall off. Can i use ssd both for freenas and ZIL?

You would need two separate devices at a minimum. I use a mirrored pair of drives for my boot pool and you could use a second mirrored pair for you SLOG.
These are pretty decent:
http://www.ebay.com/itm/Intel-32GB-2-5-SATA-SSD-SSDSA2SH032G1GN-3Gb-s/302283293441

 

[Chris Moore]

Then what do you think about those? (We want to stick with LFF not SFF disks)

The problem I have with the thin servers is they usually have fewer card slots for adding controller cards. I usually go with either a 3U or 4U rack server chassis because they usually have the ability to put 4 or more expansion cards in the system board. Many of the thin servers only allow 2 or three cards at best.

They have power generators connected to the whole building

Even with generators, there is usually a drop in power when the transfer happens. For the server, that could be a big problem if a write is in progress because you could lose data. A separate UPS on the server is just a good precaution.

 

[wblock]

Ok, I will not bother with L2ARC but ZIL is something I would gladly set for safety purposes. Also I'm strongly against installing FreeNAS on usb drives. I would like to do it on internal device, more prone to accidental fall off.

ZIL is built-in write caching. You automatically have it. A SLOG device can be added, but that is not for safety, it is for improving performance on synchronous writes. There are a few situations that benefit from it, and many that do not.

Can I use SSD both for FreeNAS and ZIL?

No. Moreover, SLOG devices need to be special low-latency SSDs. Please see http://doc.freenas.org/11/zfsprimer.html.

 

[doman18]

Ok, then no SSD ZIL.

The problem I have with the thin servers is they usually have fewer card slots for adding controller cards. I usually go with either a 3U or 4U rack server chassis because they usually have the ability to put 4 or more expansion cards in the system board. Many of the thin servers only allow 2 or three cards at best.

Ok, would this be better?
http://allegro.pl/serwer-dell-pe2900-iii-8x-3ghz-24gb-10x-3-5-i6123244794.html

 

[Chris Moore]

This would be a good base but it would take some reconfiguration and I am not sure what the price translates to.

http://allegro.pl/supermicro-3u-2x-l5630-24gb-12x-450gb-15k-3-5-i6849746125.html

 

[doman18]

This would be a good base but it would take some reconfiguration and I am not sure what the price translates to.

http://allegro.pl/supermicro-3u-2x-l5630-24gb-12x-450gb-15k-3-5-i6849746125.html

Wow, this is a little pricey. 1$ = 3,6pln (sort of) so its about 1800$, but as far as i see it comes with 12 x 450GB SAS disks. So without them price should drop.

We wouldnt want to go over 500$ (not including disks offcourse). Only if its REALLY REALLY worth it we would add some 200-300$ and i would try to convice my boss somehow but it would need strong arguments. As I said, we store documents and blueprints mostly. We have 0,5TB of important data and 1,1TB of archives which are not so much important (if we loose them it would be a shame but it wont hurt so much). So far we used Qnap 439 Pro. So yea, we dont want to spend money on some overkill hardware. We want adjust hardware to our needs and prepare for some future.

//Edit
Here you have whole servers category with added price range 600-2000pln and sorted by price growing up. First you will see items which are prioritized, and after hit price range you go from 600 again with normal items.
https://allegro.pl/kategoria/serwery-i-scsi-serwery-9061?order=p&price_from=6&price_to=2000

 

[doman18]

I really like the way its done here
https://level1techs.com/video/level1-172tb-storage-server#page=1

... i mean that they separated chasis from server itself. I even found some really nice server offer (128Gigs of ram!) ...
http://allegro.pl/dell-r900-4x-4c-2-4ghz-128gb-ram-perc6i-szyny-i6976378785.html
or a little bit weaker
http://allegro.pl/dell-r900-4x-4c-2-13ghz-64gb-ram-perc6i-szyny-i6976359518.html

... however i havent found any LSI SAS3 enclosures they used in link above and cables to connect enclosures together and with server.

Tomorrow i would have to make a final decision about hardware. Soon we will have 12U rack where we put 2 patchpanels , switch and maybe cable organizers. So there will be still enough space for NAS server and thus the rack-server idea itself is very tempting solution. But if i wont find anything that is good and dont cost a kidney (or both) i will rather stick with Z400 + Goliath chasis + H310 HBA as it seem as most trustworthy solution now.

 

[doman18]

I have two rack servers that have triple redundant power supplies that are rated at 900 watts. They draw 150 watts. The power draw is dictated by the components that are using the power, not by the power supply rating.

Can you provide names of those racks? Are they Dell r710 maybe?

I checked those dell r900 and despite they look nice but ppl complain about power consumption. The same is with quite popular IBM x3650 and few other.
http://allegro.pl/ibm-x3650-m2-2x-qc-e5530-2-4ghz-8gb-2x73gb-m1015-i6964039687.html
Only r710 was mentioned as quite reasonable power hungry.

Idea i was chasing lately was to separate storage from storage manager (server). To achieve this i need
- power savvy machine with at least 32GB RAM
- external SAS card supported by freeNAS
- 12 bay enclosure which has external SAS socket and which have HBA controller
- enclosure has to be affordable but come with SAS/SATA bays (not FC as many cheap one has)
- enclosure has to recognise at least 4TB disks

And its very difficult to meet all those requirements. Especially proper enclosures are hard to find. Many has FC connectors
http://allegro.pl/polka-dyskowa-netapp-ds14mk4-4-2tb-fvat-gwar-12-i6947459538.html
some has 2TB limit on controller
http://allegro.pl/hp-msa-20-msa20-macierz-12x-3-5-sata-i5774340781.html
others seems good and cheap but disk holders are unreasonably pricey ...
http://allegro.pl/macierz-fujitsu-siemens-fibrecat-sx40-12-zato-3-5-i5565184672.html
and those which are good (probably) with disks, SAS cables and storage manager device will be above price range.
http://allegro.pl/macierz-ibm-system-storage-exp3000-gwarancja-fv-i6249022699.html
And also was some which i wasnt sure, do they have FC or SAS/SATA disks
http://allegro.pl/macierz-hp-aj750a-msa2000-aj751a-12x-3-5-dual-i-o-i4852783207.html

Only those are somewhat fine
http://allegro.pl/polka-netapp-ds-4243-fv-gw-i6428003069.html

But i dont know how to convince boss to buy it if we need only 4 bays for now and maybe 4 or 8 in the future :D

So probably i will stick with Z400 idea :p

 

[Chris Moore]

Can you provide names of those racks? Are they Dell r710 maybe?

No, I use Supermicro components and pick the parts I want to put together. I used two chassis like this and put the system board I wanted in it.
http://www.ebay.com/itm/SuperMicro-...er-BPN-SAS2-936EL2-PWS-1K21P-1R-/172795227035

I really dislike most of the pre-built systems from the big companies like Dell, HP etc because they use proprietary components that prevent me from mixing and matching to get the system I want or need.

 

[doman18]

No, I use Supermicro components and pick the parts I want to put together. I used two chassis like this and put the system board I wanted in it.

Hmm as i see in your signature you use 32TB with 16GB ram (Irene). So its 2TB of data for 1GB of Ram. Dont you have any performance issues with that?

//PS
Here i found very cheap enclosures. No controller so it should be transparent for server controller. I asked seller what disk ports it uses, he said FC but he has adapters to mount SAS/SATA disks. Would it be a problem to FreeNAS?
http://allegro.pl/polka-dyskowa-emc-bez-dyskow-fvat-gw-i6912299754.html