Will you accept FreeBSD users ? here I am...

[borisnet]

Hi All,

I am not a FreeNAS user per se, as I have been using FreeBSD for 15 years+ now - and yes in case you are wondering, I started playing with FreeBSD when I was 5 (joke).

Anyway, I built a NAS on FreeBSD when ZFS was released a long time ago and managed to keep upgrading my zpool across different versions of FreeBSD.
I tested FreeNAS but was comfortable enough with FreeBSD to almost feel frustrated by some limitations of the Web GUI - don't get me wrong, FreeNAS is great but being behind FreeBSD release cycle may be frustrating especially when new features in FreeBSD are convenient.

So, I currently have the following box (build early 2013 currently running FreeBSD-10.0-RELEASE):

Norco RPC4224
Supermicro X9SRL-F
Xeon E5-2665 v1
128GB ECC
3 x M1015 (properly flashed)
6 x 3 TB
12 x 1TB (2.5'')
RAIDZ2 built in such a way that I can lose one of my 3 controllers and still feel like nothing happened.
4x 1Gb NIC in LACP to my switch
2 x 80 GB SSD for the system as a ZFS mirror
Many FreeBSD jails for various purposes

I have another older server - DL380 G5 - 20GB - Dual E5x00 (can't remember the exact CPU) and 8 x 1 TB 2.5 in a zfs pool as well - raidz only if I remember well.

I have other gears but not relevant to this topic.

As the HDD are getting bigger, I would love to convert my compute power (the E5 and the memory) for playing with Openstack and/or ESXi at home.

I am also interested in getting a more concentrated solution and lower power consumption.
I have done the calculation of running 24/7/365 and there is no way I will actually save anything (electricity wise) by buying something new ;-) It's more to get something compact and dedicated.

So here is what I am considering -
Main system:
ASRock C2750D4I
64GB ECC
[ Reuse my 6 x 3TB of my current system - Add 2 x 3 TB drives I have as spare and connect them all straight to the Mobo] = 8 x 3 TB = 24TB -> 18TB in RAIDZ2

Backup system - coming up online once a week or something like that:
ASRock C2750D4I
32GB ECC
[Reuse my 12 x 1TB of my current main system and 4 from the DL380 and connect them as follows: 8 on IBM-1015 and 8 on the MOBO] = 16 x 1TB = 16TB

Or will you keep a mix of 3TB and 1TB drives to just get a similar storage space between Main and backup?
something like [4x 3TB + 10 x 1TB] in each system (I have 20x1 TB 2.5 in total)

With such an approach, I address the following:
1/ I can reuse my compute power (E5, memory and DL380) only when and as needed
2/ I can reuse my current drives and still increase my storage capacity by replacing the 1TB drives in the future by 3TB drives as they (hopefully become cheaper)
3/ I still have some power thanks to the C2750D4I mobo to run a few apps on top of the storage applications
4/ it should not be power hungry (at least the CPU ;-))
5/ I have a backup server (today, I have my data copied on my different servers, but specs are different and this piggy DL380 G5 draws a lot of power while being shutdown and just accessible with IPMI - something like 30W which I found surprisingly high)

Now I might be wrong and it could be better to stick to a lower number of HDD per system, need to investigate a bit on that front.

That's it for my intro, I will create a thread under hardware to discuss my ideas/options for evolution of NAS builds.

 

[cyberjock]

Note that the 16GB DIMMs that work on the Intel Atoms are made from unicorn blood, so they are *mega* expensive and almost impossible to buy. I don't think a single forum user has bought any yet. So before you go jumping on the 64GB of RAM expectation you might want to do a little research and make 100% sure that you can and are ready to pay the price for them. ;)

 

[rogerh]

There is a German retailer that claims to have 16GB ECC SODIMMs on 2-3 days delivery. At £264, which is approximately twice the price of two 8GB SODIMMs.

 

[jgreco]

are made from unicorn blood

and mithril.

The same happened with unbuffered ECC when Sandy Bridge came around. For the first year or so, the larger sized DIMM's were wildly (like 2x-4x) more expensive, if you could even find them. The real problem with the Avotons is that I doubt that they're as popular as the Sandy Xeons, so it is unclear on when exactly a turnaround might occur. They should be available by this point

http://www.prnewswire.com/news-rele...dimms-by-im-intelligent-memory-266613731.html

The low profile variant seems to be available from some foreign sellers. IMM2G72D3LDUD8AG-B15E

 

[borisnet]

Note that the 16GB DIMMs that work on the Intel Atoms are made from unicorn blood, so they are *mega* expensive and almost impossible to buy. I don't think a single forum user has bought any yet. So before you go jumping on the 64GB of RAM expectation you might want to do a little research and make 100% sure that you can and are ready to pay the price for them. ;)

Ok thanks but how are they different exactly ? Is this because they are UDIMM ECC that you are saying this?

I just read the specs here:
http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Specifications

I mean they are not SO-DIMM.

I currently have 16GB FBDIMM in my server so unfortunately won't be able to reuse them on the C2750D4I...

 

[Ericloewe]

Ok thanks but how are they different exactly ? Is this because they are UDIMM ECC that you are saying this?

I just read the specs here:
http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Specifications

I mean they are not SO-DIMM.

I currently have 16GB FBDIMM in my server so unfortunately won't be able to reuse them on the C2750D4I...

The problem is that there are no DDR3 chips with enough density for 16GB UDIMMs, so additional logic is needed to carefully present two chips as one, to allow for 16GB UDIMMs.

This situation is unlikely to change, as Intel LGA1150 processors don't support 16GB UDIMMs and DDR4 is basically here.

 

[jgreco]

I was under the impression the Memphis stuff was actually 16. Am I wrong?

"Rare" would definitely be true though.

 

[borisnet]

Note that the 16GB DIMMs that work on the Intel Atoms are made from unicorn blood, so they are *mega* expensive and almost impossible to buy. I don't think a single forum user has bought any yet. So before you go jumping on the 64GB of RAM expectation you might want to do a little research and make 100% sure that you can and are ready to pay the price for them. ;)

The problem is that there are no DDR3 chips with enough density for 16GB UDIMMs, so additional logic is needed to carefully present two chips as one, to allow for 16GB UDIMMs.

This situation is unlikely to change, as Intel LGA1150 processors don't support 16GB UDIMMs and DDR4 is basically here.

Ok good point on the DDR4 development reducing the chance to see anything new in the DDR3 world to nil.

 

[Ericloewe]

I was under the impression the Memphis stuff was actually 16. Am I wrong?

"Rare" would definitely be true though.

They say 8Gbit per chip, with 18 of them. I have to say, the whole 16GB UDIMM story is getting rather confusing.

 

[jgreco]

Too bad, a 64GB Avoton is an interesting possibility...

 

[Ericloewe]

Too bad, a 64GB Avoton is an interesting possibility...

Definitely. Hopefully a future product with a DDR4 memory controller will open up that possibility.

 

[borisnet]

Definitely. Hopefully a future product with a DDR4 memory controller will open up that possibility.

It looks like I may skip this generation and wait for 2015 and new models, thanks for the thoughts !

 

[Ericloewe]

It looks like I may skip this generation and wait for 2015 and new models, thanks for the thoughts !

You can always say "Supersize me!" and go with a brand new Xeon E5

 

[borisnet]

You can always say "Supersize me!" and go with a brand new Xeon E5

Indeed however I was trying to build something that does not cost me an arm and the E5 is really about supersize-ing to the point I would really need to have good reason for doing so.
I already have one - first generation though, actually I have 2 since I have a special blade with an E5-2428L 1.8 Ghz which are not available as retail.

I will share more on the post I am writing now in my the hardware section.

 

[solarisguy]

[----] 8 x 3 TB = 24TB -> 18TB in RAIDZ2 [----]

I know that you know, but there could be some freshman reading the above who does not know...

8 * 3 TB = 24 TB, two disks used by parity in RAID-Z2, so 6 *3 TB = 18 TB = 16.37 TiB (and one cares only about TiB)

Out of 16.37 TiB you are likely to use only 13 TiB for data, as you might want not to use more than 80% of the pool capacity.

 

[jgreco]

Indeed however I was trying to build something that does not cost me an arm

Please do share where you get such a good discount, for most of us the E5 stuff costs an arm AND a leg.

 

[borisnet]

Please do share where you get such a good discount, for most of us the E5 stuff costs an arm AND a leg.

Black market, I traded my left arm and right kidney (to keep some sort of balance) to afford my first E5. As you can expect, I cannot sustain this kind of lifestyle anymore ;-)

 

[borisnet]

I know that you know, but there could be some freshman reading the above who does not know...

8 * 3 TB = 24 TB, two disks used by parity in RAID-Z2, so 6 *3 TB = 18 TB = 16.37 TiB (and one cares only about TiB)

Out of 16.37 TiB you are likely to use only 13 TiB for data, as you might want not to use more than 80% of the pool capacity.

I approve this message ;-) for both the TiB (good practice on you, bad on me) and the 80% capacity to keep good zpool performance.

 

[Roman]

to afford my first E5. As you can expect, I cannot sustain this kind of lifestyle anymore ;-)

Dual E5 will cost you two legs and two arms.

 

[jgreco]