Supermicro X10 series

[HolyK]

Where can you find the IT firmware for the X10SL7-F onboard LSI 2308?
Nevermind...: :)
ftp://ftp.supermicro.com/Driver/SAS/LSI/2308/Firmware/IT/

Was the flash successful? Let us know! I will probably go for the same MB. At least because of the ECC RAMs. A minute ago i found that i am missing 2GB (from 6) of the RAM in my workstation. I have 3x2GB tripplechannel OCZ Platinum Kit but the system is showing only 4GB. I did not noticed any issue so far so looks like one module died instantly ... ... and off course it is more than year after end of the warranty :/

Boot from USB3 is... meh... no biggie. What you mean Geli is still buggy? 9.1?

See this (not a bug actually, but a weak spot), this and this (will be fixed in 9.2) ticket.

I don't quite like the way how the key is stored on the USB stick ... (Actually it is a test setup in virtual environment, but it does not matter...)

[root@freenas] /# ls -l /data/geli/01e6139c-1af7-405a-b596-6d40a931a0c0.key
-rw-r--r-- 1 root wheel 64 Sep 29 13:13 /data/geli/01e6139c-1af7-405a-b596-6d40a931a0c0.key

Anyway this is an offtopic, but since you asked ... :]

 

[raidflex]

Was the flash successful? Let us know! I will probably go for the same MB. At least because of the ECC RAMs. A minute ago i found that i am missing 2GB (from 6) of the RAM in my workstation. I have 3x2GB tripplechannel OCZ Platinum Kit but the system is showing only 4GB. I did not noticed any issue so far so looks like one module died instantly ... ... and off course it is more than year after end of the warranty :/


See this (not a bug actually, but a weak spot), this and this (will be fixed in 9.2) ticket.

I don't quite like the way how the key is stored on the USB stick ... (Actually it is a test setup in virtual environment, but it does not matter...)

Anyway this is an offtopic, but since you asked ... :]

I had a buddy of mine that had a x58 motherboard and one stick was not recognized out of the three and it ended up being a bad memory slot. This was an EVGA board. Although that is much less common then bad memory.

 

[Sir.Robin]

Was the flash successful? Let us know! I will probably go for the same MB. At least because of the ECC RAMs. A minute ago i found that i am missing 2GB (from 6) of the RAM in my workstation. I have 3x2GB tripplechannel OCZ Platinum Kit but the system is showing only 4GB. I did not noticed any issue so far so looks like one module died instantly ... ... and off course it is more than year after end of the warranty :/


See this (not a bug actually, but a weak spot), this and this (will be fixed in 9.2) ticket.

I don't quite like the way how the key is stored on the USB stick ... (Actually it is a test setup in virtual environment, but it does not matter...)

Anyway this is an offtopic, but since you asked ... :]

Thanx for the bug links :) Don't use encryption though...

I haven't flashed yet... cuz i don't have the mobo yet. Just checking. It's good to be prepared... :D

Ordered it though... now lets see how long it takes to get here.

 

[rovan]

I am interested in the Supermicro Boards also. Problem I see with this is the Haswell chip only(lol) support up to 32GB DDR3 ECC... Isn't this a major flaw in the requirements section?

In terms of power consumption it should be superior, but not being able to populate more than 4x Dimm slots surely is an issue?

Example if you wanted to setup 2x ZPools of 16TB...
Each ZDev: 6x 4TB RAIDZ2 Array (32TB total usable?)

48TB Total disk space. Is the memory requirement (1GB per 1TB) based on usable space or actual space? This setup would reach the limit of recommended RAM... with 12 free HDD slots this leaves no room for expansion correct?

 

[Sir.Robin]

32GB should get you far, but if thats not far enough, you can go Xeon E5.

 

[rovan]

@Sir.Robin, Thanks for the reply! The X10 with on-board SAS controller is particularly intriguing. I look forward to hearing successful flash to IT mode! :D

Not sure what your implying. I'd rather not invest in an x10 1150 only to find out a 2011 is what I need, being a different socket that seems illogical.

I guess what I'm trying to find out is with a 32GB cap how much of a performance or otherwise effect (instability?) Will I be likely to encounter pushing the 32TB+ range.. it is likely my array will grow to surpass this size in time.

De duplication will not be necessary. I understand this increases the RAM requirements significantly.

There is a wealth of knowledge on this forum, and it appears there are quite a lot of experienced ZFS guru's on here. Cyberjocks presentation was particularly useful/interesting. Thanks!

 

[Sir.Robin]

Oh crap! Didn't read the whole post... read it on my mobile :p

The X10SL7 is no problem flashing to IT mode. Supermiocro are providing it's own FW for it.

As for your question. It depends. Although i have no experience in such a large pool on ZFS.
The thumb rule is minimum 8GB. And for pools over 8TB add 1GB RAM per TB storage.

Although this will be afftected by how much of a load you actually put on your NAS.

I have wery little going on. It's mainly documents and photo sharing for me and my girl.
So 8GB is far enough for my use. 6GB is no problem either on my secondary.

In your case i'm not sure. You might be getting along well with 32GB... and if you don't you could always upgrade again :)

 

[cyberjock]

How much RAM really depends on what you plan to use the server for, the type of loading, and the number of users.

For a home server, if you don't plan to do NFS for ESXi datastore or run a bunch of iSCSI devices, 32GB of RAM will get you probably 100TB or more of disk space. Home use just isn't that demanding. But if you want to run an ESXi datastore, have several iSCSI devices, and 10+ users, 32GB of RAM is going to be too low.

Several people here have 70TB worth of disks and 24GB of RAM and have no problems. But those are home servers.

 

[rovan]

Thanks so much! Very useful information. I will post a build log when I get all the parts :D

 

[jgreco]

I'd be a little nervous at 32GB->100TB but once you're past 16GB, I expect that a 1GB:2TB ratio becomes progressively less painful and more palatable for home users as RAM size increases. It is likely to be a matter of performance more than stability. An archival system with modest storage and retrieval might well be okay at 32GB->100TB but I'd want to bench test that by filling it and trashing it several times to make sure that there weren't any obvious gotchas in it. Since the import is the usual spot people run into trouble, definitely worth testing pressing RESET when the pool's 90% full and actively writing a bunch of times. I'd at least feel a bunch better about it having seen such testing be successful.

 

[rovan]

So it might end up being wise to build something around the expandable memory of the X9 series after all. I liked the idea of using a board based on a newer chipset (X10) and would probably fully populate (32GB) the 4 DIMM slots from the start, this would effectively leave no room for growth.

Would the difference between 32GB to say 64GB or 128GB be noticeable on a 70TB+ setup? Assuming modest use for archiving and storage by 2 - 3 users. and some concurrent streaming of HD video. Trying to keep on topic considering the limitations of the X10 series.

 

[jgreco]

It is really subjective more than anything, I think.

The guidelines suggest 1GB per TB of disk, but I can safely say that as the numbers get up there (past 16 or maybe 24), it is targeted more at a busy departmental fileserver with fairly heavy activity. Write speeds are dramatically affected by RAM, largely due to sizing of transaction groups. You can certainly have too little RAM for ARC and ZFS will thrash around and kill performance, but that strikes me as being more on the low end.

So for archival use, modest file accesses (NOT big video editing!), and some linear reads going on (streaming several 5Mbit/s streams) I would expect a 32GB system to be able to support a 70TB array. I think I'd also say that I wouldn't expect 64 to make enough of a difference to justify the wild cost differential.

 

[absence]

Can anybody of you, owning a SM X10 board, tell me whether the board supports wake-on-lan for at least one of the Intel nics? I guess so, but I'd be happy to know for sure before buying.
BTW, I'm still waiting for some power and performance benchmarks regarding the E3-1230Lv3! :)

 

[John M. Długosz]

Where can you find the IT firmware for the X10SL7-F onboard LSI 2308?

Nevermind...: :)

ftp://ftp.supermicro.com/Driver/SAS/LSI/2308/Firmware/IT/

How did that work for you? Anything to report, before I follow?

BTW, what is "IT" mode?

 

[HolyK]

Nothing to report. I did mine also without any issues. Just follow this guide and you should be fine :]

Regarding the "IT mode" question ... if i remember correctly (please someone correct me if i am wrong) it is like this:

IT = Initiator-Target ... you will get rid of the RAID so system/zfs will have direct access to the physical disks ... which is exactly what ZFS wants to have ^^
IR = Integrated RAID ... opposite situation ... you will have additional layer - RAID - between OS and physical disk, so something what ZFS does not like at all

 

[jgreco]

IR mode also allows IT style access to disks that are not in the RAID set, but at a lower performance level than pure IT mode.

 

[74m]

Got the hots for IPMI and ECC monitoring capability now :D

Sir.Robin said: ↑

Is there a log of ECC bit errors on these boards?​

Yes, there is.

I didnt know, that there is something like a ECC log. As is asked yesterday already...
So there is a way to get sure, that ECC is definitely working on this (Supermicro) boards? Or maybe just to see that ECC was activ?

That would be very nice. Because my biggest fear in buying my new equipment is, to build a new shiny NAS and ECC is not working untill i must see it when its to late...

 

[John M. Długosz]

Just follow this guide and you should be fine :]

Thanks. The instructions that came with the file are unclear at best.

1) Please copy all files from UEFI folder to bootable USB disk

2) Run SMC2308T.nsh and following the instruction to enter the last 9 digits/character of the SAS address.
⋮

I think it was just copied directly from the DOS version and a few words changed, but I didn't think that's how UEFI worked. What you pointed me to is more like what I expected (some kind of pre-boot environment built-in to the Flash memory), and includes necessary details. Thanks again.

if i remember correctly (please someone correct me if i am wrong) it is like this:

IT = Initiator-Target ... you will get rid of the RAID so system/zfs will have direct access to the physical disks ... which is exactly what ZFS wants to have ^^

Thanks. I figured what it did, but could not find any definition for what it stands for.

 

[Sir.Robin]

I didnt know, that there is something like a ECC log. As is asked yesterday already...
So there is a way to get sure, that ECC is definitely working on this (Supermicro) boards? Or maybe just to see that ECC was activ?

That would be very nice. Because my biggest fear in buying my new equipment is, to build a new shiny NAS and ECC is not working untill i must see it when its to late...

The BIOS and IPMI reports that the inserted sticks of RAM are ECC cabable.
So... you will have to trust that it is working.

The log is not as detailed as i was hoping though. My blade servers at work has detailed single and multibit error logging.
The SM board do not.
How it logs a single/multibit error i do not know... yet.

 

[74m]

Thank you for your response Sir.Robin.

So we have a log, that the RAM is ECC cabable with the SM boards, but not detailed log. I think thats only a half insurance?!

So i read in this thread that the ecc_check only works with a "few cpus", i guess, the new 1050 socket is not included?! I wont believe that we have not chance to verify that ecc is definitely working?

If i get the whole Freenas and ECC thing right, ECC might be most important if we use zfs?!

Guys, i want to buy my new hardware... 3 months ago my choice seems to be easy. No my choice is not easy cause I think the old hardware satisfy all my need... but new hardware is luring. ;)