Specific build changes to upgrade as high as 512GB of RAM

[Vegastrey]

Do you have any recommendations for affordably priced used sas enclosures? I don't want to spend a ton of money but like the idea of connecting an external enclosure to that new sas controller card I just got.

 

[racielrod]

Honestly, I had 12 drives in it, so I can't say with certainty. It was drawing around 120 watts at idle with 12 drives, so I would guess it to be around 80 to 90 watts with only 6 drives, but that depends a lot on what drives you are using.

When I upgraded the system board, I made several other changes including adding more drives. The system is drawing around 300 watts now, but I have mine in a 48 bay rack chassis with thirty drives in addition to two SAS expander backplanes and 64GB of memory. It is tough to say how much of the power is the system board, processor and memory and how much is the other. Best guess, the more sophisticated board and CPU are going to draw an extra 30 watts.

Adding data here just for the record.
I put together my build -everything except the drives- to run the memory tests

• X9SRL-F
• 8 x 8GB ECC RAM
• Processor Cooler https://noctua.at/en/nh-u9dx-i4
• Fractal Define R6 - With the 3 x 140mm case fans.

This pulls ~90W from the wall.
I when I scale back to 32GB (4 x 8 GB) it pulls ~70W from the wall.

I can report back when I have the build ready with the HDDs on it.

 

[DCswitch]

I'm wondering if these cables will work. I ask because I like the right angles for the SATA connection:
https://www.ebay.com/itm/283302569730

 

[Chris Moore]

Chris Moore updated Specific build changes to upgrade as high as 512GB of RAM with a new update entry:

This just in...

This is the same system board I suggested in the build, but it includes a processor, memory and heatsync all for one price.
It looks like a good deal to me, have a look:
https://www.ebay.com/itm/192918962664

Read the rest of this update entry...

 

[DCswitch]

@Chris Moore I was able to pick this up for $150 w/shipping (it also came with a Dynatron heatsink/fan):
https://www.ebay.com/itm/SuperMicro...-CPU-2-5-Mhz-E5-2600-v2-Zero-Ram/382939693107

I already have loads of memory laying around the apartment, so my question is what do you think of the CPU (E5-2603 v2):
https://ark.intel.com/content/www/u...-processor-e5-2603-v2-10m-cache-1-80-ghz.html

It's only 4 cores, but I'll only be using this for light home use. I still haven't solved that resilvering quagmire yet:
https://www.ixsystems.com/community...ilvering-and-then-crashing-with-reboot.76143/

but I wanted to make sure I had new, better hardware before attempting to resolve that issue. I just purchased 12 new identical HDD's and I'm using Clonezilla to do sector-to-sector copies of all the drives. The copy will be finished tomorrow afternoon.

 

[DCswitch]

Also- what do you think of the E5-2650L V2 vs E5-2650 V2? The L version has 10 cores, but a slower base clock which might be better because it uses less energy.
E5-2650L V2
https://ark.intel.com/content/www/u...processor-e5-2650l-v2-25m-cache-1-70-ghz.html
E5-2650 V2
https://ark.intel.com/content/www/u...-processor-e5-2650-v2-20m-cache-2-60-ghz.html

 

[Chris Moore]

I was able to pick this up for $150 w/shipping (it also came with a Dynatron heatsink/fan):
https://www.ebay.com/itm/SuperMicro...-CPU-2-5-Mhz-E5-2600-v2-Zero-Ram/382939693107

I think that is a good deal.

I already have loads of memory laying around the apartment, so my question is what do you think of the CPU (E5-2603 v2):
https://ark.intel.com/content/www/u...-processor-e5-2603-v2-10m-cache-1-80-ghz.html

I think it is too slow. I would upgrade the processor. In my personal home NAS, I use the E5-2650 V2. It has enough clock speed without being too expensive.

 

[Chris Moore]

Also- what do you think of the E5-2650L V2 vs E5-2650 V2? The L version has 10 cores, but a slower base clock which might be better because it uses less energy.

The number of cores doesn't help as much as the clock speed, in FreeNAS, I have a ten core processor in the system I use for virtualization because more cores helps with virtualization, but I have found that using FreeNAS for virtualization actually makes the storage slower, so it is better to have FreeNAS just do storage and use something else for virtual machines.

 

[DCswitch]

@Chris Moore Thank you- I just ordered the E5-2650 V2. As far as my resilvering issue- can you recommend someone to help me? I just wrote to CyberJock, but not sure if he's the right person. Once I turn it on- I'd like to have a plan. If there are diagnostic commands - I'd like to know what to do to gather new info in case the resilvering crashes the computer again.

 

[Chris Moore]

As far as my resilvering issue- can you recommend someone to help me?

I am not sure what caused the problem. Just changing drives from one controller to another, I have done that many times, and it should not have caused any resilver at all. There must have been some peculiar configuration with your old controller card. I don't know for sure what the old configuration was, but it may have had some underlying controller specific configuration that interfered with the data. I don't know that there is a fix at all.

 

[DCswitch]

@Chris Moore Do you know someone else I can speak to about it? I can't go in with no plan. If there's info I can gather before it crashes (if it crashes again) than perhaps I can figure out the right order of the drives. I'm not sure what commands to type to gather that info. The good news is I've succesfully made copies of all the drives with CloneZilla, so it gives me an unlimited amount of tries to get it right. If it fails a bunch of times even with the cloned drives- I'm going to go out and purchase another 11 drives and clone those as well. Failure is not an option.

 

[DCswitch]

@Chris Moore Do you know a consultant I can speak to? I would be willing to pay them. No one is answering my thread and I really don't know what to do.

 

[Chris Moore]

Do you know a consultant I can speak to? I would be willing to pay them. No one is answering my thread and I really don't know what to do.

iXsystems used to have a page on their site for it, but that page doesn't work any more. You could give them a call: (408) 943-4100

Edit: I found the link I was looking for: https://www.ixsystems.com/freenas-commercial-support/

 

[DCswitch]

@Chris Moore Thank you. I called/emailed and they already got back to me- they gave me the name of a consultant. Before I reach out to him- I'm going to fire up the new hardware. It took three days, but I just finished cloning all the drives. I also bought a SSD drive to use as the boot drive (I previously used a USB thumb drive). I have a question- since I'm going to do a fresh OS install, what version OS would you recommend (what do you think is the most stable one)? Do you think I should install 11.2 U4.1 or 11.1 U7, etc? I see you're using 11.1 U7.

 

[Chris Moore]

Do you think I should install 11.2 U4.1 or 11.1 U7, etc? I see you're using 11.1 U7.

I like 11.1-U7 because it is stable, but there are some new features in the 11.2-U4.1 build that have me seriously considering making the move to that in the near future. I would have done it this past weekend, but I had some other obligations get in the way of what I wanted to do.

 

[DCswitch]

@Chris Moore I really can't thank you enough for all your help. My FreeNAS has been running stable since I updated all the hardware. The resilver says it will be finished tomorrow morning. That was not a fun period of time- I never want to feel that way again. When its finished and backed up- I'm going to blow it out and start with a fresh install. I'm going to give 11.2 U4.1 a shot. I just hope they fixed the problems with macOS transfers (they were aware of it said they were fixing it back on U3). I was thinking about making a 13 disk vdev with ZFS-3. I know that's pushing it with 13 disks, but what do you think (seems I've read that people don't like to go over 12 disks)? This is for light, home use, so I thought it would be fine. It's also going to backed up to twice.

 

[racielrod]

Hello,

I have been using a build based in this post for quite some time now (very happy with the results).
I have been running with 8GB micron sticks, but I got access to some 16GB sticks (same speed and characteristics, different brand only - Kingston).

Questions:

1. Does this board (X9SRL-F) supports different size dimms? i.e: Can I use 4 x 16GB + 4 x 8GB for a total of 96GBs?
2. If I increase the memory density using less larger dimms to achieve the same total memory; would my system consume less? (sorry if this is a stupid question, coming from absolute ignorance here)

• The manual specifies:
  • "For best performance, please use the memory modules of the same type and speed in the same bank"
  • "Populating these DIMM modules with a pair of memory modules of the same type and same size will result in interleaved memory, which will improve memory performance"
  • "Always use DDR3 DIMM modules of the same size, type and speed"
  • "Mixed DIMM speeds can be installed. However, all DIMMs will run at the speed of the slowest DIMM"
  • "The motherboard will support odd-numbered modules installed (1 ,3 ,5 ,or 7 modules). However, for best memory performance, install DIMM modules in pairs"

Thanks!

 

[Chris Moore]

I have been running with 8GB micron sticks, but I got access to some 16GB sticks

Like the manual says, "For best performance, please use the memory modules of the same type and speed in the same bank" ...

That said, the reduction of performance should be minimal and RAM is still orders of magnitude faster than the hard drives, so it will still be better to have the memory than not have it. It might be an interesting experiment to test it in each of the three configurations, just to see if there was a noticeable difference.

What is the workload you have for this system?

The manual specifies

PS. Thanks for looking at the manual first. It doesn't say that different capacity modules won't work, and past experience makes me certain that mixed capacity modules will work, but I can't be certain what the performance impact would be. My guess is that the memory will still be fast enough that it will benefit ZFS to have the additional memory to use for ARC.

 

[pro lamer]

@racielrod mind the allowed ranks number (just from the top of my head, I haven't checked the mobo manual recently nor I checked the DIMMs specs)

Sent from my phone

 

[racielrod]

Hi @Chris Moore
While I'm extremely happy with my build based on this post, I was curious about what do you think is the next step up from here?
I'm thinking on putting together another FreeNAS build and I was looking at probably build something with DDR4.

Is that the X10SRL-F, X10SRi-F? anything else?

I guess my goals would be:
- Build another system that uses newer hardware than what I have while still being able to do it with used parts on ebay to reduce costs.
- More energy efficient
- Similar to the X9SRL-F where I can scale and have VMs and add more ram if needed.

Thanks!