First NAS System

[Bozon]

I was looking at this board because it already can connect to 8 drives through SAS and it is only $20 ( new egg has it for 279 ) more than the other board. I am not sure of any other issues with it versus the other board.

https://www.amazon.com/Supermicro-Motherboard-Micro-DDR4-X11SSL-CF/dp/B018B7BMS2

Of course really good sas controllers only seem to run $50 dollars. Of course you would have to flash the above board’s SAS controller to make it freenas friendly. Not horrible but certainly an extra step that can go wrong and you have to find the directions to do it.

 

[Bozon]

On a bit of a sidenote, I'm thinking about Raid-Z2 for the NAS. I just want to make sure I understand what that means.

If I understand it correctly, that means out of the 6x4TB drives, I will have (1-2/6)DriveEffiency*4TB = 16TB total NAS capacity, with 2 drive failure allowance, no read penalty but an increased write penalty because of parity calculation (which is all pretty much from wikipedia).

My question about the 'no read penalty', is what does that mean? Does that mean it reads off the NAS as fast as if I was reading the data off a remote hard drive? or read off the NAS as fast as if it was data on my own local hard drive? or read off the NAS at a faster rate than a local hard drive (but not faster than the network allows)? I read somewhere else that Raid 6 (which is similar to Raid-Z2?) has a read speed of N-2 of an individual dirve, or specifically in my case, 6-4=x4 the speed of an individual drive (which gets significantly slower if one drive were to fail and need replacement/rebuild). Is that correct?

Below simplified for easy digestion.
No read penalty refers to comparable nas configuration with no redundancy. Think of it this way when you write data ZFS has to write it to 3 different drives ( for Raid-Z2, 2 for Raid-Z1, 4 for Raid-Z3 ) in such a way that if 2 of the drives fail it can still find it in the third drive. It also needs to encode data to tell it if the data is actually correct on a drive. Kind of like double entry accounting can tell you if someone is stealing from a business. Now when we read we can get the data from any of the 3 places and we only need to check with the audit register entry to verify that the data is actually valid at that location. It can verify the data as quickly as it reads it so there is no penalty on the read.

To help map my "easy", explanation to Wiki's very good thorough explanation, the double entry accounting audit register translates to checksum in computer ease.

 

[Jessep]

Below simplified for easy digestion.
No read penalty refers to comparable nas configuration with no redundancy. Think of it this way when you write data ZFS has to write it to 3 different drives ( for Raid-Z2, 2 for Raid-Z1, 4 for Raid-Z3 ) in such a way that if 2 of the drives fail it can still find it in the third drive. It also needs to encode data to tell it if the data is actually correct on a drive. Kind of like double entry accounting can tell you if someone is stealing from a business. Now when we read we can get the data from any of the 3 places and we only need to check with the audit register entry to verify that the data is actually valid at that location. It can verify the data as quickly as it reads it so there is no penalty on the read.

This isn't exactly correct as parity is stored per block, not on a parity "drive". Hence why you can lose any drive and still recover the data from the remaining drives, which would be impossible if you lost the "parity drive" itself.

This explains it in much more detail:
https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz

 

[Bozon]

This isn't exactly correct as parity is stored per block, not on a parity "drive". Hence why you can lose any drive and still recover the data from the remaining drives, which would be impossible if you lost the "parity drive" itself.

This explains it in much more detail:
https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz

Please take this with a grain of salt.

I never mentioned a parity drive. In fact, I never mentioned parity at all. I talked in terms of drives, to simplify the discussion. Secondly, I was specifically staying away from detail to address the write penalty versus no read penalty, and trying to speak to the audience. I probably should have gone with a motorcycle analogy, and really pissed off the pedants in the group.

Caution Bad RAIDZ Motorcycle analogy follows. Read at your own risk.
Say you are in a war and you only have motorcycle couriers to get messages to the front line soldiers to give them vital battle plans.
You have realized that motorcycles frequently are captured or blown up delivering these messages to the front line, or just plain break down. To increase reliability, you decide to send 3 motorcycles to the front lines with the same message. This requires more time because you have to round up 3 motorcycle riders and copy the message 3 times. So you send these riders off on their 3 motorcycles on 3 different paths. The front line doesn't care which motorcycle gets to them first since they all have the same message. The first motorcycle to reach you is the one you read the message from. So no read penalty. Unfortunately, you run into another problem, the messages are sometimes being corrupted by the poor quality of the paper, pen, and courier backpack. You don't want to take a chance on the message being corrupted and the wrong order being executed. So you think about it for a minute, and you create a key that contains the first letter of each word in the message, and require that be put in the courier's back pack also. Now when a motorcycle courier arrives at the front line the captain takes out the key and the message. If they agree, the captain processes the message, and executes the orders. If they don't agree, he waits for the next message to arrive. If all of the motorcycle riders have corrupted messages then the message is lost, or if all motorcycles are captured or destroyed the message is lost also.

I know this is wrong also, because I didn't mention COW, ZIL, SHA, L2ARC.

 

[Bozon]

For really detailed description of what is really going on with RaidZ2 parity complete with a reference to the original mathematics paper describing how the method was derived, and references to the code that does the messy calculations:

http://dtrace.org/blogs/ahl/2006/06/18/double-parity-raid-z/#raidz2

 

[SearchEngine27]

...get a SAS HBA card to get you your extra drive connectors. Then you're not limited by motherboards with 8 built-in SATA connectors and are free to get with with more RAM expandability or whatever else is important to you.

So all the other hardware is on order and in the mail. I'm looking at this now, but I'm a little bit blind. I'm seeing some have, for example, 12Gb/s with 8 ports. Does that mean that the PCI-E card will be the bottleneck for any drives connected to it, assuming that I have 3+ HDD that are capable of speeds at 6Gb/s?

I don't know how huge of a deal it is for me right now, because even if that is the case, I'll just end up with my two SSD boot drives on the PCI-E and my 8 HDD will be on the MOBO directly. I guess I'm just trying to get a feel for the limitations of the SAS HBA PCI-E cards.

 

[rvassar]

Does that mean that the PCI-E card will be the bottleneck for any drives connected to it, assuming that I have 3+ HDD that are capable of speeds at 6Gb/s?

The 12Gbit/s speeds are likely for SAS drives, but otherwise it's a units problem. 6 Gbit/s SATA disks can transfer at most ~600 Megabytes per second, but spinning rust disks usually sustain much less. Usually 120 - 180 MB/sec. PCIe v.2 will transfer 500 Megabytes/sec in each direction, per lane. Most HBA's are at least x4 lanes if not x8 lanes, so they should be able to move 2 Gigabytes / second. This nearly doubles for v.3, and doubles again for v.4. Unless you're plotting a SSD array, you should be OK.

Also... SAS and SATA are electrically compatible, but only SAS controllers have the ability to interoperate. Your motherboard SATA ports will not talk to a SAS disk, but SAS HBA will almost always allow SATA fanout cables.


Serial ATA

PCIe

 

[Bozon]

Some experienced users, such as Chris Moore (search his recent posts about SAS cards), have recommended SAS cards, and reading their posts the drives are still the bottleneck even if they are rated at 12Gb/s that is only the communication speed and it isn't the actual drive performance. Also, the overall bottleneck will continue to be the network connectivity for some time. 12 GB/s per second seems to be mostly about marketing.

 

[SearchEngine27]

The 12Gbit/s speeds are likely for SAS drives, but otherwise it's a units problem. 6 Gbit/s SATA disks can transfer at most ~600 Megabytes per second, but spinning rust disks usually sustain much less. Usually 120 - 180 MB/sec.

Not to be a stickler for information, but I'm trying to get a feel for what you're trying to say through what you actually said because you clearly meant something way different - I just have no idea what. 6Gb (Gbit) is not 600 megabytes, not even if you are mixing and matching Gb and GB (gigabyte). 6Gb/s is 6000Mb/s, 6GB/s is 6144MB/s. So if you're trying to say 6000Mb/s for 6Gb/s, then I have no idea what you meant to say when you say 'Usually 120 - 180 MB/sec'

 

[rvassar]

Not to be a stickler for information, but I'm trying to get a feel for what you're trying to say through what you actually said because you clearly meant something way different - I just have no idea what. 6Gb (Gbit) is not 600 megabytes, not even if you are mixing and matching Gb and GB (gigabyte). 6Gb/s is 6000Mb/s, 6GB/s is 6144MB/s. So if you're trying to say 6000Mb/s for 6Gb/s, then I have no idea what you meant to say when you say 'Usually 120 - 180 MB/sec'

Little "b" = bits. A bit is a single 1 or 0.

Capital "B" = bytes. Bytes are made of 8 bits. These usually get padded by parity bits in I/O channels, so we usually use a divide by 10 when converting.

As for the "120 - 180 MB/sec", that's an approximate range describing the sustained transfer rate of a single non-SSD hard disk. Each disk in your ZFS pool can sustain roughly 120 to 180 Megabytes per second, which is at best 1/20th of transfer rate of a PCIe v.2 x4 HBA card.

 

[sremick]

Also, the overall bottleneck will continue to be the network connectivity for some time.

This is the point I was going to make. Unless you're running 10Gbit ethernet, or you are running apps/VMs locally on the NAS with lots of internal reading and writing, you're going to be fine. It doesn't take much of a drive to saturate standard gigabit ethernet.

 

[Bozon]

Not to be a stickler for information, but I'm trying to get a feel for what you're trying to say through what you actually said because you clearly meant something way different - I just have no idea what. 6Gb (Gbit) is not 600 megabytes, not even if you are mixing and matching Gb and GB (gigabyte). 6Gb/s is 6000Mb/s, 6GB/s is 6144MB/s. So if you're trying to say 6000Mb/s for 6Gb/s, then I have no idea what you meant to say when you say 'Usually 120 - 180 MB/sec'

rvassar is talking real world performance. if a bus is rated for x Gb/s speed, it doesn't mean that you can divide 8 to get data bytes that actually will get pushed through it, because you have to add in communication overhead to the number. This is a typical marketing trick to quote the speed that signals can be changed from 1 to 0, and not the much lower real world performance. Apparently, 6Gb looks way better than 600 megabytes. These numbers are pretty good for comparison against similar marketing hype numbers, but when you are actually calculating real world performance you need a lot more detail.

 

[SearchEngine27]

Little "b" = bits. A bit is a single 1 or 0.

Capital "B" = bytes. Bytes are made of of 8 bits. These usually get padded by parity bits in I/O channels, so we usually use a divide by 10 when converting.

As for the "120 - 180 MB/sec", that's an approximate range describing the sustained transfer rate of a single non-SSD hard disk. Each disk in your ZFS pool can sustain roughly 120 to 180 Megabytes per second, which is at best 1/20th of transfer rate of a PCIe v.2 x4 HBA card.

no I got that, but you threw out wildly off numbers. You said 6Gb/s == 600MB(yte)/s which is not close at all. 6Gb/s is 6000Mb/s. 6GB/s is 6144MB/s. So I'm trying to give you a chance to clarify the numbers '120 - 180 MB/sec', because if your first numbers are wrong then these may be too.

 

[SearchEngine27]

This is the point I was going to make. Unless you're running 10Gbit ethernet, or you are running apps/VMs locally on the NAS with lots of internal reading and writing, you're going to be fine. It doesn't take much of a drive to saturate standard gigabit ethernet.

Well I'm mainly operating over LAN. Doesn't this end up being only a little less than 100Gb/s? I'm no network expert, but I was under the impression that the big delays happened when you're communicating externally with packet collision and hopping and travel time in general, and then you end up with substantially less than your Ethernet throughput is capable of. I thought over LAN you get a lot closer to that speed. Is that not true?

I mean, if network is still the true bottleneck, then I'll just pick up any old PCI-E card I guess then. Like I said, I'm only planning on using it for the two SSD boot drives, so overall speed might not matter a whole lot anyway.

 

[Bozon]

https://www.delkin.com/blog/what-you-need-to-know-about-sata-data-transfer-rate/
http://www.tomshardware.com/forum/261621-32-internal-hard-drives-transfer-speeds

 

[rvassar]

no I got that, but you threw out wildly off numbers. You said 6Gb/s == 600MB(yte)/s which is not close at all. 6Gb/s is 6000Mb/s. 6GB/s is 6144MB/s. So I'm trying to give you a chance to clarify the numbers '120 - 180 MB/sec', because if your first numbers are wrong then these may be too.

I don't see myself using "6GB/s" anywhere in my first reply today. If it's there, it's a typo, but I can't find it. If a mod wants to double check my assertions I'm open to editing.

Current SATA standards are 1.5, 3, and 6 Gbits/sec. SAS also has a 12Gbit/s speed. I linked the relevant Wikipedia articles as well.

 

[Bozon]

Well I'm mainly operating over LAN. Doesn't this end up being only a little less than 100Gb/s? I'm no network expert, but I was under the impression that the big delays happened when you're communicating externally with packet collision and hopping and travel time in general, and then you end up with substantially less than your Ethernet throughput is capable of. I thought over LAN you get a lot closer to that speed. Is that not true?

I mean, if network is still the true bottleneck, then I'll just pick up any old PCI-E card I guess then. Like I said, I'm only planning on using it for the two SSD boot drives, so overall speed might not matter a whole lot anyway.

You have 14 Sata connections already, it seems like you are good for the sata connectivity that you need. Why do you need a PCI-E card? If it is for future expansion, you should wait. Better cards will surely become available, or you can get the current latest and greatest used when you need it.

 

[SearchEngine27]

You have 14 Sata connections already, it seems like you are good for the sata connectivity that you need. Why do you need a PCI-E card? If it is for future expansion, you should wait. Better cards will surely become available, or you can get the current latest and greatest used when you need it.

I might be misreading the spec for the board, or you think I got a different board. The board I got was Supermicro Micro ATX DDR4 LGA 1151 Motherboards X11SSM-F-O, which I thought only had 8 SATA.

 

[microserf]

I might be misreading the spec for the board, or you think I got a different board. The board I got was Supermicro Micro ATX DDR4 LGA 1151 Motherboards X11SSM-F-O, which I thought only had 8 SATA.

The board has 8 SATA3 ports. It seems that you've purchased 8 HDDs for your pool so you need to rethink how you're going to boot. An HBA, as you've been discussing above, should work. When looking at how to best use these cards, you might find it easier to use the HBA fanout/breakout cable(s) with your storage drives and only use two of the MB SATA3s for booting.

 

[Bozon]

I might be misreading the spec for the board, or you think I got a different board. The board I got was Supermicro Micro ATX DDR4 LGA 1151 Motherboards X11SSM-F-O, which I thought only had 8 SATA.

No, I misread an earlier post. Based on your latest post:

• WD Red 4TB NAS Hard Drive - 5400 RPM Class, SATA 6 Gb/s, 64 MB Cache, 3.5" - WD40EFRX (x6)

That is only 6 drives with 2 slots for mirrored boot drives. So you won't need more than 8 SATA ports now. So wait on worrying about this until you need more drives, unless there is some other reason to worry about this now.