economical home storage music production parts list proposal

[ramar]

In the context of that document, Parity, Data and Padding

COOL. ok, i have something to chew on. when closing in 20 drives, do people use some sort of drive housings and build a small computer for the Power Supply, FreeNAS Boot Drive, Motherboard, CPU, RAM and RAID Card? if so, the drive rack, probably has a power supply. can the 2 PSU be interconnected so that both of them power each other for redundant PSU power?

or ... what kind of case is used for this? it can be a little bit big and ugly, because i'm going to need to sonically isolate it and give it AC. Data from computer to NAS flows through ethernet, correct? so i can use long runs, i'm talking under 100 feet.

 

[ramar]

what kind of case is used for this?

never mind, i found it
https://www.newegg.com/Product/Prod...re=hard_drive_rack-_-9SIA5EM3HB0913-_-Product

 

[Chris Moore]

never mind, i found it
https://www.newegg.com/Product/Prod...re=hard_drive_rack-_-9SIA5EM3HB0913-_-Product

If you are going to put a high drive count system together, you might want to use a chassis like this instead of the drive bay modules:
https://www.ebay.com/itm/Supermicro...BPN-SAS2-846EL1-24x-TRAYS-NO-PWS/382277186258

 

[ramar]

If you are going to put a high drive count system together, you might want to use a chassis like this instead of the drive bay modules:

thanks
i was looking at the motherboards in the hardware list. i wasn't thinking about so many drives at first. the first one, Supermicro X9SCM-F /X9SCA says noting about how many drives it supports, but the second one X10SLL+-F supports up to 6 drives.
i'm assuming the list lists the preferred hardware in descending order with the likely preferred closest to the top. the first one with expandability as far as number of drives goes is X10SL7-F. i looked at the OS compatibility charts http://www.supermicro.com/products/motherboard/Xeon/C202_C204/X9SCM-F.cfm http://www.supermicro.com/support/resources/OS/C204.cfm which doesn't list any Mac OS X or Ubuntu 14.04. the primary reasons for me to use FreeNAS were ECC memory and file sharing Windows 7 thru 10, Mac OS X 10.9 and higher and Ubuntu 14.04 (i think this is the version of Ubuntu, but it shows no Ubuntu compatibility). no Windows 10 either.

i see one Motherboard X9SCE-F that supports Ubuntu 12.04 server. i wonder if i could have Ubuntu 14.04 running in Ubuntu 12.04 server? i am not sure how it works.

X9DR7-TF+
X9DRE-TF+ support Ubuntu 12.04 Server

do i need to look to some other operating system? https://www.samba.org/?

i need to reset. i thought it didn't matter what OS i used that i would have file sharing across platforms. is that the correct term? i need to access my project from these OSs if possible.

http://www.supermicro.com/support/resources/OS/C222.cfm

 

[Chris Moore]

A chassis like the one I linked to connects to a SAS controller, so the number of drives supported by the system board is not relevant.

 

[ramar]

https://www.ebay.com/itm/Supermicro...BPN-SAS2-846EL1-24x-TRAYS-NO-PWS/382277186258

it looks like it has 24 drive bays. i guess they are powered through the computer PSU.

you can't change from SSD to HDD because the way things work are just too different.

ok, what all changes?

 

[ramar]

A chassis like the one I linked to connects to a SAS controller, so the number of drives supported by the system board is not relevant.

REALLY? that's hard to believe. so i'm back to the Supermicro X9SCM-F motherboard. this reminds me of "deus ex machina"! lol so i need to add a SAS controller to my list, i guess. i need to see if there is a more advanced Hardware List. unless the SAS is part of chassis.
edit: i see, it's the M1015 that i already have in my list.
edit: seems that the list could mention that when using the SAS controller, the number of drives doesn't count. can NAS be built without one? so why mention the number of drives limitations in the hardware list? am i being a "nit picker"?

 

[Chris Moore]

so why mention the number of drives limitations in the hardware list? am i being a "nit picker"?

You can use those system boards to build a system that does not include a SAS controller. Then the number of drives the system board can handle directly does matter. When you use a SAS controller and a chassis that has a SAS expander (like the one I linked to) then the SAS controller handles all the drives in the chassis and the only drives that would connect to the system board are the boot drive / drives.

 

[ramar]

When you use a SAS controller and a chassis that has a SAS expander (like the one I linked to) then the SAS controller handles all the drives in the chassis and the only drives that would connect to the system board are the boot drive / drives.

in other words there needs to be 2 SAS controllers for the number of disks not to matter? and a chassis having an SAS controller is not really common in the size that most people use?
the other thing i am wondering about is why is an 18 drive NAS not super common? i need to mirror 5 drives and have backup. this would apply to all music, video and graphic design or any type of artistic endeavor. this should apply to video especially. where are all these people? either i am a total genius or they are somewhere else more tailored to these needs

 

[Chris Moore]

it looks like it has 24 drive bays. i guess they are powered through the computer PSU.

ok, what all changes?

With regard to the 24 bay chassis I linked to, it comes with a 1200 watt redundant power supply. More than enough to power anything you can put in that chassis even if you were using 1500 RPM enterprise SAS drives.

The thing about SSDs is that they are so much faster than hard drives that a configuration that might be needed to get enough speed out of hard drives would be total overkill with SSD.

As an example, you would need (estimated) 12 hard drives in mirrors to have enough transfer speed to saturate a 10GB network. With SSD, you could probably do that with 4 drives. Then when it comes to capacity, you can get 8TB hard drives easily, but to get 8 TB of SSDs it would take at least 4 of the 2TB SSDs to get there. It all depends on what you are trying to accomplish.

 

[Chris Moore]

in other words there needs to be 2 SAS controllers for the number of disks not to matter?

No. One SAS controller, depending on the model, can control 128 hard drives or more. The 24 drives in that chassis is like nothing.

and a chassis having an SAS controller is not really common in the size that most people use?

Desktop computers don't have SAS expanders, but I guess it depends on the computer you think is common, every computer in the office I work in has a SAS controller. Admittedly, those are high end systems, but it depends on what you are trying to accomplish. You have to be specific with you are describing hardware.

the other thing i am wondering about is why is an 18 drive NAS not super common?

Rack chassis usually go from 16 drive, to 20 and then to 24, because of what fits in the chassis based on the shape and size of the drive in relation to the standard width of a rack and 18 is a very odd number.

i need to mirror 5 drives and have backup.

Why do you need mirrors? The way you are thinking of doing it is not the only way, it may not even be ideal.

either i am a total genius or they are somewhere else more tailored to these needs

The number of drives you are asking for, why do you think that is the ideal number? That is only a perfect number in your way of thinking.

 

[rogerh]

There is no reason to have a separate set of disks for each set of data, unless there is a real need for different speed/size compromises for each purpose. I suspect your audio workstation should be a separate computer, unless you are going to run FreeNAS virtualised on the same computer. Even if it is the same computer, I do wonder if zfs is ideal for low latency audio multichannel recording and processing. And it is not the sort of data where single bit errors are likely to be important. So you need a fast store for audio, perhaps an SSD, a OS drive, and if you are dealing with a separate NAS all the other storage, and backup for the audio, could be in one pool of however many disks are convenient, not necessarily more than 6 or so. If you are virtualising FreeNAS then some fast storage for virtual machines may be relevant. I am not sure how important the motherboard being recommended for a given OS is, I suspect that this does not matter for a modern OS in amateur use, at least with Intel.

 

[ramar]

There is no reason to have a separate set of disks for each set of data

of coarse. i could set up volumes that span parts of 2 disks! you are a genius. that is probably what Chris was telling me by wondering what my needs are.

I suspect your audio workstation should be a separate computer

i was thinking of having a separate computer, but i am curious about having it in the same computer.

I do wonder if it is ideal for low latency audio multichannel recording and processing

i wonder if the kernel can be modified like the Real-Time Kernel for Ubuntu is? i might try it "as-is" and see what happens. or ask on GearSlutz

And it is not the sort of data where single bit errors are likely to be important

i was thinking that too. but do those errors get multiplied somehow? but perhaps i do not need ECC memory?

So you need a fast store for audio, perhaps an SSD, a OS drive,

are you saying i need SSD for storing audio and one for the OS Drive? the OS drive would be doing the audio editing (one drive) that is being stored (another, or the pool).

If you are virtualizing FreeNAS then some fast storage for virtual machines may be relevant

i'll ask on GearSlutz.com

I am not sure how important the motherboard being recommended for a given OS is, I suspect that this does not matter for a modern OS in amateur use, at least with Intel.

so i can file share between platforms: Windows, Mac and Linux with files stored in FreeNAS?
Edit: https://www.gearslutz.com/board/mus...g-windows-mac-linux-freenas.html#post13074345

 

[ramar]

Why do you need mirrors? The way you are thinking of doing it is not the only way, it may not even be ideal.

sorry for my inexperience. i do not need 18 drives at this point. i think i'll make the The fractal design Node 304 work with 6 drives, probably 5 drives and a hot spare. 5 x 8TB = 40TB way more than i need, i could do much smaller SSDs too, i use less than 2 TB of storage at the moment. it will go up, but slowly.

Why do you need mirrors? The way you are thinking of doing it is not the only way, it may not even be ideal.

Is it RAID0 that spreads the work over multiple drives for speed, but eliminates redundancy in it's array? no need to answer, i'll read up on it
here is a thread that i will try to understand https://forums.freenas.org/index.ph...d-why-we-use-mirrors-for-block-storage.44068/

 

[ramar]

2 TB of storage at the moment. it will go up, but slowly

correction: actually, i'll need at least 2 TB for my new library, better make it 3, not counting Ubuntu 14.04, which i have not downloaded yet. perhaps i'll separate Ubuntu into it's own OS Drive and Library. i do not know if the different OS, which need their own drives use libraries that need their own drives because they use differently formatted drives.

i'll have a total of 4 OS drives (Windows 10, Mac OS X HS, Linux Ubuntu 14.04 (1TB) and AVLinux and FreeNAS ... is it v11?, which should all be in RAID somehow and backed up, correct? and 4 differently formatted libraries for plugins and apps. maybe the Windows 10 library can be mixed with Linux and FreeNAS? i am about to create a few hundred GB because of collaboration with a partner. my present project and Mac OS X OS data is around 1.5 TB. with one backup, that would be 3TB, just for that data and 2 backups would be 4.5 TB. so new library 3TB, existing data, 1.5TB = 4.5TB X 2 backups = 9TB i just remembered that i have a bunch of random graphics that inadvertently downloaded by mistake somehow. maybe 1TB that i have to delete.

 

[rogerh]

My personal guess, from relative ignorance, is that you want your audio platform on the bare metal, not virtualised, and using direct access to the working disks you are actually processing audio on. This would mean you could only use one operating system at once, unless you had more than one workstation. Also, if you do multiple boot OSs you couldn't really virtualise FreeNAS on the same hardware. If, however, your experts reckon a DAW can be run as a virtual machine, then you could put everything on one computer. Then you could probably use a ZFS pool (perhaps as a zvol, q.v.) as your audio working 'disk', and this would have to be faster than your larger FreeNAS pool. I do wonder how well access to sound card drivers would work with a virtual OS, I am quite out of my depth here? If your FreeNAS machine is separate then design is much easier, you probably want something like RAIDZ2 for maximum storage if you are only using it for storage, not audio processing. ECC is practically essential for the main NAS and long term error-free storage. It is generally reckoned to be quite unnecessary for a workstation, but it is worth noting that every component from CPU to disks has ECC internally. Trouble is that using ECC for a workstation greatly restricts your MB and CPU choices. Personally if I ever build a fast workstation again I will use ECC.

 

[ramar]

you want your audio platform on the bare metal, not virtualized, and using direct access to the working disks you are actually processing audio on.

that is probably correct, here is the post i got from GearSlutz.com:
quote:
DAW's are designed to work within specific operating systems. Logic Pro X is unique to macOS. Studio One and REAPER have a version that works in macOS and in Windows. None of them have a version that works within FreeNAS.

A typical NAS topology is a set of client operating systems (macOS, Windows, Linux) that have a 1-gigabit or 10-gigabit connection to a dedicated computer that runs the FreeNAS operating system and houses the SSD's and/or HDD's. The FreeNAS presents the SSD's and HDD's as network-based drives (shares) that are usable as disk space by the client operating systems.

If you use a hypervisor (VirtualBox, VMware, etc) and run FreeNAS within it, then you've created a level of complexity. If that same physical computer with the hypervisor is also running an operating system with a DAW in it, you've created a sorrowful combination that won't work well. The DAW needs 'ownership' of the audio I/O, and the hypervisors are notoriously bad at providing usable real-time connectivity for that requirement.

Data errors for read/write are rarely a concern because the CRC checking and protocols along the way do a great job of preventing those problems. Data integrity is just as important in audio as in other applications, so don't imagine that single-bit errors are of no consequence.

The drive latency as seen through the network will affect DAWs in various (always bad) ways, the severity depending on what tuning parameters are available to soften the blow. If you're limited to 1-gigabit Ethernet for connectivity, then there's another performance bottleneck you'll encounter at high track counts.

Draw a picture of what you want to accomplish, and we can drill down to a viable solution, but in my opinion, any solution that depends on virtualizing the DAW's operating system or virtualizing the NAS function is a formula for disappointment.
~end quote
me, i'll read it a few times. does the post look ok to you? i'll read yours a few times too, thanks
edit: i cannot upload a file created in GIMP. too bad

 

[ramar]

what do i need for a 10GB ethernet connection?

 

[ramar]

here is my diagram. now that i see how good i am at illustration, i'll draw another lol! ignore the straight diagonal line with a +

 

[Nick2253]

does the post look ok to you?

I agree 100% with this advice.

I strongly recommend that you do some research on your own about NASes, Hypervisors, and networking, and gain a complete understanding of the bottlenecks that these systems would have. Many of the questions that you've asked here demonstrate a lack of basic knowledge on these topics. Furthermore, many of the questions that need to be asked can only be answered by you and your use case. On that point, I strongly recommend that you gain a thorough understanding in the workload of a DAW, and how and why system, network, and disk bottlenecks impact your DAW.

what do i need for a 10GB ethernet connection?

This is a good example of the above: only you can answer if you need a 10Gb ethernet connection. Does your use case require 10Gbps of data or is 1Gbps sufficient?

here is my diagram.

Your diagram shows a very basic setup, which conflicts with many of the goals you've stated in this thread. If this basic setup is what you are trying to accomplish, this is very easy. My only question about your diagram is that you do not show any connection between your NAS and your backup. That makes we wonder: is your backup meant to be a backup (in functionality) of your NAS, a backup of the data of your clients, a backup of the data on the NAS, or a combination of the above?