Video editing hot and warm storage server?

woods

Dabbler
Joined
Jul 27, 2018
Messages
45
What I need:

1. hot storage of about 10TB and high read/write speeds (600MB/s +) without need for redundancy.
2. warm storage of about 20TB, speed is of little importance but redundancy, really, REALLY is.

no 1 is a system for high performance video editing and no 2 is a storage system. Most vital data will be backed up in a cloud. And long term, finished projects I wish to store for the rest of my life will be burned to disks.

So I'm wondering, is it possible to combine system no 1 and no 2 into one system?

What I have:

1. editing system: i7-5930K - 16GB - GTX 970 (this is currently suitable for my needs) running Win10 but looking to migrate back to Linux
2. an unused server: HP ProLiant d380e with only one Xeon CPU and 8GB installed
3. HP H220 LSO 2308 "SATA IT Mode FreeNas Avago 9205-8i" (no idea what that means)
4. I have an old cheap RAID card I used in the editing system. But the RAID crashed and it never warned me
5. a bunch of semi-old and semi-new 3 and 4 TB HDD's

I don't really know where to start and whether it is a good idea to set up my own server and having to maintain it. Is it worth the effort? Data is pretty important to me though... but I'm wondering about the economy of running my own server vs. renting cloud space. But cloud space is no real solution for the hot storage part as I do need very high speeds to edit.

A single project is often around the 1TB+ mark when I'm working on it. And more often than not, multiple projects run at the same time. When a project is done, I am no longer responsible for the data so I can just remove it and simply cold store the finished project. The server I would build, would act as a sort of a redundancy buffer (and ideally also as a high speed hot storage server) between ongoing projects and final cold storage of the final product.

So should I even be building a server? Is it worth it?

EDIT; I would love to go the DIY route as I like studying how systems work, but I'm afraid I don't have the luxury of infinite time.. so if I would build a system, it should be easy for me to maintain for years without having to spend too much time on it. Maybe it is better to look into pre-built systems? I don't know.

I can afford to invest another 500 - 1000€ into the system.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Cloud space is not magic. It is just someone else's computer, and you have no control over whether their business plan is viable, over the quality of hardware they use, or the level of redundancy (if any) present, whether backups are performed reliably, whether they are competent to handle security against attacks, whether they're even competent to hit the power button on a server -- okay that last one's a bit harsh, or is it? It has the magic upside of making your problem someone else's problem: you feel like you don't have to worry about it. It has the dark side of when something goes wrong, you may be well and completely screwed.

When you make your own NAS, you have to invest more time, and you have to learn some stuff. If you choose that route, massive help is available here. You then gain the option of having control over your data while also being able to upload it to someone else's computer ("cloud"), which may be the best of both worlds.
 

woods

Dabbler
Joined
Jul 27, 2018
Messages
45
Ok, valid point. Although I must confess that at this particular moment in time, I think it's more likely that I'll fuck something up rather than a cloud service. But at least, if I fuck it up, it's on me, and I like that frame of mind.

From a practical point of view, could I use server to also serve as a high speed storage device to directly edit video off of? I suppose I should be looking into 10GbE, right?

Say that if I want to make a 20TB pool with good redundancy and achieve high read/write speeds, is the hardware I currently have, up to the task?


HP ProLiant DL380e Gen8 E5-2430L Xeon Hex 6-Core 8GB RAM 2U Rack Server 14x LFF

detailed specs:

  • Brand: HP
  • Model: HP ProLiant DL380e Gen8, 14 x LFF, DL380p Gen8
  • CPU: Intel Xeon Hex (6) Core E5-2430L 2.00GHz - 2.50GHz Turbo Boost - 15MB SmartCache
  • RAM: 8GB DDR3 RAM
  • RAID: HP Smart Array B120i SATA Controller (Only 4x Bays Usable Without Add-in Card)
  • HDD: 14 x 3.5" (LFF) SAS/SATA Drive Bays - Caddies/Drives/Drive Blanks Not Included (Only 4x Bays Usable Without Add-in Card, Only SATA Drives Accepted Without Added in SAS Card)
  • Graphics: Onboard Graphics
  • Optical: N/A
  • PSU: 2 x Hot-Swap 460W Platinum Plus PSUs
  • Expansion: 1 x Full-Height PCIe x16, 2 x Full-Height PCIe x8, 1 x Low-Profile PCIe x8
  • Interfaces: 1 x Serial,
    2 x Video (VGA)
    4 x 1GbE Ports
    1 x Dedicated iLO4 Port
    1 x SD Slot Internal
    7 x USB Ports
  • RAC: Integrated Lights Out 4 - iLO4
  • Form Factor: Rack-Mountable 2U
  • Rails: No Rails Included
  • Dimensions: LFF Drives: 3.44 x 17.54 x 29.5 in (8.75 x 44.55 x 74.94 cm)
  • Weight: 14KG Minimum, 32.30KG Maximum



I know I need two 10Gb/E cards (one for the server, one for my editing system) but in terms of CPU and RAM, do you think it is manageable with what I have? I don't need all the fancy stuff besides some good protocols that measure drive health and error stuff.

Perhaps I do need to invest in some decent RAM? ECC RAM seems to be what I would prefer and I think the HP ProLiant I have seems to support it.

RAM I currently have:
2x hynix 2GB 2Rx8 PC3 - 10600R - 9 -10 B0
2x samsung 2GB 2Rx! PC3 09 - 11 B1 - P2

Other than that, I would prefer to start the system with my old drives and replace them with decent drives as time progresses and the old drives are starting to deteriorate.

I currently have these drives:

4x BarraCuda 4TB ST4000DM004
8x BarraCuda 3TB ST3000DM008
4x BarraCuda 3TB ST3000DM001 (of which one or more are faulty - but could be used as intermediate disks when transferring data from the other drives to the server? Or is it possible to add to the pool a disk that already has data on it and it automatically adding the data to the pool if there is enough space?)

Most of the data I currently have, is not critical. At this point there's maybe 3-4TB that is absolutely vital.

But if using the old old drives and replacing them as time goes on is just too stupid, please say so! But I would like to put them to some good use to get some more of my investment out of it.

I essentially have 12 "healthy" drives that make up 40 TB, so that's enough to get a 20TB pool with good redundancy, right?

And my server has 12 drive slots at the front (2 more at the back). The back-plane seems to have 2x SAS connections. Currently, one SAS cable runs to the back but not connected to an HBA, it connects directly to the MOBO. So I guess I need another cable? Perhaps 2 cables because I've read there's multiple types of SAS cables and it depends on the HBA you're using. Which in my case is an LSI 2308 with two SAS connections.

To buy list:

- ECC RAM (16 GB?)
- 2x SAS cables (which ones?)
- 10 Gb/E cards (one for server, one for device, which ones?)
- SSD as FreeNas boot device
- 12x hot-swap caddy https://www.benl.ebay.be/itm/3-5-Ho...551393?hash=item21650202e1:g:t8IAAOSwz-ldLYDI
- .... ???
 
Last edited:

woods

Dabbler
Joined
Jul 27, 2018
Messages
45
usb keyboard arrived today so I could finally install freenas

currently it is running on one 12-core Xeon @ 2.00GHz, 8GB of RAM and the on-board NIC.

8x 3TB drives are running in a striped mirror configuration, controlled by the 2308 HBA.

Had some issues getting Windows 10 to connect to the system, but somehow managed to fix it.

Next step is ordering 2x 10G NIC's and a fibre cable because the server is fucking with my auditory system.

I have my eyes on 2x intel x520-SR1, a 30m SMF fiber cable and 2x intel compatible transceivers.

Tomorrow I'll place the order.

when the hardware arrives, I'll install everything and run some benchmarks. Depending on the results I'll order some extra RAM and if necessary a second CPU.

If everything works out, I'll migrate the server to the basement and drill some holes in some walls.
 
Top