New to FreeNAS, diskless installation and setup

[VladTepes]

It isn't about the number of SATA ports though. I got an expansion card to get extra ports.
The problem, for you, is that FreeNAS needs to have services like that in jails (as per snaptec's advice), which means in the pool - aka the storage drives.

This is just the way it is.

For the life of me I can''t think of any reason, aside perhaps from sheer pertinacity, that you can't accept that. Where's the downside?

 

[Ericloewe]

which means in the pool - aka the storage drives.

It can be a separate pool. In the future, it might even be the boot pool, assuming its devices are appropriate, but I don't think that's possible at the moment.

 

[jkim]

It isn't about the number of SATA ports though. I got an expansion card to get extra ports.

To get extra ports, I'd need a new machine entirely.
This is my machine: http://www8.hp.com/au/en/products/proliant-servers/product-detail.html?oid=5379860

It can be a separate pool. In the future, it might even be the boot pool

I'm guessing this is exactly what would be able to fulfill my requirements? /hope

 

[mattbbpl]

It can be a separate pool.

That's the answer, right there. "Pristine" data pool and another system/jail pool.

 

[Mirfster]

Just "spit-balling", but why not use PFSense for VPN, etc and FreeNAS just for the Data? Personally, I would rather keep the roles separated. Imagine having to down/reboot the FreeNAS Server for Maintenance, Updates, etc... So there goes your VPN too...

 

[danb35]

To get extra ports, I'd need a new machine entirely.
This is my machine: http://www8.hp.com/au/en/products/proliant-servers/product-detail.html?oid=5379860

That machine appears to have a PCI Express slot. What prohibits you from using it for a SATA card?

 

[VladTepes]

That's the answer, right there. "Pristine" data pool and another system/jail pool.

Do you mean pool or vdev?

 

[danb35]

I'm intending on doing this through the software firewall (and I'm assuming Freenas has one of these).

FreeNAS does not, by default, have a software firewall. It's designed and intended to be used on a protected LAN. It might be possible to do what you want in a jail, though (again) I'm far from certain about that--I seem to recall hearing of trouble using pf specifically in jails, and pf is what you'd need to use. I think I'd agree with @Mirfster that a separate device running your router software of choice (pfSense is a popular choice, should be able to do everything you mention, and runs on lightweight hardware) would be a much better way to handle your unique network requirements.

leaving the internal USB and microSD as the only options to service these, and the USB drives are super slow.

Apparently you can also get USB SSDs, which are pretty quick, and also considerably more reliable than a typical USB flash drive. IIRC, @depasseg has been using one.

Edit: Here's one. You could easily Velcro it to the inside of your case somewhere. I don't have any experience with them myself, but it sounds like it might be what you need. Of course, a SATA controller and a small SATA SSD might be less money.

 

[snaptec]

I build a fn Server on a gen8 basis for a Customer. I think you cant put in Another Controller, but Im not sure.
The Performance with 16gigs of ram is really good for that Money.

Im not sure If the hp gen8 supports sd and internal usb at the Same Time.
Check that please!
If its supported, you can have a usb as Boot and a sd as a Single vdev pool. The hdds as Another pool for your data. May as you want !?
Running all the jails which will produce random and Many iops on the sd will be very slow.
And with a single sd card vdev there is no redundancy.
One of the baddest configuration examples i've Seen on the freenas forums.
The only question is -> when it will fail!
Hope you have a second freenas box where you do Backups to


Edit:
To be honest, the better way would be a extra pfsense box...

Gesendet von iPhone mit Tapatalk

 

[danb35]

Do you mean pool or vdev?

No, the recommendation would be for a separate pool for jails. I'm doing that myself--data is on my 18-disk (3 x 6-disk RAIDZ2) pool, separate SSD pool for jails. The SSD is really unnecessary, but one of the jails is VirtualBox, and I wanted a little better performance for the VM.

 

[VladTepes]

I didnt know it was possible to have more than one pool

Cool I learned something. :)

 

[jkim]

Just "spit-balling", but why not use PFSense for VPN, etc and FreeNAS just for the Data?

There's multiple reasons.
1. I want to use the hardware that I have. I've spent enough budget this year to make the wife very angry if I spend any more until the next tax return (GTX1070, GTX1060, HP MS Gen 8, 4 x 3Tb HDD, 4 x 6Tb HDD, 2 sticks of 8Gb RAM, 2 x 500gb SSD, 2 x 250gb SSD).
2. Before I bought this HP, I was running an out of box NAS solution, and couldn't get it to do exactly what I wanted using the router to do the VPN and the NAS as storage only (specific NAS users such as transmission-bt to only be able to do traffic through VPN, and be firewalled out of traffic using eth0, while other users such as system update account ignoring VPN and using eth0 only). So I bought this server, installed OMV using ZFS for storage and 16Gb microSD for OS, and it was working great and meeting all my requirements, except for 1 bug in the wheezy pppd/l2tpd implementation which would bring the VPN down once every 2 or 3 weeks for about 2 hours and bring the entire server down by filling up the logs folder once every 2 months or so - longer VPN outages causing excessive logging (buffering packet 123, expected 122 or something of the sort).
After the third time, I got fed up and researched which NAS OS didn't have this bug, and found myself here. Reading the requirements, I spent the last little bit of my budget (the 2 x 8Gb sticks of RAM... OMV only needed 2Gb so that's all I had before), and ordered new disks.

So there goes your VPN too...

That's OK. 95% of use of VPN is the 2 users on the box itself. The other 5% is really adhoc - i.e. wife watching stream content from laptop while doing dishes, and a weekly rsync of 50% of the data (2 of the 4 planned vdevs) to the off-site NAS.

That machine appears to have a PCI Express slot. What prohibits you from using it for a SATA card?

1. money.
2. no room to put the disk.

Im not sure If the hp gen8 supports sd and internal usb at the Same Time.
Check that please!

I will test it this weekend!
8Gb Flash USB for FreeNAS, 64Gb microSD for apps.

 

[danb35]

no room to put the disk.

I can't speak to money (though I have trouble believing that $30 or less for a small SATA controller would bust the budget), but I can't imagine there's nowhere inside your case to Velcro an SSD, even a 2.5" unit (and a 1.8" would obviously be smaller yet).

 

[jkim]

I can't speak to money (though I have trouble believing that $30 or less for a small SATA controller would bust the budget)

I'm not in the US. An el-cheapo PCIe-SATA controller card from some unknown manufacturer (skymaster) costs $60 AUD here, and I'd need to buy a new 120gb SSD for around $60 (will have to be an unreliable cheapo like v300 or ssd plus... 120gb samsung evo 850 is closer to $100)... making it $120 (~$100 USD). The better ones (controller cards using LSI chips) looked to cost $300+. I was lucky to nab these microSDs for quite cheap (2 for $40).
There used to be some cheap and decent ones around the $30 mark, but it seems they've exited the Aussie market in the last 2 - 3 years since I can't seem to find any stock (was trying to buy them to convert my old E6600 machine into a second NAS with 10 x 3Tb disks for my off-site backup - gave up on the idea and decided to back up only 50% of my data instead).

I can't imagine there's nowhere inside your case to Velcro an SSD, even a 2.5" unit (and a 1.8" would obviously be smaller yet).

The case is quite a tight fit. Some people have succeeded in what you said... but I don't think it's that common... and even if it were, it'd have to wait until next year's tax return.

 

[snaptec]

Did you overread the parts of my last Post that you don't like?

I have a question:
What do you do If a Drive fails in 2 weeks? Wait Till tax return and have endless sleepless nights?
A single sd card as a vdev/pool should really be no Option!
There is cheap Hardware for running pfsense on ebay Australia for under 100$.
Kick out your consumer router and make it "Right"


Gesendet von iPhone mit Tapatalk

 

[jkim]

Did you overread the parts of my last Post that you don't like?

Nope... I ignored it because there's no problems in that area.

1. If the microSD fails, that's why there's 2 (different app config, but the same data config) and it should happily chug along for 12 months.
2. If 1 disk fails it's ok since it's raidz2, if 2nd disk fails then there's offsite backup box + no local data loss, if 3rd disk fails there's local data loss but offsite backup... and the critical stuff will be rsync'd to 1tb External disks both locally and at the offsite, so they'll have even more copies. What's the chances of 3 brand new disks failing within 1 year given they're not Seagates? What's the chances of 5 going bad causing loss of everything except the critical stuff? What's the chances of 7 going bad losing everything? If 4 go bad, I'll buy new disks budget be damned. Also, I don't see how it's related to the microSD argument given disk failure is unrelated to using microSD for apps?

I don't think you understand the end to end solution I have in mind, which I've mentioned earlier to a degree but perhaps not been clear, and certainly didn't mention everything.

The plan is something like below:

This NAS (onsite) = VPN server for connections between sites, VPN client for "external VPN service".
- USB OS
- microSD apps
- 4 x 6Tb in RAIDZ2
- 1 x 1.5Tb external HDD (the most critical 1.5Tb portion of the stuff on the 12Tb nas storage)
- 1 x 1Tb external HDD (the most critical 1Tb portion of the stuff being backed up to the 1.5Tb drive)
Services:
- VPN client for torrent via local user/streaming from VPN server hairpin connections
- VPN server for off-site backup and streaming clients
- Printer/Fax/Scanner
- A number of small VMs (VMDKs stored on the disks, VMware on the microSD)
- Local network plex
vdev setup:
500Gb - Super Important - back up to 1.5 Tb and 1Tb
500Gb - Important - back up to 1.5 Tb and 1Tb
500Gb - Semi Important - back up to 1.5 Tb only
3Tb - Really not important - The stuff that won't be transferred across, and not backed up anywhere. First to go if RAID fails (i.e. 3 disks go bad) or site fails (i.e. house catches fire).
remaining - Not Important

Other NAS (offsite), running... haven't decided on OS, but probably Oracle Linux Server or Windows 2012 Server ("free" through my subscription) or something - E6600, 12 Gb DDR3 non-ECC = VPN client for data synching, planned to upgrade to HP MS Gen 10 next year tax time if it's out by then.
- SSD OS/apps
- 500Gb 2.5" internal - backup drive (the most critical 500Gb portion of the stuff being backed up to the 1Tb drive)
- 4 x 3Tb in RAIDZ or RAID5 (the most critical 9Tb portion of the stuff on the main array, replicated through a daily job via VPN)
- 1 x 1Tb external HDD (the most critical 1Tb portion of the stuff being replicated to the 9Tb array)
Services:
- Backup recipient
- Local network read access to backup data (no write access to files from local network, except for perhaps 1 partition)
- Printer/Fax/Scanner
- Local network plex
partition setup:
500Gb - Super Important - sync every hour from main system, back up to 1 Tb and 500Gb
500Gb - Important - sync every 12 hours from main system, back up to 1 Tb only
500Gb - Semi Important - sync once a day from main system
500Gb - Local network accessible share (thinking whether needed or not, and whether to "reverse backup" to main).
remaining - Not Important - sync once a week, pause at x:59 of each hour and resume once super important sync finishes.

A single sd card as a vdev/pool should really be no Option!

Why not?
It has apps and app config only... there's no real loss if that goes... maybe the NAS will be down while I insert the other one? No big deal... none of my storage data is at risk, which is the important part - the part that I want clean, pristine and meticulously organised manually so I can find everything and track everything.
I do the same for every one of my PCs... never back up any of the OS and "clean reinstall, clean re-setup" in case of operational failure.
3 - 4 redundancies for all "data" that cannot be downloaded from the internet (mostly kids photos and kids videos, but also includes documents such as payroll, a scan of the receipts for all the stuff I buy, financial position spreadsheet, etc).

There is cheap Hardware for running pfsense on ebay Australia for under 100$.

Firstly, I don't see anything there for under $100. The cheapest is $450, and it doesn't support enough LAN ports or have WIFI.

More importantly, will it actually do what I want it to do?

Note that I do not want any of my other PCs on the network to go through the VPN except for very specific activities (i.e. streaming US geo-locked content for example). The client PC will have split tunneling VPN configuration with these specific sites to go via the NAS out through the external VPN, whereas all other network connections will go via the router to the WAN directly.

I was planning to achieve this by having the bt service user having access to only ppp0 and nothing else, and the vpn server user having access to ppp0 and its own connection (ppp1?). The rsync user will have access to (ppp1?) and usb ports only. If I have pfSense external box, how would I manage these security requirements? I never found a way to achieve this when running the synology + custom router (Pentium 2 based box with IPCop installed, PSU has since died and I chucked the rest given its age). Also, this will be very expensive given i'd have to change my wifi modem router to bridge mode, and purchase

 

[pirateghost]

Wait a second.

You say you plan on running VMware on a microsd card? Why are you trying to shoehorn everything you want into FreeNAS when you're going to be running a VMware server? Put all the networking functionality in VMs (pfsense will work for you there).

You aren't going to get all of your "non-negotiable" items out of FreeNAS.

 

[snaptec]

Nope... I ignored it because there's no problems in that area.

1. If the microSD fails, that's why there's 2 (different app config, but the same data config) and it should happily chug along for 12 months.

You need it in another running device for making backups ...

2. If 1 disk fails it's ok since it's raidz2, if 2nd disk fails then there's offsite backup box + no local data loss, if 3rd disk fails there's local data loss but offsite backup... and the critical stuff will be rsync'd to 1tb External disks both locally and at the offsite, so they'll have even more copies. What's the chances of 3 brand new disks failing within 1 year given they're not Seagates? What's the chances of 5 going bad causing loss of everything except the critical stuff? What's the chances of 7 going bad losing everything? If 4 go bad, I'll buy new disks budget be damned. Also, I don't see how it's related to the microSD argument given disk failure is unrelated to using microSD for apps?

apps are jails in your case.
If you are happy with a complete reconfigure...
Sorry, Im always thinking in my business criterias, where uptime is priority.

I don't think you understand the end to end solution I have in mind, which I've mentioned earlier to a degree but perhaps not been clear, and certainly didn't mention everything.

The plan is something like below:

This NAS (onsite) = VPN server for connections between sites, VPN client for "external VPN service".
- USB OS
- microSD apps

..jails in your case...

- 4 x 6Tb in RAIDZ2
- 1 x 1.5Tb external HDD (the most critical 1.5Tb portion of the stuff on the 12Tb nas storage)
- 1 x 1Tb external HDD (the most critical 1Tb portion of the stuff being backed up to the 1.5Tb drive)

Attached USB HDDs are not a good idea in freenas. Skip that!

Services:
- VPN client for torrent via local user/streaming from VPN server hairpin connections
- VPN server for off-site backup and streaming clients
- Printer/Fax/Scanner
- A number of small VMs (VMDKs stored on the disks, VMware on the microSD)
- Local network plex

Vmware on the microSD wont work.
You have BSD Jails. Please reread a Freenas beginner guid what FN is capable of.
You could use the USB for Vmware, and virtualise FN and paassthrough the disks.
That wont run on that hardware!

vdev setup:
500Gb - Super Important - back up to 1.5 Tb and 1Tb
500Gb - Important - back up to 1.5 Tb and 1Tb
500Gb - Semi Important - back up to 1.5 Tb only
3Tb - Really not important - The stuff that won't be transferred across, and not backed up anywhere. First to go if RAID fails (i.e. 3 disks go bad) or site fails (i.e. house catches fire).
remaining - Not Important

Again, reread about FN and ZFS.
Single Vdevs wouldnt be good!
One Pool out of the drives and different datasets is the way to go.

Other NAS (offsite), running... haven't decided on OS, but probably Oracle Linux Server or Windows 2012 Server ("free" through my subscription) or something - E6600, 12 Gb DDR3 non-ECC = VPN client for data synching, planned to upgrade to HP MS Gen 10 next year tax time if it's out by then.
- SSD OS/apps
- 500Gb 2.5" internal - backup drive (the most critical 500Gb portion of the stuff being backed up to the 1Tb drive)
- 4 x 3Tb in RAIDZ or RAID5 (the most critical 9Tb portion of the stuff on the main array, replicated through a daily job via VPN)
- 1 x 1Tb external HDD (the most critical 1Tb portion of the stuff being replicated to the 9Tb array)
Services:
- Backup recipient
- Local network read access to backup data (no write access to files from local network, except for perhaps 1 partition)
- Printer/Fax/Scanner
- Local network plex
partition setup:
500Gb - Super Important - sync every hour from main system, back up to 1 Tb and 500Gb
500Gb - Important - sync every 12 hours from main system, back up to 1 Tb only
500Gb - Semi Important - sync once a day from main system
500Gb - Local network accessible share (thinking whether needed or not, and whether to "reverse backup" to main).
remaining - Not Important - sync once a week, pause at x:59 of each hour and resume once super important sync finishes.


Why not?
It has apps and app config only... there's no real loss if that goes... maybe the NAS will be down while I insert the other one?

and reconfigure everything that sits on

No big deal... none of my storage data is at risk, which is the important part - the part that I want clean, pristine and meticulously organised manually so I can find everything and track everything.
I do the same for every one of my PCs... never back up any of the OS and "clean reinstall, clean re-setup" in case of operational failure.
3 - 4 redundancies for all "data" that cannot be downloaded from the internet (mostly kids photos and kids videos, but also includes documents such as payroll, a scan of the receipts for all the stuff I buy, financial position spreadsheet, etc).

the number of redundancies is good, but not the way you would like to it (single vdevs, usb disks on FN...)

Firstly, I don't see anything there for under $100. The cheapest is $450, and it doesn't support enough LAN ports or have WIFI.

Sure if you search after pfsense. Try searching after x86 hardware with multiple lan ports

More importantly, will it actually do what I want it to do?

pfsense will perfectly do your vpn and routing part.

Note that I do not want any of my other PCs on the network to go through the VPN except for very specific activities (i.e. streaming US geo-locked content for example). The client PC will have split tunneling VPN configuration with these specific sites to go via the NAS out through the external VPN, whereas all other network connections will go via the router to the WAN directly.

pfsense can handle that!

I was planning to achieve this by having the bt service user having access to only ppp0 and nothing else, and the vpn server user having access to ppp0 and its own connection (ppp1?). The rsync user will have access to (ppp1?) and usb ports only. If I have pfSense external box, how would I manage these security requirements? I never found a way to achieve this when running the synology + custom router (Pentium 2 based box with IPCop installed, PSU has since died and I chucked the rest given its age). Also, this will be very expensive given i'd have to change my wifi modem router to bridge mode, and purchase

you will always have the problem in your config that the freenas box itself will have access to usb and so on.
Its just not designed for what you want.

First read the Manual and the feature sheet.

Rethink what you really want and need AND why you need it that way you describe here. Thats not the way things or freenas work.

Sorry if my post is a bit aggressive, it isnt meant to be

 

[depasseg]

FreeNAS can be installed onto a boot device (or mirrored devices) without any data disks.
Nothing except the OS and the system dataset (system config and logs) can reside on the boot pool.
The System Dataset can be moved off the boot pool onto a different dataset.
Jails can be used for the services you would like to run.
Jails need a Dataset to reside on. This Dataset can be on any Pool except the boot pool.
You can create multiple pools. Each pool can be as small as a single vdev with a single device, or multiple vdevs with multiple devices (and everything in between).
You need a dataset for data and shares. Data Datasets can reside on any pool except the boot pool. Datasets can have sub-datasets.

Can you clarify what you mean here:

- A number of small VMs (VMDKs stored on the disks, VMware on the microSD)

And here, the term "vdev" is not being used correctly. vdevs are devices, mirrors, or RAIDZ(1,2,3) entities that comprise a pool. Pools can contain 1 or more datasets (and sub-datasets) which can have quotas, but otherwise have no size constraint except that of the underlying pool.

vdev setup:
500Gb - Super Important - back up to 1.5 Tb and 1Tb
500Gb - Important - back up to 1.5 Tb and 1Tb
500Gb - Semi Important - back up to 1.5 Tb only
3Tb - Really not important - The stuff that won't be transferred across, and not backed up anywhere. First to go if RAID fails (i.e. 3 disks go bad) or site fails (i.e. house catches fire).
remaining - Not Important

 

[jkim]

Attached USB HDDs are not a good idea in freenas. Skip that!

So how would you do backups to external drives then?

you will always have the problem in your config that the freenas box itself will have access to usb and so on.

Not sure what you mean by this. The box needs to have access to usb (back up certain partitions to usb external drive only), but certain users on the box should be banned usb access. The usb drives won't be shares... just backup targets. A script will mount the device (ntfs), perform rsync, and umount the device. The script will be called by a cron job as per the schedule set out below for each partition.

the number of redundancies is good, but not the way you would like to it (single vdevs, usb disks on FN...)

Perhaps my misuse of the vdev terminology (pointed out by depasseg) is causing some confusion.
Using traditional terminology I know:
4 disks in RAID6 equivalent = 12 Tb of usable space.
This divided into the 5 partitions mentioned = space on the NAS.
Partition 1 (0.5Tb) gets rsync'd to both usb externals and the off-site machine once an hour
Partition 2 (0.5Tb) gets rsync'd to both usb externals and the off-site machine twice a day
Partition 3 (0.5Tb) gets rsync'd to only one usb externals and the off-site machine once a day
Partition 4 (7.0Tb) gets rsync'd to only the off-site machine
Partition 5 (3.5) doesn't get rsync'd

Partition sizes set to accommodate the relevant external backup target sizes.

If no good, how would you achieve backup to external requirements?

Can you clarify what you mean here:

Some test VMs used infrequently (around 4 hrs / month average use), running on any vm app available.
Primarily to test new software under multiple OS (Solaris, Oracle Linux, Windows 2012 Server).
Can run the app on the client desktop and just have the vmdks hosted on the nas if hosting on the nas is going to be a problem, but would imagine that will run very slow.

And here, the term "vdev" is not being used correctly. vdevs are devices, mirrors, or RAIDZ(1,2,3) entities that comprise a pool.

Perhaps my misuse of this terminology has caused some confusion... I meant the equivalent of partition under ZFS... whatever that may be. Apologies for the confusion and my ignorance.