Plan to sync Work/Home servers

[zigzagman]

Hi,
My apologies if this has been addressed in previous threads.

My partner and I run a 'micro' graphic design/media business which, for both family and business reasons, is re-establishing just how we go about growing in an ever changing business (and social) environment.
I have used a HP N40L server with Freenas (version 8) in our work location for some time now, and have been impressed with it's dependability and speed. We are now splitting our work environment into a work office and a home office and it's down to me to make the new server(s) work. Unfortunately, I'm still a 'Nubie' with setting up communications between the systems.

My plan is this:

1. Purchase an additional HP N54L and install Freenas ver 9.2.1.8 (16gB memory on each machine). Match disk array (4gb) and raid type to the N40L machine, so essentially, they are equivalent machines.
2. Clone all data from the original server to the new.. Can anyone suggest the best way to do this in a 'close coupled environment. i.e using local area network.
3. Move the new server to work location (slightly better performance), move original server to home location.
4. Set up ADSL modems with static IP addresses. Should I use a VPN for this connection? Should I also set up SSH for security?
   
5. Establish sychronisation between the two servers (using RSync?).

We essentially want to be able to work on the same files from either the work location or the home location, using in either case, the local server attached to the LAN. Apart from achieving a work pattern which is continuous and efficient either at work or home, I am thinking that this will give me a much better back-up solution than the rather ad-hoc methods we have used in the past.

Can someone either confirm that I'm on the right path with all of this, or let me know if there are better ways to achieve the work / home server sychronisation.
I would appreciate any help and suggestions..

Frank

 

[solarisguy]

You are on the right track!

1. Good that it is ECC RAM. You do not need to match anything but the total size of the storage. Effectively you can use larger disks and move up from a mirror or RAID-Z1 to RAID-Z2. Do not use built-in RAID. Can you have disks being directly accessed?

2. If you feel like a Nubie, just use copy between two network drives on your Windows workstation.

3. So you had it at home? Was your partner accessing it over the Internet? How were you accessing the server in the real office? (These are not a mere curiosity questions.)

4. If the IP addresses do not change often, you may want to pay for more bandwidth instead of paying more for a static address. If VPN, then you do not another encryption layer. So you can use HTTP instead of HTTPS. With SSH you may have no choice. But you can use rsync without any encryption.

5. Yes, rsync is a good choice.

You may want to always access your data over the Internet, regardless whether you are at home office or in the real office. See my question above about current access.

Good luck!

 

[zigzagman]

Thanks Solarisguy,
Yes, I set out to make the two NAS boxes essentially the same in ECC memory, 100gb SSD cache and 4 x 1TB (ZFS RAID) and you're right, a straight copy will work ( I forgot that the FN system is on a separate USB flash drive so there should be no locked system files to content with).
I might have misled you about the arrangement so far.. up to now we have been using the existing NAS on a Windows network in our office only.

Although we have only occasionally accessed the existing NAS over the internet, the intention (dream) is to end up with two identical NAS systems, constantly mirrored between work and home, thus providing a close to identical work environment at both sites whilst ensuring we have a real-time backup at both sites. We will have in effect a LAN in both locations with effectively the same mapped drive (FreeNAS) which (by the 'grace' of RSync') will be mirrored and provide us with a consistent work environment in either location.

I guess I was looking for some assurance that I'm on the right track to achieving this and I thank you for your help and suggestions.
Cheers,

Frank

 

[Mlovelace]

Unless you want to mirror data at two locations for disaster recovery there is no reason to build a second box. Just setup site to site vpn and access your freeNAS server from home.

 

[solarisguy]

Unless you want to mirror data at two locations for disaster recovery there is no reason to build a second box. Just setup site to site vpn and access your freeNAS server from home.

There is a strong reason to have one FreeNAS machine being accessed from everywhere, and the second FreeNAS machine being only an rsynced backup/nearline disaster recovery.

@zigzagman, if you do not do that, you may have problems with your data getting out of sync, since rsync is not the same as mirror.

 

[Dennis.kulmosen]

Having a system like what you describe is not that easy. You may look into a 2-way sync like unison or something similar. I dont know if you can do something with BTSync. But both BTSync or unison needs to be installed in jails.

 

[zigzagman]

OK, I think there are more complexities than I was aware of e.g I did mistakenly think that RSync could act in a bi-directional manner.. i.e as a two-way mirror; not so, it seems.

My initial plan was really to accomplish two (simple, I thought) tasks.. Firstly, make it 'transparent' to a user as to whether they're working at home or the office. Implicit in this is to achieve similar responsiveness when doing design or publication work with linked files etc. whether in the work or home location. In my limited experience with VPN in the past, it always appeared limited by bandwidth/thoughput (maybe because our area is still operating in the copper age). The idea of two "sync'd" servers appeared to get over this. i.e. the two NAS systems would sync their data continuously despite slowish internet speeds, so we could always work on the local LAN NAS knowing they were working with the latest file. The second task was probably even more important to provide back-up via replication (whether we were diligent or not!).

I really appreciate all the input. I don't think 'the dream' was unreasonable, but I certainly think I need to delve deeper into the documentation to understand the limitations better..

 

[mjws00]

The dream isn't unreasonable at all. Especially in a media environment where VPN WAN access may be so slow as to be painful.

The implications being raised are the differences between your two servers at any point in time. The built in tools aren't really designed around "real-time" mirrors. Most involve a push or pull scenario. I don't know how well rsync works in a granular two-way scenario... I've never used it that way. But there are definitely tools that will sync your two sets of data in near real time and differentially. Btsync does a nice job. But I'm sure there are other more corporate packages.

The second task of ensuring there is a backup via replication is pretty typical.

Just keep digging and you'll get there.

 

[solarisguy]

Syncing is never the same as mirroring.

If when not together in either office, the users work with entirely separate data, then you can sync data and be OK.


Did you calculate how many gigabytes each user is changing during their work day? Worst case scenario, in an hour?

 

[zigzagman]

Yes, the data modified by users working in either location is always different. i.e. each user has a customer project distinct from other users.
Max. transfer per day 1.5gb, (say max 500 mB / Hr) but averaged over the month less than 25gb... not large by comparison with most NAS servers I would expect.

A hypothetical question (and possibly a stupid question from one who is time poor and yet to read all the information!):
Is it possible to run RSync on each NAS box each syncing to the other on a continual basis? Would this not result in exactly the status I'm after for each of the boxes?

 

[Chris Stankevitz]

Is it possible to run RSync on each NAS box each syncing to the other on a continual basis? Would this not result in exactly the status I'm after for each of the boxes?

No. Imagine if every day at noon site A rsyncs to site B. And every day at midnight site B rsyncs to site A. This means every day at noon, B will lose all of its changes over the past 12 hours and will be overwritten with A's state. And every day at midnight, A will lose all of its changes over the past 12 hours as it will be overwritten with B's state. The same problem exists even as you increase the sync frequency.

Chris

 

[leenux_tux]

Another possible way of achieving this would be to have two folders for each user on each server. More to manage I know in regards to rsync jobs, however, I think the trick with this is being able to guarantee which version of a file is on what server and where on that server. For example. User BILL has two folders on HOMEOFFICESERVER and two folders on REMOTEOFFICESERVER (HO and RO for short). Updates happen over night.

Monday
File on homeserver "/mnt/TANK/BILL/HO/DOC123.txt"
On Monday DOC123.txt gets changes made whilst BILL works at home . The file is transferred to the remote office into the same folder structure, thus mirroring the homeserver
File on remoteserver /mnt/TANK/BILL/HO/DOC123.txt

Tuesday
BILL travels to the office on Tuesday and works on /mnt/TANK/BILL/HO/DOC123.txt, HOWEVER, saves changes to /mnt/TANK/BILL/RO/DOC103.txt. The file is then rsynced to the homeserver into /mnt/TANK/BILL/RO/DOC123.txt

Wednesday
BILL, who is back at homeoffice, now has the choice of two copies of DOC123.txt to work with. The original in folder HO which he worked with on Monday, or, the one he worked on yesterday whilst in the remote office. If he continues working on the file in /RO, and wants it available on officeserver then next time he goes in, save it to /mnt/TANK/BILL/HO/DOC123.txt and it will be rsynced over.

Complex I know but I think it's workable ??

​