Two rsync issues

[postazewski]

I got rsync working with the push/pull setup. I have two issues though.

First issue is the rate at which the data on the push box gets sent to the pull box. It's very inconsistent. I can't seem to figure out what's causing it to jump all over the place. I've checked the nic's in both boxes, even had HP come out and replace the mother board on the pull box because i was certain the nic's were bad. I've tried numerous switch ports - no difference. My 2 freenas boxes are used as such - Veeam backs up to the push box and gets great speed doing so. The pull box takes the backups from the push box. The push and pull boxes are in different buildings, both connected to gigabit cisco switches and the buildings are linked by gigabit fiber. I've included a graph from my MRTG to show how erratic the speed between the boxes is.

Second issue is that I can't figure out how to make the pull box delete the older backups, which causes it to run out of space. Not sure how entirely that the push box seems to delete the older backups - pretty sure Veeam does that... So i need to figure out a way to make the pull box only keep the same backup files that the push box keeps.

 

[postazewski]

BuildFreeNAS-9.2.0-RELEASE-x64 (ab098f4)
PlatformIntel(R) Xeon(R) CPU L5520 @ 2.27GHz
Memory8162MB
System TimeTue Jan 13 12:19:01 CST 2015
Uptime12:19PM up 196 days, 3:55, 0 users
Load Average0.25, 0.31, 0.32

I tried to load newer versions of freenas at one point, but found that anything over 9.2.0 wouldn't work. I can't remember why. I think it had to do with not finding the nic? I'd have to look back at notes.

 

[cyberjock]

rsync was never meant to be consistent. It sends small amounts of data as it looks for new files or changed files (and then uploads them). You're just seeing the lulls between file transfers. That's totally normal.

As for deleting the older backups I can't give advice. rsync overwrites files typically, so there shouldn't really be a "history". If you are using Veeam and it is responsible for those files then I'd look at support from Veeam on how to delete the old files.

 

[postazewski]

Is there anyway to make it faster? Each file that rsync is sending is 1.5tb. You can imagine how long that takes. ex of a file: 1519936874496 Dec 28 17:48 Daily2014-12-27T213445.vbk

The issue is that the filenames change. Each new backup has the current day in the filename. So there's no way that rsync would overwrite and that's why they keep accumulating.

 

[cyberjock]

Nope. Rsync was never designed to be ultra fast. It was designed to be easy on network resources. You will get better throughput with ZFS replication.

 

[postazewski]

How do i set up ZFS replication?

 

[cyberjock]

doc.freenas.org has instructions on how to setup snapshots and replication. :)

 

[postazewski]

The two servers I am using have hardware raid. Does ZFS support hardware raid?

 

[saurav]

The issue is that the filenames change. Each new backup has the current day in the filename. So there's no way that rsync would overwrite and that's why they keep accumulating.

If rsync is sync'ing an entire directory and those files are inside that directory, you could probably use the "--delete" flag to remove them. There's also an option in the gui, but be sure to see the manpage first. It *is* dangerous and rsync makes it hard to use it easily.

I use rsync over ssh at home, and use HPN-SSH between an old Macbook and FreeNAS. It requires disabling encryption (via command-line) but makes a difference of 7-8 MB/s over ssh, which is acceptable for traffic entirely within home network.
http://www.psc.edu/index.php/hpn-ssh

Edit: HPN-SSH is built into FreeNAS

 

[depasseg]

The two servers I am using have hardware raid. Does ZFS support hardware raid?

Your question doesn't make sense. Apparently you missed the plethora of ZFS guides. ZFS replaces the HW raid and should have direct access to the drives. So, yes, you can run ZFS on HW raid. But it is a horribly bad idea.

 

[postazewski]

I was looking for a quick answer. Sorry. I just read about it a little and found exactly what you wrote.

I have 8GB of ram in each of these HP DL180 servers. Is that going to be enough to run ZFS? I have 6 drives configured with one hot spare.

 

[cyberjock]

8GB of RAM is the *minimum* for FreeNAS. If you have any kind of expectation above basic file sharing you'll need more.

 

[postazewski]

Is there any way to set up a cron job that ftp's the content of the share to the other freenas box? Rsync isn't going to work, and I'd have to dump at least another 6GB of ram into these boxes (guestimate) in order to get ZFS to work. I need the copy speed to be as fast as it can go. If i were dealing with small files, the speed wouldn't be an issue. But moving 1.5-2tb files at snails pace just isn't going to cut it.

 

[cyberjock]

Cron is nothing but a scheduler for scripts. So if you are handy with scripting I have no doubt you could script it to work.

Just an observation, but if 6GB of RAM is going to make this a problem I'd question if the rest of your hardware is anything close to recommended hardware for FreeNAS. Adding RAM isn't overly expensive (especially just 8GB of RAM). If adding more RAM is going to be a financial problem I'd kind of cringe at what you are using. Generally people start cutting corners and think they're going to have a viable FreeNAS server by going with cheap hardware, and then later it blows up in their face when the server up and dies one day without warning. Sometimes this leaves the user with corrupted or otherwise failed backups and the user finds they have no data.

I'd definitely read around the forums and make 100% sure you aren't cutting corners that are going to affect your server's reliability. Generally as soon as people start saything things like 'I can't spend that kind of money' they're already bargaining with the devil and the devil always wins the bet. I'd hate to see you be the next guy that loses data because of some silly mistake you didn't think was a big deal, but was, and you lost all of your data as a result.

 

[postazewski]

I work for a school district and money is tight. We don't have all sorts of money to blow. This solution was originally set up by a consultant, and that is why I don't know much about it. All i know is that it doesn't work like we need it to work, so i'm trying to find the least expensive way to make it work. The hardware is fine. It's all genuine HP branded gear (DL180 G6) and I've got carepack's on both of the servers so they're covered under warranty nbd parts. I'm not actively trying to cut corners. I just don't know much about ZFS and if i spent 200.00 on memory for these boxes and ZFS ends up not working... Then it was a waste of money and the district won't be happy.

The content saved on the freenas boxes are backups of VM servers. So if they're lost, its not the end of the world unless of course, a VM goes down or someone deletes a file and i need to restore. I don't like playing the "if" card but I also need a solution that will 100% work. I definitely understand what you're saying, I just need to know, moving forward with this, that it will 100% be the solution to my problem.

 

[cyberjock]

I work for a school district and money is tight. We don't have all sorts of money to blow. This solution was originally set up by a consultant, and that is why I don't know much about it. All i know is that it doesn't work like we need it to work, so i'm trying to find the least expensive way to make it work. The hardware is fine. It's all genuine HP branded gear (DL180 G6) and I've got carepack's on both of the servers so they're covered under warranty nbd parts. I'm not actively trying to cut corners. I just don't know much about ZFS and if i spent 200.00 on memory for these boxes and ZFS ends up not working... Then it was a waste of money and the district won't be happy.

The content saved on the freenas boxes are backups of VM servers. So if they're lost, its not the end of the world unless of course, a VM goes down or someone deletes a file and i need to restore. I don't like playing the "if" card but I also need a solution that will 100% work. I definitely understand what you're saying, I just need to know, moving forward with this, that it will 100% be the solution to my problem.

Ok. But you can't vouch for the following:

1. Did the consultant cut corners?
2. Are you even using hardware that won't shatter your dreams (and your career) when something goes wrong?
3. What actually *is* required for a 100% solution?
4. What do you need to do to get it to a 100% solution?
5. How much will it cost to do #4?
6. Will the district pay for it?

What you need to do is be able to answer ALL 6 of those questions on your own. If you can't then you shouldn't be touching the box until you can. If you can't get time (or don't want to deal with it) then the solution is to pay someone else to make it right, go buy something read-made from a company like iXsystems where the cost is higher but you have a 100% chance of having a workable system, or just go with a totally different solution. Nobody here can tell you what the 100% solution is for your problem. That's your job as the admin. You're going to have to figure that one out for yourself.

I'm not asking you to blow money. In fact, you'll find that this forum offers the no-nonsense short and inexpensive answers with regards to hardware. We could easily tell you to buy outrageously expensive hardware. Did you even look at my hardware recommendations thread? I recommended a $60 CPU! How much less expensive do you wanna get?

We're not asking to spend money on crap that doesn't matter. In fact, the #1 problem is that people *spend* money on crap that doesn't matter, then skimp in the areas that *do* matter. Then they have a box that eats their data one day (or doesn't even work on day one). Considering the people that come in here and don't know what they are doing I'm almost willing to bet cash that your consultant didn't do their due diligence and you've got a box that will eat your data one day.


Measure twice, cut once.

 

[postazewski]

Ok. But you can't vouch for the following:

1. Did the consultant cut corners?
2. Are you even using hardware that won't shatter your dreams (and your career) when something goes wrong?
3. What actually *is* required for a 100% solution?
4. What do you need to do to get it to a 100% solution?
5. How much will it cost to do #4?
6. Will the district pay for it?

What you need to do is be able to answer ALL 6 of those questions on your own. If you can't then you shouldn't be touching the box until you can. If you can't get time (or don't want to deal with it) then the solution is to pay someone else to make it right, go buy something read-made from a company like iXsystems where the cost is higher but you have a 100% chance of having a workable system, or just go with a totally different solution. Nobody here can tell you what the 100% solution is for your problem. That's your job as the admin. You're going to have to figure that one out for yourself.

I'm not asking you to blow money. In fact, you'll find that this forum offers the no-nonsense short and inexpensive answers with regards to hardware. We could easily tell you to buy outrageously expensive hardware. Did you even look at my hardware recommendations thread? I recommended a $60 CPU! How much less expensive do you wanna get?

We're not asking to spend money on crap that doesn't matter. In fact, the #1 problem is that people *spend* money on crap that doesn't matter, then skimp in the areas that *do* matter. Then they have a box that eats their data one day (or doesn't even work on day one). Considering the people that come in here and don't know what they are doing I'm almost willing to bet cash that your consultant didn't do their due diligence and you've got a box that will eat your data one day.


Measure twice, cut once.

This post is exactly why I tend to stay away from forums. I came in here seeking out a specific answer to an issue I was having and now for whatever reason I'm being preached to by the know it all of the forum.

100% to me is having a fast backup of my main Freenas box. The rest is working fine and has been for 2 years roughly. I'm not worried about losing data. So please stop bringing that up.

The hardware is fine. It's not brand new but it is reliable. I've already explained what I'm running twice now, so I'm not sure why it's still being brought up. Yes, I read your requirements.

Again... I will ask.... If I install ZFS will it give me the transfer speeds I'm looking for between boxes? I will explain again what I'm looking for: I have multiple 1.5-2tb files that need to be copied to the second freenas box every Friday when a full backup is done. I need those files to be copied quickly. Bandwidth utilization is not an issue. Rsync is too slow. It takes days to move the files. I need a faster copy option.

Can I please get an answer to that question

Thank you

 

[depasseg]

look at responses #5 and 7. ZFS replication.

Or script something and use 'cp'.

 

[depasseg]

Oh, and if you have rsync issues, why not look into that? You can try optimizing the rsync configuration (like delete files on the destination and using the -W (whole files) option).

I mean you came to a FreeNAS forum for help with something mildly related. Of course we are going to analyze the FreeNAS aspect and not rsync.

 

[Z300M]

I was looking for a quick answer. Sorry. I just read about it a little and found exactly what you wrote.

I have 8GB of ram in each of these HP DL180 servers. Is that going to be enough to run ZFS? I have 6 drives configured with one hot spare.

I don't think anyone else has mentioned it, but I don't think FreeNAS supports hot spares; i.e., it won't automatically offline a bad drive and replace it by your spare.

8GB of RAM is the minimum recommended; adequate performance may require more.