Need near-realtime file synchronization for desktop computers

[ttblum]

Hello,

Currently, I have an application that stores files on a shared Windows folder that my computers have access to. I also have a remote site that accesses this shared folder over a point to point T1 line. There are about four Windows workstations at each office.

I'd like try to replace the expensive T1 line with file synchronization software running over a VPN. The synchronization would need to be as tight as possible to prevent multiple users from working on the same file at once (each file is modified when a user starts working on it). There are a few hundred files in this directory, and all the files need to be scanned by the application when it starts up, so running the fileshare directly over a VPN causes too much of a delay.

I find that both Dropbox and LogMeIn Cubby both are able to keep remote directories consistent with about a 5 second lag time. I tried the OwnCloud plugin running on FreeNAS 9.2 Supermicro server, with built-in SQLite database, and there was usually at least 10 seconds to consistency (the Owncloud directory had about 10 files in it total).

Would OwnCloud be faster using a MariaDB database?

Or would it be better to use Dropbox or Cubby in this case, or is there some other solution that might be a good fit? Obviously the security would be easier to control if everything was running on my own server.

 

[fracai]

It sounds like what you really need is some revision control software that includes file locking. SVN includes this and it's supported by Tortoise.

 

[Joshua Parker Ruehlig]

you could also look into btsync. you can set static ports, known hosts to not rely on tracker servers. I'm experimenting on this for offsite backup to a servers with an older zfs version.

 

[ttblum]

I tested btsync and found it took 10-15 seconds for convergence after an update, a bit too slow for this project. I'm assuming TortoiseSVN would have similar performance? The concept of file-locking sounds intriguing, though. It seems odd that there's no network filesystems that are compatible with Windows that do local caching in low bandwidth situations. There is 'offline files' of course, but it doesn't seem to work very well.

 

[Joshua Parker Ruehlig]

you could try owncloud like you said. I wrote a tutorial on setup using mariadb. owncloud uses csync as the backend syncing technology, so looking into that might be fruitful as well.

 

[Joshua Parker Ruehlig]

hmm, some other thoughts. if you're over a VPN with btsync you could disable lan encryption, and tune the advanced settings. I think btsync is by default tuned to send large block sizes, but your application may only be making small change to the affected files.

 

[fracai]

SVN won't sync automatically, but from your description, you wouldn't need it to.

Each user would get a copy of the file in their local checkout from the SVN server. When a user wants to work on that file, they would update to receive any changes and then submit a request to lock the file. At this point, they are the only user that can submit changes to the file. When they're done editing the document, they would save and then checkin their changes. They can the unlock the file and another user is then free to lock and edit on their end.

As long as you institute a process of not editing a file unless you have a lock, you'd be fine. Plain text files generally wouldn't need to lock as they are easier to merge in changes, but it sounds like you're talking about binary / proprietary formats.