Group:SyncReplacement

SyncReplacement is an FSF Priority Project. Bittorrent Sync is a peer-to-peer, two-way file synchronization utility with fine-grained access controls. We need a free software version of this client or free software that can be used for the same purpose.

If you are interested, please join this group (start by leaving your name here). We should start by surveying any existing free software that is in or close to this area (TahoeLAFS might be one), and making a list of features that are needed.

The SyncReplacement IRC channel can be found at #freesync, on Freenode.

Important News!

Recently, I posted my idea for a BTSync replacement here, below. After it was posted, I began communicating with a volunteer for the German federal disaster relief organization, THW. He wants to potentially use this program in connection with their disaster relief efforts, but he needs it to have additional functionality, including the ability to synchronize between more than two computers and the ability to do block-level synchronization. I have been trying to add that functionality, but it obviously takes a good deal of time to do so, and more than I currently have. Additionally, I need multiple computers to test the latest version, a resource which I do not have. Would anyone be interested in working with me to add some of this functionality to the program and help test the program? If so, please let me know. You can contact me at jhunter (dot) dunefsky (at) gmail (dot) com.

Thanks Hunter D

Existing Solutions

We need a survey of existing solutions, their features, pros, and cons.

In development

Currently there's a protocol description implemented by Jewel in a ruby client:

https://github.com/jewel/clearskies

And there's an effort to create a portable C++ daemon for the clearskies protocol:

https://github.com/larroy/clearskies_core

Librevault

Librevault is an open-source BitTorrent Sync alternative. Written in C++, it is under active development now. The protocol uses TLS as a transport.

Implemented features

• Decentralized peer-to-peer folder synchronization via the Internet and inside a LAN.
• Adding folder by key (Secret, in terms of Librevault).
• Read-Only Secret. Using this Secret, peer can synchronize with a folder, but cannot change its contents.
• Encrypted Secret. Using this type of Secret, peer can download, distribute, but cannot decrypt (and read) the folder contents.
• Block-wise incremental synchronization. On a change of file, only the changed block will be transmitted, but not the whole file.
• inotify, fsevents, kqueue and ReadDirectoryChangesW API support for monitoring directory for changes.
• Symbolic link support (experimental, optional).
• Windows- and Unix-attribute sync support: mode, uid, gid, windows attrib (experimental, optional).
• Cross-platform: Linux, Mac OS X and Windows supported now. The code is written in a portable manner, and it will be ported to other platforms later.
• All the data (and filenames) is stored on the clients, and transmitted only in AES256-encrypted state.
• Multiple peer discovery strategies:
  • UDP multicast node discovery on the LAN.
  • UDP BitTorrent Tracker discovery on the Internet (so, existing solutions, such as OpenBitTorrent are compatible with Librevault).
  • Full DHT support. Librevault uses DHT implementation from Transmission, so it is fully compatible with Mainline DHT specification with IPv6 extensions.
• UPnP and NAT-PMP port forwarding support. PCP is in our plans.

Sync daemon uses Boost, Crypto++ and Protocol Buffers. GUI uses Qt 5.3 (min). The protocol between Daemon and GUI is WebSockets. WebUI will arrive in next releases.

Documentation and protocol specification are Work-In-Progress now.

Librevault is released under GPLv3 license.

Hive2Hive

Hive2Hive is an open-source library, written in Java, for distributed, P2P-based file synchronization and sharing. It provides a free, distributed solution that focuses on maximum security and privacy of both users and data. It supports the whole feature set known from similar centralized approaches, such as Dropbox or Google Drive, all packed in a clean, extendable API. On top of that, it has several other advantages that are described on their website. The project is hosted on GitHub:

https://github.com/Hive2Hive/Hive2Hive

syncthing

Under active development in Go, syncthing claims to be an "Open Source Continuous File Synchronization" solution. It specifies a protocol Block Exchange Protocol and a reference implementation. Syncthing is released under the GPLv3 license.

osync

A two way filesync script with fault tolerance, resume, soft deletion, conflictual file backups running on bash (linux, BSD and virtually any system supporting bash). File synchronization is bidirectional, based on rsync, can be run manually, as scheduled task, or triggered on file changes.

Non-Free

BTSync

The software that lit a fire under this project.

Cubby

A DropBox competitor that offers peer to peer file sync called 'DirectSync'.

Free

This section might need pruning, the things listed might not be entirely free.

RetroShare

A decentralised communication platform that allows file sharing.

Features

peer discovery with DHT

NAT traversal

encryption and authentication with PGP+OpenSSL

gui to manage friends+groups

additional functionality like chat, groupchat, voip... I think it is nice to have chat and shared folders at the same place. This would enable fast collaboration.

Retroshares core "libretroshare" is written in C++, and runs on every platform with sockets and a file system

Retroshare-gui uses Qt and runs on win, linux, mac

Plugins can use the secure link to send/receive data to/from peers.

Resumable transfers and swarming

Pros

no worry about finding peers or securing the connection

Retroshare has a community of developers, testers, translaters and packers. So we could concentrate on the main thing: how to get different states of a folder in sync.

Cons

Perhaps this does too much? We want a drop in replacement for BTSync, this comes with a whole raft of additional features that might not appeal to users.

git-annex assistant

git-annex can be used for basic synchronization or more sophisticated setups, such as syncing by way of an intermediate USB drive instead of over the net (see "special groups").

The underlying system is based on git and rsync, but the web interface can be used to set things up without much command-line knowledge.

The "assistant" part is the service that watches for changes, makes commits, and automatically syncs to remotes.

Our Solution

Language

• So far, either C, C++, Python, or Go seems like our best bet.

Features

• simple, everyone should be able to use this tool
• decentralised, no central server needed for operation
• secure, communication should be encrypted and authenticated
• cross-platform: win, linux, mac
• can delete files
• can cope with conflicting changes
• can cope with wrong clocks
• can cope with moved or renamed files

The Protocol

for every shared folder we have:

• name
• some id
• admin pgp id
• list of users+permissions
• could be signed

(set by admin, so update is simple with timestamp)

created by users with permissions:

• ever growing list of file metadata items
  • every item is unique by nature, so sync is simple

file metadata item:

• filepath/name
• id of previous file, this id could be the hash of the other item
• size, hash
• timestamp
• mark if file is deleted
• author id
• item could be signed

How sync works:

1. file metadata items are exchanged, so everyone has the same list
2. fileitems with same path/name are ordered, so that we know the newest state of the file
3. files are transferred

Conflicts need to be handled by the user

More thoughts

• when online, it could be signalled when files are in use
• overlay icons like tortoise svn
• differential sync
  • don't know which filetypes would take advantage of this
  • docx+odf is based on zip, zip compresses files independently
• keep old files to some point in hidden directory for undo
• sync of big list:
  • order in binary tree
  • compare checksums of treeparts
• use gxs to sync items? (gxs is a new transport for the forums inside Retroshare)
• use existing filetransfer classes?
• how to tell if folder is in sync?
  • folder is in sync when we synced with >50% of peers with write access
  • or define master peer
• maybe combine with chat client
• Sync only some file/folder in the "main" folder

People

Here's a list of people that have volunteered to be a part of this project so far. Feel free to add yourself to this list.

• Hi, I'm Franco Masotti. I'm writing a free implementation of a Dropbox-like service based on git and bash. I know it's not decentralized but there is no real server program. You just need and SSH server, git and some other packages. At the moment it works but I've not yet completed all the points I want to achieve. Contributors are needed in order to speed up the development. Licensing is GPLv3+ and all the packages used are free software (I'm using Parabola GNU/Linux-libre as one of the testing and developing machines). Project is hosted on github: https://github.com/frnmst/gnupot . Thank you. UPDATE: Most core points have been achieved but I'm looking for volunteers and testers. Please have a look at https://github.com/frnmst/gnupot/tree/dev#gnupot-is-looking-for-volunteers
• Hi, I'm Alexandre. I would like to work on this project, you can contact me at reazem@reazem.net
• Hi, I'm Hunter. I've started work on this project, using Python 2.7.3. So far, it supports bidirectional synchronization. I'll get to work adding a password system. You can find the GitHub repo [here]. You can shoot me an email at jhunter.dunefsky@gmail.com.'
• Hi. This is Rajul. I wish to work on this project. Please contact me at rajul.iitkgp@gmail.com
• Hello, I'm Pratik. I would like to contribute to this project. You can contact me at impratikb@gmail.com
• Hi, I' Isengaara - I installed GNUnet on my Trisquel System, GNUnet offers P2P File Sharing, but no sync yet. Maybe it could combined with git?

Gittorrent seems dead. contact me at tobias [ät] platen-software [döt] de

• Hi, I'm Fred Morcos - I'm interested in such a project and would like to contribute. I am experienced in C and Python. One part of my proposal would be to not try and recreate the protocol but instead write a free compatible implementation of BTSync. You can contact me at fred dot morcos at gmail dot com. *Edit:* Since people have also suggested programming languages like Go (which I'm open to looking into for this project), I would also like to have the D programming language under the radar. About licensing, I am for GPLv3.
• Hi, I'm Patrick Steinhardt - I'd be interested in contributing to this project. My relevant skillsets include C, C++ and Python. You can contact me at steinhardt.ptk [at] gmail [dot] com.
• Hi, I'm Adrien. I'd be interested in this project as well. I actually implemented a prototype for a very similar system as a class project. I'll deliver the source soon. I think it's probably better to rewrite everything from scratch (in particular I'd like to use python instead of C++), but the code I have and the lessons I learned should be very useful. adrien [dot] projectemail [at] gmail [dot] com.
• Hi, I'm Gerry. This is a great initiative. I would strongly suggest considering Go (http://golang.org/) as a language for this. It's cross platform, has excellent concurrency support and is compiled, so users won't be required to install Python. It is far better than C/C++ in terms of providing a rich standard library and garbage collection. There are already existing BitTorrent clients written in Go, under free licenses.
• Hi, I'm Jumpwah. Here is a similar group most likely looking to collaborate on the same project: https://groups.google.com/forum/#!topic/golang-nuts/7WUj3nASuLo. In addition to deciding on the language, we should also decide on the license early to resolve any conflicts/doubts. Contact: jumpwah at google's electronic mail service.
• Hi, I'm Manuel. I've been spending some time lately trying to analyze btsync's network protocol (and working on a library that should speak that protocol). Currently it's just some Qt code hacked together with the aim to being able to communicate with btsync at all. The stuff I've found out so far can be found on github: https://github.com/picosync/workingDraft/ (have a look at the wiki and feel free to contribute). Contact: picosync (a) r7r .at
• Hi, I'm also interested in doing some coding. Always wanted to do something with Go, too. Contact: foss (a) paulstaab (dot) de --Paul
• I want an open source clone of btsync, and can help with coding starting around jan 1st 2014. Computer engineering student, senior undergrad. Experienced with C and Python, and some Java and C++ (and some other stuff that isn't as relevant). Learning Go. Haven't worked on an open source team before, though. Please let me know if we get a group/irc/forum going. "syzothermy" on google's widely-used email service.
• Hi, i'm electron128 (a) yahoo (dot) com. I'm a big fan of Retroshare: http://retroshare.sourceforge.net/ My idea: create a folder sync plugin for Retroshare.

I have experience with Java, Python and C. I learn C++ and Qt.

• Hi, please take me into account for this project, you can reach me at luafran (a) gmail (dot) com.
• Hello, I'm Jewel. I've been working on a replacement for btsync for the past few weeks. So far I've created a draft protocol (which you may find useful for your own efforts) and am working on making a reference implementation. https://github.com/jewel/clearskies.
• Hi I'm William, I'm interested in helping out with this project. I'll watch this page for details of any Google Group or GitHub repos.
• Jacob Saporito, jmsaporito@ucdavis.edu. I'd like to get involved with this project.
• Hi, I'm Braydon Saporito. Sounds like a neat project to get involved with. I will keep an eye on the github and page to contribute when I can. altabraydon@gmail.com
• Hi, I'm Burcu. I've been interested in decentralized sync protocols for a long time, and spent a notable amount of time trying to alter btsync. I would like to contribute where I can. I'd suggest Go programming language, similar to some of the other commenters. burcujdogan@gmail.com
• Hi, I'm Gabor and I'd like to participate in this project on development side. Through the last decades I've collected knowledge Linux (other UN*X like OSes, such as AIX, OSF/1 and so on), programming languges (sort of C like syntax), systems engineering, et cetera.
• Hi, I'm Chris and I'm working on the Hive2Hive library project, which is a free and open-source solution for distributed, P2P-based file synchronization and sharing. In addition, it focusses on maximal security and privacy. You can contact me over the contact information on GitHub.
• Hi, I'm Joop, and I am very interested in this goal for a long time, and have been pondering for that same long time about how the PSK system in BTSync might have been implemented. My writeup about a possible solution can be found here: http://wiki.lapingvino.nl/Pre-Shared%20Key%20syncing%20blueprint , please contact me at ikojba@gmail.com if you plan to use this, just so I stay updated about all progress.
• Hi, i'm Prabodh Agarwal, email: prabodh1194[at]gmail[dot]com. I am a student and have been working on SDN and have interest in P2P networks. I am an enthusiastic coder, and have decent lot of exposure to C language. Please add me for this project.
• Hi, I'm Martin, I'd like to contribute to this project. Reach me via martinDOTnobisATgmailDOTcom. I have expertise in C but would also like to code in other languages. I'll try to keep abreast of the news for this project.
• Hi, I'm Ludovic Delporte, I'm French and I'd like to contribute to this project. I'm System and Network engineer at Ugloo. I've knowledges in shell / Bash scripting, system engineering and backup problematics. Reach me at pro.delporte.ludovic@gmail.com