Group: SyncReplacement

From LibrePlanet
Revision as of 11:03, 7 July 2013 by El128 (talk | contribs)
Jump to: navigation, search

SyncReplacement is an FSF Priority Project. Bittorrent Sync is a peer-to-peer, two-way file synchronization utility with fine-grained access controls. We need a free software version of this client or free software that can be used for the same purpose.

If you are interested, please join this group (start by leaving your name here). We should start by surveying any existing free software that is in or close to this area (TahoeLAFS might be one), and making a list of features that are needed.

--I was actually able to write a unidirectional file synchronization script in Python 2.7.3. You can find it here. --Hunter D

--Hi. This is Rajul. I wish to work on this project. Please contact me at rajul.iitkgp@gmail.com

--Hello. I'm Hunter D and I think that I would like to contribute to this project. My language of choice is Python 2.7.3, and it just so happens that I started writing a p2p two-way file synchronization utility somewhat recently. If you want to contact me, you can email me at jhunter.dunefsky@gmail.com.

--Hello, I'm Pratik. I would like to contribute to this project. You can contact me at impratikb@gmail.com

--Hi, I' Isengaara - I installed GNUnet on my Trisquel System, GNUnet offers P2P File Sharing, but no sync yet. Maybe it could combined with git? Gittorrent seems dead. contact me at tobias [ät] platen-software [döt] de

--Hi, I'm Fred Morcos - I'm interested in such a project and would like to contribute. I am experienced in C and Python. One part of my proposal would be to not try and recreate the protocol but instead write a free compatible implementation of BTSync. You can contact me at fred dot morcos at gmail dot com. *Edit:* Since people have also suggested programming languages like Go (which I'm open to looking into for this project), I would also like to have the D programming language under the radar. About licensing, I am for GPLv3.

--Hi, I'm Patrick Steinhardt - I'd be interested in contributing to this project. My relevant skillsets include C, C++ and Python. You can contact me at steinhardt.ptk [at] gmail [dot] com.

--Hi, I'm Adrien. I'd be interested in this project as well. I actually implemented a prototype for a very similar system as a class project. I'll deliver the source soon. I think it's probably better to rewrite everything from scratch (in particular I'd like to use python instead of C++), but the code I have and the lessons I learned should be very useful. adrien [dot] projectemail [at] gmail [dot] com.

--Hi, I'm Gerry. This is a great initiative. I would strongly suggest considering Go (http://golang.org/) as a language for this. It's cross platform, has excellent concurrency support and is compiled, so users won't be required to install Python. It is far better than C/C++ in terms of providing a rich standard library and garbage collection. There are already existing BitTorrent clients written in Go, under free licenses.

--Hi, I'm Jumpwah. Here is a similar group most likely looking to collaborate on the same project: https://groups.google.com/forum/#!topic/golang-nuts/7WUj3nASuLo. In addition to deciding on the language, we should also decide on the license early to resolve any conflicts/doubts. Contact: jumpwah at google's electronic mail service.

--Hi, I'm Manuel. I've been spending some time lately trying to analyze btsync's network protocol (and working on a library that should speak that protocol). Currently it's just some Qt code hacked together with the aim to being able to communicate with btsync at all. The stuff I've found out so far can be found on github: https://github.com/picosync/workingDraft/ (have a look at the wiki and feel free to contribute). Contact: picosync (a) r7r .at

--Hi, i'm electron128 (a) yahoo (dot) com. I'm a big fan of Retroshare: http://retroshare.sourceforge.net/ My idea: create a folder sync plugin for Retroshare. I have experience with Java, Python and C. I learn C++ and Qt.

What i want:

  • simple, everyone should be able to use this tool
  • decentralised, no central server needed for operation
  • secure, communication should be encryted and authenticated
  • cross-platform: win, linux, mac

What Retroshare offers:

  • peer discovery with DHT
  • NAT traversal
  • encryption and authentication with PGP+OpenSSL
  • gui to manage friends+groups
  • additional functionality like chat, groupchat, voip... I think it is nice to have chat and shared folders at the same place. This would enable fast collaboration.
  • Retroshares core "libretroshare" is written in C++, and runs on every platform with sockets and a file system
  • Retroshare-gui uses Qt and runs on win, linux, mac

Plugins can use the secure link to send/receive data to/from peers.

Advantages of using Retroshare:

  • no worry about finding peers or securing the connection
  • Retroshare has a community of developers, testers, translaters and packers.

So we could concentrate on the main thing: how to get different states of a folder in sync.

The following cases should be handled by a sync algorithm:

  • deleted files
  • conflicting changes
  • wrong clocks
  • what about moved or renamed files?
  • resumable transfers and swarming? Retroshare has these features, maybe the code could be reused

My idea of a p2p sync algorithm:

for every shared folder we have:

  • name
  • some id
  • admin pgp id
  • list of users+permissions
  • could be signed

(set by admin, so update is simple with timestamp)

created by users with permissions:

  • ever growing list of file metadata items
    • every item is unique by nature, so sync is simple

file metadata item:

  • filepath/name
  • id of previous file, this id could be the hash of the other item
  • size, hash
  • timestamp
  • mark if file is deleted
  • author id
  • item could be signed

How sync works:

  1. file metadata items are exanged, so everyone has the same list
  2. fileitems with same path/name are ordered, so that we know the newest state of the file
  3. files are transfered

Conflicts need to be handled by the user

More thoughts:

  • when online, it could be signaled when files are in use
  • overlay icons like tortoise svn
  • differential sync
    • don't know which filetypes would take advantage of this
    • docx+odf is based on zip, zip compresses files independently
  • keep old files to some point in hidden directory for undo
  • sync of big list:
    • order in binary tree
    • compare checksums of treeparts
  • use gxs to sync items? (gxs is a new transport for the forums inside Retroshare)
  • use existing filetransfer classes?
  • how to tell if folder is in sync?
    • folder is in sync when we synced with >50% of peers with write access
    • or define master peer

Existing open source sync tools:

This is everything i know, thanks for reading!