I already mentioned why I think Live Mesh is cool and that I think the most important part of it, FeedSync, is being largely ignored by reviewers. Fortunately, there's an extensive interview with the team that goes quite deep in FeedSync and how it works. Go watch it, it's good info.
At the most basic level, FeedSync is a mechanism to associate versioning "headers" to arbitrary objects (items), and an algorithm to merge and detect conflicts based on that header information. Replace "header" with "extension element" and "arbitrary object" with "RSS/Atom Item" and you have the XML feed version of it:
<
item
xmlns:sx
='http://feedsync.org/2007/feedsync'>
<title>Buy groceries</title>
<sx:sync id='0a7903db47fb0fff' updates='1'>
<sx:history sequence='1' by='kzu'/>
</sx:sync>
</item>
The sx:sync element is the versioning header. Every time the item is updated (say, by another user/device), a new sx:history element is added and the updates attribute is incremented:
<
item
>
<
title
>Buy groceries</title>
<sx:sync id='0a7903db47fb0fff' updates='2'>
<sx:history sequence='2' by='vga'/>
<sx:history sequence='1' by='kzu'/>
</sx:sync>
</item>
If v1 of the item was updated simultaneously by two users and you try to merge them, you'd end up with something like the following:
<
item
>
<
title
>Buy groceries</title>
<sx:sync id='0a7903db47fb0fff' updates='2'>
<sx:history sequence='2' by='kzu'/>
<sx:history sequence='1' by='kzu'/>
<sx:conflicts>
<item>
<title>Buy icecream</title>
<customer id='1' />
<sx:sync id='0a7903db47fb0fff' updates='2'>
<sx:history sequence='2' by='vga'/>
<sx:history sequence='1' by='kzu'/>
</sx:sync>
</item>
</sx:conflicts>
</sx:sync>
</item>
Notice the added sx:conflicts element, which contains the conflicting item in its entirety. The algorithm provides a consistent selection of a default "winner" from the merge (in this case my v2 item, rather than Victor's). This gives you a consistent state across the mesh. But note that the merge operation with conflict does NOT resolve the conflict. It just surfaces it in a predictable way.
Needless to say, the default winner might not be what the end user wants, so at that point the application can surface the conflict (i.e. a different icon for the item) and allow him to resolve it in a different way (i.e. doing a full content diff and picking the new state). Alternatively, a specific implementation of an item store/adapter (i.e. the feedsync-enabled store for files) can provide automatic resolution if it so wishes to, such as doing an auto-merge of the file contents.
Pablo has more info on how it works and the way the Microsoft Sync Framework exposes this.
It seems really simple at first glance, doesn't it? The devil is in the details, as usual. It's obviously a RESTful approach to synchronization, where the item payload is the actual representation of the state. This means that upon conflict, all you have is the state. Implementing automatic conflict resolution can be quite tricky as a consequence.
Automatic Conflict Resolution?
As part of our Mesh4x implementation with InSTEDD, we're thinking about ways to increase the chances of automatic conflict resolution, as it's generally desirable and you don't want to be bothering your users for every apparently conflicting change. Think about a very simple example: tags. Say both user A and B add a tag at the same time, and then they synchronize:
Initial state of both A and B (say, created by yet another user, C):
<
item
>
<
title
>Buy groceries</title>
<sx:sync id='0a7903db47fb0fff' updates='1'>
<sx:history sequence='1' by='C'/>
</sx:sync>
</item>
User A changes item by adding a category "to-do" (again, notice the new sx:history):
<
item
>
<
title
>Buy groceries</title>
<category>to-do</category>
<sx:sync id='0a7903db47fb0fff' updates='2'>
<sx:history sequence='2' by='A'/>
<sx:history sequence='1' by='C'/>
</sx:sync>
</item>
Whereas user B changes it by adding a different category "supermarket":
<
item
>
<
title
>Buy groceries</title>
<category>supermarket</category>
<sx:sync id='0a7903db47fb0fff' updates='2'>
<sx:history sequence='2' by='B'/>
<sx:history sequence='1' by='C'/>
</sx:sync>
</item>
When they sync, the algorithm will detect a conflict, as we have the same sequence number but with different "by". It will pick a default winner (say, A's change). But clearly this is a case where we could apply auto-merge semantics if we knew the operation each user performed.
You might be tempted to say: just do a content merge and that's it! But the semantics of a merge of state cannot be applied generically (i.e. a Company element gets an IsBankrupt='true' value from one user, and a new Invoice element from another: at the business logic level, the two changes cannot be merged. If the company went bankrup, you can no longer receive invoices from it), and it's even quite tricky with XML elements that can be moved around, renamed, deleted and re-inserted across multiple versions, etc.
So one approach we're thinking is to add command-pattern like hints that an adapter/store can use as hints for automatic conflict resolution and state reconstruction for merging. In the case above, maybe the XML adapter expresses user A operation as a combination of the item payload, the sync versioning header, and diff information about what was changed (added an element inmediately after the first child element of the root, in terms of XPath):
<
item
>
<
title
>Buy groceries</title>
<category>to-do</category>
<sx:sync id='0a7903db47fb0fff' updates='2'>
<sx:history sequence='2' by='A'/>
<sx:history sequence='1' by='C'/>
</sx:sync>
<diff xmlns=".../xmldiff">
<add match="/*[1]">
<category>to-do</category>
</add>
</diff>
</item>
The new diff element would be very similar in spirit to the XML Diff and Patch tool from Microsoft and would make it easier for the XML adapter to determine if a given change is compatible for auto-merge (i.e. it's not a change in a node that was removed, etc.).
I think different adapters might benefit from surfacing different conflict resolution hints. A database may include schema manipulation statements to perform upgrades, for example.
This is exploratory field as we try to make for the best user experience possible. We don't want to end up in the Groove approach where your choices are: save your changes as a different item, or discard your changes :S.
What do you think?
You have probably read or listened all the (maybe a bit vague) information about the Mesh Operating Environment (MOE): a platform that will allow multiple applications and devices to participate in the Live Mesh.
Let's get to the more technical details now, and leave the end-user marketing to someone else ;).
At the core of Live Mesh is FeedSync, a public spec by Microsoft, evolved from the "old" days of SSE. It didn't get its deserved attention IMO back in the day, even though you could put together the pieces from Ray's announcement and later adoption by other products and figure out Live Mesh was coming sooner or later. Ed Jezierski got me interested in it WAY back, and as a result, I created the first open source implementation of the spec.
To understand its importance you need to first understand what FeedSync is. The most succinct definition I could come up with is:
FeedSync is a definition of a versioning header (or metadata) and its associated creation, update and deletion behavior as well as a generic conflict detection algorithm based on it, which can be attached to an arbitrary piece of data.
The interesting part of it, as you are probably realizing already, is NOT the XML microformat itself, or its representation in an RSS/Atom feed, but rather the behavior and semantics associated with it, which you can rely on to be present at compliant endpoints.
The specification defines the behavior for merging (arguably the most important one) data between two "feeds" (think of these as "streams of data with associated versioning header/metadata") in a consistent manner. It's two-way sync for the masses.
Of course, specs without implementations are seldom of any use. Microsoft is providing their own with the Microsoft Synchronization Framework (which does other things in addition to supporting FeedSync), and now Live Mesh. But nothing forbids the community from building compliant cross-platform open source alternative implementations.
Being an open source fan myself, and being lucky enough to work for an organization whose goal is to produce open source software to help in the humanitarian space, I'm glad to announce that we have already been working actively in this space, and we just released early versions of our ongoing projects at Google Code.
Mesh for X
Or Mesh4x for "short", is an umbrella project by InSTEDD under which we're producing Mesh4j and Mesh4n (Java and .NET respectively) versions of a unified implementation and library design/API for FeedSync synchronization between arbitrary repositories of data (i.e. databases, files, spreadsheets, etc.). We may very well come up with Mesh4py (Phyton), Mesh4r (Ruby) or whatever we or the community need to satisfy data synchronization scenarios.
The Mesh4n version is coming straight from its old place in CodePlex, but you expect it to evolve together with the Mesh4j version. Our goal is to keep feature-parity on both. This version of FeedSync for .NET is being used by Microsoft Humanitarian Systems (MHS) in Afganistan to synchronize disparate data sources in extreme and low connectivity conditions as mentioned by Ted Okada (i.e. to achieve two-way synchronization of disconnected Access databases, Excel files and plain RSS files on a pendrive, between any of them!).
At this time, both libraries provide the basic synchronization algorithm and Sync (metadata) data model, as well as the core interfaces to create your own repository adapters to expose data from arbitrary sources for FeedSync synchronization. We'll be building concrete repository adapters during the next few weeks. The most interesting one for me is the Mobile repository, which should allow you to sync two repositories over cellphones, without cellphone internet connection or even a data plan (which needless to say is not very frequent in the under-developed or even developing world).
We'll also be looking for refactoring opportunities to simplify certain scenarios and accomodate the future roadmap for FeedSync.
Some people have argued this is just another consumer-oriented Microsoft-thinghy that will hardly make a difference in anything. I believe that's a mistake, as it's ignoring the underpinning technology, the fact that it's a public specification, and that it can be applied to really free the data once and for all, regardless of platform, application, format, language, etc. This is the very reason I'm not excited at all by all the Atom Publishing Protocolfuss that's happeningthesedays. Web 2.0/AJAX + APP is SO client-server... but that's for another post, another day.
Stay tunned as I'll be posting more detailed (and practical) information about Mesh4x in the coming weeks, as well as the cool stuff we're doing with InSTEDD as part of its goal of helping in the humanitarian space by applying technology.
For the past couple weeks, we've been very busy putting together a prototype that was running at an InSTEDD booth in the prestigious TED conference. Of anecdotal interest is the fact that it's the first time in TED's history that such a demo is run remotely from South America (Buenos Aires, Argentina, in this case :)).
The purpose was to show how deep-field data collection can be performed using ubiquitous and generally cheap (in some places even free!) text messages from cell phones (a.k.a. SMS), and how that data can be surfaced quite easily in geographically meaningful representations (i.e. Google Earth, Google Maps and Microsoft Virtual Earth) to support decision making in scenarios such as pandemic/outbreak detection and disaster/emergency responses. Also very important was to show that such a mechanism by no means has to be one-way.
You can go to the live demo site right now if you're eager to see the real thing :). *
Let's first see what's in there from the functionality point of view. Later on, I'll talk about the various technologies we used to make this possible (all of it available on the web for free).
The Prototype
- See a tag cloud generated from tags detected in the incoming messages:
- See the most recently received messages, either from direct SMS (*) to our server or Twitter:
- Visualize all messages, or those matching a given tag, on Google Earth, Google Maps and Microsoft Virtual Earth:
- Show relative "weight" according to message count by region (country or province), for the selected tag (if any):
- Enable responses to be sent to a given individual right from the geographical visualization:
(also works from Google Maps and Virtual Earth)
- See timeline with the path traveled by every individual (timestamped so you can "replay" it in Google Earth).
In a screenshot, this is just the lines connecting the bubbles as you see above. A video of Google Earth "replaying" the sequence would be more appropriate here, as it would show the messages appearing together with the line departing from the previous message from the same individual.
- Enable group replies by country and/or tag:
The demo is quite compelling I think, and the messages being exchanged simulate an H5N1 (a.k.a. bird influenza) outbreak in Southeast Asia.
The Technology
For the demo, we used C#, plus the following tools:
- SMS gateway: we connected a regular Windows Mobile phone to the server, and used the MSR IndiaSMS Toolkit to receive incoming messages sent to it.
- Twitter bot: we used the twitter bot framework we had already created for another project, which is open source.
-
KML creation: we used LINQ to XSD, which conveniently enough was released (*preview alpha 0.2*, can't get any more scary than that, eh?) about a week ago!
-
GeoRSS: for this we created our own data model. The original code was created for Golden Shadow last year, so we didn't have WCF 3.5 SyndicationFeed yet.
- VS Web Developer "server": because of some COM-related issues, we couldn't use the SMS Toolkit from IIS, so we had to do with VS built-in ASP.NET server. In order for it to "accept" requests originating outside the localhost, we used:
-
PortTunnel: this nice app allowed us to tunnel external incoming on port 80 requests as if they were originated in the localhost to port 81 :). Details on that for another post..
All in all, we had great fun developing this, and I'm convinced this mashup of existing technologies and tools can be applied to humanitarian scenarios with great success. It's been a thrilling experience putting this together for TED.
Will follow-up with details about the technologies and open source projects we're contributing to the community :).
/kzu
* sending direct SMS messages is no longer working as the TED demo finished. You can still tweet ted08 and have the messages appear.
Also, we're no longer on-site monitoring the demo server/wesite, so it may be down sporadically (see all the alpha/ctp/preview stuff we're using? ;)), so please be patient.
