Application of CRDTs to Solid

I don’t recall CRDTs (Conflict-free Replicated Data Types) being mentioned here before, and am interested to hear if people think they could be useful for Solid.

I wrote the following in response to @NoelDeMartin’s post raising an issue with performance when an LDP container holds hundreds of documents. But I think CRDTs merit a topic of their own.

An alternative which I think would be interesting to explore for Solid is local-first using CRDTs. It’s a bit like offline-first, but peer to peer rather than syncing with a central server, so each participant has a local copy of a CRDT type document which they can edit, and changes are automatically synced by exchanging edits between peers. By implementing documents as CRDTs the merging changes is completely automatic and guaranteed to produce the same result in every copy.

This tech was probably a bit late to be considered originally for Solid but is now coming of age and could be another alternative to the downsides of a server based model.

So it’s not a quick fix - more an opportunity to rethink - and probably not immediately helpful, sorry! I’ve been looking into it recently, so I could start a topic of anyone is interested, or provide links etc.

Maidsafe have begun to switch over to using CRDTs within SAFE network for individual data types, and I’m looking at using it for a self replicating filesystem that could be mounted using FUSE. These are not local-first applications at this time, but a way to ensure the network data types can be replicated for redundancy, updated arbitrarily, and remain conflict free. I am still thinking about local first though, because it’s a promising way to implement collaborative applications in its own right, and could well solve the kind of problems we are seeing with Solid because of its server based model. Lots of issues just disappear when you eliminate servers.

For anyone who wants to follow up on CRDTs, here are a few links.

  • CRDTs: The Hard Parts — Martin Kleppmann’s talks (link). See this post for notes about this talk with time stamps.
  • A highly-available move operation for replicated trees and distributed filesystems, Kleppmann et al, 2020 (move-op.pdf)
  • Local-first software: You own your data, in spite of the cloud (link)
  • Rust implementation of automerge (automerge/automerge-rs)

I particularly recommend Martin Kleppmann’s video presentation (I made notes from it which give times for different aspects, key slides and so on here).

1 Like

About a week ago there was a big HN thread about CRDTs The Hard Part:

(And there are a number of other interesting HN discussions on the topic.)

Through that discussion I found the DAT-based p2p project Hypermerge and the PushPin sample app built on top of it (both from automerge too). Very cool.

1 Like

I haven’t looked into it too deeply, but as far as I can see it’s pretty much unavoidable if you want to support offline-first and/or collaborative editing with Solid. (Don’t know why you’d need it to be peer-to-peer though, and it wouldn’t really be Solid any more if you did. Which is fine, of course :slight_smile: But Solid should be able to help keep resources available even if peers are offline.)

But yeah, I don’t see it being a quick fix either. I’d expect the first avenues to look into would be to start a vocabulary to represent the addition and removal of triples, and then to write some code to convert those into JS data structures that e.g. automerge can work with - that should get you a long way. Yet another item that’s somewhere on my personal todo list :slight_smile:

1 Like

The idea is very relevant (and we have talked about it on a couple of occasions). Note that we can do even more advanced things, given that the semantics of the data are encoded as well.

For example, here is work on CRDT and RDF:

1 Like

Thanks Ruben. I’m interested to hear views or ideas for how this can/can’t be used with Solid or to move it forward in some way. I have my own thoughts, but want to get wider inputs.

N.B. I found a link to a PDF of the paper you linked:

It sounds me possible using @jeffz solid-rest to store data locally on the device before syncing with automerge to the Pod.

I will try what is possible with Vuex store system on

1 Like