A project for live data sharing

Hi everyone, I’d like to share my project m-ld with you, because it seems a natural fit with Solid and I hope it might be useful.

https://m-ld.org/

At heart it’s a technology for sharing a live dataset between collaborating actors, like users or robots. It uses an RDF data representation, with JSON-LD for the interface. It’s decentralised, but intended to fit well with existing data architectures including both client-server and ‘local-first’.

In a Solid app, m-ld could be used to, for example, share draft structured documents with collaborative editing (sometimes called ‘multiplayer’). The document could be synced to a Pod live, or perhaps when it’s ready to be published.

I can see that this general idea has been discussed here before and I’ve added some thoughts to that topic.

So far we’ve got to a preview release with a Javascript engine that works in the browser or a Javascript back-end. We’ve got lots still to do, but I’d love to hear your thoughts. Are there Solid projects that need live data sharing (or want it in future)?

3 Likes

Hi George, I am interested in your thoughts. I developed a medical questionnaire (https://demo.mp50.sosaro.com/en/questionnaire/1/), that results in a PDF with all data presented in the way doctors learn in Med-school. That PDF is a timestamped document of 1-2 pages with the full medical history. I look for a way to help patients better connect to their doctors, improving the patient-doctor dyad that way. This could be a perfect use case for your m-ld project. What do you think? More info: https://sosaro.com/assets/press-release/press_en_final_07sep2020.pdf Hope to hear from you. Hans

1 Like

Hi Hans, thanks. Your medical questionnaire app looks interesting. If I understand correctly, the patient is in charge of their medical history as a single ‘file’, which is versioned, and can be output as a PDF. I put file in quotes because it is not necessarily in a conventional file system, but on the cloud. Where is it kept by the app at the moment?

The ‘data sharing’ part of the app is about sharing between patient and doctor. When shared, during a medical appointment or for a time afterward, the doctor should be able to edit the history. It seems that there should also be limits on what the doctor can see (“the patient chooses which parts he shares with the medical professional”), and perhaps also what the doctor can edit.

This certainly sounds like an ideal use-case for Solid, because it is important that the ownership of patient’s data is clear and controlled, ideally independently of the app provider and of the storage provider.

There are two main ways that m-ld could be useful:

  1. If the doctor and the patient will edit the structured data at the same time. For example, they may be filling in details on separate devices. I’m not sure this is required, because the doctor will probably add new details as a new entry to the history, and if they wish to edit some previous details, then it is probably not at the same time as the user. It would be okay to allow the doctor to request temporary exclusive access to such a previous record while they edit it.
  2. If the history must never be saved anywhere but the patient’s personal devices (a mobile device and a browser, perhaps). Using m-ld would allow this, and keep the devices synchronised with each other. Is this a requirement? I suspect most patients would not mind if the data is stored on the cloud, so long as it is private? Also, since they use only one device at a time, they do not mind if synchronisation takes even a few seconds, going to the data store and back when changes are detected.

I have a few more thoughts but maybe I am assuming too much about the requirements for the app, so I will slow down!

@gsvarovsky could m-ld be used to synch this kind of graph ?https://scenaristeur.github.io/solid-vue-panes/semapps (click on ‘Assemblee Virtuelle’, then on a node like pair:Project)

Yep!

Since the data is JSON-LD it is easy to load into m-ld. I have made a small example Node.js script here:

(I downloaded the JSON-LD context file and used it as the context in the configuration, and then removed the context reference from the JSON-LD data. This is just because I haven’t tested m-ld with context references.)

You can do the same in the browser.

Once the data is in m-ld, it can be edited by multiple processes at the same time.

We are just about to launch a Playground web app for m-ld which will make it easier to try out data like this, and see the synchronisation. I will test with your data and post a link when it’s ready.

Can you tell me, what do you think the users would like to do, that requires the graph sync?

1 Like

In fact, I will add creation of nodes and edges like I did on spoggy-simple

https://scenaristeur.github.io/spoggy-simple/

https://youtu.be/EE-0eoJ3dBo (how to)

And would like to allow sync graph editing.
I’m most interested in browser side do you have browser libs ?

You can bundle the npm module with Browserify. Webpack will probably work too. We haven’t provided a built bundle yet, but I can put one together if using a bundler is not right for you.

m-ld now has a Playground for trying out transactions and queries against a browser-local clone:

https://m-ld.org/playground/

If you’re familiar with JSON-LD and SPARQL it should be mostly unsurprising. The fun starts when you open two browser tabs and connect to the same domain with them.

There’s still lots of work to do (in particular, there’s a microtasks library bug that makes query results sloooow to appear – it’s not fundamental, I promise) so I’d love feedback on the most important things to get on with!

1 Like

For some light entertainment we’ve put together an infographic about CRDTs:
collaboratenable your app with CRDTs

I think having a live collaboration is going to be table stakes for all apps with write/edit features before long… Of course I do! but seriously, does this spark an idea for a Solid use-case we should be looking at?

4 Likes

It looks like Meld uses Mqtt, an Oasis standard. There has been some controversy about Oasis policy regarding patents, see https://en.wikipedia.org/wiki/OASIS_(organization)#Patent_disclosure_controversy

from https://js.m-ld.org/ :

A m-ld clone uses a ‘remotes’ object to communicate with other clones.

and

Ably provides infrastructure and APIs to power realtime experiences at scale. It is a managed service, and includes pay-as-you-go developer pricing

Indeed, as the first implemented option. Thanks for the link, interesting. MQTT and Ably are the network providers we’ve started with – easy to try out and reliable. Ultimately though, m-ld is agnostic; all it needs is a way to get messages from clone to clone. I’d be interested to know what you have in mind?

I was thinking something free and open source based on SAFE or IPFS, if that makes sense.

OK, sure. Thanks for the feedback. Given the ick factor with MQTT, we could switch to socket.io as the baseline. SAFE or IPFS definitely do-able; also Matrix (I think?). I’d like to have a true peer-to-peer option on the table.

1 Like

Being able to collaborate in real time would be awesome.

It seems like it would go together with mobiles, phones. People collaborating while moving around. Something also related to this is geolocation, gps. And of course Solid is all about permissions.

So I’m thinking it might be useful and maybe there is a way to ensure that people are in or near a certain place before they are allowed to collaborate on something. Like you can comment and contribute but only if you’re actually physically in or near a certain location.

-edit-
fwiw :smile:

iw wiw :+1:

Rolling with it… you’d need some a secure geolocation service to prevent trivial spoofing, or maybe a kind of multi-factor “he claims to be here but I see no Bluetooth from him”. Beyond that, seems straightforward. What’s the end-user superpower though…? I’m thinking medical emergency response maybe. Real-time logistics in a dynamic environment. “Need saline at this location.” Perhaps less dramatically, “I need toilet paper but Tesco is out.” “Plenty at Asda.” For consumers, gamify, to make it compelling to contribute.

1 Like

News, real estate, construction, engineering, maintenance, safety, regulatory compliance, travel, consumer, health care

Probably I missed some

1 Like

Hi listeners, a quick update from me on m-ld!

Here are the headlines; visit the website for more details…

Of course I’d also love to make progress on Live shared Pod Data – making it happen… Ideas and feedback always welcome!

4 Likes

The next version of the m-ld Javascript Engine has been published (on npm)! We’ve been working hard on usability and performance, and we’ve also included the latest prototype work from the Security Project.

We’re moving fast towards our vision of data-wrangling-free information sharing. The security work is key, not only because it boosts, um, security :stuck_out_tongue_winking_eye:, but also because it signposts the way to decentralised extensibility.

Of course, we share this goal with Solid: we want anyone to be able to create new information sharing behaviours and rules, which work well together – security rules are just one example. One superpower this gives is to be able to dynamically adapt the presentation of information as its rules evolve. But more than that, it also gives users more freedom to change which app they use, while keeping their data intact.

To us, these behaviours are also shared information. So when you use them in a m-ld domain, they will appear in the domain data itself… which means they’re also shared, live!

We’ll be writing a lot more on this topic soon. But do let me know if it piques your interest :thinking:

3 Likes