As I have built https://drive.owntech.de and many smaller apps playing around with Solid, I’ve found that for me one of the most difficult things was to figure out how and what fields/schema to use to become interoperable with other apps. I read the code from mashlib and podbrowser to figure out how to add contacts to a solid users profile or changing the name/email/job, but as the amount of apps and business domains of solid is expanding, I find that reading some other apps code/docs to become interoperable is a timely, unnecessarily complex, brittle and for non-open-source apps, dysfunctional approach.
As I understand, seamless interoperability and data portability is one of the key advantages/promises of Solid. However, I have the impression that the effort put into making it a reality is not proportional to it’s importance. I am aware of shaperepo.com, schema.org, lov and shex as attempts to distribute and use rdf schemas, but I think there is so much more potential for developing cool tooling and collaboration. I tried contacting some of the maintainers of these projects, but it seems to me that they are (justifiably) busy.
So I decided to get active on the matter myself by conceptualizing shex-codegen and ipfs-shex-server as new ways to distribute and use shex’. Now I am thinking about building a webapp to interact with these tools through the browser and hopefully enable ease-of-use, transparency and collaboration in regards to the shapes that are used in applications.
For this endeavor I am welcoming you to participate in designing, implementing and envisioning this platform and new ways to build interoperable applications. If this sounds interesting to you, you can contact me to collaborate, on this forum, gitter chat, via my solid inbox at https://ludwig.aws.owntech.de/inbox or my email ludwigschubi1@gmail.com.
Thank you for reading this far and thank you for working on Solid in whatever way, I am grateful to see all of the efforts going on and am very happy to be surrounded by so many people with good intentions. Also feel free to point me into directions that you think are important on this matter. Servus and Ciao!
Thanks for working on this, I agree with you that interoperability is very important :).
From the point of view of an app developer, I think something we’re missing in the tooling space is a way to apply reasoning, as I learned from @RubenVerborgh in his post: Shaping Linked Data apps.
It’d be awesome if apps are documented properly, and using the same shapes for the same concepts. But in practice I’m not sure how feasible that is. What I think is more feasible is declaring shape equivalences. Even though I have no idea how that works in practice, so I’d love to learn more. I may tackle it myself at some point, but right now I’m focusing my efforts in other areas.
I’m also convinced that automated conversion between shapes is the way to go, but it’s out of my comfort zone to explore further myself at the moment.
Both the app and the pod could provide shape equivalence data and ideally the conversion algorithm would be fairly generic and seems like the kind of thing that the server should provide because it potentially requires bringing together linked data across multiple documents and saving to multiple documents.
Registering shapes and building client side tooling to use it sounds useful in the meantime but feels like a stopgap to me.
Having different fields is one thing, but even representing data with same semantic meaning can be very hard. There’s this W3C internationalization best-practice on representing Personal names around the world (here discussed on HN). Things get very complex very quick.
Anyway, I came here just to mention a different take on the matter. A different approach to tackling similar problem area, by using Lenses:
I found this on fediverse here. Now, they’re not using Linked Data, but JSON Schema. But they have a large section on their finding with some interesting general observations wrt Interoperability.
@aschrijver
Couldn’t be json schema interoperable with linkeddata through jsonld?
What about a tool that analize pods resources as data fragments with machine learning and suggests pod owner some change or automatically convert data representating a Person ( or other recognized shape) to another shape?
A way could be with ml5 ( easy machine learning/tensorflow access) word2vec representating the content of a pod in the form of vectors
I am not at this moment deep-diving the subject, but have these resources listed for consideration.
We have the same problems to tackle for ActivityPub / Fediverse. In early days the decision was to KISS. Therefore there’s a very basic machine-readable ‘standardized’ NodeInfo endpoint to learn something about a remote instance. Then there’s a convention to write a FEDERATION.md in a project repo describing how interop works for the particular app.
This does not scale, and we are up to improving upon this mechanism. Grishka of Smithereen wants to build Capability Negotiation, but we need to standardize on that, and a full-blown solution would drag in all the complexity.
I am proposing - and what you seem to refer to as well - to use Conformance Profiles together with this negotiation. This is an agreed upon set of entities and properties, message formats, etc. that you guarantee your app complies with, and which can be exchanged in machine-readable way (as a string basically). What does ActivityStreams Conformance mean? - #4 by aschrijver - ActivityPub - SocialHub
Update: Note that both of these methods are concessions that help to avoid the complexity of tackling the full problem, and thus provide a practical way forward without having full-blown solutions in place.
Many years ago, I was doing some research related to interoperability in decentralized environments. Perhaps something can be tried in solid.
I used an artifact I called a ‘web blackboard,’ (honoring the old AI blackboards) a versioned RDF document describing a concept through several coexistent representations. It had an RDF adaptation of JSON-schema (for validation) and rules attached.
The idea is that the user collects concepts (commits to), which creates a relation between his personal-dataspace and a web of blackboards. Concepts are discovered by the users while interacting with other users.
At that time, I learned some stuff:
its better to see schemas as agreements between people.
people quickly agree on some things and never on others.
people describe things in many ways, at the same time.
people understand/accept flat structures quicker.
prefer lattices to hierarchies.
people want to describe things beyond reality.
I was researching this stuff because I wanted ‘meaning support system’ that constantly evolves the mappings in communities, does inference, etc., to pave the way to a (real) personal assistant.
What do you consider the relationship between ontology views and shapes?
What is the status of the web blackboard network idea? Have you/others been using it? It seems to be quite closely related to the idea of federated wikis? see Distributed Wiki's on Solid
That post is by @how who is also admin at SocialHub. He’s involved in DREAM research, I believe most recently in UPSYCLE.
I think these insights are one of the major factors that stood in the way of the Semantic Web success, as it is near impossible to describe semantics in a universal way, globally machine-readable. At TerminusDB they wrote a 5-part blog series addressing many issues and shortcomings. And in part 3 they talk about open and closed world reasoning.
On the whole set of standards surrounding RDF they say “This is where things get both really brilliant and ridiculously absurd”. SHACL is also mentioned: “the standard that came out was once again full of logical inconsistencies and impractical and wrong design decisions”.
Whatever you think of that, their decision was to focus on closed world vocabularies. And that has an appeal to me, making app design much more practical. And this fits really well with DDD strategic design where a bounded context maps onto a vocabulary. They can be agreed upon and standardized for specific application/business domains. In the case of Fediverse you can create ActivityStreams exensions also defined as separate vocabularies, specifying in this case federated msg formats, and these can be the basis for a more building-block-like approach to app development.
There’s much complexity to tackle, of course, but I hope that with Linked Data, AS/AP and Solid we can move from silo-first to task-oriented federated app design. I have my eye on Go-Fed where you start with a vocabulary defined in a subset of OWL, generate code from that, and then build your (modular) logic and UI on top of that.
That is just an old term I used in the slides. Probably now, I would call it ‘schema,’ to avoid confusion.
It was ignored; I believe the dominant idea was to only use ‘global semantics.’ I moved on to other things.
I think this is relevant stuff! Especially now the ‘tools for thoughts’ of the last two years. There is a lot of data produced by thousands of people that ‘weave’ their personal data-spaces (or text-spaces) using tools like obsidian.md, roam, foam, org-foam, to name a few. I see lots of enthusiastic people asking themselves in their respective forums: how we connect our thoughts? How we collaborate?
I’m also open to experimenting. I would start connecting existent vaults, always honoring their representation formats. Afterward, perhaps, add SOLID adapters;
…, the open-world assumption shoe does not always fit. We can use both assumptions and be explicit of which parts of the knowledge should be considered complete and which could be seen as incomplete. We lack of simple enough mechanisms to partition triples though.
This looks like a good pointer. Where can I learn more about this? Do you recommend a comprehensive source?
Funny enough, the name I always wanted to use was ‘meme’
You can compose a new concept, relating the mermaid and the tail concepts. But you pay in complexity.
Perhaps modern statistical models, like neural graph networks, can make these things easier for the user. A model trained with your data could tell if the tail is important or not for you after all,
How do you imagine this? in what sense would this be useful?
With Fediverse having grown in grassroots fashion from the 3 specs AS Core / AS Vocab and ActivityPub, I am afraid that good documentation are still lacking in depth and quality. Mastodon has been the early adopter and filling holes in the spec (with WebFinger, HTTP Signatures, NodeInfo). These things were discussed in the W3C SocialCG and SocialHub.
Onboarding for new devs is still quite hard, though there is some documentation, like Guide for new ActivityPub implementers. But if you want to see how extensions are applied in the wild, then you need to often dive into codebases. Like e.g. PeerTube defines video vocabulary extensions (e.g. a Video object) in their typescript code. Other extensions are more clearly specified, like ForgeFed protocol. Here are watchlists of apps and dev resources.
The closed world perspective simply means that your vocabulary only has a well-defined meaning for those who accept it. An agreed upon convention. You don’t worry about concerns that go beyond that, e.g. having a global standard or semantics that can be universally understood and translated to arbitrary other similar semantic models.
This may still go well beyond an (application) silo. Like e.g. for Fediverse I want agreement on what a Community entails. Create a vocabulary for that, which extends ActivityStreams. The latter offers as:Group, but there’s more to community than that. The community vocabulary (or bounded context in DDD terminology) thus created might be adopted fediverse-wide, but also elsewhere. Like e.g. for Valueflows. And in Solid apps too.
I guess the trick is not to specify too much, only the bare minimum. For instance I’ve found ‘community’ vocabularies that would only apply to a forum.
The idea of avoiding application silo’s and go towards task-oriented design is related, but different. Mostly it boils down to not consider interoperability as an afterthought, after the entire app has already been built. It is more oriented to the methodology applied to app design.
Natural language → RDF → Apps use the RDF (with queries)
People could use it to connect and expose any kind of text as RDF from a pod. Process the word documents of your folder, chat conversations, Facebook interactions. The contents of this forum. I would bet for adapters that do that (they call them ‘triplifiers’)
I can see there is a lot of discussion on the ‘schema’ aspect of interoperability, but it misses (o or I cannot find ) the ‘agreed semantics’ aspect.
To exemplify:
Suppose an app, for example, wants to use some info in the profile. In that case, it can execute queries, apply rules, follow links, consume APIs, transform data into any intermediary structure you want.
But the app will always need to be careful to interpret vocabulary one, vocabulary two, etc. Or, more precisely, the app needs to act in a way it’s expected (and accepted) by the owner of the Pod.
The pod owner will always have something to say. An app can (probably will) use statistical models to interpret or map the vocabs, but the Pod owner should agree with the result.
I miss a mechanism where the user can ‘commit’ to some vocabs.
I imagine that perhaps a Pod owner can use a community tool for interoperability’ for that; for example, in the user profile case, the user inspects examples of profiles to say afterward: yeah! That’s what I also understand as a profile. There are many techniques to do that.
I suppose currently the idea is that a user advertises the shapes they have registered and apps are encouraged to comply with them.
From that point of view, I suppose the next step is to give more visibility to the vocab choices that users and apps are actually selecting, and given that users are unlikely to be aware of the details this comes down to visibility of which shapes apps in the same domain are choosing to support.
While this partly returns to earlier points in this conversation, perhaps a simple best practice should be that apps request user consent before registering a new shape, and that the shape is described in both human and machine readable format, perhaps offering additional explanations with some disclosure about alternatives and potential tradeoffs. This might also encourage apps to support multiple competing shapes.
Asking consent for data shapes seems to go against the trend towards reducing friction in user interaction, but on the other hand might be necessary for the user to gain any meaningful control over their pod and the data within it?
Yes, it would help to know what people are using to share their data. That’s where a networked Web of shapes (or blackboards, whatever) would help. If two organizations discover what shapes are being used for the same thing; they can plan to merge them.
I think that ideally, a person should understand what is in their Pod. A person can, for example, have a map of contents (MOC) of what’s in the Pod, things like having a 'simple profile shape ’ and a ‘detailed work profile shape’ subsumed in a big circle ‘me seen by the others’ (name chosen by the person, with relevant annotations and stats ) …
The user maintains their data, curating, cleaning, augmenting, using different (interchangeable) apps. This is called ‘digital gardening’ by the current Personal Wikis applications.
… I would add to show in the screen the actual data to be shared (show the query results the app wants to consume).
Yeah!, with the right amount of detail. To be able to use the less complex and versatile one that does the job. Or move into complex ones if necessary.
… I think that ‘lack of trust’ could be a major factor of friction, and trust comes with meaningful control and understanding of what’s going on.
