I just published a blog post talking about interoperability and what it looks like in practice.
Any thoughts?
5 Likes
Very interesting article. A very hard problem to get true widespread interoperability on both data formats and protocols become a reality.
You mentioned āworking in the openā which is very important, as it allows oneās work to be discovered and more broadly shared. In that regard thereās much to be improved on the social aspects: the communities that are working on interoperability of all kinds. I notice thereās the tendency to stick within oneās own community, working on own specs, solutions and techstacks and very little time or even willingness to look beyond that see what others are doing and align efforts. Thereās very little cross-community interaction. To an extent maybe thereās even āadversarial myopiaā with āour solution is bestā or āour way or the highwayā kind of thinking (maybe unwittingly coming from a passion for the own project).
In protocol land it is often others that write protocol bridges after standards have been established. But even within a certain protocol thereās no agreement on semantics and data formats as you mention. On the Fediverse for instance - while ActivityStreams / ActivityPub have a JSON-LD vocabulary extension mechanism - thereās something like Ad-Hoc Interoperability going on: Look into the codebases of similar apps you like to integrate with and discover data formats that are used. This is probably how e.g. PeerTube and Owncast will someday interoperate.
Though it is a win-win for every FOSS developer involved they still donāt spend the time doing community work to standardize and document proper extension vocabularies. While schema.org is nice, the other big ontology library is not very useful in many cases. I feel like for Fediverse we should define Pattern Libraries of āmicro ontologiesā + additional documentation that each introduce a single concept that can be incorporated in an app as a building block. For instance I want to define a robust concept of āCommunityā that is very minimal. Another concept may be āPolicyā and by combining the two I might have community policies in my app.
I find those Lenses to be very interesting as well, and they could become part of the Pattern Libraries along with business logic / rules and other stuff needed to guarantee a level of interoperability.
Note that these concepts, such as Community, have broader applicability than just for Fediverse. Solid apps may be compatible, and also e.g. Valueflows (who have shown an interest to align).
For Fediverse I started a draft website in hopes to get an initiative started to give direction to all this. Problem, as always, is finding people willing to do community work, and look beyond their own project (a problem I also notice for the Solid community at large).
1 Like
Oh btw, the Murmurations Protocol (federated or as-is) might also be used to aggregate Pattern Libraries where the burden of maintenance is now distributed between those offering vocab extensions. For more on murmurations see also the Ideas for a possible FAQ Solid App - #2 by aschrijver topic.
i enjoyed the clarity of the text and i learned new stuff from it. thanks for sharing!
One example is the type index, which is a way to declare where you data is stored so that other apps can find it.
This reminds me of a recent conversation on this forum. It seems that type indexes arenāt really a standard. In fact there is a Solid Interoperability Panel that works on a new Solid interoperability specification. AFAIK that one doesnāt specify type indexes, but Shape Trees, something called Registrations and probably other stuff.
Do you know about this effort? What do you think about it?
iām curious, because iām somewhat skeptical to the new spec; but thatās mainly because
- i donāt understand it. it seems complex to me and i havenāt found an answer to the question Is this really necessary?
- of the text by Aaron Swartz in Programmable Web (also here):
And instead of spending time building things, theyāve convinced people interested in these ideas that the first thing we need to do is write standards. (To engineers, this is absurd from the start ā standards are things you write after youāve got something working, not before!)
Yes, Iām aware of that, but I chose not to mention it because it wasnāt the focus of the article (itās not only to Solid developers either). I almost added a footnote to clarify, but in the end I decided not to do it because it was too tangential to the main point and I thought itād just be confusing.
Now, as to what I think about it. I canāt say a lot because I havenāt used any of the new approach in practice. But I can say that the type index has been easy to implement and understand for the most part, and I worry that the new approach isnāt. In practice, most people is not even using the type index, so I worry that this becoming more complex will mean even less interoperability in practice :(.
Something that worries me in particular is that this new approach seems to require apps having a canonical url, which is a problem for apps that donāt have one (such as mobile apps, desktop apps, or apps served from a .zip or distributed filesystem).
But all of this is just my gut reaction, as I said I havenāt used it (or even read a lot about it), so I canāt say for sure. Maybe I change my mind when I use it. I also understand that we need more granularity, but if it comes at the cost of added complexity and means that nobody will use it in practice (like the type index is already not being used), Iām not sure if itās a decent trade-off. Maybe it would be nice if the type index, which is simple enough, is added to the specs together with more complex features.
I guess all this ācomplexityā wouldnāt be a problem if there were good tutorials and clear best practices, but at the moment I feel like making a Solid App is a bit of the wild west. A lot of apps are still using an authentication client that has been deprecated for a year (even new apps, not just unmaintained apps), and I think if we look at the landscape today what we have the most is what I call āobfuscated interoperabilityā.
I donāt want to sound negative, I know this ecosystem is still growing and eventually we should converge into best practices. But thatās how I feel about how things are at the moment.
4 Likes
@mrkvon I have had extensive discussions on this forum before, along similar nature of your two bullet points. I too find the specs to be seemingly overly complex, maybe too academicalā¦ idk. In the past Iāve made remarks on the positioning of the Solid Project as a whole, which was very confusing to me (this has since improved somewhat), on the need for simplifying organization structure and ceremony and focus on building a strong community.
At the outset of 2019 I felt that The biggest challenges for Solid are not technical in nature, nor are they in spec writing. And I still feel this is true.
But it may also be that Solid is following a wholly different approach to making both the standards and technology ecosystem a success. Thereās much focus on commercial viability, and the parts of the process where customer feedback is gathered and their needs are analysed (which is even more important imho than āspending time building thingsā, which comes afterwards) may not happen all in the open but at various boardroom discussions where representatives of Inrupt and other companies sit at the table to flesh things out with prospective clients. And codebases may sit in private repositories until the time they may or may not be open-sourced when these companies take a leading role and release what they hope will be killer apps.
This is all speculating, of course, but might be a valid strategy from the companiesā perspective. Having full control over the specification process (unburdened by W3C slow-going process, and big player influence (see DID-Core issues)) and technology development (early adopter access), not-yet-building strong open community in early spec stages (at the cost of losing active members)ā¦ all this enables core participants to be fast-movers, possibly gaining competitive advantage (competitive to competing specs? to fast market developments? Idk, again).
If such or a similar strategy is indeed followed, I still think it is very high-risk in terms of eventual success. Still convinced, as I said in 2019 that much hinges on: Changing perceptions of the technology and gaining [widespread] adoption.
Update:
Re: Comparing competing (?) standardsā¦
Though - just like for Solid - I only follow it at significant distance (and without any interest for the blockchain parts), the whole standards approach of the Decentralized Identity Foundation (DIF) has a certain appeal to me. From an early draft on Encrypted Data Vaults to a draft specification of Confidential Storage to early-bird implementations, I found all the work clearly positioned, and intuitively documented in easy-to-understand ways. Note, this spec is just one node in a large verifiable credentials specifications map. Note too, that hey still have a lot of work to do on the Interoperability side of things.
Overall the positioning, clarity of message, documentation and objectives, and community engagement are imho key success factors in any such efforts.
BTW, in the case of DIF the many links with Blockchain technology may be hampering adoption, while for Solid echoes and associations to the prior Semantic Web hype cycle may be inhibiting factor.
(@NoelDeMartin sorry, I realize that this side-tracked your topic with only slight reference to interoperability, while responding to @mrkvonās two points. If you want I can transfer to a new topic and cross-ref (then DM me, so as not to pollute the thread further 
))
1 Like
I have created an issue to address these data discovery concerns : Create best practices for profile predicates Ā· Issue #209 Ā· solid/data-interoperability-panel Ā· GitHub.