Interoperable Serendipity

I just published a blog post talking about interoperability and what it looks like in practice.

Any thoughts?

5 Likes

Very interesting article. A very hard problem to get true widespread interoperability on both data formats and protocols become a reality.

You mentioned “working in the open” which is very important, as it allows one’s work to be discovered and more broadly shared. In that regard there’s much to be improved on the social aspects: the communities that are working on interoperability of all kinds. I notice there’s the tendency to stick within one’s own community, working on own specs, solutions and techstacks and very little time or even willingness to look beyond that see what others are doing and align efforts. There’s very little cross-community interaction. To an extent maybe there’s even ‘adversarial myopia’ with “our solution is best” or “our way or the highway” kind of thinking (maybe unwittingly coming from a passion for the own project).

In protocol land it is often others that write protocol bridges after standards have been established. But even within a certain protocol there’s no agreement on semantics and data formats as you mention. On the Fediverse for instance - while ActivityStreams / ActivityPub have a JSON-LD vocabulary extension mechanism - there’s something like Ad-Hoc Interoperability going on: Look into the codebases of similar apps you like to integrate with and discover data formats that are used. This is probably how e.g. PeerTube and Owncast will someday interoperate.

Though it is a win-win for every FOSS developer involved they still don’t spend the time doing community work to standardize and document proper extension vocabularies. While schema.org is nice, the other big ontology library is not very useful in many cases. I feel like for Fediverse we should define Pattern Libraries of ‘micro ontologies’ + additional documentation that each introduce a single concept that can be incorporated in an app as a building block. For instance I want to define a robust concept of “Community” that is very minimal. Another concept may be “Policy” and by combining the two I might have community policies in my app.

I find those Lenses to be very interesting as well, and they could become part of the Pattern Libraries along with business logic / rules and other stuff needed to guarantee a level of interoperability.

Note that these concepts, such as Community, have broader applicability than just for Fediverse. Solid apps may be compatible, and also e.g. Valueflows (who have shown an interest to align).

For Fediverse I started a draft website in hopes to get an initiative started to give direction to all this. Problem, as always, is finding people willing to do community work, and look beyond their own project (a problem I also notice for the Solid community at large).

1 Like

Oh btw, the Murmurations Protocol (federated or as-is) might also be used to aggregate Pattern Libraries where the burden of maintenance is now distributed between those offering vocab extensions. For more on murmurations see also the Ideas for a possible FAQ Solid App - #2 by aschrijver topic.

i enjoyed the clarity of the text and i learned new stuff from it. thanks for sharing!


One example is the type index, which is a way to declare where you data is stored so that other apps can find it.

This reminds me of a recent conversation on this forum. It seems that type indexes aren’t really a standard. In fact there is a Solid Interoperability Panel that works on a new Solid interoperability specification. AFAIK that one doesn’t specify type indexes, but Shape Trees, something called Registrations and probably other stuff.

Do you know about this effort? What do you think about it?


i’m curious, because i’m somewhat skeptical to the new spec; but that’s mainly because

  1. i don’t understand it. it seems complex to me and i haven’t found an answer to the question Is this really necessary?
  2. of the text by Aaron Swartz in Programmable Web (also here):

    And instead of spending time building things, they’ve convinced people interested in these ideas that the first thing we need to do is write standards. (To engineers, this is absurd from the start – standards are things you write after you’ve got something working, not before!)

Yes, I’m aware of that, but I chose not to mention it because it wasn’t the focus of the article (it’s not only to Solid developers either). I almost added a footnote to clarify, but in the end I decided not to do it because it was too tangential to the main point and I thought it’d just be confusing.

Now, as to what I think about it. I can’t say a lot because I haven’t used any of the new approach in practice. But I can say that the type index has been easy to implement and understand for the most part, and I worry that the new approach isn’t. In practice, most people is not even using the type index, so I worry that this becoming more complex will mean even less interoperability in practice :(.

Something that worries me in particular is that this new approach seems to require apps having a canonical url, which is a problem for apps that don’t have one (such as mobile apps, desktop apps, or apps served from a .zip or distributed filesystem).

But all of this is just my gut reaction, as I said I haven’t used it (or even read a lot about it), so I can’t say for sure. Maybe I change my mind when I use it. I also understand that we need more granularity, but if it comes at the cost of added complexity and means that nobody will use it in practice (like the type index is already not being used), I’m not sure if it’s a decent trade-off. Maybe it would be nice if the type index, which is simple enough, is added to the specs together with more complex features.

I guess all this “complexity” wouldn’t be a problem if there were good tutorials and clear best practices, but at the moment I feel like making a Solid App is a bit of the wild west. A lot of apps are still using an authentication client that has been deprecated for a year (even new apps, not just unmaintained apps), and I think if we look at the landscape today what we have the most is what I call “obfuscated interoperability”.

I don’t want to sound negative, I know this ecosystem is still growing and eventually we should converge into best practices. But that’s how I feel about how things are at the moment.

4 Likes

@mrkvon I have had extensive discussions on this forum before, along similar nature of your two bullet points. I too find the specs to be seemingly overly complex, maybe too academical… idk. In the past I’ve made remarks on the positioning of the Solid Project as a whole, which was very confusing to me (this has since improved somewhat), on the need for simplifying organization structure and ceremony and focus on building a strong community.

At the outset of 2019 I felt that The biggest challenges for Solid are not technical in nature, nor are they in spec writing. And I still feel this is true.

But it may also be that Solid is following a wholly different approach to making both the standards and technology ecosystem a success. There’s much focus on commercial viability, and the parts of the process where customer feedback is gathered and their needs are analysed (which is even more important imho than “spending time building things”, which comes afterwards) may not happen all in the open but at various boardroom discussions where representatives of Inrupt and other companies sit at the table to flesh things out with prospective clients. And codebases may sit in private repositories until the time they may or may not be open-sourced when these companies take a leading role and release what they hope will be killer apps.

This is all speculating, of course, but might be a valid strategy from the companies’ perspective. Having full control over the specification process (unburdened by W3C slow-going process, and big player influence (see DID-Core issues)) and technology development (early adopter access), not-yet-building strong open community in early spec stages (at the cost of losing active members)… all this enables core participants to be fast-movers, possibly gaining competitive advantage (competitive to competing specs? to fast market developments? Idk, again).

If such or a similar strategy is indeed followed, I still think it is very high-risk in terms of eventual success. Still convinced, as I said in 2019 that much hinges on: Changing perceptions of the technology and gaining [widespread] adoption.


Update:

Re: Comparing competing (?) standards…

Though - just like for Solid - I only follow it at significant distance (and without any interest for the blockchain parts), the whole standards approach of the Decentralized Identity Foundation (DIF) has a certain appeal to me. From an early draft on Encrypted Data Vaults to a draft specification of Confidential Storage to early-bird implementations, I found all the work clearly positioned, and intuitively documented in easy-to-understand ways. Note, this spec is just one node in a large verifiable credentials specifications map. Note too, that hey still have a lot of work to do on the Interoperability side of things.

Overall the positioning, clarity of message, documentation and objectives, and community engagement are imho key success factors in any such efforts.

BTW, in the case of DIF the many links with Blockchain technology may be hampering adoption, while for Solid echoes and associations to the prior Semantic Web hype cycle may be inhibiting factor.

(@NoelDeMartin sorry, I realize that this side-tracked your topic with only slight reference to interoperability, while responding to @mrkvon’s two points. If you want I can transfer to a new topic and cross-ref (then DM me, so as not to pollute the thread further :slight_smile: ))

1 Like

I have created an issue to address these data discovery concerns : Create best practices for profile predicates · Issue #209 · solid/data-interoperability-panel · GitHub.