Data Discovery on Community Solid Server

[mrkvon]

The profile/card document created by Community Solid Server (e.g. https://solidweb.me/mrkvon/profile/card#me) has many “default” fields missing, as compared to Node Solid Server (inbox, storage, publicTypeIndex, privateTypeIndex, ?). Are we expected to create these ourselves when we need them (perhaps from within an app)? Or they’re just not part of a spec and we shouldn’t rely on them when building an app?

With an app I build, I’ve relied on data discovery proposal to find relevant data of a currently signed in user.

Well, from the profile/card, i can’t discover

• relevant data directly by looking into publicTypeIndex, privateTypeIndex
• storage Container(s), which I could use to search through the whole Container hierarchy (edit: pod root is one of the pim:Storages, actually, and…)
• “pod root”. Going backwards through parents folders of https://solidweb.me/username/profile/card I can’t tell directly which one is the pod root. https://solidweb.me/username/ has no indication of being a pod root that I’m aware of. Only that it is the last parent that contains its children. (edit2 … we can find pod root precisely by going backwards through parent folders until we locate a container with header link: <http://www.w3.org/ns/pim/space#Storage>; rel="type".)

Just to clarify

What is a developer supposed to do? Create these files? (Where? I don’t know which container I can use.) Or do something else?

I suppose the Community Solid Server v1.1.0 is a minimal but correct implementation of Solid specification. Therefore it’s a very interesting playground to solve these interoperability issues and I want to make it work! <3

[jeffz]

This is a really good and important question. The short answer is that CSS aims to stick as closely as possible to the specs and the specs aim to specify as little as possible until further discussion. So yes, things like typeIndexes and preferences and even the inbox are not available . Neither is any information about contained resources (e.g. mtime, mime-type) in container listings.

I really hope this does not turn into the wild west where every app decides for itself how to handle these parts of discovery. I would really like to see the specs handle this in a way that implementers can handle but I am afraid I have no confidence that the specs will do that anytime soon. So maybe what is needed is a strong statement of best practices from the interoperability and discovery folks.

If we have a “the spec says nothing yet but best practices say include a pointer to a preferences file in your profile”, that will at least allow apps to look in the profile first and fallback to something else if nothing is found. Since users can edit their profiles, there is never a guarantee that any given item will be there, but I think we should specify - if it’s there, this is how it should be represented.

My $0.02.

[jeffz]

Also note that although things like preferences and type indexes do not come by default with CSS, it is quite possible to include them by modifying the default profile and by provisioning the prefs.ttl and other files on pod creation. This requires actions on the part of the pod provider. The fact that CSS comes with almost nothing doesn’t mean that pod providers should offer CSS pods with almost nothing - it is (in my opinion) the pod provider’s responsibility to provision pods adequately. The lack of specs and best practices means that we’ll have the wild west there too with each pod provider doing something different.

[gibsonf1]

Trinpod.us makes all those as well, so I think it would be an important standard to have so apps can depend on it.

[timbl]

Yes, this needs to be written in a solid standard. Not the Solid Protocol, as it is a client-client spec, which the server does not have to know about.
"What is a developer supposed to do? Create these files?

Yes

Where

The same place they are now.

If necessary, an app may have to create the user’s preferences file. But be careful you don’t accidentally zap an existing one just because you couldn’t read it.

Yes. Someone needs an action item to explain that and give the spec a name.

The interp panel should have done that as a first order of business. But they got distracted by a much more complicated system.

[mrkvon]

Hello, thank you for the thoughtful responses!

To clarify a mistake: from Solid Protocol Section 4.1 (It’s a draft? (edit: well everything in the spec seems to be a draft ))

Clients can determine the storage of a resource by moving up the URI path hierarchy until the response includes a Link header with rel="type" targeting http://www.w3.org/ns/pim/space#Storage .

This part is indeed implemented on the Community Server, as we can show by curl -D - https://solidweb.me/mrkvon/. So we can determine the pod storage, i was just unaware of it.

I’m content with the solution (which everybody here seems to agree on) of keeping the best practices document structure in the pods and create it where it’s not found. (publicTypeIndex stuff etc.) hopefully the recommended structure won’t change (much).

I’m just confused by:

be careful you don’t accidentally zap an existing [preference file] just because you couldn’t read it.

Does it mean that I may be able to overwrite stuff I don’t have read access to? Sounds troublesome.

Next steps:
I may give an update with the actual approach i will have taken. Also i’d perhaps further discuss the pod structure we would consider the best practices.

[NoelDeMartin]

We’re making a list of differences for app developers coming from NSS who want to make their app compatible with CSS, make sure to check it out and let us know if you find something else: Add migration instructions for older apps · Issue #607 · solid/solidproject.org · GitHub

[ewingson]

@NoelDeMartin your Media Kraken works fine with CSS

[Vincent]

@mrkvon

So we can determine the pod storage, i was just unaware of it.

Well, somewhat. You can move up from the user’s WebID to find that link header, but that only works if that WebID is in the same URL space as the storage root. This is e.g. not the case for @timbl’s WebID: https://www.w3.org/People/Berners-Lee/card#i

I brought up this question in the solid/specification chat a while ago, and the conclusion was roughly that at this point in time every app should implement the following three steps:

1. Check the user’s profile document (i.e. the resource at their WebID) for a http://www.w3.org/ns/pim/space#storage value. If found, one of those can be used as a starting point for creating the other resources. If no value is found, proceed to:
2. Keep moving up the URL hierarchy based on the user’s WebID to find a resource with a Link header with rel="type" targeting http://www.w3.org/ns/pim/space#Storage. If reaching the root without finding on, proceed to:
3. Ask the user where their storage root is

(In my opinion step 3 is very unfortunate, and needing two steps is a bit of a hassle, but that appears to be the current concensus among the spec editors. Step 1 isn’t specified by the spec but implemented by every non-CSS server, step 2 is specced but AFAIK not yet implemented by all servers, and is the reason step 3 is needed too.)

Only after you’ve done that can you create the files yourself.

[elf-pavlik]

Hi @mrkvon, you might like to take a look at Solid Application Interoperability

Type indexes didn’t provide any control for which data get exposed to which application. With Data Registrations we do provide a clear way to control what information gets disclosed to every application.

Users often will not want that some game app, restaurant table booking app, etc. to even know that they store some very specific kinds of sensitive medical, financial, or other information.

We also have initial TS implementation for the parts needed by regular application sai-js/packages/application at main · janeirodigital/sai-js · GitHub

SAI spec introduces the role of Authorization Agent and we are currently working on implementing that part in TS.

Interop panel meets every Tuesday, everyone is very welcome to bring up their use cases and share feedback. GitHub - solid/data-interoperability-panel: Repository for the Solid Data Interoperability Panel

[mrkvon]

@elf-pavlik funny that i just finished rough reading through Solid Application Interoperability (SAI) document when you mention it

Is the SAI draft The next standard for interoperability? Is it reasonably safe to start refactoring the apps according to it?

After reading the spec draft, here are my initial questions and impressions:

• I tend to learn by example, so:

  • Does an app exist that implements the specification? And tests it against reality? If yes, I’m very interested to have a look at it and at its source code.

  • Is there a Solid Pod (either a whole server, or a single POD) that showcases how the future is supposed to look?

  • Is there any guide written in informal words by someone who used the draft?
    In particular, I’d start with the questions:

    • How does an app discover, whether a person (Social Agent) has anything relevant for it in their POD?
    • How to create a new resource that app could use to store its data and optionally share it with other people (or groups of people) or applications?

• I don’t understand where the app’s identity profile document is supposed to live, and how it relates to the actual web or mobile app. I’d like to be able to advertise to people Check this Cool App at https://check.this.cool.app.example/, not https://check.this.cool.app.example/#id. I want them to see some action, not a profile document. And i hope apps can still remain serverless (i.e. hosted on github pages). Again, an example or an informal explanation would go a long way.

[jeffz]

Media Kraken works fine with CSS

Yes because it creates resources like the preferences file using the defacto standard of what NSS does. If that is what all apps are supposed to do, then it should (as Tim agrees above) become more than a defacto standard because otherwise every app is free to invent its own which will lead to chaos.

[jeffz]

I do not understand the rationale for putting the onus on an app to create its own discovery resources. Does every app then need write permission on the profile and settings locations? I would think that it would be a best practice for either servers or pod providers to provision pods with the basic profile and settings locations already set up with needed permissions (e.g. append but not write). In addition there should be best practices for apps in case the server/provider fails to provision or user erases after provisioning. An app creating its own discovery resources is reasonable as a fall-back but it seems that preventing the need for that fallback could be accomplished at the server or pod-provider level.

[jeffz]

@elf-pavlik - thanks much for your reply and your great work on the panels. The Data Registration stuff looks much more robust and will be a great benefit in the long run. The short run (the next year or so) is a different question. It seems like there are three choices

1. no one creates any apps until the discovery spec is done
2. everyone invents their own approach until the discovery spec is done
3. there is a best practices document which explains how to do discovery right now in a way least likely to be a problem when the specs are done.

I understand that a spec editor’s first job is to edit the specs so taking the time to work on #3 is burdensome. But unless spec editors get involved in it, the jobs of implementers and of those creating support materials for implementers are going to be very chaotic.

[elf-pavlik]

Is there any guide written in informal words by someone who used the draft?

Besides TS implementation I also wrote this short primer: Solid Application Interoperability - Application Primer

How does an app discover, whether a person ( Social Agent ) has anything relevant for it in their POD?

The application should publish their Access Needs which will be used during the authorization flow. If the user grants this application required access Application Registration will be created for that application and it will be following their nose from there.

How to create a new resource that app could use to store its data

The app should use shape trees from some common place, we consider putting one out in the interop panel. Having that user would use their Authorization Agent to create Data Registration (max one per storage their own) which than they can authorize an app to access.

and optionally share it with other people (or groups of people) or applications?

Users own their data so only users can share it with other users. Users use applications to access data they authorized it to access, but the application being just a tool and not owning any data can’t really share it.

I don’t understand where the app’s identity profile document is supposed to live, and how it relates to the actual web or mobile app. I’d like to be able to advertise to people Check this Cool App at https://check.this.cool.app.example/, not https://check.this.cool.app.example/#id. I want them to see some action, not a profile document. And i hope apps can still remain serverless (i.e. hosted on github pages). Again, an example or an informal explanation would go a long way.

Solid-OIDC also introduces a public app profile document. Currently, we only looked there into verifying app identity based on the redirect url in that profile. We definitely need to address how app identity can be verified for mobile apps. So far we discussed using client credentials flow where each user can create themselves static registration for app they are using.
I don’t think there should be any problem with static hosting, apps could be hosted on github pages or even better in solid storage.

I would think that it would be a best practice for either servers or pod providers to provision pods with the basic profile and settings locations already set up with needed permissions (e.g. append but not write).

In interop spec we have the very important role of User’s Authorization Agent, we can see it as something similar to Identity Provider. I would expect that deployments would commonly combine those two. As a party fully trusted by their user (just as the IdP) it will be able to assist user in tasks that regular applications should not be involved in.

everyone invents their own approach until the discovery spec is done

Implementing TS library to be used by applications helped us improve current draft quite a bit. By the rest of this year I will implement module to be used by Authorization Agent which will help us improve remaining bits. I know that @justin will follow with Java implementations. SAI should be pretty stable by the end of this year.

Just to give some perspective, in AuthN panel we are still considering some breaking changes to Solid-OIDC and AuthZ panel seems to be moving pretty slowly for quite some time. I would be surprised if interop spec was the one holding developers back.

[jeffz]

SAI should be pretty stable by the end of this year.

That’s great news. Will it then be actually usable or will we also need to wait for libraries and/or server changes?

Just to give some perspective, in AuthN panel we are still considering some breaking changes to Solid-OIDC and AuthZ panel seems to be moving pretty slowly for quite some time. I would be surprised if interop spec was the one holding developers back.

I am not trying to play a blame game and do not mean to imply anything about the speed of the interop or any other panel. You have a daunting task in front of you and only people actively involved in doing the work have any ground to judge from. But your answer only makes my point more relevant - there are multiple panels that have left implementers in a perpetual state of not knowing.

[elf-pavlik]

GitHub - janeirodigital/sai-js: Typescript library for Solid Application Interoperability already includes the initial version of the module for app developers. Currently, I’m working on the module for authorization agents.
There will be no required changes to the storage server, though authorization proposals are still in flux and known issues with WAC haven’t been addressed yet. An Updated ACP proposal was recently submitted.
There will be a need to write an actual Authorization Agent server which in TS would use modules I’m currently working on. If everything goes as planned I should be able to start working on that in January.

Last but not least we should already start working on the public repository of common shape trees. I know that we could use GitHub - shapetrees/shapetrees.pub for that. I’ll try to bring it up during today’s panel meeting.

[jeffz]

I have created an issue to address these data discovery concerns : Create best practices for profile predicates · Issue #209 · solid/data-interoperability-panel · GitHub.