Problem with retrieving pieces of data from separate files

Hello everyone, here is the problem I came across.

  1. End goal:
    I want my things to be stored within a specified folder in separate files. And then I want to able to easily fetch them from the user’s POD in a single fetch call.
  2. Here is the story so far:
    I use the @inrupt/solid-client library, here is how I create a folder:
 createContainerAt(notesFolder, {
      fetch: fetch

This would create a container where I store my files.
Next I save files in this container, using the following method:
I first fetch the dataset:

 let dataSet = await getSolidDataset(notesFolder, {
    fetch: fetch

Then I build a Thing:

 const newNote = buildThing(createThing({ url: `${notesFolder}${id}.ttl`})).addUrl(RDF.type, schema.TextDigitalDocument)
    .addStringNoLocale(DCTERMS.title, note.title)
    .addStringNoLocale(schema.text, note.content)
    .addInteger(schema.identifier, id)

Then I set this thing in a data set I fetched before and call save in container function:

 dataSet = setThing(dataSet, newNote);
  const updDataSet = saveSolidDatasetInContainer(noteUrl, dataSet, { fetch: fetch });

This is how a resulting file looks like in a user’s POD:

this is how the container folder looks like:

when I try to fetch the files as an array (there will be many files like this)
this is how I fetch them:

let dataSet = await getSolidDataset("", { fetch: fetch });
    let allThings = getThingAll(dataSet);

Now, in this array of things, the first thing would be container itself, and the second thing would be the note that I want to fetch.
this is how the note thing looks like if output it to the console:
Screen Shot 2022-05-16 at 8.30.38 PM

Now here is where the problem comes:
If I try to get one of the values, from this note, I get null, for example:

console.log(getInteger(allThings[1], "")); 
console.log(getStringNoLocale(allThings[1], ""));

Both would result in a null.
I would really appreciate any help as I am completely lost in this.
Thank you in advance

The problem here is that when you fetch the Container, you don’t get the Resources that are present in the Container, but instead you get a SolidDataset that contains references to where you can find the Resources in that Container.

In other words: it is almost by definition not possible to have your data in separate files, and fetch them in a single request, because pretty much all that a file is is something that can be fetched in a single request.

So the question is: why do you want your data to be in separate files, and would it not be possible to solve that use case by storing them as separate Things in a single SolidDataset? I’m expecting that there’s a way to make that work (and we can help you with that if you describe your use case), but if not, I’m afraid you’ll have to learn to live with the separate requests.

(Also, one other important thing to realise: there is no guaranteed order of Things in a SolidDataset. Probably doesn’t matter in this case, but good to be aware of.)

1 Like

Thank you for your reply, this clarifies a lot.
The things that concern me, ie why I wanted to go for a “each note in a separate file” approach are the following things:

  1. I want to set an access type to each note, which might be different for different notes. For example:
    some notes, might be public, some notes might be private, and some notes might be “shared”, ie the access will be granted only to users chosen by the note’s owner.
    Is it possible to do so, with all notes stored as Things in a single SolidDataset?
  2. Also, if someone other app will want to edit a single note in the user’s POD, it will have to fetch the whole SolidDataset with all notes stored in it.

Now the only use case when I fetch all the notes, is when I display all notes that the user has in his POD as a list, and after some change has been made (new note created, old note deleted, etc) I would fetch all notes again, obviously having let’s say 20 fetch call’s on app load, and 20 fetch calls each time a new note is created or deleted is going to result in long waiting times for the user.
The question is: Is there a way to “subscribe” to updates of the Container, so I don’t have to fetch all the notes each time a single change occurs, but rather to only fetch that one change.

P.S some clarification: I don’t have any strong preference for either approach, just trying to figure out what would be the best for my situation.

P.S.S not related to topic, but is there a way to notify a user that for example access to a note was granted to him? Ie by sending a message to his inbox container?
Thank you for your help!

you could use getcontainedresourceurlall() function : Module: resource/solidDataset — Inrupt solid-client API Documentation

 dataset = await getSolidDataset( path, { fetch: sc.fetch });
          let remotesUrl  = await getContainedResourceUrlAll(dataset,{fetch: sc.fetch} )

then fetch each of remotesUrl

Ah yes, different access controls are indeed a valid reason to use separate files, and unfortunately that does indeed mean you will need as many requests. And no, there currently is no standard and widely supported way to subscribe to updates to a Container, as far as I know.

You can indeed send a message to someone’s inbox Container (if present and allowed) the same way you’d create any Resource. That said, they’ll only see that if they’re opening an app that processes those messages (might be yours).

I don’t know about the current implementations. The specification says that servers SHOULD implement the Solid Websockets API for live updates, but also mentions transitioning to a new protocol which is in an unofficial draft. Personally, based on this spec, I currently wouldn’t use any of them for something that’s meant to last some time (without requiring multiple changes)

1 Like

@A_A - are you aware of the very recent new specs on this topic : Solid Notifications Protocol, Solid Notifications: WebSocketSubscription2021

Yes, that’s the new specification Solid is transitioning to. I haven’t looked closely at it, so I can’t speak about the capabilities of it. But it’s still described as a WIP with an expected completion at 2022-12-30. So for experimenting it may be interesting, but for building something stable, production-ready I would still wait until the end of the year.

This document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

@A_A - Fortunately, not everyone takes the position you do. If they did the specs would operate in a vacuum uninformed by implementations. We are at a phase in Solid where no, not everything is specified or permanent, but, yes, we need developers trying out the specs as they exist to see what works and what needs improvement. Individuals such as yourself are free to opt out until things are stable, but advising other people to do the same as a blanket statement is counter-productive for this project.

Rather than “you probably don’t want to even try this until it’s fully specified”, a much more helpful suggestion would be "the specs are still not written in stone, if you are unsure about a specific feature, talk to the spec editors and/or look at the issues on the spec and report back to the spec editors on your implementation experience. "

It is true that someone building an app meant to be in production, stable, and last for years should not count on things that are not specified. It is a mistake however, to assume that most people in this forum are working on such apps or that those who are would need a warning about the state of the specifications.


@jeffz Thank you for highlighting the importance of experimenting. I think this can be useful for both, the developers to gain experience and insights on how stuff can work, and the specification to gain valuable feedback. I also said above that it’s interesting for experimenting, and that’s why I even brought up the existence of these specs. Looking back at my writing, I buried this behind some warnings about non-production-readiness.

1 Like

Thx Vinvent , i recognize that point you’re making…in some respects is it not good that such functionality of segmentation can increase compartmentalization? I can see it’s beneficial effect in this context… namely, because it will influence the way that I think about doling out my personal information… which for me can be handy in terms of influencing personal behaviour modification is relative to better data sovereignty