Fetching Multiple Child Resources in Solid/Inrupt App: Best Practices?

Build a Solid App
1

I’m currently developing a Solid App using React and Inrupts js SDK. I’ve encountered a problem when I want to fetch all child resources of parent resources. Let’s say I have this structure in my pod:

  • parent/
    • child-1
    • child-2
    • …
    • child-n

Both the parent and the child are RDF resources. Currently, to fetch all the child resources, I first fetch the parent resource, get all the things defined in it, loop over the array of things containing the URL to the individual child resources which I use to fetch them:

async function getAllChildResources(session: Session, parentPath: string) {
  const parentDataset = await getSolidDataset(parentPath, {
    fetch: session.fetch,
  });
  const parentThings = getThingAll(dataset)

  const childResources = await Promise.all(
    parentThings.map(async (thing) => {
      const childResource = await getSolidDataset(thing.url, {
        fetch: session.fetch,
      });
      return getThingAll(childResouce)[0]
    })
  );

  return childResources;
}

The problem with this approach is that I’m sending n+1 request to the pod provider. When the number of child resources grows, this results in 429 too many requests as I’m rate-limited by the pod provider. Is it possible to send one request that returns all child resources? Is there a way to paginate so that I can fetch a subset? I’ve read about both LDP and SPARQL, but from my understanding, LDP is a bit limited, and SPARQL is not supported by all pod providers.

2

Unfortunately that is correct, Solid does not mandate a way to fetch all that data in a single request. I think your two options are either accepting the latency, or changing the data model to spread the data out over fewer Resources.

1 Like
3

Thanks for the reply. Unfortunately, I cannot segregate the data any further, as users who upload child resources will only have append access. I guess I have to figure out something clever to solve my problem.

4

There are some search things in the works but none are implemented on open source Solid servers. You might want to look at TrinPod which I believe can support your use case.

1 Like
5

Things depend on your use case a bit, but am I understanding correctly that each child Resource represents a single user, and that that user has Append access to that particular Resource? One other thing you could do is to have your own bot running that has access to all child Resources, and that indexes them into a single Resource. It’s a bit of extra work, which doesn’t feel great, but I think it should work.

Or indeed, if all those users are writing to a single server that you control anyway, you could indeed look at using a particular server that supports this. (Though in that case, you could also consider just not using Solid for that part? Since for the users there would be no material difference, I think.)

6

Depending on the app you’re making, you could also consider hosting each user’s resource on their own pod, then keep the links in an index.

This has some implications for control and discovery and may not be well suited for your use case. Otoh it could be an exercise of decentralization and user control, and may address your rate limit error, since you’d be fetching from many different resources.

7

To be more concrete, my app has a feature where users can send notification to each other via an inbox. When you create an account, the inbox RDF is created with public append enabled. If user A wants to send a notification to user B, user A can create a new resource and add it to the inbox. The inbox RDF lives inside user B’s pod, and B has full control over this resource.

Because A only has append access, I’m limited in how the data can be structure:

  • User A cannot create a new thing and add it directly to the inbox RDF. He has to create a new resource and append it as a child RDF with the notification as a thing inside the child RDF.
  • User A cannot chose the name of the child resource (at least when testing with Inrupt, the child resource is named with a random UUID)
  • User A cannot create sub-containers inside the Inbox.

The problem with fetching multiple children appears when user B logs into the app and wants to see his inbox. That’s when I have to fetch the inbox RDF to get the link to all the children, then send a request per children to get the details about the notifications.

Edit: Can also add that when listing the inbox to the users, I prefer that the notification are sorted by the time they were sent. The notification thing inside the child resources has this information, however, this means that all child resources has to be fetched into memory before it can be sorted. Just seems like this approach scales poorly.

8

Your best option here is to have an automaton which listens to the inbox resource, and that would handle the process of listening for new appended resources and retrieving them, at which point it can sort the notifications and dispatch them in appropriate order.

The problem with fetching multiple children appears when user B logs into the app and wants to see his inbox. That’s when I have to fetch the inbox RDF to get the link to all the children, then send a request per children to get the details about the notifications.

One way to address this is store a cache using webworkers, since your model will support it. Since other users have append only, you just need to fetch it once then cache that request. The next time the user logs in and the request is executed, you only need to fetch the new ones while the old ones won’t be changed.