Add PRD for data abstraction layer by taobojlen · Pull Request #71 · roostorg/community

taobojlen · 2026-06-19T18:43:23Z

This is an early draft of a PRD for the data abstraction layer (DAL).

It's a bit of a mix of a vision doc / PRD, really! The early conversations I've seen covered a lot of ground so I wanted to narrow things down.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

juanmrad · 2026-07-06T14:57:51Z

+7. A template repo with resolver scaffolding, a test harness, examples, and support for dataloaders to prevent N+1s.
+8. When configured, Coop enriches from the DAL instead of Partial Items.
+9. Coop's Partial Items API is deprecated with a migration path.
+10. Bearer-token auth -- a token grants access to the entire API.


Is "full API access per token" an intentional v1 design decision, or just a simplification?

Since the DAL may expose sensitive organizational data, it would be useful to call out that the authentication model is intentionally coarse-grained for v1 and designed to evolve toward field/object-level authorization later. Otherwise readers may interpret this as a long-term architectural choice.

An intentional design decision! Field-level authorization is mentioned in the Future Work section but lmk if I can make that more clear!

juanmrad · 2026-07-06T14:59:49Z

+- Integration with Osprey for ingest-time enrichment
+- Ingest-time enrichment for Coop, i.e. fetching more data when an item is first sent in, before running rules
+- Agentic moderators that can actually explore data via the DAL
+- Caching queries for performance (with good headers so clients know if the response was cached)


I wonder if it's worth calling this out earlier in the document as part of the design philosophy.

Something like:

The DAL is intentionally read-only, which allows implementations and clients to safely leverage aggressive caching where appropriate.

I think that helps explain why caching is a natural fit for the architecture, rather than introducing it later as an optimization.

I added a note in the Approach section!

juanmrad · 2026-07-06T15:05:31Z

+An org running Coop decides that they want to show more data in the reviewer UI. This is the flow I envision:
+
+1. A developer clones a Roost template repo, e.g. `roostorg/data-abstraction-layer-template`.
+2. Inside this template, they first define their ontology (what object types exist in their org, what properties do they have, and how do they relate to each other).


Is the ontology intended to be the canonical source of truth, with the GraphQL schema generated from it?

If so, it might be worth stating that explicitly. Right now it reads as "define the ontology, then implement GraphQL resolvers," but it's not entirely clear whether the ontology exists independently of GraphQL or is simply another way of describing the GraphQL schema.

Clarifying that relationship would make the architecture easier to reason about. It also leaves the door open for generating other representations from the same ontology in the future (e.g. JSON Schema, MCP, protobuf) if different organizations or integrations need them, without making GraphQL the only representation of the underlying model.

great question! i think we'd want to have the ontology be specified on its own, like in some YAML or JSON-ish format. then we can scaffold the graphQL resolvers from that (or something else).

that way, we can ensure that the generated graphQL schema also follows conventions.

calebmcquaid

One question. Looks great!

calebmcquaid · 2026-07-06T15:11:16Z

+5. Query depth is bounded by a configurable maximum.
+6. Developers write resolvers per object type/field. This is source-system agnostic.
+7. A template repo with resolver scaffolding, a test harness, examples, and support for dataloaders to prevent N+1s.
+8. When configured, Coop enriches from the DAL instead of Partial Items.


Can we talk about how Coop would consume the DAL? My understanding of Partial Items is that its responses are validated against the Item Type schema and downstream code depends on that shap. Would the DAL be similar? How would the Item Type map to a DAL object?

this is a great question! i laid out three options here: https://github.com/roostorg/community/pull/71/changes#r3539117701

i kind of prefer the strict version, but that makes the DAL a prerequisite for deploying Coop and makes the lift of building this larger.

julietshen

I have some questions but I like the direction this is going in. Not going to mark it as Approval quite yet

julietshen · 2026-07-06T13:17:47Z

+- Thread (an ordered collection of Content)
+- User (an account or profile on the org's platform)
+
+This is flexible, but has limitations. You cannot easily model that e.g. one user follows another, or that a comment is a reply to a specific other comment in the same Thread.


this is based on my understanding of the code, which might be wrong, but I think this IS possible through "Roles" that Items can have.

Unrelated to this PR, I was taking @selena-lustig through Coop and found that our documentation has little coverage of Roles - @cassidyjames let's figure out what info we can add given the Role an Item plays can actually show relations like comment replies and relationships between users (this one i'm less confident about)

ah interesting! AFAICT:

item types have fields

fields can have roles (like "title" or "IP address")

they also have a field type like string/number/image

in addition, they can have a field type of "related item" which basically makes them a foreign key

so this means that you might have, say, an item type called Comment, with a field called "reply_to_id" which is a "related item" field. so i stand corrected there!

but i don't think this allows you to easily model many-to-many relationships like users following each other. to do so you'd need to create an intermediate item for each relationship (e.g. an item called Follow) -- that seems counterintuitive to me and not super smooth to work with!

i've updated this sentence in the doc.

julietshen · 2026-07-06T13:19:43Z

+
+1. A developer clones a Roost template repo, e.g. `roostorg/data-abstraction-layer-template`.
+2. Inside this template, they first define their ontology (what object types exist in their org, what properties do they have, and how do they relate to each other).
+3. Then, they write e.g. Python code to implement resolvers that serve this ontology as a GraphQL API.


How much custom code would need to be written by adopters? Especially if they are not using AI for engineering?

A minimal example, just reading from a single DB, after the initial scaffolding, might only be in the ~100s of lines? Writing resolvers is usually quite straightforward. It gets more complicated if their data is split across many systems, or if they start hitting performance problems like N+1s.

julietshen · 2026-07-06T16:09:47Z

+
+## Open questions
+
+1. How should this interact with Coop's existing Item Types concept? Can we replace that entirely to avoid duplication? That would expand the scope but may be worth doing.


if we punt this question, does that change the requirements for the DAL?

I think it's impossible to punt fully. Coop needs to be able to communicate with the DAL (otherwise the DAL will be pretty useless) and this decision is crucial for how the API will work.

The weak version would be that we put the onus on the user to implement an ontology that matches what they have in Coop. But this is fragile and error-prone.

The medium version is that Coop somehow checks if the DAL's ontology is compatible with its defined Item Types, and errors if not.

The strict version is that the ontology lives entirely in the DAL and Coop just consumes from that. I.e. the DAL becomes the new source of truth for Item Types.

julietshen · 2026-07-06T16:12:02Z

+
+1. How should this interact with Coop's existing Item Types concept? Can we replace that entirely to avoid duplication? That would expand the scope but may be worth doing.
+2. Where will agentic review live -- in Coop's manual review tool, in Osprey's rules engine, or in some new service?
+3. How important is ingestion-time enrichment in Coop? Do we actually need it in v1?


this is a good question that i want to raise with the adopters. @dom-notion @ThatKoffe

julietshen · 2026-07-06T16:12:25Z

+
+## Risks
+
+1. Solidifying an ontology and writing resolvers is an additional engineering lift that orgs may be reluctant to take on.


this is my biggest concern - are we adding more friction and work for adopters and needing them to write more custom code?

if the adopters don't use the partial items API -- no, if we keep the DAL as an optional bolt-on, they don't need to do anything.

if the adopters do want to use the partial items API -- not really. i'd say that the DAL is a more generalised version of implementing the same idea. of course, if we do deprecate the partial items API, any existing adopters would need to migrate eventually per some deprecation timeline. (i would not recommend maintaining both pathways indefinitely).

though since the DAL will be deployed as a new service, there is some operational overhead to deploying it. we can work to minimize this.

julietshen · 2026-07-06T16:13:03Z

+## Risks
+
+1. Solidifying an ontology and writing resolvers is an additional engineering lift that orgs may be reluctant to take on.
+    - This means that we need to make it worth it. Initially, replacing Partial Items will be that carrot, but soon after (post-this-PRD) we should add support for Osprey/Coop ingest-time enrichment and/or agentic moderation.


let's separate agentic moderation from this, as not all adopters will see that as a carrot. We can define the pain points of Partial Items in more detail to make sure the value is super clear

julietshen · 2026-07-06T16:13:21Z

+
+1. Solidifying an ontology and writing resolvers is an additional engineering lift that orgs may be reluctant to take on.
+    - This means that we need to make it worth it. Initially, replacing Partial Items will be that carrot, but soon after (post-this-PRD) we should add support for Osprey/Coop ingest-time enrichment and/or agentic moderation.
+2. If the DAL is optional infrastructure, there is a risk that orgs will just not use it.


and if it's optional while we deprecate Partial Items, are we making Coop worse out of the box?

i'd say no -- there's still a way to achieve the same thing (i.e. via the DAL), and though the tech stack might be more opinionated (e.g. requiring graphQL) it is fundamentally not very different.

julietshen · 2026-07-06T16:14:51Z

+    - This can be a real problem, especially in high-volume use cases!
+    - We cannot control these source systems so the best thing we can do here is around observability, so it's easy to see *what* is slow.
+6. The DAL may overload source systems if used heavily for T&S.
+    - This is an inherent risk. Orgs will have to manage this, and again we can ensure that it's easy to understand which queries the DAL is executing.


i'd like to see an aggregated list as part of the Risks of what new problems/pain points might we be introducing to adopters

writing custom code as resolvers for the DAL

is overloading source systems an existing risk or a new one we'd introduce

requiring APIs that provide broad access to their data

(as in - a lot of that is here already, but call it out specifically as potential challenges for adopters)

i think these risks are basically all potential problems or pain points for adopters. but some also apply to the Partial Items API (i.e. they are pre-existing!).

the new problems that the DAL, as written here, could introduce are:

graphQL may be unfamiliar dev experience/tooling

the overhead of deploying a new service

it's easy to create N+1s in graphQL

the ones that also apply to the partial items API:

there is eng lift involved

source systems may be slow to respond with data

source systems may get overloaded

the API allows broad access to data

taobojlen and others added 4 commits June 19, 2026 19:41

initial data abstraction layer PRD draft

4f37ed3

data abstraction layer PRD: expand scope, risks, and GraphQL framing

635d961

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

flesh out PRD

530b026

flesh out requirements

9ae390e

taobojlen marked this pull request as ready for review June 23, 2026 14:20

cassidyjames requested review from cassidyjames, juanmrad and julietshen June 23, 2026 14:28

cassidyjames added 2 commits June 23, 2026 08:32

Merge branch 'main' into tao/data-abstraction-layer-prd

3522f94

Merge branch 'main' into tao/data-abstraction-layer-prd

e399c66

cassidyjames requested a review from a team as a code owner June 25, 2026 01:57

juanmrad reviewed Jul 6, 2026

View reviewed changes

calebmcquaid approved these changes Jul 6, 2026

View reviewed changes

julietshen reviewed Jul 6, 2026

View reviewed changes

address comments, round 1

dab14a4


		## Open questions

		1. How should this interact with Coop's existing Item Types concept? Can we replace that entirely to avoid duplication? That would expand the scope but may be worth doing.


		## Risks

		1. Solidifying an ontology and writing resolvers is an additional engineering lift that orgs may be reluctant to take on.

Uh oh!

Conversation

taobojlen commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

calebmcquaid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julietshen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

taobojlen commented Jun 19, 2026 •

edited

Loading