Saltar al contenido
ClickHouse Loom
Esc
↑↓navegar↵abrir⌘Jvista previa
En esta página

Document sources

A document source pulls files from a document store (a shared-folder store, a page store, a project tracker, a meeting store) into a namespace’s documents and keeps them in step, on any shared namespace or collection. This page is the design of sources of many items given as pasted links, per-user connected accounts, an org-wide crawl, and access that follows the store’s own permissions. The caller’s view of today’s sources is in knowledge: document sources; parsing is in document extraction. Optional provider notifications request the same sync path; source push describes their setup, authentication and lifecycle.

What changes

Today With this design
A source holds 1 to 20 folders, projects, pages or databases, and no op changes its config. A source holds up to 300 items of mixed types, given as links or ids, and knowledge.source.update adds and removes them (items).
One shared credential per source, typed by an admin. A shared credential, a person’s own connected account, or a delegated service account that reads many people’s drives (credentials).
Every reader of the namespace reads every synced document. A source’s access is namespace or mirrored; a mirrored document is shown only to a reader the store says may read it (access).
A linear source reads project documents only. Issues and comments are a later design of their own (issues).
A recording source reads meeting-store transcripts (meet) or cloud-recording transcripts (zoom). Meeting notes and transcripts that the meeting store saves to the shared-folder store come in through a gdrive source; recording keeps zoom only (meetings).

Any namespace

A source can be registered on any active shared namespace or collection; who may register which kind is in credentials. add on an inactive namespace answers NAMESPACE_INACTIVE, and add and update answer ERASURE_RUNNING (409, retryable) while an erasure that will still run names the namespace, so no source writes into a mirror the erasure is clearing. A source on a repository’s trunk code namespace still tags each document repo:<id>, so the repository’s code.context and loom_search legs read them as today; on any other namespace, a branch namespace included, it does not. A person’s own place (a col-me-… collection, a personal-… or u-… namespace, a namespace claimed at sign-up) holds no source (PERSONAL_NAMESPACE): a user_id erasure erases what the person wrote, and a source’s documents are the sync worker’s. The sync skips a source whose namespace is inactive; the cleanup of a stopped source runs whatever its namespace’s state.

A source ends with its namespace: code.repo.remove, namespace.delete, collection.delete and a namespace erasure each retire the namespace’s sources under its source lease, clearing their credentials, as code.repo.remove does today, so no source outlives its namespace with a sealed credential or writes back what an erasure took.

A source’s sync state is keyed by its namespace and name, as today, and a mirrored source’s by its mirror and source id, so the namespace’s erasure covers it and a later source of the same name resumes none of an earlier one’s listing. It lives where storage layout places it (loom.code_index_state today, Postgres cursors from wave 3).

On the dashboard, the Library’s Sources button opens the sources of the namespace or collection in view; the repository card keeps its own.

A source’s items are its things to read, of mixed types:

Kind Item type Fields Reads
gdrive folder id, recursive (default false) the folder’s files; with recursive, every subfolder’s too, to a depth of 8 and at most 500 subfolders per source (both not settings)
gdrive file id one file of a kind include names
gdrive shared_drive id every file of a shared drive
gdrive filter any of name_contains, kinds, modified_after (a date), and optionally in (a folder id) the files that match, as a query the server builds
gdrive meet_recordings none the meeting store’s recordings folder of each account a connected or crawl source reads (meetings)
notion page, database id as today’s page_ids and database_ids
linear project id or slug_id the project’s documents, as today’s project_ids

Every id keeps today’s alphabet check, since an id goes into a store query or a request path. A filter never takes a raw query string: each field has a pattern (a name fragment of printable characters, an ISO date, kinds from the include table), quotes and backslashes are escaped, and server/clients/gdrive.py builds the query, so a caller cannot widen a listing past what the fields say. include stays the source-wide file-kind filter. meet_recordings is refused on a shared source, whose service account has no recordings folder of its own.

A source holds at most 300 items (a wire contract, not a setting; a linear or notion source keeps its 20 until its turn bounds what it lists), and at most LOOM_KNOWLEDGE_SOURCE_MAX_DOCUMENTS documents (default 50,000). The bound counts the source’s stored documents plus the new ones a turn writes (at most 200,000, since the paged read of the stored copies costs about the square of the bound over 10,000). A turn over it (a listing grown past it, an item swap whose new files are listed before the old copies retire, or a lowered setting) holds back new documents only: it finishes its listing, rewrites changed lists, retires, and lands its listing row, so its copies drain back under the bound and no list goes stale; inspect and loom doctor show too_many_documents meanwhile, with the count, for a namespace source; for a mirrored source, its connected owner sees the code without the count, and a platform admin sees both (exception 4). The read of a source’s stored copies goes in pages by source_uri, so it never cuts one off. A listing asks the store about at most 50 parent folders per query (not a setting), so a source of many folders lists in several queries; a recursive folder’s subfolders are walked on every listing, so a new file in a new subfolder arrives with the next turn, and files moved in with it (a move keeps a file’s change time) with the next full listing; a walk the pass deadline cuts leaves the listing incomplete, so the turn keeps its stamp. A walk cut at its depth or folder bound still lets a full turn retire a file the store reports deleted or trashed, or one of a kind include no longer names, but never one the listing merely lacks, since it may sit below the cut.

A probe checks a bounded sample: every item while there are at most 20, otherwise the 20 items added last, and the answer says sampled: true with the number checked, which is shown wherever the item count is.

On a mirrored source, the probe and the status hold back what depends on what a person can read:

  • A mirrored source’s probe answers reachability, the number of items checked and codes, never a count, a sample of titles or a summary that names a file, since it lists as the owner or as crawled people; which items were checked goes only to a connected source’s owner and a platform admin.
  • To anyone else, a mirrored source’s probe, status and sync_now answer only source-level codes, whether the credential and its scopes work and owner_inactive, and depend only on source-level conditions, so a held-back code reads as a clean sync. rate_limited, which grows with how much a source lists, an item refusal, a per-file or per-item code (item <n>: http_404, meet_recordings: not found, too_large, extraction_failed:<code>), the oldest listing start and every last-sync time go only to the owner and a platform admin.
  • A refused item of the sample refuses add or update only on a namespace source and on a connected source probed by its owner; on any other mirrored probe, a platform admin’s included, items never refuse, and only the source-level codes do. The next sync turn reports a refused item as item <n>: http_404, to those who see per-item codes, and goes on with the rest.

server/knowledge/domain/source_links.py turns a pasted link into an item, with no request to the store:

Link Item
https://drive.google.com/drive/folders/<id>, with or without u/<n>/ or mobile/ gdrive folder
https://drive.google.com/file/d/<id>/… gdrive file
https://docs.google.com/document/d/<id>/…, …/presentation/d/<id>/… gdrive file
https://www.notion.so/<slug>-<32 hex>, https://www.notion.so/<workspace>/<32 hex>, https://<workspace>.notion.site/<slug>-<32 hex> notion page (database with ?v=, unless p= names a page)
https://linear.app/<workspace>/project/<slug>-<id> linear project

A shared drive’s own link has the folder form: a folder link names a folder, read without its subfolders, even when it is a shared drive’s root; a recursive folder at a drive’s root is turned into a shared_drive item by the probe. A link of another host, with no id, or to something the sync does not read (a spreadsheet, a single issue) answers unreadable with the reason, as does an open?id= link, which names a folder or a file alike. The parser takes a list and answers each link, so a paste of fifty links shows which three it could not read. It is knowledge.source.parse_links (viewer; no store request, nothing written), which agents use to preview pasted links; add and update take links directly. A page link that turns out to be a database is reported by the probe. A project link carries the project’s short id, which add and update resolve to the UUID the store’s API takes, in one lookup before the probe.

Wire shape

knowledge.source.add takes items, or links it parses (an unreadable link refuses the call naming the first 10 with their reasons and counting the rest, BAD_LINK), and still takes config, which it turns into items (folder_ids into folder items, and so on), so an N-1 client keeps working. list and inspect answer items, and config in today’s form when the items can be written in it, else {}; a mirrored source answers config {} to everyone, and its audit rows carry every field of its items and a crawl’s subjects as keyed hashes only (below). A filter has no owner field: it would keep a person’s address in the registry, which outlives their erasure. Its name_contains is an admin’s text and, like the source’s name, stays in the registry and the audit log after an erasure.

knowledge.source.update takes add_items, remove_items (each also as links) and a new include, probes what it adds with the stored credential (the same sample rule), and writes the source at its next version under the namespace’s source lease. Its audit row, and add’s, names the namespace, the name and the items added and removed, with a crawl’s subjects and every field of a mirrored source’s items as keyed hashes (below). Removing every item is refused (BAD_CONFIG); remove ends a source. Changing access is a new source. For a crawl, update also takes subjects (the new list, probed like added items) and directory_subject. Every update and rotate of a crawl needs the same roles as add, since its credential reads every listed person’s drive: a namespace admin without an org-wide role cannot change whose drives it reads, what it reads of them (items, include) or the key it reads with.

One document, one owner

In a namespace, one document per address, as today, owned by one source: the source whose source:<name> tag it carries, written by the sync worker (today’s ownership check). Only the owner rewrites the document when its file changes and retires it on positive evidence (today’s retirement rule), and a document another writer (a person, an agent) wrote is never overwritten or retired by any source, as today. A mirrored document also carries source-id:<id>, the id of the source that wrote it, and in a mirror that tag, not the name, decides ownership: a later source of the same name owns none of an earlier one’s documents, and the cleanup of a stopped mirrored source retires by it. Only N writes a mirror, so every mirror document carries the tag. Inside a mirror a document’s source_uri is its address with the source id appended (<address>#<source id>; the address is the part before the last #, and citations and answers use it), so each mirrored source writes its own copy with its own list under the table’s key, heads and document machinery as they are, and no source’s write or retirement touches another’s copy. The legs answer the address itself, dropping duplicates after the check (below), and a get or download of an address answers the namespace copy when the reader may see it, else the newest mirror copy the reader may see. Only the owning source’s turn writes a mirror document’s list; any other writer of a mirror row (a retention or tier move, a sweep, a reindex) writes a tombstone or unread, never a list it carried. Only those writers and the erasure write rows under a mirror’s name; no request does. How a stopped source’s documents and rows are found and cleaned when several sources, builds or names meet is the implementing pull request’s to specify.

In a namespace, a second source that lists a file at an address where a document is counts it skipped, as today, and fetches no content. No source reads whether another is live: an address is free exactly when no document is there. Once the owner’s document is retired (by today’s retirement rule, its remove with delete_documents), the next source that lists the file writes it as a new file under its own tag at its next full listing (knowledge_source_sync_full_interval_seconds). A kept document (one a remove without delete_documents left) is skipped by every source until knowledge.delete removes it, as today.

So a file that two namespace sources reach is written once and read through one document. Between the owner’s retirement and the other source’s next listing of the file, the document is missing; that gap is the cost of keeping one owner per address there. A mirror pays duplicate chunks for a file two mirrored sources reach instead.

A mirrored predecessor never makes add of its name answer SOURCE_DELETE_PENDING, since mirror ownership goes by source id, and it counts toward no source quota while it waits; its pending retirement shows to a platform admin only. remove with delete_documents retires the source’s documents, as today, and without it they stay as kept documents, which add of the same name refuses to take over (SOURCE_DOCUMENTS_KEPT, as today). A mirrored source’s documents are always retired (below).

Credentials

A source has a credential_mode:

Mode Credential Registers it Writes to
shared Today’s: a service-account key, an integration token, an API key an admin of the namespace the namespace, or its mirror when access: mirrored
connected The registering person’s own connected account for the kind’s store a person with writer on the namespace; the source is theirs that namespace’s mirror
crawl A service account with domain-wide delegation that reads, as each listed person, what that person can read an admin of the namespace who also holds org_admin or platform_admin the namespace’s mirror

The gate of add, update, check, sync_now and remove drops from admin to writer. The handler then requires admin for a shared or crawl source. On a connected source only its owner adds items or changes include, since the items read with the owner’s account; its owner or an admin of the namespace may check, sync_now and remove it, and only its owner, who sees them, removes its items. rotate stays admin (for a crawl, also an org-wide role) and is refused for a connected source, whose credential is its owner’s connection. Owners are compared as canonical subjects (server/auth/aliases.py canonical_subject), so a person’s other spellings are the same owner.

LOOM_KNOWLEDGE_SOURCES_MAX_PER_NAMESPACE (default 8) bounds a namespace’s shared and crawl sources, as today. LOOM_KNOWLEDGE_CONNECTED_SOURCES_MAX_PER_PERSON (default 8) bounds a person’s connected sources in one namespace, and LOOM_KNOWLEDGE_CONNECTED_SOURCES_MAX_PER_NAMESPACE (default 64) all of a namespace’s, so writers cannot fill an admin’s quota or the sync’s passes. A crawl’s credential reads every listed person’s drive, which is why it needs an org-wide role; a turn stops a crawl whose registrant no longer holds admin on the namespace and an org-wide role, as the owner check below stops a connected source.

Connected accounts

A person connects their own account from the dashboard (Settings → Connected accounts) or with loom connect gdrive|notion|linear. Connecting needs LOOM_PUBLIC_BASE_URL: the redirect URI registered with the store is built from it, never from a request’s Host.

  • The flow is bound to the browser that starts it, as SSO account linking binds its ticket (_link_cookie in server/auth/oauth.py). The dashboard’s page calls POST /v1/connect/<kind>/start with its bearer token, which must carry a sign-in session (sid), which a service-account token lacks; the cookie below, not the token, is what ties the connection to one browser. The server keeps a single-use state in Keeper for 10 minutes, bound to the subject and to a random binding it sets in that browser as an HttpOnly, SameSite=Lax cookie, and answers the store’s consent URL, which the page opens. The store redirects to GET /v1/connect/<kind>/callback, which stores the connection only when the request carries the cookie whose binding the state names; the subject is the one recorded at start. A consent URL sent to someone else attaches nothing: their browser has no such cookie. loom connect opens the dashboard’s Connected accounts page for the kind, where the person signs in if they have not. PKCE where the store takes it; include_granted_scopes=false, so an earlier grant of the same client is not folded in. After consent succeeds, a callback whose HTTP Accept header prefers HTML returns a redirect to the fixed same-origin /settings/connected-accounts path. Other clients receive {"connected": true} as JSON. Both responses disable caching, carry Referrer-Policy: no-referrer, vary on Accept, and clear the browser binding cookie. The dashboard URL carries no code, state, token or caller-supplied return address.

  • Scopes, the narrowest each store has:

    • gdrive: https://www.googleapis.com/auth/drive.readonly, openid and email (which the token answer spells https://www.googleapis.com/auth/userinfo.email). The Drive scope is a restricted one, so the operator registers an OAuth client that the store treats as internal to the org’s workspace; a public client needs the store’s verification first. drive.file with the store’s file picker reads only files picked in a Loom page, never a folder’s later files or the recordings folder, so it is not offered. The store’s device grant does not issue Drive scopes.
    • notion: the public-integration OAuth, where the person picks the pages to share on the consent page. The operator registers read-content capability only, without write, comment or user-information capabilities. The OAuth flow has no documented scope parameter; token introspection can report an opaque scope, without a documented mapping to capabilities. Neither that value nor its absence proves the registration is read-only. The registration is the permission bound; a display email is optional because reading it needs a separate user-information capability.
    • linear: an OAuth application with the read scope and actor=user.
  • GET /v1/connect/<kind>/callback exchanges the code. Each granted scope is mapped to its canonical name, and a granted scope outside the kind’s set refuses the connection. The callback stores in loom.connected_accounts the canonical subject, the kind, the store account’s id, optional display email, scopes and refresh token. The store id and email describe the connected account; neither changes the person’s Loom identity or grants namespace access. A usable refresh token is required for ongoing sync, including for notion; a missing token refuses the connection instead of falling back to storing an access token. The refresh token is sealed under the purpose connected_accounts.credential, which loom secrets reseal covers. An access token is held in memory only. The optional display email is sealed separately under connected_accounts.email; neither value is copied into a source row.

  • Refresh for a source retains its account generation and source binding across provider work. One reservation owns the account generation at a time. A successful rotation is sealed only while the owner, account fence and reservation remain current; source or requesting administrator loss then refuses delivery without discarding that rotation. Disconnect, replacement, erasure or deactivation cannot be undone by an older refresh. A cancellation or lost response does not prove the provider left its token unchanged, so a retry needs the provider’s documented recovery guarantee. Missing expiry metadata does not justify an invented lifetime or permanent access-token fallback.

  • Each native HTTP call receives an internal CheckedAccess checkout rather than an ordinary credential string. It retains the owner, source id and connection generation, rechecks standing after pacing and before dispatch, and rechecks after the bounded response stream closes. Its deadline covers those waits and the body read. It cannot be converted into a shared credential or serialized, and it forwards credentials only to the provider’s fixed native API paths, without redirects. A source changed during the request discards the result.

  • Provider token contracts differ:

    • gdrive can omit a replacement refresh token. Retain the existing refresh token in that case. Revoking a token affects the person’s grant across the OAuth project’s clients. Use a project dedicated to document connections; unrelated applications must not share that revocation boundary. Other affected connections stop when refresh is refused.
    • notion refresh replaces both access and refresh tokens. The authorization guide requires the pair, while the response reference permits a null refresh token; a reply without usable replacement material is unavailable. The documented REST flow supplies no token lifetime, PKCE or lost-response recovery guarantee to assume.
    • linear refresh replaces both tokens and renews the refresh token’s 30-day lifetime. Track access expiry from expires_in. The provider documents a 30-minute recovery window for a lost successful refresh response, excluding revoked or replaced credentials; this does not permit retrying another provider’s rotation.

    These contracts follow the stores’ authorization references: Drive, page-store authorization, page-store refresh, capabilities, token introspection and tracker authorization.

  • A connection belongs to the subject that made it, and only that subject’s sources use it. connection.list and connection.disconnect act on the caller’s own; an erasure of the person deletes its connections and retires their sources (erasure). An org admin sees in admin Users which kinds each person has connected, never a token or the store account’s email.

  • Connecting again with another store account replaces the connection and stops the sources that used the old one, as a disconnect does: their items were chosen for the old account.

  • Every path that ends a connection (connection.disconnect, connecting again with another store account, a SCIM deactivation for every subject it revokes, an erasure of the person, a restore) ends it in one order: it unseals the credential into memory, clears the sealed credential and marks every source that uses it inactive (the part that fails closed), and only then revokes at the store with the token in memory, best effort, under a short timeout and outside any lease or claim, as SCIM revokes tokens past its leases; a revoke that hangs or fails leaves the credential cleared. Local disconnection is complete independently of remote revocation. The page store’s revocation reference documents access-token revocation, without promising refresh-token or whole-grant revocation. An access-token revoke therefore cannot be reported as invalidating its refresh family. If only refresh material is available, bounded refresh solely to obtain an access token for revocation may follow local inactivation; no replacement credential is persisted. An unavailable or unproven remote revocation remains so, and the person can remove the integration’s grant at the store. The sync worker then retires those sources’ mirrored documents. A turn also checks the owner: a source whose owner subject is revoked, deactivated, or no longer holds writer on the source’s namespace is stopped (written inactive under the source lease, status: owner_inactive) and goes as a removed source does, so a missed hook fails closed.

A connected source reads the person’s own files, the files and folders shared with them, their recordings folder, and the pages and projects they can read. It reads no directory, so it expands no group (below).

Org-wide crawl

A crawl source’s credential is a service-account key whose client id the org’s workspace administrator granted domain-wide delegation for drive.readonly, admin.directory.group.member.readonly and admin.directory.user.readonly. Its crawl_options field holds directory_subject (a workspace administrator the directory reads impersonate) and subjects: people (emails) and groups (group emails), whose members the directory read lists at each turn, at most 2000 people (not a setting). For each person the sync exchanges the key for a token that impersonates them (the JWT bearer grant with sub) and lists the source’s items as them. A typical config is one meet_recordings item and a filter with kinds: [docs], over a group: every member’s meeting notes, without anyone connecting. Every list and members row the sync writes, a crawl’s expansion included, leaves out the people an erasure excluded (erasure).

The probe checks that the delegation grants no write on what the crawl reads: it asks for each of the three scopes, which must be granted, and for https://www.googleapis.com/auth/drive, admin.directory.user, admin.directory.group and the distinct admin.directory.group.member write scope, which the store must refuse; a credential the store grants any of them is refused (CREDENTIAL_REFUSED). It cannot prove the delegation grants nothing else (a mail scope, say), and the page says so to the admin who registers it. The audit rows of add, update and rotate record the configured subjects as keyed hashes (an HMAC under the root secret, as SCIM’s audit rows hash a subject, so a list of the org’s addresses does not reverse them). Audit rows also record the scopes granted and the number of members resolved: the audit trail outlives erasure and keeps no crawled person’s address.

Registration, update and rotation require a current org administrator who also administers the source namespace. Each approval binds the signing-key lifecycle, canonical registrant, exact selectors, items, credential hash, verified Workspace customer and source generation. Workers recheck that authority after provider and model waits and under the publication leases. An indexed read uses the same local approval and exclusion fence without opening a credential or contacting Google. Only a complete union listing stamped with both the source generation and current exclusion epoch admits copies; updating a registration cannot reuse its old listing. A check reports delegation standing without document titles or counts.

knowledge.source.update.crawl_readmit explicitly readmits named, Directory-verified people after SCIM reactivation. It requires the same current org and namespace administration, refuses a still-deactivated contact and advances the durable refusal epoch. It never republishes old copies: a complete subsequent crawl must establish their permissions and freshness. Ordinary selector edits do not readmit anyone. A dry run writes no registry or approval; an org whose refusal key has not been initialized must complete its first real registration before a dry-run probe.

Access follows the source

A source’s access:

  • namespace: every reader of the namespace reads every document, as today. The only mode for a shared source of a store whose permissions the sync does not read (notion, recording), and the default for shared gdrive and linear sources.
  • mirrored: a document is shown only to a reader the store says may read it. Required for every crawl and every connected source.

The model has four parts: mirrored documents live in a companion namespace no grant opens; each document carries its permission list; every read through that namespace checks the list and fails closed; and every complete listing reads the lists again. One rule holds them together: nothing derived from a mirrored document (its address, title, text, counts or existence) enters a Loom record outside the mirror or an answer to a caller. An op that is not a checked leg (a source’s probe, status or remove, an erasure’s receipt) therefore answers codes for what a mirror holds, never a count, title, file id or address, and a code that depends on the mirror’s rows (a residue failure) reads as the same code for any cause. The exceptions are these:

  1. A checked leg’s answer to its reader, every field of which depends, of the mirror, only on the items that reader was admitted to, and otherwise only on the request and on whether the leg’s reads succeeded.
  2. The rows a marked request leaves, and every read of them (its own answer falls under exception 1, so a marked memory.ask still answers its synth_* fields to its caller):
    • Its spend (input, output and embedding tokens, and cost) depends only on what exception 1 lets the answer depend on: no model or embedding call runs, or is chosen, on a mirror candidate before the check. It counts in its namespace’s totals and cap. Except to a verified platform admin without namespace-role restrictions, it shows only in the per-namespace-day total and in cost.budget_status’s live namespace or org total, and every other cost.summary dimension leaves it out. The fold carries ops, those tokens and cost only, and leaves marked requests out of errors and the latency figures. A viewer can still difference a live or daily total around another person’s marked request and so learn that request’s spend; that residual is accepted. Named-user caps also count marked spend. Their public status exposes the cap and whether the user or namespace ceiling is reached, but exact user spend and percentage are null except to that operator. User warnings expose the selected user, cap, warning share and threshold crossing, with exact spend withheld from every recipient, including an operator’s configured inbox or webhook. Namespace warnings retain the allowed namespace total. These policy signals expose bounds; an admin changing caps can infer tighter bounds. That residual is accepted, not a claim of full noninterference or a grant to marked work, models or outcome fields.
    • Its work figures (statements, rows and bytes read, chunks injected, llm_calls, response_tokens, the stored duration), its model and the outcome fields a model call can change (success, error_code, status_class) go to a platform admin only, and the error-rate and model metrics leave marked requests out. The stored argument map keeps only numeric statement/read/injection work fields. Result identifiers, document tags, argument digests and error prose are omitted.
  3. A connected source’s per-item and per-file codes and too_many_documents, to its owner, and its listing age, to its owner as a code (fresh or stale).
  4. A platform admin’s view of a mirror’s sources (every mirrored source’s status, the probe’s sampled items and their codes, and the oldest listing start), its sync and spend, and its erasure counts, as codes and counts, through loom doctor, knowledge.source.inspect and the erasure ops.

The rule covers Loom’s records, application logs, spans, metrics and caller-visible answers. Database-native query, error, mutation, text, profile and trace logs remain privileged deployment infrastructure, under the same trust as the plaintext tables and backups. They can retain mirrored addresses, counts and values quoted in errors after erasure; the database operator controls their access and retention, and any export of them keeps that privileged boundary. This is the existing database-log boundary and erasure limit. The SQL surface and analyst must refuse access to these logs. Loom’s statement-capture files, provider and extractor telemetry, and caller-readable exports remain subject to the mirror rule.

A deployment with auth off (LOOM_AUTH_REQUIRED=false) has no reader identity to check. It refuses to add a mirrored source with AUTH_REQUIRED, which wins over MIRRORED_SOURCES_OFF, the answer of every deployment until LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED is set. That setting turned off again once mirrors exist makes the mirror legs answer nothing and stops the sync apart from the cleanup of a stopped source. Auth turned off once mirrors exist does the same, and the gate still refuses every mirror name; the cleanup only deletes.

The mirror

A mirrored document is not stored in the source’s namespace but in that namespace’s mirror: a namespace named acl-<24 random hex>, made under the namespace’s source lease with its first mirrored source and recorded in loom.knowledge_mirrors (org, namespace, mirror). The gate admits no token to a name of that form (acl-[0-9a-f]{24}): check_authorized and CachedGate refuse the name before any other rule, the auth-off return included, and namespace.register and namespace.restore refuse the form. erasure.request refuses an acl- name in its scope for every caller but a platform admin with auth on, and erasure.export for every caller; with auth off both refuse it for everyone. So no token searches, reads or exports a mirror, and the threat model’s rows hold for every token. A mirror has no directory row and appears in no namespace listing or usage view; a deleted name keeps its mirror and counts as in use (namespace_census.in_use, with the same code as any other reason, whatever the mirror holds) until a namespace erasure, or the mirror-erase command, writes its loom.knowledge_mirrors row deleted. A namespace erasure does that last, after the mirror’s rows (the row holds names only), so an erased name can be registered again and gets a new mirror. A name that has ever had a mirror is always erased as a purge, a new owner’s erasure included, since the registry keeps every mirror it had. The migration that enables the refusal first looks for namespaces of the reserved form, registered or holding rows, and refuses to run, naming them, if it finds any.

A knowledge read taught the mirror adds a leg over the namespace’s mirror, with the check below, while the namespace’s source list holds an active mirrored source (a fact that list shows every viewer); a mirror with no active mirrored source in that list runs no leg. Every other read path reads the namespace and finds nothing of the mirror. So a new op, a context-pack leg or an export that nobody taught the check shows no mirrored document; an N-1 replica has no mirror leg, and the rollback procedure keeps it off a mirror that holds rows. A permission column on loom.documents read by every path would make the opposite the default; sharing rejected that for collections, since one missed path leaks.

Search discovers chunk keys and scores without returning chunk text. Each leg checks candidates in batches of at most 64 until its admitted pool fills or its stream ends; denied candidates consume no pool position or fusion rank. Content reads bind the exact admitted namespace, address, chunk index, version and revision under the requested embedding model. A missing key is omitted without substituting a later version. Duplicate or unexpected rows discard all mirrored legs. Permissions are checked again after content is fetched. Listing evidence and any group membership relied on must still be fresh before the content leaves the checked leg. Independent email, owner, domain or public grants can admit a document whose group evidence expired.

A marked private-first search gathers both possible passes from its authorized namespace list and visibility settings before choosing which to return. It refreshes all retained mirror permissions and reranks before applying the first-pass similarity threshold. A provisional private hit that loses permission therefore cannot suppress the public fallback. No reranking provider receives the provisional pools.

  • Legs with the check: knowledge.search, get, outline, list, download, synthesize, knowledge.inspect (titles), knowledge.source.inspect (a mirrored source’s documents), the search.query knowledge and repository-docs legs, code.context’s docs section, context.pack, memory.stack, memory.ask and the assistant (which calls the same handlers under its answer token, delegated from the caller’s). A get, outline or download of a mirrored address the reader may not see answers found=false, as for an absent one.

  • Nothing to read: decision mining and dataset extraction skip mirrored documents, since a decision row is a memory every reader of the namespace reads and a dataset is read by knowledge.query without the check.

  • A request that touches a mirror is marked. Nothing a marked request writes outside the mirror depends on what the mirror holds, except as exception 2 allows: no cache shared across readers (the repository-docs listing cache, memory.ask’s precedents, the provider response and embedding caches) is read or written with text a mirror leg gave, and no field or row of its agent_events, trace, exposure, metric or span records, of recall credit (it writes none to loom.memory_usage_events), nor a generation’s captured text, depends on the mirror. Every request that runs a mirror leg is marked, whether or not anything was admitted, since marking on admission would tell existence. The costs are accepted:

    • knowledge.reads names no address a marked request read, a namespace document’s included;
    • every marked knowledge read drops out of cost.summary by user_id, operation and agent;
    • a marked request writes no recall credit or exposure row, so the usage signals (recall_count, last_recalled_at, the usage totals) leave out every marked read;
    • the assistant’s support check reads a marked call as unknown, so a memory it cites from one gets the reason that the check could not run (unchecked), not unsupported;
    • the assistant’s step list cannot say a marked tool call failed;
    • every session whose request runs a mirror leg is flagged, and a prompt-recall context.pack runs one, so while a namespace lists an active mirrored source every Loom hook of a session over it stops after its first recall, reads included, and a dashboard view of a flagged answer sends no ui.vitals (the session rule).

    The assistant front end’s token record of a marked answer is a spend row under exception 2 that only the deployment’s operator reads; the conversation itself lives with the caller (the dashboard’s browser storage, or the message list an SDK or HTTP caller sends).

  • The flag. The answer of every marked request carries mirrored: true as its first field or line, whether or not anything was admitted, for the same reason as the mark, and each admitted item carries it again, so a truncated copy still carries it. Every assistant answer on a namespace with an active mirrored source (the assistant.ask op and the chat route, POST /v1/assistant/chat/completions, which carries the flag in its stream’s first event or its response’s first field) is marked and flagged, since its answer is model text written from tool results and need not carry the field the tools answered. A mirror leg answers nothing to a client that does not declare it honours the flag. Loom’s plugins and CLI declare it, and so do the dashboard, which shows the document, and the assistant, whose answer token declares the flag only when its caller’s request did: a caller that does not declare gets no mirrored text through the assistant, and the assistant writes nothing from that turn into anything it shares. An SDK declares it only when the app built on it opts in.

  • The session rule keeps a session that saw a mirrored answer from sending it on (the session rule).

  • The sync and the workers write nothing outside the mirror. The mirrored sync’s spend charges no namespace’s cap, and LOOM_KNOWLEDGE_SOURCE_MIRROR_DOCUMENTS_PER_PASS bounds it. A mirrored source’s sync state, the sync’s audit rows, error text and per-file wait state are filed under the mirror’s namespace and go with it when it is erased; whatever of a mirror leaves it (a source’s status, a worker’s pass notes, logs, spans and metrics, the extractor service’s logs and traces) carries codes only (exceptions 3 and 4 apart). Worker pass summaries exclude mirrored turns’ completed-source counts, document figures and cut flags. A failed mirrored turn contributes only SYNC_FAILED, for every cause; returned turn details remain private. Mirrored per-file failures emit no log record, since the number of records would disclose a document figure. A sync turn enters its private diagnostic scope before taking its source lease or listing files; retirement enters it before reading its queue. That scope suppresses client and provider instrumentation, database work figures, shared metric calls and log records, including their volume. Provider health and embedding caches receive no private contribution. The extractor receives a generic filename; its service emits no per-request access, stderr or fault logs. A pass that touched private work records codes without duration, outcome figures or failure counters, including when invoked from the CLI. Its retry history cannot advance a later ordinary pass’s diagnostic streak. Ordinary work keeps its own cost accumulator; private spend is written directly under the mirror, without entering the shared event queue or its statistics.

  • No scope reaches a mirror. Every statement a request runs that reads a table holding mirror rows other than by a namespace the caller names leaves out the acl-[0-9a-f]{24} form; the workers named here are the exception, since they must reach the mirror’s rows. resolve_org_scope (its auth-off filter included), every_namespace_scope and rollup_member_namespaces (server/auth/context.py) leave it out for a request’s caller; the sync, the erasure saga, the mirror-erase command (which finds mirrors by the acl- form in every store, not only by their registry rows, so a restore that loses a loom.knowledge_mirrors row orphans nothing) and the maintenance workers (reindex and embedding-model rebuilds, tier and retention moves, the blob and originals sweeps, which keep a mirror’s rows referenced) reach a mirror through the stores, never the gate, under the rule of the bullet “the sync and the workers write nothing outside the mirror”. Every sql_v1_ view (each replaced by a migration), the analytics export’s statements and the org-wide reads of erasure.export (Shown.clause carries the form) and of a user_id erasure’s steps and census leave it out too; a saga whose scope holds only acl- names is not found for anyone but a platform admin; every scope a saga answers to anyone else, and every audit row a saga writes, leaves acl- names out (a mirror-only saga’s audit rows show only that a platform admin erased something); so audit.list, the insights, usage and cost reads, the SQL surface and a person’s export name no mirror. check_authorized refuses the form before its auth-off return, so no id-addressed check (row_reach) or audit.list of a named namespace opens one. The deployment-wide admin reads (/v1/admin/, outside the gate) leave a mirror’s ledger rows out whole: until the fold below carries a marked request’s spend, a mirror’s spend is in no fleet or consumer total.

  • Private events have separate physical storage. Ordinary events stay in agent_events; marked requests and mirror workers write agent_events_private. The ordinary table’s CHECK excludes marked rows and mirror namespaces. An analyst keeps ordinary signed, namespace-scoped reads and cannot read the private table or its physical metadata. Provisioning verifies the CHECK on every writer, drains the finite set of prepared INSERTs and refuses any existing private contamination before granting access. Checked per-writer HTTP sessions read the native access catalog and prove that policies cannot hide the process or event populations. Unknown visibility, writer routes or metadata refuse. It preserves existing rows for operator resolution. Administrative application uses loom ch-setup --apply-analyst-sql; unchecked analyst SQL files refuse.

  • Private event reads are folded and masked. The analytics export and sql_v1_usage_daily union both stores and fold marked spend into a namespace/day row with agent and operation -. sql_v1_agent_events and the signed analysis_agent_events view mask every field that can depend on an admitted item before predicates run: model, tokens, cost, work, llm_calls, success, error code and status class. Trusted budget and history-policy reads cover both streams too. User erasure scrubs both stores under agent_events.scrub; the private store has the ordinary ledger’s retention and tiering policy. The storage cutover preserves ordinary analysis across builds. The assistant boundary describes the view and key-hiding requirements. Exception 2 accepts this fold beside cost.summary’s.

History predicates cannot use a marked call’s outcome or projected arguments. A potentially matching marked event in a rule’s identity, operation and time scope makes that leaf unknown before success or argument filters run. This includes an unresolved argument reference and a negated condition. Breadth rules check marked participation under the same principal pin because marked reads write no exposure row. The compiler reads coarse participation and the ordinary aggregate in one statement. Protected unknown carries one fixed reason and no count or timestamp; every affected enforcing operation refuses, reads included. Shadow rules retain their non-enforcing behavior. policy.check uses the same protected-history decision. Ordinary trace-backend errors keep their existing read-availability policy.

A checked answer retains its access proofs through required policy witnesses and other work that must finish before publication. All provider work for its complete ordinary alternative finishes before cost and event snapshots. After required tail work, publication renews the exact private dependencies of the already selected frame and checks every retained deadline after the last renewal await. This boundary does not render, rerank, call a provider or choose an ordinary alternative. A lost proof refuses the whole frame; removing a citation from private prose is insufficient. After awaiting the response headers, publication renews access again before sending the body. A checked-proof or publication failure after headers aborts with a constant failure and sends no unchecked body. Discarded candidate widths and optional sections retain no publication guard; returned metadata, counts and cursor lookahead retain their contributing proofs. A raw download owns the same required marked accounting attempt as a registered operation. Cancellation and disconnects still propagate through owned cleanup. These checks end at the server’s response-send boundary.

The assistant’s incremental stream has a terminal accounting boundary because final usage is unavailable before its early text. Its server-owned first event is data: {"mirrored":true} when marked. Partial text remains provisional until the response closes and joins its producers, captures final usage and confirms its required private cost row before [DONE] or a terminal error. Disconnect cleanup retains and joins that required delivery. Whole answers and checked source frames keep the accounting-before-publication boundary above. The answer token’s distinct answer_mirrors bit fixes mirror admission before model work, so a source activated during a negative answer cannot introduce an unmarked tool leg. Missing admission is false; ordinary delegation drops it. See the assistant boundary.

A multi-entry MCP batch refuses every response-bearing marked entry with the same instruction to send it individually, including denied and empty entries. Notifications remain silent. It discards each marked entry’s complete frame before buffering it or awaiting another entry. Single calls and one-entry arrays retain checked reads. Each entry gets fresh local state before routing or screening; the enclosing HTTP request keeps a monotonic mark through later ordinary entries, streaming and cleanup. Middleware task boundaries do not clear that owner.

A mirrored source’s documents are never kept past it: remove retires every one it owns (delete_documents is implied), and so do a disconnect, a SCIM deactivation, the owner check and every retirement of a namespace’s sources. Each of them marks the source inactive (in the request, or in the turn for the owner check) and leaves the deleting to the sync worker, so no request’s work depends on how much the mirror holds. The worker retires a stopped source’s documents by its source-id: tag outside the content bound, with a per-pass bound of its own, so a large removal does not stall other sources’ content. In a mirror, ownership and every per-source document read go by the source-id:<id> tag, not by name. The retention overrides give each mirror its namespace’s retention_days and warm_days.

Stopped-source deletion has a durable source-id queue in code_index_state under the mirror, with keys knowledge_source_retire:<source id>. A queue row contains only the original namespace, source name, kind and registration id. The request writes it under the sources lease before inactivating the source. A replacement registration can reuse the name while the old queue row and copies remain; cleanup uses the row’s mirror and source-id ownership, independently of the current mirror registry.

The knowledge_source_retire worker runs even with authentication or syncing off. It shares the sync cadence, with its own deadline, cursor and LOOM_KNOWLEDGE_SOURCE_RETIRE_DOCUMENTS_PER_PASS bound. Its cursor contains registration metadata only. A queued source still active under the same id is left alone, so a crash between enqueue and inactivation deletes nothing. Completion requires both no live owned documents and no pending or parked publication while holding the source’s turn lease. Clearing the queue writes an empty version row; erasure removes it with the mirror’s other state. Cleanup exports no document counts or cut flags.

A namespace source and a mirrored source that reach the same file own two documents, one in the namespace and one in its mirror. The namespace copy is readable by every reader, because an admin registered a source that makes it so; every leg answers the address once, dropping duplicates after the check. When both copies matched and the reader may see the namespace copy, the hit is the namespace copy’s, with its own chunks and scores and no item flag. When only a mirror copy the reader is admitted to matched, the hit is that copy’s, with the item flag. Otherwise the mirror copy is dropped.

The session rule

A session-content op records a session’s text, or a figure derived from it, that a hook or client sends without the agent choosing it; an op the agent calls with text it chose is the reader’s own write. Each op declares its class in the op registry (server/operations/registry.py), and an op both a hook and an agent call is session content. They include conversation.append (which the CLI’s ingest-transcript calls), conversation.save_message, the title and metadata of conversation.create and conversation.update, memory.set_from_messages, session.brief_save, observe.record, observe.crystallize, working_memory.set, working_memory.append, run.begin, run.complete, code.outcome, cost.record_turns and ui.vitals, and every read a hook sends with a session’s prompt, files or transcript (context.pack, code.search, code.impact, memory.for_file, namespace.validate). Framework callbacks also send the latest user message through memory.search and persist conversation items through memory.set, so both are session content. Its coalesced delivery through memory.bulk_set has the same class. Automatic tool interception also sends code.test_gate, code.symbol, code.enclosing and tracker.duplicate_check with session-derived inputs. The rule decides; the registry’s classes and a test over what the shipped hooks send pin the full set.

Once a session has seen the flag, no Loom hook sends any op from it, reads included, and no client sends a session-content op on its own; the calls the agent makes go on, dual-use ops (code.search, context.pack) included. A session is also flagged when a subagent or a child process it started saw the flag: the child records it in the session state it shares with the parent, which the parent’s hooks read, since the child’s final text, reworded, need not carry mirrored: true. A session resumed, forked or compacted from a flagged one is flagged, and so is one started from a flagged session’s process (claude -p or codex exec run through its shell): the CLI and the hooks set an inherited marker in the environment and read it, and keep the flag in the session’s own state beside what they keep for resume and fork. A worker an orchestrator starts outside that process tree (through tmux, ssh or a job queue) inherits nothing; what its own session records is the orchestrating agent’s choice, out of scope (threat model). The CLI’s ingest-transcript refuses a transcript that carries the flag, or whose session’s state records it.

The refusal: from the first time LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED is set until it is off again and the mirror-erase command has run, every session-content op refuses a client that does not declare the flag, in every org, whatever mix of clients and tokens a session runs; the declaration is the client’s promise, carried by the transport and never by an op argument a model fills in. The exact transport declaration is one X-Loom-Session-Protection: v1 header; missing, repeated or unknown values declare nothing. An assistant answer token carries its caller’s declaration in the Boolean session_protection claim the bridge fixes at delegation. An answer or its descendant cannot replace that claim with its own header, and ordinary delegation does not inherit the claim. A client that does not make it (a third-party MCP client connected straight to Loom) is refused like any other. The condition is the one row of loom.knowledge_mirror_latch or the process’s own switch: a process refuses while its switch is on, row or no row, and writes the row whenever it reads the switch on and finds it absent, so neither a lost write nor a restore of a copy without it lifts the refusal. Only the mirror-erase command deletes the row, last, once no namespace of the acl- form holds rows in any store, and it refuses while any live process reports the switch on. The check fails closed: a process that cannot read the row refuses, and the first mirrored add waits until every live process reports the switch on, as the schema check does. Since the refusal lifts only then, it says nothing of when another org’s mirrors went.

A process with the switch on confirms durable protection before startup can admit clients or start workers. Its existing instance heartbeat reasserts that state, so a lost row is repaired without waiting for a session-content call. Startup refuses an unconfirmed write. The common gate runs after token verification. HTTP checks before reading the operation body; MCP checks after resolving the operation and before content validation, identity defaults, history, replay or implicit session runs. Its fixed SESSION_PROTECTION_REQUIRED refusal is HTTP 409 and is not retryable without changing the client or completing the documented cleanup.

The latch is one fixed id = 0 KeeperMap row on the default Keeper ensemble, with set_at and no TTL, tenant identity or content. It is deployment protection, so tenant erasure never removes it. A fresh negative check uses a checked HTTP session under the service principal. It validates the local latch and lease engines, primes the latch, and brackets an acknowledged nonempty strict write to a unique expiring lease key and the latch read with the same live Keeper client identity. Every proof SELECT returns its execution’s server UUID, including an empty latch, and the barrier INSERT computes the writer’s UUID into its own lease row. The final exact holder and write UUID must match. This matters because a retried initial HTTP session creation can leave the same session name on two backends; the name alone does not pin execution. Query caches are disabled within this protocol. Any missing metadata, changed identity, uncertain write, expired nonce or cleanup failure refuses the negative decision. Only its exact own lease is disposable; a cleanup fault leaves an expiring row and never admits a request. A positive repair computes its fixed row key in the INSERT from a check of the writing server’s UUID against the validated backend. A differently routed write refuses; an acknowledged strict insert or duplicate is followed by a fresh validated positive read. No negative result is cached. The named-session transport alone is not a Keeper proof.

Mirrored sources are off for a deployment until an operator turns them on, after every Loom client and hook that sends session content honours the flag. Each pull request that adds a mirror leg, writer, client or hook lists what it writes and tests each against these rules.

The permission list

Each mirrored document carries its permission list in an access_json column of its row (loom.documents; under the Postgres authority it rides in the head’s row jsonb, with no Postgres migration), written by the turn of the source that owns the document:

  • emails: the people the store grants any role on the whole file, lower-cased, and, by credential mode:
    • connected: the owner, whose account listed the file in that turn, entered as the source’s owner_subject (an owner entry, never an email, and never the connection’s store email), since the owner chose the item and can open it. When the store will not give the file’s list, the list is the owner alone, not unread.
    • crawl: a person the crawl listed the file as, only when they own it or the permissions read as them name them, never from a read by an item or folder id or from a shared-with-me entry, which the store also gives a link-only share someone opened. An unreadable list holds the file’s owner if a crawled person owns it, and is otherwise unread.
    • shared: nobody more; its service account is no reader, so its unreadable lists are unread.
  • groups: group emails, which a crawl’s directory read expands into loom.knowledge_group_members (one row per source and group, with its members and read_at). A connected or shared mirrored source reads no directory and expands no group, so a file shared with its readers only through a group shows only to the people whose own account listed it.
  • domains and anyone: a domain or public permission the store marks discoverable. A link-only share (one the store marks not discoverable) is left out: it lets someone who holds the link open the file, not find it.
  • unread: no current list is known; such a list carries no owner, emails, groups, domains or anyone, and admits nobody.

For gdrive the list is the file’s permissions, read with the listing, or with permissions.list when the listing leaves it out; for linear, the active members of every team the project belongs to, a member of unknown state admitting nobody; for a connected notion source, the owner alone, so only the owner sees a page. The list lives and dies with its document: a retirement’s tombstone carries no list, and no read op returns the column.

Who may read

A reader sees a mirrored document when all of these hold:

  1. The reader holds viewer on the source’s namespace (the ordinary gate).
  2. The document’s owning source (the registry row whose source_id matches its source-id tag) is active and has a last complete listing that started within LOOM_KNOWLEDGE_SOURCE_ACL_STALE_SECONDS (default 3600) of now, read from its row in loom.knowledge_source_listings.
  3. The person the token reads as (below) is the list’s owner entry, compared through canonical_subject, or one of that person’s verified emails is in the list’s emails, or in the members of one of its groups from a members row of the document’s owning source read within LOOM_KNOWLEDGE_SOURCE_ACL_STALE_SECONDS (by its source-id tag), or that email’s domain is in its domains, or its anyone is set.

The reader’s verified emails are its subject’s directory emails that VERIFIED_EMAIL_SQL admits (an SSO account’s or a linked one’s, server/core/grant_sql.py), and nothing else: a subject that merely spells an address is not one, and neither is a connection’s store email. The predicate’s passwordless arm covers nonempty contacts from verified SSO provisioning or operator linking. Password signup hashes the supplied password before storing its claimed address; ordinary namespace registration cannot write a contact address or password hash. The internal reader resolver retains the uncached canonical subject independently of verified emails and reads actual active contact rows for email proof.

Signed hosted-domain evidence lives in sso_links.hosted_domain and issuer, written after the final SCIM check under the account ownership lock. A domain matches only through a current linked row naming the active passwordless account’s contact, with the configured issuer and its signed hosted-domain claim. Empty defaults, missing claims, issuer changes and generic link changes deny domain access. Neither a linked password account’s email nor a bare verified address proves domain membership. These identity inputs alone expose no mirrored document. A request keeps a monotonic mirror mark in its existing work bag, shared with child legs. An unbound direct helper serves no mirrored content. Active registrations, mirror bindings and listing starts are read in sequence before candidate permissions. A failed unconditional read marks mirror_unread; a failed candidate read invalidates the whole mirror plan without that marker. Checked adapters cover search, get, outline, download and listing. Other readers, marked-request sinks and session protections remain prerequisites to enabling mirrored sources.

documents.access_keys and its bloom index select coarse candidates under the ClickHouse authority. Owner and group buckets retain candidates needing identity resolution; Unicode buckets retain matches whose database and Python normalization can differ. Every candidate still needs the current strict permission check. Under the Postgres authority these keys do not filter candidates: an unpublished head may widen permission. Both chunk-frequency views exclude mirrors, and a lexical-statistics rebuild refuses their names before reading or writing statistics. Mirrored lexical scores use fixed corpus constants.

The checked list projection reads title, description, tags, blob metadata and the stored preview only at an admitted live document version and revision. Under Postgres, the current head supplies these fields and computes only the first 220 content bytes in the database, including for encoded text; no full body crosses into the list reader. Chunk counts belong to that same storage copy and revision. They count distinct embedding-model and section-index pairs, including sections awaiting embeddings. A Postgres count also requires the head’s recorded chunk version; a current revision without recorded chunks counts zero. Permission and serving-state proofs must remain current through the list’s final read before any count or lookahead is returned. Metadata discovery groups complete public address families before a public cursor or limit. Postgres reads internal pages of 64 rows, releasing the connection before permission checks; it reads every page without a candidate cutoff. Ordinary shadow checks read bounded batches independently of public filters and cursors. They cover the ordinary listing’s published rows and its unpublished Postgres heads; requesting superseded rows also lets an ordinary tombstone shadow a mirrored copy. Mirrored candidates remain live-only. Listing chooses the newest admitted complete copy of a public address, with a stable storage-identity tie break, before applying public tags, URI prefixes or the cursor. An ordinary copy wins its entire family before these filters. Selection retains only one current family, bounded shadow batches and the requested page plus one lookahead. After selected projections and counts are read, ordinary shadows, permissions, identity, document versions and recorded chunk serving state are checked again for every retained copy, including the lookahead. A lost proof drops the whole mirrored leg, so it cannot leave a stale cursor or reveal a removed row through the page’s continuation.

A token reads as one person, for both the owner entry and the emails, or as nobody:

  • a person’s own session token (a primary token whose own sid claim names a sign-in session, with no creator claim) reads as that person; a token minted for a subject without one reads as nobody;
  • a token minted for one person alone that says so (a reads_as_creator claim, which only the person’s own primary first-party sign-in session can mint through POST /v1/me/personal_tokens.create, for the dashboard’s own-agent connection flow; delegation never copies it) reads as its creator;
  • the assistant’s answer token, delegated from the caller’s for that answer alone, carries a reads-as claim that the bridge sets at delegation to the person the caller’s token reads as, or to nobody, and reads as that; the check never infers it from the token’s jti or consumer, and no token delegated from it reads as anyone;
  • every other token (a service account made for tools, CI or a team, a bot, an end-user token, whatever their creator) and a request sent on behalf of another person (X-Loom-On-Behalf-Of) reads as nobody, so it sees only anyone documents.

The check fails closed. A stale listing, an unread list, an unexpanded or stale group, a link-only permission, a permission the store marks deleted (its address may now belong to someone else), a grant to one view of the file only (view), and a permission type or role the sync does not know admit nobody. A missing registry or listing row admits nobody. A leg whose per-request reads (the statements it issues whatever matched, the reader’s canonical subject and verified emails among them) fail shows no mirrored document and marks the answer degraded: mirror_unread; a read it issues only for candidates (the lists, the members rows, the heads) that fails drops every mirrored result without the marker, so the marker says nothing about what matched. Each leg reads the registry row and the listing row first and then the members rows and the lists, under sequential_reads(), so a lagging replica cannot pair a fresh listing row with an older list. It computes the addresses the reader may see before its LIMIT, so a page is never cut short by the check, and caches no admitted set. No score depends on a document the reader may not see: a mirror’s chunks add nothing to any corpus statistic (the lexical leg’s document counts, lengths and term frequencies), and a mirror leg scores its lexical hits with fixed corpus constants, which depend on no document, so a namespace fed only by mirrored sources still ranks by both legs. Under the Postgres authority each leg judges the addresses it is about to answer by their heads, so a narrowing or a retirement committed and not yet published already applies, and a missing head drops the address; a leg that cannot read the heads shows no mirrored document. The implementing pull request measures the leg on a mirror of a million documents under both storage authorities and puts the figures here.

Every complete listing reads the lists again

Every complete listing of a mirrored source reads every file, metadata only (id, revision, permissions): the store does not change a file’s modification time when its sharing changes, so an incremental listing would miss a permission removed. A listing that reads a list unlike the stored one rewrites the document’s row with the same content and the new list, as knowledge.update_metadata without reindex does, so no chunk or embedding is rebuilt; an unchanged list costs no write. A stored copy behind its file’s revision (a fetch deferred, given up or over the size bound) takes only the entries both its stored list and the new one hold, since its text was shared under that list (entry by entry, so a person the stored list admitted only through a domain or anyone is dropped, which fails closed); the content write that brings it up to date carries the wider one. A behind copy that once reads unread stays so until that write. A stored document the complete listing no longer has is rewritten unread, with every entry cleared, then retired on positive evidence, as today. The listing row’s start lands only once every changed list and every such unread rewrite of that listing is written, so a row is never newer than the lists it vouches for. These rewrites are outside the pass’s document cap; the permission reads they need stop at half of what is left of the listings’ share when a turn starts them, so the turn’s new and changed files still get time. Content is fetched only for a file whose revision changed, and a content write carries the list of the latest listing that read the file, so a fetch the pass bound deferred never writes an older list; a listing row lands only once every document written during that listing carries a list it read.

A pass runs the listings first, rotating across the mirrored sources, and gives them at most a third of the time the pass starts work in (a quarter of knowledge_source_sync_max_pass_seconds) and their own document bound, while the namespace sources then have the rest of the pass under theirs. Mirrored files consume none of the namespace sources’ document bound. Their earlier work can reduce the time left for namespace writes, so namespace progress can vary under the threat model’s timing exclusion. A listing that does not finish goes on from its cursor in the next pass, and until it completes the source’s listing row keeps its older start, so a large source needs a stale bound longer than the time to list it all. inspect and loom doctor report each source’s oldest listing start, a mirrored source’s to a platform admin only (its connected owner sees fresh or stale). A stopped sync hides every mirrored document once the stale bound passes. The implementing pull request measures the writes of a crawl over 2000 people and puts the figures here.

How lists from several accounts of one source, a person a crawl stops reading, a stopped source’s rows, and restores of either store are handled is the implementing pull request’s to specify against these four parts; each must leave no list or row that admits someone the store no longer names.

Threat model

Threat Boundary Enforced by
A connected person leaks a private file into a team namespace Every connected source is mirrored; its documents live in the mirror, and a teammate sees one only when the store names them. access cannot change. add and update; the mirror gate; test_document_source_leakage.py
A crawl’s registrant reads crawled people’s private files No rule admits a source’s registrant or owner as such; a registrant reads a file only when the store names them or their own connected account lists it. the access check
An admin points someone’s connected account at files they did not choose Only a connected source’s owner adds its items or changes include. update
A link-only share becomes a search hit for everyone A permission the store marks not discoverable admits nobody. the access check
A person stops reading a file and Loom keeps showing it (stale ACL) Every complete listing of a mirrored source reads every file’s list and rewrites a changed one; a source whose last complete listing is older than the stale bound admits nobody. the sync’s mirrored listing; the access check
A person leaves a group and keeps reading through it (group churn) Groups are read with every complete listing into a members row with its read_at; a stale or unread group admits nobody. the access check
Auth is switched off after mirrors exist The acl- refusal runs before the auth-off return; mirror legs answer nothing and the sync skips mirrored sources while auth is off. check_authorized; the mirror legs
A person leaves the org and their connected sources keep syncing SCIM deactivation and disconnect stop the sources and clear the credential, then revoke it at the store; each turn’s owner check skips a source whose owner is revoked, deactivated or no longer a writer. server/auth/scim.py; the sync’s turn
A crawl reads more than its registrant meant, or can write An org-wide role to register, update or rotate; the probe requires the store to refuse a write scope; subjects and scopes in the audit row. the crawl probe; loom.audit_log
An id-addressed read confirms a mirrored document exists get, outline, download and the inspects answer found=false or leave out an address the reader may not see. the mirror legs
A viewer or admin learns what a mirror holds from an op that is not a checked leg A source’s probe, status and remove, and an erasure’s receipt, answer codes for a mirror; its counts go to a platform admin only. the source ops; the erasure handlers
A new read path shows mirrored documents to everyone Mirrored documents live in a namespace no grant opens and no listing names; a path that does not read the mirror shows none, and no org-wide scope names an acl- namespace. check_authorized (acl-[0-9a-f]{24}); the scope helpers (the mirror)
An older build shows mirrored documents N-1 has no mirror leg, but its org-wide reads return acl- rows and so hand out a mirror’s name, and its gate admits a wildcard admin token that names it. So no N-1 may serve while a mirror holds rows: the new modes wait for N everywhere, the rollback procedure erases the mirrors with every replica stopped first, and N reports an N-1 heartbeat while a mirror holds rows. An N-1 replica started without that procedure (a bare helm rollback) exposes mirror text to an org admin; that residual rests on the procedure. the rollback procedure; loom doctor
A mirrored document’s text reaches every reader through a derived row Decision mining and dataset extraction skip mirrored documents; a request that touches a mirror writes nothing derived from it outside the mirror. the sync; the mark
A store account is linked into another person’s Loom account, either way The connection is stored only from the browser that started it, with that subject’s own dashboard session and the state cookie. A connection’s email is never a reader identity. the connect routes
A caller injects store query syntax through a filter Filters are fields with patterns, escaped and built by the server; no raw query is taken. server/clients/gdrive.py
A stolen connected credential changes the store Only read scopes are accepted; the credential is sealed and never returned. the connect callback; server/core/secrets_at_rest.py

Out of scope: response timing, and when a pass reaches the namespace sources, which the mirrored sources’ turns before them delay; a resource limit (memory, rows read) that a mirror’s size can trip on a per-request statement, which like timing depends on how much the mirror holds; how fast the shared pass gets to a connected owner’s own source, which its fresh or stale code and per-item status reflect; that a namespace erasure is still retrying, which a namespace admin sees whatever its cause; a worker an orchestrating agent starts outside a flagged session’s process tree; a person who hands their own reads_as_creator token to a team bot or a CI job, which then reads as them; a session still open from before an operator lifted the refusal that saw mirrored text then; a reader who already saw a document and kept its text in a record only they read, or one their agent chose to write or sent in a call it chose (a follow-up memory.search whose terms come from what it read, recorded in a trace log the namespace’s viewers read), which is that reader’s write; that a person able to add a file where a mirrored source lists can tell, by the file never appearing, that the source is over LOOM_KNOWLEDGE_SOURCE_MAX_DOCUMENTS; the store itself reporting a wrong permission list.

Meetings through the shared-folder store

Meet transcripts and notes are documents in Drive. A gdrive source reads them as files: a meet_recordings item finds the recordings folder in an account’s own Drive, and a folder item reads a supplied folder. Folder discovery uses the fixed table of localized names. When no folder is found, the turn reports meet_recordings: not found without naming an account; a folder item with the folder’s link reads it instead. Individual document links use the same source.

A transcript Doc in the recognized English export layout is tagged ingested_format: transcript. Its elapsed-time headings and speaker paragraphs are retained unchanged; the document chunker uses those headings for timed sections and citation context. The export supplies coarse elapsed points, rather than a start and end time for each speaker turn. Loom does not infer missing intervals or use the meeting-ended footer as an utterance endpoint. Edited, untimed or unrecognized layouts, including meeting notes, remain ordinary Docs with their content preserved.

The source keeps its existing connected-account and mirrored-access checks. A folder name, transcript shape or attendee paragraph does not grant access.

The recording source accepts Zoom only. A stored Meet source remains listed and inspectable, and keeps its configuration and documents. Its check, rotate, sync_now and update operations refuse with BAD_CONFIG and direct the caller to gdrive. Its worker turn records provider_removed, contributes no pass failure and preserves the cursor, pending full request and file waits. Removing the source deletes documents only when requested. No stored configuration is rewritten to a sentinel, and no native Meet API fallback remains.

Project-tracker issues

Feeding a project tracker’s issues and comments into the tracker plane is a design of its own, written in the pull request that builds it, after the items and access pull requests. That design has to answer what this page found:

  • The tracker plane reads a team’s records live and copies none (tracker plane); issues with no team database would be kept by Loom, in its own tables, under the erasure census.
  • The plane’s statements filter by repos (and, in a standard namespace, by is_private = 0), never by org or namespace, so a Loom-kept table needs an (org, tracker namespace) predicate on every read.
  • A source serves only with github_issues, github_prs and edges (record contract); the contract’s source_id names a repository; and tracker namespaces belong to LOOM_TRACKER_ORG alone.
  • An issue of a private team, or read through a person’s connected account, is private (is_private = 1). The contract derives an item’s id from source_id, which changes when an issue moves to another team; the design changes the contract to take the issue’s own UUID, so a moved issue keeps its item.

Surfaces

  • Ops. knowledge.source.add takes items or links, credential_mode and, for crawl, directory_subject and subjects; update, parse_links, connection.list and connection.disconnect are new (a connection starts in the dashboard, POST /v1/connect/<kind>/start); list and inspect show items, modes and, through the check, the documents each owns. A mirrored source shows its document counts to nobody but a platform admin (exception 4), apart from what a checked leg lists to an admitted reader, and a connected source’s items only to its owner (an admin can still remove the source). To a viewer who is neither the owner nor an admin, a mirrored source shows its kind, its owner as “a connected account” for a connected one, its item count and its last sync’s status, not its items, config or created_by; its error text carries codes only, for every viewer. Python and TypeScript SDKs are regenerated.
  • CLI. loom knowledge source add|update|list --namespace <ns> --link <url>... (--credential-file, --connected), loom connect gdrive|notion|linear, loom connection list|disconnect, loom knowledge mirror erase --all (restores and rollback), and loom doctor’s line for the oldest listing start of each mirrored source (for a platform admin; a connected owner sees fresh or stale).
  • Dashboard. The Library’s Sources section: paste many links, a preview of what each resolves to and which could not be read, Check, Add, edit items, the access mode, and each source’s last sync (a mirrored source’s status code only). Settings → Connected accounts: connect, see, disconnect. The repository card keeps its own sources.
  • Agents. Every op reaches an agent through loom_op. The mentor reference gains a line: handed a store link and asked to bring it in, call knowledge.source.update with links on the caller’s connected source for that store, or add one; with no connection, connection.list answers the dashboard link <public base>/settings/connected-accounts?kind=<kind> to hand back.

Data model

ClickHouse, additive migrations:

Table Key Holds
loom.knowledge_sources (columns) (org, namespace, name) source_kind, credential_mode (default shared), access (default namespace), owner_subject, items_json, options_json (a crawl’s directory_subject and subjects), secret_enc
loom.knowledge_mirrors (org, namespace, mirror) deleted (written once a namespace erasure or the mirror-erase command has erased the mirror); at most one live mirror per namespace
loom.knowledge_mirror_latch one row set_at: the deployment once had the switch on; the mirror-erase command deletes it last. It stays in ClickHouse and holds no memory content.
loom.documents (column) as today; in a mirror source_uri is <address>#<source id> access_json: a mirrored document’s permission list (emails, owner, groups, domains, anyone, unread); empty elsewhere; under the Postgres authority it rides in the head’s row jsonb, no migration
loom.knowledge_source_listings (org, namespace, source_id) name, started_at (the start of the source’s last complete listing), version; namespace is the mirror
loom.knowledge_group_members (org, namespace, source_id, group) name, members, read_at, deleted, version; namespace is the mirror
loom.connected_accounts (org, subject, kind) connection_id, account_id, workspace_id, account_email_enc, scopes, credential_enc, active, refresh_id, connected_at, disconnected_at, version

This is an authentication control registry, keyed by canonical sign-in owner across namespaces, like the sign-in and revocation registries. It is not a memory-content row: namespace, agent, end-user user_id and session_id would falsely bind one provider connection to one requesting session or namespace. Source registrations retain their namespace and worker provenance; account changes use the authenticated actor in audit. The explicit owner erasure axis covers sealed identity and credential history rather than guessing a memory user_id from the sign-in subject. Auth invokes a required lifecycle port; the HTTP and CLI composition roots bind its knowledge implementation before accepting mutations. An unbound process refuses retirement rather than skipping it.

The account registry seals its optional display email in account_email_enc. Its workspace_id distinguishes provider workspaces, connection_id fences replacement and disconnection, and refresh_id reserves one rotating refresh. Source rows retain that exact connection id with their canonical owner_subject; created_by remains audit attribution and supplies no credential authority. Indexed mirror discovery retains an exact source and account generation for each connected source. Preparing and refreshing a read plan revalidates the owner’s standing without refreshing a provider token. The existing document permission list still decides which readers may see each copy; a verified reader need not own its source. A replacement source is not added to a retained plan. Worker publication checks owner and source standing again after extraction and embedding, before committing documents or a complete listing.

The browser routes and account operations implement the connection backend. The dashboard connection page, interactive CLI flow and crawl support require their own integration; the backend does not implement those surfaces. Connect routes are excluded from HTTP tracing before request attributes or headers can be captured.

Account restore cleanup discovers raw account and connected-source history, including replaced registrations and source-only orphan owners. It stops every proven random Keeper key under its exact owner lease, queues every historical source id, clears only matching current registrations, and physically deletes account and source history with raw residue verification. Ordinary disconnect still refuses another live generation. The cleanup runs before publication in loom reconcile for either restored store. For a ClickHouse-only deployment, stop every serving process, restore its configured stores, then run python dev/restore_connected_accounts.py and repeat with --apply. The script accepts only the configured deployment and preserves its authority and cell checks; it does not attach cleanup to the generic loader’s arbitrary database or scratch target. Recent worker activity refuses cleanup. Local completion precedes bounded best-effort provider revocation outside leases, and revocation failure cannot restore credentials. Complete the separate mirror-content restore cleanup before serving; account cleanup alone does not prove every mirror document, chunk, head and original was erased.

A source N-1 can operate on (shared, access: namespace, with items today’s config can express) keeps today’s columns as they are: kind, its credential in credential_enc and its items in config_json, which N writes alongside items_json. Every other source (mirrored, connected, crawl, or with an item today’s config cannot express) is stored with kind empty and its kind in source_kind, its credential in secret_enc (purpose knowledge_sources.secret, which loom secrets reseal covers) and its items in items_json, with credential_enc empty and config_json {}. N reads source_kind first and kind when it is empty, and items_json first and config_json when it is empty (rolling upgrades). The mirror’s documents and chunks live in loom.documents and loom.chunks under the mirror’s namespace, so the document machinery (heads under the Postgres authority, deliveries, originals, the blob sweep) applies unchanged. Each new table is placed in storage layout by its placement rule, in the pull request that adds it: loom.knowledge_mirrors and loom.connected_accounts are registrations read by key, so they follow knowledge_sources to Postgres in wave 3; the listing and members rows are read by the mirror legs beside the lists in loom.documents, so they stay in ClickHouse. An implementing pull request that needs another table (for a stopped source’s cleanup, or a crawl’s exclusions) adds it with its placement and erasure classification.

Erasure and audit

What every implementing pull request must hold; each specifies its steps, leases, statements and census keys against these, and its tests pin each one:

  • Every new table is classified in server/erasure/store/census.py, and every table that can hold a person’s address or a document’s text is covered by an erasure.
  • A namespace erasure, namespace.delete, collection.delete and code.repo.remove retire the namespace’s sources first, clearing their credentials, so no turn writes back what the erasure took. A namespace or namespace_list erasure covers each named namespace’s mirrors as more names of the same erasure: a scope that holds a mirror is erased as a purge, in one pass over the named namespaces and their mirrors, since no token reads or restores an acl- name, and the census counts it the same way. The last Loom step, knowledge_mirrors.forget, writes the mirrors deleted, so the name registered again gets a new mirror; the registry keeps them, so a resume or a replay after a restore still reaches them. A user_id erasure touches no mirror row (the sync writes them as -), and its census and export name none.
  • Each namespace step resolves its mirrors once and persists mirror_marked in loom.erasure_requests and, under the Postgres authority, loom.erasure_ledger before counting or deleting that scope. Failure-tail verification does the same; if marking fails it counts nothing. The mark only moves from false to true and survives registry loss, resume, abandonment and restore replay. A restore promotion preserves the receipt, checkpoint time and pending-replay fence. Marked checkpoints use the high bit of their UInt64 version; they supersede stale unmarked writers even after a merge, while timestamps order marked writes.
  • Marked erasure.request replies carry codes only for every caller, since the idempotent answer can be replayed after a role change. erasure.status and erasure.list carry codes only unless the current caller is a flat platform admin; namespace and org admins receive zero counts, empty residue, no census and no verification timestamp. These placeholders never prove verified absence. Masking the whole receipt prevents a caller subtracting its namespace’s counts to discover a mirror’s size. Ordinary receipts keep their counts. Marked completion and abandonment audit rows persist no counts. Failed marked receipts and audit rows answer ERASURE_FAILED for every cause and withhold step progress, so a residue failure reveals no special code or failed stage. Full internal evidence remains available for recovery and authorized platform-admin status reads.
  • A person’s connected accounts, the sources they own with every mirrored copy those wrote, and their entries in every crawl are erased when an org admin asks; afterwards no source row, list or members row names them, and a crawl reads them again only once an org admin readmits them. Canonical-person deletion runs from the required SCIM lifecycle; it never infers that a memory user_id names a sign-in subject.
  • Private canonical-owner cleanup uses the exact (org, subject, kind) key independently of namespace and end-user erasure. Disconnect and deactivation fence the generation, clear the current credential, queue every owned source for retirement, then physically delete historical provider identity and credentials. They retain only an empty inactive canonical-owner marker. SCIM deletion removes that marker too. The canonical-owner census counts all historical rows, including sealed values; a failed delete, missing census answer or residue refuses completion. The user_id erasure scope continues to mean an end-user principal and never guesses a sign-in subject from an id or email. Source retirement remains queued until the existing mirror retirement worker finishes; account cleanup alone does not certify that every source copy or crawl entry is erased.
  • Canonical-person deletion captures verified sign-in subject aliases and contact ties before removing them. A bounded Directory lookup separately proves Workspace customer and provider user ids; neither an email nor an end-user user_id is a canonical subject. A required cleanup capsule fences crawls until raw document ACL, group membership, source selector, approval and capsule histories have been scrubbed and their censuses pass. Other writers’ content and the registering administrator’s service-account key remain intact. The capsule is physically erased; only keyed refusal intent survives until explicit readmission. Reactivation cannot clear it.
  • The canonical-person control registry is cross-namespace and uses its real identity axes rather than inventing a namespace or memory user_id. SCIM capsules have an identity-lease-protected history reseal; exclusion keys and pending manifests reseal without changing the keyed identities, cleanup phase or Keeper fence. An unknown historical deletion remains a refusal pending verified identity repair, rather than guessing whom an old email denoted.
  • erasure.request refuses an acl- scope for every caller but a platform admin with auth on, and erasure.export refuses one for every caller (an export would read the mirror’s chunks with no permission check); with auth off both refuse it for everyone.
  • Audit rows: add, update, rotate, remove, sync_now as today, plus update’s item changes; connect and disconnect (subject, kind, scopes, never a credential); a crawl’s subjects (as keyed hashes), granted scopes and member count; a SCIM-driven stop names the sources it stopped.

Rolling upgrades

  • Clients. A client built before mirrored: true declares nothing, so a mirror leg answers it nothing until it is upgraded, and the refusal refuses its session-content ops; a third-party MCP client is refused the same way. Mirrored sources stay off (LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED) until the operator has rolled out the plugins, CLI, SDKs and hooks that honour the flag.
  • N-1 cannot operate on a source it must not. N-1 reads kind, unseals credential_enc and reads config_json, and knows none of the new columns. A source stored with kind empty is one N-1 cannot use: its check, rotate and sync turn fail before any request and retire nothing, and N-1 syncs only sources on a trunk namespace.
  • The new modes wait for N everywhere. add and update answer 503 OVERLOADED (retry_after_s) for a source stored with kind empty while any live process of any deployment on the ClickHouse service reports a schema version below the migration that adds the columns, or no heartbeat has been read, as rule 24 does for namespace guides (rolling upgrades).
  • Rollback and restores. N-1 has no mirror leg, but its gate admits a wildcard admin token that names a mirror, its org-wide reads (insights, usage, cost, the analytics export) include acl- rows, and nothing on N-1 ages the lists. N reports the condition (loom doctor, and the schema heartbeat check above) when a process below the mirror migration heartbeats while loom.knowledge_mirrors holds a live mirror (one not written deleted), as a rolling rollback (a Helm rollback included) would start one. So the rollback procedure (upgrades) stops every replica, stops every mirrored source and erases every mirror (loom knowledge mirror erase --all) before any N-1 replica starts. A restore of either store runs the same command before the deployment starts, since a restored copy cannot be trusted to say who may read; owners connect and add their sources again. The implementing pull request adds both steps and specifies the command.
  • Ownership. In a namespace, one owner per address, found by its tag, is today’s model, so N-1 and N agree on whose a document is; N-1 has no mirror leg.
  • Lists. N-1 never meets a list: the build that adds the column writes none and carries none forward, and the first build that writes one runs beside it (rule 53).
  • meet. N-1 still syncs a stored meet source on its turns; N skips it.
  • Erasure. N-1’s erasure misses the mirror and the new tables (rule 14): run again with N every erasure completed while an N-1 replica served.

Each implementing pull request adds its rule to rolling upgrades, and pins this section’s N-1 behaviour with a test that runs today’s check, rotate, remove and sync turn over a row N wrote with kind empty.

Breaks

  • Mirrored sources need LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED, which an operator sets once every Loom client and hook honours mirrored: true.
  • While the refusal holds, every session-content op, the hook-sent reads included (context.pack, code.search, code.impact, memory.for_file, namespace.validate), refuses a client that does not declare mirrored: true in every org: an SDK app that has not opted in, and a third-party MCP client, which cannot.
  • recording sources lose the meet provider.
  • The gate of add, update, check, sync_now and remove takes writer, with admin checked by the handler for shared and crawl sources; rotate is refused for a connected source.

Settings

Setting Default Meaning
LOOM_KNOWLEDGE_SOURCE_ACL_STALE_SECONDS 3600 a source’s listing or group members row older than this admits nobody
LOOM_KNOWLEDGE_SOURCE_MAX_DOCUMENTS 50000 a source’s stored documents plus the new ones a turn writes, at most 200,000; at it a turn tries no new file
LOOM_KNOWLEDGE_CONNECTED_SOURCES_MAX_PER_PERSON 8 a person’s connected sources in one namespace
LOOM_KNOWLEDGE_CONNECTED_SOURCES_MAX_PER_NAMESPACE 64 a namespace’s connected sources
LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED false mirrored sources may be added; set once every Loom client and hook honours mirrored: true
LOOM_KNOWLEDGE_SOURCE_MIRROR_DOCUMENTS_PER_PASS 40 documents a pass fetches, writes and retires for mirrored sources, apart from the namespace sources’ bound
LOOM_CONNECT_GDRIVE_CLIENT_ID, _SECRET unset the shared-folder store’s OAuth client; unset, those connections are off
LOOM_CONNECT_NOTION_CLIENT_ID, _SECRET unset the page store’s public integration
LOOM_CONNECT_LINEAR_CLIENT_ID, _SECRET unset the project tracker’s OAuth application

The redirect URI to register with each store is <LOOM_PUBLIC_BASE_URL>/v1/connect/<kind>/callback.

Rejected

  • Mirrored documents in the namespace itself, filtered by every read. A missed path would show the document to every reader. The list sits on the document only inside the mirror, which no token opens.
  • A namespace per permission list. A list changes with every share, which would move the document, its chunks and embeddings on each change.
  • A mirror name derived from the namespace. Anyone could compute it, and a known name makes every error and refusal about it a probe.
  • A permission list per reading account within a source. An account that read the list before a reader was removed would keep admitting them until its row aged out. Across sources each copy keeps its own list, bounded by the stale bound and dropped at its own source’s next listing.
  • Resolving the store’s groups through Loom’s SCIM groups. They hold a service’s end users (user:<id>), not people’s emails.
  • A separate meeting-store path. The meeting store already saves transcripts and notes to the shared-folder store, with the participants’ permissions.
  • drive.file with the picker. It cannot read a folder’s later files.
  • Incremental listings for mirrored sources. A sharing change does not move a file’s modification time.

Invariants

  • A mirrored document is read only through a leg that applies the permission check; no token reads or exports a mirror, and erasure.request takes a mirror’s name only from a platform admin, with auth on.
  • Every crawl and connected source is mirrored.
  • Every permission the sync cannot trust admits nobody, and so does an unread list (who may read).
  • Every complete listing of a mirrored source reads every file’s list again, and only a complete listing’s start vouches for the lists.
  • No caller-supplied host or query string reaches a store.
  • A connected source stops on every path that ends its connection or its owner’s standing (connected accounts), and its mirrored documents go with it.
  • Nothing derived from a mirrored document leaves the mirror except as the four exceptions allow, and no mirrored source exists until LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED is set.
  • N-1 never operates on a source it cannot honour: such a source is stored with kind empty.

Source entry points

  • server/knowledge/domain/sources.py: kinds, configs, items.
  • server/knowledge/domain/source_links.py: the link parser.
  • server/knowledge/app/sources.py: the ops.
  • server/knowledge/app/source_sync.py and its kind modules: the sync.
  • server/knowledge/app/source_access.py: the permission check and the mirror legs.
  • server/knowledge/app/connection_routes.py: the browser consent routes.
  • server/clients/gdrive.py, notion.py, linear.py, recording.py: the stores.
  • server/clients/connected_oauth.py (OAuthClient): fixed-provider consent, code exchange, refresh, account metadata and revocation requests. Calls share the caller’s deadline, cap response bodies, refuse redirects and never retry. The caller consumes browser-bound state before exchange, authorizes the Loom owner/source and commits refresh generations; this HTTP helper neither supplies nor bypasses those checks.

Tests

Each implementing pull request lists its own; these pin the model:

  • Unit:
    • Items, links and the sources’ ops:
      • The link parser on every link form and every unreadable one.
      • Item validation.
      • The query builder escaping every field.
      • A non-owner admin’s list and inspect of a connected source showing config {}.
      • A connected source’s items, and its document counts beyond those a checked leg lists to that admin when admitted, hidden from an admin who is not its owner; a non-owner admin’s check of a connected source answering no item ids.
      • A failed read of the reader’s canonical subject or emails giving the same marker whether or not a candidate exists.
      • A mirrored source’s probe (check, and the add, update and rotate probes, by a non-owner admin and a crawl registrant), status, remove answer and a namespace erasure’s receipt carrying no count, title, file id or address of what the mirror holds.
      • A crawl update probe by any caller answering the same code whether or not the file exists.
      • A crawl’s status carrying no item <n> code and no meet_recordings: not found.
      • A non-owner admin’s remove of mirrored sources of 10 and 10,000 documents leaving the same request-side figures.
      • add of a removed mirrored source’s name answering as for a fresh name.
      • A mirrored add during an open namespace erasure refused with ERASURE_RUNNING.
      • The mixed-version refusal, and N-1’s check, rotate, remove and sync turn over a row N wrote with kind empty.
      • A crawl update by an admin with an org-wide role who is not the registrant, adding a file or filter item, accepted whether or not any crawled person can read it.
    • The mirror and its gate:
      • The acl- gate in check_authorized and CachedGate, with auth on and off, and for a platform admin on a read op.
      • namespace.register and namespace.restore refusing the form.
      • erasure.request and erasure.export refusing an acl- scope with auth on and off.
      • row_reach and audit.list refusing an acl- name with auth off.
      • row_reach, row_in_scope and namespace_for_id opening nothing of a mirror with auth off.
      • erasure.list and erasure.status treating an acl--scoped saga as not found for an org admin.
      • erasure.status and erasure.list by a viewer, and a saga that fails on mirror residue, showing no mirror count.
      • resolve_org_scope with auth off, every_namespace_scope, rollup_member_namespaces, sql_v1_agent_events under a * grant and the analytics export leaving out every acl- name.
      • The insights agents and cost reads leaving acl- names out.
      • audit.list after a connected add, by an admin who is not the owner, showing no item id other than its keyed hash.
      • erasure.export and a user_id erasure’s census naming no acl- row.
      • No row carrying a reader’s user_id under an acl- name after an admitted reader’s get, search and ask.
      • An erased namespace’s name registered again with a new mirror.
      • N reporting a below-schema heartbeat while a mirror holds rows.
    • The list and who may read:
      • The access check on fresh and stale listings, unread, group, stale group, domain, link-only, anyone and unknown permissions, and for a delegated token, a bot and an end-user token.
      • An email-shaped subject with no verified directory email, and an inheriting service account, seeing no emails document.
      • A group matched only through the owning source’s members row.
      • A connected source’s unreadable-list entry admitting the owner through the list’s owner entry, never an email or the connection’s store email.
      • A crawl file item of a link-only file admitting no crawled person.
      • A verified email at a domain without the identity provider’s domain claim refused a domains document.
      • A crawled person who opened a link-only file only through its link not admitted.
      • A deactivated team member admitted by no linear list.
      • Against an emails document: a person’s own session token and a reads_as_creator agent token admitted, a Settings-made service-account token (with creator), a delegated child of a reads_as_creator token other than the assistant’s answer token, and a request on behalf of another person not.
      • A permission the store marks deleted, whose address now belongs to another verified user, admitting nobody.
      • An answer token delegated from the listed person’s session token admitted, an answer token delegated from a reads_as_creator token admitted as its creator, and a token delegated from an answer token refused.
      • An assistant answer token delegated from a Settings-made service-account token refused an emails document.
      • A token with consumer loom-assistant and an assistant-answer- id but no reads-as claim reading as nobody.
      • A caller of assistant.ask, loom_assistant_ask or the chat route that does not declare the flag getting no mirrored text; an assistant.ask and a chat-route answer on a namespace with an active mirrored source and no admitted item answering mirrored: true, the chat route in its stream’s first event and, unstreamed, in its response’s first field; a plugin session flagged once its loom_assistant_ask answer carries the flag; a marked assistant.ask and chat-route row folded and masked like a marked tool-call row.
      • A knowledge.search, context.pack and memory.ask on a namespace with an active mirrored source and no admitted item answering mirrored: true as the first field; the same reads after the last mirrored source is removed, and while the only one listed is inactive (its owner gone, its cleanup pending), answering no flag and running no mirror leg.
      • A marked memory.ask whose candidates are all hidden making the same model and embedding calls as one with no candidates.
      • A caller admitted to nothing seeing the same cost.budget_status change over 10 and over 10,000 hidden matching documents.
      • A file another mirrored source holds, added by a second source, written as that source’s own copy with its own list, so the adder sees what they would with no other source.
      • A hidden mirrored chunk holding the query term leaving an admitted hit’s lexical_score and rank unchanged.
      • A namespace whose documents are all mirrored ranking by both legs.
    • The check fails closed:
      • A leg that cannot read the listing row or the owning source’s registry row showing no mirrored document, marked degraded: mirror_unread, and one that cannot read the lists, the members rows or the heads dropping mirrored results without the marker.
      • The same degraded marker for a query that matches a hidden document and one that matches nothing while a mirror read fails.
      • A failed list, members or heads read with one hidden candidate answering exactly as a query with no candidate: no marker.
      • A leg on a lagging replica that sees a fresh listing row and an older list admitting nobody.
      • A document missing from a complete listing admitting nobody once its listing row lands.
      • A stopped source’s documents admitting nobody before their retirement.
    • Every complete listing reads the lists again:
      • A turn that finds a narrowed list rewriting the row without rebuilding a chunk, and one that finds the same list writing nothing.
      • A listing row landing only after its changed lists.
      • A fetch the pass bound deferred writing no list older than the listing row that lands after it.
      • After a person’s erasure, no source’s next turn writing a list or members row that names them.
      • A namespace source’s per-pass progress unchanged by a mirror’s size.
      • A large removal leaving another mirrored source’s content fetches unchanged.
      • A mirror’s originals surviving a blob sweep and its chunks re-embedded after a model change.
    • The rule and its exceptions:
      • Two readers of one namespace, one admitted to a mirrored document, through the repository-docs listing cache and memory.ask’s precedent fast path: the other gets none of it.
      • A request a mirror leg answered keeping no argument or result-side field the leg shaped in its agent_events row.
      • A marked request’s trace and exposure rows unchanged by a hidden mirrored hit.
      • A follow-up query a mirror leg shaped leaving the embedding cache untouched.
      • A generation over a mirrored chunk capturing no text.
      • The sync’s audit rows, status, pass notes and log lines naming no mirrored file id or address.
      • No reader but a platform admin reading the work figures on a marked row.
      • A viewer’s cost or insights read showing no per-request token spend of another person’s marked request.
      • A viewer’s cost.summary grouped by user_id and operation showing no single marked request’s tokens.
      • cost.budget_status and cost.summary by a viewer reading the same before and after a mirrored sync pass.
      • A namespace-scoped SQL user reading sql_v1_agent_events and sql_v1_usage_daily, and the analytics export’s usage rows, after another person’s marked request, finding its spend and work only as the fold and the mask allow.
      • A mirrored source’s document counts shown to nobody but a platform admin.
      • knowledge.reads after an admitted reader’s knowledge.get of a mirrored address not naming it, nor a namespace document’s address read in the same marked request.
      • The dashboard’s knowledge.get and the assistant’s memory.ask of an admitted mirrored document answered with mirrored: true.
      • A later turn of a session that saw mirrored: true not captured, and a client that does not declare the flag answered nothing by a mirror leg.
      • A capture hook recording no turn whose answer carries mirrored: true.
      • A plugin session over a collection with an active mirrored source and no admitted hit flagged all the same, its next turn not captured.
      • loom CLI output holding an admitted mirrored item, and a subagent’s mirror read, not captured into the parent session.
      • A loom CLI run through a shell and a prompt-recall injection carrying mirrored: true stopping every hook of the session.
      • The CLI’s ingest-transcript refusing a transcript that carries the flag.
      • A dashboard view of an answer carrying mirrored: true, with or without an admitted item, sending no ui.vitals.
      • A forked or resumed session not captured after its parent saw the flag.
      • One test over every op the registry classifies as session content, each refusing a client that does not declare the flag from the first time LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED is set until it is off again and the mirror-erase command has run, in every org, with the switch on and off; and a test that collects every op the shipped hooks, their shared library and the CLI send without the agent choosing the content and asserts each is classed as session content.
      • A session with clients holding tokens for two orgs, one reading a mirror, capturing nothing into the other.
      • The session-content refusal holding once the switch was set with no mirror yet, when the deployment row cannot be read and on a replica whose view lags it, and lifting only once the switch is off and the mirror-erase command has deleted the row.
      • A mirror-erase run with the switch still on refused, a restore that loses the deployment row, and a mirrored add after either, with the refusal still holding.
      • A viewer’s insights.query over analysis_agent_events after another person’s marked request finding its tokens, cost, model and outcome fields only as the mask allows, the caller’s own rows included.
      • A caller admitted to nothing seeing no work figure on its own marked row over a mirror of 10 and of 10,000 hidden documents.
      • A checked leg’s every answer field unchanged by a hidden mirrored document.
      • The mirror-erase command cut short after deleting mirrors but before the latch row, with the refusal still holding.
      • An assistant memory citation from a marked memory.ask returned unchecked.
      • A viewer’s cost.budget_status read before and after another person’s marked request differing only by that request’s folded spend, as exception 2 accepts.
      • The folded per-namespace-day row’s errors count unchanged by a marked request that fails in the model call.
      • The session-content refusal of a hook-sent context.pack from a client that does not declare the flag.
      • An admitted mirrored item in context.pack writing no recall credit or exposure row.
      • A later ingest-transcript of a session flagged only by inheritance refused from an unflagged shell.
      • A session started from a flagged session’s process (claude -p through its shell) flagged through the inherited marker, and a hook’s code.search and context.pack sending nothing after the flag.
      • The owner’s reads_as_creator token admitted to the owner-only list of a connected source; an on-behalf request for the owner and a token minted for the owner’s subject without sid refused.
      • A claude -p child that read a mirror flagging its parent session, whose hooks then send nothing.
      • A namespace copy winning a dedupe carrying only its own chunks and scores; a reader only the namespace copy admits keeping it; a reader whose query matches only the mirror copy’s revision getting that hit with mirrored: true.
      • A restore that drops a loom.knowledge_mirrors row while acl- rows stay in the documents, chunks, heads and bucket, then loom knowledge mirror erase --all deleting every one of them before the latch.
      • A non-owner admin’s sync_now on a mirrored source with a refused item answering as on a clean source.
  • Integration: tests/integration/test_document_source_leakage.py, beside test_cross_principal_leakage.py: for every leg with the check and every id-addressed op, a reader the list leaves out sees no mirrored document, its title, its address, its count or its existence, through the legs, knowledge.reads, audit.list, the org-wide rollups or a source’s status (an admin who is not the owner included), under both storage authorities; a reader the list names sees it; a reader removed from the store’s list stops seeing it after the next listing of every source that holds a copy; one mirrored source’s retirement leaving another source’s copy of the same address readable; a shared mirrored source’s status, read by a viewer no list names, showing no per-file code.
  • Integration: a connected source is mirrored wherever it writes; a SCIM deactivation and a disconnect stop the person’s sources and retire their mirrored documents; namespace.delete and collection.delete retire the namespace’s sources and clear their credentials.

¿Te ha resultado útil esta página?