Schedule source ingestion

Create a daemon to keep reading a source into Brain on an hourly schedule. Use Brain’s Ingest page in the web app to configure it and inspect its runs.

Before you start

You need access to your product and a source connection attached to it. Public GitHub repositories need no connection. Imported memories are shared with the product.

For MCP, use a product-scoped credential with write access to create, update, run, or re-ingest; read to inspect; and admin to remove a daemon. The tool reference lists the full arguments.

Ingestion uses embedding and language-model tokens. Under blocking enforcement, an exhausted organization allowance can skip a run without failing it.

Set up a daemon

On Ingest, create a daemon and choose GitHub, Google Drive, or ClickUp.
Select its scope. For GitHub, choose the account, repository, and branch; each daemon reads exactly one branch. For Drive, select the account whose picked files and folders it should read. For ClickUp, select a workspace, optionally one space or list, and any chat channels.
Enable the content categories you want. GitHub Markdown also supports include/exclude patterns; the default is **/*.md with nothing excluded.
Choose source trust and load mode as described below, then create the daemon. It starts enabled.
Select Run now, or wait for the hourly cycle. Open the daemon’s page to watch its run and item outcomes.

Over MCP, the corresponding operations are brain_daemon_create and brain_daemon_run. Keep the daemon ID for subsequent calls.

Choose trust and load mode

Automatic source trust uses the category defaults. An explicit setting applies one tier to everything the daemon imports: policy, official, team, or informal. Existing memories keep their recorded authority.

Mode	Processing	Source deletions
Hot	Processes content as it is read, usually in seconds to minutes. A large first sync can use batch processing; a small first sync and all later syncs use hot processing.	A run processed hot removes memories for items that disappeared from the source.
Batch	Defers every sync; it can take hours and costs less for large volumes.	Imports and updates, but never applies removals.

How the schedule works

The ingestion cycle runs hourly for the whole product, not on a configurable schedule per daemon. It considers enabled daemons and leaves any with a run in progress alone. Run now is the only other trigger; a push, file edit, or new task does not immediately start ingestion.

A run discovers source items, fetches them, and stores changed content. Unchanged items are skipped. A stale run is closed automatically so the daemon can resume on a later cycle.

Inspect a run

Open the daemon’s page for its latest run, item list, and recent history. Over MCP, use brain_daemon_status, brain_daemon_runs, and brain_daemon_run_items. Run history returns 20 entries by default, up to 100; item pages return 100 by default, up to 500, with a cursor for pagination.

Run state	Meaning
`queued`	Accepted and waiting.
`running`	In progress.
`done`	Finished; inspect items to see what was imported or denied.
`failed`	At least one retryable failure, such as a timeout, rate limit, network error, or server error. The next cycle tries again.

Item outcome	Meaning
`discovered`	Found in scope.
`fetched`	Content retrieved.
`ingested`	Written to memory.
`skipped_unchanged`	Content has not changed since the previous run.
`denied`	Source permanently refused the item: not found or not permitted.
`error`	A retryable failure.

A denied item does not fail the run. Fix its connection access rather than expecting a retry to grant permission. A quota-blocked run can also end done with no ingestion.

Two daemon counts accumulate across runs: documents whose text was stored but meaning was not fully extracted, and unmeasured documents, including older imports and unfinished batches. Neither count means those documents were checked and found complete.

Repair an incomplete ingestion

Brain automatically retries documents whose meaning could not be extracted. The retry schedule is immediate, then after one hour, six hours, and 24 hours, up to four consecutive attempts. Each run repairs at most 100 documents; the rest carry over. An edit at the source uses the normal changed-content path regardless of that budget.

To force specific documents to be read again:

Find the daemon ID and the exact source paths it recorded.
Call brain_daemon_reingest over MCP with id and paths. Supply 1–100 paths per call, such as a repository path, Drive file ID, or ClickUp item ID.
Check the response: cleared lists paths prepared for re-ingestion; notFound lists unknown paths; skippedPending lists paths still in an unfinished batch. triggered says whether a run started, and runId identifies it. Inspect that run and the stored text.

A paused daemon or one with a run already in progress may not start another run. Read triggerError when present. Wait for an unfinished batch before requesting its skipped paths again.

Forced re-ingestion is not available in the web app. For deferred processing, brain_daemon_batch_status reports progress and the batch outcome.

Pause or remove a daemon

To pause, disable the daemon from Ingest, or use brain_daemon_update with enabled: false. It keeps its configuration and imported memories. It refuses manual runs until enabled again, then resumes from its previous state.

Removing a daemon, including through brain_daemon_remove, deletes its configuration and source-tracking records, not its memories. Those memories remain without future source updates. Recreating the daemon imports everything again; the old and new memories are reconciled by content rather than daemon identity.

Verify it works

Check that the first run has ingested items, then open Browse and search for a statement from one of those items. A done status alone is insufficient: inspect denied items and quota-blocked results.

For missing documents, check the selected scope and content categories. For access failures, confirm the connection still reaches the source and is attached to the product. For a daemon that never runs, check it is enabled and select Run now.

Was this page helpful?