For the complete BrainAPI documentation index, see llms.txt. A markdown version of any docs page is available by appending .md to its URL. Docs MCP: /docs/mcp.
Saving URLs
How to ingest a public website into a brain from a seed URL
URL ingest is for public websites. BrainAPI does not fetch the seed URL itself. It asks the configured crawler (Lumen Crawler by default) to render pages, wait for cleanup JSON, and then ingest only the fields that matter: title, headings, core content, contacts, and CTAs. Chrome, navigation, and raw Markdown are dropped.
POST /ingest/url returns 202 Accepted with { "message": "URL ingestion accepted", "task_id": "…" }. Poll with ingestion tasks.
For agents
POST /ingest/urlwith JSON{ "url": "https://…" }, auth withBrainPATurlmust be publichttps;max_pagesdefaults to 10 and caps at 999- Optional
allowed_path_prefixeslimits the crawl to matching paths and must include the seed path - Requires
LUMEN_CRAWLER_KEY; missing config is 503 - 202 +
task_id→ pollGET /tasks/{id}; 404 ≠ pending
Standard URL ingestion processes each cleaned page as a unit. Hosted progressive ingestion can split pages into sections. Set skip_enrichment=true to store
passages without Scout/Architect. Optional allowed_domains adds extra eTLD+1 hosts
(max 20). group_tracking rolls up ingest cost the same way as text and file ingest.
Path scope
In the console, open Link → Advanced → Allowed path prefixes and enter one path per line. For API requests, send allowed_path_prefixes as an array of up to 20 paths. Start each prefix with / and leave out the host, query, and fragment.
Prefixes apply across all allowed domains and subdomains. Matching is case-sensitive and uses whole path segments. For example, /docs matches /docs and /docs/install. It excludes /docsite and /Docs. Trailing slashes on prefixes and queries or fragments on page URLs are ignored.
The seed URL must match at least one prefix. Leave the setting blank to use domain scope alone. See Lumen Crawler’s path matching rules.
Usage
Post a seed URL to POST /ingest/url. Expect HTTP 202 and a task_id.
curl -X POST "<DEPLOYMENT_URL>/ingest/url" \ -H "Content-Type: application/json" \ -H "X-Brain-ID: mybrain123" \ -H "BrainPAT: YOUR_BRAIN_PAT" \ -d '{"url":"https://example.com/docs","max_pages":10,"allowed_path_prefixes":["/docs","/guides"]}'# → {"message":"URL ingestion accepted","task_id":"…"}After 202, poll GET /tasks/{task_id} with the returned ID. Legacy imports can expose page task IDs in counts.task_ids. Staged imports report publication at the root. Hosted progressive imports also expose section activity.
See Ingestion tasks.