# Saving URLs (https://brainapi.lumen-labs.ai/docs/v2/ingestion/urls)

> For the complete BrainAPI documentation index, see [llms.txt](https://brainapi.lumen-labs.ai/docs/llms.txt). A markdown version of any docs page is available by appending `.md` to its URL (e.g. https://brainapi.lumen-labs.ai/docs/v2/ingestion/urls.md).

How to ingest a public website into a brain from a seed URL

URL ingest is for public websites. BrainAPI does not fetch the seed URL itself. It asks the configured crawler (Lumen Crawler by default) to render pages, wait for cleanup JSON, and then ingest only the fields that matter: title, headings, core content, contacts, and CTAs. Chrome, navigation, and raw Markdown are dropped.

`POST /ingest/url` returns **202 Accepted** with `{ "message": "URL ingestion accepted", "task_id": "…" }`. Poll with [ingestion tasks](https://brainapi.lumen-labs.ai/docs/v2/ingestion/tasks).

<AgentNote>
- `POST /ingest/url` with JSON `{ "url": "https://…" }`, auth with `BrainPAT`
- `url` must be public `https`; `max_pages` defaults to 10 and caps at 999
- Optional `allowed_path_prefixes` limits the crawl to matching paths and must include the seed path
- Requires `LUMEN_CRAWLER_KEY`; missing config is **503**
- **202** + `task_id` → poll `GET /tasks/{id}`; **404** ≠ pending
</AgentNote>

<Callout type="info">
  Standard URL ingestion processes each cleaned page as a unit. Hosted progressive ingestion can split pages into sections. Set `skip_enrichment=true` to store
  passages without Scout/Architect. Optional `allowed_domains` adds extra eTLD+1 hosts
  (max 20). `group_tracking` rolls up ingest cost the same way as text and file ingest.
</Callout>

## Path scope

In the console, open **Link → Advanced → Allowed path prefixes** and enter one path per line. For API requests, send `allowed_path_prefixes` as an array of up to 20 paths. Start each prefix with `/` and leave out the host, query, and fragment.

Prefixes apply across all allowed domains and subdomains. Matching is case-sensitive and uses whole path segments. For example, `/docs` matches `/docs` and `/docs/install`. It excludes `/docsite` and `/Docs`. Trailing slashes on prefixes and queries or fragments on page URLs are ignored.

The seed URL must match at least one prefix. Leave the setting blank to use domain scope alone. See [Lumen Crawler’s path matching rules](https://crawler-docs.lumen-labs.ai/docs/v2/doors#path-scope).

## Usage

<Tabs items={["cURL"]}>
  <Tab value="cURL">
    Post a seed URL to `POST /ingest/url`. Expect HTTP 202 and a `task_id`.
    <DynamicCodeBlock
      lang="bash"
      code={`curl -X POST "<DEPLOYMENT_URL>/ingest/url" \\
      -H "Content-Type: application/json" \\
      -H "X-Brain-ID: mybrain123" \\
      -H "BrainPAT: YOUR_BRAIN_PAT" \\
      -d '{"url":"https://example.com/docs","max_pages":10,"allowed_path_prefixes":["/docs","/guides"]}'
# → {"message":"URL ingestion accepted","task_id":"…"}`}
    />
  </Tab>
</Tabs>

<Callout type="info">
  After 202, poll `GET /tasks/{task_id}` with the returned ID. Legacy imports can expose page task IDs in `counts.task_ids`. Staged imports report publication at the root. Hosted progressive imports also expose section activity.
  See [Ingestion tasks](https://brainapi.lumen-labs.ai/docs/v2/ingestion/tasks).
</Callout>