> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcatalog.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl

> Async discovery of collections and product listings for a vendor. Returns immediately with an execution_id to check status.

<Tip>
  **When to use:** Discover and index all product listings from a vendor website. The crawl process will:

  1. Discover collections from the vendor
  2. Extract product listings from each collection

  This is ideal for initial discovery of a new vendor. Use [`POST /v2/extract`](/v2/api-reference/endpoints/extract/extract) to get full product details from discovered listings.
</Tip>

<Note>
  **Async Processing:** This endpoint starts processing asynchronously and returns an `execution_id` immediately. Use [`GET /v2/crawl/{execution_id}`](/v2/api-reference/endpoints/crawl/get-crawl-status) to check progress and retrieve results when processing completes.

  **Billing Requirement:** Crawl requests require auto top-up to be enabled in your billing settings. This ensures you have sufficient credits to complete the crawl operation.

  **View All Executions:** You can view all your crawl executions using [`GET /v2/crawl`](/v2/api-reference/endpoints/crawl/list-crawl-executions) to see all crawl jobs, their statuses, and when they were created.

  **Cancel Execution:** You can cancel a running crawl execution using [`DELETE /v2/crawl/{execution_id}`](/v2/api-reference/endpoints/crawl/cancel-crawl-execution) if you need to stop a crawl that is currently processing.
</Note>

## Request

<ParamField header="x-api-key" type="string" required>
  Your API key for authentication
</ParamField>

### Request Body

<ParamField body="url" type="string" required>
  The vendor website URL or domain to crawl

  **Note:** The URL will be normalized to a canonical form. If a crawl is already running for the same hostname, the existing execution ID will be returned.

  **URL sanitization:** URLs are sanitized and normalized on the backend; client-side sanitization is not required.
</ParamField>

## Response

<ResponseField name="execution_id" type="string">
  Unique execution identifier for this crawl job. Use this ID with [`GET /v2/crawl/{execution_id}`](/v2/api-reference/endpoints/crawl/get-crawl-status) to check progress.

  **Format:** `crawl-{hostname}-{uuid}`

  **Note:** If a crawl is already running for the same hostname, this will return the existing execution ID.
</ResponseField>

<ResponseField name="status" type="string">
  Initial execution status. Always `"pending"` when the crawl is first started.
</ResponseField>

<ResponseField name="meta" type="object">
  Metadata about the request

  <Expandable title="Meta Object">
    <ResponseField name="credits_used" type="number">
      Number of credits charged for starting this crawl operation
    </ResponseField>

    <ResponseField name="request_id" type="string" optional>
      Unique request identifier for support purposes (also available in the `Request-ID` response header)
    </ResponseField>
  </Expandable>
</ResponseField>

<Note>
  **HTTP Status Code:** This endpoint returns `202 Accepted` to indicate the request has been accepted for processing. The response body contains the execution details you need to track the crawl progress.
</Note>

<RequestExample dropdown>
  ```powershell cURL theme={null}
  curl -X POST https://api.getcatalog.ai/v2/crawl \
    -H "Content-Type: application/json" \
    -H "x-api-key: $CATALOG_API_KEY" \
    -d '{
      "url": "skims.com"
    }'
  ```

  ```javascript JavaScript theme={null}
  // Start crawl
  const response = await fetch('https://api.getcatalog.ai/v2/crawl', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-api-key': 'YOUR_API_KEY'
    },
    body: JSON.stringify({
      url: 'skims.com'
    })
  });

  const data = await response.json();
  console.log(`Crawl started with execution ID: ${data.execution_id}`);
  console.log(`Credits used: ${data.meta.credits_used}`);

  // Poll for results: GET /v2/crawl/{execution_id}
  ```

  ```python Python theme={null}
  import requests

  # Start crawl
  response = requests.post(
      'https://api.getcatalog.ai/v2/crawl',
      headers={
          'Content-Type': 'application/json',
          'x-api-key': 'YOUR_API_KEY'
      },
      json={
          'url': 'skims.com'
      }
  )

  data = response.json()
  execution_id = data['execution_id']
  print(f"Crawl started with execution ID: {execution_id}")
  print(f"Credits used: {data['meta']['credits_used']}")

  # Poll for results: GET /v2/crawl/{execution_id}
  ```
</RequestExample>

<ResponseExample>
  ```json POST Response theme={null}
  {
    "execution_id": "crawl-skims-com-4ddfa1c9",
    "status": "pending",
    "meta": {
      "credits_used": 0
    }
  }
  ```
</ResponseExample>