Skip to main content
GET
/
v1
/
crawl
/
{execution_id}
cURL
# Check execution status
curl -X GET "https://api.getcatalog.ai/v1/crawl/crawl-skims-com-a1b2c3d4" \
  -H "x-api-key: $CATALOG_API_KEY"

{
  "status": "completed",
  "total_listings_found": 1234
}

Documentation Index

Fetch the complete documentation index at: https://docs.getcatalog.ai/llms.txt

Use this file to discover all available pages before exploring further.

When to use: After starting a crawl job with the POST endpoint, use this endpoint to:
  • Check if the crawl is complete
  • Monitor real-time progress
  • Get the total number of listings found when the crawl completes
Authentication Required: This endpoint requires a valid API key. The API key is verified, but execution IDs are not restricted to specific API keys. Keep your execution IDs secure.Cancel Running Executions: If you need to stop a crawl that is currently running, use DELETE /v1/crawl/{execution_id} to stop the execution.

​
Request

​
x-api-key
string
required
Your API key for authentication
​
execution_id
string
required
The execution ID returned from POST /v1/crawlFormat: crawl-{hostname}-{uuid}Note: If the execution ID does not exist, the endpoint returns a 404 Not Found error.

​
Response

​
status
string
Current execution statusPossible values:
  • "pending" - Execution has been created but not yet started
  • "running" - Execution is currently processing
  • "completed" - Execution finished successfully
  • "failed" - Execution failed or was aborted
​
total_listings_found
number | null
Total number of product listings discovered during the crawl (only available when status is “completed”)Note: This value is extracted from the execution output and may be null if the crawl is still running or if the count cannot be determined.
cURL
# Check execution status
curl -X GET "https://api.getcatalog.ai/v1/crawl/crawl-skims-com-a1b2c3d4" \
  -H "x-api-key: $CATALOG_API_KEY"

{
  "status": "completed",
  "total_listings_found": 1234
}

​
Polling Strategy

For best results when waiting for completion:
  1. Initial Poll: Check status immediately after receiving execution_id
  2. Polling Interval: Wait 10-30 seconds between polls for running executions (crawls can take longer than product processing)
  3. Exponential Backoff: Consider increasing wait time for long-running crawls
  4. Timeout: Set a maximum wait time based on the size of the vendor website
  5. Cancellation: If a crawl is taking too long or you need to stop it, use DELETE /v1/crawl/{execution_id} to stop running executions