> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcatalog.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Building a Product Index
> How to build and maintain your own product index using the Catalog API
Build and maintain your own product index using Catalog's crawl and extract endpoints. The typical workflow is: **crawl** a vendor to discover their product listings, **browse** the discovered data to curate which products you want, then **extract** full product data for your selected listings.
This guide covers the core workflow: discovering product listings via [`POST /v2/crawl`](/v2/api-reference/endpoints/crawl/crawl), browsing discovered data via [`GET /v2/vendors`](/v2/api-reference/endpoints/get-vendors), [`POST /v2/collections`](/v2/api-reference/endpoints/get-collections), and [`POST /v2/listings`](/v2/api-reference/endpoints/get-listings), and extracting products via [`POST /v2/extract`](/v2/api-reference/endpoints/extract/extract).
## Step 1: Crawl a Vendor
Start by crawling a vendor website to discover all their collections and product listings. Use [`POST /v2/crawl`](/v2/api-reference/endpoints/crawl/crawl) with the vendor URL:
```json theme={null}
{
"url": "https://www.example.com"
}
```
The crawl process:
* Discovers collections from the vendor (e.g., "New In", "Sale", "Shoes")
* Extracts product listings from each collection
**Typical flow:**
1. Start a crawl by calling `POST /v2/crawl` with the vendor URL
2. Receive an `execution_id` immediately (format: `crawl-{hostname}-{uuid}`)
3. Poll [`GET /v2/crawl/{execution_id}`](/v2/api-reference/endpoints/crawl/get-crawl-status) to check progress:
* Status will be `"pending"`, `"running"`, `"completed"`, or `"failed"`
* When `status === "completed"`, you'll see `total_listings_found` indicating how many products were discovered
**Billing Requirement:** Crawl requests require auto top-up to be enabled in your billing settings. This ensures you have sufficient credits to complete the crawl operation.
## Step 2: Browse Discovered Data
After crawling, use the three browse endpoints to explore and curate which products you want to extract. These endpoints return data from vendors you have crawled.
### List your vendors
Use [`GET /v2/vendors`](/v2/api-reference/endpoints/get-vendors) to see all vendors you have crawled and their product counts:
* Review which vendors you have indexed
* Check `product_count` to see how many listings were discovered
* Use `latest_product_update_by_catalog` to see when data was last refreshed
### Browse collections
Use [`POST /v2/collections`](/v2/api-reference/endpoints/get-collections) to explore a vendor's collections:
* Retrieve collections (e.g., "New In", "Sale", "Shoes")
* Decide which collections to include in your index (e.g., only "New Arrivals" or "Top Sellers")
This helps you build a more structured index (vendor → collection → products).
### Curate product listings
Use [`POST /v2/listings`](/v2/api-reference/endpoints/get-listings) to page through product listings for a vendor or collection. As you browse:
* Review the lightweight listing data (title, URL, collection, timestamps)
* Curate which products you want to extract full data for
* Store the **canonical product URLs** for listings you want to index
This step lets you select a subset of products rather than extracting everything—useful when you only need certain collections, price ranges, or product types.
## Step 3: Extract Full Product Data
Once you have curated your list of product URLs, use [`POST /v2/extract`](/v2/api-reference/endpoints/extract/extract) to get full product data with AI enrichment, reviews, and image tags.
**Typical flow:**
1. Pull a batch of URLs from your curated list (up to 1000 URLs per request)
2. Call `POST /v2/extract` with your URLs:
```json theme={null}
{
"urls": [
"https://www.example.com/product/1",
"https://www.example.com/product/2"
],
"enable_enrichment": true,
"enable_reviews": true,
"enable_image_tags": true,
"country_code": "us"
}
```
3. Receive an `execution_id` (format: `extract-urls-{uuid}`) and poll [`GET /v2/extract/{execution_id}`](/v2/api-reference/endpoints/extract/get-extract-status)
4. Upsert the extracted products into your index (search engine, DB, vector store, etc.)
## Shortcut: Extract All Products from a Vendor
If you want to extract **all** products from a vendor without browsing and curating, you can use [`POST /v2/extract`](/v2/api-reference/endpoints/extract/extract) with the `vendor` parameter instead of `urls`. This skips the browse step entirely.
**Typical flow:**
1. Start extraction by calling `POST /v2/extract` with the vendor:
```json theme={null}
{
"vendor": "example.com",
"enable_enrichment": true,
"enable_reviews": true,
"enable_image_tags": true,
"country_code": "us"
}
```
2. Optionally provide a `crawl_id` to wait for crawl completion before starting extraction:
```json theme={null}
{
"vendor": "example.com",
"crawl_id": "crawl-example-com-a1b2c3d4",
"enable_enrichment": true,
"country_code": "us"
}
```
3. Receive an `execution_id` immediately (format: `extract-{vendor}-{uuid}`)
4. Poll [`GET /v2/extract/{execution_id}`](/v2/api-reference/endpoints/extract/get-extract-status):
* Check `status` and `meta.progress` for real-time progress
* When `status === "completed"`, results are available with pagination
* Use `page` and `page_size` query parameters to retrieve results in chunks
5. Upsert the extracted products into your index (search engine, DB, vector store, etc.)
**When to use vendor-based extraction:** This shortcut is ideal when you want a complete catalog from a vendor. If you only need specific collections or a curated subset of products, use the browse + URL extraction flow instead.
## Extracting from Any URL Source
The [`POST /v2/extract`](/v2/api-reference/endpoints/extract/extract) endpoint accepts product URLs from any source—not just URLs discovered through crawling. This gives you flexibility to build your index from multiple sources:
**Common use cases:**
* **Affiliate feeds:** Extract products from affiliate network URLs
* **Merchant feeds:** Process product URLs from partner data feeds
* **Internal catalogs:** Index products from your own product database
* **Hand-curated lists:** Extract specific products you've manually selected
* **Competitor monitoring:** Track products from URLs you've collected
**Example:**
```json theme={null}
{
"urls": [
"https://www.nike.com/t/air-force-1-07-mens-shoes-5QFp5Z/CW2288-111",
"https://www.adidas.com/us/gazelle-shoes/BB5476.html",
"https://www.newbalance.com/pd/574-core/ML574EVG.html"
],
"enable_enrichment": true,
"country_code": "us"
}
```
This works the same as extracting crawled URLs—you receive an `execution_id` and poll for results.
## Keeping Your Index Fresh
To maintain a high-quality product index:
**Schedule re-crawling:** Periodically re-crawl vendor websites to discover new product listings and collections.
**Schedule re-extraction:** Re-run extraction to capture price, availability, and content changes for existing products.
**Monitor failures:** Use `success` and `outcome` fields to detect:
* Non-product URLs
* Unsupported vendors
* Products that have been removed
**Prune stale products:** Remove (or downgrade) products that consistently fail to process or are no longer available.
**Use URLs for targeted updates:** For specific products that need frequent updates (e.g., featured items), use the Extract endpoint with URLs to refresh them more frequently than a full re-extraction.
## Related endpoints
Discover collections and product listings
Extract high-quality, real-time product data
List your crawled vendors
List collections for a vendor
List product listings for a vendor