> ## Documentation Index
> Fetch the complete documentation index at: https://supermemory-temp-snowcone-command.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawler Connector

> Crawl and sync websites automatically with scheduled recrawling and robots.txt compliance

Connect websites to automatically crawl and sync web pages into your Supermemory knowledge base. The web crawler respects robots.txt rules, includes SSRF protection, and automatically recrawls sites on a schedule.

<Warning>
  The web crawler connector requires a **Scale Plan** or **Enterprise Plan**.
</Warning>

## Quick Setup

### 1. Create Web Crawler Connection

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    import Supermemory from 'supermemory';

    const client = new Supermemory({
      apiKey: process.env.SUPERMEMORY_API_KEY!
    });

    const connection = await client.connections.create('web-crawler', {
      redirectUrl: 'https://yourapp.com/callback',
      containerTags: ['user-123', 'website-sync'],
      documentLimit: 5000,
      metadata: {
        startUrl: 'https://docs.example.com'
      }
    });

    // Web crawler doesn't require OAuth - connection is ready immediately
    console.log('Connection ID:', connection.id);
    console.log('Connection created:', connection.createdAt);
    // Note: connection.authLink is undefined for web-crawler
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from supermemory import Supermemory
    import os

    client = Supermemory(api_key=os.environ.get("SUPERMEMORY_API_KEY"))

    connection = client.connections.create(
        'web-crawler',
        redirect_url='https://yourapp.com/callback',
        container_tags=['user-123', 'website-sync'],
        document_limit=5000,
        metadata={
            'startUrl': 'https://docs.example.com'
        }
    )

    # Web crawler doesn't require OAuth - connection is ready immediately
    print(f'Connection ID: {connection.id}')
    print(f'Connection created: {connection.created_at}')
    # Note: connection.auth_link is None for web-crawler
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    curl -X POST "https://api.supermemory.ai/v3/connections/web-crawler" \
      -H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "redirectUrl": "https://yourapp.com/callback",
        "containerTags": ["user-123", "website-sync"],
        "documentLimit": 5000,
        "metadata": {
          "startUrl": "https://docs.example.com"
        }
      }'

    # Response: {
    #   "id": "conn_wc123",
    #   "redirectsTo": "https://yourapp.com/callback",
    #   "authLink": null,
    #   "expiresIn": null
    # }
    ```
  </Tab>
</Tabs>

### 2. Connection Established

Unlike other connectors, the web crawler doesn't require OAuth authentication. The connection is established immediately upon creation, and crawling begins automatically.

### 3. Monitor Sync Progress

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    // Check connection details
    const connection = await client.connections.getByTags('web-crawler', {
      containerTags: ['user-123', 'website-sync']
    });

    console.log('Start URL:', connection.metadata?.startUrl);
    console.log('Connection created:', connection.createdAt);

    // List synced web pages
    const documents = await client.connections.listDocuments('web-crawler', {
      containerTags: ['user-123', 'website-sync']
    });

    console.log(`Synced ${documents.length} web pages`);
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    # Check connection details
    connection = client.connections.get_by_tags(
        'web-crawler',
        container_tags=['user-123', 'website-sync']
    )

    print(f'Start URL: {connection.metadata.get("startUrl")}')
    print(f'Connection created: {connection.created_at}')

    # List synced web pages
    documents = client.connections.list_documents(
        'web-crawler',
        container_tags=['user-123', 'website-sync']
    )

    print(f'Synced {len(documents)} web pages')
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    # Get connection details by provider and tags
    curl -X POST "https://api.supermemory.ai/v3/connections/web-crawler/connection" \
      -H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"containerTags": ["user-123", "website-sync"]}'

    # Response includes connection details:
    # {
    #   "id": "conn_wc123",
    #   "provider": "web-crawler",
    #   "createdAt": "2024-01-15T10:00:00Z",
    #   "documentLimit": 5000,
    #   "metadata": {"startUrl": "https://docs.example.com", ...}
    # }

    # List synced documents
    curl -X POST "https://api.supermemory.ai/v3/connections/web-crawler/documents" \
      -H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"containerTags": ["user-123", "website-sync"]}'

    # Response: Array of document objects
    # [
    #   {"title": "Home Page", "type": "webpage", "status": "done", "url": "https://docs.example.com"},
    #   {"title": "Getting Started", "type": "webpage", "status": "done", "url": "https://docs.example.com/getting-started"}
    # ]
    ```
  </Tab>
</Tabs>

## Supported Content Types

### Web Pages

* **HTML content** extracted and converted to markdown
* **Same-domain crawling** only (respects hostname boundaries)
* **Robots.txt compliance** - respects disallow rules
* **Content filtering** - only HTML pages (skips non-HTML content)

### URL Requirements

The web crawler only processes valid public URLs:

* Must be a public URL (not localhost, private IPs, or internal domains)
* Must be accessible from the internet
* Must return HTML content (non-HTML files are skipped)

## Sync Mechanism

The web crawler uses **scheduled recrawling** rather than real-time webhooks:

* **Initial Crawl**: Begins immediately after connection creation
* **Scheduled Recrawling**: Automatically recrawls sites that haven't been synced in 7+ days
* **No Real-time Updates**: Unlike other connectors, web crawler doesn't support webhook-based real-time sync

<Note>
  The recrawl schedule is automatically assigned when the connection is created. Sites are recrawled periodically to keep content up to date, but updates are not instantaneous.
</Note>

## Connection Management

### List All Connections

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    // List all web crawler connections
    const connections = await client.connections.list({
      containerTags: ['user-123']
    });

    const webCrawlerConnections = connections.filter(
      conn => conn.provider === 'web-crawler'
    );

    webCrawlerConnections.forEach(conn => {
      console.log(`Start URL: ${conn.metadata?.startUrl}`);
      console.log(`Connection ID: ${conn.id}`);
      console.log(`Created: ${conn.createdAt}`);
    });
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    # List all web crawler connections
    connections = client.connections.list(container_tags=['user-123'])

    web_crawler_connections = [
        conn for conn in connections if conn.provider == 'web-crawler'
    ]

    for conn in web_crawler_connections:
        print(f'Start URL: {conn.metadata.get("startUrl")}')
        print(f'Connection ID: {conn.id}')
        print(f'Created: {conn.created_at}')
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    # List all connections
    curl -X POST "https://api.supermemory.ai/v3/connections/list" \
      -H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"containerTags": ["user-123"]}'

    # Response: [
    #   {
    #     "id": "conn_wc123",
    #     "provider": "web-crawler",
    #     "createdAt": "2024-01-15T10:30:00.000Z",
    #     "documentLimit": 5000,
    #     "metadata": {"startUrl": "https://docs.example.com", ...}
    #   }
    # ]
    ```
  </Tab>
</Tabs>

### Delete Connection

Remove a web crawler connection when no longer needed:

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    // Delete by connection ID
    const result = await client.connections.delete('connection_id_123');
    console.log('Deleted connection:', result.id);

    // Delete by provider and container tags
    const providerResult = await client.connections.deleteByProvider('web-crawler', {
      containerTags: ['user-123']
    });
    console.log('Deleted web crawler connection for user');
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    # Delete by connection ID
    result = client.connections.delete('connection_id_123')
    print(f'Deleted connection: {result.id}')

    # Delete by provider and container tags
    provider_result = client.connections.delete_by_provider(
        'web-crawler',
        container_tags=['user-123']
    )
    print('Deleted web crawler connection for user')
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    # Delete by connection ID
    curl -X DELETE "https://api.supermemory.ai/v3/connections/connection_id_123" \
      -H "Authorization: Bearer $SUPERMEMORY_API_KEY"

    # Delete by provider and container tags
    curl -X DELETE "https://api.supermemory.ai/v3/connections/web-crawler" \
      -H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"containerTags": ["user-123"]}'
    ```
  </Tab>
</Tabs>

<Note>
  Deleting a connection will:

  * Stop all future crawls from the website
  * Keep existing synced documents in Supermemory (they won't be deleted)
  * Remove the connection configuration
</Note>

## Advanced Configuration

### Content Filtering

Control which web pages get synced using the settings API:

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    // Configure intelligent filtering for web content
    await client.settings.update({
      shouldLLMFilter: true,
      includeItems: {
        urlPatterns: ['*docs*', '*documentation*', '*guide*'],
        titlePatterns: ['*Getting Started*', '*API Reference*', '*Tutorial*']
      },
      excludeItems: {
        urlPatterns: ['*admin*', '*private*', '*test*'],
        titlePatterns: ['*Draft*', '*Archive*', '*Old*']
      },
      filterPrompt: "Sync documentation pages, guides, and API references. Skip admin pages, private content, drafts, and archived pages."
    });
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    # Configure intelligent filtering for web content
    client.settings.update(
        should_llm_filter=True,
        include_items={
            'urlPatterns': ['*docs*', '*documentation*', '*guide*'],
            'titlePatterns': ['*Getting Started*', '*API Reference*', '*Tutorial*']
        },
        exclude_items={
            'urlPatterns': ['*admin*', '*private*', '*test*'],
            'titlePatterns': ['*Draft*', '*Archive*', '*Old*']
        },
        filter_prompt="Sync documentation pages, guides, and API references. Skip admin pages, private content, drafts, and archived pages."
    )
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    # Configure intelligent filtering for web content
    curl -X PATCH "https://api.supermemory.ai/v3/settings" \
      -H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "shouldLLMFilter": true,
        "includeItems": {
          "urlPatterns": ["*docs*", "*documentation*", "*guide*"],
          "titlePatterns": ["*Getting Started*", "*API Reference*", "*Tutorial*"]
        },
        "excludeItems": {
          "urlPatterns": ["*admin*", "*private*", "*test*"],
          "titlePatterns": ["*Draft*", "*Archive*", "*Old*"]
        },
        "filterPrompt": "Sync documentation pages, guides, and API references. Skip admin pages, private content, drafts, and archived pages."
      }'
    ```
  </Tab>
</Tabs>

## Security & Compliance

### SSRF Protection

Built-in protection against Server-Side Request Forgery (SSRF) attacks:

* Blocks private IP addresses (10.x.x.x, 192.168.x.x, 172.16-31.x.x)
* Blocks localhost and internal domains
* Blocks cloud metadata endpoints
* Only allows public, internet-accessible URLs

### URL Validation

All URLs are validated before crawling:

* Must be valid HTTP/HTTPS URLs
* Must be publicly accessible
* Must return HTML content
* Response size limited to 10MB

<Warning>
  **Important Limitations:**

  * Requires Scale Plan or Enterprise Plan
  * Only crawls same-domain URLs
  * Scheduled recrawling means updates are not real-time
  * Large websites may take significant time to crawl initially
  * Robots.txt restrictions may prevent crawling some pages
  * URLs must be publicly accessible (no authentication required)
</Warning>