Websearch Plugin

Multi-provider web search with automatic provider fallback, in-memory caching, and AI-optimized output formats.

Installation

Install this tool individually:

beige tools install github:matthias-hausberger/beige-toolkit/plugins/websearch

Or install all tools:

# From npm
beige tools install npm:@matthias-hausberger/beige-toolkit

# From GitHub
beige tools install github:matthias-hausberger/beige-toolkit

Configuration

Key	Default	Description
`providerPriority`	`[]`	Array of providers. Array order determines priority (first = highest). Each entry requires an `apiKey`. Supports `tavily`, `brave`, `exa`, `websearchapi`. Multiple instances of same provider allowed (e.g., separate primary and backup keys).
`fallbackBehavior`	`try-all`	How to handle provider failures: `"try-all"` continues to next provider, `"fail-fast"` stops on first error.
`maxProvidersToTry`	`3`	Maximum number of providers to attempt before giving up.
`timeoutSeconds`	`30`	Request timeout in seconds for all provider API calls.
`maxResults`	`10`	Default maximum number of results to return per provider (1-20).
`enableCache`	`true`	Enable in-memory request caching.
`cacheTTLSeconds`	`300`	Cache time-to-live in seconds (default: 5 minutes).
`enableLocalExtraction`	`true`	Use local content extraction for providers without built-in content.
`extractionTimeout`	`15`	Content extraction timeout in seconds.
`extractionMaxResults`	`3`	Maximum number of results to extract content from (to avoid hitting rate limits).
`defaultFormat`	`readable`	Default output format: `"readable"`, `"json"`, `"markdown"`.
`aiOptimized`	`false`	Output optimized for AI consumption (JSON with metadata when `true`).
`maxRetries`	`3`	Maximum retry attempts for failed requests.
`retryDelayMs`	`1000`	Delay between retries in milliseconds (exponential backoff: delay * 2^attempt).

Provider Priority Entry

Each entry in providerPriority supports:

Field	Required	Description
`provider`	✅	Provider identifier: `tavily`, `brave`, `exa`, `websearchapi`
`enabled`	✅	Whether this provider is active
`apiKey`	✅	API key for this provider

API keys are set directly in config. No environment variables are read automatically by this plugin. If you prefer not to hardcode keys, you can reference env vars via your config file’s variable interpolation (e.g. "apiKey": "${TAVILY_API_KEY}").

Prerequisites

Provider	Where to get a key	Free tier
Tavily	app.tavily.com	1,000 requests/month
Brave	brave.com/search/api	~2,000 requests/month
Exa	dashboard.exa.ai	1,000 requests/month
WebSearchAPI	websearchapi.ai	2,000 requests/month

Config Examples

Basic setup with Tavily and Brave:

{
  "plugins": {
    "websearch": {
      "config": {
        "providerPriority": [
          { "provider": "tavily", "enabled": true, "apiKey": "tvly-..." },
          { "provider": "brave",  "enabled": true, "apiKey": "BSA-..." }
        ]
      }
    }
  }
}

Using environment variable interpolation for keys:

{
  "plugins": {
    "websearch": {
      "config": {
        "providerPriority": [
          { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY}" },
          { "provider": "brave",  "enabled": true, "apiKey": "${BRAVE_API_KEY}" }
        ]
      }
    }
  }
}

High availability with a backup Tavily key:

{
  "plugins": {
    "websearch": {
      "config": {
        "providerPriority": [
          { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY}" },      // Primary
          { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY_2}" },    // Backup
          { "provider": "brave",  "enabled": true, "apiKey": "${BRAVE_API_KEY}" }        // Fallback
        ],
        "maxProvidersToTry": 3,
        "fallbackBehavior": "try-all",
        "enableCache": true
      }
    }
  }
}

Cost-optimized with longer cache:

{
  "plugins": {
    "websearch": {
      "config": {
        "providerPriority": [
          { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY}" }
        ],
        "enableCache": true,
        "cacheTTLSeconds": 600
      }
    }
  }
}

AI-optimized output for direct LLM consumption:

{
  "plugins": {
    "websearch": {
      "config": {
        "providerPriority": [
          { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY}" }
        ],
        "defaultFormat": "json",
        "aiOptimized": true
      }
    }
  }
}

Usage

The websearch plugin provides a websearch tool that agents can use to perform web searches.

Basic search (tries providers in priority order)

websearch search <query>

Tries providers in array order until first success or all providers exhausted.

Search with specific provider

websearch search <query> --provider tavily
websearch search <query> --provider brave

Use a specific provider directly (bypasses priority system). Available providers: tavily, brave, exa, websearchapi.

Limit results

websearch search <query> --count 5

Limit number of results (1-20, default: 10).

Custom output format

websearch search <query> --format json
websearch search <query> --format markdown

Output format options:

--format readable - Human-friendly text with titles, URLs, snippets (default)
--format json - Machine-readable JSON with full metadata
--format markdown - Markdown format with frontmatter for documentation

AI-generated direct answer

websearch answer <query>

Returns direct answer with source citations (currently Tavily-only).

Extract content from URL

websearch extract <url>

Uses local content extraction (Mozilla Readability) to extract readable content as Markdown.

Commands Reference

Command	Description
`search <query>`	Search web (tries providers in priority order)
`search <query> --provider <name>`	Use specific provider (`tavily`, `brave`, `exa`, `websearchapi`)
`search <query> --count <n>`	Limit results (1-20, default: 10)
`search <query> --format <fmt>`	Output format: `readable`, `json`, `markdown`
`answer <query>`	Get AI-generated direct answer with citations (Tavily)
`extract <url>`	Extract content from URL (local extraction)

Architecture

Priority-Based Fallback

Query Request
    ↓
Check Cache (normalized)
    ↓ hit → Return Cached Results (fast)
    ↓ miss → Try Provider[0] (highest priority)
    ↓ Success → Return Results
    ↓ Failure (retryable) → Retry (up to 3 times with backoff)
    ↓ Failure (non-retryable) or Exhausted Retries
    ↓ Try Provider[1] (next in array)
    ↓ Success → Return Results
    ↓ Failure (retryable) → Retry
    ↓ Failure (non-retryable) or Exhausted Retries
    ↓ Try Provider[2] ...
    ↓ Continue until maxProvidersToTry or all providers exhausted
    ↓ All Failed → Return Aggregate Error

Circuit Breaker Pattern

Each provider has a circuit breaker that:

Opens after 3 consecutive failures
Remains open for 1 minute
Prevents cascading failures
Resets on success or after timeout
State stored in-memory (clean slate on restart)

Request Caching

In-memory cache with TTL:

Normalizes query (lowercase, trimmed)
Returns cached results if hit
Expires after TTL (default: 5 minutes)
Automatic cleanup on cache miss
Reduces API usage by 50-90% for repeated queries

Content Extraction Pipeline

URL → Fetch (browser headers) → HTML → JSDOM → Readability → Turndown → Markdown

Mozilla Readability: Extracts main article content, removes clutter
Turndown + GFM: Converts HTML to clean Markdown with code block support

Features

✅ Multi-provider support: Tavily, Brave, Exa (planned), WebSearchAPI (planned)
✅ Provider priority system: Array order determines priority
✅ Automatic fallback: Try next provider on failure (configurable: try-all or fail-fast)
✅ Circuit breaker pattern: Prevents cascading failures
✅ Request caching: In-memory caching with configurable TTL
✅ Retry with exponential backoff: Handle transient errors gracefully
✅ Local content extraction: Mozilla Readability + Turndown for Markdown
✅ Multiple output formats: Human-readable, JSON, Markdown
✅ AI-optimized output: JSON format with rich metadata for direct LLM consumption
✅ Config-only API keys: No implicit environment variable reads

Performance

Caching Efficiency

Memory overhead: ~20 bytes per cached query
Cache hit speedup: 250x faster (<1ms vs 250ms API call)
API cost reduction: 50-90% for repeated queries (typical agent workloads)

Provider Priority Efficiency

Configuration overhead: Zero (array order, no sorting at runtime)
Runtime overhead: O(n) where n = providers tried (usually 1 due to early exit)
Automatic fallback: Self-healing without manual intervention

License

MIT

​Installation

​Configuration

​Provider Priority Entry

​Prerequisites

​Config Examples

​Basic setup with Tavily and Brave:

​Using environment variable interpolation for keys:

​High availability with a backup Tavily key:

​Cost-optimized with longer cache:

​AI-optimized output for direct LLM consumption:

​Usage

​Basic search (tries providers in priority order)

​Search with specific provider

​Limit results

​Custom output format

​AI-generated direct answer

​Extract content from URL

​Commands Reference

​Architecture

​Priority-Based Fallback

​Circuit Breaker Pattern

​Request Caching

​Content Extraction Pipeline

​Features

​Performance

​Caching Efficiency

​Provider Priority Efficiency

​License