> ## Documentation Index > Fetch the complete documentation index at: https://beige.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Websearch Plugin {/* Run: pnpm run docs:sync-toolkit */} Multi-provider web search with automatic provider fallback, in-memory caching, and AI-optimized output formats. ## Installation Install this tool individually: ```bash theme={null} beige tools install github:matthias-hausberger/beige-toolkit/plugins/websearch ``` Or install all tools: ```bash theme={null} # From npm beige tools install npm:@matthias-hausberger/beige-toolkit # From GitHub beige tools install github:matthias-hausberger/beige-toolkit ``` ## Configuration | Key | Default | Description | | ----------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `providerPriority` | `[]` | Array of providers. **Array order determines priority** (first = highest). Each entry requires an `apiKey`. Supports `tavily`, `brave`, `exa`, `websearchapi`. Multiple instances of same provider allowed (e.g., separate primary and backup keys). | | `fallbackBehavior` | `try-all` | How to handle provider failures: `"try-all"` continues to next provider, `"fail-fast"` stops on first error. | | `maxProvidersToTry` | `3` | Maximum number of providers to attempt before giving up. | | `timeoutSeconds` | `30` | Request timeout in seconds for all provider API calls. | | `maxResults` | `10` | Default maximum number of results to return per provider (1-20). | | `enableCache` | `true` | Enable in-memory request caching. | | `cacheTTLSeconds` | `300` | Cache time-to-live in seconds (default: 5 minutes). | | `enableLocalExtraction` | `true` | Use local content extraction for providers without built-in content. | | `extractionTimeout` | `15` | Content extraction timeout in seconds. | | `extractionMaxResults` | `3` | Maximum number of results to extract content from (to avoid hitting rate limits). | | `defaultFormat` | `readable` | Default output format: `"readable"`, `"json"`, `"markdown"`. | | `aiOptimized` | `false` | Output optimized for AI consumption (JSON with metadata when `true`). | | `maxRetries` | `3` | Maximum retry attempts for failed requests. | | `retryDelayMs` | `1000` | Delay between retries in milliseconds (exponential backoff: delay \* 2^attempt). | ### Provider Priority Entry Each entry in `providerPriority` supports: | Field | Required | Description | | ---------- | -------- | ------------------------------------------------------------- | | `provider` | ✅ | Provider identifier: `tavily`, `brave`, `exa`, `websearchapi` | | `enabled` | ✅ | Whether this provider is active | | `apiKey` | ✅ | API key for this provider | > **API keys are set directly in config.** No environment variables are read automatically by this plugin. If you prefer not to hardcode keys, you can reference env vars via your config file's variable interpolation (e.g. `"apiKey": "${TAVILY_API_KEY}"`). ## Prerequisites | Provider | Where to get a key | Free tier | | ------------ | ----------------------------------------------------- | ---------------------- | | Tavily | [app.tavily.com](https://app.tavily.com) | 1,000 requests/month | | Brave | [brave.com/search/api](https://brave.com/search/api/) | \~2,000 requests/month | | Exa | [dashboard.exa.ai](https://dashboard.exa.ai) | 1,000 requests/month | | WebSearchAPI | [websearchapi.ai](https://websearchapi.ai) | 2,000 requests/month | ## Config Examples ### Basic setup with Tavily and Brave: ```json5 theme={null} { "plugins": { "websearch": { "config": { "providerPriority": [ { "provider": "tavily", "enabled": true, "apiKey": "tvly-..." }, { "provider": "brave", "enabled": true, "apiKey": "BSA-..." } ] } } } } ``` ### Using environment variable interpolation for keys: ```json5 theme={null} { "plugins": { "websearch": { "config": { "providerPriority": [ { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY}" }, { "provider": "brave", "enabled": true, "apiKey": "${BRAVE_API_KEY}" } ] } } } } ``` ### High availability with a backup Tavily key: ```json5 theme={null} { "plugins": { "websearch": { "config": { "providerPriority": [ { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY}" }, // Primary { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY_2}" }, // Backup { "provider": "brave", "enabled": true, "apiKey": "${BRAVE_API_KEY}" } // Fallback ], "maxProvidersToTry": 3, "fallbackBehavior": "try-all", "enableCache": true } } } } ``` ### Cost-optimized with longer cache: ```json5 theme={null} { "plugins": { "websearch": { "config": { "providerPriority": [ { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY}" } ], "enableCache": true, "cacheTTLSeconds": 600 } } } } ``` ### AI-optimized output for direct LLM consumption: ```json5 theme={null} { "plugins": { "websearch": { "config": { "providerPriority": [ { "provider": "tavily", "enabled": true, "apiKey": "${TAVILY_API_KEY}" } ], "defaultFormat": "json", "aiOptimized": true } } } } ``` ## Usage The websearch plugin provides a `websearch` tool that agents can use to perform web searches. ### Basic search (tries providers in priority order) ``` websearch search ``` Tries providers in array order until first success or all providers exhausted. ### Search with specific provider ``` websearch search --provider tavily websearch search --provider brave ``` Use a specific provider directly (bypasses priority system). Available providers: `tavily`, `brave`, `exa`, `websearchapi`. ### Limit results ``` websearch search --count 5 ``` Limit number of results (1-20, default: 10). ### Custom output format ``` websearch search --format json websearch search --format markdown ``` Output format options: * `--format readable` - Human-friendly text with titles, URLs, snippets (default) * `--format json` - Machine-readable JSON with full metadata * `--format markdown` - Markdown format with frontmatter for documentation ### AI-generated direct answer ``` websearch answer ``` Returns direct answer with source citations (currently Tavily-only). ### Extract content from URL ``` websearch extract ``` Uses local content extraction (Mozilla Readability) to extract readable content as Markdown. ## Commands Reference | Command | Description | | ---------------------------------- | ---------------------------------------------------------------- | | `search ` | Search web (tries providers in priority order) | | `search --provider ` | Use specific provider (`tavily`, `brave`, `exa`, `websearchapi`) | | `search --count ` | Limit results (1-20, default: 10) | | `search --format ` | Output format: `readable`, `json`, `markdown` | | `answer ` | Get AI-generated direct answer with citations (Tavily) | | `extract ` | Extract content from URL (local extraction) | ## Architecture ### Priority-Based Fallback ``` Query Request ↓ Check Cache (normalized) ↓ hit → Return Cached Results (fast) ↓ miss → Try Provider[0] (highest priority) ↓ Success → Return Results ↓ Failure (retryable) → Retry (up to 3 times with backoff) ↓ Failure (non-retryable) or Exhausted Retries ↓ Try Provider[1] (next in array) ↓ Success → Return Results ↓ Failure (retryable) → Retry ↓ Failure (non-retryable) or Exhausted Retries ↓ Try Provider[2] ... ↓ Continue until maxProvidersToTry or all providers exhausted ↓ All Failed → Return Aggregate Error ``` ### Circuit Breaker Pattern Each provider has a circuit breaker that: * Opens after 3 consecutive failures * Remains open for 1 minute * Prevents cascading failures * Resets on success or after timeout * State stored in-memory (clean slate on restart) ### Request Caching In-memory cache with TTL: * Normalizes query (lowercase, trimmed) * Returns cached results if hit * Expires after TTL (default: 5 minutes) * Automatic cleanup on cache miss * Reduces API usage by 50-90% for repeated queries ### Content Extraction Pipeline ``` URL → Fetch (browser headers) → HTML → JSDOM → Readability → Turndown → Markdown ``` **Mozilla Readability**: Extracts main article content, removes clutter\ **Turndown + GFM**: Converts HTML to clean Markdown with code block support ## Features * ✅ **Multi-provider support**: Tavily, Brave, Exa (planned), WebSearchAPI (planned) * ✅ **Provider priority system**: Array order determines priority * ✅ **Automatic fallback**: Try next provider on failure (configurable: try-all or fail-fast) * ✅ **Circuit breaker pattern**: Prevents cascading failures * ✅ **Request caching**: In-memory caching with configurable TTL * ✅ **Retry with exponential backoff**: Handle transient errors gracefully * ✅ **Local content extraction**: Mozilla Readability + Turndown for Markdown * ✅ **Multiple output formats**: Human-readable, JSON, Markdown * ✅ **AI-optimized output**: JSON format with rich metadata for direct LLM consumption * ✅ **Config-only API keys**: No implicit environment variable reads ## Performance ### Caching Efficiency * **Memory overhead**: \~20 bytes per cached query * **Cache hit speedup**: 250x faster (\<1ms vs 250ms API call) * **API cost reduction**: 50-90% for repeated queries (typical agent workloads) ### Provider Priority Efficiency * **Configuration overhead**: Zero (array order, no sorting at runtime) * **Runtime overhead**: O(n) where n = providers tried (usually 1 due to early exit) * **Automatic fallback**: Self-healing without manual intervention ## License MIT