Search API

Perform semantic search across your indexed pages using three different approaches: path-based suggestions, query-based semantic search, or RAG retrieval for LLMs.

Overview

The Search API provides three endpoints for finding relevant content:

• Suggest - Path-based search with spell correction and path sanitization
• Semantic Search - Query-based search using natural language queries
• Retrieve - RAG retrieval API that returns full document content with citations and metadata

The suggest and semantic-search endpoints return suggestions with similarity scores and support metadata filtering and metadata return, while the retrieve endpoint is designed for RAG systems and returns full document content.

Important: All endpoints must be called from your server, never from the browser. Your API keys must remain private on your server. The typical architecture is:

Browser → Your Application Server → deadend.ai API

Suggest (Path-based Search)

Endpoint: POST /v1/suggest

Purpose: Retrieve suggestions based on a URL path (same as original search-api implementation)

Authentication: Required (Bearer token)

Request

curl -X POST https://api.deadend.ai/v1/suggest \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/products/blue-widget",
    "limit": 3,
    "options": {
      "excludeUrls": ["/products/old-widget"],
      "searchThreshold": 0.7
    }
  }'

Parameters

Options Object

Response (200 OK)

{
  "suggestions": [
    {
      "url": "/products/blue-widget-v2",
      "title": "Blue Widget V2",
      "score": 0.92,
      "metadata": {
        "category": "Electronics",
        "in_stock": true,
        "price_range": "premium"
      }
    }
  ],
  "search_id": "a1b2c3d4-e5f6-a7b8-c9d0-e1f2a3b4c5d6"
}

Response Fields

Features

• Path sanitization and spell correction
• Vector similarity search
• Configurable result limit and threshold
• Metadata filtering - Filter results by custom metadata fields before vector search
• Metadata in response - Receive metadata fields in search results for display context

Semantic Search (Query-based Search)

Endpoint: POST /v1/semantic-search

Purpose: Perform semantic search using a natural language query

Authentication: Required (Bearer token)

Request

curl -X POST https://api.deadend.ai/v1/semantic-search \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "blue widget products",
    "limit": 3,
    "options": {
      "excludeUrls": ["/products/old-widget"],
      "searchThreshold": 0.7
    }
  }'

Parameters

Options Object

Response (200 OK)

{
  "suggestions": [
    {
      "url": "/products/blue-widget-v2",
      "title": "Blue Widget V2",
      "score": 0.92,
      "metadata": {
        "category": "Electronics",
        "in_stock": true,
        "price_range": "premium"
      }
    }
  ],
  "search_id": "a1b2c3d4-e5f6-a7b8-c9d0-e1f2a3b4c5d6"
}

Response Fields

Features

• Direct query processing (no path sanitization)
• Vector similarity search
• Same options as suggest endpoint
• Metadata filtering - Filter results by custom metadata fields before vector search
• Metadata in response - Receive metadata fields in search results for display context

Retrieve (RAG Retrieval API)

Endpoint: POST /v1/retrieve

Purpose: Retrieval API for RAG (Retrieval-Augmented Generation) systems. Returns full document content with citations and metadata, designed for building AI chatbots, support agents, or internal search tools.

Authentication: Required (Bearer token)

Request

curl -X POST https://api.deadend.ai/v1/retrieve \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is the return policy for items bought on sale?",
    "top_k": 3,
    "filters": {
      "category": "Support",
      "in_stock": true
    }
  }'

Parameters

Response (200 OK)

{
  "documents": [
    {
      "id": "page_id_123",
      "url": "https://example.com/support/returns",
      "title": "Return Policy | Example Co",
      "content": "Our return policy is 30 days... Sale items are final sale...",
      "score": 0.91,
      "metadata": {
        "category": "Support"
      }
    },
    {
      "id": "page_id_456",
      "url": "https://example.com/support/faq",
      "title": "Frequently Asked Questions",
      "content": "How do I make a return? You can visit our returns portal...",
      "score": 0.85,
      "metadata": {
        "category": "Support"
      }
    }
  ]
}

Response Fields

Features

• Natural language query processing
• Filtered vector search with metadata pre-filtering
• Returns full document content (not just URLs/titles)
• Includes similarity scores for ranking
• Supports custom metadata filtering for scoped searches

Use Cases

• Building AI chatbots that need context from your indexed content
• Creating support agents that search your knowledge base
• Internal search tools that return full content snippets
• RAG systems that need citations and source documents

Error Responses

• 400 Bad Request: Missing query parameter or invalid request body
• 401 Unauthorized: Invalid or missing API key
• 403 Forbidden: Rate limit exceeded
• 500 Internal Server Error: Vectorization or database errors

Metadata Filtering and Return Support

Both /v1/suggest and /v1/semantic-search endpoints now support metadata filtering and metadata return, bringing feature parity with the /v1/retrieve endpoint.

Metadata Filtering

You can filter search results by custom metadata fields using the options.filters parameter. Filters are applied before vector search for optimal performance, and multiple filters are combined with AND logic (all must match).

How It Works:

• Filter keys map to customMetadata.<key> in MongoDB (e.g., {"category": "Electronics"} filters by customMetadata.category)
• Filters are applied before vector search for optimal query performance
• Multiple filters are combined with AND logic (all must match)

Example Request with Filters:

curl -X POST https://api.deadend.ai/v1/suggest \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/products",
    "limit": 5,
    "options": {
      "filters": {
        "category": "Electronics",
        "in_stock": true,
        "price_range": "premium"
      }
    }
  }'

Metadata in Response

Search results now include metadata when available. The metadata field is optional and will be omitted if no metadata exists for a page.

Example Response with Metadata:

{
  "suggestions": [
    {
      "url": "/products/laptop",
      "title": "Gaming Laptop",
      "score": 0.95,
      "metadata": {
        "category": "Electronics",
        "brand": "TechCorp",
        "in_stock": true,
        "price_range": "premium"
      }
    }
  ],
  "search_id": "a1b2c3d4-e5f6-a7b8-c9d0-e1f2a3b4c5d6"
}

Cache Behavior

Important: When filters are present in the request, the search cache is bypassed. This ensures that filtered results are always accurate and not served from potentially unfiltered cached results.

• Without filters: Cache is checked first (fast path, ~10ms)
• With filters: Cache is skipped, full vector search is performed

Backward Compatibility

✅ Fully backward compatible:

• Existing requests without filters continue to work unchanged
• filters field is optional
• metadata field in response is optional (omitted if not present)
• No breaking changes to existing API contracts

Comparison with /v1/retrieve

Understanding Similarity Scores

Scores indicate semantic relevance:

Recommended threshold: 0.7 for most use cases

Example Usage

cURL Examples

Suggest Endpoint:

curl -X POST https://api.deadend.ai/v1/suggest \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/products/blue-widget",
    "limit": 3
  }'

Suggest Endpoint with Metadata Filtering:

curl -X POST https://api.deadend.ai/v1/suggest \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/products",
    "limit": 5,
    "options": {
      "filters": {
        "category": "Electronics",
        "in_stock": true
      }
    }
  }'

Semantic Search Endpoint:

curl -X POST https://api.deadend.ai/v1/semantic-search \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "blue widget products",
    "limit": 3
  }'

Semantic Search Endpoint with Metadata Filtering:

curl -X POST https://api.deadend.ai/v1/semantic-search \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "wireless headphones",
    "limit": 3,
    "options": {
      "filters": {
        "category": "Electronics",
        "in_stock": true,
        "price_range": "mid"
      }
    }
  }'

Retrieve Endpoint (RAG):

curl -X POST https://api.deadend.ai/v1/retrieve \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is the return policy?",
    "top_k": 3,
    "filters": {
      "category": "Support"
    }
  }'

Error Responses

See the Error Handling Guide for detailed error codes and examples.

Common errors include:

• 400 Bad Request - Missing or invalid parameters (e.g., missing path, query, or invalid request body)
• 401 Unauthorized - Invalid API key
• 403 Forbidden - Plan not set or insufficient permissions
• 429 Too Many Requests - Rate limit exceeded
• 500 Internal Server Error - Server error (vectorization or database errors)

Next Steps

• Pages & Content API - Index your content
• Authentication - Get your API keys
• Error Handling - Handle errors gracefully