Add live web search to a Haystack agent

Haystack 2.x (deepset's LLM pipeline framework) builds pipelines from @component blocks you connect by named input/output sockets, and runs tool-using Agent loops. Give it live web search two ways: write a tiny custom @component over the REST API with a free key, or load Superhighway's MCP server with MCPToolset so the agent pays per call via an x402 wallet.

Two paths

Path 1 — a custom @component (REST API + free key)

pip install haystack-ai httpx

Create a free-tier key at superhighway.walls.sh — no credit card. A Haystack component is a class decorated with @component; its run() method declares output sockets with @component.output_types(...) and returns a dict keyed by those names. Here the component calls /search and emits a results list.

from haystack import component
import httpx

@component
class SuperhighwaySearch:
    def __init__(self, api_key: str):
        self.api_key = api_key

    @component.output_types(results=list)
    def run(self, query: str, limit: int = 5):
        response = httpx.get(
            "https://superhighway.walls.sh/search",
            params={"q": query, "limit": limit},
            headers={"X-Api-Key": self.api_key},
        )
        response.raise_for_status()
        results = response.json()["results"]
        return {"results": results}

Now wire it into a Pipeline: search → build a chat prompt from the results → answer with a generator. Connections are made by "<component>.<socket>" names. ChatPromptBuilder exposes its rendered messages on the prompt output socket, which feeds the generator's messages input.

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

template = [ChatMessage.from_user(
    "Based on these search results:\n"
    "{% for r in results %}{{ r.title }}: {{ r.description }}\n{% endfor %}\n"
    "Answer: {{ question }}"
)]

pipeline = Pipeline()
pipeline.add_component("search", SuperhighwaySearch(api_key="YOUR_FREE_API_KEY"))
pipeline.add_component("prompt", ChatPromptBuilder(
    template=template,
    required_variables=["results", "question"],
))
pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))

# Connect components by socket name
pipeline.connect("search.results", "prompt.results")
pipeline.connect("prompt.prompt", "llm.messages")

result = pipeline.run({
    "search": {"query": "latest AI agent frameworks 2024"},
    "prompt": {"question": "What are the top agent frameworks?"},
})
print(result["llm"]["replies"][0].text)

Two details that trip people up: the ChatPromptBuilder output socket is named prompt (so the connection is prompt.prompt → llm.messages), and any Jinja variable that doesn't come from a connection — here question — is passed at run() time and should be listed in required_variables so the pipeline declares it as an input. The generator returns its output on replies as a list of ChatMessage; read .text.

Path 2 — MCPToolset for an autonomous Agent (x402 wallet)

Haystack ships first-party MCP support in the mcp-haystack integration. MCPToolset connects to an MCP server over stdio and loads all of its tools into one object you hand to a Haystack Agent — no per-tool wiring. Point it at npx -y superhighway-mcp and the agent gets web_search, news_search, image_search, scrape, and research at once.

pip install haystack-ai mcp-haystack

import os
from haystack_integrations.tools.mcp import MCPToolset, StdioServerInfo
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.agents import Agent
from haystack.dataclasses import ChatMessage

server_info = StdioServerInfo(
    command="npx",
    args=["-y", "superhighway-mcp"],
    env={
        "AGENT_PRIVATE_KEY": os.environ["AGENT_PRIVATE_KEY"],
        "X402_NETWORK": "base",
    },
)

# Loads every tool the MCP server exposes into one toolset
toolset = MCPToolset(server_info=server_info)

agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-4o"),
    tools=toolset,
    exit_conditions=["text"],
)

response = agent.run(
    messages=[ChatMessage.from_user(
        "What are the latest developments in AI agent frameworks?"
    )]
)
print(response["messages"][-1].text)

AGENT_PRIVATE_KEY is the private key of a wallet holding USDC on Base. Each call costs $0.001 and settles automatically per request via the x402 protocol — no signup, no API key, no human in the loop. To load only specific tools instead of all five, pass tool_names=["web_search", "research"] to MCPToolset. For a single tool rather than the whole set, use MCPTool(name="web_search", server_info=server_info) from the same module.

Single tool over the REST path

If you'd rather give an Agent a function-backed tool with the free API key (no wallet), wrap the REST call in a Haystack Tool built from your component or a plain function — the Agent accepts a list mixing MCP tools and Tool objects. The @component in Path 1 also works standalone: SuperhighwaySearch(api_key=...).run(query="...") returns {"results": [...]} directly, no pipeline required.

Which tools are available

Full API reference: /openapi.json. Get a free API key or learn the x402 wallet flow.