How to integrate Archive.org MCP with CrewAI

This guide walks you through connecting Archive.org to CrewAI using the Composio tool router. By the end, you'll have a working Archive.org agent that can search collections, download item files, inspect wayback captures, fetch iiif manifests, and manage item metadata through natural language commands. This guide will help you understand how to give your CrewAI agent real control over a Archive.org account through Composio's Archive.org MCP server. Before we dive in, let's take a quick look at the key ideas and tools involved.

Get started for free Get a demo

Archive.org

Api Key

Archive.org is the Internet Archive's digital library of websites, books, audio, video, and images. Use it to preserve, explore, and access historical web captures and public-domain media.

26 Tools

Managed auth

Connect Archive.org without auth hassles

We manage OAuth, API keys, token refresh, and scopes — you just build.

Try for free

Introduction

This guide will help you understand how to give your CrewAI agent real control over a Archive.org account through Composio's Archive.org MCP server.

Before we dive in, let's take a quick look at the key ideas and tools involved.

Also integrate Archive.org with

OpenAI Agents SDK Claude Agent SDK Claude Code Claude Cowork Codex OpenClaw Hermes CLI Google ADK LangChain Vercel AI SDK Mastra AI LlamaIndex

TL;DR

Here's what you'll learn:

Get a Composio API key and configure your Archive.org connection
Set up CrewAI with an MCP enabled agent
Create a Tool Router session or standalone MCP server for Archive.org
Build a conversational loop where your agent can execute Archive.org operations

What is CrewAI?

CrewAI is a powerful framework for building multi-agent AI systems. It provides primitives for defining agents with specific roles, creating tasks, and orchestrating workflows through crews.

Key features include:

Agent Roles: Define specialized agents with specific goals and backstories
Task Management: Create tasks with clear descriptions and expected outputs
Crew Orchestration: Combine agents and tasks into collaborative workflows
MCP Integration: Connect to external tools through Model Context Protocol

What is the Archive.org MCP server, and what's possible with it?

The Archive.org MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Archive.org account. It provides structured and secure access to archived content, Wayback captures, item records, usage analytics, and account tasks, so your agent can search collections, retrieve files, manage items, inspect activity, and handle reviews on your behalf.

Content search and retrieval: Have your agent search Archive.org by keywords and metadata, inspect complete item records, and download public files.
Wayback history research: Instruct your agent to find the closest archived version of a web page or list available captures for a website.
Item and file management: Direct your agent to check identifier availability, upload, copy, or delete files, update item metadata, and request new derivatives.
Images, manifests, and analytics: Let the agent inspect IIIF image details and manifests, render selected image regions, and review item views, history, geography, and referrers.
Account tasks and reviews: Have your agent check your account details, monitor catalog tasks and limits, read task logs, and create, update, retrieve, or delete your reviews.

What is the Composio tool router, and how does it fit here?

What is Composio SDK?

Composio's Composio SDK helps agents find the right tools for a task at runtime. You can plug in multiple toolkits (like Gmail, HubSpot, and GitHub), and the agent will identify the relevant app and action to complete multi-step workflows. This can reduce token usage and improve the reliability of tool calls. Read more here: Getting started with Composio SDK

The tool router generates a secure MCP URL that your agents can access to perform actions.

How the Composio SDK works

The Composio SDK follows a three-phase workflow:

Discovery: Searches for tools matching your task and returns relevant toolkits with their details.
Authentication: Checks for active connections. If missing, creates an auth config and returns a connection URL via Auth Link.
Execution: Executes the action using the authenticated connection.

Step-by-step Guide

Step by step08 STEPS

Prerequisites

Before starting, make sure you have:

Python 3.9 or higher
A Composio account and API key
A Archive.org connection authorized in Composio
An OpenAI API key for the CrewAI LLM
Basic familiarity with Python

Getting API Keys for OpenAI and Composio

OpenAI API Key

Go to the OpenAI dashboard and create an API key. You'll need credits to use the models, or you can connect to another model provider.
Keep the API key safe.

Composio API Key

Log in to the Composio dashboard.
Navigate to your API settings and generate a new API key.
Store this key securely as you'll need it for authentication.

Install dependencies

bash

pip install composio crewai crewai-tools[mcp] python-dotenv

What's happening:

composio connects your agent to Archive.org via MCP
crewai provides Agent, Task, Crew, and LLM primitives
crewai-tools[mcp] includes MCP helpers
python-dotenv loads environment variables from .env

Set up environment variables

bash

COMPOSIO_API_KEY=your_composio_api_key_here
USER_ID=your_user_id_here
OPENAI_API_KEY=your_openai_api_key_here

Create a .env file in your project root.

What's happening:

COMPOSIO_API_KEY authenticates with Composio
USER_ID scopes the session to your account
OPENAI_API_KEY lets CrewAI use your chosen OpenAI model

Import dependencies

python

import os
from composio import Composio
from crewai import Agent, Task, Crew
from crewai_tools import MCPServerAdapter
import dotenv

dotenv.load_dotenv()

COMPOSIO_API_KEY = os.getenv("COMPOSIO_API_KEY")
COMPOSIO_USER_ID = os.getenv("COMPOSIO_USER_ID")

if not COMPOSIO_API_KEY:
    raise ValueError("COMPOSIO_API_KEY is not set")
if not COMPOSIO_USER_ID:
    raise ValueError("COMPOSIO_USER_ID is not set")

What's happening:

CrewAI classes define agents and tasks, and run the workflow
MCPServerHTTP connects the agent to an MCP endpoint
Composio will give you a short lived Archive.org MCP URL

Create a Composio Tool Router session for Archive.org

python

composio_client = Composio(api_key=COMPOSIO_API_KEY)
session = composio_client.create(user_id=COMPOSIO_USER_ID, toolkits=["archive_org"])

url = session.mcp.url

What's happening:

You create a Archive.org only session through Composio
Composio returns an MCP HTTP URL that exposes Archive.org tools

Initialize the MCP Server

python

server_params = {
    "url": url,
    "transport": "streamable-http",
    "headers": {"x-api-key": COMPOSIO_API_KEY},
}

with MCPServerAdapter(server_params) as tools:
    agent = Agent(
        role="Search Assistant",
        goal="Help users search the internet effectively",
        backstory="You are a helpful assistant with access to search tools.",
        tools=tools,
        verbose=False,
        max_iter=10,
    )

What's Happening:

Server Configuration: The code sets up connection parameters including the MCP server URL, streamable HTTP transport, and Composio API key authentication.
MCP Adapter Bridge: MCPServerAdapter acts as a context manager that converts Composio MCP tools into a CrewAI-compatible format.
Agent Setup: Creates a CrewAI Agent with a defined role (Search Assistant), goal (help with internet searches), and access to the MCP tools.
Configuration Options: The agent includes settings like verbose=False for clean output and max_iter=10 to prevent infinite loops.
Dynamic Tool Usage: Once created, the agent automatically accesses all Composio Search tools and decides when to use them based on user queries.

Create a CLI Chatloop and define the Crew

python

print("Chat started! Type 'exit' or 'quit' to end.\n")

conversation_context = ""

while True:
    user_input = input("You: ").strip()

    if user_input.lower() in ["exit", "quit", "bye"]:
        print("\nGoodbye!")
        break

    if not user_input:
        continue

    conversation_context += f"\nUser: {user_input}\n"
    print("\nAgent is thinking...\n")

    task = Task(
        description=(
            f"Conversation history:\n{conversation_context}\n\n"
            f"Current request: {user_input}"
        ),
        expected_output="A helpful response addressing the user's request",
        agent=agent,
    )

    crew = Crew(agents=[agent], tasks=[task], verbose=False)
    result = crew.kickoff()
    response = str(result)

    conversation_context += f"Agent: {response}\n"
    print(f"Agent: {response}\n")

What's Happening:

Interactive CLI Setup: The code creates an infinite loop that continuously prompts for user input and maintains the entire conversation history in a string variable.
Input Validation: Empty inputs are ignored to prevent processing blank messages and keep the conversation clean.
Context Building: Each user message is appended to the conversation context, which preserves the full dialogue history for better agent responses.
Dynamic Task Creation: For every user input, a new Task is created that includes both the full conversation history and the current request as context.
Crew Execution: A Crew is instantiated with the agent and task, then kicked off to process the request and generate a response.
Response Management: The agent's response is converted to a string, added to the conversation context, and displayed to the user, maintaining conversational continuity.

Complete Code

Here's the complete code to get you started with Archive.org and CrewAI:

python

from crewai import Agent, Task, Crew, LLM
from crewai_tools import MCPServerAdapter
from composio import Composio
from dotenv import load_dotenv
import os

load_dotenv()

GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY")
COMPOSIO_API_KEY = os.getenv("COMPOSIO_API_KEY")
COMPOSIO_USER_ID = os.getenv("COMPOSIO_USER_ID")

if not GOOGLE_API_KEY:
    raise ValueError("GOOGLE_API_KEY is not set in the environment.")
if not COMPOSIO_API_KEY:
    raise ValueError("COMPOSIO_API_KEY is not set in the environment.")
if not COMPOSIO_USER_ID:
    raise ValueError("COMPOSIO_USER_ID is not set in the environment.")

# Initialize Composio and create a session
composio = Composio(api_key=COMPOSIO_API_KEY)
session = composio.create(
    user_id=COMPOSIO_USER_ID,
    toolkits=["archive_org"],
)
url = session.mcp.url

# Configure LLM
llm = LLM(
    model="gpt-5",
    api_key=os.getenv("OPENAI_API_KEY"),
)

server_params = {
    "url": url,
    "transport": "streamable-http",
    "headers": {"x-api-key": COMPOSIO_API_KEY},
}

with MCPServerAdapter(server_params) as tools:
    agent = Agent(
        role="Search Assistant",
        goal="Help users with internet searches",
        backstory="You are an expert assistant with access to Composio Search tools.",
        tools=tools,
        llm=llm,
        verbose=False,
        max_iter=10,
    )

    print("Chat started! Type 'exit' or 'quit' to end.\n")

    conversation_context = ""

    while True:
        user_input = input("You: ").strip()

        if user_input.lower() in ["exit", "quit", "bye"]:
            print("\nGoodbye!")
            break

        if not user_input:
            continue

        conversation_context += f"\nUser: {user_input}\n"
        print("\nAgent is thinking...\n")

        task = Task(
            description=(
                f"Conversation history:\n{conversation_context}\n\n"
                f"Current request: {user_input}"
            ),
            expected_output="A helpful response addressing the user's request",
            agent=agent,
        )

        crew = Crew(agents=[agent], tasks=[task], verbose=False)
        result = crew.kickoff()
        response = str(result)

        conversation_context += f"Agent: {response}\n"
        print(f"Agent: {response}\n")

Conclusion

You now have a CrewAI agent connected to Archive.org through Composio's Tool Router. The agent can perform Archive.org operations through natural language commands.

Next steps:

Add role-specific instructions to customize agent behavior
Plug in more toolkits for multi-app workflows
Chain tasks for complex multi-step operations

TOOLS

Supported Tools

Every Archive.org action and event your agent gets out of the box.

Advanced Search Archive.org Items

Search Archive.

Check Identifier Availability

Check whether an Archive.

Check Archive.org Upload Limit

Check whether Archive.

Copy Item File

Copy one Archive.

Delete Item File

Submit asynchronous deletion of one named file from an Archive.

Delete My Archive.org Review

Permanently delete the connected Archive.

Derive Archive.org Item

Queue Archive.

Download Item File

Download one public file from an Archive.

Find Wayback Snapshots

Find the closest archived Wayback Machine snapshot for each requested URL, using one before, after, or either direction for the batch.

Get Connected Archive.org User

Return the authenticated Archive.

Get Detailed Archive.org Views

Get bounded geographic view analytics and, for items, top referrers over a recent window or date range.

Get IIIF Image Info

Get normalized IIIF Image API capabilities through Archive.

Get IIIF Manifest

Get a compact IIIF Presentation manifest for an Archive.

Get Item Metadata

Get an Archive.

Get Item View Counts

Get all-time, last-30-day, and last-7-day view counts for one or more Archive.

Get Item View History

Get a bounded daily view history for one Archive.

Get My Review

Get the connected Archive.

Get Archive.org Task Log

Download a plain-text log for a task visible to the connected account.

Get Task Rate Limits

Report the connected account's current task capacity and in-flight counts for one Archive.

List Archive.org Tasks

Return any documented combination of the connected account's task summary, active catalog, and resource-authorized history, one page at a time.

List Wayback Captures

Return one bounded structured CDX page of Wayback captures for a URL or host.

Render IIIF Image

Render a bounded crop, resize, rotation, quality, and format from a fixed Archive.

Search Archive.org Items

Search Archive.

Update Item Metadata

Apply JSON Patch changes to one Archive.

Upload Item File

Create or replace one file through Archive.

Upsert My Archive.org Review

Create or overwrite the connected Archive.

FRAMEWORKS

How to build Archive.org MCP Agent with another framework

OpenAI Agents SDK

Use Archive.org MCP with OpenAI Agents SDK

Claude Agent SDK

Use Archive.org MCP with Claude Agent SDK

Claude Code

Use Archive.org MCP with Claude Code

Claude Cowork

Use Archive.org MCP with Claude Cowork

Codex

Use Archive.org MCP with Codex

OpenClaw

Use Archive.org MCP with OpenClaw

Hermes

Use Archive.org MCP with Hermes

CLI

Use Archive.org MCP with CLI

Google ADK

Use Archive.org MCP with Google ADK

LangChain

Use Archive.org MCP with LangChain

Vercel AI SDK

Use Archive.org MCP with Vercel AI SDK

Mastra AI

Use Archive.org MCP with Mastra AI

LlamaIndex

Use Archive.org MCP with LlamaIndex

MORE TOOLKITS

Explore Other Toolkits

Toolkit marketplace

ActiveMerge

Api Key

ActiveMerge is a document automation platform for generating documents, images, merged files, and e-signature workflows. It helps teams turn templates and stored files into finished, shareable documents faster.

Affinda

Api Key

Affinda is an AI-powered document processing platform that automates data extraction from resumes, invoices, and more. It streamlines document-heavy workflows by turning files into structured, actionable data.

Airparser

Api Key

Airparser is an AI-powered parser for extracting structured data from emails, documents, PDFs, and files. It turns messy inbound content into clean fields your apps and workflows can use.

Api2pdf

Api Key

Api2Pdf is a REST API for generating PDFs from HTML, URLs, and documents using powerful engines like wkhtmltopdf and Headless Chrome. It streamlines document conversion and automation for developers and businesses.

FAQ

Frequently asked questions

With a standalone Archive.org MCP server, the agents and LLMs can only access a fixed set of Archive.org tools tied to that server. However, with the Composio Tool Router, agents can dynamically load tools from Archive.org and many other apps based on the task at hand, all through a single MCP endpoint.

Yes, you can. CrewAI fully supports MCP integration. You get structured tool calling, message history handling, and model orchestration while Tool Router takes care of discovering and serving the right Archive.org tools.

Yes, absolutely. You can configure which Archive.org scopes and actions are allowed when connecting your account to Composio. You can also bring your own OAuth credentials or API configuration so you keep full control over what the agent can do.

All sensitive data such as tokens, keys, and configuration is fully encrypted at rest and in transit. Composio is SOC 2 Type 2 compliant and follows strict security practices so your Archive.org data and credentials are handled as safely as possible.

Start with Archive.org.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Archive.org tool your agent needs.Free to start.

Start building

Customer stories

Teams ship real agents on Composio

GET STARTED FOR FREE BOOK A DEMO

Karan skipped his own birthday party to fix our critical issue. It was 10pm and he diverted his Waymo to help us instead. This really sets the bar — it shows the commitment you need when users rely on your software.

Harsha Gaddipati

Co-founder, Slashy

A lot of students tell us that the moment their connected tools start talking to each other inside Opennote feels almost magical. The agent just knows them, and it's immensely helped keep new users on the platform.

Abhi Arya

Co-founder, Opennote

We chose Composio over Pipedream because it delivered depth where it mattered — niche tools and tricky edge cases other platforms simply ignored. That gave us confidence to scale without compromising.

Nirman Dave

CEO, Zams

As a solo builder, shipping fast is life or death. The only way I can outcompete incumbents is by outmanoeuvring them. Getting bogged down managing agent auth would have been a death sentence.

Ryan Yu

Founder, Extra Thursday

Before Composio, adding tool integrations was slow and resource-intensive. Each one could take weeks of engineering time, and maintaining them meant constantly keeping up with API changes.

Tomisin Jenrola

Founder & CEO, SwarmZero

With hands-on help from their founder, we integrated Gmail and Google Drive in just 30 minutes. This level of personal support and commitment is exactly what startups should strive for.

Jerome Leclanche

Co-Founder, Ingram Technologies

How to integrate Archive.org MCP with CrewAI

Connect Archive.org without auth hassles

Introduction

Also integrate Archive.org with

TL;DR

What is CrewAI?

What is the Archive.org MCP server, and what's possible with it?

What is the Composio tool router, and how does it fit here?

What is Composio SDK?

How the Composio SDK works

Step-by-step Guide

Prerequisites

Getting API Keys for OpenAI and Composio

Install dependencies

Set up environment variables

Import dependencies

Create a Composio Tool Router session for Archive.org

Initialize the MCP Server

Create a CLI Chatloop and define the Crew

Complete Code

Conclusion

Supported Tools

How to build Archive.org MCP Agent with another framework

OpenAI Agents SDK

Claude Agent SDK

Claude Code

Claude Cowork

Codex

OpenClaw

Hermes

CLI

Google ADK

LangChain

Vercel AI SDK

Mastra AI

LlamaIndex

Explore Other Toolkits

ActiveMerge

Affinda

Airparser

Api2pdf

Frequently asked questions

What are the differences in Tool Router MCP and Archive.org MCP?+

Can I use Tool Router MCP with CrewAI?+

Can I manage the permissions and scopes for Archive.org while using Tool Router?+

How safe is my data with Composio Tool Router?+

Start with Archive.org.It takes 30 seconds.

Teams ship real agents on Composio