The Crawlbase MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Crawlbase account. It provides structured and secure access to web crawling, cloud storage, account usage, and user agent generation, so your agent can crawl public pages, retrieve stored content, delete saved pages, monitor usage, and generate user agents on your behalf.
- Public web page crawling: Have your agent fetch content from any public HTTP or HTTPS page and review whether the crawl succeeded.
- Stored page retrieval: Let the agent find and retrieve individual or grouped pages saved in Crawlbase Cloud Storage.
- Stored content cleanup: Direct your agent to permanently delete selected pages from cloud storage and confirm each deletion outcome.
- Usage and storage monitoring: Instruct your agent to check current crawling usage, review previous month statistics, and count stored pages.
- User agent generation: Have your agent generate up to ten realistic randomized user agent strings for a selected device class.