New IteraTools Endpoint: POST /crawl — BFS Web Crawler for AI Agents
IteraTools has released a new breadth-first web crawler API endpoint that extracts structured content from multiple pages, useful for AI agents that need to digest entire documentation sites or product catalogs.
Why it matters
This new IteraTools endpoint provides a powerful tool for AI agents and other applications that need to quickly digest large amounts of web-based information.
Key Points
- 1The new POST /crawl endpoint performs BFS (Breadth-First Search) web crawling, starting from a seed URL and returning page titles, Markdown content, and outbound links
- 2It's designed to stay on the same domain by default, but can be configured to include or exclude specific URL patterns
- 3The API response includes up to 20 pages of content, with a limit of 20,000 characters per page
Details
The new IteraTools POST /crawl endpoint provides a convenient way for AI agents and other applications to extract structured content from multiple web pages in a single API call. By performing a breadth-first search starting from a seed URL, the crawler can efficiently gather information from documentation sites, product catalogs, or other multi-page websites. The API returns the page title, full Markdown content (up to 20,000 characters), and outbound links for each crawled page. Developers can also configure the crawler to only follow same-domain links or to include/exclude specific URL patterns using regular expressions. This makes it easy to focus the crawl on the most relevant content for the AI agent's needs.
No comments yet
Be the first to comment