crawler Tag

Browse 57 tools tagged with crawler (tag). Filter by pricing, platform, language and more. The list below shows every tool matching this tag. Each entry links to a full page with pricing, platforms, integrations, and API availability.

All 57 tools

• 🔍 JetOctopus

  SaaS crawler and log analyzer providing real-time data visualization for massive enterprise websites

  paidAPI
• 🔍 Lumar

  Enterprise-scale website intelligence platform for technical SEO and accessibility audits

  paidAPI
• 🔎 Sitechecker

  Real-time website monitoring and technical audit platform for ongoing SEO health tracking

  paidAPI
• 🚀 Aiohttp-Crawler

  Asynchronous HTTP client for ultra-fast SEO crawling and API requests

  open-sourceAPI
• 🧪 Trafilatura

  Python package and command-line tool to gather text and metadata on the web

  open-sourceAPI
• 🔧 Audisto Crawl

  Enterprise-grade crawler for structural SEO analysis and complex website migrations for experts

  paidAPI
• 🧪 Bing Robots.txt Tester

  Validate robots.txt syntax against the Bing crawler agent

  free
• 🛡️ Bing Site Scan

  Automated site auditing tool to find technical errors and SEO issues

  free
• 🔍 Botify

  Enterprise SEO platform for large-scale crawl analysis and search performance

  paidAPI
• ⚙️ Botify Activation

  Technical platform monitoring bot crawl budgets and AI indexing efficiency for large-scale websites

  paidAPI
• 🕷️ Deepcrawl API

  Enterprise-grade API for automating technical audits within customized dashboard environments

  paidAPI
• 📊 Integrity

  Mac website link checker that generates visual sitemaps and checks for broken links and SEO data

  freemium
• 🔗 Link-Checker

  W3C tool for finding broken links and validating URI syntax on web pages

  free
• 💻 Netpeak Software

  Tool for technical SEO audits and rank tracking designed for specialists and agencies

  paid
• 🏗️ Netpeak Spider

  Personalized crawler for day-to-day SEO audits and comprehensive technical site analysis

  paid
• 📉 Oncrawl

  Enterprise SEO platform combining log file analysis and crawl data for technical optimization

  paidAPI
• 🔍 ParseHub

  Powerful desktop-based web scraper capable of extracting data from complex AJAX-heavy websites

  freemiumAPI
• 🤖 Robots.txt Tester

  Test whether your robots.txt file is blocking Google web crawlers from specific URLs on your site

  free
• 🖥️ Screaming Frog Automation

  CommandLine interface for the SEO Spider allowing for fully customized scheduled tasks

  paid
• 🐚 Screaming Frog Guides

  The definitive guide to using the SEO Spider for site crawling and technical analysis

  free
• 🌍 Screaming Frog Hreflang Checker

  Specialized audit module for validating regional URL mapping and routing consistency

  freemium
• 🏗️ Screaming Frog Log Analyser

  Stand-alone tool that syncs with GSC to verify if Googlebot is actually crawling high-traffic pages

  paid
• 🧪 Screaming Frog SEO Spider (Cloud)

  Headless automation of the popular spider for scheduled technical audits in the cloud

  paid
• 🛡️ Screaming Frog XML Sitemap Generator

  Command-line tool for automating the generation of complex XML sitemaps for large sites

  paid
• 📦 Seolyzer

  Free real-time log analysis tool to see how Googlebot crawls your website instantly

  freemium
• 🧬 SiteAnalyzer Backlink Tool

  On-page and off-page SEO suite that tracks backlink quality and competitor link profiles

  paidAPI
• 🗺️ Sitemap Generator (XML-Sitemaps)

  Visual and automated tool to generate XML sitemaps for domain structure validation

  freemium
• 💎 Xenu's Link Sleuth

  Legendary lightweight desktop tool for crawling sites and generating simple sitemaps and link reports

  free
• 🕸️ GPT-Crawler

  Crawler that crawls websites to generate knowledge files from web content for custom GPTs and LLM training

  open-sourceAPI
• 🛠️ SEOEstate

  Open-source website crawler and technical SEO analyzer for developers and technical consultants.

  open-source
• 📦 Zeno

  Open-source crawler and SEO toolset for developers built with Node.js

  open-sourceAPI
• 🔧 SEOnaut

  Comprehensive open-source SEO audit tool for deep site scans

  open-source
• 🦖 RustySEO

  Cross-platform marketing toolkit for deep crawl and server log analysis

  open-sourceAPI
• 🔧 Greenflare

  Open-source desktop SEO crawler for privacy-conscious technical website audits

  open-source
• 🔍 Isivono Hreflang Checker

  Deep-scan crawler that validates cross-page hreflang reciprocity and reports on broken international links

  free
• 🛠️ Sitemap Pro

  Professional cloud-based sitemap builder with automated crawling and broken link detection features

  paidAPI
• 🌐 WonderWeb Sitemap

  Cloud generator with high-speed crawling for sites with over 1 million pages and cloud storage export

  paidAPI
• 📂 Photon

  Incredibly fast crawler designed for OSINT and domain profiling

  open-source
• 📉 Broken-Link-Checker

  Find broken links on websites with this highly configurable CLI tool

  open-source
• 🧪 Gevent-SEO-Crawler

  High-performance concurrent crawler using gevent for large scale site audits

  open-source
• 🧩 A11y Machine

  Automated accessibility testing tool that crawls a website and generates a visual report of issues

  open-source
• 📜 Sitemap-Generator-CLI

  Create comprehensive XML sitemaps for large websites from the command line

  open-source
• 🕷️ Abot

  Open-source crawler for developers to build custom scheduled technical SEO audits

  open-sourceAPI
• ⚡ Scrapy-Cluster

  Distributed crawling system for massive SEO datasets using Redis and Kafka

  open-sourceAPI
• 🗺️ Sitemap Generator

  Open-source tool to generate massive XML sitemaps for enterprise eCommerce catalogs

  open-sourceAPI
• 🗺️ Ultimate Sitemap Parser

  Robust Python sitemap parser to extract URLs and detect errors in XML sitemap files

  open-sourceAPI
• 🔧 Site-Crawler

  Fast multi-threaded web crawler for site mapping and technical audits

  open-sourceAPI
• 📂 Site-Inspector

  Open-source tool to find broken links and common technical SEO errors

  open-source
• 🔗 Link Checker

  Verify internal and external links for broken connections and redirect loops

  open-source
• 🦴 Skeleton

  Open-source crawler for identifying broken links and SEO issues in a CLI environment

  open-source
• 🦖 Dino

  Simple CLI for checking status codes and broken links on small to medium sites

  open-source
• 🏗️ Beam Us Up

  Free desktop SEO crawler for Windows providing an alternative to Screaming Frog

  free
• 🔬 Scrutiny

  Mac-based website crawler that performs scheduled technical, link, and sitemap audits

  paid
• 🔍 SearchBlox

  Enterprise search and web crawling solution with built-in SEO and metadata analysis capabilities

  paidAPI
• 🏗️ Site Condor

  Visual site auditor that maps internal routing and link distribution

  paid
• 🏁 Spotibo

  Web-based SEO crawler that allows for recurring technical checks and on-page analysis

  freemium
• 🌍 Zadroweb

  Suite of free SEO tools including a scheduled technical crawler for site maintenance

  free

Frequently asked questions