GPT-Crawler
Technical SEO · github.com trending
Crawler that crawls websites to generate knowledge files from web content for custom GPTs and LLM training
Specifications
- Pricing
- open-source
- Price range
- 0
- Billing
- one-time
- License
- MIT
- Platform
- cli, api
- Languages
- en
- Target user
- intermediate
- Company size
- small-team, agency
- API
- Yes
- Data export
- Yes
- Last update
- 2025-12-15
Tags
Related tools
- 🕸️ Apify SEO Crawler
Programmable cloud-based crawler for highly custom scheduled technical SEO audits
- 🔍 JetOctopus
SaaS crawler and log analyzer providing real-time data visualization for massive enterprise websites
- 🔍 Lumar
Enterprise-scale website intelligence platform for technical SEO and accessibility audits
- 🔎 Sitechecker
Real-time website monitoring and technical audit platform for ongoing SEO health tracking
- 🔧 Audisto Crawl
Enterprise-grade crawler for structural SEO analysis and complex website migrations for experts
- 🔍 Botify
Enterprise SEO platform for large-scale crawl analysis and search performance