Website Migration Scraper
A responsible Node.js crawler for authorized website migrations, structured content export, media discovery, and SEO preservation.
Current portfolio record
Verified against the public repository and current documentation
Purpose and use case
Why this project was built
Built for developers and site owners who need a repeatable way to collect content and metadata from websites they own or are authorized to migrate.
Overview
What the project is
Website Migration Scraper is designed for controlled migrations and archival work rather than unrestricted crawling. It discovers pages through sitemaps and internal links, extracts a stable page schema, writes inspectable files, and produces migration and SEO reports that can be transformed for WordPress, Next.js, Astro, Hugo, or another CMS.
Problem
What it is designed to solve
A website migration can lose titles, canonical URLs, headings, navigation, taxonomy, structured data, redirects, and media when content is copied manually or through a platform-specific export.
Approach
How the project addresses it
The crawler respects robots rules by default, discovers XML sitemaps, normalizes URLs, applies bounded concurrency and delays, saves resumable state, and exports one canonical page format with supporting reports.
Key features
- robots.txt enforcement and sitemap-index discovery
- Internal-link fallback crawling with depth, page, include, and exclude limits
- Rate limiting, retries, redirect tracking, timeouts, and bounded concurrency
- Per-page JSON and optional Markdown exports
- SEO, headings, links, media, navigation, taxonomy, dates, authors, and JSON-LD extraction
- Checkpoint resume, migration reports, SEO reports, optional media download, and optional Playwright rendering
Architecture highlights
- Small CommonJS modules separate configuration, HTTP, robots, sitemaps, extraction, state, output, and reporting
- Filesystem state keeps migration output portable and easy to inspect without requiring a database
- One canonical page schema allows platform-specific importers to remain outside the crawler
- Normal HTTP and Cheerio remain the default, with browser rendering isolated as an optional path
Engineering challenges
- Respecting crawl permissions while handling missing, blocked, or malformed robots and sitemap responses
- Normalizing and deduplicating URLs across redirects, canonical forms, query strings, and internal links
- Capturing enough migration context without turning the tool into a large general-purpose crawler
Implemented outcomes
- Implemented a resumable website-migration workflow with stable per-page exports and reports
- Added explicit responsible-use rules, safe defaults, tests, and troubleshooting documentation
- Kept the architecture small enough to adapt outputs for different CMS and static-site targets