Web scraping & data pipelines that keep running

Scraping is easy; reliable scraping is not#

A script that pulls data from a page once is quick to write. A pipeline that runs every day for a year, survives layout changes, rate limits, timeouts and outages, and never loads garbage into your database is a different job entirely. Most scraping projects fail not on day one, but on day forty, when a source quietly changes and the pipeline keeps running, filling your database with empty or wrong values that nobody notices for weeks.

That second kind of pipeline is what I build: extraction designed for the sources you actually depend on, validation that knows what good data looks like, and monitoring that tells you the moment something changes.

Signs you need a proper data pipeline#

• Someone copies data from websites or portals by hand, every day or every week.
• A source you depend on has no export or API, but the data is public on its pages.
• Your current scrapers break often, and you usually find out from users or bad reports.
• Data from different sources doesn't line up, so it can't be compared or combined.
• You need history, not just today's snapshot, to spot changes and trends.

What I build#

Custom crawlers and scrapers#

Using plain HTTP requests where possible, and headless browsers such as Playwright or Puppeteer where pages need JavaScript, interaction or a login you're authorised to use.

API extraction#

Many modern pages load their data from an API behind the scenes. Using that endpoint directly is usually faster, cheaper and far more reliable than parsing HTML, so it's the first thing I look for.

ETL pipelines#

Extracted records are normalised into consistent formats (dates, addresses, currencies, identifiers), validated against rules, de-duplicated, and only then loaded.

Scheduled syncs#

Pipelines run on a schedule into PostgreSQL, MySQL, MongoDB or BigQuery, with incremental runs that fetch only what changed.

Monitoring and alerts#

Alerts for failed runs, sudden drops in record counts, unexpected empty fields and layout changes, so problems are fixed before anyone relies on bad data.

How I make scrapers resilient#

• Validation on every run. Each source has rules for what a good record looks like; runs that fall outside them are flagged instead of loaded.
• Selectors that survive small changes, based on stable attributes and structure rather than fragile positions.
• Polite, paced requests with retries and backoff, so sources aren't overloaded and temporary errors don't fail a whole run.
• Raw copies kept for a while, so a parsing bug can be fixed and data re-processed without fetching everything again.
• Idempotent loading, so re-running a job never creates duplicates.

What a typical pipeline looks like#

1. Fetch: a scheduled job requests only new or changed pages or records from each source, paced to respect its limits.
2. Store raw: the original response is kept for a short period, so parsing can be fixed and re-run without fetching again.
3. Parse: fields are extracted into a structured record for that source.
4. Validate: each record is checked against rules for required fields, formats and sensible values; failures are quarantined, not loaded.
5. Normalise and de-duplicate: formats are standardised and records matched to existing ones.
6. Load: clean records are written to your database idempotently.
7. Report: counts, errors and anomalies are recorded, and alerts fire when something looks wrong.

Responsible data collection#

I collect public data only, respect rate limits, and design pipelines to be considerate to the sources they depend on. If a source's terms or the type of data involved raise legal questions, particularly around personal data, I'll say so before building, and suggest alternatives such as official APIs or data licensing where they exist.

Data pipeline projects#

• Violerts: ingestion of NYC Open Data and other municipal sources into a PropTech compliance platform, growing coverage from 3–4 to 12+ datasets and agencies, processed in the background with Laravel Horizon. Scraper development was part of the engagement, and the client, Daniel Lee of AZARK, described me as a "Great developer to work with, very knowledgeable".

What you get#

• pipelines running on your infrastructure, with source code in your repository;
• clean, validated data in the database or format you need;
• a record of every run, with counts, errors and changes;
• alerts to the right person when a source breaks or changes;
• documentation of each source and how to add new ones.

Common scraping mistakes I fix#

• Scraping HTML when an API exists, making the pipeline slow and fragile for no reason.
• No validation, so a changed page silently produces empty or wrong fields.
• Full re-scrapes every time, instead of incremental runs that only fetch what changed.
• One long script for everything, where a single failing source stops all the others.
• No history, so changes over time are lost and mistakes can't be corrected.

Related services#

Scraped data usually feeds a larger product. See real estate and data platforms, database optimization for keeping large datasets fast, and AI integration for analysing what you collect.

Working together#

We start with a free 30-minute call where you share the sources and what you need from them. I review each source for structure, APIs, limits and terms, then send a written plan and estimate. Pipelines are built as fixed-scope projects, with maintenance available on a retainer.

Need data collected reliably? Book a free call and share the sources.