An industry with no market data now has one

Overview

Nobody knew what competitors charged. We ended the guessing.

Our client's industry had no market data. Nobody knew what competitors were charging, what was actually selling, or which breeders were genuinely licensed. Pricing decisions were made from phone calls, trade gossip and the occasional afternoon spent clicking through a rival's website.

We built the platform that ended the guessing. It visits 188 competitor websites on a schedule, reads every listing, keeps every price change as history, detects when a rival sells something, and feeds the results into the client's own store software through a private interface.

The hard part was never the reading. It was building something that keeps telling the truth when a competitor redesigns their site on a Tuesday.

The failure nobody watches for

The most dangerous failure in a data platform is not a crash. It is a crawler that returns nothing, and a business that reads the empty screen as good news.

The challenges

Six problems, and the most dangerous one looks like good news.

Every website is built differently, and none of them want to be read

188 sites means 188 page structures. Some are hand-built and decades old. Some are built on shop platforms that assemble the page in the browser after it loads, so the page source contains nothing useful. Several actively detect and block repeat automated visitors.

There is no shared format in this industry, no public data feed and no standard. Every site is its own problem.

Websites change without warning, and a broken reader looks exactly like an honest empty result

A competitor redesigns on a Tuesday. The reader built for the old layout starts returning nothing. Nothing is a legitimate answer, because a shop really can have no stock, so a naive system records it as fact.

Worse, it then overwrites months of good data with blanks. A single bad run can destroy a price history that took two years to accumulate, and nobody notices until they need it.

Silence is the failure mode nobody watches for

This is the same problem as the one above, one level up. A crawler that dies loudly gets fixed within the hour. A crawler that quietly returns empty fields for six weeks corrupts every decision made from its data in that period. It generates no alert at all, because from the outside it looks like it is working.

Images are the expensive part, and they never stop arriving

Listings are mostly photographs, hundreds of thousands of them. Each needs downloading, checking that it is genuinely an image, converting to a smaller format, uploading to cloud storage, and deleting when the listing it belongs to disappears.

Done in line with the crawling, this either overwhelms the servers or falls days behind. Done badly, storage costs grow without limit because nothing ever gets cleaned up.

The licensing data that mattered most lived in a government portal

The national licensing authority publishes breeder and dealer licences and inspection reports, as web pages and as scanned PDFs. They are perfectly usable by a human reading one at a time, and completely unusable by a store assistant who needs to verify a breeder while a customer is standing in front of them.

Politeness is a technical requirement, not a courtesy

A platform that hits 188 sites as fast as it can is indistinguishable from an attack, gets blocked within a week, and deserves to be. The system had to be built to move slowly on purpose, at scale, forever.

The circuit breaker · no human awake

How the platform shuts itself off

The single hardest requirement in this platform is not collecting data. It is refusing to collect bad data, automatically, at three in the morning, with no human awake.

First, the platform never overwrites good data with a blank. When a reader fails to extract a field, the system keeps the value it already had. The comment in the source is unusually direct about why: if an error is thrown and the scraper cannot get data from the page, return the old data rather than lose it.

The consequence is that a broken reader produces stale data, never destroyed data. Years of price history survive a competitor's redesign. That one decision is the reason the archive is worth anything.

Second, the platform reports ten distinct kinds of failure, because "it didn't work" is not actionable. Each is a different problem with a different fix, and each goes to a Slack channel and an email rather than to a log file nobody opens.

Ten failures, each reported on its own

• server error
• crawler error
• empty page body
• empty extracted values
• empty database response
• non-numeric response
• failed file download
• outdated file structure
• missing file
• inspection-report count over its ceiling

That is the whole design

Empty is treated as an incident, not as an answer.

Third, the platform takes sites offline by itself.

Every fault is recorded against the site that caused it. When a single site accumulates 100 unresolved faults, the platform stops every crawler for that site immediately, marks the site as deactivated because of faults, sends an email titled Action Required naming the domain, and posts to the operations Slack channel.

It does not ask permission and it does not wait for a person. A site that has started lying is taken out of the data set until somebody fixes it.

And it lets them back in by itself too.

When the unresolved fault count drops back below 100, the site is reactivated and its crawlers resume. The recovery is as automatic as the shutdown, so clearing a backlog of faults is the only action needed. There is no second manual step to forget.

What we built

Six chapters, from a reader per site to a search box in the client's store.

Chapter one · 45 shared helpers

A reader per site, and a toolkit underneath all of them

Each competitor gets its own small set of instructions, broken into steps: find the listings, read a listing, read the breeder, check the result. When a site changes, one reader is repaired and the other 187 keep running.

But the readers are deliberately thin. The value sits underneath them, in a shared extraction toolkit of 45 helper operations across roughly a thousand lines: get the text at a position, get an attribute, get the text between two markers, get everything after the last occurrence, read a value out of a URL, parse a date into a timestamp, validate a link. Writing a new reader is assembling these, not writing a parser.

Chapter two · 8 readers, 115 sites

One reader for many sites

Sites built on the same underlying platform share a single reader. Eight grouped readers cover 115 of the 188 sites, including one that alone handles 75 franchise stores, and one written specifically for shops built on a popular commercial platform.

Seventy-three sites need their own bespoke reader. The other 115 do not. That ratio is the difference between a platform and a very large pile of scripts.

Chapter three · Price and availability

History, not snapshots

For fields that matter commercially, price above all, the system does not simply store the new value. When the value changes, it appends the old one to a change log with a timestamp.

That is how a series of crawls becomes a price history, and how the client can point at the exact day a competitor discounted. The same mechanism tracks availability, which is how the platform knows when a rival has actually sold something rather than merely stopped advertising it.

Chapter four · Up to 10 seconds per request

Moving slowly, at scale, on purpose

Before every outbound request the platform pauses for a random interval of up to ten seconds. Requests go out through rotating regional connections. Sites that only assemble their pages in a browser are fetched through a real headless browser or a rendering service.

The effect is a platform that reads 188 sites continuously and behaves, from each site's point of view, like a slow and unremarkable visitor.

Chapter five · Laravel · a real job queue

The image pipeline, rebuilt properly

Image handling was the part that strained, so it is the part we modernised. It was rebuilt as a Laravel application with a real job queue. Work is broken into small units, each attempted twice with a delay between attempts, duplicate jobs are refused outright, and long-running work is given a generous ceiling rather than being killed mid-upload.

It is also defensive in a way that only comes from operating a system rather than just building it. If a job cannot find its record by identifier, because some historical records carry the wrong one, it falls back to finding the record by its image address instead of failing. That is not a design decision made at a whiteboard. That is a decision made after seeing the failure.

The pipeline runs on a tight schedule: new images collected and uploaded to cloud storage every minute, validated every five minutes, statistics and database backups every six hours, with housekeeping daily. Each stage reports into its own Slack channel, so a backlog is visible in minutes rather than days.

Chapter six · Text recognition · private interface

Turning a government portal into a search box

We pointed a reader at the national licensing authority's own licence and inspection portal, ran the scanned inspection documents through text recognition to make them searchable, and normalised the result into a register that can be queried by name, licence number or address, filtered to currently active licences only.

Then we exposed it as a private interface. When a store assistant in the client's own software types a breeder's name, the request reaches this platform and returns verified licence holders.

That closes the loop. Data gathered from a government portal, by a crawler platform, is what stops a wrong licence number being printed on a legal contract in a store hundreds of miles away. It is the least visible thing we built and arguably the most valuable.

Related case study: Multi-Store Retail Operations Platform

The shop software at the end of this loop.

Operations deep-dive · the staff control panel

Inside the control room

A platform reading 188 sites will always be partly broken. Something is always redesigning. The question is not whether faults exist. It is whether the people running it can see them.

So a substantial share of the work is not crawling at all. It is the staff control panel: 47 routes built for the people who keep the data honest.

Separate control panels per animal category

Nine of them, so a specialist can work on one category without wading through the rest. Alongside them: breeders, breed reviews, inspection reports and rescue organisations.

Crawler health, front and centre

Which crawlers are active, their full run history, current faults, and failed jobs. The faults are the ones the automatic reporting raised, so the fault list and the alerting are the same system, not two views that can disagree.

Data quality as a staffed activity, not an assumption

Three separate screens exist purely to catch bad data that did not announce itself: a review of a site's collected records, a report of records with fields missing, and a random sample review that shows arbitrary records for a human to eyeball. That last one exists because some kinds of wrongness cannot be detected by a rule.

Per-site field mapping, editable by staff

Which fields a site produces, and which of the platform's columns they land in, is configuration rather than code. Onboarding a site that resembles an existing one is largely an administrative task.

A database monitor and a statistics layer

Totals per category and per site are watched over time. A site that quietly halves its output is visible as a trend before anyone thinks to check it.

And the whole thing backs itself up

Nightly database backups to cloud storage, with a matching job that deletes old ones so storage does not grow without limit, plus a cleaner that removes orphaned files and a job that rotates the signed links protecting stored images.

How we work

Eight rules, built for the failure nobody watches for.

Start with the hardest case, not the easiest

The first readers written were for the worst-behaved sites. If the architecture survives those, it survives everything. Building the easy ones first tells you nothing and produces an architecture that collapses at site forty.

Treat every component as replaceable except the engine

Readers are expected to break and are built to be thrown away. The investment goes into the shared toolkit, the fault reporting and the control panel, the things all 188 depend on.

Build the monitoring before scaling up

Crawler logs, fault reporting, failed-job views and the data-review screens existed before the tenth site was added. Adding monitoring to a platform already running at scale means you are always looking at a system you cannot see.

Protect stored data above incoming data

When the two conflict, what is already stored wins. A gap in collection is an inconvenience; a corrupted history is permanent.

Make failure loud and specific

Ten distinct failure reports rather than one generic error, each routed to a channel a human actually reads. "Something went wrong" is not a report.

Let the system take its own dangerous parts offline

The 100-fault circuit breaker acts without waiting for a person, and restores without needing one either. Anything that depends on someone noticing will eventually depend on someone who did not.

Modernise the part that is straining, not the part that is old

The original crawler engine still runs the readers, because it works. The image pipeline was rebuilt in Laravel with a proper queue, because that is where the actual strain was. Rewriting working software for its own sake spends the client's money on our preferences.

Be a good citizen of other people's servers

Deliberate delays before every request, rotating connections, and no more load on any one site than a slow human browsing. It is the right thing to do, and it is also the only version that still works in two years.

Technologies used

Eight layers, and what each one is for.

Storage

Every site produces a different shape of listing. A flexible store means a new site does not require a schema change.

• MongoDB

Crawler engine

Runs all due readers at once rather than one after another, so a slow site does not hold up the other 187. A new reader is assembled from existing parts rather than written from scratch.

• Custom PHP
• Concurrent request dispatch
• 45 shared helper operations

Difficult sites and access

Reads pages that only exist once a browser has built them, and reaches sites that block repeat visitors without behaving like an attack.

• Headless Chrome
• ScrapingBee
• Rotating regional proxies
• Randomised delays

Image pipeline

Photo processing as a managed queue: retries, duplicate refusal, visible backlogs. Photos are converted to a smaller format before storage. Both runtime versions are now past end-of-life; see the status note.

• PHP 8.1
• Laravel 9.19
• Horizon 5
• Redis
• Intervention Image
• WebP conversion

Cloud storage

Images and nightly backups, in separate stores for galleries, files and backups, with signed links rotated on a schedule.

• DigitalOcean Spaces

Documents

Makes scanned government inspection reports searchable.

• OCR text recognition

Alerting

A backlog or a fault is visible in minutes, in the place the team already is.

• Slack, a channel per pipeline stage
• Postmark email

Control panel

Where staff actually run the platform.

• Custom PHP
• 47 routes

Automated testing

Automated tests check each change before it reaches production.

How the platform works

Seven steps, from a clock that runs every minute to an answer in the client's store.

Step 1

Schedule

A clock runs every minute. It asks the database which readers are due, sets their next run time, and dispatches them all concurrently.

Step 2

Collect

Readers collect listing links, then visit each listing: pausing before each request, going out through a rotating connection, and using a real browser for sites that need one.

Step 3

Compare

Results are compared against what is already stored. A new listing is added. A changed price appends the old value to that listing's history. A listing that has vanished is treated as sold. A field that failed to extract keeps its previous value rather than being blanked.

Step 4

Classify

Anything that went wrong is classified and reported into one of ten specific fault types, and recorded against the site. Hit 100 unresolved faults and that site's crawlers stop automatically until the faults are cleared.

Step 5

Queue the photos

Photographs are queued, not processed on the spot. A separate pipeline downloads, validates, converts and uploads them to cloud storage every minute, validates every five, and cleans up after listings that have gone.

Step 6

Watch

Staff watch the control room: nine category panels, crawler health, fault lists, failed jobs, missing-data reports and a random-sample review screen for the kinds of wrongness no rule catches.

Step 7

Answer

The client's store software asks a question. A staff member types a breeder's name into their own system; the request reaches this platform's licence register and returns verified, currently-licensed breeders, dealers and carriers.

Results

What the business can do now that it couldn't before.

See the whole market instead of a corner of it

188 competitor sites across nine animal categories, read continuously by one system.

Point at the day a competitor discounted

Price changes are kept as history, not overwritten. The archive gets more valuable every month rather than being replaced every night.

Know what is actually selling

Not just what is being advertised. Availability changes are detected and dated.

Verify a breeder's licence during a customer conversation

A government portal of web pages and scanned PDFs became a search box in the client's own store software, filtered to active licences.

Survive a competitor's redesign without losing anything

A broken reader costs freshness, never history.

Find out something is wrong before the data is used

Ten specific fault reports, a fault list staff work through, and an automatic shutdown for any site that crosses 100 unresolved faults.

Add a similar site as an admin task

Field mapping is configuration. Sites sharing a platform share a reader: 115 of the 188 are covered by eight.

Run it with a small team

47 operational routes exist so the people watching the platform are looking at dashboards rather than log files.

The engineering, in numbers

• 188 Competitor websites monitored
• 73 Bespoke site readers
• 8 Grouped readers, covering the other 115 sites
• 75 Sites handled by the largest grouped reader
• 9 Animal categories tracked
• 10 Failure types reported separately
• 100 Unresolved faults before automatic shutdown
• 45 Helper operations in the shared extraction toolkit
• 47 Staff control-panel routes
• 30 Scheduled maintenance jobs in the original engine
• 7 / 11 Queued job types / console commands in the image pipeline
• Every minute Fastest scheduled task

Honest status

This platform is live and operating, and the numbers above are counted from the running codebase, a snapshot last modified in May 2023.

It is two generations running side by side, on purpose. The original PHP engine runs the readers and the control panel; the newer Laravel application runs the image pipeline. That is a deliberate decision, not a migration that stalled.

The versions are now old. PHP 8.1 lost security support in December 2025 and Laravel 9 in February 2024. The MongoDB package the pipeline uses has since been handed over to MongoDB and renamed, so the name in this codebase is no longer the maintained one.

The 188 figure counts configured readers, which is what the code can prove. How many are enabled on any given day is an operational setting, not a code fact.

Related work

All work

Next step

Making decisions from data that could be quietly wrong?

Work with an engineering team that builds for the failure nobody watches for, protects the data you already hold, and operates what it builds.

Book a discovery call