The estate.sh Scraper Is a Work of Art

You’re not supposed to say this about your own code, so let me just say it: the scraper behind estate.sh is a work of art.

I intend to earn that sentence, because it did not start as one. It started as the obvious thing.

Property hunting in Georgia means bouncing between four portals, learning four searches, and seeing the same apartment more than once. I wanted one clear view of the whole market. One search. No four-tab ritual.

That became estate.sh. As I write this, the public stats endpoint reports roughly 316,000 listings and 3.45 million images. One scraper collects, normalises, groups, and refreshes all of it every night.

Nobody babysits this thing. That’s the art.

It started as the obvious scraper

The sources are MyHome, SS.ge, Korter, and Binebi. They cover different parts of the Georgian market, and estate.sh keeps the original source visible on every property.

The first shape of the scraper was exactly what you would expect: fetch the list pages, parse the cards, fetch the property pages, write the rows.

That works until a portal has more inventory than it will let you page through. The result count says thousands of properties exist, but deep pages eventually stop giving you useful results. A normal crawler does not fail. It finishes successfully with half the market missing.

That is worse.

So the search became a tree

The fix was not a bigger page loop. Every source now defines search segments that know how to split themselves when they get too large.

MyHome starts with 300 cities. If one city reports more than 20,000 listings, it splits into five deal types. Still too big? It splits again into six property types. Still too big? 101 area bands, mostly two square metres wide, from under 2m² to over 200m².

Tbilisi apartments for sale get cut all the way down. A village with 40 listings stays as it is.

My favourite part sits at the deepest split. The scraper counts all 101 children and checks that they add back to the parent within five listings. If the maths is wrong, it throws away the clever split and keeps the parent.

The scraper doesn’t trust its own cleverness.

Every portal needed a different version of the trick. SS.ge splits large searches by city, Tbilisi subdistrict, municipality, and eventually individual street. Its page hides a bearer token inside __NEXT_DATA__, so the scraper pulls that out and calls the same count API as the site’s own frontend. Korter puts its data in window.INITIAL_STATE = {...}, which needed a small balanced-brace parser to lift the JSON out of the HTML.

The ugly source-specific parts stay inside four adapters. Everything after them sees one property shape.

The market was finally visible. It was also thousands of pages, and the only way to finish before morning was to fetch them in parallel.

Going fast without being a dick

List pages, detail pages, and images fan out through Cloudflare Queues. The consumers can scale to 250 concurrent invocations.

Point 250 fetchers at a local property site without coordination and you have built an outage, not a scraper.

So every source gets its own clock. A Durable Object stores the next time a request is allowed to leave. Consumers ask for a slot. They either get one or get told how long to wait.

The clocks are different because the sites are different. MyHome detail requests spread across 24 throttle shards. Korter discovery gets one. If a source starts failing, only that source gets a 30-second penalty. The other three keep moving.

If the wait gets too long, the consumer does not sit there burning time. It puts the remaining work back on the queue with a delay and gets out of the way.

The politeness is load-bearing. These are local websites, and the entire design assumes the scraper is a guest.

Parallel work also means interrupted work. Jobs get retried, delayed, and occasionally cut off halfway through. Once the scraper became fast, the next problem was making failure boring.

Failure needed a memory

Every raw page body is archived in R2 alongside the normalised result. Failed detail fetches get their own prefix. Every discovery job checks whether its output already exists before doing the work again.

Requeue a job, replay it, or crash it halfway through: finished work stays finished. If seven retries are not enough, the message lands in a dead-letter queue instead of disappearing.

The scraper also writes compact NDJSON load files. The database import reads those stable artifacts instead of going back to the portals. When a parser or import goes wrong, the exact raw response and the exact normalised record are still there to inspect.

Once R2 became the scraper’s memory, images followed. They are capped at 12 per property and served back through the Cloudflare Worker with a one-year immutable cache header.

The bucket even watches its own bill. A readiness check projects monthly write costs from recent runs and fails its guardrail above the configured $30 Class A budget.

I have never met another scraper that can tell me when it cannot afford itself.

At this point the pipeline could find the whole market, fetch it without being rude, and survive its own failures. Then all that clean data reached the database and found the actual hard problem.

The same flat, three times

Sellers cross-post. The same apartment appears on MyHome, SS.ge, and Binebi with different photos, slightly different prices, and sometimes slightly different facts.

Showing all three would make estate.sh another messy portal. Hiding two without enough evidence could hide a genuinely different property. The scraper has to know when two listings are the same without getting enthusiastic about it.

The write path into PostgreSQL already assumes the inputs are unreliable. Duplicate rows collapse in memory, then again in SQL. Sparse list-page fields cannot wipe richer detail-page data. Concurrent imports for the same source serialise on an advisory lock. Older observations cannot overwrite newer ones:

on conflict(source, source_id) do update set ...
where excluded.scraped_at > estate_sh_properties.scraped_at

Cross-portal matching is deliberately harder. Manually confirmed groups win. Strong automatic matches need the same location, deal type, property type, rounded area, room count, floor layout, and a matching coordinate, street, or address across more than one source.

Below that is a looser statistical rule. It uses canonical location, area to 0.1m², inferred room count, and this:

round(ln(p.price_usd::numeric) / ln(1.04::numeric))::text

That is a logarithmic price band. Candidate listings still have to sit within 4% of each other in both directions, with tight tolerances for bedrooms and floors. Missing room counts are pulled from titles in English, Georgian, and Russian.

Enough evidence and the listings become one canonical property. Not enough and they stay separate. Every source link survives either way.

Now the scraper had a clean version of the market. Rebuilding that version every night created the next problem: people were already searching it.

The swap nobody notices

You cannot rebuild a live search table row by row and hope nobody catches it halfway through.

So estate.sh builds the next version beside the current one. The post-process creates shadow tables named for the run, fills them, builds their indexes, and checks that they are not empty.

Only then does it swap them in. Constraints and indexes are recreated from the Postgres catalog, ANALYZE teaches the query planner about the new data, and a short table rename replaces the old version.

The database sees a metadata swap. Users see the old complete market, then the new complete market.

There is no moment where estate.sh is half-updated.

Everything now worked from start to finish. Unfortunately, the entire chain still began with a schedule firing at 01:00 UTC.

A cron schedule is a promise, not proof.

So the scraper got a watchdog

A second Cloudflare Workflow checks at 01:15, 03:00, 05:15, 07:00, and 09:00. It asks one question: did the run we expected actually happen?

If the run is alive or finished, the watchdog leaves it alone. If it looks failed or stuck, it checks the real counters and archived outputs. If scraping finished but the production import never started, it queues the missing import. If the nightly never started at all, it creates a fallback run with watchdog in the ID so the morning-after evidence explains itself.

The monitor is skeptical during the run too. The pipeline has to look drained for three consecutive checks before the next stage begins. One quiet poll proves nothing.

I briefly had GitHub Actions owning the production load and watchdog. That was wrong. GitHub Actions is good at building code; it is not where a three-hour data pipeline should live. The Worker already had the queues, database connection through Hyperdrive, and schedules, so I deleted the long jobs and moved the runtime into Cloudflare.

The final readiness audit checks the sources, queues, database, latest full run, and R2 budget. It also enforces one rule I genuinely love: exactly one system is allowed to own automatic production imports. Not zero. Not two.

So yes, art

My definition: a system is a work of art when every piece exists because something real demanded it and nothing exists for show.

The 101 area bands exist because pagination gives out. The source clocks exist because going faster could hurt the sites estate.sh depends on. The price bands exist because sellers cross-post the same flat at slightly different prices. The shadow tables exist because readers should never watch the database being rebuilt. The watchdog exists because that 01:00 promise needed proof.

I can point at almost any odd-looking part of this thing and tell you which problem put it there.

If you’re wondering how one person maintains all of it, the honest answer starts with Building a Fully Agentic Coding Setup and sessions like 13 Prompts, 366 Tool Calls. Same workflow, pointed at a scraper that has to survive the public internet instead of a work backlog.

The output is public. Search estate.sh, inspect the live numbers, read where every listing comes from, or build against the API through estate.sh/developers.

The interesting part is not that it scrapes. It’s that it knows when it hasn’t.