01 The Problem
SalesRepository is the production-grade successor to my earlier SalesScraper prototype. Same starting problem — county governments publish rich property and sale data, then bury it behind ArcGIS map viewers and clunky public lookup sites — but where the prototype just scraped one county into a generic object store, this is a full application: a multi-county ingestion engine feeding a structured commercial real-estate sales-comparable repository, the kind an appraiser uses to value a property against recent comparable sales.
The job splits into two stubborn halves. Every county is messy in its own way — different field names, different land-use codes, even a different access method (some expose a JSON REST service, others only an HTML site). Downstream, though, an appraiser wants the opposite of that: one uniform comp they can search and compare regardless of where it came from. The whole design is about quarantining the per-county mess on the way in so everything past the front door is uniform.
02 Built on WorkPath
The biggest change from the prototype is the foundation. SalesRepository is built as a full WorkPath client. The scraping and ingestion code is custom — that's the genuinely novel part — but everything downstream is declared on top of the platform rather than hand-built: the typed sale objects, forms, list/detail views, search, exports, access control, email, and the scheduled jobs.
┌───────────────────────────────────────────────┐
│ SalesRepository (this app) │
│ • custom scrapers / ingestion (IDataProvider) │
│ • sale-type templates · forms · views │
└───────────────────────┬───────────────────────┘
│ declared on top of
▼
┌───────────────────────────────────────────────┐
│ WorkPath platform │
│ dynamic objects · search · email · tasks │
└──────────┬───────────────────────┬─────────────┘
▼ ▼
┌────────────────────┐ ┌──────────────────────┐
│ MongoDB │ │ Elasticsearch │
│ staging + records │ │ search index │
└────────────────────┘ └──────────────────────┘
The object templates, choice lists, forms, and views are all provisioned by an idempotent ObjectBuilder that checks for each definition and creates it if missing — installing the whole schema in-process through WorkPath's data layer, so a fresh environment builds itself.
03 The Domain Model
The prototype stored everything as a single generic county_property. SalesRepository instead models a real taxonomy of commercial sale types as WorkPath object templates, so a warehouse comp and a farm comp each carry the fields that actually matter to them.
sales_comparable (base sale)
├─ base_land_sale ──► land_sale
└─ improved_sale ──► commercial_sale · industrial_sale
office_sale · multi_family_sale
self_storage_sale · hospitality_sale
mobile_home_park_sale · mixed_use_sale
institutional_sale · speciality_sale
embedded history: sale_history · lease_history · listing_history
Each record also carries embedded sale history (plus lease and listing history), so a comp isn't a point-in-time snapshot but a chain of transactions on the parcel — which is what you actually need when a property has traded more than once.
04 The Ingestion Pipeline
The heart of the system is a two-stage scrape → stage → normalize pipeline, run per county. Raw data lands first; mapping into typed objects happens as a separate pass over the staging collection. That separation is the same bet the prototype proved: a mapping bug is a cheap local re-run, never another slow, throttled scrape of a county server.
┌──────────────────────────────────────────────────┐
│ DATA PROVIDER (per county) │
│ ArcGIS REST scraper ──or── HTML site scraper │
│ paginate 1000/page · 2s throttle · proxy → DTO │
└──────────────────────┬───────────────────────────┘
│ Insert (only newer than latest stored)
▼
┌──────────────────────────────────────────────────┐
│ MongoDB staging: SiteDTO │
│ { TaxId, SaleDate, County, State, │
│ Status: NotImported|Skipped|Success| │
│ Failed|MappingFailure } │
└──────────────────────┬───────────────────────────┘
│ page over UNPROCESSED records
▼
┌──────────────────────────────────────────────────┐
│ NORMALIZE (DataImportPipeline) │
│ • pick sale-type template by land-use code │
│ • map fields → WorkPath DynamicObject │
│ • enrich: Google Street View → sale_images │
│ • attach embedded sale_history │
│ • dedupe on unique keys → Save │
│ • stamp staging record Status + date │
└──────────────────────────────────────────────────┘
A few details worth calling out:
- Incremental by default. Each run asks the store for its latest sale date and only scrapes back to there (county tasks default to a 2-year floor), so re-runs are cheap and bounded.
- Polite & resilient. 1,000-record pages, a 2-second throttle, and a FastestWebProxy that picks a working proxy keep scrapes from hammering — or getting blocked by — county servers.
- Two scraper strategies. ArcGIS counties (Cumberland, Franklin, Adams, York) hit JSON REST endpoints; others (Dauphin, Lancaster) use HTML site scrapers. Both implement the same IDataProvider contract, so the pipeline doesn't care which a county uses.
- Code-driven classification. Each scraper maps that county's raw land-use codes (e.g. Cumberland's LUC / P_TYPE) to the right sale-type template via a switch — that's where messy source data becomes a clean, uniform comp.
- Street View enrichment. During normalization the address is sent to Google's Street View API; a returned photo is stored on the record (sale_images), with a file cache to avoid repeat lookups.
- Idempotent. Unique-key validation skips comps that already exist, so the whole pipeline can be re-run safely.
05 Scheduling & Reporting
Ingestion runs as WorkPath background tasks. A BaseCountyAction wires up the pipeline (data provider + Mongo staging + import user) and reads run arguments — minimum date, processing date range, skip-scrape — and each county is a thin subclass (PaCumberlandCountyAction, PaYorkCountyAction, …).
WorkPath task runner
├─ PaCumberland / Franklin / Adams / York / Lancaster
│ CountyAction (scrape + import)
│
└─ ImportEmailStatusAction (weekly)
reads 7 days of task history
→ per-county success / skipped / failed counts
→ email status report
A separate ImportEmailStatusAction runs weekly: it reads the last seven days of task history, tallies each county's success / skipped / failed / mapping-failure counts, and sends a status email through WorkPath's workflow email system — so the health of the whole ingestion fleet arrives in an inbox without anyone tailing logs.
06 Reflections & Honest Limitations
The prototype proved the spine — scrape to a raw landing zone, then normalize as a separate pass — and SalesRepository keeps that bet while fixing everything the prototype left rough. The hand-edited Main that toggled scrape vs. import by commenting code is gone, replaced by real scheduled tasks with arguments and a weekly status report. The single generic object became a proper sale-type taxonomy with transaction history. And the "clean domain model defined but unused" gap closed by leaning on WorkPath: instead of writing forms, search, exports, and access control, the project declares them and inherits a working app.
What I like most is that the per-county messiness stays quarantined. Every county brings its own DTO, its own field names, its own land-use codes, and its own scrape strategy — but all of it is sealed behind the IDataProvider contract and a classification switch. Downstream, every record is a uniform WorkPath comp, so cross-county search and valuation work without caring where the data came from.
The honest limitation is reach: it's five Pennsylvania counties today, and each new one is still a real unit of work — a scraper plus a code map — not a config change.