SalesScraper

01 The Problem

County governments publish an enormous amount of real-estate data — parcels, ownership, assessed values, sale prices and dates — but they bury it behind clunky GIS map viewers rather than offering a clean download. SalesScraper is my pipeline for prying that data out: it scrapes a county's ArcGIS REST service, lands the raw records in MongoDB, then normalizes them into a generic, schema-flexible object store where the data becomes queryable and comparable across sources.

The first (and so far only) county wired up is Cumberland County, PA (gis.ccpa.net). The design intent, though, is that each county is just another scraper plugged into the same downstream pipeline — hence the Counties/ folder and the abstract base class.

Status: early-stage working prototype. The Cumberland scraper and the import pipeline both run end-to-end; a clean Property/Sale/Address domain model is sketched but not yet wired in, and multi-county support is scaffolding only.

02 Architecture

The solution splits cleanly into two halves: a scrape half that deals with one county's messy source format, and an import half that turns that raw data into a clean, uniform object. MongoDB sits in the middle as the handoff point — raw records land first, normalization happens second.

  ┌──────────────────────────────────────────────────┐
  │  SalesScraper.Base                                │
  │  abstract Scrapper.ProcessNextRecords()           │
  └──────────────────────────────────────────────────┘
                         ▲
  ┌──────────────────────┴───────────────────────────┐
  │  SalesScraper.ArcGis                              │
  │  AcrGisScraper / AcrGisUpdateScraper              │
  │  paginate ArcGIS → map ~40 fields → AcrGisDTO     │
  └──────────────────────┬───────────────────────────┘
                         │ Insert
                         ▼
  ┌──────────────────────────────────────────────────┐
  │  MongoDB (SalesDbTest)   raw AcrGisDTO documents  │
  └──────────────────────┬───────────────────────────┘
                         │ read
                         ▼
  ┌──────────────────────────────────────────────────┐
  │  SalesScraper.Importer                            │
  │  SalesImporter → WorkPath DynamicObject           │
  │  ("county_property" template) + derived fields    │
  └──────────────────────┬───────────────────────────┘
                         │ Save
                         ▼
  ┌──────────────────────────────────────────────────┐
  │  MongoDB (WorkPathDB)  normalized county_property │
  └──────────────────────────────────────────────────┘

03 Scraping the ArcGIS Service

ArcGIS MapServer layers expose a /query endpoint that returns features as JSON. AcrGisScraper walks it page by page, copying each feature's ~40 attributes (OWNER, SITUS, SALEPRICE, SALEDATE, SQFT, …) field-by-field into a strongly-typed AcrGisDTO. The source is inconsistent about which fields are populated, so every read is null-guarded with a default.

  loop while features returned AND offset < max:

      GET .../MapServer/42/query
          ?resultRecordCount=1000
          &resultOffset={offset}
          &outFields=*&f=json

      → deserialize dynamic JSON
      → for each feature:
            ConvertRecordAttributes() → AcrGisDTO
      → repository.Insert(dto)

      offset += 1000
      Thread.Sleep(2000)   // be polite to the server

AcrGisUpdateScraper is the incremental variant: it orders the query by SALEDATE DESC (newest first) so a bounded run picks up recent sales, and it converts ArcGIS's epoch-millisecond SALEDATE into a real DateTime rather than storing the raw long. The 1,000-record page size and 2-second throttle are the two knobs that keep the scrape from hammering the county server.

04 Normalizing into WorkPath

The raw AcrGisDTO mirrors Cumberland's field names exactly — useless for comparing against another county that names things differently. SalesImporter is where the data gets cleaned up: it reads raw records from MongoDB and projects each one onto a WorkPath dynamic object — a runtime-defined, schemaless entity stored in its own collection.

  AcrGisDTO (raw, county-specific)    county_property (normalized)
  ────────────────────────────────   ────────────────────────────
  PIN        ──────────────────────► tax_id
  OWNER      ──────────────────────► current_owner
  SITUS      ──────────────────────► address_line_1
  SALEPRICE  ──────────────────────► sale_price
  SQFT       ──────────────────────► square_feet
  ... (18 mapped fields total) ...
                                     price_per_square_foot  ◄── derived

On startup the importer ensures a county_property template exists with ~19 typed property definitions (string / int / double / date), creating it if missing. For each raw record it applies a name-mapping dictionary (PIN → tax_id, SITUS → address_line_1, …), then computes a derived price_per_square_foot — guarding against the junk $1 placeholder sales that public records are full of.

The payoff of the dynamic-object approach: a second county's scraper only needs its own DTO and field-mapping; the destination schema and storage stay identical, so cross-county queries (price-per-square-foot trends, say) work without a migration.

05 Supporting Layers

• DataAccess.MongoDb — a generic MongoDbRepository<T> with full sync/async CRUD over the specification pattern, plus a MongoDbDynamicRepository for schemaless documents. Collection names resolve by reflection (a [CollectionName] attribute or the type name).
• Pattern — the repository and ISpecification interfaces the storage layer implements.
• Models / Specifications — the AcrGisDTO raw entity, the (currently unused) clean Property/Sale/Address domain model, and an AllSales specification.

06 Reflections & Honest Rough Edges

The part I think paid off is using MongoDB as a raw landing zone before normalizing. Scraping and cleaning are different problems with different failure modes — letting the raw, county-shaped data hit disk first means a mapping bug in the importer is a re-run over local data, not another slow, throttled scrape of someone's server. The dynamic-object destination is the other bet: pushing per-county quirks entirely into the DTO and field-map, so the normalized schema never has to know which county a record came from.

What's honestly still rough: the runner is a hand-edited Main that toggles between scrape and import by commenting code, the clean domain model is defined but unused, and there are some acknowledged hacks around async template creation. It's a working prototype, not a finished product — but the scrape → store → normalize spine is solid and the per-county extension point is real.

That proof of concept did its job. Once it concluded — the spine validated and the per-county extension point proven — this prototype was productionized into SalesRepository: the same scrape → stage → normalize bet, rebuilt as a full multi-county commercial sales-comparable database on WorkPath, with a real sale-type taxonomy, scheduled per-county tasks, and a weekly status report in place of the hand-edited runner.