Publish the error the way assessors do: ratio studies, time splits and coverage checks for valuation models

In June 2022 the Office of Property Assessment in Philadelphia published its ratio study for tax year 2023. Across 27,382 residential sales it reported a coefficient of dispersion (COD) of 13.1% and a price-related differential (PRD) of 1.024. Both sit inside the bands of the International Association of Assessing Officers (IAAO). The two numbers are a model audit that anyone can recompute, and they are the yardstick a machine-learning valuation model should be held to before anyone prices a loan off it.

Assessors publish two numbers that a single error percentage hides

A ratio study divides each appraised value by the price the property later sold for, then summarises the ratios. The COD is the average percentage deviation of those ratios from the median ratio, so it measures uniformity. The PRD is the mean ratio divided by the weighted mean ratio, so it measures whether expensive and cheap properties are valued evenly.

The IAAO Standard on Ratio Studies asks for a COD between 5.0 and 15.0 for single-family homes and condominiums in older or more heterogeneous areas. It asks for a PRD between 0.98 and 1.03, and it cautions that PRD means less when samples are small. A PRD above 1.03 says high-priced homes are undervalued relative to cheap ones. A model can look fine on average and still fail on the larger assets a lender cares about.

Philadelphia's own document is explicit that this is a backward-looking audit: it compares a valuation date with sales that have already happened. That is the right frame for a model too. The question is how it would have done on sales it never saw.

Split by time and by building, and label what each price is

A random holdout answers an easier question than a lender has. Two flats in one building, or one flat listed three times, land on both sides of a shuffled split, so the model scores well by recognising the building. The same leak runs through time: a sale in March helps predict a sale in April. Random splits are the default in common tooling, so the first question for any headline accuracy figure is how the holdout was drawn.

A rolling-origin test is one direct fix. Train on everything before a cutoff, test on the period after it, move the cutoff forward and keep each window's error separate. Group rows by building or parcel so no group straddles the split. Other designs, such as spatial block holdouts, address related leaks, but a time-ordered window is the one that mimics deployment.

The target needs the same care as the split. In Turin, our reading of the court auction notices published for the city shows later rounds offered at a discount to the first asking price. The median is 80.9% of the first price in round 2, 76.3% in round 3, 78.4% in round 4 and 81.0% in round 5. An asking price is a seller's position. A model trained on it as if it were a clearing price learns how notices are written. Tag each training row with the kind of price (asking, reserve, appraised, transacted) and the date it was set.

Check interval coverage period by period, because markets step

A point estimate without a range invites false precision. Split conformal prediction attaches a range to any pre-trained model. In the framing of Angelopoulos and Bates, it produces sets that contain the truth with a user-chosen probability, such as 90%. You hold out a calibration set, measure the absolute errors there, and use roughly their 90th percentile, with a small finite-sample correction, as the half-width of the interval.

The guarantee assumes the calibration data and the new cases are exchangeable, in effect drawn from the same process. A market that reprices between calibration and use can break that. Barber and co-authors, in "Conformal prediction beyond exchangeability", relax the assumption with weighted methods. The practical defence is simpler: calibrate on the latest window, then measure realised coverage in every later one.

Turin gives a concrete shape of change to test against. Our series is the median of the zone mid-points in the city's half-yearly official property value quotations for ordinary homes. It read 2,288 EUR per m2 in 2025 H1 and 2,312 in 2025 H2. In 2026 H1 it reached 2,425, above every half-year since 2021 H2, whose previous high was 2,350 in 2023 H2. These are official reference values, not transactions, and one step of this size does not show that any interval would fail. We have not run that test. It is the kind of move a calibration window ending in 2025 H2 does not contain, and it is what a per-period coverage check exists to catch.

Inputs move too. The ECB's interest rate statistics for Italian banks put the rate on new house-purchase loans with a one-to-five-year fixation at 3.11% in September 2025 and 3.75% in September 2026. The floating rate was 2.99% in July 2026, 3.37% in August and 3.29% in September. A step within one month is the pattern a window-by-window report shows and a pooled score averages away.

What to watch

Three readings will say whether this audit matters this winter.

First, the ECB's next monthly release for Italian new house-purchase loans. If the floating rate falls back toward the 2.99% of July 2026, August's 3.37% was a blip. If it stays above 3.3%, the financing input has moved and models calibrated before August deserve a fresh test.

Second, the half-yearly quotations for Turin's zones covering 2026 H2. A median near 2,425 EUR per m2 would confirm the step. A return toward 2,312 would suggest the 2026 H1 reading was an outlier.

Third, your own rolling test. As a rule of thumb of mine, not an IAAO threshold, retrain when the PRD leaves the 0.98 to 1.03 band, or when the realised coverage of a nominal 90% interval falls below 90% in two consecutive windows. The Philadelphia figures of COD 13.1% and PRD 1.024 are a fair reference point for what a published audit looks like.