Linking Milling Establishments Across Time

During the time period of our microdata the Census Bureau did not assign time-invariant unique identifiers for the recorded establishments, making long-run analysis of specific manufactures impossible without ex post record linking. In order to study the historical dynamic characteristics of mills, we link lumber and flour mills across decades and create a panel of milling establishments.

This process involved the creation of training data by the hand-linking a relatively small number of flour and lumber mill establishments across a pair of decades, training a machine learning (ML) on linked training data, using the ML model to identify likely cross-decade matches in the entire set of milling establishments, and then hand-checking potential matches in order to create links between decades. Chaining these links together, we make a panel of mills for 1850-1880.

To use this panel one needs to merge the panel identifiers from the first download to the main CMF micro data.

Download

• Mill Panel Identifiers

• Full Set of Mill Links

• Mill Panel Documentation

Methodology

Establishment Definitions

We define a stable manufacturing establishment by owner name, industry, and county. Cross-county moves, industry switches out of milling, and complete ownership changes are treated as closures; partial ownership changes (e.g., a son taking over from his father) are treated as continuations.

Initial Hand-Linking

We first hand-linked establishments in 97 relatively small counties, matching all 2,709 establishments in 1860 to 5,518 candidate establishments in 1870, to generate initial training data. Links were made using establishment name, industry, and nearest post office, with original manuscript images available to resolve transcription errors. Each sheet was completed by two people and reconciled by a third.

Match Types

We distinguish three kinds of matches: same owner and industry (“y”), same owner with industry change (“s”), ownership transfer (“o”).

Feature Construction

For each candidate pair we construct features across three categories: name similarity, industry overlap, and post office similarity. Because establishment names follow different conventions depending on ownership structure, we build a name classifier that flags impersonal (company) names using location and business-related tokens, and otherwise extracts, standardizes, and reorders owner names from person-based names.

Machine Learning

We train two random forest models, one matching on individual owner names and one matching on the establishment’s full name, since some establishments are named after their owners and others are named after the company or water power source. Both models predict whether a pair is a same-owner match, an ownership-transfer match, an industry-change match, or a non-match. For multi-owner establishments, the owner model takes the maximum predicted probability across all owner pairs to produce an establishment-level score. We take the maximum of the two models’ predicted probabilities as the final linking probability.

Hand Verification at Scale

Using the trained models, we sent candidate pairs above a probability threshold — the top 20 candidates per establishment above 9%, or the top 5 above 5% when fewer candidates existed — to Digital Divide Data (DDD) for large-scale hand-linking, following the same two-reviewer-plus-reconciliation protocol as the initial round. Link probabilities were withheld from reviewers.

Iteration

After hand-linking, we retrained the model on the full set of matches and flagged three categories of links for a second review: hand-links with predicted probability below 40%, unlinked establishments with a candidate above 40%, and cases where the hand-linked match differed substantially from the model’s top-predicted match. Flagged establishments were resent to DDD with their full candidate sets for re-review.