Library: Splink - Record Linkage Without a Training Set

Two tables, one real world, no common key. Splink answers “are these the same entity?” with a probability instead of a guess - and then shows you the arithmetic.

The problem

name	date of birth	city
Jon Smith	1984-03-12	Leeds
John Smyth	1984-03-12	Leeds
J. Smith	12/03/84	LEEDS

One person, three rows, zero keys in common. An exact join finds nothing. A fuzzy string score with a threshold you picked by squinting finds something, and you will never know how much of it is wrong.

Splink is an open-source Python library from the UK Ministry of Justice that does this properly: probabilistic record linkage, at scale, without labelled training data.

Clean first

Look at that table again. Two of its three problems are not linkage problems at all. 12/03/84 and 1984-03-12 are the same date in two formats; LEEDS and Leeds are the same city in two casings. Splink compares whatever you hand it, and to a string comparator 12/03/84 shares nothing with 1984-03-12. This model will not learn its way out of that: Fellegi-Sunter weighs outcomes, it does not parse. You could push the parsing into a comparison level, but then every candidate pair pays for it and the verdicts get harder to read. Cheaper and clearer to make the columns comparable before the model sees them. Parse every date into one format. Lowercase every string. Strip the punctuation from J.. Turn N/A, - and 1900-01-01 into real nulls, because a placeholder that agrees on every row is evidence for nothing, and a comparison that counts it will learn that it means everything. In Serbian data the rule bites harder: Pavlović, Pavlovic and Павловић are one surname in three scripts, and the Cyrillic one has zero characters in common with the other two. Transliterate, fold đ to dj and ć to c, then link.

Cleaning is also where you get to be clever, because some columns carry more than they show. The Serbian JMBG is thirteen digits: the last is a check digit computed from the other twelve, the first seven are the date of birth. So before any model runs you know which JMBGs are mistyped, and a valid one can fill a missing birth date or flag a contradicting one. What you do with an invalid one is a decision, not a rule. Leave it in and let Splink weigh it (a single-digit typo still lands in a “one digit off” level). Null it, so the row falls back on name and date. Or fix it, if date part and name give you enough. Each choice trades one kind of error for another, and the model cannot make it for you.

None of this is specific to Splink. It is ordinary data-cleaning discipline, in this order:

Profile before you touch anything. Null count, distinct count, top twenty values, min and max, string lengths - per column. That is where N/A, -, 0, 1900-01-01, test and the column that is 95% empty show up.
Drop exact duplicates first. DISTINCT on every column. They are free to remove and only distort the statistics later.
Nulls as nulls. Empty strings, whitespace, N/A, the text NULL, placeholder dates - all become real nulls. Then per column: leave it (null is silence to the model), fill it from another column (JMBG gives a birth date), or drop the row. Never invent a value; for linkage that is fabricating evidence.
One type and one format per column. Dates parsed with an explicit format per source, never guessed - 12/03/84 is March or December depending on who exported it, and a guess fails silently. Numbers as numbers, not "1,234".
Strings normalised. Trim, lowercase, collapse whitespace, unicode-normalise (é as one character, not e plus accent), transliterate, fold diacritics. Names: titles out, nicknames mapped (Bob to Robert). Companies: d.o.o. / DOO / d o o into one form or their own column.
Composite columns split into parts. Address into street, number, city, postcode. Full name into forename and surname, knowing the order varies. Compare the parts; the whole string is noise.
Validate against the domain. Check digits (JMBG, IBAN), email shape, postcode lists, birth dates in a possible range. Flag failures in a boolean column; do not silently delete them, you want the count.
New columns, never the original. Every step writes to surname_norm, dob_parsed, jmbg_valid, or a new table. The raw value stays for display, audit, and the day you discover the cleaning was wrong.
Cleaning is code, not clicking. A script that runs on tomorrow’s export and gives the same result, with assertions (dob.min() > 1900). Not a hand-fixed spreadsheet.
Count before and after. Nulls before and after; rows dropped and why; one log line per rule. Otherwise you do not know whether you helped or wrecked it.

What is left after all of that, Jon against John, Smith against Smyth, is the part Splink is for: genuine variation, not formatting noise.

Built by the British government

The origin is not a startup, and not a company either. It is the UK Ministry of Justice - the government department that runs the courts and prisons of England and Wales. In 2019 its data linking team, led by Robin Linacre, was told to link the justice system to itself - criminal courts, prisons, probation, civil and family courts - so the result could go to academic researchers under a government-funded research programme called Data First. None of those systems share a person ID. The same defendant appears as a different record, often spelled differently, in each system.

The free tool that implemented the right model, the R package fastLink, stalled above a few hundred thousand records. The ministry had over a hundred million. Linacre had done this by hand once: around 100,000 records, about a year. So the team wrote their own, first on Apache Spark, and published it as open source.

Then it escaped the building. The Office for National Statistics, the UK’s national statistics agency, linked the 2021 Census of England and Wales against itself with Splink and found roughly 420,000 duplicate responses, an overcount of 0.96%. NHS England, the public health service, is building its patient record linkage on it. Marie Curie, a British hospice charity, replaced a year of hand-written SQL matching in two months. By late 2024: nine million downloads, sixty-plus contributors, staff from the Australian Bureau of Statistics and Databricks among them.

A civil service analytics team shipped the reference implementation of a field. Take a moment with that.

The model is from 1969

The cleaning section was long because the model below assumes you already did it. The model does two things: it checks whether two values agree, and it counts how often they agree to decide how much that is worth. 12/03/84 and 1984-03-12 are the same date, but the model sees them disagree. Two N/As are nothing, but the model sees them agree. Every such false answer goes into the counting, and the weights come out wrong. You clean first because the model cannot tell a real agreement from a formatting accident.

The idea is older than most of the people using it. In 1959 Howard Newcombe, a geneticist at Canada’s Chalk River nuclear laboratory, needed to follow families through birth and marriage records to study inherited disease, and published in Science how to make a computer do it: score each agreement by odds, and count a rare surname as stronger evidence than a common one. Ten years later Ivan Fellegi and Alan Sunter, at what is now Statistics Canada, turned his practice into a theorem - a formal model with a decision rule proven optimal for a given error rate.

Splink implements that model, Fellegi-Sunter. For every column comparison it holds two numbers:

m - the probability the values agree when the records are the same entity
u - the probability they agree when the records are not

The ratio is the evidence. Date of birth agrees on true matches about 90% of the time (typos exist) and on random non-matches about 0.1%, so a match is worth log2(0.9 / 0.001), roughly +10 in match weight. Gender agrees on 98% of matches and 50% of non-matches: log2(0.98 / 0.5), about +1. Same “both columns match”, ten times the evidence.

Weights add up across columns, and the sum converts to a match probability. Nobody hand-tunes that date of birth matters more than gender - it falls out of m and u.

Comparisons are not binary either. Term frequency adjustments go further still: in Serbian data, agreeing on Jovanović, the commonest surname in the country, is weaker evidence than agreeing on Žarković, and the model prices that in - from the frequencies in your data, not a dictionary. Put the same model on a UK file and Smith becomes the weak one.

What you compare, and how

A comparison is a stack of levels, checked top to bottom, first hit wins - a switch statement. Null first (a missing value is silence, not disagreement), exact match, then looser and looser (Jaro-Winkler above 0.92, above 0.88), then “everything else”. Each level gets its own m and u. Most of the modelling judgment lives in choosing which stack belongs on each column, and the library ships the usual ones ready-made: ExactMatch; LevenshteinAtThresholds, DamerauLevenshteinAtThresholds, JaroWinklerAtThresholds for strings; DateOfBirthComparison, EmailComparison, PostcodeComparison, ForenameSurnameComparison for columns that fail in known ways; CustomComparison when they don’t fit, with any SQL expression as a level.

Which stack goes where is the practice, and it follows how the data goes wrong:

Names get typos and nicknames. Jaro-Winkler, because it rewards a shared prefix and people misspell the end of a word more than the start. ForenameSurnameComparison also catches the columns swapped.
Dates fail in two ways: typos and genuine near-values. 1984 to 2984 is a typo, nobody is born in 2984; 1984 to 1985 may be a different person. String distance and numeric distance therefore mean different things, and the built-in date comparison stacks both, each level with its own weight.
Identifiers (JMBG, NHS number, account number) are exact or a single-digit typo; numeric closeness means nothing, 1234567 and 1234568 are strangers. Exact, then Levenshtein 1, then else.
Addresses are parsed into parts and compared part by part; a raw address string is noise.
Scripts and diacritics are normalised before the comparison - see above.

Get the stack wrong and training cannot fix it: the model only weighs outcomes you named.

No labels

Nobody is going to hand-label ten thousand pairs. Splink does not ask.

u is estimated from random pairs of records: pick two rows at random and they are almost certainly different entities, so how often their columns agree is u. m is estimated with expectation maximisation - guess which pairs are matches, re-estimate m from those, repeat until the numbers stop moving. Each EM pass runs inside a blocking rule, and the column you block on agrees by construction, so it cannot be estimated in that pass: train once blocked on surname, once on date of birth, and every column gets its m.

Blocking

A million records is half a trillion pairs. Nobody compares them all. Blocking rules generate only the candidate pairs worth scoring: same surname, or same first name and date of birth, or same postcode.

This is where the quiet failures live. A pair that no blocking rule produces is never scored, so it is never a match, and nothing tells you. Prediction recall is therefore bounded by blocking recall: the classifier cannot recover a pair it never sees. Several loose rules beat one tight one.

It is just SQL

import splink.comparison_library as cl
from splink import DuckDBAPI, Linker, SettingsCreator, block_on

settings = SettingsCreator(
    link_type="dedupe_only",
    comparisons=[
        cl.JaroWinklerAtThresholds("first_name"),
        cl.JaroWinklerAtThresholds("surname"),
        cl.DateOfBirthComparison("dob", input_is_string=True),
        cl.ExactMatch("city").configure(term_frequency_adjustments=True),
    ],
    blocking_rules_to_generate_predictions=[
        block_on("first_name", "dob"),
        block_on("surname"),
    ],
)

linker = Linker(df, settings, DuckDBAPI())
linker.training.estimate_probability_two_random_records_match(
    [block_on("first_name", "surname")], recall=0.7
)
linker.training.estimate_u_using_random_sampling(max_pairs=1e6)
linker.training.estimate_parameters_using_expectation_maximisation(block_on("surname"))
linker.training.estimate_parameters_using_expectation_maximisation(block_on("dob"))

pairs = linker.inference.predict(threshold_match_probability=0.9)
clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(pairs, 0.95)

Underneath, Splink does not execute the matching itself. It compiles the model into SQL and hands it to a backend. With DuckDB and sensible blocking that means millions of records on a laptop. When the data outgrows the laptop, swap the backend for Spark or Athena - same settings, same model, different engine.

You can read the verdict

For any pair, the waterfall chart starts at the prior and stacks one bar per column: surname +6.1, date of birth +9.8, city -1.4, final probability 0.997. When someone asks why two records were merged, the answer is a picture, not a shrug at a neural network. A probability is only as good as the model behind it: that 0.997 is conditional on the comparisons, prior and estimated parameters being sensible. Blocking decides whether the pair gets scored at all.

This is the feature that decides it. A linkage model that cannot explain a merge cannot be trusted with one.

Where it bites

The threshold is yours. Splink gives probabilities; where to cut between “match” and “review” is a business decision about which error is more expensive.
Clusters chain. A matches B, B matches C, and connected-components puts A and C in one cluster though they share nothing. Raise the clustering threshold and inspect the big clusters.

The 1969 math was never the hard part. Making it run on a hundred million rows and explain itself afterwards was.