Measured against Squawk and pgfence
56 labelled migration files, three linters, default configuration. Built and published by the author of one of the three, which is a conflict of interest, so everything that could be used to rig the result is written down and the corpus ships in the repository.
Generated 12 August 2026 by bench/run.mjs. Platform win32, Node v24.13.0.
Results
| Tool | Hazards named | Any hazard flagged | False positives | Unparseable |
|---|---|---|---|---|
| MigrationPilot | 30/33 (90.9%) | 31/33 | 1/17 (5.9%) | 1/56 |
| Squawk | 20/33 (60.6%) | 24/33 | 1/17 (5.9%) | 0/56 |
| pgfence | 25/33 (75.8%) | 28/33 | 3/17 (17.6%) | 3/56 |
Detection is over the 33 dangerous files; false positives are over the 17 safe ones. “Hazards named” is the strict measure: the tool has to name the specific hazard the file was written to contain, not merely flag the file. “Any hazard flagged” is the looser one, which needs almost no rule mapping and is therefore the hardest number to rig. MigrationPilot finds the most, and ties with Squawk for the fewest false positives.
How the corpus was built
| Category | Files | Verdict | What it tests |
|---|---|---|---|
| unsafe/ | 26 | dangerous | One primary named hazard per file, drawn from handbook entries MPH-001 to MPH-020 |
| safe/ | 16 | safe | The handbook's own safe SQL. Expand/contract, NOT VALID, CONCURRENTLY. False-positive bait |
| context/ | 6 | context-dependent | Real hazards that are harmless at the stated scale. Not scored in either direction |
| agent-flavored/ | 8 | 7 dangerous, 1 safe | Multi-statement migrations in the register coding agents actually emit |
The corpus is derived from the handbook, not from the rule list: each file cites the handbook entry whose hazard it exhibits. Hazards MigrationPilot does not catch stay in the corpus and are reported as misses below. Ground truth lives in a YAML header in each file, so changing a label shows up in the diff. The 33 dangerous files assert 46 (file, hazard) pairs across 25 distinct hazard classes.
What MigrationPilot missed
| File | Hazard missed | Handbook | Caught by |
|---|---|---|---|
| a02-order-status-enum.sql | unbatched-backfill | MPH-010, MPH-018 | nobody |
| a04-soft-delete.sql | ddl-plus-backfill-same-txn | MPH-016, MPH-018, MPH-003 | nobody |
| a04-soft-delete.sql | unbatched-backfill | MPH-016, MPH-018, MPH-003 | nobody |
| u13-concurrently-if-not-exists-retry.sql | invalid-index-retry | MPH-012 | nobody |
| u18-ddl-and-backfill-same-transaction.sql | ddl-plus-backfill-same-txn | MPH-016 | nobody |
Five misses across four files, and no tool in the comparison caught any of them. Two hazard classes account for all five: unbatched backfills, and DDL sharing a transaction with a backfill. Both are open.
These rows are (file, hazard) pairs, which is why four files produce five of them and why the count does not line up with the 30/33 above. That figure is scored per file: 30 of the 33 dangerous files had the hazard they were written to contain named. The 33 files assert 46 pairs between them, so a file carrying several hazards can miss one and still be counted as detected.
The build that was tested
| Tool | Version | Invocation |
|---|---|---|
| MigrationPilot | v1.5.1 (3a07864), v1.6.0 release candidate | node dist/cli.cjs analyze <file> --format json --offline --pg-version <v> |
| Squawk | 2.62.0 | npx squawk-cli@2.62.0 --reporter json --pg-version <v>.0 <files...> |
| pgfence | 0.6.1 | npx @flvmnt/pgfence@0.6.1 analyze --output json --format sql --min-pg-version <v> <files...> |
All three run with their default rule sets: nothing enabled, disabled or configured. That is what a team gets on the day they install the tool, and it is the only configuration comparable across three products with different opinions about defaults. Every tool targets PostgreSQL 17, except on the one corpus file that declares 18 in its header, where all three are told 18.
Atlas is not in the table. It ships a real migration linter and belongs here, but since v0.38 atlas migrate lint aborts without a paid account, so it had no chance to earn a number that a reader could reproduce.
Run it yourself
git clone https://github.com/mickelsamuel/migrationpilot cd migrationpilot && pnpm install && pnpm build node bench/run.mjs