Your Cart
Loading

Data Quality Toolkit - SQL & Python Scripts

On Sale
A$47.00
A$47.00
Added to cart

Data profiling and monitoring scripts adaptable for any SQL database. Stop guessing about data quality and start measuring it.

Ever inherited a data warehouse and wondered if anything in it actually works? These are the scripts I've built and refined over years of engagements where teams needed to go from "we think our data is okay" to "here's exactly what's wrong, and here's how we're fixing it." Profile a new dataset in minutes, then turn the checks that matter into production monitoring.


Works where your data lives

No new infrastructure and no vendor lock-in. The SQL sticks to the ANSI standard wherever it can, and each script's header explains the differences for PostgreSQL, SQL Server, MySQL, Oracle, Snowflake, Trino and Delta Lake. Every check gives you a work list rather than a dashboard: the records that failed, the duplicates to merge, the fields to fix.


What you get

  • 15 SQL scripts covering the six data quality dimensions: completeness, uniqueness, timeliness, validity, accuracy and consistency
  • 9 profiling scripts for getting to know a dataset: table statistics with dynamic query generators, missing values, duplicates, value distributions, data types, formats (email, phone, postcode and your own patterns), outliers, cross-column rules, and freshness over time
  • 6 production scripts for ongoing monitoring: checks for completeness, uniqueness, timeliness, validity, and accuracy and consistency, each returning pass, warn or fail, plus the metadata framework that logs their results
  • The metadata framework: seven linked tables covering check ownership, thresholds, results history, failed records, trends and ROI, and three views for current status, 30-day trends and a remediation work queue
  • A Python script for fuzzy duplicates, the "Acme Corp" vs "ACME Corporation" cases SQL misses, using Levenshtein, Jaro-Winkler and phonetic matching with multi-field weighted scoring
  • A business case calculator in Excel, with a worked example of putting a dollar figure on poor data quality


Who it's for

Data engineers and analysts who need to know what's really in a dataset before they build on it, and data quality leads setting up monitoring with owners and thresholds. If you have to justify the work to stakeholders, the calculator helps you put a number on it.


What you need

A SQL database you can query. Read access to information_schema lets the generators write queries for you, and write access lets you create the metadata tables. The fuzzy matching script needs Python 3.8 or later.


Licence

A personal, non-transferable licence for your own use, including at work. Please don't share the scripts publicly, redistribute them, or include them in something you sell. Your Australian Consumer Law rights are protected.

You will get a ZIP (744KB) file