A reusable collection of validation step definitions.
Usage
A Steps object is a list of validation steps that isn’t attached to any table. You build it with the same methods you’d call on a Validate object (col_vals_not_null(), rows_distinct(), col_vals_regex(), and so on), but each call only records the step. Nothing is checked until the collection is added to a validation with Validate.add_steps() and that validation is interrogated.
This makes it possible to write a group of checks once and use it in many places. Many tables share the same rules: every table with a customer ID needs that ID to be present and unique, and every table with an email column needs valid addresses. Without Steps, those rules get copied into each validation and drift apart over time. With Steps, you define them in one place (a shared module, for example), name them for what they check (id_checks, contact_checks), and add them to whichever validations need them. Each validation keeps its own table, thresholds, actions, and metadata, and can mix shared steps with its own.
Since the methods have the same signatures as the Validate methods they mirror, you get the same argument names, defaults, and editor completion, and a misspelled argument raises a TypeError as soon as the step is recorded.
steps: list[Step] | None = NoneStep objects to start the collection with. This is useful when step definitions come from data (a configuration file, for example) rather than from method calls.
StepsValidate.add_steps().
Each validation method records one step and returns the same Steps object, so calls can be chained. All of the row-based and table-level validation methods are available, from col_vals_gt() to tbl_reconcile(). The aggregation methods (col_sum_*(), col_avg_*(), and col_sd_*()), specially(), and prompt() aren’t, so add those directly to the validation.
Because each method adds to the object it’s called on, a collection isn’t copied when you add to it. Calling a method on a shared collection (e.g., extended = id_checks.col_exists(...)) adds the step to id_checks itself, and extended is the same object. To build variations, keep the shared parts in separate collections and add several of them to a validation.
Printing a Steps object lists its steps with the arguments that differ from the defaults, and len() gives the number of steps.
Validate.add_steps() appends the recorded steps to a validation, in order, as if each method had been called on it directly. It takes any number of Steps objects (and also Validate objects, whose steps it copies), and has options for adapting the steps to the validation at hand:
columns_map= renames columns in the steps, so that the same collection works on tables that name a column differently (e.g., {"customer_id": "cust_id"})thresholds= and active= replace the thresholds and active state of every added stepexclude= skips steps by method name or by their 1-based position in a collectionA Step is a single step stored as data: a method name and a dictionary of arguments. A Steps object holds a list of them, built with method calls instead. A Contract also holds a list of Step objects, together with a schema and metadata (an owner, a version, consumers), and can be saved as YAML. Use Steps for reusable building blocks inside Python code, and a Contract for an agreement about a table that’s shared, versioned, and enforced at a pipeline boundary. The two work together: a contract compiled with to_validate() is an ordinary validation, so collections can be added to it with add_steps().
Let’s build two collections of checks for customer data. The first covers identifiers (each customer ID must be present and appear only once), and the second covers contact details (email addresses must look valid, and the country must be one of three codes).
import pointblank as pb
import polars as pl
id_checks = (
pb.Steps()
.col_vals_not_null(columns="customer_id")
.rows_distinct(columns_subset="customer_id")
)
contact_checks = (
pb.Steps()
.col_vals_regex(columns="email", pattern=r"^[^@]+@[^@]+\.[a-z]+$")
.col_vals_in_set(columns="country", set=["CA", "MX", "US"])
)
id_checksDisplaying a collection lists its steps and their arguments. No table is involved yet.
Now let’s apply both collections to a table of customers from a CRM system. The validation is created as usual, and add_steps() appends the four recorded steps before interrogation.
crm_customers = pl.DataFrame(
{
"customer_id": [1, 2, 3, 3, 5],
"email": ["a@x.com", "b@y.org", "c-at-z.com", "d@w.net", "e@v.io"],
"country": ["US", "CA", "US", "MX", "BR"],
}
)
(
pb.Validate(data=crm_customers, tbl_name="crm_customers")
.add_steps(id_checks, contact_checks)
.interrogate()
)The report has one step per recorded step, in the order they were added. Customer ID 3 appears twice, so both of its rows fail the rows_distinct step. The address "c-at-z.com" fails the email pattern, and the country "BR" isn’t in the allowed set.
The same collections can check a table from a billing system, where the ID column is called cust_id. The columns_map= argument renames the column in the added steps, and thresholds= gives every added step a warning threshold of one failing test unit.
billing_accounts = pl.DataFrame(
{
"cust_id": [10, 11, None, 13],
"email": ["f@x.com", "g@y.com", "h@z.com", "i@w"],
"country": ["US", "US", "CA", "US"],
}
)
(
pb.Validate(data=billing_accounts, tbl_name="billing_accounts")
.add_steps(
id_checks,
contact_checks,
columns_map={"customer_id": "cust_id"},
thresholds=pb.Thresholds(warning=1),
)
.interrogate()
)The ID steps now check cust_id, and one account is missing its ID. The address "i@w" has no domain suffix, so it fails the email pattern. Both failing steps reach the warning threshold.
Add the steps in one or more collections to a validation.
A single validation step stored as data.
A schema, steps, and metadata that make up an agreement about a table.
| Name | Description |
|---|---|
| col_vals_gt() | Add a col_vals_gt() step to the collection. |
| col_vals_lt() | Add a col_vals_lt() step to the collection. |
| col_vals_ge() | Add a col_vals_ge() step to the collection. |
| col_vals_le() | Add a col_vals_le() step to the collection. |
| col_vals_eq() | Add a col_vals_eq() step to the collection. |
| col_vals_ne() | Add a col_vals_ne() step to the collection. |
| col_vals_between() | Add a col_vals_between() step to the collection. |
| col_vals_outside() | Add a col_vals_outside() step to the collection. |
| col_vals_in_set() | Add a col_vals_in_set() step to the collection. |
| col_vals_not_in_set() | Add a col_vals_not_in_set() step to the collection. |
| col_vals_increasing() | Add a col_vals_increasing() step to the collection. |
| col_vals_decreasing() | Add a col_vals_decreasing() step to the collection. |
| col_vals_null() | Add a col_vals_null() step to the collection. |
| col_vals_not_null() | Add a col_vals_not_null() step to the collection. |
| col_vals_regex() | Add a col_vals_regex() step to the collection. |
| col_vals_within_spec() | Add a col_vals_within_spec() step to the collection. |
| col_vals_str_len() | Add a col_vals_str_len() step to the collection. |
| col_vals_in_table() | Add a col_vals_in_table() step to the collection. |
| col_vals_expr() | Add a col_vals_expr() step to the collection. |
| col_exists() | Add a col_exists() step to the collection. |
| col_pct_null() | Add a col_pct_null() step to the collection. |
| col_pct_missing() | Add a col_pct_missing() step to the collection. |
| col_missing_coded() | Add a col_missing_coded() step to the collection. |
| col_missing_only_coded() | Add a col_missing_only_coded() step to the collection. |
| col_missing_consistent() | Add a col_missing_consistent() step to the collection. |
| rows_distinct() | Add a rows_distinct() step to the collection. |
| rows_complete() | Add a rows_complete() step to the collection. |
| col_schema_match() | Add a col_schema_match() step to the collection. |
| row_count_match() | Add a row_count_match() step to the collection. |
| col_count_match() | Add a col_count_match() step to the collection. |
| data_freshness() | Add a data_freshness() step to the collection. |
| tbl_match() | Add a tbl_match() step to the collection. |
| tbl_reconcile() | Add a tbl_reconcile() step to the collection. |
| conjointly() | Add a conjointly() step to the collection. |
Add a col_vals_gt() step to the collection.
Usage
The step records a call to Validate.col_vals_gt(), which checks whether column values are greater than a fixed value or the values in another column. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_lt() step to the collection.
Usage
The step records a call to Validate.col_vals_lt(), which checks whether column values are less than a fixed value or the values in another column. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_ge() step to the collection.
Usage
The step records a call to Validate.col_vals_ge(), which checks whether column values are greater than or equal to a fixed value or the values in another column. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_le() step to the collection.
Usage
The step records a call to Validate.col_vals_le(), which checks whether column values are less than or equal to a fixed value or the values in another column. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_eq() step to the collection.
Usage
The step records a call to Validate.col_vals_eq(), which checks whether column values are equal to a fixed value or the values in another column. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_ne() step to the collection.
Usage
The step records a call to Validate.col_vals_ne(), which checks whether column values are not equal to a fixed value or the values in another column. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_between() step to the collection.
Usage
The step records a call to Validate.col_vals_between(), which checks whether column values lie between two bounds, given as fixed values or other columns. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_outside() step to the collection.
Usage
The step records a call to Validate.col_vals_outside(), which checks whether column values lie outside two bounds, given as fixed values or other columns. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_in_set() step to the collection.
Usage
The step records a call to Validate.col_vals_in_set(), which checks whether column values belong to a set of allowed values. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_not_in_set() step to the collection.
Usage
The step records a call to Validate.col_vals_not_in_set(), which checks whether column values avoid a set of disallowed values. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_increasing() step to the collection.
Usage
The step records a call to Validate.col_vals_increasing(), which checks whether column values increase from row to row. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_decreasing() step to the collection.
Usage
The step records a call to Validate.col_vals_decreasing(), which checks whether column values decrease from row to row. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_null() step to the collection.
Usage
The step records a call to Validate.col_vals_null(), which checks whether column values are all Null. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_not_null() step to the collection.
Usage
The step records a call to Validate.col_vals_not_null(), which checks whether column values are free of Nulls. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_regex() step to the collection.
Usage
The step records a call to Validate.col_vals_regex(), which checks whether column values match a regular expression. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_within_spec() step to the collection.
Usage
The step records a call to Validate.col_vals_within_spec(), which checks whether column values fit a specification such as an email address, URL, or postal code. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_str_len() step to the collection.
Usage
The step records a call to Validate.col_vals_str_len(), which checks whether the lengths of string values fall within given bounds. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_in_table() step to the collection.
Usage
The step records a call to Validate.col_vals_in_table(), which checks whether column values exist in a column of a reference table (referential integrity). It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_vals_expr() step to the collection.
Usage
The step records a call to Validate.col_vals_expr(), which checks column values with a custom expression. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_exists() step to the collection.
Usage
The step records a call to Validate.col_exists(), which checks whether one or more columns exist in the table. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_pct_null() step to the collection.
Usage
The step records a call to Validate.col_pct_null(), which checks whether the percentage of Null values in a column matches an expected value. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_pct_missing() step to the collection.
Usage
The step records a call to Validate.col_pct_missing(), which checks whether the percentage of structured missing values in a column stays within a limit. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_missing_coded() step to the collection.
Usage
The step records a call to Validate.col_missing_coded(), which checks whether all missing values in a column are coded (no uncoded Nulls). It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_missing_only_coded() step to the collection.
Usage
The step records a call to Validate.col_missing_only_coded(), which checks whether a column contains only documented missing-value codes and legitimate values. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_missing_consistent() step to the collection.
Usage
The step records a call to Validate.col_missing_consistent(), which checks whether related columns share a consistent pattern of missingness for a given reason. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a rows_distinct() step to the collection.
Usage
The step records a call to Validate.rows_distinct(), which checks whether rows are distinct, across all columns or a subset of them. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a rows_complete() step to the collection.
Usage
The step records a call to Validate.rows_complete(), which checks whether rows are complete, with no missing values across all columns or a subset of them. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_schema_match() step to the collection.
Usage
The step records a call to Validate.col_schema_match(), which checks whether the table’s columns and their types match a schema. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a row_count_match() step to the collection.
Usage
The step records a call to Validate.row_count_match(), which checks whether the table’s row count matches an expected count or the count of another table. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a col_count_match() step to the collection.
Usage
The step records a call to Validate.col_count_match(), which checks whether the table’s column count matches an expected count or the count of another table. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a data_freshness() step to the collection.
Usage
The step records a call to Validate.data_freshness(), which checks whether the latest value in a datetime column is recent enough. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a tbl_match() step to the collection.
Usage
The step records a call to Validate.tbl_match(), which checks whether the table matches a comparison table. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a tbl_reconcile() step to the collection.
Usage
The step records a call to Validate.tbl_reconcile(), which checks whether aggregates of the table reconcile with those of a comparison table. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
StepsAdd a conjointly() step to the collection.
Usage
The step records a call to Validate.conjointly(), which checks whether several row-wise conditions all hold for each row. It takes the same arguments as that method, so see its reference page for what each one does. Nothing is checked when the step is recorded: it runs when the collection is added to a validation with Validate.add_steps() and the validation is interrogated.
The expressions are recorded together in the step’s exprs= argument and passed back as positional arguments when the step is applied.
Steps