----------------------------------------------------------------------
This is the API documentation for the pointblank library.
----------------------------------------------------------------------


## Validate

When performing data validation, use the `Validate` class to get the process started. It takes the target table and options for metadata and failure thresholds (using the `Thresholds` class or shorthands). The `Validate` class has numerous methods for defining validation steps and for obtaining post-interrogation metrics and data.



Validate(data: 'IntoDataFrame', reference: 'IntoFrame | None' = None, tbl_name: 'str | None' = None, label: 'str | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, final_actions: 'FinalActions | None' = None, brief: 'str | bool | None' = None, lang: 'str | None' = None, locale: 'str | None' = None, owner: 'str | None' = None, consumers: 'str | list[str] | None' = None, version: 'str | None' = None) -> None

Workflow for defining a set of validations on a table and interrogating for results.

The `Validate` class is used for defining a set of validation steps on a table and interrogating
the table with the *validation plan*. This class is the main entry point for the *data quality
reporting* workflow. The overall aim of this workflow is to generate comprehensive reporting
information to assess the level of data quality for a target table.

We can supply as many validation steps as needed, and having a large number of them should
increase the validation coverage for a given table. The validation methods (e.g.,
[`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_between()`](`pointblank.Validate.col_vals_between`), etc.) translate to discrete
validation steps, where each step will be sequentially numbered (useful when viewing the
reporting data). This process of calling validation methods is known as developing a
*validation plan*.

The validation methods, when called, are merely instructions up to the point the concluding
[`interrogate()`](`pointblank.Validate.interrogate`) method is called. That kicks off the
process of acting on the *validation plan* by querying the target table getting reporting
results for each step. Once the interrogation process is complete, we can say that the workflow
now has reporting information. We can then extract useful information from the reporting data
to understand the quality of the table. Printing the `Validate` object (or using the
[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) method) will return a table
with the results of the interrogation and
[`get_sundered_data()`](`pointblank.Validate.get_sundered_data`) allows for the splitting of the
table based on passing and failing rows.

Parameters
----------
data
    The table to validate, which could be a DataFrame object, an Ibis table object, a CSV
    file path, a Parquet file path, a GitHub URL pointing to a CSV or Parquet file, or a
    database connection string. When providing a CSV or Parquet file path (as a string or
    `pathlib.Path` object), the file will be automatically loaded using an available DataFrame
    library (Polars or Pandas). Parquet input also supports glob patterns, directories
    containing .parquet files, and Spark-style partitioned datasets. GitHub URLs are
    automatically transformed to raw content URLs and downloaded. Connection strings enable
    direct database access via Ibis with optional table specification using the `::table_name`
    suffix. Read the *Supported Input Table Types* section for details on the supported table
    types.
reference
    An optional second table to compare the target table against, such as an earlier extract
    of the same data or a trusted baseline. It can be any of the input types accepted by
    `data=`. Reference data is used by the aggregate validation methods (the `col_sum_*`,
    `col_avg_*`, and `col_sd_*` families, such as
    [`col_sum_eq()`](`pointblank.Validate.col_sum_eq`)): a `value=` given with
    [`ref()`](`pointblank.ref`) takes the aggregate of a column in this table, and leaving out
    `value=` compares against the same column here. A file path or connection string given
    here is kept, so that [`to_code()`](`pointblank.Validate.to_code`) and
    [`to_yaml()`](`pointblank.Validate.to_yaml`) can write it back out.
tbl_name
    An optional name to assign to the input table object. If no value is provided, a name will
    be generated based on whatever information is available. This table name will be displayed
    in the header area of the tabular report.
label
    An optional label for the validation plan. If no value is provided, a label will be
    generated based on the current system date and time. Markdown can be used here to make the
    label more visually appealing (it will appear in the header area of the tabular report).
thresholds
    Generate threshold failure levels so that all validation steps can report and react
    accordingly when exceeding the set levels. The thresholds are set at the global level and
    can be overridden at the validation step level (each validation step has its own
    `thresholds=` parameter). The default is `None`, which means that no thresholds will be set.
    Look at the *Thresholds* section for information on how to set threshold levels.
actions
    The actions to take when validation steps meet or exceed any set threshold levels. These
    actions are paired with the threshold levels and are executed during the interrogation
    process when there are exceedances. The actions are executed right after each step is
    evaluated. Such actions should be provided in the form of an `Actions` object. If `None`
    then no global actions will be set. View the *Actions* section for information on how to set
    actions.
final_actions
    The actions to take when the validation process is complete and the final results are
    available. This is useful for sending notifications or reporting the overall status of the
    validation process. The final actions are executed after all validation steps have been
    processed and the results have been collected. The final actions are not tied to any
    threshold levels, they are executed regardless of the validation results. Such actions
    should be provided in the form of a `FinalActions` object. If `None` then no finalizing
    actions will be set. Please see the *Actions* section for information on how to set final
    actions.
brief
    A global setting for briefs, which are optional brief descriptions for validation steps
    (they be displayed in the reporting table). For such a global setting, templating elements
    like `"{step}"` (to insert the step number) or `"{auto}"` (to include an automatically
    generated brief) are useful. If `True` then each brief will be automatically generated. If
    `None` (the default) then briefs aren't globally set.
lang
    The language to use for various reporting elements. By default, `None` will select English
    (`"en"`) as the but other options include French (`"fr"`), German (`"de"`), Italian
    (`"it"`), Spanish (`"es"`), and several more. Have a look at the *Reporting Languages*
    section for the full list of supported languages and information on how the language setting
    is utilized.
locale
    An optional locale ID to use for formatting values in the reporting table according the
    locale's rules. Examples include `"en-US"` for English (United States) and `"fr-FR"` for
    French (France). More simply, this can be a language identifier without a designation of
    territory, like `"es"` for Spanish.
owner
    An optional string identifying the owner of the data being validated. This is useful for
    governance purposes, indicating who is responsible for the quality and maintenance of the
    data. For example, `"data-platform-team"` or `"analytics-engineering"`.
consumers
    An optional string or list of strings identifying who depends on or consumes this data.
    This helps document data dependencies and can be useful for impact analysis when data
    quality issues are detected. For example, `"ml-team"` or `["ml-team", "analytics"]`.
version
    An optional string representing the version of the validation plan or data contract. This
    supports semantic versioning (e.g., `"1.0.0"`, `"2.1.0"`) and is useful for tracking changes
    to validation rules over time and for organizational governance.

Returns
-------
Validate
    A `Validate` object with the table and validations to be performed.

Supported Input Table Types
---------------------------
The `data=` parameter can be given any of the following table types:

- Polars DataFrame (`"polars"`)
- Pandas DataFrame (`"pandas"`)
- PySpark table (`"pyspark"`)
- DuckDB table (`"duckdb"`)*
- MySQL table (`"mysql"`)*
- PostgreSQL table (`"postgresql"`)*
- SQLite table (`"sqlite"`)*
- Microsoft SQL Server table (`"mssql"`)*
- Snowflake table (`"snowflake"`)*
- Databricks table (`"databricks"`)*
- BigQuery table (`"bigquery"`)*
- Parquet table (`"parquet"`)*
- CSV files (string path or `pathlib.Path` object with `.csv` extension)
- Parquet files (string path, `pathlib.Path` object, glob pattern, directory with `.parquet`
extension, or partitioned dataset)
- Database connection strings (URI format with optional table specification)

The table types marked with an asterisk need to be prepared as Ibis tables (with type of
`ibis.expr.types.relations.Table`). Furthermore, the use of `Validate` with such tables requires
the Ibis library v9.5.0 and above to be installed. If the input table is a Polars or Pandas
DataFrame, the Ibis library is not required.

To use a CSV file, ensure that a string or `pathlib.Path` object with a `.csv` extension is
provided. The file will be automatically detected and loaded using the best available DataFrame
library. The loading preference is Polars first, then Pandas as a fallback.

Connection strings follow database URL formats and must also specify a table using the
`::table_name` suffix. Examples include:

```
"duckdb:///path/to/database.ddb::table_name"
"sqlite:///path/to/database.db::table_name"
"postgresql://user:password@localhost:5432/database::table_name"
"mysql://user:password@localhost:3306/database::table_name"
"bigquery://project/dataset::table_name"
"snowflake://user:password@account/database/schema::table_name"
```

When using connection strings, the Ibis library with the appropriate backend driver is required.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for all validation
steps. They are set here at the global level but can be overridden at the validation step level
(each validation step has its own local `thresholds=` parameter).

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values can
either be set as a proportion failing of all test units (a value between `0` to `1`), or, the
absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is the
'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units for
the 'warning' level only

If the number of failing test units for a validation step exceeds set thresholds, the validation
step will be marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need
to be set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to take
for each level of failure (using the `actions=` parameter).

Actions
-------
The `actions=` and `final_actions=` parameters provide mechanisms to respond to validation
results. These actions can be used to notify users of validation failures, log issues, or
trigger other processes when problems are detected.

*Step Actions*

The `actions=` parameter allows you to define actions that are triggered when validation steps
exceed specific threshold levels (warning, error, or critical). These actions are executed
during the interrogation process, right after each step is evaluated.

Step actions should be provided using the [`Actions`](`pointblank.Actions`) class, which lets
you specify different actions for different severity levels:

```python
# Define an action that logs a message when warning threshold is exceeded
def log_warning():
    metadata = pb.get_action_metadata()
    print(f"WARNING: Step {metadata['step']} failed with type {metadata['type']}")

# Define actions for different threshold levels
actions = pb.Actions(
    warning = log_warning,
    error = lambda: send_email("Error in validation"),
    critical = "CRITICAL FAILURE DETECTED"
)

# Use in Validate
validation = pb.Validate(
    data=my_data,
    actions=actions  # Global actions for all steps
)
```

You can also provide step-specific actions in individual validation methods:

```python
validation.col_vals_gt(
    columns="revenue",
    value=0,
    actions=pb.Actions(warning=log_warning)  # Only applies to this step
)
```

Step actions have access to step-specific context through the
[`get_action_metadata()`](`pointblank.get_action_metadata`) function, which provides details
about the current validation step that triggered the action.

*Final Actions*

The `final_actions=` parameter lets you define actions that execute after all validation steps
have completed. These are useful for providing summaries, sending notifications based on
overall validation status, or performing cleanup operations.

Final actions should be provided using the [`FinalActions`](`pointblank.FinalActions`) class:

```python
def send_report():
    summary = pb.get_validation_summary()
    if summary["status"] == "CRITICAL":
        send_alert_email(
            subject=f"CRITICAL validation failures in {summary['tbl_name']}",
            body=f"{summary['critical_steps']} steps failed with critical severity."
        )

validation = pb.Validate(
    data=my_data,
    final_actions=pb.FinalActions(send_report)
)
```

Final actions have access to validation-wide summary information through the
[`get_validation_summary()`](`pointblank.get_validation_summary`) function, which provides a
comprehensive overview of the entire validation process.

The combination of step actions and final actions provides a flexible system for responding to
data quality issues at both the individual step level and the overall validation level.

Reporting Languages
-------------------
Various pieces of reporting in Pointblank can be localized to a specific language. This is done
by setting the `lang=` parameter in `Validate`. Any of the following languages can be used (just
provide the language code):

- English (`"en"`)
- French (`"fr"`)
- German (`"de"`)
- Italian (`"it"`)
- Spanish (`"es"`)
- Portuguese (`"pt"`)
- Dutch (`"nl"`)
- Swedish (`"sv"`)
- Danish (`"da"`)
- Norwegian Bokmål (`"nb"`)
- Icelandic (`"is"`)
- Finnish (`"fi"`)
- Polish (`"pl"`)
- Czech (`"cs"`)
- Romanian (`"ro"`)
- Greek (`"el"`)
- Russian (`"ru"`)
- Turkish (`"tr"`)
- Arabic (`"ar"`)
- Hindi (`"hi"`)
- Simplified Chinese (`"zh-Hans"`)
- Traditional Chinese (`"zh-Hant"`)
- Japanese (`"ja"`)
- Korean (`"ko"`)
- Vietnamese (`"vi"`)
- Indonesian (`"id"`)
- Ukrainian (`"uk"`)
- Bulgarian (`"bg"`)
- Croatian (`"hr"`)
- Estonian (`"et"`)
- Hungarian (`"hu"`)
- Irish (`"ga"`)
- Latvian (`"lv"`)
- Lithuanian (`"lt"`)
- Maltese (`"mt"`)
- Slovak (`"sk"`)
- Slovenian (`"sl"`)
- Hebrew (`"he"`)
- Thai (`"th"`)
- Persian (`"fa"`)

Automatically generated briefs (produced by using `brief=True` or `brief="...{auto}..."`) will
be written in the selected language. The language setting will also used when generating the
validation report table through
[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) (or printing the `Validate`
object in a notebook environment).

Examples
--------
### Creating a validation plan and interrogating

Let's walk through a data quality analysis of an extremely small table. It's actually called
`"small_table"` and it's accessible through the [`load_dataset()`](`pointblank.load_dataset`)
function.

```{python}
import pointblank as pb

# Load the `small_table` dataset
small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

# Preview the table
pb.preview(small_table)
```

We ought to think about what's tolerable in terms of data quality so let's designate
proportional failure thresholds to the 'warning', 'error', and 'critical' states. This can be
done by using the [`Thresholds`](`pointblank.Thresholds`) class.

```{python}
thresholds = pb.Thresholds(warning=0.10, error=0.25, critical=0.35)
```

Now, we use the `Validate` class and give it the `thresholds` object (which serves as a default
for all validation steps but can be overridden). The static thresholds provided in `thresholds=`
will make the reporting a bit more useful. We also need to provide a target table and we'll use
`small_table` for this.

```{python}
validation = (
    pb.Validate(
        data=small_table,
        tbl_name="small_table",
        label="`Validate` example.",
        thresholds=thresholds
    )
)
```

Then, as with any `Validate` object, we can add steps to the validation plan by using as many
validation methods as we want. To conclude the process (and actually query the data table), we
use the [`interrogate()`](`pointblank.Validate.interrogate`) method.

```{python}
validation = (
    validation
    .col_vals_gt(columns="d", value=100)
    .col_vals_le(columns="c", value=5)
    .col_vals_between(columns="c", left=3, right=10, na_pass=True)
    .col_vals_regex(columns="b", pattern=r"[0-9]-[a-z]{3}-[0-9]{3}")
    .col_exists(columns=["date", "date_time"])
    .interrogate()
)
```

The `validation` object can be printed as a reporting table.

```{python}
validation
```

The report could be further customized by using the
[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) method, which contains
options for modifying the display of the table.

### Adding briefs

Briefs are short descriptions of the validation steps. While they can be set for each step
individually, they can also be set globally. The global setting is done by using the
`brief=` argument in `Validate`. The global setting can be as simple as `True` to have
automatically-generated briefs for each step. Alternatively, we can use templating elements
like `"{step}"` (to insert the step number) or `"{auto}"` (to include an automatically generated
brief). Here's an example of a global setting for briefs:

```{python}
validation_2 = (
    pb.Validate(
        data=pb.load_dataset(),
        tbl_name="small_table",
        label="Validation example with briefs",
        brief="Step {step}: {auto}",
    )
    .col_vals_gt(columns="d", value=100)
    .col_vals_between(columns="c", left=3, right=10, na_pass=True)
    .col_vals_regex(
        columns="b",
        pattern=r"[0-9]-[a-z]{3}-[0-9]{3}",
        brief="Regex check for column {col}"
    )
    .interrogate()
)

validation_2
```

We see the text of the briefs appear in the `STEP` column of the reporting table. Furthermore,
the global brief's template (`"Step {step}: {auto}"`) is applied to all steps except for the
final step, where the step-level `brief=` argument provided an override.

If you should want to cancel the globally-defined brief for one or more validation steps, you
can set `brief=False` in those particular steps.

### Post-interrogation methods

The `Validate` class has a number of post-interrogation methods that can be used to extract
useful information from the validation results. For example, the
[`get_data_extracts()`](`pointblank.Validate.get_data_extracts`) method can be used to get
the data extracts for each validation step.

```{python}
validation_2.get_data_extracts()
```

We can also view step reports for each validation step using the
[`get_step_report()`](`pointblank.Validate.get_step_report`) method. This method adapts to the
type of validation step and shows the relevant information for a step's validation.

```{python}
validation_2.get_step_report(i=2)
```

The `Validate` class also has a method for getting the sundered data, which is the data that
passed or failed the validation steps. This can be done using the
[`get_sundered_data()`](`pointblank.Validate.get_sundered_data`) method.

```{python}
pb.preview(validation_2.get_sundered_data())
```

The sundered data is a DataFrame that contains the rows that passed or failed the validation.
The default behavior is to return the rows that failed the validation, as shown above.

### Working with CSV Files

The `Validate` class can directly accept CSV file paths, making it easy to validate data stored
in CSV files without manual loading:

```{python}
# Get a path to a CSV file from the package data
csv_path = pb.get_data_path("global_sales", "csv")

validation_3 = (
    pb.Validate(
        data=csv_path,
        label="CSV validation example"
    )
    .col_exists(["customer_id", "product_id", "revenue"])
    .col_vals_not_null(["customer_id", "product_id"])
    .col_vals_gt(columns="revenue", value=0)
    .interrogate()
)

validation_3
```

You can also use a Path object to specify the CSV file. Here's an example of how to do that:

```{python}
from pathlib import Path

csv_file = Path(pb.get_data_path("game_revenue", "csv"))

validation_4 = (
    pb.Validate(data=csv_file, label="Game Revenue Validation")
    .col_exists(["player_id", "session_id", "item_name"])
    .col_vals_regex(
        columns="session_id",
        pattern=r"[A-Z0-9]{8}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{12}"
    )
    .col_vals_gt(columns="item_revenue", value=0, na_pass=True)
    .interrogate()
)

validation_4
```

The CSV loading is automatic, so when a string or Path with a `.csv` extension is provided,
Pointblank will automatically load the file using the best available DataFrame library (Polars
preferred, Pandas as fallback). The loaded data can then be used with all validation methods
just like any other supported table type.

### Working with Parquet Files

The `Validate` class can directly accept Parquet files and datasets in various formats. The
following examples illustrate how to validate Parquet files:

```{python}
# Single Parquet file from package data
parquet_path = pb.get_data_path("nycflights", "parquet")

validation_5 = (
    pb.Validate(
        data=parquet_path,
        tbl_name="NYC Flights Data"
    )
    .col_vals_not_null(["carrier", "origin", "dest"])
    .col_vals_gt(columns="distance", value=0)
    .interrogate()
)

validation_5
```

You can also use glob patterns and directories. Here are some examples for how to:

1. load multiple Parquet files
2. load a Parquet-containing directory
3. load a partitioned Parquet dataset

```python
# Multiple Parquet files with glob patterns
validation_6 = pb.Validate(data="data/sales_*.parquet")

# Directory containing Parquet files
validation_7 = pb.Validate(data="parquet_data/")

# Partitioned Parquet dataset
validation_8 = (
    pb.Validate(data="sales_data/")  # Contains year=2023/quarter=Q1/region=US/sales.parquet
    .col_exists(["transaction_id", "amount", "year", "quarter", "region"])
    .interrogate()
)
```

When you point to a directory that contains a partitioned Parquet dataset (with subdirectories
like `year=2023/quarter=Q1/region=US/`), Pointblank will automatically:

- discover all Parquet files recursively
- extract partition column values from directory paths
- add partition columns to the final DataFrame
- combine all partitions into a single table for validation

Both Polars and Pandas handle partitioned datasets natively, so this works seamlessly with
either DataFrame library. The loading preference is Polars first, then Pandas as a fallback.

### Working with Database Connection Strings

The `Validate` class supports database connection strings for direct validation of database
tables. Connection strings must specify a table using the `::table_name` suffix:

```{python}
# Get path to a DuckDB database file from package data
duckdb_path = pb.get_data_path("game_revenue", "duckdb")

validation_9 = (
    pb.Validate(
        data=f"duckdb:///{duckdb_path}::game_revenue",
        label="DuckDB Game Revenue Validation"
    )
    .col_exists(["player_id", "session_id", "item_revenue"])
    .col_vals_gt(columns="item_revenue", value=0)
    .interrogate()
)

validation_9
```

For comprehensive documentation on supported connection string formats, error handling, and
installation requirements, see the [`connect_to_table()`](`pointblank.connect_to_table`)
function. This function handles all the connection logic and provides helpful error messages
when table specifications are missing or backend dependencies are not installed.

Thresholds(warning: 'int | float | bool | None' = None, error: 'int | float | bool | None' = None, critical: 'int | float | bool | None' = None) -> None

Definition of threshold values.

Thresholds are used to set limits on the number of failing test units at different levels. The
levels are 'warning', 'error', and 'critical'. These levels correspond to different levels of
severity when a threshold is reached. The threshold values can be set as absolute counts or as
fractions of the total number of test units. When a threshold is reached, an action can be taken
(e.g., displaying a message or calling a function) if there is an associated action defined for
that level (defined through the [`Actions`](`pointblank.Actions`) class).

Parameters
----------
warning
    The threshold for the 'warning' level. This can be an absolute count or a fraction of the
    total. Using `True` will set this threshold value to `1`.
error
    The threshold for the 'error' level. This can be an absolute count or a fraction of the
    total. Using `True` will set this threshold value to `1`.
critical
    The threshold for the 'critical' level. This can be an absolute count or a fraction of the
    total. Using `True` will set this threshold value to `1`.

Returns
-------
Thresholds
    A `Thresholds` object. This can be used when using the [`Validate`](`pointblank.Validate`)
    class (to set thresholds globally) or when defining validation steps like
    [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`) (so that threshold values are scoped to
    individual validation steps, overriding any global thresholds).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```
In a data validation workflow, you can set thresholds for the number of failing test units at
different levels. For example, you can set a threshold for the 'warning' level when the number
of failing test units exceeds 10% of the total number of test units:

```{python}
thresholds_1 = pb.Thresholds(warning=0.1)
```

You can also set thresholds for the 'error' and 'critical' levels:

```{python}
thresholds_2 = pb.Thresholds(warning=0.1, error=0.2, critical=0.05)
```

Thresholds can also be set as absolute counts. Here's an example where the 'warning' level is
set to `5` failing test units:

```{python}
thresholds_3 = pb.Thresholds(warning=5)
```

The `thresholds` object can be used to set global thresholds for all validation steps. Or, you
can set thresholds for individual validation steps, which will override the global thresholds.
Here's a data validation workflow example where we set global thresholds and then override with
different thresholds at the [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`) step:

```{python}
validation = (
    pb.Validate(
        data=pb.load_dataset(dataset="small_table"),
        label="Example Validation",
        thresholds=pb.Thresholds(warning=0.1, error=0.2, critical=0.3)
    )
    .col_vals_not_null(columns=["c", "d"])
    .col_vals_gt(columns="a", value=3, thresholds=pb.Thresholds(warning=5))
    .interrogate()
)

validation
```

As can be seen, the last step ([`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)) has its own
thresholds, which override the global thresholds set at the beginning of the validation workflow
(in the [`Validate`](`pointblank.Validate`) class).

Actions(warning: 'str | Callable | list[str | Callable] | None' = None, error: 'str | Callable | list[str | Callable] | None' = None, critical: 'str | Callable | list[str | Callable] | None' = None, default: 'str | Callable | list[str | Callable] | None' = None, highest_only: 'bool' = True) -> None

Definition of action values.

Actions complement threshold values by defining what action should be taken when a threshold
level is reached. The action can be a string or a `Callable`. When a string is used, it is
interpreted as a message to be displayed. When a `Callable` is used, it will be invoked at
interrogation time if the threshold level is met or exceeded.

There are three threshold levels: 'warning', 'error', and 'critical'. These levels correspond
to different levels of severity when a threshold is reached. Those thresholds can be defined
using the [`Thresholds`](`pointblank.Thresholds`) class or various shorthand forms. Actions
don't have to be defined for all threshold levels; if an action is not defined for a level in
exceedance, no action will be taken. Likewise, there is no negative consequence (other than a
no-op) for defining actions for thresholds that don't exist (e.g., setting an action for the
'critical' level when no corresponding 'critical' threshold has been set).

Parameters
----------
warning
    A string, `Callable`, or list of `Callable`/string values for the 'warning' level. Using
    `None` means no action should be performed at the 'warning' level.
error
    A string, `Callable`, or list of `Callable`/string values for the 'error' level. Using
    `None` means no action should be performed at the 'error' level.
critical
    A string, `Callable`, or list of `Callable`/string values for the 'critical' level. Using
    `None` means no action should be performed at the 'critical' level.
default
    A string, `Callable`, or list of `Callable`/string values for all threshold levels. This
    parameter can be used to set the same action for all threshold levels. If an action is
    defined for a specific threshold level, it will override the action set for all levels.
highest_only
    A boolean value that, when set to `True` (the default), results in executing only the action
    for the highest threshold level that is exceeded. Useful when you want to ensure that only
    the most severe action is taken when multiple threshold levels are exceeded.

Returns
-------
Actions
    An `Actions` object. This can be used when using the [`Validate`](`pointblank.Validate`)
    class (to set actions for meeting different threshold levels globally) or when defining
    validation steps like [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`) (so that actions
    are scoped to individual validation steps, overriding any globally set actions).

Types of Actions
----------------
Actions can be defined in different ways:

1. **String**: A message to be displayed when the threshold level is met or exceeded.
2. **Callable**: A function that is called when the threshold level is met or exceeded.
3. **List of Strings/Callables**: Multiple messages or functions to be called when the threshold
   level is met or exceeded.

The actions are executed at interrogation time when the threshold level assigned to the action
is exceeded by the number or proportion of failing test units. When providing a string, it will
simply be printed to the console. A callable will also be executed at the time of interrogation.
If providing a list of strings or callables, each item in the list will be executed in order.
Such a list can contain a mix of strings and callables.

String Templating
-----------------
When using a string as an action, you can include placeholders for the following variables:

- `{type}`: The validation step type where the action is executed (e.g., 'col_vals_gt',
'col_vals_lt', etc.)
- `{level}`: The threshold level where the action is executed ('warning', 'error', or
'critical')
- `{step}` or `{i}`: The step number in the validation workflow where the action is executed
- `{col}` or `{column}`: The column name where the action is executed
- `{val}` or `{value}`: An associated value for the validation method (e.g., the value to
compare against in a 'col_vals_gt' validation step)
- `{time}`: A datetime value for when the action was executed

The first two placeholders can also be used in uppercase (e.g., `{TYPE}` or `{LEVEL}`) and the
corresponding values will be displayed in uppercase. The placeholders are replaced with the
actual values during interrogation.

For example, the string `"{LEVEL}: '{type}' threshold exceeded for column {col}."` will be
displayed as `"WARNING: 'col_vals_gt' threshold exceeded for column a."` when the 'warning'
threshold is exceeded in a 'col_vals_gt' validation step involving column `a`.

Crafting Callables with `get_action_metadata()`
-----------------------------------------------
When creating a callable function to be used as an action, you can use the
[`get_action_metadata()`](`pointblank.get_action_metadata`) function to retrieve metadata about
the step where the action is executed. This metadata contains information about the validation
step, including the step type, level, step number, column name, and associated value. You can
use this information to craft your action message or to take specific actions based on the
metadata provided.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Let's define both threshold values and actions for a data validation workflow. We'll set these
thresholds and actions globally for all validation steps. In this specific example, the only
actions we'll define are for the 'critical' level:

```{python}
import pointblank as pb

validation = (
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),
        actions=pb.Actions(critical="Major data quality issue found in step {step}."),
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}[0-9]{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .col_vals_gt(columns="session_duration", value=15)
    .interrogate()
)

validation
```

Because we set the 'critical' action to display `"Major data quality issue found."` in the
console, this message will be displayed if the number of failing test units exceeds the
'critical' threshold (set to 15% of the total number of test units). In step 3 of the validation
workflow, the 'critical' threshold is exceeded, so the message is displayed in the console.

Actions can be defined locally for individual validation steps, which will override any global
actions set at the beginning of the validation workflow. Here's a variation of the above example
where we set global threshold values but assign an action only for an individual validation
step:

```{python}
def dq_issue():
    from datetime import datetime

    print(f"Data quality issue found ({datetime.now()}).")

validation = (
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}[0-9]{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .col_vals_gt(
        columns="session_duration",
        value=15,
        actions=pb.Actions(warning=dq_issue),
    )
    .interrogate()
)

validation
```

In this case, the 'warning' action is set to call the `dq_issue()` function. This action is
only executed when the 'warning' threshold is exceeded in the 'session_duration' column. Because
all three thresholds are exceeded in step 3, the 'warning' action of executing the function
occurs (resulting in a message being printed to the console). If actions were set for the other
two threshold levels, they would also be executed.

See Also
--------
get_action_metadata : Get metadata about the step where an action is executed.
FinalActions : Define actions that run after all validation steps are done.

FinalActions(*actions) -> 'None'

Define actions to be taken after validation is complete.

Final actions are executed after all validation steps have been completed. They provide a
mechanism to respond to the overall validation results, such as sending alerts when critical
failures are detected or generating summary reports.

Parameters
----------
*actions
    One or more actions to execute after validation. An action can be (1) a callable function
    that will be executed with no arguments, or (2) a string message that will be printed to the
    console.

Returns
-------
FinalActions
    An `FinalActions` object. This can be used when using the
    [`Validate`](`pointblank.Validate`) class (to set final actions for the validation
    workflow).

Types of Actions
----------------
Final actions can be defined in two different ways:

1. **String**: A message to be displayed when the validation is complete.
2. **Callable**: A function that is called when the validation is complete.

The actions are executed at the end of the validation workflow. When providing a string, it will
simply be printed to the console. A callable will also be executed at the time of validation
completion. Several strings and callables can be provided to the `FinalActions` class, and
they will be executed in the order they are provided.

Crafting Callables with `get_validation_summary()`
-------------------------------------------------
When creating a callable function to be used as a final action, you can use the
[`get_validation_summary()`](`pointblank.get_validation_summary`) function to retrieve the
summary of the validation results. This summary contains information about the validation
workflow, including the number of test units, the number of failing test units, and the
threshold levels that were exceeded. You can use this information to craft your final action
message or to take specific actions based on the validation results.

Examples
--------
Final actions provide a powerful way to respond to the overall results of a validation workflow.
They're especially useful for sending notifications, generating reports, or taking corrective
actions based on the complete validation outcome.

The following example shows how to create a final action that checks for critical failures
and sends an alert. The `d` column of the `small_table` dataset has many values that aren't
greater than `1000`, so the step reaches the 'critical' threshold:

```{python}
import pointblank as pb

def send_alert():
    summary = pb.get_validation_summary()
    if summary["highest_severity"] == "critical":
        print(f"ALERT: Critical validation failures found in {summary['tbl_name']}")

validation = (
    pb.Validate(
        data=pb.load_dataset("small_table"),
        tbl_name="small_table",
        thresholds=(1, 2, 3),
        final_actions=pb.FinalActions(send_alert),
    )
    .col_vals_gt(columns="d", value=1000)
    .interrogate()
)
```

In this example, the `send_alert()` function is defined to check the validation summary for
critical failures. Since some were found, the alert message was printed. The function is
passed to the `FinalActions` class, which ensures it will be executed after all validation steps
are complete. Note that we used the
[`get_validation_summary()`](`pointblank.get_validation_summary`) function to retrieve the
summary of the validation results to help craft the alert message.

Multiple final actions can be provided in a sequence. They will be executed in the order they
are specified after all validation steps have completed. Here, a string message is followed by
the `send_alert()` function and a second function that prints a count of the failing steps:

```{python}
def log_counts():
    summary = pb.get_validation_summary()
    print(f"{summary['n_failing_steps']} of {summary['n_steps']} steps had failures")

validation = (
    pb.Validate(
        data=pb.load_dataset("small_table"),
        tbl_name="small_table",
        thresholds=(1, 2, 3),
        final_actions=pb.FinalActions(
            "Validation complete.",  # a string message
            send_alert,              # a callable function
            log_counts,              # another callable function
        ),
    )
    .col_vals_gt(columns="d", value=1000)
    .col_vals_not_null(columns="c")
    .col_exists(columns="a")
    .interrogate()
)
```

The three outputs appear in the order the actions were given. A string action is printed as
is, while the functions can use the summary to decide what to do. Two of the three steps have
failures (`d` has values of `1000` or less and `c` has missing values), and only the
`col_exists()` step passes.

See Also
--------
get_validation_summary : Get a summary of the validation results inside a final action.
Actions : Define actions that run when a step exceeds a threshold.

Schema(columns: 'str | list[str] | list[tuple[str, str]] | list[tuple[str]] | dict[str, str] | None' = None, tbl: 'Any | None' = None, **kwargs) -> 'None'

Definition of a schema object.

The schema object defines the structure of a table. Once it is defined, the object can be used
in a validation workflow, using `Validate` and its methods, to ensure that the structure of a
table matches the expected schema. The validation method that works with the schema object is
called [`col_schema_match()`](`pointblank.Validate.col_schema_match`).

A schema for a table can be constructed with the `Schema` class in a number of ways:

1. providing a list of column names to `columns=` (to check only the column names)
2. using a list of one- or two-element tuples in `columns=` (to check both column names and
optionally dtypes, should be in the form of `[(column_name, dtype), ...]`)
3. providing a dictionary to `columns=`, where the keys are column names and the values are
dtypes
4. providing individual column arguments in the form of keyword arguments (constructed as
`column_name=dtype`)

The schema object can also be constructed by providing a DataFrame or Ibis table object (using
the `tbl=` parameter) and the schema will be collected from either type of object. The schema
object can be printed to display the column names and dtypes. Note that if `tbl=` is provided
then there shouldn't be any other inputs provided through either `columns=` or `**kwargs`.

Parameters
----------
columns
    A list of strings (representing column names), a list of tuples (for column names and column
    dtypes), or a dictionary containing column and dtype information. If any of these inputs are
    provided here, it will take precedence over any column arguments provided via `**kwargs`.
tbl
    A DataFrame (Polars or Pandas) or an Ibis table object from which the schema will be
    collected. Read the *Supported Input Table Types* section for details on the supported
    table types.
**kwargs
    Individual column arguments that are in the form of `column=dtype` or
    `column=[dtype1, dtype2, ...]`. These will be ignored if the `columns=` parameter is not
    `None`.

Returns
-------
Schema
    A schema object.

Supported Input Table Types
---------------------------
The `tbl=` parameter, if used, can be given any of the following table types:

- Polars DataFrame (`"polars"`)
- Pandas DataFrame (`"pandas"`)
- PySpark table (`"pyspark"`)
- DuckDB table (`"duckdb"`)*
- MySQL table (`"mysql"`)*
- PostgreSQL table (`"postgresql"`)*
- SQLite table (`"sqlite"`)*
- Microsoft SQL Server table (`"mssql"`)*
- Snowflake table (`"snowflake"`)*
- Databricks table (`"databricks"`)*
- BigQuery table (`"bigquery"`)*
- Parquet table (`"parquet"`)*

The table types marked with an asterisk need to be prepared as Ibis tables (with type of
`ibis.expr.types.relations.Table`). Furthermore, using `Schema(tbl=)` with these types of tables
requires the Ibis library (`v9.5.0` or above) to be installed. If the input table is a Polars or
Pandas DataFrame, the availability of Ibis is not needed.

Additional Notes on Schema Construction
---------------------------------------
While there is flexibility in how a schema can be constructed, there is the potential for some
confusion. So let's go through each of the methods of constructing a schema in more detail and
single out some important points.

When providing a list of column names to `columns=`, a
[`col_schema_match()`](`pointblank.Validate.col_schema_match`) validation step will only check
the column names. Any arguments pertaining to dtypes will be ignored.

When using a list of tuples in `columns=`, the tuples could contain the column name and dtype
or just the column name. This construction allows for more flexibility in constructing the
schema as some columns will be checked for dtypes and others will not. This method is the only
way to have mixed checks of column names and dtypes in
[`col_schema_match()`](`pointblank.Validate.col_schema_match`).

When providing a dictionary to `columns=`, the keys are the column names and the values are the
dtypes. This method of input is useful in those cases where you might already have a dictionary
of column names and dtypes that you want to use as the schema.

If using individual column arguments in the form of keyword arguments, the column names are the
keyword arguments and the dtypes are the values. This method emphasizes readability and is
perhaps more convenient when manually constructing a schema with a small number of columns.

Finally, multiple dtypes can be provided for a single column by providing a list or tuple of
dtypes in place of a scalar string value. Having multiple dtypes for a column allows for the
dtype check via [`col_schema_match()`](`pointblank.Validate.col_schema_match`) to make multiple
attempts at matching the column dtype. Should any of the dtypes match the column dtype, that
part of the schema check will pass. Here are some examples of how you could provide single and
multiple dtypes for a column:

```python
# list of tuples
schema_1 = pb.Schema(columns=[("name", "String"), ("age", ["Float64", "Int64"])])

# dictionary
schema_2 = pb.Schema(columns={"name": "String", "age": ["Float64", "Int64"]})

# keyword arguments
schema_3 = pb.Schema(name="String", age=["Float64", "Int64"])
```

All of the above examples will construct the same schema object.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
A schema can be constructed via the `Schema` class in multiple ways. Let's use the following
Polars DataFrame as a basis for constructing a schema:

```{python}
import pointblank as pb
import polars as pl

df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "height": [5.6, 6.0, 5.8]
})
```

You could provide `Schema(columns=)` a list of tuples containing column names and data types:

```{python}
schema = pb.Schema(columns=[("name", "String"), ("age", "Int64"), ("height", "Float64")])
```

Alternatively, a dictionary containing column names and dtypes also works:

```{python}
schema = pb.Schema(columns={"name": "String", "age": "Int64", "height": "Float64"})
```

Another input method involves using individual column arguments in the form of keyword
arguments:

```{python}
schema = pb.Schema(name="String", age="Int64", height="Float64")
```

Finally, could also provide a DataFrame (Polars and Pandas) or an Ibis table object to `tbl=`
and the schema will be collected:

```python
schema = pb.Schema(tbl=df)
```

Whichever method you choose, you can verify the schema inputs by printing the `schema` object:

```{python}
print(schema)
```

The `Schema` object can be used to validate the structure of a table against the schema. The
relevant `Validate` method for this is
[`col_schema_match()`](`pointblank.Validate.col_schema_match`). In a validation workflow, you'll
have a target table (defined at the beginning of the workflow) and you might want to ensure that
your expectations of the table structure are met. The
[`col_schema_match()`](`pointblank.Validate.col_schema_match`) method works with a `Schema`
object to validate the structure of the table. Here's an example of how you could use
[`col_schema_match()`](`pointblank.Validate.col_schema_match`) in a validation workflow:

```{python}
# Define the schema
schema = pb.Schema(name="String", age="Int64", height="Float64")

# Define a validation that checks the schema against the table (`df`)
validation = (
    pb.Validate(data=df)
    .col_schema_match(schema=schema)
    .interrogate()
)

# Display the validation results
validation
```

The [`col_schema_match()`](`pointblank.Validate.col_schema_match`) validation method will
validate the structure of the table against the schema during interrogation. If the structure of
the table does not match the schema, the single test unit will fail. In this case, the defined
schema matched the structure of the table, so the validation passed.

We can also choose to check only the column names of the target table. This can be done by
providing a simplified `Schema` object, which is given a list of column names:

```{python}
schema = pb.Schema(columns=["name", "age", "height"])

validation = (
    pb.Validate(data=df)
    .col_schema_match(schema=schema)
    .interrogate()
)

validation
```

In this case, the schema only checks the column names of the table against the schema during
interrogation. If the column names of the table do not match the schema, the single test unit
will fail. In this case, the defined schema matched the column names of the table, so the
validation passed.

If you wanted to check column names and dtypes only for a subset of columns (and just the column
names for the rest), you could use a list of mixed one- or two-item tuples in `columns=`:

```{python}
schema = pb.Schema(columns=[("name", "String"), ("age", ), ("height", )])

validation = (
    pb.Validate(data=df)
    .col_schema_match(schema=schema)
    .interrogate()
)

validation
```

Not specifying a dtype for a column (as is the case for the `age` and `height` columns in the
above example) will only check the column name.

There may also be the case where you want to check the column names and specify multiple dtypes
for a column to have several attempts at matching the dtype. This can be done by providing a
list of dtypes where there would normally be a single dtype:

```{python}
schema = pb.Schema(
  columns=[("name", "String"), ("age", ["Float64", "Int64"]), ("height", "Float64")]
)

validation = (
    pb.Validate(data=df)
    .col_schema_match(schema=schema)
    .interrogate()
)

validation
```

For the `age` column, the schema will check for both `Float64` and `Int64` dtypes. If either of
these dtypes is found in the column, the portion of the schema check will succeed.

See Also
--------
Validate.col_schema_match : The validation step that checks a table against a `Schema`.

DraftValidation(data: 'Any', model: 'str', api_key: 'str | None' = None, verify_ssl: 'bool' = True, max_reprompts: 'int' = 1) -> None

Draft a validation plan for a given table using an LLM.

By using a large language model (LLM) to draft a validation plan, you can quickly generate a
starting point for validating a table. This can be useful when you have a new table and you
want to get a sense of how to validate it (and adjustments could always be made later). The
`DraftValidation` class uses the `chatlas` package to draft a validation plan for a given table
using an LLM from the `"anthropic"`, `"openai"`, `"ollama"`, `"bedrock"`, or `"azure-openai"`
provider. You can install all requirements for the class through an optional 'generate' install
of Pointblank via `pip install pointblank[generate]`.

:::{.callout-warning}
The `DraftValidation` class is still experimental. Please report any issues you encounter in
the [Pointblank issue tracker](https://github.com/posit-dev/pointblank/issues).
:::

Parameters
----------
data
    The data to be used for drafting a validation plan.
model
    The model to be used. This should be in the form of `provider:model` (e.g.,
    `"anthropic:claude-opus-4-6"`). Supported providers are `"anthropic"`, `"openai"`,
    `"ollama"`, `"bedrock"`, and `"azure-openai"`. For `"azure-openai"`, the value after the
    colon is the Azure *deployment id*, not an OpenAI model id.
api_key
    The API key to be used for the model.
verify_ssl
    Whether to verify SSL certificates when making requests to the LLM provider. Set to `False`
    to disable SSL verification (e.g., when behind a corporate firewall with self-signed
    certificates). Defaults to `True`. Use with caution as disabling SSL verification can pose
    security risks.
max_reprompts
    The maximum number of times to ask the model for a corrected plan when the one it returned
    doesn't pass a basic check: the code must parse as Python, and every method that looks like
    a validation method (a name starting with `col_`, `row_`, or `rows_`) must exist, since
    models sometimes invent method names. The retry tells the model what the problem was. Each
    retry is another request to the provider, so a higher value can cost more. Set to `0` to
    accept the first reply as is, and check it yourself with the object's `validate_syntax()`
    method. Defaults to `1`.

Returns
-------
str
    The drafted validation plan.

Constructing the `model` Argument
---------------------------------
The `model=` argument should be constructed using the provider and model name separated by a
colon (`provider:model`). The provider text can any of:

- `"anthropic"` (Anthropic)
- `"openai"` (OpenAI)
- `"ollama"` (Ollama)
- `"bedrock"` (Amazon Bedrock)
- `"azure-openai"` (Azure OpenAI)

The model name should be the specific model to be used from the provider. Model names are
subject to change so consult the provider's documentation for the most up-to-date model names.
For `"azure-openai"`, the value after the colon is the Azure *deployment id* (the name you
assigned when deploying the model in your Azure OpenAI resource). It also requires the
environment variables `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` (e.g.,
`https://<resource>.openai.azure.com`), and `OPENAI_API_VERSION` (e.g., `"2024-06-01"`) to
be set.

Notes on Authentication
-----------------------
Providing a valid API key as a string in the `api_key` argument is adequate for getting started
but you should consider using a more secure method for handling API keys.

One way to do this is to load the API key from an environent variable and retrieve it using the
`os` module (specifically the `os.getenv()` function). Places to store the API key might
include `.bashrc`, `.bash_profile`, `.zshrc`, or `.zsh_profile`.

Another solution is to store one or more model provider API keys in an `.env` file (in the root
of your project). If the API keys have correct names (e.g., `ANTHROPIC_API_KEY` or
`OPENAI_API_KEY`) then DraftValidation will automatically load the API key from the `.env` file
and there's no need to provide the `api_key` argument. An `.env` file might look like this:

```plaintext
ANTHROPIC_API_KEY="your_anthropic_api_key_here"
OPENAI_API_KEY="your_openai_api_key_here"
```

There's no need to have the `python-dotenv` package installed when using `.env` files in this
way.

Notes on SSL Certificate Verification
--------------------------------------
By default, SSL certificate verification is enabled for all requests to LLM providers. However,
in certain network environments (such as corporate networks with self-signed certificates or
firewall proxies), you may encounter SSL certificate verification errors.

To disable SSL verification, set the `verify_ssl` parameter to `False`:

```python
import pointblank as pb

data = pb.load_dataset(dataset="nycflights", tbl_type="duckdb")

# Disable SSL verification for networks with self-signed certificates
pb.DraftValidation(
    data=data,
    model="anthropic:claude-opus-4-6",
    verify_ssl=False
)
```

:::{.callout-warning}
Disabling SSL verification (through `verify_ssl=False`) can expose your API keys and data to
man-in-the-middle attacks. Only use this option in trusted network environments and when
absolutely necessary.
:::

Notes on Data Sent to the Model Provider
----------------------------------------
The data sent to the model provider is a JSON summary of the table. This data summary is
generated internally by `DraftValidation` using the `DataScan` class. The summary includes the
following information:

- the number of rows and columns in the table
- the type of dataset (e.g., Polars, DuckDB, Pandas, etc.)
- the column names and their types
- column level statistics such as the number of missing values, min, max, mean, and median, etc.
- a short list of data values in each column

The JSON summary is used to provide the model with the necessary information to draft a
validation plan. As such, even very large tables can be used with the `DraftValidation` class
since the contents of the table are not sent to the model provider.

The Amazon Bedrock is a special case since it is a self-hosted model and security controls are
in place to ensure that data is kept within the user's AWS environment. If using an Ollama
model all data is handled locally, though only a few models are capable enough to perform the
task of drafting a validation plan.

Examples
--------
Let's look at how the `DraftValidation` class can be used to draft a validation plan for a
table. The table to be used is `"nycflights"`, which is available here via the
[`load_dataset()`](`pointblank.load_dataset`) function. The model to be used is
`"anthropic:claude-opus-4-6"` (which performs very well compared to other LLMs). The
example assumes that the API key is stored in an `.env` file as `ANTHROPIC_API_KEY`.

```python
import pointblank as pb

# Load the "nycflights" dataset as a DuckDB table
data = pb.load_dataset(dataset="nycflights", tbl_type="duckdb")

# Draft a validation plan for the "nycflights" table
pb.DraftValidation(data=data, model="anthropic:claude-opus-4-6")
```

The output will be a drafted validation plan for the `"nycflights"` table and this will appear
in the console.

````plaintext
```python
import pointblank as pb

# Define schema based on column names and dtypes
schema = pb.Schema(columns=[
    ("year", "int64"),
    ("month", "int64"),
    ("day", "int64"),
    ("dep_time", "int64"),
    ("sched_dep_time", "int64"),
    ("dep_delay", "int64"),
    ("arr_time", "int64"),
    ("sched_arr_time", "int64"),
    ("arr_delay", "int64"),
    ("carrier", "string"),
    ("flight", "int64"),
    ("tailnum", "string"),
    ("origin", "string"),
    ("dest", "string"),
    ("air_time", "int64"),
    ("distance", "int64"),
    ("hour", "int64"),
    ("minute", "int64")
])

# The validation plan
validation = (
    pb.Validate(
        data=your_data,  # Replace your_data with the actual data variable
        label="Draft Validation",
        thresholds=pb.Thresholds(warning=0.10, error=0.25, critical=0.35)
    )
    .col_schema_match(schema=schema)
    .col_vals_not_null(columns=[
        "year", "month", "day", "sched_dep_time", "carrier", "flight",
        "origin", "dest", "distance", "hour", "minute"
    ])
    .col_vals_between(columns="month", left=1, right=12)
    .col_vals_between(columns="day", left=1, right=31)
    .col_vals_between(columns="sched_dep_time", left=106, right=2359)
    .col_vals_between(columns="dep_delay", left=-43, right=1301, na_pass=True)
    .col_vals_between(columns="air_time", left=20, right=695, na_pass=True)
    .col_vals_between(columns="distance", left=17, right=4983)
    .col_vals_between(columns="hour", left=1, right=23)
    .col_vals_between(columns="minute", left=0, right=59)
    .col_vals_in_set(columns="origin", set=["EWR", "LGA", "JFK"])
    .col_count_match(count=18)
    .row_count_match(count=336776)
    .rows_distinct()
    .interrogate()
)

validation
```
````

The drafted validation plan can be copied and pasted into a Python script or notebook for
further use. In other words, the generated plan can be adjusted as needed to suit the specific
requirements of the table being validated.

Note that the output does not know how the data was obtained, so it uses the placeholder
`your_data` in the `data=` argument of the `Validate` class. When adapted for use, this should
be replaced with the actual data variable.

EditValidation(validation: 'Any', instruction: 'str', model: 'str', data: 'Any' = None, api_key: 'str | None' = None, verify_ssl: 'bool' = True, max_reprompts: 'int' = 1) -> None

Edit an existing validation plan with a plain-English instruction using an LLM.

While [`DraftValidation`](`pointblank.DraftValidation`) generates a validation plan from
scratch by profiling a table, `EditValidation` takes an *existing* plan plus a natural-language
instruction (e.g., "add a freshness check on `updated_at`, tighten the email regex, and drop
the `price > 0` check") and produces a revised plan. You review the change as a diff and
explicitly [`accept()`](`pointblank.EditValidation.accept`) it before anything runs, keeping a
human in control.

`EditValidation` reuses the same provider abstraction as `DraftValidation` (via the `chatlas`
package), supporting the `"anthropic"`, `"openai"`, `"ollama"`, `"bedrock"`, and
`"azure-openai"` providers. Install the optional requirements with
`pip install pointblank[generate]`.

:::{.callout-warning}
The `EditValidation` class is experimental. Please report any issues you encounter in the
[Pointblank issue tracker](https://github.com/posit-dev/pointblank/issues).
:::

Parameters
----------
validation
    The existing validation plan to edit. This can be a [`Validate`](`pointblank.Validate`)
    object, a Python code string (such as the output of
    [`Validate.to_code()`](`pointblank.Validate.to_code`)), a YAML configuration string, or a
    path to a `.py` or `.yaml`/`.yml` file. All inputs are normalized to Pointblank code before
    being sent to the model.
instruction
    A plain-English description of the changes to make to the plan.
model
    The model to be used. This should be in the form of `provider:model` (e.g.,
    `"anthropic:claude-opus-4-8"`). Supported providers are `"anthropic"`, `"openai"`,
    `"ollama"`, `"bedrock"`, and `"azure-openai"`.
data
    Optional data table. When provided, a [`DataScan`](`pointblank.DataScan`) summary of the
    table is included in the prompt so the model can make *informed* edits (e.g., tightening a
    range using observed min/max values). It is also used as the default data source when
    building a [`Validate`](`pointblank.Validate`) via
    [`accept()`](`pointblank.EditValidation.accept`).
api_key
    The API key to be used for the model.
verify_ssl
    Whether to verify SSL certificates when making requests to the LLM provider. Defaults to
    `True`. Use with caution as disabling SSL verification can pose security risks.
max_reprompts
    The maximum number of times to automatically re-prompt the model when the returned code
    fails the syntax/lint check, before surfacing the result to the user. Defaults to `1`.

Returns
-------
EditValidation
    An `EditValidation` object exposing the revised plan through
    [`to_code()`](`pointblank.EditValidation.to_code`),
    [`diff()`](`pointblank.EditValidation.diff`),
    [`changed_steps()`](`pointblank.EditValidation.changed_steps`), and
    [`accept()`](`pointblank.EditValidation.accept`).

Notes on Data Sent to the Model Provider
----------------------------------------
As with `DraftValidation`, only a JSON summary of the table (generated by `DataScan`) is ever
sent to the model, never the underlying rows. If `data=` is omitted, no table information is
sent at all; the model edits the plan using only the plan code and your instruction.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
```
Editing a plan with a model needs the `chatlas` package and an API key, so this first example
isn't run here. We start with a plan for the `small_table` dataset and ask for two changes:

```python
import pointblank as pb

# An existing plan
validation = (
    pb.Validate(data=pb.load_dataset("small_table"))
    .col_vals_gt(columns="d", value=100)
    .col_vals_regex(columns="b", pattern="[0-9]-[a-z]{3}-[0-9]{3}")
)

# Edit it with a plain-English instruction
edited = pb.EditValidation(
    validation=validation,
    instruction="Loosen the `d` check to value=50 and add a not-null check on column `a`.",
    model="anthropic:claude-opus-4-8",
    data=pb.load_dataset("small_table"),
)

print(edited.diff())        # review the change
plan = edited.accept()      # a Validate you can .interrogate()
```

Nothing runs when the `EditValidation` object is created, apart from the call to the model.
The diff shows the revised plan line by line against the original, and only `accept()` turns
the revised code into a `Validate` object, using the `data=` given here because none is passed
to `accept()`. Because `data=` was given, the model also saw a summary of the table, which
helps with edits like choosing a new bound.

The review methods don't depend on the model, and
[`from_plans()`](`pointblank.EditValidation.from_plans`) builds an `EditValidation` from two
plans you already have. Here the revised plan makes the same changes that the instruction
above asked for:

```{python}
small_table = pb.load_dataset("small_table")

original = (
    pb.Validate(data=small_table)
    .col_vals_gt(columns="d", value=100)
    .col_vals_regex(columns="b", pattern="[0-9]-[a-z]{3}-[0-9]{3}")
)
revised = (
    pb.Validate(data=small_table)
    .col_vals_gt(columns="d", value=50)
    .col_vals_regex(columns="b", pattern="[0-9]-[a-z]{3}-[0-9]{3}")
    .col_vals_not_null(columns="a")
)

edit = pb.EditValidation.from_plans(original, revised, instruction="Loosen d; require a")

edit.review()
```

The review table lists one row per changed step: the `col_vals_gt()` step is modified (its
`value=` went from `100` to `50`), the `col_vals_not_null()` step is added, and the unchanged
`col_vals_regex()` step isn't shown. Once the changes look right,
[`accept()`](`pointblank.EditValidation.accept`) gives the revised plan as a `Validate` object
that's ready to interrogate.

See Also
--------
DraftValidation : Draft a new validation plan for a table with an LLM.
Validate.to_code : Get a validation plan as Python code.

MissingSpec(reasons: 'dict[Any, str]', categories: 'dict[str, list[str]] | None' = None, null_is_missing: 'bool' = True, null_reason: 'str' = 'unknown', description: 'str | None' = None) -> None

Specification for structured missing values in a column.

Real-world data rarely encodes missingness as a single `null` value. Survey data distinguishes
*refused* from *don't know* from *not applicable*; clinical data uses codes like `"NOT DONE"`;
statistical packages use sentinel values such as `-99`, `".A"`, or `""`. A `MissingSpec`
captures these sentinel values, the *reason* each one represents, and how they should be
handled during validation and analysis.

This brings the idea of *structured missingness* (a missing value carries a reason for its
absence) into Pointblank's runtime validation layer. Without a spec, a sentinel like `-99` is
just a number: a range check on an `age` column fails it, and a null check passes it, so both
checks give the wrong answer. Declaring the sentinels once and passing the spec to each check
keeps the codes out of the value checks, while still letting you check the codes themselves.

A spec can be used in these places:

- the `missing=` parameter of the `col_vals_*()` methods (such as
  [`Validate.col_vals_between()`](`pointblank.Validate.col_vals_between`)), where values that
  the spec treats as missing pass the check, so only the real values are tested
- [`Validate.col_pct_missing()`](`pointblank.Validate.col_pct_missing`), to limit how much of a
  column is missing, overall or for one reason or category
- [`Validate.col_missing_coded()`](`pointblank.Validate.col_missing_coded`),
  [`Validate.col_missing_only_coded()`](`pointblank.Validate.col_missing_only_coded`), and
  [`Validate.col_missing_consistent()`](`pointblank.Validate.col_missing_consistent`), to check
  how the missing values are coded
- [`missing_vals_tbl()`](`pointblank.missing_vals_tbl`), whose `missing=` mapping of column
  names to specs gives a breakdown of each column's missing values by reason
- YAML configurations, through a top-level `missing_specs:` block or an inline `missing:`
  mapping on a step

The factory methods build common specs: [`from_sas()`](`pointblank.MissingSpec.from_sas`) and
[`from_spss()`](`pointblank.MissingSpec.from_spss`) for statistical packages,
[`from_cdisc_null_flavors()`](`pointblank.MissingSpec.from_cdisc_null_flavors`) for clinical
data, and [`from_variable_metadata()`](`pointblank.MissingSpec.from_variable_metadata`) for
variables read with [`import_metadata()`](`pointblank.import_metadata`).

Parameters
----------
reasons
    A dictionary mapping sentinel values to reason labels. Keys are the actual values present
    in the data (e.g., `-99`, `"NA"`, `".A"`), and they must have the same type as the column:
    use `"-99"` rather than `-99` for a string column. Values are reason identifiers (e.g.,
    `"refused"`, `"not_asked"`). Several sentinel values can share a reason. The dictionary can
    be empty if `null_is_missing=True`.
categories
    Optional grouping of reasons into categories (e.g., an `"item_nonresponse"` category that
    groups `"refused"` and `"dont_know"`). Useful for aggregate reporting and for checking
    missingness rates by category. Each value is a list of reason labels that appear in
    `reasons` (or `null_reason`, when `null_is_missing=True`); any other label raises a
    `ValueError`. Default is `None`.
null_is_missing
    Whether actual null values (`None`, and `NaN` in floating-point columns) should also be
    treated as missing, with the reason given by `null_reason`. Default is `True`. With
    `False`, nulls are left to the method's usual null handling (such as `na_pass=` in the
    `col_vals_*()` methods).
null_reason
    The reason label assigned to actual null values when `null_is_missing=True`. Default is
    `"unknown"`.
description
    Optional human-readable description of the overall missingness pattern. Default is `None`.

Returns
-------
MissingSpec
    A missing-value specification that can be passed to validation methods (via `missing=`)
    and to [`missing_vals_tbl()`](`pointblank.missing_vals_tbl`).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Suppose a survey records the respondent's age, with three codes for an age that wasn't given:
`-99` when the question wasn't asked, `-98` when the respondent refused, and `-97` when they
didn't know. The last two are both *item nonresponse* (the question was asked but not
answered), so we'll group them in a category:

```{python}
import pointblank as pb
import polars as pl

age_missing = pb.MissingSpec(
    reasons={-99: "not_asked", -98: "refused", -97: "dont_know"},
    categories={"item_nonresponse": ["refused", "dont_know"]},
)

print(age_missing.sentinel_values())
print(age_missing.reason_for(-98))
print(age_missing.values_for_category("item_nonresponse"))
```

The spec lists its sentinel values, gives the reason for any of them, and finds the values in a
category. Here's some survey data that uses these codes, along with a null and an age of `150`
that's a genuine error:

```{python}
survey = pl.DataFrame({"age": [34, -98, 51, -99, None, 27, 150, -97, 45, 62]})

pb.preview(survey)
```

We'll check the ages with and without the spec, and then limit the item nonresponse to 10% of
the rows:

```{python}
validation = (
    pb.Validate(data=survey)
    .col_vals_between(columns="age", left=0, right=120)
    .col_vals_between(columns="age", left=0, right=120, missing=age_missing)
    .col_pct_missing(
        columns="age", missing=age_missing, category="item_nonresponse", max_pct=0.1
    )
    .interrogate()
)

validation
```

Step 1 has 5 failing rows: the three codes and the null fail the range check along with the
`150`, so the real error is lost among the codes. With `missing=age_missing`, step 2 lets the
codes and the null pass and only the `150` fails. Step 3 looks at the codes instead of the
ages: 2 of the 10 rows (`-98` and `-97`) are item nonresponse, which is 20% and over the 10%
limit, so it fails. Together the steps separate bad values from values that are missing for a
known reason, and check each on its own terms.

See Also
--------
Validate.col_pct_missing : Limit the share of a column that is missing.
Validate.col_missing_coded : Check that every missing value has a declared code.
missing_vals_tbl : Show each column's missing values by reason.
import_metadata : Read variable metadata, including missing-value codes, from a file.


## Contracts and Pipelines

Use `Contract` and `Step` to define declarative data quality contracts that specify what valid data looks like. Use `Steps` to build reusable step libraries with a fluent API and compose them into `Validate` plans via `add_steps()`. Use `Pipeline` to enforce contracts at both boundaries of a data transformation (source and target), producing a `PipelineResult` with full introspection into what passed and what failed.



Contract(name: 'str', direction: "Literal['source', 'target']" = 'source', schema: 'Schema | None' = None, steps: 'list[Step]' = <factory>, version: 'str | None' = None, owner: 'str | None' = None, consumers: 'str | list[str] | None' = None, description: 'str | None' = None, thresholds: 'Thresholds | None' = None, on_violation: "Literal['warn', 'raise', 'log']" = 'warn') -> None

A declarative boundary contract for pipeline data.

A `Contract` is a written-down agreement about what a table must look like at a specific point
in a data pipeline. It bundles three things into one object: a
[`Schema`](`pointblank.Schema`) with the expected column names and types (the structural
expectations), a list of [`Step`](`pointblank.Step`) objects with the rules the values must
follow (the semantic expectations), and metadata saying who owns the data, who depends on it,
and which version of the agreement is in force.

With a plain [`Validate`](`pointblank.Validate`) chain, the rules exist only as code that runs
against one table. A contract holds the same rules as data. It isn't tied to any table, so you
can apply it to each day's delivery of a feed, check it into version control, review changes to
it like any other change, and share it as a YAML file. Teams on both sides of a handoff can
then refer to the same definition of "valid". The producer of a table can check its output
against the contract before publishing, and the consumer can check its input against the same
contract on arrival.

Use a contract when the expectations matter more than any one validation run: data crossing a
team or system boundary, inputs and outputs of a scheduled job, or a dataset with named owners
and downstream consumers. For a one-off look at a table, a `Validate` chain is simpler.

Validating data against a contract gives you an ordinary interrogated `Validate` object. That
means everything you already use with validations works here: the tabular report, step reports,
[`all_passed()`](`pointblank.Validate.all_passed`),
[`assert_passing()`](`pointblank.Validate.assert_passing`), data extracts, and so on.

Parameters
----------
name
    A short, unique name for the contract (e.g., `"raw_clickstream_feed"`). It becomes the
    table name in the validation report and is part of the report label. It must be a non-empty
    string.
direction
    Where the contract sits relative to a transformation: `"source"` (the default) for data
    flowing in, or `"target"` for the data product flowing out. The direction is descriptive
    metadata that appears in violation messages from
    [`Pipeline`](`pointblank.Pipeline`). It doesn't change what gets checked.
schema
    An optional [`Schema`](`pointblank.Schema`) giving the expected column names and data types.
    When a schema is given, it's checked first, with
    [`col_schema_match()`](`pointblank.Validate.col_schema_match`) and its default settings.
    These defaults are strict: the table must have exactly the schema's columns, in the same
    order, with matching types.
steps
    A list of [`Step`](`pointblank.Step`) objects, each naming a validation method and its
    arguments (e.g., `pb.Step("col_vals_gt", columns="amount", value=0)`). They run in list
    order after the schema check. Only a subset of validation methods can be used. See the
    *Supported Step Methods* section below.
version
    A version string for the contract, such as `"1.2.0"`. Semantic versioning is a good
    convention, but any string is accepted. The version is added to the report label (e.g.,
    `"Contract: orders v1.2.0"`) and recorded in the report's governance metadata.
owner
    The person or team responsible for the contract and the data it describes. It's recorded in
    the report's governance metadata.
consumers
    Who depends on data meeting this contract, as a string or a list of strings. It's recorded
    in the report's governance metadata.
description
    An optional longer description of the contract's purpose. It's kept with the contract and
    serialized with it, but doesn't appear in validation reports.
thresholds
    Default [`Thresholds`](`pointblank.Thresholds`) for the validation built from this contract.
    These set when a failing step is considered a 'warning', 'error', or 'critical' issue. If
    the contract is used in a `Pipeline` that has its own thresholds, the pipeline's thresholds
    take precedence.
on_violation
    What a [`Pipeline`](`pointblank.Pipeline`) should do when data fails this contract:
    `"warn"` (the default) issues a `UserWarning`, `"raise"` raises a
    [`ContractViolationError`](`pointblank.ContractViolationError`) (a `RuntimeError`), and
    `"log"` writes a warning to the `"pointblank.contract"` logger. This setting applies only
    when the contract is enforced through a `Pipeline`. Calling
    [`validate()`](`pointblank.Contract.validate`) directly always returns the validation
    without warning or raising, so check the result yourself (e.g., with
    [`assert_passing()`](`pointblank.Validate.assert_passing`)).

Returns
-------
Contract
    A `Contract` object. Apply it to data with
    [`validate()`](`pointblank.Contract.validate`), or with
    [`to_validate()`](`pointblank.Contract.to_validate`) when you want to add more steps before
    interrogating.

How a Contract Becomes a Validation
-----------------------------------
A contract holds no data and runs nothing until it's applied to a table. Calling
[`to_validate()`](`pointblank.Contract.to_validate`) (or
[`validate()`](`pointblank.Contract.validate`), which also interrogates) compiles the
contract into a `Validate` object:

1. a `Validate` is created for the table, with `tbl_name=` set to the contract name, a label of
the form `"Contract: <name> v<version>"`, and the contract's `thresholds=`, `owner=`,
`consumers=`, and `version=`
2. if the contract has a schema, a `col_schema_match()` step is added first
3. each [`Step`](`pointblank.Step`) is added in order, by calling the validation method it
names with its stored arguments

A step that lists several columns expands into one validation step per column, just as it would
in a `Validate` chain. Mistakes in a step's arguments (a misspelled parameter name, for example)
surface when the contract is compiled, not when it's created.

Supported Step Methods
----------------------
These validation methods can be used in a contract's steps:

- column value checks: `col_vals_gt`, `col_vals_lt`, `col_vals_ge`, `col_vals_le`,
`col_vals_eq`, `col_vals_ne`, `col_vals_between`, `col_vals_outside`, `col_vals_in_set`,
`col_vals_not_in_set`, `col_vals_increasing`, `col_vals_decreasing`, `col_vals_null`,
`col_vals_not_null`, `col_vals_regex`, `col_vals_within_spec`, and `col_vals_expr`
- column-level checks: `col_exists` and `col_pct_null`
- table-level checks: `col_schema_match`, `col_count_match`, `row_count_match`,
`rows_distinct`, and `rows_complete`
- multi-condition checks: `conjointly`, with its expressions given as a list in `exprs=`

Each takes the same arguments as the [`Validate`](`pointblank.Validate`) method of the same
name. A step that names any other method is rejected with a `ValueError` when the contract is
created. For steps that a contract can't express, compile the contract with
[`to_validate()`](`pointblank.Contract.to_validate`) and chain the extra methods onto the
result.

Serialization
-------------
Contracts are meant to live outside your code, so they round-trip through YAML and plain
dictionaries. [`to_yaml()`](`pointblank.Contract.to_yaml`) and
[`from_yaml()`](`pointblank.Contract.from_yaml`) work with YAML files, and
[`to_dict()`](`pointblank.Contract.to_dict`) and
[`from_dict()`](`pointblank.Contract.from_dict`) work with dictionaries (useful for JSON or a
configuration system). The schema is stored as a mapping of column names to type strings, the
thresholds as a mapping of levels to values, and each step as a single-key mapping from method
name to arguments. Step arguments must be plain data (strings, numbers, booleans, dates,
lists, and dictionaries) for a contract to be written to YAML, although step-level
`Thresholds` objects and tuples are converted to dictionaries automatically. A contract with
other objects in its steps, such as a column selector (e.g., `pb.starts_with("amt_")`), a
Polars expression, or a function, can't be written: `to_yaml()` raises a `ValueError` that
says where the object is. Keep contracts like that in Python.

Working With Other Pointblank Tools
-----------------------------------
A contract is the building block for several other parts of the API:

- [`Pipeline`](`pointblank.Pipeline`) pairs a source contract and a target contract around a
transformation function. It validates the input, runs the transformation, validates the output,
and applies each contract's `on_violation=` setting. The results come back in a
[`PipelineResult`](`pointblank.PipelineResult`).
- [`export_contract()`](`pointblank.export_contract`) converts a contract into an external
format (JSON Schema, Frictionless Table Schema, dbt, or the Open Data Contract Standard), and
[`import_contract()`](`pointblank.import_contract`) reads one of those formats. Call
[`to_contract()`](`pointblank.ContractImport.to_contract`) on the result to get a `Contract`.
Each format supports a different set of rules, so checks with no equivalent in the target
format are left out of the export.
- Because [`to_validate()`](`pointblank.Contract.to_validate`) returns an ordinary `Validate`
object, a contract can be combined with reusable [`Steps`](`pointblank.Steps`) libraries
through [`Validate.add_steps()`](`pointblank.Validate.add_steps`).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's write a contract for a daily orders extract. The schema fixes the four columns and their
types, and the steps say that IDs must be present, each order must appear once, amounts must
be positive, and the status must be one of three known values. Ownership and versioning go in
the metadata, and the thresholds flag any failing test unit as a warning and a failure rate of
10% or more as an error.

```{python}
import pointblank as pb
import polars as pl

orders_contract = pb.Contract(
    name="orders",
    version="1.2.0",
    owner="sales-data-team",
    consumers=["finance", "analytics"],
    description="Daily order extract delivered by the sales platform.",
    schema=pb.Schema(
        order_id="String",
        customer_id="Int64",
        amount="Float64",
        status="String",
    ),
    steps=[
        pb.Step("col_vals_not_null", columns=["order_id", "customer_id"]),
        pb.Step("rows_distinct", columns_subset=["order_id"]),
        pb.Step("col_vals_gt", columns="amount", value=0),
        pb.Step("col_vals_in_set", columns="status", set=["pending", "shipped", "delivered"]),
    ],
    thresholds=pb.Thresholds(warning=1, error=0.10),
)

orders_contract
```

The contract itself is just a definition. Printing it shows a short summary: the name,
direction, version, whether a schema is set, and the number of steps.

Now let's apply the contract to a day's extract that meets every expectation. The
[`validate()`](`pointblank.Contract.validate`) method compiles the contract against the table
and interrogates it, returning a `Validate` object that displays as a validation report.

```{python}
orders_day_1 = pl.DataFrame(
    {
        "order_id": ["A1", "A2", "A3", "A4"],
        "customer_id": [101, 102, 101, 103],
        "amount": [25.00, 110.50, 42.00, 8.99],
        "status": ["pending", "shipped", "delivered", "shipped"],
    }
)

orders_contract.validate(orders_day_1)
```

The report header shows the label built from the contract (`Contract: orders v1.2.0`) along
with the owner, consumers, and version. The first step is the schema check, followed by one step
per contract rule. The `col_vals_not_null` step names two columns, so it becomes two validation
steps. All six steps pass.

The next day's extract has problems: a missing customer ID, a repeated order ID, a negative
amount, and a status the contract doesn't allow. The same contract catches all of them.

```{python}
orders_day_2 = pl.DataFrame(
    {
        "order_id": ["A5", "A6", "A6", "A8"],
        "customer_id": [104, None, 101, 105],
        "amount": [25.00, -5.00, 42.00, 8.99],
        "status": ["pending", "shipped", "returned", "shipped"],
    }
)

validation = orders_contract.validate(orders_day_2)

validation
```

The schema still matches, but four of the rule steps have failing test units. With only four
rows, one failure is already 25% of the table, so those steps exceed both the warning and the
error thresholds. The `rows_distinct` step counts two failures because both rows with order
`"A6"` are duplicates. Since the result is an ordinary `Validate` object, you can check it in
code, for example to stop a scheduled job:

```{python}
validation.all_passed()
```

To share the contract with another team or keep it under version control, write it out as YAML.
Called without a path, [`to_yaml()`](`pointblank.Contract.to_yaml`) returns the YAML text.

```{python}
print(orders_contract.to_yaml())
```

The YAML lists the metadata, schema, steps, and thresholds in a form that's easy to read and
review. [`from_yaml()`](`pointblank.Contract.from_yaml`) reads it back into an equivalent
`Contract`.

Finally, a contract can be the starting point for a stricter, local validation. Here,
[`to_validate()`](`pointblank.Contract.to_validate`) compiles the contract without
interrogating, and we add one more check (orders of 100 or less) before calling
[`interrogate()`](`pointblank.Validate.interrogate`).

```{python}
(
    orders_contract.to_validate(orders_day_1)
    .col_vals_le(columns="amount", value=100)
    .interrogate()
)
```

The six contract steps pass as before, and the added seventh step flags the order of 110.50.
The contract itself is unchanged, so other users of it aren't affected by the extra rule.

See Also
--------
Step : A single declarative validation step, used in a contract's `steps=`.
Schema : The column names and types used in a contract's `schema=`.
Pipeline : Enforce source and target contracts around a transformation.
pointblank.export_contract : Convert a contract to JSON Schema, Frictionless, dbt, or ODCS.
pointblank.import_contract : Read an external format, then call `to_contract()` on it.

Step(method: 'str', **kwargs: 'Any') -> 'None'

A single validation step, stored as data.

A `Step` records one validation rule: the name of a [`Validate`](`pointblank.Validate`)
validation method and the keyword arguments to call it with. `pb.Step("col_vals_gt",
columns="amount", value=0)` describes the same check as
`Validate.col_vals_gt(columns="amount", value=0)`, but nothing runs when the step is created.
The step is a small, inert value that's applied later, when it's compiled into a validation
for a specific table.

Holding a rule as data rather than as a method call makes it possible to keep rules apart from
the code that runs them. Steps can be collected into a [`Contract`](`pointblank.Contract`),
written to YAML or JSON and read back, compared with each other, or built from configuration
files and external schema formats. The steps in a contract imported with
[`import_contract()`](`pointblank.import_contract`), for example, are `Step` objects
generated from the source format.

The step's parts are available as the `method` attribute (the method name) and the `kwargs`
attribute (a dictionary of arguments). Two steps are equal when both their method names and
arguments are equal.

Parameters
----------
method
    The name of the validation method to call, such as `"col_vals_gt"` or `"rows_distinct"`.
    A `Step` doesn't check the name itself. A [`Contract`](`pointblank.Contract`) accepts only
    its supported methods (listed in its *Supported Step Methods* section) and raises a
    `ValueError` for any other name. Steps used through [`Steps`](`pointblank.Steps`) and
    [`Validate.add_steps()`](`pointblank.Validate.add_steps`) can name any validation method.
**kwargs
    The arguments for the validation method, given by keyword (e.g., `columns="amount"`,
    `value=0`). They're stored as given and passed to the method when the step is compiled, so
    a misspelled or missing argument raises a `TypeError` at that point rather than when the
    step is created. Positional arguments aren't supported. The one validation method that
    takes positional arguments, `conjointly()`, receives its expressions as a list in `exprs=`
    (e.g., `pb.Step("conjointly", exprs=[lambda df: pl.col("a") > 0])`).

Returns
-------
Step
    A `Step` object.

Writing Steps That Can Be Saved
-------------------------------
A contract's steps are written to YAML as a mapping from the method name to its arguments, so
a step can be saved only if its arguments are plain data: strings, numbers, booleans, dates,
lists, and dictionaries. Step-level thresholds are the exception, since a
[`Thresholds`](`pointblank.Thresholds`) object or a tuple is converted to a dictionary of
levels automatically. Other objects can't be saved, and
[`Contract.to_yaml()`](`pointblank.Contract.to_yaml`) raises a `ValueError` that names the
step and argument holding one. Keep these in mind for steps you plan to save:

- name columns with strings or lists of strings, rather than with selectors like
`pb.starts_with()` or `pb.col()`
- steps that take an expression or a function, such as `col_vals_expr` and `conjointly`, can
only be used in contracts kept in Python

All of these work when the step is used directly in Python.

Steps and the Steps Class
-------------------------
[`Steps`](`pointblank.Steps`) is a fluent builder for lists of `Step` objects. It has a
method for each validation method (`pb.Steps().col_vals_gt(columns="amount", value=0)`), so
you get the real method signatures, with argument names and defaults, while writing the
steps. Use `Step` when the method name and arguments are data (when they come from a
configuration file, for example) or when building a contract's `steps=` list. A list of
`Step` objects can also be turned into a `Steps` object with `pb.Steps(steps=...)` and added
to any validation with [`Validate.add_steps()`](`pointblank.Validate.add_steps`).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's create a step that checks that ages fall between 0 and 120. Printing it shows the method
name and the stored arguments.

```{python}
import pointblank as pb

age_step = pb.Step("col_vals_between", columns="age", left=0, right=120)

age_step
```

The two parts of the step are available as attributes:

```{python}
print(age_step.method)
print(age_step.kwargs)
```

Steps are normally used in groups. Here are three more rules for a table of patients: each
patient ID must appear only once, IDs must follow a pattern, and the `visit` column must be
present. Note that `rows_distinct` takes `columns_subset=` rather than `columns=`, because
each step's arguments are exactly those of its `Validate` method.

```{python}
patient_steps = [
    pb.Step("rows_distinct", columns_subset="patient_id"),
    pb.Step("col_vals_regex", columns="patient_id", pattern=r"^PT-\d{3}$"),
    pb.Step("col_exists", columns="visit"),
    age_step,
]
```

Placing the steps in a [`Contract`](`pointblank.Contract`) lets us apply them to a table. The
steps run in the order they're listed.

```{python}
import polars as pl

patients = pl.DataFrame(
    {
        "patient_id": ["PT-001", "PT-002", "PT-003", "PT-03"],
        "age": [34, 51, 130, 28],
        "visit": [1, 1, 2, 1],
    }
)

patients_contract = pb.Contract(name="patients", steps=patient_steps)

patients_contract.validate(patients)
```

The ID check flags `"PT-03"`, which is missing a digit, and the age check flags the value
`130`. The other two steps pass.

The same steps can be added to an ordinary validation by wrapping them in a
[`Steps`](`pointblank.Steps`) object. That route also accepts validation methods that
contracts don't support, such as
[`col_vals_str_len()`](`pointblank.Validate.col_vals_str_len`):

```{python}
id_length_step = pb.Step("col_vals_str_len", columns="patient_id", min_val=6, max_val=6)

(
    pb.Validate(data=patients, tbl_name="patients")
    .add_steps(pb.Steps(steps=patient_steps + [id_length_step]))
    .interrogate()
)
```

The first four steps give the same results as in the contract. The new length check, in
step 5, also catches `"PT-03"`, which has five characters instead of six. Defining the steps
once as a list lets a contract and an ordinary validation share them.

See Also
--------
Contract : Collect steps, a schema, and metadata into a reusable contract.
Steps : Build lists of steps with the signatures of the validation methods.
Validate.add_steps : Add steps from a `Steps` object to a validation.

Steps(steps: 'list[Step] | None' = None) -> 'None'

A reusable collection of validation step definitions.

A `Steps` object is a list of validation steps that isn't attached to any table. You build it
with the same methods you'd call on a [`Validate`](`pointblank.Validate`) object
(`col_vals_not_null()`, `rows_distinct()`, `col_vals_regex()`, and so on), but each call only
records the step. Nothing is checked until the collection is added to a validation with
[`Validate.add_steps()`](`pointblank.Validate.add_steps`) and that validation is
interrogated.

This makes it possible to write a group of checks once and use it in many places. Many tables
share the same rules: every table with a customer ID needs that ID to be present and unique,
and every table with an email column needs valid addresses. Without `Steps`, those rules get
copied into each validation and drift apart over time. With `Steps`, you define them in one
place (a shared module, for example), name them for what they check (`id_checks`,
`contact_checks`), and add them to whichever validations need them. Each validation keeps its
own table, thresholds, actions, and metadata, and can mix shared steps with its own.

Since the methods have the same signatures as the `Validate` methods they mirror, you get the
same argument names, defaults, and editor completion, and a misspelled argument raises a
`TypeError` as soon as the step is recorded.

Parameters
----------
steps
    An optional list of [`Step`](`pointblank.Step`) objects to start the collection with.
    This is useful when step definitions come from data (a configuration file, for example)
    rather than from method calls.

Returns
-------
Steps
    A `Steps` object. Add steps to it with its validation methods, and apply it with
    [`Validate.add_steps()`](`pointblank.Validate.add_steps`).

Recording Steps
---------------
Each validation method records one step and returns the same `Steps` object, so calls can be
chained. All of the row-based and table-level validation methods are available, from
`col_vals_gt()` to `tbl_reconcile()`. The aggregation methods (`col_sum_*()`, `col_avg_*()`,
and `col_sd_*()`), `specially()`, and `prompt()` aren't, so add those directly to the
validation.

Because each method adds to the object it's called on, a collection isn't copied when you add
to it. Calling a method on a shared collection (e.g., `extended = id_checks.col_exists(...)`)
adds the step to `id_checks` itself, and `extended` is the same object. To build variations,
keep the shared parts in separate collections and add several of them to a validation.

Printing a `Steps` object lists its steps with the arguments that differ from the defaults,
and `len()` gives the number of steps.

Applying Steps to a Validation
------------------------------
[`Validate.add_steps()`](`pointblank.Validate.add_steps`) appends the recorded steps to a
validation, in order, as if each method had been called on it directly. It takes any number of
`Steps` objects (and also `Validate` objects, whose steps it copies), and has options for
adapting the steps to the validation at hand:

- `columns_map=` renames columns in the steps, so that the same collection works on tables that
name a column differently (e.g., `{"customer_id": "cust_id"}`)
- `thresholds=` and `active=` replace the thresholds and active state of every added step
- `exclude=` skips steps by method name or by their 1-based position in a collection

Steps, Step, and Contract
-------------------------
A [`Step`](`pointblank.Step`) is a single step stored as data: a method name and a dictionary
of arguments. A `Steps` object holds a list of them, built with method calls instead. A
[`Contract`](`pointblank.Contract`) also holds a list of `Step` objects, together with a schema
and metadata (an owner, a version, consumers), and can be saved as YAML. Use `Steps` for
reusable building blocks inside Python code, and a `Contract` for an agreement about a table
that's shared, versioned, and enforced at a pipeline boundary. The two work together: a
contract compiled with [`to_validate()`](`pointblank.Contract.to_validate`) is an ordinary
validation, so collections can be added to it with `add_steps()`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's build two collections of checks for customer data. The first covers identifiers (each
customer ID must be present and appear only once), and the second covers contact details
(email addresses must look valid, and the country must be one of three codes).

```{python}
import pointblank as pb
import polars as pl

id_checks = (
    pb.Steps()
    .col_vals_not_null(columns="customer_id")
    .rows_distinct(columns_subset="customer_id")
)

contact_checks = (
    pb.Steps()
    .col_vals_regex(columns="email", pattern=r"^[^@]+@[^@]+\.[a-z]+$")
    .col_vals_in_set(columns="country", set=["CA", "MX", "US"])
)

id_checks
```

Displaying a collection lists its steps and their arguments. No table is involved yet.

Now let's apply both collections to a table of customers from a CRM system. The validation is
created as usual, and `add_steps()` appends the four recorded steps before interrogation.

```{python}
crm_customers = pl.DataFrame(
    {
        "customer_id": [1, 2, 3, 3, 5],
        "email": ["a@x.com", "b@y.org", "c-at-z.com", "d@w.net", "e@v.io"],
        "country": ["US", "CA", "US", "MX", "BR"],
    }
)

(
    pb.Validate(data=crm_customers, tbl_name="crm_customers")
    .add_steps(id_checks, contact_checks)
    .interrogate()
)
```

The report has one step per recorded step, in the order they were added. Customer ID `3`
appears twice, so both of its rows fail the `rows_distinct` step. The address `"c-at-z.com"`
fails the email pattern, and the country `"BR"` isn't in the allowed set.

The same collections can check a table from a billing system, where the ID column is called
`cust_id`. The `columns_map=` argument renames the column in the added steps, and
`thresholds=` gives every added step a warning threshold of one failing test unit.

```{python}
billing_accounts = pl.DataFrame(
    {
        "cust_id": [10, 11, None, 13],
        "email": ["f@x.com", "g@y.com", "h@z.com", "i@w"],
        "country": ["US", "US", "CA", "US"],
    }
)

(
    pb.Validate(data=billing_accounts, tbl_name="billing_accounts")
    .add_steps(
        id_checks,
        contact_checks,
        columns_map={"customer_id": "cust_id"},
        thresholds=pb.Thresholds(warning=1),
    )
    .interrogate()
)
```

The ID steps now check `cust_id`, and one account is missing its ID. The address `"i@w"`
has no domain suffix, so it fails the email pattern. Both failing steps reach the warning
threshold.

See Also
--------
Validate.add_steps : Add the steps in one or more collections to a validation.
Step : A single validation step stored as data.
Contract : A schema, steps, and metadata that make up an agreement about a table.

add_steps(self, *steps: 'Steps | Validate', active: 'bool | Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, exclude: 'list[str | int] | None' = None, columns_map: 'dict[str, str] | None' = None) -> 'Validate'

Add the steps from one or more `Steps` collections to the validation.

This method appends previously recorded validation steps to the validation plan. The main
source of steps is a [`Steps`](`pointblank.Steps`) collection: a reusable group of steps
built with the same methods as `Validate` but not attached to any table. Each recorded step
is applied to this validation as if its method had been called directly, in the order the
steps were recorded. Several collections can be added in one call, and the result can be
chained with other validation methods, so shared steps and steps specific to this table
can be combined in one plan. A `Validate` object can also be given, in which case its
steps are copied into this validation as they were defined, whether or not it has been
interrogated (a step that interrogation split into one step per column or segment is copied
once).

Steps written for one table don't always fit another one exactly. The optional arguments
adapt the added steps without changing the collections themselves: `columns_map=` renames
columns, `thresholds=` and `active=` replace those settings on every added step, and
`exclude=` leaves out steps that don't apply.

Parameters
----------
*steps
    One or more [`Steps`](`pointblank.Steps`) or `Validate` objects whose steps should be
    added, in the order given.
active
    Replace the `active=` setting of every added step. If `None` (the default), each step
    keeps its own setting. Use `False` to add the steps in an inactive state, or a callable
    (such as [`has_columns()`](`pointblank.has_columns`)) to decide at interrogation time.
thresholds
    Replace the `thresholds=` setting of every added step. If `None` (the default), each
    step keeps its own thresholds and otherwise uses the thresholds of this validation.
    Accepts the same values as the `thresholds=` argument of `Validate`.
exclude
    Steps to leave out, as a list of method names (e.g., `["col_vals_regex"]` skips every
    regex step) or of 1-based positions (e.g., `[2]` skips the second step). Positions
    count within each collection, so `[2]` skips the second step of every collection given.
columns_map
    A dictionary that renames columns in the added steps, mapping the column names used in
    the collection to the names in this table (e.g., `{"customer_id": "cust_id"}`). It
    applies to column names given as strings or lists of strings in a step's `columns=`,
    `column=`, or `columns_subset=` argument. Column selectors such as
    [`starts_with()`](`pointblank.starts_with`) are left as they are.

Returns
-------
Validate
    The `Validate` object with the steps added, for further method chaining.

Raises
------
ValueError
    If no `Steps` or `Validate` objects are given.
TypeError
    If an argument in `*steps` isn't a `Steps` or `Validate` object, if an item in
    `exclude=` isn't a string or an integer, or if a step's arguments don't fit its
    validation method.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's define two collections of steps for order data: one for completeness and one for
valid amounts.

```{python}
import pointblank as pb
import polars as pl

completeness = (
    pb.Steps()
    .col_vals_not_null(columns="order_id")
    .col_vals_not_null(columns="email")
)

valid_amounts = (
    pb.Steps()
    .col_vals_gt(columns="amount", value=0)
    .col_vals_le(columns="amount", value=10_000)
)
```

Both collections can be added to a validation of an orders table, along with a check
that only applies to this table (the `region` values):

```{python}
orders = pl.DataFrame(
    {
        "order_id": ["A1", "A2", None, "A4"],
        "email": ["a@x.com", None, "c@z.com", "d@w.net"],
        "amount": [25.0, 120.0, 0.0, 18_500.0],
        "region": ["east", "west", "east", "north"],
    }
)

(
    pb.Validate(data=orders, tbl_name="orders")
    .add_steps(completeness, valid_amounts)
    .col_vals_in_set(columns="region", set=["east", "west"])
    .interrogate()
)
```

The first four steps come from the two collections, in order, and the fifth is the
table-specific check. Each step finds one failing row.

A table of refunds uses the same rules, but with different column names, no email
column, and a stricter tolerance for failures. Here, `columns_map=` renames the columns,
`exclude=` skips the second completeness step (the email check), and `thresholds=` makes
any failure an error.

```{python}
refunds = pl.DataFrame(
    {"refund_ref": ["R1", "R2", "R3"], "refund_amt": [15.0, -4.0, 60.0]}
)

(
    pb.Validate(data=refunds, tbl_name="refunds")
    .add_steps(
        completeness,
        valid_amounts,
        columns_map={"order_id": "refund_ref", "amount": "refund_amt"},
        exclude=[2],
        thresholds=pb.Thresholds(error=1),
    )
    .interrogate()
)
```

Because positions count within each collection, `exclude=[2]` also skips the second step
of `valid_amounts` (the upper limit on the amount). The negative refund fails the
remaining amount check and reaches the error threshold.

Pipeline(source: 'Contract | None' = None, target: 'Contract | None' = None, thresholds: 'Thresholds | None' = None, actions: 'Actions | None' = None, final_actions: 'FinalActions | None' = None, label: 'str | None' = None, short_circuit: 'bool' = True) -> None

Enforce data contracts at both ends of a data transformation.

Most data problems are introduced at a boundary: an upstream system starts sending nulls or a
new category, or a transformation step drops, duplicates, or miscalculates rows. A `Pipeline`
guards those boundaries. It pairs a *source* [`Contract`](`pointblank.Contract`), describing
the data a transformation expects to receive, with a *target* contract, describing the data it
promises to produce. Data is validated against the source contract on the way in and against
the target contract on the way out, so a failure tells you whether the input was bad or the
transformation went wrong.

The pipeline doesn't run or schedule your transformation logic beyond calling the function you
give it. The transformation can use any DataFrame library, and the pipeline can sit inside any
orchestration tool. You can also check just one boundary with
[`validate_source()`](`pointblank.Pipeline.validate_source`) or
[`validate_target()`](`pointblank.Pipeline.validate_target`), for example when the
transformation runs somewhere else. Only one of the two contracts is required, so a pipeline
can also act as an ingestion guard (source only) or an output guard (target only).

Parameters
----------
source
    The [`Contract`](`pointblank.Contract`) that incoming data must meet. Leave it as `None`
    for a pipeline that only checks output.
target
    The [`Contract`](`pointblank.Contract`) that the transformed data must meet. Leave it as
    `None` for a pipeline that only checks input. At least one of `source=` and `target=` must
    be given.
thresholds
    [`Thresholds`](`pointblank.Thresholds`) to use for both boundary validations. When given,
    they replace each contract's own `thresholds=` (they aren't merged), which lets the same
    contracts be applied more or less strictly in different settings. Thresholds set on
    individual steps still take precedence. Thresholds determine the warning, error, and
    critical levels shown in the reports and the [`Actions`](`pointblank.Actions`) they
    trigger, but not whether a boundary passes (see *Passing and Failing* below).
actions
    [`Actions`](`pointblank.Actions`) to take when a step in either boundary validation reaches
    a threshold level. The same actions are used for both boundaries.
final_actions
    [`FinalActions`](`pointblank.FinalActions`) to run when a boundary validation finishes.
    They run once for each boundary that's validated, so twice in a full run.
label
    A label for the pipeline. Each boundary validation is labeled `"<label> [source]"` or
    `"<label> [target]"`. Without a label, the contract name is used instead, as in
    `"Pipeline: raw_orders [source]"`.
short_circuit
    Whether [`run()`](`pointblank.Pipeline.run`) should stop when the source validation fails.
    With `True` (the default), the transformation isn't called and the target isn't validated,
    so bad input never reaches your transformation code or anything it writes to. Use `False`
    to always run the transformation and validate both boundaries, for example to see every
    problem at once while developing. A source contract with `on_violation="raise"` still
    raises with `False`, but only after the target has been validated.

Returns
-------
Pipeline
    A `Pipeline` object. Use [`run()`](`pointblank.Pipeline.run`) to validate, transform, and
    validate again in one call, getting back a [`PipelineResult`](`pointblank.PipelineResult`).

Raises
------
TypeError
    If `source=` or `target=` is given and isn't a `Contract`.
ValueError
    If neither `source=` nor `target=` is given.

How a Pipeline Runs
-------------------
Calling [`run()`](`pointblank.Pipeline.run`) with a table and a transformation function goes
through these steps:

1. the source contract is compiled into a [`Validate`](`pointblank.Validate`) object for the
table and interrogated
2. if the source validation has any failing test units and `short_circuit=True`, the run stops
here
3. the transformation function is called with the original table, and its output is kept
4. the target contract is compiled for the output and interrogated

Each boundary validation is built the same way as in
[`Contract.to_validate()`](`pointblank.Contract.to_validate`): the table name is the contract
name, the contract's owner, consumers, and version are recorded, the schema check (if any)
comes first, and the steps follow in order. The pipeline adds its own label, thresholds, and
actions. If one of the contracts is missing, its boundary is simply skipped.

Passing and Failing
-------------------
A boundary *passes* only when every test unit in every step passes, which is the same test as
[`Validate.all_passed()`](`pointblank.Validate.all_passed`). Thresholds don't change this: a
single failing row makes the boundary fail, even if it's well below the warning level. For
decisions based on threshold levels instead, inspect the boundary validations directly, for
example with [`above_threshold()`](`pointblank.Validate.above_threshold`).

When a boundary fails, the pipeline applies that contract's `on_violation=` setting. With
`"warn"` (the default), a `UserWarning` is issued. With `"log"`, a warning-level message is
written to the `"pointblank.contract"` logger. With `"raise"`, a
[`ContractViolationError`](`pointblank.ContractViolationError`) (a `RuntimeError`) is raised,
which stops the code that called the pipeline. When it's raised by
[`run()`](`pointblank.Pipeline.run`), its `result` attribute holds the
[`PipelineResult`](`pointblank.PipelineResult`) of the run, so both validations can still be
inspected.

Serialization
-------------
A pipeline can be stored as YAML with [`to_yaml()`](`pointblank.Pipeline.to_yaml`) and read
back with [`from_yaml()`](`pointblank.Pipeline.from_yaml`), or converted to and from a
dictionary with [`to_dict()`](`pointblank.Pipeline.to_dict`) and
[`from_dict()`](`pointblank.Pipeline.from_dict`). The stored form holds the label, thresholds,
and `short_circuit=` setting under a `pipeline` key, plus the two contracts under `source` and
`target`. Actions and final actions are functions or messages tied to your environment, so
they aren't stored. Pass them again when you recreate the pipeline. The transformation isn't
part of the pipeline either: it's given to `run()` each time.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
import warnings
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
warnings.formatwarning = lambda message, category, *args, **kwargs: (
    f"{category.__name__}: {message}\n"
)
```

Let's build a pipeline around a transformation that prepares raw order records for reporting.
The raw records store amounts in cents with lowercase currency codes. The source contract says
what the transformation needs from them: an order ID on every record, no negative amounts, and
a known currency.

```{python}
import pointblank as pb
import polars as pl

raw_orders_contract = pb.Contract(
    name="raw_orders",
    steps=[
        pb.Step("col_vals_not_null", columns="order_id"),
        pb.Step("col_vals_ge", columns="amount_cents", value=0),
        pb.Step("col_vals_in_set", columns="currency", set=["usd", "eur", "gbp"]),
    ],
)
```

The target contract describes what the transformation promises to deliver: a fixed schema,
one row per order, positive amounts, and uppercase currency codes.

```{python}
orders_contract = pb.Contract(
    name="orders",
    direction="target",
    schema=pb.Schema(order_id="String", amount="Float64", currency="String"),
    steps=[
        pb.Step("rows_distinct", columns_subset="order_id"),
        pb.Step("col_vals_gt", columns="amount", value=0),
        pb.Step("col_vals_in_set", columns="currency", set=["USD", "EUR", "GBP"]),
    ],
)

pipeline = pb.Pipeline(source=raw_orders_contract, target=orders_contract, label="Orders")

pipeline
```

The transformation is an ordinary function that takes a table and returns a table. This one
removes zero-amount test orders, converts cents to whole currency units, and uppercases the
currency codes.

```{python}
def prepare_orders(df: pl.DataFrame) -> pl.DataFrame:
    return df.filter(pl.col("amount_cents") > 0).select(
        "order_id",
        amount=pl.col("amount_cents") / 100,
        currency=pl.col("currency").str.to_uppercase(),
    )
```

Now we can give the pipeline a day of raw orders and the transformation function.
[`run()`](`pointblank.Pipeline.run`) validates the input, transforms it, and validates the
output.

```{python}
raw_orders = pl.DataFrame(
    {
        "order_id": ["A1", "A2", "A3", "A4"],
        "amount_cents": [2500, 0, 11050, 899],
        "currency": ["usd", "eur", "usd", "gbp"],
    }
)

result = pipeline.run(data=raw_orders, transform=prepare_orders)

result
```

Both boundaries pass, so the [`PipelineResult`](`pointblank.PipelineResult`) reports
`passed=True`. Its `transform_output` attribute holds the prepared table, ready to be written
out:

```{python}
result.transform_output
```

Each boundary validation is kept as an interrogated `Validate` object, so the full report is
available. Here's the target validation, with the label built from the pipeline label and the
contract's direction.

```{python}
result.target_validation
```

The next day's raw orders have problems: a record with no order ID, a negative amount, and a
currency the transformation can't handle. Since `short_circuit=` is `True` by default, the
pipeline stops after the source validation fails. The source contract uses the default
`on_violation="warn"`, so a warning is also issued.

```{python}
bad_raw_orders = pl.DataFrame(
    {
        "order_id": ["B1", None, "B3", "B4"],
        "amount_cents": [1200, 3400, -500, 760],
        "currency": ["usd", "usd", "cad", "eur"],
    }
)

result = pipeline.run(data=bad_raw_orders, transform=prepare_orders)

print(f"passed: {result.passed}")
print(f"transform output: {result.transform_output}")
```

The transformation never ran, so there's no output and no target validation. The source
validation shows why the input was rejected, with one failing test unit in each step:

```{python}
result.source_validation
```

The batch was caught at the source boundary, so nothing reached the target table. Once the
upstream data is fixed, the same pipeline can be run again on it without any changes.

See Also
--------
PipelineResult : The result of `run()`, holding both boundary validations.
Contract : Define the source and target contracts.
Step : A single validation rule within a contract.

PipelineResult(source_validation: 'Validate | None' = None, target_validation: 'Validate | None' = None, transform_output: 'Any' = None, _source_passed: 'bool | None' = None, _target_passed: 'bool | None' = None) -> None

The outcome of running a data pipeline.

A `PipelineResult` is what [`Pipeline.run()`](`pointblank.Pipeline.run`) returns. It collects
everything a run produced: the interrogated validation for each boundary, the transformed data,
and pass/fail properties for a quick go/no-go decision. Keeping all of it in one object means
the code that runs a pipeline can decide what to do next (publish the output, stop the job, or
alert someone) and still have the full reports at hand to explain the decision.

You don't normally create a `PipelineResult` yourself. Get one from `run()` and read its
attributes and properties. When `run()` raises a
[`ContractViolationError`](`pointblank.ContractViolationError`) instead, the error's `result`
attribute holds the `PipelineResult`.

Attributes
----------
source_validation
    The interrogated [`Validate`](`pointblank.Validate`) object for the source boundary, or
    `None` if the pipeline has no source contract. Display it to see the validation report, or
    use any of its methods (e.g., [`get_step_report()`](`pointblank.Validate.get_step_report`)
    or [`get_data_extracts()`](`pointblank.Validate.get_data_extracts`)) to dig into
    failures.
target_validation
    The interrogated `Validate` object for the target boundary. It's `None` if the pipeline
    has no target contract, or if the run stopped after a failing source validation (with
    `short_circuit=True`).
transform_output
    The table returned by the transformation function, or `None` if the run stopped before
    the transformation was called. This is the data to publish when the result passes.

Passing and Failing
-------------------
The [`passed`](`pointblank.PipelineResult.passed`),
[`source_passed`](`pointblank.PipelineResult.source_passed`), and
[`target_passed`](`pointblank.PipelineResult.target_passed`) properties give a boolean verdict.
A boundary passes when every test unit in every step passes (the same test as
[`Validate.all_passed()`](`pointblank.Validate.all_passed`)), so thresholds don't affect it. A
boundary that wasn't validated counts as passing. That covers a pipeline without that
contract, and also a target that was skipped because the source failed: in that case
`target_passed` is `True`, but `passed` is still `False` because of the source. For a verdict
based on threshold levels instead, use the validations directly, for example with
[`above_threshold()`](`pointblank.Validate.above_threshold`).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
import warnings
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
warnings.formatwarning = lambda message, category, *args, **kwargs: (
    f"{category.__name__}: {message}\n"
)
```

Let's run a small pipeline that turns raw temperature readings in Fahrenheit into Celsius. The
source contract requires a station for every reading, and the target contract requires
temperatures in a plausible range.

```{python}
import pointblank as pb
import polars as pl

pipeline = pb.Pipeline(
    source=pb.Contract(
        name="raw_readings", steps=[pb.Step("col_vals_not_null", columns="station")]
    ),
    target=pb.Contract(
        name="readings",
        direction="target",
        steps=[pb.Step("col_vals_between", columns="temp_c", left=-60, right=60)],
    ),
)

raw_readings = pl.DataFrame({"station": ["N1", "N2", "S1"], "temp_f": [68.0, 41.0, 95.0]})

result = pipeline.run(
    data=raw_readings,
    transform=lambda df: df.select("station", temp_c=(pl.col("temp_f") - 32) * 5 / 9),
)

result
```

Printing the result shows which boundaries were validated and whether the run passed. In a
scheduled job, `passed` is the natural thing to branch on:

```{python}
if result.passed:
    print(result.transform_output)
else:
    print("Not publishing: the pipeline failed.")
```

For a plain-text summary of both boundaries, suitable for a log or a notification, use
[`get_report()`](`pointblank.PipelineResult.get_report`):

```{python}
print(result.get_report())
```

The full validation report for either boundary is in `source_validation` or
`target_validation`:

```{python}
result.target_validation
```

All three converted readings pass the target's single step, which is why `passed` is `True`.
When a run fails, these reports show which steps failed and on which rows, and
[`get_data_extracts()`](`pointblank.Validate.get_data_extracts`) pulls out the failing rows
themselves.

See Also
--------
Pipeline : Define the boundaries and run the transformation.
Validate.all_passed : The test used to decide whether a boundary passes.

ContractViolationError(message: 'str', contract: 'Contract', validation: 'Validate', result: 'PipelineResult | None' = None) -> 'None'

The error raised when data fails a contract with `on_violation="raise"`.

A [`Pipeline`](`pointblank.Pipeline`) raises this error when a boundary validation has any
failing test units and that boundary's [`Contract`](`pointblank.Contract`) was created with
`on_violation="raise"`. It's raised by [`run()`](`pointblank.Pipeline.run`),
[`validate_source()`](`pointblank.Pipeline.validate_source`), and
[`validate_target()`](`pointblank.Pipeline.validate_target`). The message names the contract
and its direction, and the error carries the failing validation, so the code that catches it
can report or log the details before stopping.

It's a subclass of `RuntimeError`, so `except RuntimeError:` catches it as well.

Attributes
----------
contract
    The [`Contract`](`pointblank.Contract`) that the data failed.
validation
    The interrogated [`Validate`](`pointblank.Validate`) object for the failing boundary.
    Display it to see the validation report.
result
    The [`PipelineResult`](`pointblank.PipelineResult`) of the run, holding everything the run
    produced before it stopped (the source validation, and the transformed data and target
    validation if they were reached). It's `None` when the error comes from
    `validate_source()` or `validate_target()`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's make a pipeline whose target contract must hold for the output to be published. It
uses `on_violation="raise"`, so a failing output stops the run with an error.

```{python}
import pointblank as pb
import polars as pl

pipeline = pb.Pipeline(
    target=pb.Contract(
        name="prices",
        direction="target",
        steps=[pb.Step("col_vals_gt", columns="price", value=0)],
        on_violation="raise",
    ),
)

raw_prices = pl.DataFrame({"item": ["a", "b", "c"], "price": [4.5, 0.0, 12.0]})
```

The transformation passes a zero price through, so the run raises. The error holds the
result of the run, which still has the transformed data and the target validation:

```{python}
try:
    pipeline.run(data=raw_prices, transform=lambda df: df)
except pb.ContractViolationError as e:
    print(e)
    failed_run = e.result

failed_run.target_validation
```

The report shows the one failing test unit in the `col_vals_gt` step, the row for item
`"b"`. Nothing after the `run()` call was reached, so the prices weren't published.

See Also
--------
Pipeline : Enforce contracts at both ends of a transformation.
Contract : Set `on_violation=` to choose what happens when data fails.


## Contract Import/Export

Import external schema definitions (JSON Schema, Frictionless Table Schema, and more) into Pointblank validation workflows, or export Pointblank contracts to those formats. Use `import_contract()` as the entry point, `export_contract()` for the reverse, and `register_adapter()` to add support for custom formats.



import_contract(source: 'Any', *, format: 'str | None' = None, **kwargs: 'Any') -> 'ContractImport'

Import a contract or schema written in another format.

Data is often already described somewhere outside of Pointblank: a JSON Schema for an API
payload, a Frictionless Table Schema inside a Data Package, the `schema.yml` file of a dbt
project, or an Open Data Contract Standard (ODCS) document. `import_contract()` reads such a
definition and translates its rules into Pointblank validation steps. That way the existing
definition can be checked against real data without rewriting it by hand, and the rules stay
in the place where they're already maintained.

The result is a [`ContractImport`](`pointblank.ContractImport`) object. It holds the column
names and types, the mapped constraints, any metadata from the source (such as a title and a
description), and a warning for every rule that has no Pointblank equivalent. From there you
can:

- get a [`Validate`](`pointblank.Validate`) object, ready to interrogate, with
[`to_validate()`](`pointblank.ContractImport.to_validate`)
- get a reusable [`Contract`](`pointblank.Contract`) (for a
[`Pipeline`](`pointblank.Pipeline`), or to save as YAML) with
[`to_contract()`](`pointblank.ContractImport.to_contract`)
- generate equivalent Python code with
[`to_python()`](`pointblank.ContractImport.to_python`)
- review what was and wasn't translated with
[`summary()`](`pointblank.ContractImport.summary`)

The reverse direction is handled by [`export_contract()`](`pointblank.export_contract`).
Support for further formats can be added with
[`register_adapter()`](`pointblank.register_adapter`).

Parameters
----------
source
    The definition to import. This can be a path to a file, given as a string, or a dictionary
    holding content that has already been loaded (e.g., with `json.load()` or
    `yaml.safe_load()`). `pathlib.Path` objects aren't accepted, so pass `str(path)` instead.
format
    The name of the adapter to use: `"json_schema"`, `"frictionless"`, `"dbt"`, or `"odcs"`
    for the built-in formats, or the name of a custom adapter. If `None` (the default), the
    format is detected from the file name or the content (see the *Format Detection* section).
**kwargs
    Options for the adapter. The Frictionless adapter takes `resource=` (the name or position
    of the resource to use in a Data Package), the dbt adapter takes `model=` (the name of a
    model or source table), and the ODCS adapter takes `table=` (the name of a dataset table).
    Each defaults to the first entry in the document. The JSON Schema adapter takes no
    options. Options that an adapter doesn't recognize are ignored.

Returns
-------
ContractImport
    The imported columns, constraints, metadata, and warnings, along with methods for turning
    them into a `Validate` object, a `Contract`, Python code, or a text summary.

Raises
------
ValueError
    If the format can't be detected, if no adapter is registered under `format=`, or if the
    adapter doesn't support importing. Adapters also raise `ValueError` when the document lacks
    the content they need (e.g., a dbt file with no models) or when a `resource=`, `model=`, or
    `table=` value doesn't match anything in it.
FileNotFoundError
    If `source` is a path to a file that doesn't exist.

Supported Formats
-----------------
Each built-in adapter reads the column names and types and the following rules:

- **JSON Schema** (`"json_schema"`): the `properties` of an object schema become the columns.
`required` maps to `col_vals_not_null()`, `minimum`/`maximum` to `col_vals_ge()`/
`col_vals_le()`, `exclusiveMinimum`/`exclusiveMaximum` to `col_vals_gt()`/`col_vals_lt()`,
`enum` to `col_vals_in_set()`, `pattern` to `col_vals_regex()`, and `const` to
`col_vals_eq()`. The `format` values `email`, `uri`, `uri-reference`, `ipv4`, and `ipv6` map
to `col_vals_within_spec()`; other formats (such as `date`) produce a warning. The `title` and
`description` keywords are kept as metadata and `$schema` is recorded as the source version.
- **Frictionless Table Schema** (`"frictionless"`): a standalone Table Schema (with `fields`)
or a Data Package (with `resources`). The field constraints `required`, `unique`, `minimum`,
`maximum`, `enum`, and `pattern` are mapped, and a `primaryKey` becomes `col_vals_not_null()`
on each key column plus `rows_distinct()` over all of them. Foreign keys produce a warning.
- **dbt** (`"dbt"`): the models and source tables of a `schema.yml` file. The column tests
`not_null`, `unique`, and `accepted_values` are mapped (from either `data_tests:` or the older
`tests:` key), as is a model-level `unique` test with `combination_of_columns`.
`relationships` and any other tests produce a warning. The model name becomes the title.
- **Open Data Contract Standard** (`"odcs"`): documents with a `dataset:` list of tables and
their columns. `isNullable: false`, `isUnique`, `isPrimaryKey`, `enum` (or `values`),
`pattern`, and `minimum`/`maximum` are mapped. Length limits and `checks`/`quality` entries
produce a warning.

Format Detection
----------------
When `format=` isn't given, a file path is first matched against the file extensions of the
registered adapters: `.schema.json` (JSON Schema), `.resource.json` and `.datapackage.json`
(Frictionless), `.yml` and `.yaml` (dbt), and `.odcs.yml`, `.odcs.yaml`, and `.odcs.json`
(ODCS). If no extension matches, each adapter in turn inspects the content of the file or
dictionary (e.g., a `fields` list identifies a Table Schema and a `models` list a dbt file).
Since the dbt adapter claims every `.yml` and `.yaml` file, pass `format="odcs"` when importing
an ODCS document stored as YAML. Use [`list_adapters()`](`pointblank.list_adapters`) to see the
registered formats and their extensions.

Coverage and Warnings
---------------------
Not every rule in an external format can be expressed as a Pointblank validation step. When the
adapter recognizes a rule but can't translate it, it adds a message to `warnings` and leaves the
rule out. The `coverage` value is the share of recognized rules that were translated, so
`1.0` means nothing recognized was dropped. Keywords that an adapter doesn't look at all (such
as `minLength` in a JSON Schema) are skipped without a warning and don't count toward coverage,
so it's worth checking [`summary()`](`pointblank.ContractImport.summary`) against the source
when importing a definition for the first time.

Examples
--------
```{python}
#| echo: false
#| output: false
import os
import tempfile
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
os.chdir(tempfile.mkdtemp())
```

Suppose an API team publishes a JSON Schema for user profiles and we want to check a table of
profiles against it. The schema can be passed as a dictionary (here it's written inline, but it
would usually come from `json.load()`). Because it has a `$schema` key, the format is detected
without needing `format=`.

```{python}
import pointblank as pb
import polars as pl

user_schema = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "user_profile",
    "description": "A registered user of the app.",
    "type": "object",
    "properties": {
        "user_id": {"type": "integer", "minimum": 1},
        "email": {"type": "string", "format": "email"},
        "country": {"type": "string", "pattern": "^[A-Z]{2}$"},
        "plan": {"type": "string", "enum": ["free", "pro", "team"]},
    },
    "required": ["user_id", "email"],
}

imported = pb.import_contract(user_schema)

imported
```

The printed form gives the counts at a glance: four columns, six constraints, and nothing left
untranslated. The [`summary()`](`pointblank.ContractImport.summary`) method gives a little more
detail, including the version of the source specification.

```{python}
print(imported.summary())
```

Now let's validate some data. The [`to_validate()`](`pointblank.ContractImport.to_validate`)
method builds a `Validate` object with a schema check followed by one step per constraint, and
it accepts the usual `Validate` arguments such as `tbl_name=` and `label=`.

```{python}
users = pl.DataFrame(
    {
        "user_id": [1, 2, 3, 0],
        "email": ["ana@example.com", "ben@example", "cy@example.org", None],
        "country": ["US", "CA", "gb", "DE"],
        "plan": ["free", "pro", "enterprise", "team"],
    }
)

imported.to_validate(users, tbl_name="users", label="Checks from user_profile").interrogate()
```

The table has the right columns and types, so the schema check passes. The remaining failures
come straight from the JSON Schema rules: a `user_id` of `0` (below the minimum of `1`), a
missing email (which fails both the `required` check and the `email` format check), an email
without a valid domain, a lowercase country code, and a plan that isn't in the `enum`.

Files work the same way. In this next example a dbt `schema.yml` file with two models is
written to the working directory. The `.yml` extension identifies it as a dbt file, and
`model=` picks the model to import (without it, the first model would be used).

```{python}
from pathlib import Path

dbt_yml = '''
version: 2
models:
  - name: customers
    columns:
      - name: customer_id
        data_tests: [not_null, unique]
  - name: orders
    description: One row per order.
    columns:
      - name: order_id
        data_type: integer
        data_tests: [not_null, unique]
      - name: customer_id
        data_type: integer
        data_tests:
          - not_null
          - relationships:
              to: ref('customers')
              field: customer_id
      - name: status
        data_type: varchar(20)
        data_tests:
          - accepted_values:
              values: [placed, shipped, returned]
'''

Path("schema.yml").write_text(dbt_yml)

orders_import = pb.import_contract("schema.yml", model="orders")

print(orders_import.summary())
```

Four of the five tests were translated. The `relationships` test checks values against another
model, which a single-table validation can't do, so it was reported as a warning and the
coverage is 80%. Note also that the dbt type `varchar(20)` was read as Pointblank's `String`
type.

The import can also be turned into a `Contract` with
[`to_contract()`](`pointblank.ContractImport.to_contract`), which accepts any `Contract`
argument. Printing the contract as YAML shows that the schema, the steps, and the model's
description were all carried over.

```{python}
orders_contract = orders_import.to_contract(name="orders", owner="analytics")

print(orders_contract.to_yaml())
```

The contract now has a name and an owner, and the dbt model's four translated tests are its
steps. From here it works like any other contract: it can be saved as YAML next to the dbt
project, enforced with [`Contract.validate()`](`pointblank.Contract.validate`), or used as a
boundary in a [`Pipeline`](`pointblank.Pipeline`).

See Also
--------
pointblank.export_contract : Export a `Validate` object or `Contract` to an external format.
ContractImport : The object returned by this function.
list_adapters : List the formats available for import and export.
register_adapter : Add support for another format.
Contract : Pointblank's own declarative contract.

export_contract(validation_or_contract: 'Any', destination: 'str | None' = None, *, format: 'str', **kwargs: 'Any') -> 'str | dict[str, Any]'

Export a Pointblank validation or contract to another format.

Data quality rules written in Pointblank are often useful to tools that don't run Pointblank.
An API team may want a JSON Schema, an analytics engineering team may want the tests in a dbt
`schema.yml` file, and a data catalog may accept Open Data Contract Standard (ODCS) documents.
`export_contract()` translates a [`Contract`](`pointblank.Contract`) or a
[`Validate`](`pointblank.Validate`) object into one of these formats, so a single set of rules
can be published wherever it's needed.

Contracts export most completely because they carry a name, a description, and a
[`Schema`](`pointblank.Schema`) with column types. A `Validate` object is exported from the
steps in its plan; it doesn't need to be interrogated first. The translation is lossy by
nature: only checks that have an equivalent in the target format are written, and the rest are
left out without a warning (see the *What Gets Exported* section). To bring a definition back
into Pointblank, use [`import_contract()`](`pointblank.import_contract`).

Parameters
----------
validation_or_contract
    The `Contract` or `Validate` object to export.
destination
    A file path to write the exported document to. Parent directories are created as needed.
    If `None` (the default), nothing is written and the document is only returned.
format
    The name of the adapter to use: `"json_schema"`, `"frictionless"`, `"dbt"`, or `"odcs"`
    for the built-in formats, or the name of a custom adapter. This argument is required and
    must be given by keyword.
**kwargs
    Options for the adapter. None of the built-in adapters currently take any options.

Returns
-------
str | dict
    The exported document. The built-in adapters return it as a dictionary (whether or not it's
    also written to `destination=`), which can be serialized with `json.dumps()` or
    `yaml.dump()`.

Raises
------
ValueError
    If no adapter is registered under `format=` (the message lists the available ones), or if
    the adapter doesn't support exporting.
TypeError
    If `validation_or_contract` is neither a `Contract` nor a `Validate` object.

What Gets Exported
------------------
The column types come from the contract's schema, and each built-in adapter translates these
checks:

- **JSON Schema** (`"json_schema"`): a draft 2020-12 object schema whose `title` and
`description` come from the contract. `col_vals_not_null()` adds the column to `required`;
`col_vals_ge()`/`col_vals_le()` become `minimum`/`maximum`, `col_vals_gt()`/`col_vals_lt()`
become `exclusiveMinimum`/`exclusiveMaximum`, and `col_vals_between()` sets both `minimum` and
`maximum`. `col_vals_in_set()` becomes `enum`, `col_vals_regex()` becomes `pattern`,
`col_vals_eq()` becomes `const`, and `col_vals_within_spec()` becomes a `format` for the
`email`, `url`, `ipv4`, and `ipv6` specs.
- **Frictionless Table Schema** (`"frictionless"`): a `fields` list with field constraints.
`col_vals_not_null()` becomes `required`, `rows_distinct()` becomes `unique`,
`col_vals_ge()`/`col_vals_le()` become `minimum`/`maximum`, `col_vals_in_set()` becomes `enum`,
and `col_vals_regex()` becomes `pattern`.
- **dbt** (`"dbt"`): a `version: 2` document with one model named after the contract.
`col_vals_not_null()`, `rows_distinct()`, and `col_vals_in_set()` become the generic tests
`not_null`, `unique`, and `accepted_values`.
- **Open Data Contract Standard** (`"odcs"`): a `DataContract` document with an `info` block
and one dataset table. `col_vals_not_null()` sets `isNullable: false`, `rows_distinct()` sets
`isUnique`, `col_vals_in_set()` becomes `enum`, `col_vals_regex()` becomes `pattern`, and
`col_vals_ge()`/`col_vals_le()` become `minimum`/`maximum`.

Any other step, along with thresholds, actions, and owner information, isn't exported.

When `destination=` is given, the JSON Schema and Frictionless documents are written as JSON
and the dbt document as YAML. ODCS documents are written as JSON if the path ends in `.json`
and as YAML otherwise.

Examples
--------
```{python}
#| echo: false
#| output: false
import os
import tempfile
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
os.chdir(tempfile.mkdtemp())
```

Let's start with a `Contract` for a table of orders. It has a schema with column types, a
description, and a mix of checks.

```{python}
import pointblank as pb
import json

orders_contract = pb.Contract(
    name="orders",
    description="One row per customer order.",
    schema=pb.Schema(order_id="Int64", status="String", amount="Float64"),
    steps=[
        pb.Step("col_vals_not_null", columns=["order_id", "status"]),
        pb.Step("rows_distinct", columns_subset="order_id"),
        pb.Step("col_vals_in_set", columns="status", set=["placed", "shipped", "returned"]),
        pb.Step("col_vals_between", columns="amount", left=0, right=10000),
    ],
)
```

Exporting it as a JSON Schema gives us a dictionary, which we can print with `json.dumps()`.

```{python}
orders_json_schema = pb.export_contract(orders_contract, format="json_schema")

print(json.dumps(orders_json_schema, indent=2))
```

The Pointblank types became JSON types, the not-null check filled the `required` list, the set
of statuses became an `enum`, and the range check became `minimum` and `maximum` values. The
`rows_distinct()` step was left out because JSON Schema has no way to say that a value must be
unique across records.

The same contract can be written out as a dbt `schema.yml` file. Here we write it to a
`models` directory (which is created if it doesn't exist) and print the file's contents.

```{python}
from pathlib import Path

pb.export_contract(orders_contract, "models/schema.yml", format="dbt")

print(Path("models/schema.yml").read_text())
```

This time the uniqueness check is kept (as dbt's `unique` test) but the range check is not,
since dbt has no generic test for it. The `amount` column is still listed, with its type and an
empty list of tests.

An exported document can be imported again with `import_contract()`. Doing so is a quick way to
see which rules survive the round trip.

```{python}
print(pb.import_contract(orders_json_schema).summary())
```

All five constraints in the JSON Schema were mapped back, so the coverage is 100%. The range
check returns as two steps (one each for `minimum` and `maximum`), and the uniqueness check is
gone because it was dropped on export. Coverage measures how much of the *document* was
understood, not how much of the original contract survived. To find what an export loses,
compare the steps before and after.

See Also
--------
pointblank.import_contract : Import an external definition into Pointblank.
list_adapters : List the formats available for import and export.
register_adapter : Add support for another format.
Contract : The declarative contract that exports most completely.

list_adapters() -> 'dict[str, dict[str, Any]]'

List the formats available for contract import and export.

Every format that [`import_contract()`](`pointblank.import_contract`) and
[`export_contract()`](`pointblank.export_contract`) can work with is provided by a registered
adapter. `list_adapters()` reports what's registered in the current session: the four built-in
formats plus any custom adapters added with
[`register_adapter()`](`pointblank.register_adapter`). It's useful for finding the right value
for `format=`, for checking whether a format supports import, export, or both before calling
either function, and for seeing which file extensions are used to detect a format.

Returns
-------
dict[str, dict]
    A dictionary, sorted by format name, that maps each format name to a dictionary with these
    keys:

    - `"class"`: the name of the adapter class
    - `"file_extensions"`: the list of file extensions used to detect the format
    - `"supports_import"`: whether the format can be imported with `import_contract()`
    - `"supports_export"`: whether the format can be exported with `export_contract()`

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

With no custom adapters registered, the result lists the four built-in formats.

```{python}
import pointblank as pb

adapters = pb.list_adapters()

list(adapters)
```

Each entry describes one adapter. Here's the entry for the dbt adapter, which claims the `.yml`
and `.yaml` file extensions and supports both directions.

```{python}
adapters["dbt"]
```

Since the result is an ordinary dictionary, it's easy to filter. This finds the formats that
a contract can be exported to.

```{python}
[name for name, info in adapters.items() if info["supports_export"]]
```

All four built-in formats support export, so the list is the same as before. An adapter added
with [`register_adapter()`](`pointblank.register_adapter`) may be import-only, though, so a
check like this is useful before calling [`export_contract()`](`pointblank.export_contract`)
with a format chosen at run time.

See Also
--------
register_adapter : Add a custom format to this list.
pointblank.import_contract : Import a definition in one of the listed formats.
pointblank.export_contract : Export a contract to one of the listed formats.

register_adapter(format_name: 'str | None' = None)

Register a contract adapter so that it can be used for import and export.

Pointblank ships with adapters for JSON Schema, Frictionless Table Schema, dbt, and the Open
Data Contract Standard, but many organizations describe their data in other ways: an in-house
data dictionary, a spreadsheet export, or a schema format from another tool. Writing a
[`ContractAdapter`](`pointblank.ContractAdapter`) subclass for such a format and registering it
with `register_adapter()` makes the format a first-class citizen.
[`import_contract()`](`pointblank.import_contract`) and
[`export_contract()`](`pointblank.export_contract`) can then use it by name with `format=`,
and [`list_adapters()`](`pointblank.list_adapters`) includes it.

Registration creates one instance of the class (by calling it with no arguments) and stores it
under the format name for the rest of the session. Registering a second adapter under a name
that's already taken replaces the first one, and that includes the built-in adapters. When a
format has to be detected, adapters are tried in the order they were registered, so the
built-in adapters are always tried before custom ones.

Parameters
----------
format_name
    The name to register the adapter under, which becomes the value to pass to `format=`. If
    `None` (the default), the class's own `format_name` attribute is used.

Returns
-------
Callable or type
    When called with a name (or with no arguments), a decorator that registers the class it's
    applied to and returns that class unchanged. When applied directly to a class (as a bare
    `@register_adapter`), the class itself, unchanged.

Raises
------
ValueError
    If no name is given and the class has an empty `format_name` attribute.

Ways to Register
----------------
There are three equivalent ways to register an adapter class:

- `@register_adapter("name")` above the class definition registers it under `"name"`
- a bare `@register_adapter` above the class definition uses the class's `format_name`
- `register_adapter("name")(AdapterClass)` registers a class that's already been defined (the
name can be left out here too, as in `register_adapter()(AdapterClass)`)

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose a team keeps a simple list of required columns in a dictionary with a `required_columns`
key. A minimal adapter for this format only needs to say how to recognize it and how to turn it
into a [`ContractImport`](`pointblank.ContractImport`). Since this adapter can't export, it sets
`supports_export = False`.

```{python}
import pointblank as pb
from pointblank.adapters import MappedConstraint


@pb.register_adapter("required_columns")
class RequiredColumnsAdapter(pb.ContractAdapter):
    format_name = "required_columns"
    supports_export = False

    @staticmethod
    def detect(source):
        return isinstance(source, dict) and "required_columns" in source

    def import_contract(self, source, **kwargs):
        columns = source["required_columns"]
        return pb.ContractImport(
            source_format="required_columns",
            columns=[(col, None) for col in columns],
            constraints=[
                MappedConstraint("col_vals_not_null", {"columns": col}, f"{col} is required")
                for col in columns
            ],
        )
```

The new format now appears next to the built-in ones in `list_adapters()`.

```{python}
for name, info in pb.list_adapters().items():
    print(f"{name:<18} import={info['supports_import']}  export={info['supports_export']}")
```

And it can be used with `import_contract()`, either by name or (since it implements
`detect()`) by letting the format be detected from the content.

```{python}
imported = pb.import_contract({"required_columns": ["id", "email"]})

imported.constraints
```

Exporting to this format isn't possible, so `export_contract()` raises an error that says so.

```{python}
contract = pb.Contract(name="users", steps=[pb.Step("col_vals_not_null", columns="id")])

try:
    pb.export_contract(contract, format="required_columns")
except ValueError as e:
    print(e)
```

The error is about the adapter, not the contract. To make the format work in both directions,
the adapter class would also implement
[`export_contract()`](`pointblank.ContractAdapter.export_contract`) and drop the
`supports_export = False` line.

See Also
--------
ContractAdapter : The base class to subclass when writing an adapter.
ContractImport : What an adapter's `import_contract()` method returns.
list_adapters : List the registered adapters.
pointblank.import_contract : Import a definition using a registered adapter.
pointblank.export_contract : Export a contract using a registered adapter.

ContractImport(source_format: 'str', source_path: 'str | None' = None, source_version: 'str | None' = None, columns: 'list[tuple[str, str | None]]' = <factory>, constraints: 'list[MappedConstraint]' = <factory>, metadata: 'dict[str, Any]' = <factory>, warnings: 'list[str]' = <factory>, coverage: 'float' = 1.0) -> None

The result of importing a contract or schema from another format.

[`import_contract()`](`pointblank.import_contract`) returns a `ContractImport`. It's the
halfway point between an external definition (a JSON Schema, a Frictionless Table Schema, a dbt
`schema.yml` file, and so on) and a Pointblank validation. The definition has been read and
translated, but nothing has been run yet, so this is the place to review the translation and
decide how to use it.

The object holds what the adapter found: the columns and their types, the rules translated into
[`MappedConstraint`](`pointblank.adapters.MappedConstraint`) objects, the source's metadata, and
a warning for each rule that couldn't be translated. Its methods turn this into something
usable:

- [`to_validate()`](`pointblank.ContractImport.to_validate`) builds a
[`Validate`](`pointblank.Validate`) object for a table, ready to interrogate or to extend with
more steps
- [`to_contract()`](`pointblank.ContractImport.to_contract`) builds a
[`Contract`](`pointblank.Contract`), which can be used in a [`Pipeline`](`pointblank.Pipeline`)
or saved as YAML
- [`to_python()`](`pointblank.ContractImport.to_python`) writes equivalent Python code, for
when the imported rules should become a starting point that's edited by hand
- [`to_yaml()`](`pointblank.ContractImport.to_yaml`) lists the steps as YAML
- [`summary()`](`pointblank.ContractImport.summary`) describes what was imported and what
wasn't

`ContractImport` is a dataclass, so its fields can be read and changed directly. For instance,
an unwanted constraint can be removed from `constraints` before calling `to_validate()`.
Custom adapters (see [`ContractAdapter`](`pointblank.ContractAdapter`)) create these objects
by calling the constructor with the parameters below.

Parameters
----------
source_format
    The name of the adapter that produced the import, such as `"json_schema"` or `"dbt"`.
source_path
    The path of the imported file, or `None` if the definition was given as a dictionary.
source_version
    The version of the source format, when the document states one (e.g., the `$schema` URL
    of a JSON Schema). Otherwise `None` or an empty string.
columns
    The columns found in the source, as a list of `(name, dtype)` tuples. The `dtype` is a
    Pointblank type name such as `"Int64"` or `"String"`, or `None` when the source doesn't give
    a type that maps to one.
constraints
    The translated rules, as a list of `MappedConstraint` objects, in the order in which they
    become validation steps.
metadata
    Other information from the source, such as a `"title"` and a `"description"`.
warnings
    A message for each rule that was recognized but couldn't be translated.
coverage
    The share of recognized rules that were translated, from `0.0` to `1.0`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's import a Frictionless Data Package that describes two tables. The `resource=` option
selects the `readings` table, which has a primary key and a foreign key to the `stations`
table.

```{python}
import pointblank as pb
import polars as pl

package = {
    "name": "weather",
    "resources": [
        {
            "name": "stations",
            "schema": {"fields": [{"name": "station_id", "type": "string"}]},
        },
        {
            "name": "readings",
            "schema": {
                "fields": [
                    {"name": "station_id", "type": "string"},
                    {"name": "reading_no", "type": "integer"},
                    {
                        "name": "temp_c",
                        "type": "number",
                        "constraints": {"minimum": -90, "maximum": 60},
                    },
                ],
                "primaryKey": ["station_id", "reading_no"],
                "foreignKeys": [
                    {
                        "fields": "station_id",
                        "reference": {"resource": "stations", "fields": "station_id"},
                    }
                ],
            },
        },
    ],
}

readings = pb.import_contract(package, resource="readings")

readings
```

The fields hold the details. The columns come with their Frictionless types translated to
Pointblank types:

```{python}
readings.columns
```

The constraints show how the temperature range became a pair of bounds, and how the primary
key was expressed as not-null checks plus a `rows_distinct()` check over both key columns.

```{python}
for c in readings.constraints:
    print(c)
```

The foreign key couldn't be translated, since checking it would need the `stations` table. It
shows up in `warnings`, and it's why the coverage is below 100%.

```{python}
readings.warnings
```

Because the fields can be changed, the import can be adjusted before it's used. Here the
temperature range is dropped (perhaps the sensors are known to report out-of-range values
that are dealt with later) and the remaining checks are run against a table.

```{python}
readings.constraints = [c for c in readings.constraints if c.kwargs.get("columns") != "temp_c"]

readings_tbl = pl.DataFrame(
    {
        "station_id": ["S1", "S1", "S2", "S2"],
        "reading_no": [1, 2, 1, 1],
        "temp_c": [12.5, 13.1, 75.0, 8.4],
    }
)

readings.to_validate(readings_tbl, tbl_name="readings").interrogate()
```

The duplicated `("S2", 1)` key is caught by the `rows_distinct()` step, while the `75.0` reading
no longer fails any step because the range check was removed.

See Also
--------
pointblank.import_contract : Create a `ContractImport` from an external definition.
adapters.MappedConstraint : The objects in the `constraints` list.
ContractAdapter : The base class for adapters, which create these objects.
Contract : The Pointblank contract that `to_contract()` produces.

ContractAdapter()

The base class for adapters that translate contracts to and from other formats.

An adapter is what connects an external format to
[`import_contract()`](`pointblank.import_contract`) and
[`export_contract()`](`pointblank.export_contract`). Pointblank's built-in support for JSON
Schema, Frictionless Table Schema, dbt, and the Open Data Contract Standard is made of four
`ContractAdapter` subclasses. Writing another subclass adds a new format, such as an
organization's own data dictionary or a schema format from another tool. Once it's registered
with [`register_adapter()`](`pointblank.register_adapter`), the new format works everywhere the
built-in ones do.

A subclass sets a few class attributes and implements up to three methods:

- [`detect()`](`pointblank.ContractAdapter.detect`) decides whether a source is in this format,
which lets `import_contract()` find the adapter without being told `format=` (optional)
- [`import_contract()`](`pointblank.ContractAdapter.import_contract`) reads a source and
returns a [`ContractImport`](`pointblank.ContractImport`) whose rules are expressed as
[`MappedConstraint`](`pointblank.adapters.MappedConstraint`) objects (needed for import)
- [`export_contract()`](`pointblank.ContractAdapter.export_contract`) translates a
[`Contract`](`pointblank.Contract`) or [`Validate`](`pointblank.Validate`) object into the
format (needed for export)

The base class versions of these methods raise `NotImplementedError`. An adapter that only
goes in one direction should set `supports_import` or `supports_export` to `False`, so that the
unsupported direction fails with a clear error message.

Attributes
----------
format_name
    The name of the format, used as the value of `format=`. It's also the registration name
    when `register_adapter()` is used without one.
file_extensions
    File name endings that identify the format, such as `[".schema.json"]`. They're matched
    against file paths during format detection, before any `detect()` method is tried.
supports_import
    Whether the adapter can import. The default is `True`.
supports_export
    Whether the adapter can export. The default is `True`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose a team documents its tables in a "data dictionary" format of its own: a dictionary
with the table name under `data_dictionary` and a list of `columns`. Each column has a name and
a type and can say whether it's nullable, whether it's unique, and which values are allowed. It
can also have a `pii` flag, which matters for governance but isn't a data quality rule.

The adapter below supports both directions. Importing maps the column properties to
`MappedConstraint` objects and records a warning for the `pii` flag, which has no equivalent.
Exporting goes the other way, starting from a contract's schema and steps.

```{python}
import json
import pointblank as pb
from pointblank.adapters import MappedConstraint

TYPES = {"integer": "Int64", "decimal": "Float64", "text": "String"}


@pb.register_adapter
class DataDictionaryAdapter(pb.ContractAdapter):
    format_name = "data_dictionary"
    file_extensions = [".dd.json"]

    @staticmethod
    def detect(source):
        return isinstance(source, dict) and "data_dictionary" in source

    def import_contract(self, source, **kwargs):
        path = None
        if isinstance(source, str):
            path = source
            with open(source) as f:
                source = json.load(f)

        columns, constraints, warnings = [], [], []
        for fld in source["columns"]:
            name = fld["name"]
            columns.append((name, TYPES.get(fld.get("type"))))
            if fld.get("nullable") is False:
                constraints.append(
                    MappedConstraint("col_vals_not_null", {"columns": name}, "nullable: false")
                )
            if fld.get("unique"):
                constraints.append(
                    MappedConstraint("rows_distinct", {"columns_subset": [name]}, "unique")
                )
            if "allowed" in fld:
                constraints.append(
                    MappedConstraint(
                        "col_vals_in_set", {"columns": name, "set": fld["allowed"]}, "allowed"
                    )
                )
            if fld.get("pii"):
                warnings.append(f"Column '{name}': the pii flag has no Pointblank equivalent.")

        n_rules = len(constraints) + len(warnings)
        return pb.ContractImport(
            source_format=self.format_name,
            source_path=path,
            columns=columns,
            constraints=constraints,
            metadata={"title": source["data_dictionary"]},
            warnings=warnings,
            coverage=len(constraints) / n_rules if n_rules else 1.0,
        )

    def export_contract(self, validation_or_contract, destination=None, **kwargs):
        if not isinstance(validation_or_contract, pb.Contract):
            raise TypeError("This adapter can only export a Contract.")
        contract = validation_or_contract

        to_dd_type = {pb_type: dd_type for dd_type, pb_type in TYPES.items()}
        entries = {name: {"name": name} for name, _ in contract.schema.columns}
        for name, dtype in contract.schema.columns:
            entries[name]["type"] = to_dd_type.get(dtype, "text")
        for step in contract.steps:
            columns = step.kwargs.get("columns")
            columns = [columns] if isinstance(columns, str) else columns or []
            for col in columns:
                if step.method == "col_vals_not_null":
                    entries[col]["nullable"] = False
                elif step.method == "col_vals_in_set":
                    entries[col]["allowed"] = list(step.kwargs["set"])

        doc = {"data_dictionary": contract.name, "columns": list(entries.values())}
        if destination is not None:
            with open(destination, "w") as f:
                json.dump(doc, f, indent=2)
        return doc
```

The bare `@pb.register_adapter` decorator registered the class under its `format_name`. Now
a data dictionary can be imported with `import_contract()`, and its `detect()` method means
there's no need for `format=`.

```{python}
customers_dd = {
    "data_dictionary": "customers",
    "columns": [
        {"name": "customer_id", "type": "integer", "nullable": False, "unique": True},
        {"name": "tier", "type": "text", "allowed": ["bronze", "silver", "gold"]},
        {"name": "email", "type": "text", "nullable": False, "pii": True},
    ],
}

imported = pb.import_contract(customers_dd)

print(imported.summary())
```

Four of the five column properties became constraints and the `pii` flag became a warning,
which the coverage figure reflects. From here the import behaves like any other: it can be
validated against a table, turned into a contract, or written out as code.

Exporting runs in the other direction. A contract written in Pointblank can be shared as a
data dictionary with `export_contract()`.

```{python}
orders_contract = pb.Contract(
    name="orders",
    schema=pb.Schema(order_id="Int64", status="String", total="Float64"),
    steps=[
        pb.Step("col_vals_not_null", columns=["order_id", "status"]),
        pb.Step("col_vals_in_set", columns="status", set=["open", "closed"]),
        pb.Step("col_vals_ge", columns="total", value=0),
    ],
)

print(json.dumps(pb.export_contract(orders_contract, format="data_dictionary"), indent=2))
```

The schema supplied the columns and their types, and the not-null and set-membership steps
became `nullable` and `allowed` properties. The data dictionary format has nothing that
corresponds to `col_vals_ge()`, so that step was left out.

See Also
--------
register_adapter : Register an adapter so that it can be used by name.
ContractImport : The object an adapter's `import_contract()` method returns.
adapters.MappedConstraint : The objects that hold translated rules.
list_adapters : List the registered adapters.

MappedConstraint(method: 'str', kwargs: 'dict[str, Any]' = <factory>, source_description: 'str' = '') -> None

A rule from an external format, translated into a Pointblank validation step.

When an adapter imports a definition, each rule it can translate becomes a `MappedConstraint`:
the name of a [`Validate`](`pointblank.Validate`) method and the arguments to call it with.
Together they make up the `constraints` list of a
[`ContractImport`](`pointblank.ContractImport`), which turns them into real validation steps
in [`to_validate()`](`pointblank.ContractImport.to_validate`), into
[`Step`](`pointblank.Step`) objects in
[`to_contract()`](`pointblank.ContractImport.to_contract`), and into method calls in
[`to_python()`](`pointblank.ContractImport.to_python`).

Most users only read these objects, to see what an import produced. Anyone writing a
[`ContractAdapter`](`pointblank.ContractAdapter`) creates them, and that's the reason for the
`source_description=` field: it records the original rule in the source's own terms (such as
`"minimum: 0"`), which helps when tracing a validation step back to where it came from.

`MappedConstraint` is available from the `pointblank.adapters` module.

Parameters
----------
method
    The name of a `Validate` validation method, such as `"col_vals_gt"` or `"rows_distinct"`.
kwargs
    The keyword arguments for that method, such as `{"columns": "age", "value": 0}`.
source_description
    A human-readable description of the rule as written in the source format.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Here's a constraint that requires values in the `age` column to be at least `0`, written the
way an adapter would create it from a JSON Schema `minimum` keyword.

```{python}
from pointblank.adapters import MappedConstraint

constraint = MappedConstraint(
    method="col_vals_ge",
    kwargs={"columns": "age", "value": 0},
    source_description="minimum: 0 on age",
)

constraint
```

The printed form shows the method call that the constraint stands for. Constraints are what an
imported definition is made of, as we can see by importing a small JSON Schema and looking at
its `constraints` list.

```{python}
import pointblank as pb

imported = pb.import_contract(
    {
        "type": "object",
        "properties": {"age": {"type": "integer", "minimum": 0}},
        "required": ["age"],
    }
)

for c in imported.constraints:
    print(f"{c.method:<18} {c.kwargs}  (from {c.source_description!r})")
```

Each JSON Schema keyword became one constraint: the `required` entry gave a not-null check
and `minimum` gave a lower bound. The `source_description` records where each constraint
came from, which helps when tracing a failing step back to the definition that produced it.

See Also
--------
ContractImport : The import result that holds a list of constraints.
ContractAdapter : The base class for adapters, which create constraints.
Step : The equivalent declarative object used in a `Contract`.


## Validation Steps

Validation steps are sequential validations on the target data. Call `Validate`'s validation methods to build up a validation plan: a collection of steps that provides good validation coverage.



col_vals_gt(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', value: 'float | int | Column', na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Are column data greater than a fixed value or data in another column?

The `col_vals_gt()` validation method checks whether column values in a table are
*greater than* a specified `value=` (the exact comparison used in this function is
`col_val > value`). The `value=` can be specified as a single, literal value or as a column
name given in [`col()`](`pointblank.col`). This validation will operate over the number of
test units that is equal to the number of rows in the table (determined after any `pre=`
mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
value
    The value to compare against. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison. For more information on which types of values are allowed, see the
    *What Can Be Used in `value=`?* section.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
    A `NaN` in a float column also counts as missing on every backend, and that includes a
    `NaN` in a column given to `value=` via [`col()`](`pointblank.col`).
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

What Can Be Used in `value=`?
-----------------------------
The `value=` argument allows for a variety of input types. The most common are:

- a single numeric value
- a single date or datetime value
- A [`col()`](`pointblank.col`) object that represents a column name

When supplying a number as the basis of comparison, keep in mind that all resolved columns
must also be numeric. Should you have columns that are of the date or datetime types, you
can supply a date or datetime value as the `value=` argument. There is flexibility in how
you provide the date or datetime value, as it can be:

- a string-based date or datetime (e.g., `"2023-10-01"`, `"2023-10-01 13:45:30"`, etc.),
  optionally with a UTC offset for a timezone-aware datetime (e.g.,
  `"2023-10-01T13:45:30+05:00"` or `"2023-10-01T13:45:30Z"`)
- a date or datetime object using the `datetime` module (e.g., `datetime.date(2023, 10, 1)`,
  `datetime.datetime(2023, 10, 1, 13, 45, 30)`, etc.)

Finally, when supplying a column name in the `value=` argument, it must be specified within
[`col()`](`pointblank.col`). This is a column-to-column comparison and, crucially, the
columns being compared must be of the same type (e.g., both numeric, both date, etc.).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns=` and `value=col(...)` that are expected to be present in the
transformed table, but may not exist in the table before preprocessing. Regarding the
lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three numeric columns (`a`,
`b`, and `c`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 6, 5, 7, 6, 5],
        "b": [1, 2, 1, 2, 2, 2],
        "c": [2, 1, 2, 2, 3, 4],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are all greater than the value of `4`. We'll
determine if this validation had any failing test units (there are six test units, one for
each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=4)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_gt()`. All test units passed, and there are no failing test units.

Aside from checking a column against a literal value, we can also use a column name in the
`value=` argument (with the helper function [`col()`](`pointblank.col`) to perform a
column-to-column comparison. For the next example, we'll use `col_vals_gt()` to check
whether the values in column `c` are greater than values in column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="c", value=pb.col("b"))
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are:

- Row 1: `c` is `1` and `b` is `2`.
- Row 3: `c` is `2` and `b` is `2`.

col_vals_lt(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', value: 'float | int | Column', na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Are column data less than a fixed value or data in another column?

The `col_vals_lt()` validation method checks whether column values in a table are
*less than* a specified `value=` (the exact comparison used in this function is
`col_val < value`). The `value=` can be specified as a single, literal value or as a column
name given in [`col()`](`pointblank.col`). This validation will operate over the number of
test units that is equal to the number of rows in the table (determined after any `pre=`
mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
value
    The value to compare against. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison. For more information on which types of values are allowed, see the
    *What Can Be Used in `value=`?* section.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
    A `NaN` in a float column also counts as missing on every backend, and that includes a
    `NaN` in a column given to `value=` via [`col()`](`pointblank.col`).
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

What Can Be Used in `value=`?
-----------------------------
The `value=` argument allows for a variety of input types. The most common are:

- a single numeric value
- a single date or datetime value
- A [`col()`](`pointblank.col`) object that represents a column name

When supplying a number as the basis of comparison, keep in mind that all resolved columns
must also be numeric. Should you have columns that are of the date or datetime types, you
can supply a date or datetime value as the `value=` argument. There is flexibility in how
you provide the date or datetime value, as it can be:

- a string-based date or datetime (e.g., `"2023-10-01"`, `"2023-10-01 13:45:30"`, etc.),
  optionally with a UTC offset for a timezone-aware datetime (e.g.,
  `"2023-10-01T13:45:30+05:00"` or `"2023-10-01T13:45:30Z"`)
- a date or datetime object using the `datetime` module (e.g., `datetime.date(2023, 10, 1)`,
  `datetime.datetime(2023, 10, 1, 13, 45, 30)`, etc.)

Finally, when supplying a column name in the `value=` argument, it must be specified within
[`col()`](`pointblank.col`). This is a column-to-column comparison and, crucially, the
columns being compared must be of the same type (e.g., both numeric, both date, etc.).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns=` and `value=col(...)` that are expected to be present in the
transformed table, but may not exist in the table before preprocessing. Regarding the
lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three numeric columns (`a`,
`b`, and `c`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 6, 5, 9, 7, 5],
        "b": [1, 2, 1, 2, 2, 2],
        "c": [2, 1, 1, 4, 3, 4],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are all less than the value of `10`. We'll
determine if this validation had any failing test units (there are six test units, one for
each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_lt(columns="a", value=10)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_lt()`. All test units passed, and there are no failing test units.

Aside from checking a column against a literal value, we can also use a column name in the
`value=` argument (with the helper function [`col()`](`pointblank.col`) to perform a
column-to-column comparison. For the next example, we'll use `col_vals_lt()` to check
whether the values in column `b` are less than values in column `c`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_lt(columns="b", value=pb.col("c"))
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are:

- Row 1: `b` is `2` and `c` is `1`.
- Row 2: `b` is `1` and `c` is `1`.

col_vals_ge(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', value: 'float | int | Column', na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Are column data greater than or equal to a fixed value or data in another column?

The `col_vals_ge()` validation method checks whether column values in a table are
*greater than or equal to* a specified `value=` (the exact comparison used in this function
is `col_val >= value`). The `value=` can be specified as a single, literal value or as a
column name given in [`col()`](`pointblank.col`). This validation will operate over the
number of test units that is equal to the number of rows in the table (determined after any
`pre=` mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
value
    The value to compare against. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison. For more information on which types of values are allowed, see the
    *What Can Be Used in `value=`?* section.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
    A `NaN` in a float column also counts as missing on every backend, and that includes a
    `NaN` in a column given to `value=` via [`col()`](`pointblank.col`).
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

What Can Be Used in `value=`?
-----------------------------
The `value=` argument allows for a variety of input types. The most common are:

- a single numeric value
- a single date or datetime value
- A [`col()`](`pointblank.col`) object that represents a column name

When supplying a number as the basis of comparison, keep in mind that all resolved columns
must also be numeric. Should you have columns that are of the date or datetime types, you
can supply a date or datetime value as the `value=` argument. There is flexibility in how
you provide the date or datetime value, as it can be:

- a string-based date or datetime (e.g., `"2023-10-01"`, `"2023-10-01 13:45:30"`, etc.),
  optionally with a UTC offset for a timezone-aware datetime (e.g.,
  `"2023-10-01T13:45:30+05:00"` or `"2023-10-01T13:45:30Z"`)
- a date or datetime object using the `datetime` module (e.g., `datetime.date(2023, 10, 1)`,
  `datetime.datetime(2023, 10, 1, 13, 45, 30)`, etc.)

Finally, when supplying a column name in the `value=` argument, it must be specified within
[`col()`](`pointblank.col`). This is a column-to-column comparison and, crucially, the
columns being compared must be of the same type (e.g., both numeric, both date, etc.).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns=` and `value=col(...)` that are expected to be present in the
transformed table, but may not exist in the table before preprocessing. Regarding the
lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three numeric columns (`a`,
`b`, and `c`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 6, 5, 9, 7, 5],
        "b": [5, 3, 1, 8, 2, 3],
        "c": [2, 3, 1, 4, 3, 4],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are all greater than or equal to the value of `5`.
We'll determine if this validation had any failing test units (there are six test units, one
for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_ge(columns="a", value=5)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_ge()`. All test units passed, and there are no failing test units.

Aside from checking a column against a literal value, we can also use a column name in the
`value=` argument (with the helper function [`col()`](`pointblank.col`) to perform a
column-to-column comparison. For the next example, we'll use `col_vals_ge()` to check
whether the values in column `b` are greater than values in column `c`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_ge(columns="b", value=pb.col("c"))
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are:

- Row 0: `b` is `2` and `c` is `3`.
- Row 4: `b` is `3` and `c` is `4`.

col_vals_le(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', value: 'float | int | Column', na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Are column data less than or equal to a fixed value or data in another column?

The `col_vals_le()` validation method checks whether column values in a table are
*less than or equal to* a specified `value=` (the exact comparison used in this function is
`col_val <= value`). The `value=` can be specified as a single, literal value or as a column
name given in [`col()`](`pointblank.col`). This validation will operate over the number of
test units that is equal to the number of rows in the table (determined after any `pre=`
mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
value
    The value to compare against. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison. For more information on which types of values are allowed, see the
    *What Can Be Used in `value=`?* section.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
    A `NaN` in a float column also counts as missing on every backend, and that includes a
    `NaN` in a column given to `value=` via [`col()`](`pointblank.col`).
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

What Can Be Used in `value=`?
-----------------------------
The `value=` argument allows for a variety of input types. The most common are:

- a single numeric value
- a single date or datetime value
- A [`col()`](`pointblank.col`) object that represents a column name

When supplying a number as the basis of comparison, keep in mind that all resolved columns
must also be numeric. Should you have columns that are of the date or datetime types, you
can supply a date or datetime value as the `value=` argument. There is flexibility in how
you provide the date or datetime value, as it can be:

- a string-based date or datetime (e.g., `"2023-10-01"`, `"2023-10-01 13:45:30"`, etc.),
  optionally with a UTC offset for a timezone-aware datetime (e.g.,
  `"2023-10-01T13:45:30+05:00"` or `"2023-10-01T13:45:30Z"`)
- a date or datetime object using the `datetime` module (e.g., `datetime.date(2023, 10, 1)`,
  `datetime.datetime(2023, 10, 1, 13, 45, 30)`, etc.)

Finally, when supplying a column name in the `value=` argument, it must be specified within
[`col()`](`pointblank.col`). This is a column-to-column comparison and, crucially, the
columns being compared must be of the same type (e.g., both numeric, both date, etc.).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns=` and `value=col(...)` that are expected to be present in the
transformed table, but may not exist in the table before preprocessing. Regarding the
lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three numeric columns (`a`,
`b`, and `c`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 6, 5, 9, 7, 5],
        "b": [1, 3, 1, 5, 2, 5],
        "c": [2, 1, 1, 4, 3, 4],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are all less than or equal to the value of `9`.
We'll determine if this validation had any failing test units (there are six test units, one
for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_le(columns="a", value=9)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_le()`. All test units passed, and there are no failing test units.

Aside from checking a column against a literal value, we can also use a column name in the
`value=` argument (with the helper function [`col()`](`pointblank.col`) to perform a
column-to-column comparison. For the next example, we'll use `col_vals_le()` to check
whether the values in column `c` are less than values in column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_le(columns="c", value=pb.col("b"))
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are:

- Row 0: `c` is `2` and `b` is `1`.
- Row 4: `c` is `3` and `b` is `2`.

col_vals_eq(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', value: 'float | int | Column', na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Are column data equal to a fixed value or data in another column?

The `col_vals_eq()` validation method checks whether column values in a table are
*equal to* a specified `value=` (the exact comparison used in this function is
`col_val == value`). The `value=` can be specified as a single, literal value or as a column
name given in [`col()`](`pointblank.col`). This validation will operate over the number of
test units that is equal to the number of rows in the table (determined after any `pre=`
mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
value
    The value to compare against. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison. For more information on which types of values are allowed, see the
    *What Can Be Used in `value=`?* section.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
    A `NaN` in a float column also counts as missing on every backend, and that includes a
    `NaN` in a column given to `value=` via [`col()`](`pointblank.col`).
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

What Can Be Used in `value=`?
-----------------------------
The `value=` argument allows for a variety of input types. The most common are:

- a single numeric value
- a single date or datetime value
- A [`col()`](`pointblank.col`) object that represents a column name

When supplying a number as the basis of comparison, keep in mind that all resolved columns
must also be numeric. Should you have columns that are of the date or datetime types, you
can supply a date or datetime value as the `value=` argument. There is flexibility in how
you provide the date or datetime value, as it can be:

- a string-based date or datetime (e.g., `"2023-10-01"`, `"2023-10-01 13:45:30"`, etc.),
  optionally with a UTC offset for a timezone-aware datetime (e.g.,
  `"2023-10-01T13:45:30+05:00"` or `"2023-10-01T13:45:30Z"`)
- a date or datetime object using the `datetime` module (e.g., `datetime.date(2023, 10, 1)`,
  `datetime.datetime(2023, 10, 1, 13, 45, 30)`, etc.)

Finally, when supplying a column name in the `value=` argument, it must be specified within
[`col()`](`pointblank.col`). This is a column-to-column comparison and, crucially, the
columns being compared must be of the same type (e.g., both numeric, both date, etc.).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns=` and `value=col(...)` that are expected to be present in the
transformed table, but may not exist in the table before preprocessing. Regarding the
lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with two numeric columns (`a` and
`b`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 5, 5, 5, 5, 5],
        "b": [5, 5, 5, 6, 5, 4],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are all equal to the value of `5`. We'll determine
if this validation had any failing test units (there are six test units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_eq(columns="a", value=5)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_eq()`. All test units passed, and there are no failing test units.

Aside from checking a column against a literal value, we can also use a column name in the
`value=` argument (with the helper function [`col()`](`pointblank.col`) to perform a
column-to-column comparison. For the next example, we'll use `col_vals_eq()` to check
whether the values in column `a` are equal to the values in column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_eq(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are:

- Row 3: `a` is `5` and `b` is `6`.
- Row 5: `a` is `5` and `b` is `4`.

col_vals_ne(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', value: 'float | int | Column', na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Are column data not equal to a fixed value or data in another column?

The `col_vals_ne()` validation method checks whether column values in a table are
*not equal to* a specified `value=` (the exact comparison used in this function is
`col_val != value`). The `value=` can be specified as a single, literal value or as a column
name given in [`col()`](`pointblank.col`). This validation will operate over the number of
test units that is equal to the number of rows in the table (determined after any `pre=`
mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
value
    The value to compare against. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison. For more information on which types of values are allowed, see the
    *What Can Be Used in `value=`?* section.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
    A `NaN` in a float column also counts as missing on every backend, and that includes a
    `NaN` in a column given to `value=` via [`col()`](`pointblank.col`).
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

What Can Be Used in `value=`?
-----------------------------
The `value=` argument allows for a variety of input types. The most common are:

- a single numeric value
- a single date or datetime value
- A [`col()`](`pointblank.col`) object that represents a column name

When supplying a number as the basis of comparison, keep in mind that all resolved columns
must also be numeric. Should you have columns that are of the date or datetime types, you
can supply a date or datetime value as the `value=` argument. There is flexibility in how
you provide the date or datetime value, as it can be:

- a string-based date or datetime (e.g., `"2023-10-01"`, `"2023-10-01 13:45:30"`, etc.),
  optionally with a UTC offset for a timezone-aware datetime (e.g.,
  `"2023-10-01T13:45:30+05:00"` or `"2023-10-01T13:45:30Z"`)
- a date or datetime object using the `datetime` module (e.g., `datetime.date(2023, 10, 1)`,
  `datetime.datetime(2023, 10, 1, 13, 45, 30)`, etc.)

Finally, when supplying a column name in the `value=` argument, it must be specified within
[`col()`](`pointblank.col`). This is a column-to-column comparison and, crucially, the
columns being compared must be of the same type (e.g., both numeric, both date, etc.).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns=` and `value=col(...)` that are expected to be present in the
transformed table, but may not exist in the table before preprocessing. Regarding the
lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with two numeric columns (`a` and
`b`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 5, 5, 5, 5, 5],
        "b": [5, 6, 3, 6, 5, 8],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are not equal to the value of `3`. We'll determine
if this validation had any failing test units (there are six test units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_ne(columns="a", value=3)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_ne()`. All test units passed, and there are no failing test units.

Aside from checking a column against a literal value, we can also use a column name in the
`value=` argument (with the helper function [`col()`](`pointblank.col`) to perform a
column-to-column comparison. For the next example, we'll use `col_vals_ne()` to check
whether the values in column `a` aren't equal to the values in column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_ne(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are in rows
0 and 4, where `a` is `5` and `b` is `5` in both cases (i.e., they are equal to each other).

col_vals_between(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', left: 'float | int | Column', right: 'float | int | Column', inclusive: 'tuple[bool, bool]' = (True, True), na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Do column data lie between two specified values or data in other columns?

The `col_vals_between()` validation method checks whether column values in a table fall
within a range. The range is specified with three arguments: `left=`, `right=`, and
`inclusive=`. The `left=` and `right=` values specify the lower and upper bounds. These
bounds can be specified as literal values or as column names provided within
[`col()`](`pointblank.col`). The validation will operate over the number of test units that
is equal to the number of rows in the table (determined after any `pre=` mutation has been
applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
left
    The lower bound of the range. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison for this bound. See the *What Can Be Used in `left=` and `right=`?* section
    for details on this.
right
    The upper bound of the range. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison for this bound. See the *What Can Be Used in `left=` and `right=`?* section
    for details on this.
inclusive
    A tuple of two boolean values indicating whether the comparison should be inclusive. The
    position of the boolean values correspond to the `left=` and `right=` values,
    respectively. By default, both values are `True`.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
    A `NaN` in a float column also counts as missing on every backend, and that includes a
    `NaN` in a bound column given to `left=` or `right=` via [`col()`](`pointblank.col`).
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

What Can Be Used in `left=` and `right=`?
-----------------------------------------
The `left=` and `right=` arguments both allow for a variety of input types. The most common
are:

- a single numeric value
- a single date or datetime value
- A [`col()`](`pointblank.col`) object that represents a column in the target table

When supplying a number as the basis of comparison, keep in mind that all resolved columns
must also be numeric. Should you have columns that are of the date or datetime types, you
can supply a date or datetime value within `left=` and `right=`. There is flexibility in how
you provide the date or datetime values for the bounds; they can be:

- string-based dates or datetimes (e.g., `"2023-10-01"`, `"2023-10-01 13:45:30"`, etc.),
  optionally with a UTC offset for timezone-aware datetimes (e.g.,
  `"2023-10-01T13:45:30+05:00"` or `"2023-10-01T13:45:30Z"`)
- date or datetime objects using the `datetime` module (e.g., `datetime.date(2023, 10, 1)`,
`datetime.datetime(2023, 10, 1, 13, 45, 30)`, etc.)

Finally, when supplying a column name in either `left=` or `right=` (or both), it must be
specified within [`col()`](`pointblank.col`). This facilitates column-to-column comparisons
and, crucially, the columns being compared to either/both of the bounds must be of the same
type as the column data (e.g., all numeric, all dates, etc.).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns=` and `left=col(...)`/`right=col(...)` that are expected to be present
in the transformed table, but may not exist in the table before preprocessing. Regarding the
lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three numeric columns (`a`,
`b`, and `c`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [2, 3, 2, 4, 3, 4],
        "b": [5, 6, 1, 6, 8, 5],
        "c": [9, 8, 8, 7, 7, 8],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are all between the fixed boundary values of `1`
and `5`. We'll determine if this validation had any failing test units (there are six test
units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_between(columns="a", left=1, right=5)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_between()`. All test units passed, and there are no failing test units.

Aside from checking a column against two literal values representing the lower and upper
bounds, we can also provide column names to the `left=` and/or `right=` arguments (by using
the helper function [`col()`](`pointblank.col`). In this way, we can perform three
additional comparison types:

1. `left=column`, `right=column`
2. `left=literal`, `right=column`
3. `left=column`, `right=literal`

For the next example, we'll use `col_vals_between()` to check whether the values in column
`b` are between than corresponding values in columns `a` (lower bound) and `c` (upper
bound).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_between(columns="b", left=pb.col("a"), right=pb.col("c"))
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are:

- Row 2: `b` is `1` but the bounds are `2` (`a`) and `8` (`c`).
- Row 4: `b` is `8` but the bounds are `3` (`a`) and `7` (`c`).

col_vals_outside(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', left: 'float | int | Column', right: 'float | int | Column', inclusive: 'tuple[bool, bool]' = (True, True), na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Do column data lie outside of two specified values or data in other columns?

The `col_vals_between()` validation method checks whether column values in a table *do not*
fall within a certain range. The range is specified with three arguments: `left=`, `right=`,
and `inclusive=`. The `left=` and `right=` values specify the lower and upper bounds. These
bounds can be specified as literal values or as column names provided within
[`col()`](`pointblank.col`). The validation will operate over the number of test units that
is equal to the number of rows in the table (determined after any `pre=` mutation has been
applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
left
    The lower bound of the range. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison for this bound. See the *What Can Be Used in `left=` and `right=`?* section
    for details on this.
right
    The upper bound of the range. This can be a single value or a single column name given
    in [`col()`](`pointblank.col`). The latter option allows for a column-to-column
    comparison for this bound. See the *What Can Be Used in `left=` and `right=`?* section
    for details on this.
inclusive
    A tuple of two boolean values indicating whether the comparison should be inclusive. The
    position of the boolean values correspond to the `left=` and `right=` values,
    respectively. By default, both values are `True`.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
    A `NaN` in a float column also counts as missing on every backend, and that includes a
    `NaN` in a bound column given to `left=` or `right=` via [`col()`](`pointblank.col`).
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

What Can Be Used in `left=` and `right=`?
-----------------------------------------
The `left=` and `right=` arguments both allow for a variety of input types. The most common
are:

- a single numeric value
- a single date or datetime value
- A [`col()`](`pointblank.col`) object that represents a column in the target table

When supplying a number as the basis of comparison, keep in mind that all resolved columns
must also be numeric. Should you have columns that are of the date or datetime types, you
can supply a date or datetime value within `left=` and `right=`. There is flexibility in how
you provide the date or datetime values for the bounds; they can be:

- string-based dates or datetimes (e.g., `"2023-10-01"`, `"2023-10-01 13:45:30"`, etc.),
  optionally with a UTC offset for timezone-aware datetimes (e.g.,
  `"2023-10-01T13:45:30+05:00"` or `"2023-10-01T13:45:30Z"`)
- date or datetime objects using the `datetime` module (e.g., `datetime.date(2023, 10, 1)`,
`datetime.datetime(2023, 10, 1, 13, 45, 30)`, etc.)

Finally, when supplying a column name in either `left=` or `right=` (or both), it must be
specified within [`col()`](`pointblank.col`). This facilitates column-to-column comparisons
and, crucially, the columns being compared to either/both of the bounds must be of the same
type as the column data (e.g., all numeric, all dates, etc.).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns=` and `left=col(...)`/`right=col(...)` that are expected to be present
in the transformed table, but may not exist in the table before preprocessing. Regarding the
lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three numeric columns (`a`,
`b`, and `c`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 6, 5, 7, 5, 5],
        "b": [2, 3, 6, 4, 3, 6],
        "c": [9, 8, 8, 9, 9, 7],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are all outside the fixed boundary values of `1`
and `4`. We'll determine if this validation had any failing test units (there are six test
units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_outside(columns="a", left=1, right=4)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_outside()`. All test units passed, and there are no failing test units.

Aside from checking a column against two literal values representing the lower and upper
bounds, we can also provide column names to the `left=` and/or `right=` arguments (by using
the helper function [`col()`](`pointblank.col`). In this way, we can perform three
additional comparison types:

1. `left=column`, `right=column`
2. `left=literal`, `right=column`
3. `left=column`, `right=literal`

For the next example, we'll use `col_vals_outside()` to check whether the values in column
`b` are outside of the range formed by the corresponding values in columns `a` (lower bound)
and `c` (upper bound).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_outside(columns="b", left=pb.col("a"), right=pb.col("c"))
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are:

- Row 2: `b` is `6` and the bounds are `5` (`a`) and `8` (`c`).
- Row 5: `b` is `6` and the bounds are `5` (`a`) and `7` (`c`).

col_vals_in_set(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', set: 'Collection[Any]', missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether column values are in a set of values.

The `col_vals_in_set()` validation method checks whether column values in a table are part
of a specified `set=` of values. This validation will operate over the number of test units
that is equal to the number of rows in the table (determined after any `pre=` mutation has
been applied).

It suits categorical columns (status values, country codes, units, and the like) where
only a known list of values is allowed, and it catches typos and inconsistent spellings
such as `"Active"` next to `"active"`. A null fails the check unless `None` is in the
set. For the reverse check (values that must not appear), use
[`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`); when the allowed
values come from another table, such as a lookup table, use
[`col_vals_in_table()`](`pointblank.Validate.col_vals_in_table`); and for values that
follow a pattern rather than a list, use
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
set
    A collection of values to compare against. Can be a list of values, a Python Enum class,
    or a collection containing Enum instances. When an Enum class is provided, all enum
    values will be used. When a collection contains Enum instances, their values will be
    extracted automatically.
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass as
    well (create it with `null_is_missing=False` to change this).
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Missing Values
--------------
A missing value (`None`, `NA`, or Null) counts as a failing test unit unless `set=` contains
`None`. Alternatively, `missing=MissingSpec(null_is_missing=True)` lets missing values pass
(a [`MissingSpec`](`pointblank.MissingSpec`) can also declare sentinel values, such as
`-99`, that pass as documented missing values).

A `NaN` in a float column counts as missing on every backend (as in comparison methods such
as [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)), so it passes when `set=` contains
`None`.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
a column via `columns=` that is expected to be present in the transformed table, but may not
exist in the table before preprocessing. Regarding the lifetime of the transformed table, it
only exists during the validation step and is not stored in the `Validate` object or used in
subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with two numeric columns (`a` and
`b`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 2, 4, 6, 2, 5],
        "b": [5, 8, 2, 6, 5, 1],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are all in the set of `[2, 3, 4, 5, 6]`. We'll
determine if this validation had any failing test units (there are six test units, one for
each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_in_set(columns="a", set=[2, 3, 4, 5, 6])
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_in_set()`. All test units passed, and there are no failing test units.

Now, let's use that same set of values for a validation on column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_in_set(columns="b", set=[2, 3, 4, 5, 6])
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are for the
column `b` values of `8` and `1`, which are not in the set of `[2, 3, 4, 5, 6]`.

**Using Python Enums**

The `col_vals_in_set()` method also supports Python Enum classes and instances, which can
make validations more readable and maintainable:

```{python}
from enum import Enum

class Color(Enum):
    RED = "red"
    GREEN = "green"
    BLUE = "blue"

# Create a table with color data
tbl_colors = pl.DataFrame({
    "product": ["shirt", "pants", "hat", "shoes"],
    "color": ["red", "blue", "green", "yellow"]
})

# Validate using an Enum class (all enum values are allowed)
validation = (
    pb.Validate(data=tbl_colors)
    .col_vals_in_set(columns="color", set=Color)
    .interrogate()
)

validation
```

This validation will fail for the `"yellow"` value since it's not in the `Color` enum.

You can also use specific Enum instances or mix them with regular values:

```{python}
# Validate using specific Enum instances
validation = (
    pb.Validate(data=tbl_colors)
    .col_vals_in_set(columns="color", set=[Color.RED, Color.BLUE])
    .interrogate()
)

# Mix Enum instances with regular values
validation = (
    pb.Validate(data=tbl_colors)
    .col_vals_in_set(columns="color", set=[Color.RED, Color.BLUE, "yellow"])
    .interrogate()
)

validation
```

In this case, the `"green"` value will cause a failing test unit since it's not part of the
specified set.

col_vals_not_in_set(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', set: 'Collection[Any]', na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether column values are not in a set of values.

The `col_vals_not_in_set()` validation method checks whether column values in a table are
*not* part of a specified `set=` of values. This validation will operate over the number of
test units that is equal to the number of rows in the table (determined after any `pre=`
mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
set
    A collection of values to compare against. Can be a list of values, a Python Enum class,
    or a collection containing Enum instances. When an Enum class is provided, all enum
    values will be used. When a collection contains Enum instances, their values will be
    extracted automatically. It can't contain `None`; see the *Missing Values* section.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring values that encode
    missingness (and whether nulls count as missing). Those values are excluded from the
    check, so they pass. See the *Missing Values* section.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Missing Values
--------------
A missing value (`None`, `NA`, or Null) isn't in the set, but it isn't a valid value either,
so by default it counts as a failing test unit. There are two ways to let missing values
pass: `na_pass=True`, or `missing=MissingSpec(null_is_missing=True)` (a `MissingSpec` can
also declare sentinel values, such as `-99`, that pass as documented missing values). As
missing values are handled with these options, `set=` can't contain `None`.

A `NaN` in a float column counts as missing on every backend (as in comparison methods such
as [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)), so it fails unless `na_pass=True`.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
a column via `columns=` that is expected to be present in the transformed table, but may not
exist in the table before preprocessing. Regarding the lifetime of the transformed table, it
only exists during the validation step and is not stored in the `Validate` object or used in
subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with two numeric columns (`a` and
`b`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [7, 8, 1, 9, 1, 7],
        "b": [1, 8, 2, 6, 9, 1],
    }
)

pb.preview(tbl)
```

Let's validate that none of the values in column `a` are in the set of `[2, 3, 4, 5, 6]`.
We'll determine if this validation had any failing test units (there are six test units, one
for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_not_in_set(columns="a", set=[2, 3, 4, 5, 6])
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_not_in_set()`. All test units passed, and there are no failing test
units.

Now, let's use that same set of values for a validation on column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_not_in_set(columns="b", set=[2, 3, 4, 5, 6])
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are for the
column `b` values of `2` and `6`, both of which are in the set of `[2, 3, 4, 5, 6]`.

**Using Python Enums**

Like `col_vals_in_set()`, this method also supports Python Enum classes and instances:

```{python}
from enum import Enum

class InvalidStatus(Enum):
    DELETED = "deleted"
    ARCHIVED = "archived"

# Create a table with status data
status_table = pl.DataFrame({
    "product": ["widget", "gadget", "tool", "device"],
    "status": ["active", "pending", "deleted", "active"]
})

# Validate that no values are in the invalid status set
validation = (
    pb.Validate(data=status_table)
    .col_vals_not_in_set(columns="status", set=InvalidStatus)
    .interrogate()
)

validation
```

This `"deleted"` value in the `status` column will fail since it matches one of the invalid
statuses in the `InvalidStatus` enum.

col_vals_increasing(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', allow_stationary: 'bool' = False, decreasing_tol: 'float | None' = None, na_pass: 'bool' = False, order_by: 'str | list[str] | None' = None, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Are column data increasing by row?

The `col_vals_increasing()` validation method checks whether column values in a table are
increasing when moving down a table. The column can be numeric or hold dates, datetimes, or
durations; for any other column (e.g., a string column) the step is marked as having an
evaluation error. There are options for allowing missing values in the target column,
allowing stationary phases (where consecutive values don't change), and even one for
allowing decreasing movements up to a certain threshold. This validation will operate over
the number of test units that is equal to the number of rows in the table (determined after
any `pre=` mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
allow_stationary
    An option to allow pauses in increasing values. For example, if the values for the test
    units are `[80, 82, 82, 85, 88]` then the third unit (`82`, appearing a second time)
    would be marked as failing when `allow_stationary` is `False`. Using
    `allow_stationary=True` will result in all the test units in `[80, 82, 82, 85, 88]` to
    be marked as passing.
decreasing_tol
    An optional threshold value that allows for movement of numerical values in the negative
    direction. By default this is `None` but using a numerical value will set the absolute
    threshold of negative travel allowed across numerical test units. Note that setting a
    value here also has the effect of setting `allow_stationary` to `True`. It needs a
    numeric column: with a date, datetime, or duration column, the step is marked as having
    an evaluation error.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values. A `NaN`
    in a float column also counts as missing on every backend.
order_by
    The column (or list of columns) that puts the rows in order before the values are
    compared, in ascending order. If `None` (the default), the table's own row order is
    used. Tables without an inherent row order (e.g., database tables, Ibis tables, or
    PySpark DataFrames) need `order_by=`. See the *Row Order and Missing Values* section.
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units) and, like missing values, don't break the sequence, so only the real values are
    validated, and a note in the validation report lists the codes. By default a
    `MissingSpec` also treats nulls as missing, so nulls pass whatever the value of
    `na_pass=`. Create it with `null_is_missing=False` to leave nulls to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Row Order and Missing Values
----------------------------
Each value is compared with the value before it, so the rows need an order. Polars and
pandas DataFrames (and Polars LazyFrames) have one, which is used unless `order_by=` is
given. Database tables, Ibis tables, and PySpark DataFrames have no inherent row order, so
`order_by=` is needed for them; without it, the step is marked as having an evaluation
error. The results are reported against the table's own rows either way. If the
`order_by=` columns have ties, the order of the tied rows (and so the result) may vary.

A missing value fails unless `na_pass=True`, and it doesn't break the sequence: the next
non-missing value is compared with the last non-missing value before it. For example, with
`[1, 2, None, 5]` the `5` is compared with the `2`. The first non-missing value has nothing
to be compared with, so it always passes. The codes of a `missing=` `MissingSpec` are
skipped in the same way: with `-99` as a code, `[1, 3, -99, 2]` fails the `2`, since it's
compared with the `3`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

For the examples here, we'll use a simple Polars DataFrame with a numeric column (`a`). The
table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5, 6],
        "b": [1, 2, 2, 3, 4, 5],
        "c": [1, 2, 1, 3, 4, 5],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are increasing. We'll determine if this validation
had any failing test units (there are six test units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_increasing(columns="a")
    .interrogate()
)

validation
```

The validation passed as all values in column `a` are increasing. Now let's check column
`b` which has a stationary value:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_increasing(columns="b")
    .interrogate()
)

validation
```

This validation fails at the third row because the value `2` is repeated. If we want to
allow stationary values, we can use `allow_stationary=True`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_increasing(columns="b", allow_stationary=True)
    .interrogate()
)

validation
```

With `allow_stationary=True`, the repeated `2` counts as a passing value, so all six test
units pass. Column `c`, where the value drops from `2` to `1`, would still fail with this
setting; only the `decreasing_tol=` parameter can allow drops like that.

col_vals_decreasing(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', allow_stationary: 'bool' = False, increasing_tol: 'float | None' = None, na_pass: 'bool' = False, order_by: 'str | list[str] | None' = None, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Are column data decreasing by row?

The `col_vals_decreasing()` validation method checks whether column values in a table are
decreasing when moving down a table. The column can be numeric or hold dates, datetimes, or
durations; for any other column (e.g., a string column) the step is marked as having an
evaluation error. There are options for allowing missing values in the target column,
allowing stationary phases (where consecutive values don't change), and even one for
allowing increasing movements up to a certain threshold. This validation will operate over
the number of test units that is equal to the number of rows in the table (determined after
any `pre=` mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
allow_stationary
    An option to allow pauses in decreasing values. For example, if the values for the test
    units are `[88, 85, 85, 82, 80]` then the third unit (`85`, appearing a second time)
    would be marked as failing when `allow_stationary` is `False`. Using
    `allow_stationary=True` will result in all the test units in `[88, 85, 85, 82, 80]` to
    be marked as passing.
increasing_tol
    An optional threshold value that allows for movement of numerical values in the positive
    direction. By default this is `None` but using a numerical value will set the absolute
    threshold of positive travel allowed across numerical test units. Note that setting a
    value here also has the effect of setting `allow_stationary` to `True`. It needs a
    numeric column: with a date, datetime, or duration column, the step is marked as having
    an evaluation error.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values. A `NaN`
    in a float column also counts as missing on every backend.
order_by
    The column (or list of columns) that puts the rows in order before the values are
    compared, in ascending order. If `None` (the default), the table's own row order is
    used. Tables without an inherent row order (e.g., database tables, Ibis tables, or
    PySpark DataFrames) need `order_by=`. See the *Row Order and Missing Values* section.
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units) and, like missing values, don't break the sequence, so only the real values are
    validated, and a note in the validation report lists the codes. By default a
    `MissingSpec` also treats nulls as missing, so nulls pass whatever the value of
    `na_pass=`. Create it with `null_is_missing=False` to leave nulls to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Row Order and Missing Values
----------------------------
Each value is compared with the value before it, so the rows need an order. Polars and
pandas DataFrames (and Polars LazyFrames) have one, which is used unless `order_by=` is
given. Database tables, Ibis tables, and PySpark DataFrames have no inherent row order, so
`order_by=` is needed for them; without it, the step is marked as having an evaluation
error. The results are reported against the table's own rows either way. If the
`order_by=` columns have ties, the order of the tied rows (and so the result) may vary.

A missing value fails unless `na_pass=True`, and it doesn't break the sequence: the next
non-missing value is compared with the last non-missing value before it. For example, with
`[1, 2, None, 5]` the `5` is compared with the `2`. The first non-missing value has nothing
to be compared with, so it always passes. The codes of a `missing=` `MissingSpec` are
skipped in the same way: with `-99` as a code, `[5, 3, -99, 4]` fails the `4`, since it's
compared with the `3`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

For the examples here, we'll use a simple Polars DataFrame with a numeric column (`a`). The
table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [6, 5, 4, 3, 2, 1],
        "b": [5, 4, 4, 3, 2, 1],
        "c": [5, 4, 5, 3, 2, 1],
    }
)

pb.preview(tbl)
```

Let's validate that values in column `a` are decreasing. We'll determine if this validation
had any failing test units (there are six test units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_decreasing(columns="a")
    .interrogate()
)

validation
```

The validation passed as all values in column `a` are decreasing. Now let's check column
`b` which has a stationary value:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_decreasing(columns="b")
    .interrogate()
)

validation
```

This validation fails at the third row because the value `4` is repeated. If we want to
allow stationary values, we can use `allow_stationary=True`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_decreasing(columns="b", allow_stationary=True)
    .interrogate()
)

validation
```

With `allow_stationary=True`, the repeated `4` counts as a passing value, so all six test
units pass. Column `c`, where the value rises from `4` to `5`, would still fail with this
setting; only the `increasing_tol=` parameter can allow rises like that.

col_vals_null(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether values in a column are Null.

The `col_vals_null()` validation method checks whether column values in a table are Null.
This validation will operate over the number of test units that is equal to the number
of rows in the table.

Use it for columns that should be empty: a field that's no longer filled in, or one that
only applies to some rows. For the second case, a `pre=` function that keeps only the
rows in question (e.g., `lambda df: df.filter(pl.col("status") == "active")` for a
`closed_on` column) turns it into a conditional check. The opposite check is
[`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`), and to check the
proportion of nulls in a column rather than every value, use
[`col_pct_null()`](`pointblank.Validate.col_pct_null`).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
a column via `columns=` that is expected to be present in the transformed table, but may not
exist in the table before preprocessing. Regarding the lifetime of the transformed table, it
only exists during the validation step and is not stored in the `Validate` object or used in
subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with two numeric columns (`a` and
`b`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [None, None, None, None],
        "b": [None, 2, None, 9],
    }
).with_columns(pl.col("a").cast(pl.Int64))

pb.preview(tbl)
```

Let's validate that values in column `a` are all Null values. We'll determine if this
validation had any failing test units (there are four test units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_null(columns="a")
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_null()`. All test units passed, and there are no failing test units.

Now, let's use that same set of values for a validation on column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_null(columns="b")
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are for the
two non-Null values in column `b`.

col_vals_not_null(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether values in a column are not Null.

The `col_vals_not_null()` validation method checks whether column values in a table are not
Null. This validation will operate over the number of test units that is equal to the number
of rows in the table.

This is the usual check for required fields such as identifiers, keys, and timestamps.
Most other checks (e.g., [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`) and
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)) already fail null values
unless `na_pass=True` is set, but they count them together with the other failures. A
separate `col_vals_not_null()` step reports the missing values on their own. To check
whole rows across several columns, use
[`rows_complete()`](`pointblank.Validate.rows_complete`); to allow some nulls, check
their proportion with [`col_pct_null()`](`pointblank.Validate.col_pct_null`); and when
missing values are recorded with codes such as `-99`, see
[`col_missing_coded()`](`pointblank.Validate.col_missing_coded`).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
a column via `columns=` that is expected to be present in the transformed table, but may not
exist in the table before preprocessing. Regarding the lifetime of the transformed table, it
only exists during the validation step and is not stored in the `Validate` object or used in
subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with two numeric columns (`a` and
`b`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [4, 7, 2, 8],
        "b": [5, None, 1, None],
    }
)

pb.preview(tbl)
```

Let's validate that none of the values in column `a` are Null values. We'll determine if
this validation had any failing test units (there are four test units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_not_null(columns="a")
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_not_null()`. All test units passed, and there are no failing test units.

Now, let's use that same set of values for a validation on column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_not_null(columns="b")
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are for the
two Null values in column `b`.

col_vals_regex(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', pattern: 'str', na_pass: 'bool' = False, inverse: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether column values match a regular expression pattern.

The `col_vals_regex()` validation method checks whether column values in a table
correspond to a `pattern=` matching expression. This validation will operate over the number
of test units that is equal to the number of rows in the table (determined after any `pre=`
mutation has been applied).

Regular expressions check the format of string values, such as identifiers and codes
(e.g., `"^[A-Z]{3}-[0-9]{4}$"`), or dates and phone numbers stored as strings. A value
passes when the pattern matches anywhere in it, so use `^` and `$` to match the whole
value. With `inverse=True`, values must *not* match, which finds unwanted content such as
leading or trailing spaces. For common formats such as email addresses, URLs, and postal
codes, [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`) has
ready-made checks; to check only string lengths, use
[`col_vals_str_len()`](`pointblank.Validate.col_vals_str_len`); and for a fixed list of
allowed values, [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`) is simpler.

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
pattern
    A regular expression pattern to compare against.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
inverse
    Should the validation step be inverted? If `True`, then the expectation is that column
    values should *not* match the specified `pattern=` regex.
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
a column via `columns=` that is expected to be present in the transformed table, but may not
exist in the table before preprocessing. Regarding the lifetime of the transformed table, it
only exists during the validation step and is not stored in the `Validate` object or used in
subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with two string columns (`a` and
`b`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": ["rb-0343", "ra-0232", "ry-0954", "rc-1343"],
        "b": ["ra-0628", "ra-583", "rya-0826", "rb-0735"],
    }
)

pb.preview(tbl)
```

Let's validate that all of the values in column `a` match a particular regex pattern. We'll
determine if this validation had any failing test units (there are four test units, one for
each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_regex(columns="a", pattern=r"r[a-z]-[0-9]{4}")
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_regex()`. All test units passed, and there are no failing test units.

Now, let's use the same regex for a validation on column `b`.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_regex(columns="b", pattern=r"r[a-z]-[0-9]{4}")
    .interrogate()
)

validation
```

The validation table reports two failing test units. The specific failing cases are for the
string values of rows 1 and 2 in column `b`.

col_vals_within_spec(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', spec: 'str', na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether column values fit within a specification.

The `col_vals_within_spec()` validation method checks whether column values in a table
correspond to a specification (`spec=`) type (details of which are available in the
*Specifications* section). Specifications include common data types like email addresses,
URLs, postal codes, vehicle identification numbers (VINs), International Bank Account
Numbers (IBANs), and more. This validation will operate over the number of test units that
is equal to the number of rows in the table.

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
spec
    A specification string for defining the specification type. Examples are `"email"`,
    `"url"`, and `"postal_code[USA]"`. See the *Specifications* section for all available
    options.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Specifications
--------------
A specification type must be used with the `spec=` argument. This is a string-based keyword
that corresponds to the type of data in the specified columns. The following keywords can
be used:

- `"isbn"`: The International Standard Book Number (ISBN) is a unique numerical identifier
  for books. This keyword validates both 10-digit and 13-digit ISBNs.

- `"vin"`: A vehicle identification number (VIN) is a unique code used by the automotive
  industry to identify individual motor vehicles.

- `"postal_code[<country_code>]"`: A postal code (also known as postcodes, PIN, or ZIP
  codes) is a series of letters, digits, or both included in a postal address. Because the
  coding varies by country, a country code in either the 2-letter (ISO 3166-1 alpha-2) or
  3-letter (ISO 3166-1 alpha-3) format needs to be supplied (e.g., `"postal_code[US]"` or
  `"postal_code[USA]"`). The keyword alias `"zip"` can be used for US ZIP codes.

- `"credit_card"`: A credit card number can be validated across a variety of issuers. The
  validation uses the Luhn algorithm.

- `"iban[<country_code>]"`: The International Bank Account Number (IBAN) is a system of
  identifying bank accounts across countries. Because the length and coding varies by
  country, a country code needs to be supplied (e.g., `"iban[DE]"` or `"iban[DEU]"`).

- `"swift"`: Business Identifier Codes (also known as SWIFT-BIC, BIC, or SWIFT code) are
  unique identifiers for financial and non-financial institutions.

- `"phone"`, `"email"`, `"url"`, `"ipv4"`, `"ipv6"`, `"mac"`: Phone numbers, email
  addresses, Internet URLs, IPv4 or IPv6 addresses, and MAC addresses can be validated with
  their respective keywords.

Only a single `spec=` value should be provided per function call.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
a column via `columns=` that is expected to be present in the transformed table, but may not
exist in the table before preprocessing. Regarding the lifetime of the transformed table, it
only exists during the validation step and is not stored in the `Validate` object or used in
subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

For the examples here, we'll use a simple Polars DataFrame with an email column. The table
is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "email": [
            "user@example.com",
            "admin@test.org",
            "invalid-email",
            "contact@company.co.uk",
        ],
    }
)

pb.preview(tbl)
```

Let's validate that all of the values in the `email` column are valid email addresses.
We'll determine if this validation had any failing test units (there are four test units,
one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_within_spec(columns="email", spec="email")
    .interrogate()
)

validation
```

The validation table shows that one test unit failed (the invalid email address in row 3).

col_vals_str_len(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', min_val: 'int | None' = None, max_val: 'int | None' = None, na_pass: 'bool' = False, missing: 'MissingSpec | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether the length of string values falls within specified bounds.

The `col_vals_str_len()` validation method checks whether the character length of string
values in a column meets the specified minimum and/or maximum bounds. This validation will
operate over the number of test units that is equal to the number of rows in the table
(determined after any `pre=` mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
min_val
    The minimum acceptable string length (inclusive). If `None`, no lower bound is applied.
    At least one of `min_val=` or `max_val=` must be provided.
max_val
    The maximum acceptable string length (inclusive). If `None`, no upper bound is applied.
    At least one of `min_val=` or `max_val=` must be provided.
na_pass
    Should any encountered None, NA, or Null values be considered as passing test units? By
    default, this is `False`. Set to `True` to pass test units with missing values.
missing
    An optional [`MissingSpec`](`pointblank.MissingSpec`) declaring the codes that stand in
    for missing values in the column (such as `-99` for a survey question that was refused).
    Rows holding one of these codes are excluded from the check (they count as passing test
    units), so only the real values are validated, and a note in the validation report lists
    the codes. By default a `MissingSpec` also treats nulls as missing, so nulls pass
    whatever the value of `na_pass=`. Create it with `null_is_missing=False` to leave nulls
    to `na_pass=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Raises
------
ValueError
    If neither `min_val=` nor `max_val=` is provided.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with a string column (`a`).
The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": ["short", "medium str", "a very long string value", "ok"],
    }
)

pb.preview(tbl)
```

Let's validate that all values in column `a` have a string length between 2 and 10
characters.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_str_len(columns="a", min_val=2, max_val=10)
    .interrogate()
)

validation
```

The validation table shows one failing test unit: `"a very long string value"` (24 chars)
exceeds `max_val=10`. The value `"ok"` (2 chars) passes since the lower bound is inclusive.

We can also validate with only a minimum length:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_str_len(columns="a", min_val=3)
    .interrogate()
)

validation
```

Without an upper bound, the long string now passes, but `"ok"` fails because its two
characters fall short of `min_val=3`. So there's again one failing test unit out of four.

col_vals_in_table(self, columns: 'str | list[str]', ref_table: 'Any', ref_column: 'str | list[str]', na_pass: 'bool' = False, pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate that column values exist in a reference table (referential integrity).

The `col_vals_in_table()` validation method checks whether each value (or composite key) in
the specified column(s) of this table exists in the corresponding column(s) of a reference
table. This is a referential integrity check: it catches orphaned foreign keys, broken
references, and values that fall outside a controlled vocabulary maintained in another
table.

Each row is a test unit. Rows whose key values are found in the reference table pass. Rows
with orphaned values fail. Failing rows are extractable via `get_data_extracts()` just like
any other column-level validation.

Cross-backend matching is supported: the reference table can be from a different backend
(e.g., a Polars DataFrame checked against a DuckDB table, or a MySQL table checked against a
SQLite table). The lighter table is automatically materialized to match the heavier table's
backend before comparison.

Parameters
----------
columns
    The column (or list of columns for composite keys) in this table that should reference
    the other table. For composite keys, supply a list of column names.
ref_table
    The reference table containing the valid values. Can be a DataFrame, a database table,
    a file path (e.g., to a CSV or Parquet file), a database connection string (e.g.,
    `"duckdb:///path/to/db.ddb::table_name"`), or a callable that returns one of these. A
    path or connection string is loaded at interrogation time, as with
    [`tbl_match()`](`pointblank.Validate.tbl_match`); if the reference table can't be
    loaded, the step is marked as having an evaluation error.
ref_column
    The column (or list of columns for composite keys) in the reference table to check
    against. Must have the same length as ``columns``.
na_pass
    If `True`, rows where the key column(s) are null will pass. If `False` (default), null
    key values are treated as failing (not found in reference).
pre
    An optional preprocessing function to apply to the data table before validation.
thresholds
    Failure-condition thresholds for this step (overrides global thresholds).
actions
    Actions to take when thresholds are exceeded.
brief
    A brief description for this validation step.
active
    Whether this step is active (can be a bool or callable returning bool).
dimension
    The data quality dimension tag for this step.

Returns
-------
Validate
    The `Validate` object with this step added (for method chaining).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Check that every `customer_id` in orders exists in the customers table:

```{python}
import pointblank as pb
import polars as pl

customers = pl.DataFrame({"id": [1, 2, 3, 4, 5]})
orders = pl.DataFrame({
    "order_id": [101, 102, 103, 104],
    "customer_id": [1, 2, 3, 99],
})

validation = (
    pb.Validate(data=orders)
    .col_vals_in_table(
        columns="customer_id",
        ref_table=customers,
        ref_column="id",
    )
    .interrogate()
)

validation
```

The last row fails because `customer_id=99` does not exist in the customers table.

Composite foreign keys are also supported:

```{python}
catalog = pl.DataFrame({
    "region": ["US", "US", "EU"],
    "sku": ["A1", "B2", "A1"],
})

orders = pl.DataFrame({
    "region": ["US", "EU", "US"],
    "sku": ["A1", "A1", "C3"],
})

validation = (
    pb.Validate(data=orders)
    .col_vals_in_table(
        columns=["region", "sku"],
        ref_table=catalog,
        ref_column=["region", "sku"],
    )
    .interrogate()
)

validation
```

With composite keys, each row's combination of values must appear as a row in the reference
table. The third row fails because `("US", "C3")` isn't in the catalog. The second row,
`("EU", "A1")`, passes because the catalog has that exact pair, even though `"A1"` is also
listed under `"US"`.

col_vals_expr(self, expr: 'Any', pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate column values using a custom expression.

The `col_vals_expr()` validation method checks whether column values in a table satisfy a
custom `expr=` expression. This validation will operate over the number of test units that
is equal to the number of rows in the table (determined after any `pre=` mutation has been
applied).

Use it when no other validation method expresses the rule, such as a condition that
involves several columns (e.g., `(pl.col("end") - pl.col("start")) <= 5`) or one that
combines conditions. For a simple comparison of two columns, the comparison methods take
a column as the value (e.g., `col_vals_ge(columns="end", value=pb.col("start"))`), which
also names both columns in the report. To require that each row passes several
expressions at once, use [`conjointly()`](`pointblank.Validate.conjointly`), and for
custom logic that doesn't fit a row-by-row expression (such as a check of the whole
table), use [`specially()`](`pointblank.Validate.specially`).

Parameters
----------
expr
    A column expression that will evaluate each row in the table, returning a boolean value
    per table row. A Narwhals column expression (e.g., `nw.col("a") > 1`) works with every
    table type. Otherwise, use an expression from the table's own library: a Polars
    expression for a Polars table, an Ibis expression for an Ibis table, or a PySpark
    column for a PySpark DataFrame. A function (such as a lambda) can also be given; it
    receives the table and should return one of those expressions or a boolean Series
    (e.g., `lambda df: df["a"] > 1` for a pandas DataFrame). For Polars, Narwhals, and
    Ibis expressions, the columns used in the expression are shown in the validation
    report, as is any name given to the expression (e.g., through `.alias("a_is_int")`),
    which helps to tell apart several `col_vals_expr()` steps. The expression as written
    in the code (when its source text is available, which requires Python 3.11 or later)
    appears in a note in the report footer.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Missing Values and Evaluation Errors
------------------------------------
A row where the expression evaluates to a missing value (e.g., `pl.col("a") > 1` when `a`
is null) counts as a failing test unit. To let such rows pass, allow for them in the
expression itself, as in `(pl.col("a") > 1) | pl.col("a").is_null()`.

If the expression can't be evaluated against the table (e.g., it refers to a column that
doesn't exist, comes from a library that doesn't match the table, or doesn't produce boolean
values), the step is marked as having an evaluation error and a note explains why. The
other validation steps are still interrogated.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Regarding the lifetime of the
transformed table, it only exists during the validation step and is not stored in the
`Validate` object or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 1, 7, 8, 6],
        "b": [0, 0, 0, 1, 1, 1],
        "c": [0.5, 0.3, 0.8, 1.4, 1.9, 1.2],
    }
)

pb.preview(tbl)
```

Let's validate that the values in column `a` are all integers. We'll determine if this
validation had any failing test units (there are six test units, one for each row).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_expr(expr=pl.col("a") % 1 == 0)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_vals_expr()`. All test units passed, with no failing test units.

col_exists(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether one or more columns exist in the table.

The `col_exists()` method checks whether one or more columns exist in the target table. The
only requirement is specification of the column names. Each validation step or expectation
will operate over a single test unit, which is whether the column exists or not.

It's worth adding when a column might be missing, as in data from a source that may change
its layout. Without it, a step on a missing column can't be evaluated: it's marked with an
evaluation error and has no test units, so it can't reach a threshold. With
`col_exists()`, the missing column is a failing test unit, which thresholds and actions
can react to. To check the full set of columns along with their types, use
[`col_schema_match()`](`pointblank.Validate.col_schema_match`), and to check only the
number of columns, use [`col_count_match()`](`pointblank.Validate.col_count_match`). To
skip the steps on a column when it's absent, give them
`active=pb.has_columns("<column>")` (see [`has_columns()`](`pointblank.has_columns`)).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with a string columns (`a`) and a
numeric column (`b`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": ["apple", "banana", "cherry", "date"],
        "b": [1, 6, 3, 5],
    }
)

pb.preview(tbl)
```

Let's validate that the columns `a` and `b` actually exist in the table. We'll determine if
this validation had any failing test units (each validation will have a single test unit).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_exists(columns=["a", "b"])
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows two entries (one check per column) generated by the
`col_exists()` validation step. Both steps passed since both columns provided in `columns=`
are present in the table.

Now, let's check for the existence of a different set of columns.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_exists(columns=["b", "c"])
    .interrogate()
)

validation
```

The validation table reports one passing validation step (the check for column `b`) and one
failing validation step (the check for column `c`, which doesn't exist).

col_pct_null(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', p: 'float', tol: 'Tolerance' = 0, thresholds: 'int | float | None | bool | tuple | dict | Thresholds' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether a column has a specific percentage of Null values.

The `col_pct_null()` validation method checks whether the percentage of Null values in a
column matches a specified percentage `p=` (within an optional tolerance `tol=`). This
validation operates at the column level, generating a single validation step per column that
passes or fails based on whether the actual percentage of Null values falls within the
acceptable range defined by `p ± tol`.

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
p
    The expected percentage of Null values in the column, expressed as a decimal between
    `0.0` and `1.0`. For example, `p=0.5` means 50% of values should be Null.
tol
    The tolerance allowed when comparing the actual percentage of Null values to the
    expected percentage `p=`. The validation passes if the actual percentage falls within
    the range `[p - tol, p + tol]`. Default is `0`, meaning an exact match is required. See
    the *Tolerance* section for details on all supported formats (absolute, relative,
    symmetric, and asymmetric bounds).
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Tolerance
---------
The `tol=` parameter accepts several different formats to specify the acceptable deviation
from the expected percentage `p=`. The tolerance can be expressed as:

1. *single integer* (absolute tolerance): the exact number of test units that can deviate.
For example, `tol=2` means the actual count can differ from the expected count by up to 2
units in either direction.

2. *single float between 0 and 1* (relative tolerance): a proportion of the expected
count. For example, if the expected count is 50 and `tol=0.1`, the acceptable range is
45 to 55 (50 ± 10% of 50 = 50 ± 5).

3. *tuple of two integers* (absolute bounds): explicitly specify the lower and upper
bounds as absolute deviations. For example, `tol=(1, 3)` means the actual count can be
1 unit below or 3 units above the expected count.

4. *tuple of two floats between 0 and 1* (relative bounds): explicitly specify the lower
and upper bounds as proportional deviations. For example, `tol=(0.05, 0.15)` means the
lower bound is 5% below and the upper bound is 15% above the expected count.

When using a single value (integer or float), the tolerance is applied symmetrically in both
directions. When using a tuple, you can specify asymmetric tolerances where the lower and
upper bounds differ.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three columns (`a`, `b`,
and `c`) that have different percentages of Null values. The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5, 6, 7, 8],
        "b": [1, None, 3, None, 5, None, 7, None],
        "c": [None, None, None, None, None, None, 1, 2],
    }
)

pb.preview(tbl)
```

Let's validate that column `a` has 0% Null values (i.e., no Null values at all).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_pct_null(columns="a", p=0.0)
    .interrogate()
)

validation
```

Printing the `validation` object shows the validation table in an HTML viewing environment.
The validation table shows the single entry that corresponds to the validation step created
by using `col_pct_null()`. The validation passed since column `a` has no Null values.

Now, let's check that column `b` has exactly 50% Null values.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_pct_null(columns="b", p=0.5)
    .interrogate()
)

validation
```

This validation also passes, as column `b` has exactly 4 out of 8 values as Null (50%).

Finally, let's validate column `c` with a tolerance. Column `c` has 75% Null values, so
we'll check if it's approximately 70% Null with a tolerance of 10%.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_pct_null(columns="c", p=0.70, tol=0.10)
    .interrogate()
)

validation
```

This validation passes because the actual percentage (75%) falls within the acceptable
range of 60% to 80% (70% ± 10%).

The `tol=` parameter supports multiple formats to express tolerance. Let's explore all the
different ways to specify tolerance using column `b`, which has exactly 50% Null values
(4 out of 8 values).

*Using an absolute tolerance (integer)*: Specify the exact number of rows that can
deviate. With `tol=1`, we allow the count to differ by 1 row in either direction.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_pct_null(columns="b", p=0.375, tol=1)  # Expect 3 nulls, allow ±1 (range: 2-4)
    .interrogate()
)

validation
```

This passes because column `b` has 4 Null values, which falls within the acceptable range
of 2 to 4 (3 ± 1).

*Using a relative tolerance (float)*: Specify the tolerance as a proportion of the
expected count. With `tol=0.25`, we allow a 25% deviation from the expected count.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_pct_null(columns="b", p=0.375, tol=0.25)  # Expect 3 nulls, allow ±25% (range: 2.25-3.75)
    .interrogate()
)

validation
```

This passes because 4 Null values falls within the acceptable range (3 ± 0.75 calculates
to 2.25 to 3.75, which rounds down to 2 to 3 rows).

*Using asymmetric absolute bounds (tuple of integers)*: Specify different lower and
upper bounds as absolute values. With `tol=(0, 2)`, we allow no deviation below but up
to 2 rows above the expected count.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_pct_null(columns="b", p=0.25, tol=(0, 2))  # Expect 2 Nulls, allow +0/-2 (range: 2-4)
    .interrogate()
)

validation
```

This passes because 4 Null values falls within the acceptable range of 2 to 4.

*Using asymmetric relative bounds (tuple of floats)*: Specify different lower and upper
bounds as proportions. With `tol=(0.1, 0.3)`, we allow 10% below and 30% above the
expected count.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_pct_null(columns="b", p=0.375, tol=(0.1, 0.3))  # Expect 3 Nulls, allow -10%/+30%
    .interrogate()
)

validation
```

This passes because 4 Null values falls within the acceptable range (3 - 0.3 to 3 + 0.9
calculates to 2.7 to 3.9, which rounds down to 2 to 3 rows).

col_pct_missing(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', missing: 'MissingSpec', max_pct: 'float', reason: 'str | None' = None, category: 'str | None' = None, thresholds: 'int | float | None | bool | tuple | dict | Thresholds' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate that the percentage of *structured* missing values stays within a limit.

The `col_pct_missing()` validation method checks whether the percentage of missing values
in a column is at most `max_pct=`. Unlike [`col_pct_null()`](`pointblank.Validate.col_pct_null`),
which only considers actual null values, this method uses a
[`MissingSpec`](`pointblank.MissingSpec`) to define which values count as missing: declared
sentinel values (e.g., `-99` for `"refused"`) and, when `null_is_missing=True`, actual null
values. This validation operates at the column level, generating a single validation step
per column that passes when the missing percentage does not exceed `max_pct=`.

You can narrow the check to a single reason (via `reason=`) or a category of reasons (via
`category=`), making it possible to assert things like "at most 10% of values were refused"
or "at most 15% are item nonresponse".

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
missing
    A [`MissingSpec`](`pointblank.MissingSpec`) describing the sentinel values (and their
    reasons) that encode missingness for this column.
max_pct
    The maximum allowable percentage of missing values, expressed as a decimal between
    `0.0` and `1.0`. For example, `max_pct=0.20` means at most 20% of values may be missing.
reason
    If provided, only count missing values whose reason matches this label. Cannot be
    combined with `category=`.
category
    If provided, only count missing values whose reason falls in this category (as defined
    in `MissingSpec.categories`). Cannot be combined with `reason=`.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged).
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Survey data often encodes missingness with sentinel values rather than nulls. Here, the
`age` column uses `-99` (`"not_asked"`), `-98` (`"refused"`), and `-97` (`"dont_know"`):

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {"age": [34, -98, 41, -99, 29, -98, 55, 38]},
)

age_missing = pb.MissingSpec(
    reasons={-99: "not_asked", -98: "refused", -97: "dont_know"},
    categories={"item_nonresponse": ["refused", "dont_know"]},
)

validation = (
    pb.Validate(data=tbl)
    .col_pct_missing(columns="age", missing=age_missing, max_pct=0.5)
    .col_pct_missing(columns="age", missing=age_missing, reason="refused", max_pct=0.30)
    .interrogate()
)

validation
```

Both steps pass. Three of the eight values are missing codes, so the overall missing share
of 37.5% is within `max_pct=0.5`. The `reason="refused"` step counts only the two `-98`
values, and their share of 25% is within `max_pct=0.30`.

col_missing_coded(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', missing: 'MissingSpec', pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate that all missing values in a column are *coded* (no uncoded nulls).

The `col_missing_coded()` validation method checks that every absent value in a column is
expressed with an explicit missing-value code, rather than a raw null. Under the structured
missingness model (see [`MissingSpec`](`pointblank.MissingSpec`)), every absence should
carry a *reason* — encoded as a sentinel value such as `-99` for `"not_asked"`. A raw null
represents *uncoded* (unknown) missingness, so this validation treats raw nulls as failing
test units while declared sentinel values and real values pass.

This validation operates over the number of test units equal to the number of rows in the
table (determined after any `pre=` mutation has been applied).

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns. If
    multiple columns are supplied or resolved, there will be a separate validation step
    generated for each column.
missing
    A [`MissingSpec`](`pointblank.MissingSpec`) describing the sentinel values (and their
    reasons) that encode missingness for this column. The spec documents which codes are
    considered valid expressions of missingness.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment).
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`.
actions
    Optional actions to take when the validation step(s) meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged).
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column. Providing a single column name results in a separate validation step for each unique
value in that column; a tuple of `(column, values)` restricts segmentation to the listed
values. The segmentation is performed after any `pre=` preprocessing.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Here, the `age` column codes its missingness with sentinel values, except for one row that
has a raw null (an uncoded absence):

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame({"age": [34, -98, 41, None, 29, -99, 55, 38]})

age_missing = pb.MissingSpec(
    reasons={-99: "not_asked", -98: "refused", -97: "dont_know"},
)

validation = (
    pb.Validate(data=tbl)
    .col_missing_coded(columns="age", missing=age_missing)
    .interrogate()
)

validation
```

The validation reports a single failing test unit: the row where `age` is a raw null, which
represents missingness without a documented reason.

col_missing_only_coded(self, columns: 'str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals', missing: 'MissingSpec', allowed: 'Collection[Any] | None' = None, min_val: 'float | int | None' = None, max_val: 'float | int | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate that a column contains only documented codes and legitimate values.

The `col_missing_only_coded()` method checks that every value in a column is *accounted
for*: it is either a declared missing-value code (a sentinel in the
[`MissingSpec`](`pointblank.MissingSpec`), or a null when `null_is_missing=True`), or a
legitimate "real" value. Legitimate real values are defined by `allowed=` (an explicit set)
and/or a `[min_val, max_val]` range. Any value that is neither a documented code nor a
legitimate real value is flagged — this catches *undocumented* sentinel codes (e.g., a
stray `-95`) that aren't part of the spec.

At least one of `allowed=`, `min_val=`, or `max_val=` must be provided so that legitimate
real values can be distinguished from undocumented codes. This validation operates over the
number of test units equal to the number of rows in the table.

Parameters
----------
columns
    A single column or a list of columns to validate. Can also use
    [`col()`](`pointblank.col`) with column selectors to specify one or more columns.
missing
    A [`MissingSpec`](`pointblank.MissingSpec`) declaring the documented sentinel codes.
allowed
    An explicit set of legitimate real values. A value in this set passes. Can be combined
    with `min_val=`/`max_val=` (a value passes if it satisfies either constraint).
min_val
    Lower bound (inclusive) of the legitimate real-value range.
max_val
    Upper bound (inclusive) of the legitimate real-value range.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment).
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged).
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column. Providing a single column name results in a separate validation step for each unique
value in that column; a tuple of `(column, values)` restricts segmentation to the listed
values. The segmentation is performed after any `pre=` preprocessing.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
The `age` column should contain real ages in `[0, 120]` or the documented codes `-99`/`-98`.
The value `-95` is an *undocumented* code and should be flagged:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame({"age": [34, -98, 41, -95, 29, -99, 55]})

age_missing = pb.MissingSpec(reasons={-99: "not_asked", -98: "refused"})

validation = (
    pb.Validate(data=tbl)
    .col_missing_only_coded(columns="age", missing=age_missing, min_val=0, max_val=120)
    .interrogate()
)

validation
```

The validation reports one failing test unit: the row where `age` is `-95`, which is
neither a real age in range nor a declared sentinel.

col_missing_consistent(self, columns: 'list[str]', missing: 'MissingSpec', when_reason: 'str', pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate that related columns share a consistent missingness pattern for a given reason.

The `col_missing_consistent()` method checks that, across a set of related columns, the
"missing for a specific reason" status is *consistent*: for each row, either *none* of the
columns are missing for `when_reason=`, or *all* of them are. This is useful for structured
survey or clinical data where a skip pattern should propagate across related fields — for
example, if a question wasn't asked (`"not_asked"`) then all of its dependent fields should
also be coded `"not_asked"`.

A value is considered "missing for the reason" when it is one of the sentinel values mapped
to `when_reason=` in the [`MissingSpec`](`pointblank.MissingSpec`) (and, when the reason is
the spec's `null_reason` and `null_is_missing=True`, an actual null). This validation
operates over the number of test units equal to the number of rows in the table. A row fails
when some — but not all — of the columns are missing for the given reason.

Parameters
----------
columns
    A list of related columns to check for consistent missingness.
missing
    A [`MissingSpec`](`pointblank.MissingSpec`) describing the sentinel values and their
    reasons for the columns.
when_reason
    The reason label whose presence should be consistent across `columns=`. If one column
    in a row is missing for this reason, all of them should be.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment).
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged).
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column. Providing a single column name results in a separate validation step for each unique
value in that column; a tuple of `(column, values)` restricts segmentation to the listed
values. The segmentation is performed after any `pre=` preprocessing.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Here, `income_source` and `income_amount` should both be coded `"not_asked"` (`-99`) together
when the income question wasn't asked. The last row is inconsistent — only one field is
coded `-99`:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "income_source": [1, -99, 2, -99],
        "income_amount": [50000, -99, 42000, 38000],
    }
)

income_missing = pb.MissingSpec(reasons={-99: "not_asked", -98: "refused"})

validation = (
    pb.Validate(data=tbl)
    .col_missing_consistent(
        columns=["income_source", "income_amount"],
        missing=income_missing,
        when_reason="not_asked",
    )
    .interrogate()
)

validation
```

The validation reports one failing test unit: the final row, where `income_source` is coded
`-99` (`"not_asked"`) but `income_amount` is a real value.

rows_distinct(self, columns_subset: 'str | list[str] | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether rows in the table are distinct.

The `rows_distinct()` method checks whether rows in the table are distinct. This validation
will operate over the number of test units that is equal to the number of rows in the table
(determined after any `pre=` mutation has been applied).

Duplicate rows often come from loading the same data twice or from a join that matched
more rows than expected. With `columns_subset=`, only the given columns are compared, which
checks that a key (a single column such as an ID, or a combination of columns) is unique.
Every copy of a duplicated row is a failing test unit, so two rows with the same key give
two failures. To check that rows have no missing values instead, use
[`rows_complete()`](`pointblank.Validate.rows_complete`).

Parameters
----------
columns_subset
    A single column or a list of columns to use as a subset for the distinct comparison.
    If `None`, then all columns in the table will be used for the comparison. If multiple
    columns are supplied, the distinct comparison will be made over the combination of
    values in those columns.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns_subset=` that are expected to be present in the transformed table, but
may not exist in the table before preprocessing. Regarding the lifetime of the transformed
table, it only exists during the validation step and is not stored in the `Validate` object
or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three string columns
(`col_1`, `col_2`, and `col_3`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "col_1": ["a", "b", "c", "d"],
        "col_2": ["a", "a", "c", "d"],
        "col_3": ["a", "a", "d", "e"],
    }
)

pb.preview(tbl)
```

Let's validate that the rows in the table are distinct with `rows_distinct()`. We'll
determine if this validation had any failing test units (there are four test units, one for
each row). A failing test units means that a given row is not distinct from every other row.

```{python}
validation = (
    pb.Validate(data=tbl)
    .rows_distinct()
    .interrogate()
)

validation
```

From this validation table we see that there are no failing test units. All rows in the
table are distinct from one another.

We can also use a subset of columns to determine distinctness. Let's specify the subset
using columns `col_2` and `col_3` for the next validation.

```{python}
validation = (
    pb.Validate(data=tbl)
    .rows_distinct(columns_subset=["col_2", "col_3"])
    .interrogate()
)

validation
```

The validation table reports two failing test units. The first and second rows are
duplicated when considering only the values in columns `col_2` and `col_3`. There's only
one set of duplicates but there are two failing test units since each row is compared to all
others.

rows_complete(self, columns_subset: 'str | list[str] | None' = None, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether row data are complete by having no missing values.

The `rows_complete()` method checks whether rows in the table are complete. Completeness
of a row means that there are no missing values within the row. This validation will operate
over the number of test units that is equal to the number of rows in the table (determined
after any `pre=` mutation has been applied). A subset of columns can be specified for the
completeness check. If no subset is provided, all columns in the table will be used.

Parameters
----------
columns_subset
    A single column or a list of columns to use as a subset for the completeness check. If
    `None` (the default), then all columns in the table will be used.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list). Read the *Segmentation* section for usage information.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that you can refer to
columns via `columns_subset=` that are expected to be present in the transformed table, but
may not exist in the table before preprocessing. Regarding the lifetime of the transformed
table, it only exists during the validation step and is not stored in the `Validate` object
or used in subsequent validation steps.

Segmentation
------------
The `segments=` argument allows for the segmentation of a validation step into multiple
segments. This is useful for applying the same validation step to different subsets of the
data. The segmentation can be done based on a single column or specific fields within a
column.

Providing a single column name will result in a separate validation step for each unique
value in that column. For example, if you have a column called `"region"` with values
`"North"`, `"South"`, and `"East"`, the validation step will be applied separately to each
region.

Alternatively, you can provide a tuple that specifies a column name and its corresponding
values to segment on. For example, if you have a column called `"date"` and you want to
segment on only specific dates, you can provide a tuple like
`("date", ["2023-01-01", "2023-01-02"])`. Any other values in the column will be disregarded
(i.e., no validation steps will be created for them).

A list with a combination of column names and tuples can be provided as well. This allows
for more complex segmentation scenarios. The following inputs are both valid:

```
# Segments from all unique values in the `region` column
# and specific dates in the `date` column
segments=["region", ("date", ["2023-01-01", "2023-01-02"])]

# Segments from all unique values in the `region` and `date` columns
segments=["region", "date"]
```

The segmentation is performed during interrogation, and the resulting validation steps will
be numbered sequentially. Each segment will have its own validation step, and the results
will be reported separately. This allows for a more granular analysis of the data and helps
identify issues within specific segments. If no segments are found (e.g., the table has no
rows), a single step is kept with a note explaining why, so the steps after it keep their
numbers.

Importantly, the segmentation process will be performed after any preprocessing of the data
table. Because of this, one can conceivably use the `pre=` argument to generate a column
that can be used for segmentation. For example, you could create a new column called
`"segment"` through use of `pre=` and then use that column for segmentation.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three string columns
(`col_1`, `col_2`, and `col_3`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "col_1": ["a", None, "c", "d"],
        "col_2": ["a", "a", "c", None],
        "col_3": ["a", "a", "d", None],
    }
)

pb.preview(tbl)
```

Let's validate that the rows in the table are complete with `rows_complete()`. We'll
determine if this validation had any failing test units (there are four test units, one for
each row). A failing test units means that a given row is not complete (i.e., has at least
one missing value).

```{python}
validation = (
    pb.Validate(data=tbl)
    .rows_complete()
    .interrogate()
)

validation
```

From this validation table we see that there are two failing test units. This is because
two rows in the table have at least one missing value (the second row and the last row).

We can also use a subset of columns to determine completeness. Let's specify the subset
using columns `col_2` and `col_3` for the next validation.

```{python}
validation = (
    pb.Validate(data=tbl)
    .rows_complete(columns_subset=["col_2", "col_3"])
    .interrogate()
)

validation
```

The validation table reports a single failing test units. The last row contains missing
values in both the `col_2` and `col_3` columns.
others.

col_schema_match(self, schema: 'Schema', complete: 'bool' = True, in_order: 'bool' = True, case_sensitive_colnames: 'bool' = True, case_sensitive_dtypes: 'bool' = True, full_match_dtypes: 'bool' = True, pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Do columns in the table (and their types) match a predefined schema?

The `col_schema_match()` method works in conjunction with an object generated by the
[`Schema`](`pointblank.Schema`) class. That class object is the expectation for the actual
schema of the target table. The validation step operates over a single test unit, which is
whether the schema matches that of the table (within the constraints enforced by the
`complete=`, and `in_order=` options). Every column in the schema must be present in the
table, so a schema that names the same column more than once never matches.

Parameters
----------
schema
    A `Schema` object that represents the expected schema of the table. This object is
    generated by the [`Schema`](`pointblank.Schema`) class.
complete
    Should the schema match be complete? If `True`, then the schema must account for every
    column in the target table. If `False`, then the table can have additional columns not
    in the schema (i.e., the schema is a subset of the target table's columns).
in_order
    Should the schema match be in order? If `True`, then the columns in the schema must
    appear in the same order as they do in the target table. If `False`, then the order of
    columns in the schema and the target table can differ.
case_sensitive_colnames
    Should the schema match be case-sensitive with regard to column names? If `True`, then
    the column names in the schema and the target table must match exactly. If `False`, then
    the column names are compared in a case-insensitive manner.
case_sensitive_dtypes
    Should the schema match be case-sensitive with regard to column data types? If `True`,
    then the column data types in the schema and the target table must match exactly. If
    `False`, then the column data types are compared in a case-insensitive manner.
full_match_dtypes
    Should the schema match require a full match of data types? If `True`, then the column
    data types in the schema and the target table must match exactly. If `False` then
    substring matches are allowed, so a schema data type of `Int` would match a target table
    data type of `Int64`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. Regarding the lifetime of the transformed table, it only exists during the
validation step and is not stored in the `Validate` object or used in subsequent validation
steps.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

For the examples here, we'll use a simple Polars DataFrame with three columns (string,
integer, and float). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": ["apple", "banana", "cherry", "date"],
        "b": [1, 6, 3, 5],
        "c": [1.1, 2.2, 3.3, 4.4],
    }
)

pb.preview(tbl)
```

Let's validate that the columns in the table match a predefined schema. A schema can be
defined using the [`Schema`](`pointblank.Schema`) class.

```{python}
schema = pb.Schema(
    columns=[("a", "String"), ("b", "Int64"), ("c", "Float64")]
)
```

You can print the schema object to verify that the expected schema is as intended.

```{python}
print(schema)
```

Now, we'll use the `col_schema_match()` method to validate the table against the expected
`schema` object. There is a single test unit for this validation step (whether the schema
matches the table or not).

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_schema_match(schema=schema)
    .interrogate()
)

validation
```

The validation table shows that the schema matches the table. The single test unit passed
since the table columns and their types match the schema.

row_count_match(self, count: 'int | Any', tol: 'Tolerance' = 0, inverse: 'bool' = False, pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether the row count of the table matches a specified count.

The `row_count_match()` method checks whether the row count of the target table matches a
specified count. This validation will operate over a single test unit, which is whether the
row count matches the specified count.

We also have the option to invert the validation step by setting `inverse=True`. This will
make the expectation that the row count of the target table *does not* match the specified
count.

Parameters
----------
count
    The expected row count of the table. This can be an integer value, a Polars or Pandas
    DataFrame object, or an Ibis backend table. If a DataFrame/table is provided, the row
    count of that object will be used as the expected count.
tol
    The tolerance allowable for the row count match. This can be specified as a single
    numeric value (integer or float) or as a tuple of two integers representing the lower
    and upper bounds of the tolerance range. If a single integer value (`1` or greater) is
    provided, it represents the absolute bounds of the tolerance, ie. plus or minus the value.
    If a float value (between 0-1) is provided, it represents the relative tolerance, ie.
    plus or minus the relative percentage of the target. If a tuple is provided, it represents
    the lower and upper absolute bounds of the tolerance range. See the examples for more.
inverse
    Should the validation step be inverted? If `True`, then the expectation is that the row
    count of the target table should not match the specified `count=` value.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Regarding the lifetime of the
transformed table, it only exists during the validation step and is not stored in the
`Validate` object or used in subsequent validation steps.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

For the examples here, we'll use the built in dataset `"small_table"`. The table can be
obtained by calling `load_dataset("small_table")`.

```{python}
import pointblank as pb

small_table = pb.load_dataset("small_table")

pb.preview(small_table)
```

Let's validate that the number of rows in the table matches a fixed value. In this case, we
will use the value `13` as the expected row count.

```{python}
validation = (
    pb.Validate(data=small_table)
    .row_count_match(count=13)
    .interrogate()
)

validation
```

The validation table shows that the expectation value of `13` matches the actual count of
rows in the target table. So, the single test unit passed.

The `tol=` argument allows the row count to differ from `count=` by some amount. A tuple
sets separate lower and upper bounds, so here `tol=(2, 0)` accepts anywhere from 11 to 13
rows. A sample of 12 rows from the table passes:

```{python}
smaller_small_table = small_table.sample(n=12)

validation = (
    pb.Validate(data=smaller_small_table)
    .row_count_match(count=13, tol=(2, 0))
    .interrogate()
)

validation
```

A float between `0` and `1` is a relative tolerance: `tol=0.1` allows a difference of 10%
of `13`, which is rounded down to one row. So the 12-row sample passes again:

```{python}
validation = (
    pb.Validate(data=smaller_small_table)
    .row_count_match(count=13, tol=0.1)
    .interrogate()
)

validation
```

A smaller relative tolerance such as `tol=0.05` would make this step fail, since 5% of `13`
rounds down to zero rows. An integer of `1` or more is an absolute tolerance in both
directions. With `tol=5`, the row count can be anywhere from 8 to 18, so a two-row table
fails:

```{python}
even_smaller_table = small_table.sample(n=2)

validation = (
    pb.Validate(data=even_smaller_table)
    .row_count_match(count=13, tol=5)
    .interrogate()
)

validation
```

The single test unit fails because two rows is well outside the range of 8 to 18. When a
table should be about the same size as another table, `count=` can also take that table
directly and use its row count as the expected value.

col_count_match(self, count: 'int | Any', inverse: 'bool' = False, pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether the column count of the table matches a specified count.

The `col_count_match()` method checks whether the column count of the target table matches a
specified count. This validation will operate over a single test unit, which is whether the
column count matches the specified count.

We also have the option to invert the validation step by setting `inverse=True`. This will
make the expectation that column row count of the target table *does not* match the
specified count.

Parameters
----------
count
    The expected column count of the table. This can be an integer value, a Polars or Pandas
    DataFrame object, or an Ibis backend table. If a DataFrame/table is provided, the column
    count of that object will be used as the expected count.
inverse
    Should the validation step be inverted? If `True`, then the expectation is that the
    column count of the target table should not match the specified `count=` value.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Regarding the lifetime of the
transformed table, it only exists during the validation step and is not stored in the
`Validate` object or used in subsequent validation steps.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

For the examples here, we'll use the built in dataset `"game_revenue"`. The table can be
obtained by calling `load_dataset("game_revenue")`.

```{python}
import pointblank as pb

game_revenue = pb.load_dataset("game_revenue")

pb.preview(game_revenue)
```

Let's validate that the number of columns in the table matches a fixed value. In this case,
we will use the value `11` as the expected column count.

```{python}
validation = (
    pb.Validate(data=game_revenue)
    .col_count_match(count=11)
    .interrogate()
)

validation
```

The validation table shows that the expectation value of `11` matches the actual count of
columns in the target table. So, the single test unit passed.

data_freshness(self, column: 'str', max_age: 'str | datetime.timedelta', reference_time: 'datetime.datetime | str | None' = None, timezone: 'str | None' = None, allow_tz_mismatch: 'bool' = False, allow_future: 'bool | str | datetime.timedelta' = False, pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate that data in a datetime column is not older than a specified maximum age.

The `data_freshness()` validation method checks whether the most recent timestamp in the
specified datetime column is within the allowed `max_age=` from the `reference_time=` (which
defaults to the current time). This is useful for ensuring data pipelines are delivering
fresh data and for enforcing data SLAs.

This method helps detect stale data by comparing the maximum (most recent) value in a
datetime column against an expected freshness threshold. Data dated later than the
reference time fails too, unless `allow_future=` permits it (see the *Future-Dated Data*
section).

Parameters
----------
column
    The name of the datetime column to check for freshness. This column should contain
    date or datetime values.
max_age
    The maximum allowed age of the data. Can be specified as: (1) a string with a
    human-readable duration like `"24 hours"`, `"1 day"`, `"30 minutes"`, `"2 weeks"`, etc.
    (supported units: `seconds`, `minutes`, `hours`, `days`, `weeks`), or (2) a
    `datetime.timedelta` object for precise control.
reference_time
    The reference point in time to compare against. Defaults to `None`, which uses the
    current time (UTC if `timezone=` is not specified). Can be: (1) a `datetime.datetime`
    object (timezone-aware recommended), (2) a string in ISO 8601 format (e.g.,
    `"2024-01-15T10:30:00"` or `"2024-01-15T10:30:00+05:30"`), or (3) `None` to use the
    current time.
timezone
    The timezone to use for interpreting the data and reference time. Accepts IANA
    timezone names (e.g., `"America/New_York"`), hour offsets (e.g., `"-7"`), or ISO 8601
    offsets (e.g., `"-07:00"`). When `None` (default), naive datetimes are treated as UTC.
    See the *The `timezone=` Parameter* section for details.
allow_tz_mismatch
    Whether to allow timezone mismatches between the column data and reference time.
    By default (`False`), a warning note is added when comparing timezone-naive with
    timezone-aware datetimes. Set to `True` to suppress these warnings.
allow_future
    How far the most recent timestamp may be later than the reference time. The default,
    `False`, makes the step fail when any timestamp is later than the reference time. A
    duration (a string like `"5 minutes"` or a `datetime.timedelta`) allows timestamps up to
    that far ahead, which absorbs small differences between the clocks that wrote the data
    and the clock that runs the validation. Use `True` to allow future-dated data of any
    amount. See the *Future-Dated Data* section for details.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

How Timezones Affect Freshness Checks
-------------------------------------
Freshness validation involves comparing two times: the **data time** (the most recent
timestamp in your column) and the **execution time** (when and where the validation runs).
Timezone confusion typically arises because these two times may originate from different
contexts.

Consider these common scenarios:

- your data timestamps are stored in UTC (common for databases), but you're running
  validation on your laptop in New York (Eastern Time)
- you develop and test validation locally, then deploy it to a cloud workflow that runs
  in UTC—suddenly your 'same' validation behaves differently
- your data comes from servers in multiple regions, each recording timestamps in their
  local timezone

The `timezone=` parameter exists to solve this problem by establishing a single, explicit
timezone context for the freshness comparison. When you specify a timezone, Pointblank
interprets both the data timestamps (if naive) and the execution time in that timezone,
ensuring consistent behavior whether you run validation on your laptop or in a cloud
workflow.

**Scenario 1: Data has timezone-aware datetimes**

```python
# Your data column has values like: 2024-01-15 10:30:00+00:00 (UTC)
# Comparison is straightforward as both sides have explicit timezones
.data_freshness(column="updated_at", max_age="24 hours")
```

**Scenario 2: Data has naive datetimes (no timezone)**

```python
# Your data column has values like: 2024-01-15 10:30:00 (no timezone)
# Specify the timezone the data was recorded in:
.data_freshness(column="updated_at", max_age="24 hours", timezone="America/New_York")
```

**Scenario 3: Ensuring consistent behavior across environments**

```python
# Pin the timezone to ensure identical results whether running locally or in the cloud
.data_freshness(
    column="updated_at",
    max_age="24 hours",
    timezone="UTC",  # Explicit timezone removes environment dependence
)
```

The `timezone=` Parameter
---------------------------
The `timezone=` parameter accepts several convenient formats, making it easy to specify
timezones in whatever way is most natural for your use case. The following examples
illustrate the three supported input styles.

**IANA Timezone Names** (recommended for regions with daylight saving time):

```python
timezone="America/New_York"   # Eastern Time (handles DST automatically)
timezone="Europe/London"      # UK time
timezone="Asia/Tokyo"         # Japan Standard Time
timezone="Australia/Sydney"   # Australian Eastern Time
timezone="UTC"                # Coordinated Universal Time
```

**Simple Hour Offsets** (quick and easy):

```python
timezone="-7"    # UTC-7 (e.g., Mountain Standard Time)
timezone="+5"    # UTC+5 (e.g., Pakistan Standard Time)
timezone="0"     # UTC
timezone="-12"   # UTC-12
```

**ISO 8601 Offset Format** (precise, including fractional hours):

```python
timezone="-07:00"   # UTC-7
timezone="+05:30"   # UTC+5:30 (e.g., India Standard Time)
timezone="+00:00"   # UTC
timezone="-09:30"   # UTC-9:30
```

When a timezone is specified:

- naive datetime values in the column are assumed to be in this timezone.
- the reference time (if naive) is assumed to be in this timezone.
- the validation report will show times in this timezone.

When `None` (default):

- if your column has timezone-aware datetimes, those timezones are used
- if your column has naive datetimes, they're treated as UTC
- the current time reference uses UTC

Note that IANA timezone names are preferred when daylight saving time transitions matter, as
they automatically handle the offset changes. Fixed offsets like `"-7"` or `"-07:00"` do not
account for DST.

Recommendations for Working with Timestamps
-------------------------------------------
When working with datetime data, storing timestamps in UTC in your databases is strongly
recommended since it provides a consistent reference point regardless of where your data
originates or where it's consumed. Using timezone-aware datetimes whenever possible helps
avoid ambiguity—when a datetime has an explicit timezone, there's no guessing about what
time it actually represents.

If you're working with naive datetimes (which lack timezone information), always specify the
`timezone=` parameter so Pointblank knows how to interpret those values. When providing
`reference_time=` as a string, use ISO 8601 format with the timezone offset included (e.g.,
`"2024-01-15T10:30:00+00:00"`) to ensure unambiguous parsing. Finally, prefer IANA timezone
names (like `"America/New_York"`) over fixed offsets (like `"-05:00"`) when daylight saving
time transitions matter, since IANA names automatically handle the twice-yearly offset
changes. To see all available IANA timezone names in Python, use
`zoneinfo.available_timezones()` from the standard library's `zoneinfo` module.

Future-Dated Data
-----------------
When the most recent timestamp is later than the reference time, the data's age is negative.
This usually points to a problem: a clock that is set wrong, a timezone mix-up when the data
was loaded, or a snapshot validated against a `reference_time=` that comes before it. By
default, such a step fails, and the report notes how far ahead of the reference time the
data is.

The `allow_future=` parameter changes this. A small duration such as `allow_future="1m"` is
useful when the reference time is the current time and the data comes from a system whose
clock may run slightly ahead. Timestamps up to that far ahead pass, while anything further
ahead still fails. With `allow_future=True`, any amount of future-dated
data passes, and only `max_age=` is checked.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

The simplest use of `data_freshness()` requires just two arguments: the `column=` containing
your timestamps and `max_age=` specifying how old the data can be. In this first example,
we create sample data with an `"updated_at"` column containing timestamps from 1, 12, and
20 hours ago. By setting `max_age="24 hours"`, we're asserting that the most recent
timestamp should be within 24 hours of the current time. Since the newest record is only
1 hour old, this validation passes.

```{python}
import pointblank as pb
import polars as pl
from datetime import datetime, timedelta

# Create sample data with recent timestamps
recent_data = pl.DataFrame({
    "id": [1, 2, 3],
    "updated_at": [
        datetime.now() - timedelta(hours=1),
        datetime.now() - timedelta(hours=12),
        datetime.now() - timedelta(hours=20),
    ]
})

validation = (
    pb.Validate(data=recent_data)
    .data_freshness(column="updated_at", max_age="24 hours")
    .interrogate()
)

validation
```

The `max_age=` parameter accepts human-readable strings with various time units. You can
chain multiple `data_freshness()` calls to check different freshness thresholds
simultaneously—useful for tiered SLAs where you might want warnings at 30 minutes but
errors at 2 days.

```{python}
# Check data is fresh within different time windows
validation = (
    pb.Validate(data=recent_data)
    .data_freshness(column="updated_at", max_age="30 minutes")  # Very fresh
    .data_freshness(column="updated_at", max_age="2 days")      # Reasonably fresh
    .data_freshness(column="updated_at", max_age="1 week")      # Within a week
    .interrogate()
)

validation
```

Only the first step fails: the newest record is an hour old, which is older than 30 minutes
but well within 2 days and 1 week. Each step has a single test unit, so these tiers show up
as separate rows in the report.

When your data contains naive datetimes (timestamps without timezone information), use the
`timezone=` parameter to specify what timezone those values represent. Here we have event
data recorded in Eastern Time, so we set `timezone="America/New_York"` to ensure the
freshness comparison is done correctly.

```{python}
# Data with naive datetimes (assume they're in Eastern Time)
eastern_data = pl.DataFrame({
    "event_time": [
        datetime.now() - timedelta(hours=2),
        datetime.now() - timedelta(hours=5),
    ]
})

validation = (
    pb.Validate(data=eastern_data)
    .data_freshness(
        column="event_time",
        max_age="12 hours",
        timezone="America/New_York"  # Interpret times as Eastern
    )
    .interrogate()
)

validation
```

The step passes here. Note that `datetime.now()` gives the local time of the machine
running the code, so the age computed for this example depends on how far that machine's
time zone is from Eastern Time. With real data, `timezone=` should name the time zone in
which the timestamps were recorded.

For reproducible validations or historical checks, you can use `reference_time=` to compare
against a specific point in time instead of the current time. This is particularly useful
for testing or when validating data snapshots. The reference time should include a timezone
offset (like `+00:00` for UTC) to avoid ambiguity.

```{python}
snapshot_data = pl.DataFrame({
    "id": [1, 2, 3],
    "updated_at": [
        datetime(2024, 1, 15, 10, 0),
        datetime(2024, 1, 14, 22, 0),
        datetime(2024, 1, 13, 9, 0),
    ]
})

validation = (
    pb.Validate(data=snapshot_data)
    .data_freshness(
        column="updated_at",
        max_age="24 hours",
        reference_time="2024-01-15T12:00:00+00:00"
    )
    .data_freshness(
        column="updated_at",
        max_age="1 hour",
        reference_time="2024-01-15T12:00:00+00:00"
    )
    .interrogate()
)

validation
```

The newest timestamp is two hours before the reference time, so the 24-hour step passes and
the 1-hour step fails. Because neither the data nor the reference time depend on the current
time, this validation gives the same result whenever it runs.

If the reference time comes before the newest timestamp, the data is future-dated. Here the
reference time is two hours before the newest record. The first step fails because of
that, the second step allows timestamps up to three hours ahead, and the third allows
future-dated data of any amount.

```{python}
validation = (
    pb.Validate(data=snapshot_data)
    .data_freshness(
        column="updated_at",
        max_age="24 hours",
        reference_time="2024-01-15T08:00:00+00:00"
    )
    .data_freshness(
        column="updated_at",
        max_age="24 hours",
        reference_time="2024-01-15T08:00:00+00:00",
        allow_future="3 hours"
    )
    .data_freshness(
        column="updated_at",
        max_age="24 hours",
        reference_time="2024-01-15T08:00:00+00:00",
        allow_future=True
    )
    .interrogate()
)

validation
```

The steps that allow future-dated data say so below `max_age=` in the report, and the note
for each step gives how far ahead of the reference time the newest record is.

tbl_match(self, tbl_compare: 'Any', keys: 'str | list[str] | None' = None, columns: 'str | list[str] | None' = None, column_map: 'dict[str, str] | None' = None, tolerance: 'Any' = None, normalize: 'dict[str, Any] | None' = None, null_equal: 'bool' = True, schema: "Literal['strict', 'common']" = 'strict', dup_keys: "Literal['flag', 'compare']" = 'flag', order_by: 'str | list[str] | None' = None, partitions: 'int | None' = None, pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate whether the target table matches a comparison table.

The `tbl_match()` method compares the target table, row by row, against a comparison table
that holds the expected data (the *source*). Rows of the two tables are aligned and every
aligned row becomes a test unit: it passes if the row is present in both tables with equal
values, and fails if its values changed, if it's missing from the target, if it's an extra
row in the target, or if its key is duplicated. Schema differences (columns added, removed,
retyped, or reordered) are checked too.

Rows are aligned in one of three ways:

- **by key** (`keys=` given): rows are matched on one or more key columns, so row order
  doesn't matter; this is the most informative mode
- **by position** (`keys=None`, the default): the first row of each table is compared, then
  the second, and so on
- **as multisets** (`keys="*"`): rows are compared as whole values regardless of order, with
  duplicates counted (for tables without a key)

The comparison runs inside the table's own engine (it is backend-agnostic, and database
tables are compared in the database). Use
[`get_step_report()`](`pointblank.Validate.get_step_report`) to see a detailed report of
the differences, and [`get_data_extracts()`](`pointblank.Validate.get_data_extracts`) to
get the failing rows. The same comparison is available outside of a validation with
[`compare()`](`pointblank.compare`).

Parameters
----------
tbl_compare
    The comparison table (the expected data) to validate against. This can be a DataFrame
    object (Polars or Pandas), an Ibis table object, a path to a CSV or Parquet file, or a
    callable that returns a table. If a callable is provided, it will be executed during
    interrogation to obtain the comparison table.
keys
    One or more columns that uniquely identify a row, used to align the rows of the two
    tables. Key names refer to the comparison table (use `column_map=` if the target names
    them differently). If `None` (the default), rows are aligned by position. Use `"*"` to
    compare the tables as multisets of rows.
columns
    The comparison-table columns whose values should be compared. By default, all columns
    present in both tables (other than the keys) are compared. Giving `columns=` also limits
    the schema check to the keys and these columns.
column_map
    A mapping of comparison-table column names to target column names, for renamed columns.
tolerance
    Tolerance for numeric and datetime comparisons. A number is an absolute tolerance for all
    numeric columns. A dict with `"atol"` and/or `"rtol"` keys applies to all numeric
    columns, where values are equal if `|expected - actual| <= atol + rtol * |expected|`. A
    dict mapping column names to either of the above sets per-column tolerances. Datetime
    tolerances are given as a `datetime.timedelta` or a string like `"1s"` or `"500ms"`.
normalize
    Per-column transformations applied to both tables before comparing, as a dict mapping a
    column name to a list of operations: `"strip"`, `"lower"`, `"upper"`,
    `"collapse_whitespace"`, or `{"round": n}`. The string operations only apply to string
    columns and `round` only to numeric ones.
null_equal
    Should two null values count as equal? By default, they do. NaN values in floating-point
    columns are treated as null.
schema
    How schema differences affect the result. With `"strict"` (the default), any schema
    difference (other than renames given in `column_map=`) means the tables don't match,
    and every test unit fails. With `"common"`, only the shared columns are compared and
    schema differences are reported without failing any test units.
dup_keys
    Rows whose key appears more than once in either table always fail. With `"flag"` (the
    default), the step report lists the duplicated keys. With `"compare"`, the step report
    also compares the duplicated rows pairwise to show which columns differ.
order_by
    Column(s) defining the row order for positional alignment on backends without an
    inherent row order (e.g., PySpark or database tables). Without it, such tables are
    collected and compared in the order in which their rows are returned; use `keys=` for
    a reliable alignment. Ignored when `keys=` is given.
partitions
    Compare the tables in this many passes, each covering a disjoint subset of the key
    values (by key hash), to bound memory use for very large tables. Each pass reads the
    tables again, so this trades time for memory; the results are identical to a
    single-pass comparison. Requires `keys=` (not `"*"`), and Polars, DuckDB, or Ibis
    tables (including CSV/Parquet paths). Sundering isn't available for the step when
    partitions are used.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Test Units
----------
Each aligned row is a test unit. With `keys=`, that's every key value found in either table
(a duplicated key counts as many units as its larger number of occurrences); by position,
it's the number of rows in the longer table; as multisets, it's every row copy. A test unit
passes only if its row is in both tables with all compared values equal. If the strict
schema check fails, all test units fail. When both tables are empty, the step has a single
test unit, which passes if the schemas match.

Because failures are counted per row, thresholds can be expressed as proportions (e.g., a
`warning` when more than 1% of rows differ). To fail on any difference, use an absolute
threshold of `1`.

:::{.callout-important}
In Pointblank `v1.0.1` and earlier, `tbl_match()` had a single test unit that failed if the
tables differed in any way. Whether a step passes or fails is unchanged, but the counts
(`n`, `n_passed`, `n_failed`, and the fractions) now reflect rows. This changes the meaning
of fractional thresholds: `thresholds=0.1` used to be exceeded by any difference, and is
now exceeded only when at least 10% of the rows fail. To keep the earlier behavior of
reacting to any difference, use an absolute threshold of `1` (e.g., `thresholds=1` or
`Thresholds(warning=1)`).
:::

The data extract for the step (from
[`get_data_extracts()`](`pointblank.Validate.get_data_extracts`)) contains the failing rows
of *both* tables: the key column(s), a `_status_` column (`"changed"`, `"missing"`,
`"extra"`, or `"dup_key"`), and a pair of `<column>_source`/`<column>_target` columns for
each compared column. For [`get_sundered_data()`](`pointblank.Validate.get_sundered_data`),
target rows with a matching row in the comparison table pass; rows missing from the target
can't be part of the sundered data. Sundering for this step is available with in-memory
tables (not lazy or database tables) and isn't available in multiset mode.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that the same preprocessing
is **not** applied to the comparison table; only the target table is preprocessed. Regarding
the lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Cross-Backend Validation
------------------------
The two tables can come from different backends (e.g., a Polars DataFrame and a Pandas
DataFrame, or a database table and a CSV file). Tables on the same backend are compared in
place (so two tables in the same database are compared inside the database). Tables on
different backends are both converted to Polars (or to Pandas, if Polars isn't installed)
before the comparison. Key columns whose types differ between the tables (e.g., integer
keys that became floats in Pandas because of a missing value) are cast to a common type.

Values in columns whose types differ are compared numerically when both columns are numeric
and as strings otherwise; note that such differences also fail the strict schema check
(use `schema="common"` to only report them).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

For the examples here, we'll create two simple tables to demonstrate the `tbl_match()`
validation.

```{python}
import pointblank as pb
import polars as pl

# Create the first table
tbl_1 = pl.DataFrame({
    "a": [1, 2, 3, 4],
    "b": ["w", "x", "y", "z"],
    "c": [4.0, 5.0, 6.0, 7.0]
})

# Create an identical table
tbl_2 = pl.DataFrame({
    "a": [1, 2, 3, 4],
    "b": ["w", "x", "y", "z"],
    "c": [4.0, 5.0, 6.0, 7.0]
})

pb.preview(tbl_1)
```

Let's validate that `tbl_1` matches `tbl_2`. Since these tables are identical, the
validation should pass.

```{python}
validation = (
    pb.Validate(data=tbl_1)
    .tbl_match(tbl_compare=tbl_2)
    .interrogate()
)

validation
```

The validation table shows that all four test units (one per row) passed, indicating that
the two tables match completely.

Now, let's compare against a table that has one different value, is missing a row, and has
an extra row. Since the rows can be identified by column `a`, we'll use it as the key.

```{python}
tbl_3 = pl.DataFrame({
    "a": [1, 2, 4, 5],
    "b": ["w", "x", "z", "v"],
    "c": [4.0, 5.5, 7.0, 8.0]  # Changed 5.0 to 5.5
})

validation = (
    pb.Validate(data=tbl_3)
    .tbl_match(tbl_compare=tbl_1, keys="a")
    .interrogate()
)

validation
```

Three of the five aligned rows failed: one changed, one missing from the target, and one
extra. The step report shows exactly what differs.

```{python}
validation.get_step_report(i=1)
```

The report groups the differences: column `c` is the only column with mismatched values, the
row with key `2` changed (`5.0` became `5.5`), the row with key `3` is missing from the
target, and the row with key `5` is extra. The key values identify each row, so you can look
up the problem rows in either table.

tbl_reconcile(self, tbl_compare: 'Any', keys: 'str | list[str] | None' = None, measures: 'str | list[str] | None' = None, agg: 'str | dict[str, str | list[str]]' = 'sum', tolerance: 'Any' = None, column_map: 'dict[str, str] | None' = None, partitions: 'int | None' = None, pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate that aggregates of the target table reconcile with a comparison table.

The `tbl_reconcile()` method checks that the target table agrees with a comparison table
(the *source*, or expected data) on aggregate values: totals, counts, averages, minimums,
or maximums of *measure* columns, computed per group of *key* columns (or over the whole
table). It's the check to reach for when the two tables don't have the same rows but must
add up to the same numbers, for example a warehouse fact table against the source system it
was loaded from, or a monthly summary against the detailed transactions.

Each key group is a test unit: it passes if the group is in both tables and all of its
aggregates are equal (within the tolerance), and fails if an aggregate differs, if the
group is missing from the target, or if it's an extra group in the target. To compare the
rows themselves rather than aggregates, use
[`tbl_match()`](`pointblank.Validate.tbl_match`).

Parameters
----------
tbl_compare
    The comparison table (the expected data) to reconcile against. This can be a DataFrame
    object (Polars or Pandas), an Ibis table object, a path to a CSV or Parquet file, or a
    callable that returns a table.
keys
    The column(s) to group by. Aggregates are compared group by group. If `None` (the
    default), the aggregates are computed over the whole table and compared as a single
    group (a grand total).
measures
    The column(s) to aggregate. Required unless `agg="count"`, which (without `measures=`)
    compares the number of rows in each group.
agg
    The aggregation(s): `"sum"` (the default), `"count"` (the number of non-null values),
    `"mean"`, `"min"`, or `"max"`. Use a dict to aggregate measures differently, mapping a
    measure to an aggregation or a list of aggregations (e.g.,
    `{"amount": ["sum", "max"], "qty": "sum"}`).
tolerance
    Tolerance for comparing the aggregates. A number is an absolute tolerance for all
    aggregates. A dict with `"atol"` and/or `"rtol"` keys applies to all aggregates, where
    values are equal if `|expected - actual| <= atol + rtol * |expected|`. A dict mapping
    measures to either of the above sets per-measure tolerances (applied to all of a
    measure's aggregates other than counts).
column_map
    A mapping of comparison-table column names to target column names, for keys or
    measures that are named differently in the target.
partitions
    Compare the aggregated tables in this many passes, each covering a disjoint subset of
    the key values, to bound memory use when there are very many groups. Requires `keys=`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Test Units
----------
Each key group found in either table is a test unit (with `keys=None`, there's a single
unit for the whole table). A unit passes if its aggregates are equal (within the tolerance)
in both tables. Rows with null keys form a group of their own, which is matched with the
null-key group of the other table.

The data extract for the step (from
[`get_data_extracts()`](`pointblank.Validate.get_data_extracts`)) contains the failing
groups: the key column(s), a `_status_` column (`"changed"`, `"missing"`, or `"extra"`),
and a pair of `<measure>_<agg>_source`/`<measure>_<agg>_target` columns for each aggregate.
The step report shows the full comparison of the aggregates. Since test units are groups
rather than rows, this step doesn't take part in
[`get_sundered_data()`](`pointblank.Validate.get_sundered_data`).

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Note that the same preprocessing
is **not** applied to the comparison table; only the target table is preprocessed. Regarding
the lifetime of the transformed table, it only exists during the validation step and is not
stored in the `Validate` object or used in subsequent validation steps.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

Let's reconcile a small table of loaded orders against the source system's orders. The
loaded table names the amount column differently, has a different number of rows (one order
was split into two rows), and is missing the orders for one region.

```{python}
import pointblank as pb
import polars as pl

source_orders = pl.DataFrame(
    {
        "region": ["north", "north", "south", "east", "west"],
        "amount": [100.0, 250.0, 75.5, 30.0, 12.0],
    }
)

loaded_orders = pl.DataFrame(
    {
        "region": ["north", "north", "north", "south", "east"],
        "amt": [100.0, 200.0, 50.0, 75.5, 31.0],
    }
)

validation = (
    pb.Validate(data=loaded_orders)
    .tbl_reconcile(
        tbl_compare=source_orders,
        keys="region",
        measures="amount",
        column_map={"amount": "amt"},
    )
    .interrogate()
)

validation
```

Of the four regions, `north` and `south` reconcile; `east` has a different total, and
`west` is missing from the loaded orders. The step report shows the details:

```{python}
validation.get_step_report(i=1)
```

Grand totals don't need any keys. Here, the total amount matches within a tolerance of
`20`, and so the step passes:

```{python}
(
    pb.Validate(data=loaded_orders)
    .tbl_reconcile(
        tbl_compare=source_orders,
        measures="amount",
        column_map={"amount": "amt"},
        tolerance=20,
    )
    .interrogate()
    .all_passed()
)
```

The method chain ends with `all_passed()`, which returns `True`. The totals are `467.5` in
the source and `456.5` in the loaded table, a difference of `11` that falls within the
tolerance. A grand total can hide offsetting differences, though: the missing `west` orders
and the larger `east` total partly cancel out, which is why the per-region check above is
the stricter one.

conjointly(self, *exprs: 'Callable', pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Perform multiple row-wise validations for joint validity.

The `conjointly()` validation method checks whether each row in the table passes multiple
validation conditions simultaneously. This enables compound validation logic where a test
unit (typically a row) must satisfy all specified conditions to pass the validation.

This method accepts multiple validation expressions as callables, which should return
boolean expressions when applied to the data. You can use lambdas that incorporate
Polars/Pandas/Ibis expressions (based on the target table type) or create more complex
validation functions. The validation will operate over the number of test units that is
equal to the number of rows in the table (determined after any `pre=` mutation has been
applied).

Parameters
----------
*exprs
    Multiple validation expressions provided as callable functions. Each callable should
    accept a table as its single argument and return a boolean expression or Series/Column
    that evaluates to boolean values for each row.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Missing Values and Evaluation Errors
------------------------------------
A row where any expression evaluates to a missing value (e.g., a comparison on a null)
counts as a failing test unit. To let such rows pass, allow for them in the expression
itself (e.g., `lambda df: (pl.col("a") > 1) | pl.col("a").is_null()`).

If any of the expressions can't be evaluated against the table (e.g., one refers to a column
that doesn't exist), the step is marked as having an evaluation error and a note says which
expression failed and why. The other validation steps are still interrogated.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Regarding the lifetime of the
transformed table, it only exists during the validation step and is not stored in the
`Validate` object or used in subsequent validation steps.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use a simple Polars DataFrame with three numeric columns (`a`,
`b`, and `c`). The table is shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 7, 1, 3, 9, 4],
        "b": [6, 3, 0, 5, 8, 2],
        "c": [10, 4, 8, 9, 10, 5],
    }
)

pb.preview(tbl)
```

Let's validate that the values in each row satisfy multiple conditions simultaneously:

1. Column `a` should be greater than 2
2. Column `b` should be less than 7
3. The sum of `a` and `b` should be less than the value in column `c`

We'll use `conjointly()` to check all these conditions together:

```{python}
validation = (
    pb.Validate(data=tbl)
    .conjointly(
        lambda df: pl.col("a") > 2,
        lambda df: pl.col("b") < 7,
        lambda df: pl.col("a") + pl.col("b") < pl.col("c")
    )
    .interrogate()
)

validation
```

The validation table shows that not all rows satisfy all three conditions together. For a
row to pass the conjoint validation, all three conditions must be true for that row.

We can also use preprocessing to filter the data before applying the conjoint validation:

```{python}
# Define preprocessing function for serialization compatibility
def filter_by_c_gt_5(df):
    return df.filter(pl.col("c") > 5)

validation = (
    pb.Validate(data=tbl)
    .conjointly(
        lambda df: pl.col("a") > 2,
        lambda df: pl.col("b") < 7,
        lambda df: pl.col("a") + pl.col("b") < pl.col("c"),
        pre=filter_by_c_gt_5
    )
    .interrogate()
)

validation
```

This allows for more complex validation scenarios where the data is first prepared and then
validated against multiple conditions simultaneously.

Or, you can use the backend-agnostic column expression helper
[`expr_col()`](`pointblank.expr_col`) to write expressions that work across different table
backends:

```{python}
tbl = pl.DataFrame(
    {
        "a": [5, 7, 1, 3, 9, 4],
        "b": [6, 3, 0, 5, 8, 2],
        "c": [10, 4, 8, 9, 10, 5],
    }
)

# Using backend-agnostic syntax with expr_col()
validation = (
    pb.Validate(data=tbl)
    .conjointly(
        lambda df: pb.expr_col("a") > 2,
        lambda df: pb.expr_col("b") < 7,
        lambda df: pb.expr_col("a") + pb.expr_col("b") < pb.expr_col("c")
    )
    .interrogate()
)

validation
```

Using [`expr_col()`](`pointblank.expr_col`) allows your validation code to work consistently
across Pandas, Polars, and Ibis table backends without changes, making your validation
pipelines more portable.

See Also
--------
expr_col : Build column expressions that work across table backends.

specially(self, expr: 'Callable', pre: 'Callable | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Perform a specialized validation with customized logic.

The `specially()` validation method allows for the creation of specialized validation
expressions that can be used to validate specific conditions or logic in the data. This
method provides maximum flexibility by accepting a custom callable that encapsulates
your validation logic.

The callable function can have one of two signatures:

- a function accepting a single parameter (the data table): `def validate(data): ...`
- a function with no parameters: `def validate(): ...`

The second form is particularly useful for environment validations that don't need to
inspect the data table.

The callable function must ultimately return one of:

1. a single boolean value or boolean list
2. a table where the final column contains boolean values (column name is unimportant)

The validation will operate over the number of test units that is equal to the number of
rows in the data table (if returning a table with boolean values). If returning a scalar
boolean value, the validation will operate over a single test unit. For a return of a list
of boolean values, the length of the list constitutes the number of test units.

Parameters
----------
expr
    A callable function that defines the specialized validation logic. This function should:
    (1) accept the target data table as its single argument (though it may ignore it), or
    (2) take no parameters at all (for environment validations). The function must
    ultimately return boolean values representing validation results. Design your function
    to incorporate any custom parameters directly within the function itself using closure
    variables or default parameters.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
    Have a look at the *Preprocessing* section for more information on how to use this
    argument.
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect. Look at the *Thresholds*
    section for information on how to set threshold levels.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Preprocessing
-------------
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data
table during interrogation. This function should take a table as input and return a modified
table. This is useful for performing any necessary transformations or filtering on the data
before the validation step is applied.

The preprocessing function can be any callable that takes a table as input and returns a
modified table. For example, you could use a lambda function to filter the table based on
certain criteria or to apply a transformation to the data. Regarding the lifetime of the
transformed table, it only exists during the validation step and is not stored in the
`Validate` object or used in subsequent validation steps.

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values
can either be set as a proportion failing of all test units (a value between `0` to `1`),
or, the absolute number of failing test units (as integer that's `1` or greater).

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

If the number of failing test units exceeds set thresholds, the validation step will be
marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be
set, you're free to set any combination of them.

Aside from reporting failure conditions, thresholds can be used to determine the actions to
take for each level of failure (using the `actions=` parameter).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
The `specially()` method offers maximum flexibility for validation, allowing you to create
custom validation logic that fits your specific needs. The following examples demonstrate
different patterns and use cases for this powerful validation approach.

### Simple validation with direct table access

This example shows the most straightforward use case where we create a function that
directly checks if the sum of two columns is positive.

```{python}
import pointblank as pb
import polars as pl

simple_tbl = pl.DataFrame({
    "a": [5, 7, 1, 3, 9, 4],
    "b": [6, 3, 0, 5, 8, 2]
})

# Simple function that validates directly on the table
def validate_sum_positive(data):
    return data.select(pl.col("a") + pl.col("b") > 0)

(
    pb.Validate(data=simple_tbl)
    .specially(expr=validate_sum_positive)
    .interrogate()
)
```

The function returns a Polars DataFrame with a single boolean column indicating whether
the sum of columns `a` and `b` is positive for each row. Each row in the resulting DataFrame
is a distinct test unit. This pattern works well for simple validations where you don't need
configurable parameters.

### Advanced validation with closure variables for parameters

When you need to make your validation configurable, you can use the function factory pattern
(also known as closures) to create parameterized validations:

```{python}
# Create a parameterized validation function using closures
def make_column_ratio_validator(col1, col2, min_ratio):
    def validate_column_ratio(data):
        return data.select((pl.col(col1) / pl.col(col2)) > min_ratio)
    return validate_column_ratio

(
    pb.Validate(data=simple_tbl)
    .specially(
        expr=make_column_ratio_validator(col1="a", col2="b", min_ratio=0.5)
    )
    .interrogate()
)
```

This approach allows you to create reusable validation functions that can be configured with
different parameters without modifying the function itself.

### Validation function returning a list of booleans

This example demonstrates how to create a validation function that returns a list of boolean
values, where each element represents a separate test unit:

```{python}
import pointblank as pb
import polars as pl
import random

# Create sample data
transaction_tbl = pl.DataFrame({
    "transaction_id": [f"TX{i:04d}" for i in range(1, 11)],
    "amount": [120.50, 85.25, 50.00, 240.75, 35.20, 150.00, 85.25, 65.00, 210.75, 90.50],
    "category": ["food", "shopping", "entertainment", "travel", "utilities",
                "food", "shopping", "entertainment", "travel", "utilities"]
})

# Define a validation function that returns a list of booleans
def validate_transaction_rules(data):
    # Create a list to store individual test results
    test_results = []

    # Check each row individually against multiple business rules
    for row in data.iter_rows(named=True):
        # Rule: transaction IDs must start with "TX" and be 6 chars long
        valid_id = row["transaction_id"].startswith("TX") and len(row["transaction_id"]) == 6

        # Rule: Amounts must be appropriate for their category
        valid_amount = True
        if row["category"] == "food" and (row["amount"] < 10 or row["amount"] > 200):
            valid_amount = False
        elif row["category"] == "utilities" and (row["amount"] < 20 or row["amount"] > 300):
            valid_amount = False
        elif row["category"] == "entertainment" and row["amount"] > 100:
            valid_amount = False

        # A transaction passes if it satisfies both rules
        test_results.append(valid_id and valid_amount)

    return test_results

(
    pb.Validate(data=transaction_tbl)
    .specially(
        expr=validate_transaction_rules,
        brief="Validate transaction IDs and amounts by category."
    )
    .interrogate()
)
```

This example shows how to create a validation function that applies multiple business rules
to each row and returns a list of boolean results. Each boolean in the list represents a
separate test unit, and a test unit passes only if all rules are satisfied for a given row.

The function iterates through each row in the data table, checking:

1. if transaction IDs follow the required format
2. if transaction amounts are appropriate for their respective categories

This approach is powerful when you need to apply complex, conditional logic that can't be
easily expressed using the built-in validation functions.

### Table-level validation returning a single boolean

Sometimes you need to validate properties of the entire table rather than row-by-row. In
these cases, your function can return a single boolean value:

```{python}
def validate_table_properties(data):
    # Check if table has at least one row with column 'a' > 10
    has_large_values = data.filter(pl.col("a") > 10).height > 0

    # Check if mean of column 'b' is positive
    has_positive_mean = data.select(pl.mean("b")).item() > 0

    # Return a single boolean for the entire table
    return has_large_values and has_positive_mean

(
    pb.Validate(data=simple_tbl)
    .specially(expr=validate_table_properties)
    .interrogate()
)
```

This example demonstrates how to perform multiple checks on the table as a whole and combine
them into a single validation result.

### Environment validation that doesn't use the data table

The `specially()` validation method can even be used to validate aspects of your environment
that are completely independent of the data:

```{python}
def validate_pointblank_version():
    try:
        import importlib.metadata
        version = importlib.metadata.version("pointblank")
        version_parts = version.split(".")

        # Get major and minor components regardless of how many parts there are
        major = int(version_parts[0])
        minor = int(version_parts[1])

        # Check both major and minor components for version `0.9+`
        return (major > 0) or (major == 0 and minor >= 9)

    except Exception as e:
        # More specific error handling could be added here
        print(f"Version check failed: {e}")
        return False

(
    pb.Validate(data=simple_tbl)
    .specially(
        expr=validate_pointblank_version,
        brief="Check Pointblank version `>=0.9.0`."
    )
    .interrogate()
)
```

This pattern shows how to validate external dependencies or environment conditions as part
of your validation workflow. Notice that the function doesn't take any parameters at all,
which makes it cleaner when the validation doesn't need to access the data table.

By combining these patterns, you can create sophisticated validation workflows that address
virtually any data quality requirement in your organization.

prompt(self, prompt: 'str', model: 'str', columns_subset: 'str | list[str] | None' = None, attachments: 'list | None' = None, batch_size: 'int' = 1000, max_concurrent: 'int' = 3, pre: 'Callable | None' = None, segments: 'SegmentSpec | None' = None, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, actions: 'Actions | None' = None, brief: 'str | bool | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Validate rows using AI/LLM-powered analysis.

The `prompt()` validation method uses Large Language Models (LLMs) to validate rows of data
based on natural language criteria. Similar to other Pointblank validation methods, this
generates binary test results (pass/fail) that integrate seamlessly with the standard
reporting framework.

Like `col_vals_*()` methods, `prompt()` evaluates data against specific criteria, but
instead of using programmatic rules, it uses natural language prompts interpreted by an LLM.
Like `rows_distinct()` and `rows_complete()`, it operates at the row level and allows you to
specify a subset of columns for evaluation using `columns_subset=`.

The system automatically combines your validation criteria from the `prompt=` parameter with
the necessary technical context, data formatting instructions, and response structure
requirements. This is all so you only need to focus on describing your validation logic in
plain language.

Each row becomes a test unit that either passes or fails the validation criteria, producing
the familiar True/False results that appear in Pointblank validation reports. This method
is particularly useful for complex validation rules that are difficult to express with
traditional validation methods, such as semantic checks, context-dependent validation, or
subjective quality assessments.

Parameters
----------
prompt
    A natural language description of the validation criteria. This prompt should clearly
    describe what constitutes valid vs invalid rows. Some examples:
    `"Each row should contain a valid email address and a realistic person name"`,
    `"Values should indicate positive sentiment"`,
    `"The description should mention a country name"`.
columns_subset
    A single column or list of columns to include in the validation. If `None`, all columns
    will be included. Specifying fewer columns can improve performance and reduce API costs
    so try to include only the columns necessary for the validation.
attachments
    An optional list of reference files (images or PDFs) to attach as global context for
    every batch. Each item can be a local file path, a URL (`http://` / `https://`), a
    `pathlib.Path`, or a pre-built chatlas `Content` object. Supported extensions are
    `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, and `.pdf`. The attachments apply to the whole
    validation step, not per-row, so they are well-suited for things like brand guides,
    schema diagrams, or sample documents the LLM should consult when scoring each row.
    **Cost note**: attachments are re-sent on every batch — see the *Multi-modal
    attachments* section below for cost-management tips.
model
    The model to be used. This should be in the form of `provider:model` (e.g.,
    `"anthropic:claude-opus-4-6"`). Supported providers are `"anthropic"`, `"openai"`,
    `"ollama"`, `"bedrock"`, and `"azure-openai"`. The model name should be the specific
    model to be used from the provider (for `"azure-openai"`, the value after the colon is
    the Azure *deployment id*). Model names are subject to change so consult the provider's
    documentation for the most up-to-date model names.
batch_size
    Number of rows to process in each batch. Larger batches are more efficient but may hit
    API limits. Default is `1000`.
max_concurrent
    Maximum number of concurrent API requests. Higher values speed up processing but may
    hit rate limits. Default is `3`.
pre
    An optional preprocessing function or lambda to apply to the data table during
    interrogation. This function should take a table as input and return a modified table.
segments
    An optional directive on segmentation, which serves to split a validation step into
    multiple (one step per segment). Can be a single column name, a tuple that specifies a
    column name and its corresponding values to segment on, or a combination of both
    (provided as a list).
thresholds
    Set threshold failure levels for reporting and reacting to exceedences of the levels.
    The thresholds are set at the step level and will override any global thresholds set in
    `Validate(thresholds=...)`. The default is `None`, which means that no thresholds will
    be set locally and global thresholds (if any) will take effect.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value. The callable is evaluated *before* any `pre=` processing. Inspection
    functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate a step
    based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the
    dimension is inferred automatically from the assertion type. This label appears in the
    validation report and feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Constructing the `model` Argument
---------------------------------
The `model=` argument should be constructed using the provider and model name separated by a
colon (`provider:model`). The provider text can any of:

- `"anthropic"` (Anthropic)
- `"openai"` (OpenAI)
- `"ollama"` (Ollama)
- `"bedrock"` (Amazon Bedrock)
- `"azure-openai"` (Azure OpenAI)

The model name should be the specific model to be used from the provider. Model names are
subject to change so consult the provider's documentation for the most up-to-date model
names. For `"azure-openai"`, the value after the colon is the Azure *deployment id* (the
name you assigned when deploying the model in your Azure OpenAI resource), not an OpenAI
model id.

Notes on Authentication
-----------------------
API keys are automatically loaded from environment variables or `.env` files and are **not**
stored in the validation object for security reasons. You should consider using a secure
method for handling API keys.

One way to do this is to load the API key from an environment variable and retrieve it using
the `os` module (specifically the `os.getenv()` function). Places to store the API key might
include `.bashrc`, `.bash_profile`, `.zshrc`, or `.zsh_profile`.

Another solution is to store one or more model provider API keys in an `.env` file (in the
root of your project). If the API keys have correct names (e.g., `ANTHROPIC_API_KEY` or
`OPENAI_API_KEY`) then the AI validation will automatically load the API key from the `.env`
file. An `.env` file might look like this:

```plaintext
ANTHROPIC_API_KEY="your_anthropic_api_key_here"
OPENAI_API_KEY="your_openai_api_key_here"
```

There's no need to have the `python-dotenv` package installed when using `.env` files in
this way.

**Provider-specific setup**:

- **OpenAI**: set `OPENAI_API_KEY` environment variable or create `.env` file
- **Anthropic**: set `ANTHROPIC_API_KEY` environment variable or create `.env` file
- **Ollama**: no API key required, just ensure Ollama is running locally
- **Bedrock**: configure AWS credentials through standard AWS methods
- **Azure OpenAI**: set `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` (e.g.,
  `https://<resource>.openai.azure.com`), and `OPENAI_API_VERSION` (e.g., `"2024-06-01"`)

AI Validation Process
---------------------
The AI validation process works as follows:

1. data batching: the data is split into batches of the specified size
2. row deduplication: duplicate rows (based on selected columns) are identified and only
unique combinations are sent to the LLM for analysis
3. json conversion: each batch of unique rows is converted to JSON format for the LLM
4. prompt construction: the user prompt is embedded in a structured system prompt
5. llm processing: each batch is sent to the LLM for analysis
6. response parsing: LLM responses are parsed to extract validation results
7. result projection: results are mapped back to all original rows using row signatures
8. result aggregation: results from all batches are combined

**Performance Optimization**: the process uses row signature memoization to avoid redundant
LLM calls. When multiple rows have identical values in the selected columns, only one
representative row is validated, and the result is applied to all matching rows. This can
dramatically reduce API costs and processing time for datasets with repetitive patterns.

The LLM receives data in this JSON format:

```json
{
  "columns": ["col1", "col2", "col3"],
  "rows": [
    {"col1": "value1", "col2": "value2", "col3": "value3", "_pb_row_index": 0},
    {"col1": "value4", "col2": "value5", "col3": "value6", "_pb_row_index": 1}
  ]
}
```

The LLM returns validation results in this format:
```json
[
  {"index": 0, "result": true},
  {"index": 1, "result": false}
]
```

Prompt Design Tips
------------------
For best results, design prompts that are:

- boolean-oriented: frame validation criteria to elicit clear valid/invalid responses
- specific: clearly define what makes a row valid/invalid
- unambiguous: avoid subjective language that could be interpreted differently
- context-aware: include relevant business rules or domain knowledge
- example-driven: consider providing examples in the prompt when helpful

**Critical**: Prompts must be designed so the LLM can determine whether each row passes or
fails the validation criteria. The system expects binary validation responses, so avoid
open-ended questions or prompts that might generate explanatory text instead of clear
pass/fail judgments.

Good prompt examples:

- "Each row should contain a valid email address in the 'email' column and a non-empty name
in the 'name' column"
- "The 'sentiment' column should contain positive sentiment words (happy, good, excellent,
etc.)"
- "Product descriptions should mention at least one technical specification"

Poor prompt examples (avoid these):

- "What do you think about this data?" (too open-ended)
- "Describe the quality of each row" (asks for description, not validation)
- "How would you improve this data?" (asks for suggestions, not pass/fail)

Multi-modal Attachments
-----------------------
Use `attachments=` to give the LLM a reference image or PDF that applies to every row. The
attachment is sent as global context, while each row's values are still serialized as JSON
and validated against your `prompt=`. Useful patterns:

- validating descriptions against a brand-style image: `attachments=["brand_guide.pdf"]`
- cross-referencing rows with a schema diagram: `attachments=["schema.png"]`
- matching free-text fields to a sample document: `attachments=["sample_invoice.pdf"]`

Accepted values per list item:

- local path strings or `pathlib.Path` objects (e.g., `"docs/diagram.png"`)
- URLs (e.g., `"https://example.com/diagram.png"`)
- pre-built chatlas `Content` objects (e.g., `chatlas.content_image_plot()`)

Supported extensions: `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.pdf`.

Note: local image paths auto-coerced through `attachments=` use `resize="low"` (chatlas's
default downscale to 512x512) to keep token costs predictable. For higher fidelity,
pre-build the content yourself with `chatlas.content_image_file(path, resize="high")`
(or `"none"`) and pass that object inside `attachments=`.

**Cost / batching note**: attachments are re-sent on *every* batch (one batch = one LLM API
call). For a table that requires N batches, each attachment's input tokens are billed N
times. To control costs:

- keep attachments small (downscale images, crop PDFs to the relevant page)
- use `columns_subset=` aggressively to maximize row-signature memoization (fewer unique
  rows means fewer batches)
- raise `batch_size=` when the combined system prompt, attachment, and row JSON fit
  comfortably under the model's context window

Performance Considerations
--------------------------
AI validation is significantly slower than traditional validation methods due to API calls
to LLM providers. However, performance varies dramatically based on data characteristics:

**High Memoization Scenarios** (seconds to minutes):

- data with many duplicate rows in the selected columns
- low cardinality data (repeated patterns)
- small number of unique row combinations

**Low Memoization Scenarios** (minutes to hours):

- high cardinality data with mostly unique rows
- large datasets with few repeated patterns
- all or most rows requiring individual LLM evaluation

The row signature memoization optimization can reduce processing time significantly when
data has repetitive patterns. For datasets where every row is unique, expect longer
processing times similar to validating each row individually.

**Strategies to Reduce Processing Time**:

- test on data slices: define a sampling function like `def sample_1000(df): return df.head(1000)`
and use `pre=sample_1000` to validate on smaller samples
- filter relevant data: define filter functions like `def active_only(df): return df.filter(df["status"] == "active")`
and use `pre=active_only` to focus on a specific subset
- optimize column selection: use `columns_subset=` to include only the columns necessary
for validation
- start with smaller batches: begin with `batch_size=100` for testing, then increase
gradually
- reduce concurrency: lower `max_concurrent=1` if hitting rate limits
- use faster/cheaper models: consider using smaller or more efficient models for initial
testing before switching to more capable models

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
The following examples demonstrate how to use AI validation for different types of data
quality checks. These examples show both basic usage and more advanced configurations with
custom thresholds and actions.

**Basic AI validation example:**

This first example shows a simple validation scenario where we want to check that customer
records have both valid email addresses and non-empty names. Notice how we use
`columns_subset=` to focus only on the relevant columns, which improves both performance
and cost-effectiveness.

```python
import pointblank as pb
import polars as pl

# Sample data with email and name columns
tbl = pl.DataFrame({
    "email": ["john@example.com", "invalid-email", "jane@test.org"],
    "name": ["John Doe", "", "Jane Smith"],
    "age": [25, 30, 35]
})

# Validate using AI
validation = (
    pb.Validate(data=tbl)
    .prompt(
        prompt="Each row should have a valid email address and a non-empty name",
        columns_subset=["email", "name"],  # Only check these columns
        model="openai:gpt-6-luna",
    )
    .interrogate()
)

validation
```

In this example, the AI will identify that the second row fails validation because it has
an invalid email format (`"invalid-email"`) and the third row also fails because it has an
empty name field. The validation results will show 2 out of 3 rows failing the criteria.

**Advanced example with custom thresholds:**

This more sophisticated example demonstrates how to use AI validation with custom thresholds
and actions. Here we're validating phone number formats to ensure they include area codes,
which is a common data quality requirement for customer contact information.

```python
customer_data = pl.DataFrame({
    "customer_id": [1, 2, 3, 4, 5],
    "name": ["John Doe", "Jane Smith", "Bob Johnson", "Alice Brown", "Charlie Davis"],
    "phone_number": [
        "(555) 123-4567",  # Valid with area code
        "555-987-6543",    # Valid with area code
        "123-4567",        # Missing area code
        "(800) 555-1234",  # Valid with area code
        "987-6543"         # Missing area code
    ]
})

validation = (
    pb.Validate(data=customer_data)
    .prompt(
        prompt="Do all the phone numbers include an area code?",
        columns_subset="phone_number",  # Only check the `phone_number` column
        model="openai:gpt-6-luna",
        batch_size=500,
        max_concurrent=5,
        thresholds=pb.Thresholds(warning=0.1, error=0.2, critical=0.3),
        actions=pb.Actions(error="Too many phone numbers missing area codes.")
    )
    .interrogate()
)
```

This validation will identify that 2 out of 5 phone numbers (40%) are missing area codes,
which exceeds all threshold levels. The validation will trigger the specified error action
since the failure rate (40%) is above the error threshold (20%). The AI can recognize
various phone number formats and determine whether they include area codes.

**Multi-modal example with `attachments=`:**

Suppose you have a table of product descriptions and a brand-style PDF that describes the
approved tone and vocabulary. Pass the PDF as a global attachment so the LLM can compare
each description against it.

```python
validation = (
    pb.Validate(data=products)
    .prompt(
        prompt="Each product description must match the tone and vocabulary in the brand guide.",
        columns_subset=["description"],
        attachments=["docs/brand_guide.pdf"],
        model="anthropic:claude-opus-4-6",
    )
    .interrogate()
)
```

The brand guide is sent alongside the row JSON on every batch, so the LLM evaluates each
description with the same reference document in view.


## Aggregation Steps

These validation methods check aggregated column values (sums, averages, standard deviations) against fixed values or column references.



col_sum_gt(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column sum satisfy a greater than comparison?

The `col_sum_gt()` validation method checks whether the sum of a column is greater than a
target `value=`. The whole column is reduced to a single number and compared with the target
(`sum(column) > value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the sum is computed over the whole
table, so filter the table before passing it to `Validate` if you need the sum of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the sum to be computed.
value
    The target to compare the column sum against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    sum of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the sum of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sum_gt()`, the step passes when `sum(column) > target - lower`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the sum, so a column with no non-missing values sums to `0`. When the sum is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The sum of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the sum of column `a` is greater than `10`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_gt(columns="a", value=10)
    .interrogate()
)

validation
```

The sum of `a` is `15`, and `15 > 10` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_gt(columns=["a", "b"], value=10)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The sum of `b` is `10`, so step 2 fails. The report doesn't show the computed sum, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the sum computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `0`, with positive values above the
boundary and negative values below it. A difference of `0` fails here because the comparison is strict.

Next, a target that the sum of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_gt(columns="a", value=15.5)
    .col_sum_gt(columns="a", value=15.5, tol=0.05)
    .interrogate()
)

validation
```

The sum of `a` is `15`, so `15 > 15.5` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.775`), so the sum only has to be greater than `14.725`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_gt(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the sum of `b` (`10`) rather than a fixed number. The sum of `a` is `15`, so `15 > 10` holds and the step passes. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sum_gt(columns="revenue")
    .col_sum_gt(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `446.7 > 451` (the sum of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sum_lt(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column sum satisfy a less than comparison?

The `col_sum_lt()` validation method checks whether the sum of a column is less than a
target `value=`. The whole column is reduced to a single number and compared with the target
(`sum(column) < value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the sum is computed over the whole
table, so filter the table before passing it to `Validate` if you need the sum of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the sum to be computed.
value
    The target to compare the column sum against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    sum of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the sum of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sum_lt()`, the step passes when `sum(column) < target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the sum, so a column with no non-missing values sums to `0`. When the sum is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The sum of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the sum of column `a` is less than `20`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_lt(columns="a", value=20)
    .interrogate()
)

validation
```

The sum of `a` is `15`, and `15 < 20` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_lt(columns=["a", "b"], value=20)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The sum of `b` is `10`, so step 2 passes. The report doesn't show the computed sum, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the sum computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-10`, with positive values above the
boundary and negative values below it.

Next, a target that the sum of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_lt(columns="a", value=14.5)
    .col_sum_lt(columns="a", value=14.5, tol=0.05)
    .interrogate()
)

validation
```

The sum of `a` is `15`, so `15 < 14.5` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.725`), so the sum only has to be less than `15.225`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_lt(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the sum of `b` (`10`) rather than a fixed number. The sum of `a` is `15`, so `15 < 10` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sum_lt(columns="revenue")
    .col_sum_lt(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `446.7 < 451` (the sum of `revenue` in `current`, then in `baseline`), so it passes. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sum_ge(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column sum satisfy a greater than or equal to comparison?

The `col_sum_ge()` validation method checks whether the sum of a column is at least a
target `value=`. The whole column is reduced to a single number and compared with the target
(`sum(column) >= value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the sum is computed over the whole
table, so filter the table before passing it to `Validate` if you need the sum of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the sum to be computed.
value
    The target to compare the column sum against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    sum of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the sum of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sum_ge()`, the step passes when `sum(column) >= target - lower`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the sum, so a column with no non-missing values sums to `0`. When the sum is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The sum of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the sum of column `a` is at least `15`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_ge(columns="a", value=15)
    .interrogate()
)

validation
```

The sum of `a` is `15`, and `15 >= 15` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_ge(columns=["a", "b"], value=15)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The sum of `b` is `10`, so step 2 fails. The report doesn't show the computed sum, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the sum computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-5`, with positive values above the
boundary and negative values below it.

Next, a target that the sum of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_ge(columns="a", value=15.5)
    .col_sum_ge(columns="a", value=15.5, tol=0.05)
    .interrogate()
)

validation
```

The sum of `a` is `15`, so `15 >= 15.5` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.775`), so the sum only has to be at least `14.725`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_ge(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the sum of `b` (`10`) rather than a fixed number. The sum of `a` is `15`, so `15 >= 10` holds and the step passes. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sum_ge(columns="revenue")
    .col_sum_ge(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `446.7 >= 451` (the sum of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sum_le(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column sum satisfy a less than or equal to comparison?

The `col_sum_le()` validation method checks whether the sum of a column is at most a
target `value=`. The whole column is reduced to a single number and compared with the target
(`sum(column) <= value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the sum is computed over the whole
table, so filter the table before passing it to `Validate` if you need the sum of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the sum to be computed.
value
    The target to compare the column sum against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    sum of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the sum of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sum_le()`, the step passes when `sum(column) <= target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the sum, so a column with no non-missing values sums to `0`. When the sum is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The sum of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the sum of column `a` is at most `15`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_le(columns="a", value=15)
    .interrogate()
)

validation
```

The sum of `a` is `15`, and `15 <= 15` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_le(columns=["a", "b"], value=15)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The sum of `b` is `10`, so step 2 passes. The report doesn't show the computed sum, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the sum computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-5`, with positive values above the
boundary and negative values below it.

Next, a target that the sum of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_le(columns="a", value=14.5)
    .col_sum_le(columns="a", value=14.5, tol=0.05)
    .interrogate()
)

validation
```

The sum of `a` is `15`, so `15 <= 14.5` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.725`), so the sum only has to be at most `15.225`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_le(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the sum of `b` (`10`) rather than a fixed number. The sum of `a` is `15`, so `15 <= 10` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sum_le(columns="revenue")
    .col_sum_le(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `446.7 <= 451` (the sum of `revenue` in `current`, then in `baseline`), so it passes. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sum_eq(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column sum satisfy an equal to comparison?

The `col_sum_eq()` validation method checks whether the sum of a column equals a
target `value=`. The whole column is reduced to a single number and compared with the target
(`sum(column) == value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the sum is computed over the whole
table, so filter the table before passing it to `Validate` if you need the sum of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the sum to be computed.
value
    The target to compare the column sum against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    sum of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the sum of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sum_eq()`, the step passes when `target - lower <= sum(column) <= target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the sum, so a column with no non-missing values sums to `0`. When the sum is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The sum of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the sum of column `a` equals `15`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_eq(columns="a", value=15)
    .interrogate()
)

validation
```

The sum of `a` is `15`, and `15 == 15` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_eq(columns=["a", "b"], value=15)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The sum of `b` is `10`, so step 2 fails. The report doesn't show the computed sum, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the sum computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-5`, with positive values above the
boundary and negative values below it.

Next, a target that the sum of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_eq(columns="a", value=15.5)
    .col_sum_eq(columns="a", value=15.5, tol=0.05)
    .interrogate()
)

validation
```

The sum of `a` is `15`, so `15 == 15.5` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.775`), so any sum from `14.725` to `16.275` is accepted, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sum_eq(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the sum of `b` (`10`) rather than a fixed number. The sum of `a` is `15`, so `15 == 10` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sum_eq(columns="revenue")
    .col_sum_eq(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `446.7 == 451` (the sum of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_avg_gt(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column average satisfy a greater than comparison?

The `col_avg_gt()` validation method checks whether the average of a column is greater than a
target `value=`. The whole column is reduced to a single number and compared with the target
(`average(column) > value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the average is computed over the whole
table, so filter the table before passing it to `Validate` if you need the average of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the average to be computed.
value
    The target to compare the column average against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    average of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the average of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_avg_gt()`, the step passes when `average(column) > target - lower`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the average, so a column with no non-missing values has no average. When the average is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The average of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the average of column `a` is greater than `2`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_gt(columns="a", value=2)
    .interrogate()
)

validation
```

The average of `a` is `3`, and `3 > 2` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_gt(columns=["a", "b"], value=2)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The average of `b` is `2`, so step 2 fails. The report doesn't show the computed average, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the average computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `0`, with positive values above the
boundary and negative values below it. A difference of `0` fails here because the comparison is strict.

Next, a target that the average of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_gt(columns="a", value=3.1)
    .col_avg_gt(columns="a", value=3.1, tol=0.05)
    .interrogate()
)

validation
```

The average of `a` is `3`, so `3 > 3.1` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.155`), so the average only has to be greater than `2.945`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_gt(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the average of `b` (`2`) rather than a fixed number. The average of `a` is `3`, so `3 > 2` holds and the step passes. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_avg_gt(columns="revenue")
    .col_avg_gt(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `111.675 > 112.75` (the average of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_avg_lt(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column average satisfy a less than comparison?

The `col_avg_lt()` validation method checks whether the average of a column is less than a
target `value=`. The whole column is reduced to a single number and compared with the target
(`average(column) < value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the average is computed over the whole
table, so filter the table before passing it to `Validate` if you need the average of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the average to be computed.
value
    The target to compare the column average against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    average of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the average of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_avg_lt()`, the step passes when `average(column) < target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the average, so a column with no non-missing values has no average. When the average is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The average of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the average of column `a` is less than `4`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_lt(columns="a", value=4)
    .interrogate()
)

validation
```

The average of `a` is `3`, and `3 < 4` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_lt(columns=["a", "b"], value=4)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The average of `b` is `2`, so step 2 passes. The report doesn't show the computed average, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the average computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-2`, with positive values above the
boundary and negative values below it.

Next, a target that the average of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_lt(columns="a", value=2.9)
    .col_avg_lt(columns="a", value=2.9, tol=0.05)
    .interrogate()
)

validation
```

The average of `a` is `3`, so `3 < 2.9` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.145`), so the average only has to be less than `3.045`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_lt(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the average of `b` (`2`) rather than a fixed number. The average of `a` is `3`, so `3 < 2` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_avg_lt(columns="revenue")
    .col_avg_lt(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `111.675 < 112.75` (the average of `revenue` in `current`, then in `baseline`), so it passes. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_avg_ge(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column average satisfy a greater than or equal to comparison?

The `col_avg_ge()` validation method checks whether the average of a column is at least a
target `value=`. The whole column is reduced to a single number and compared with the target
(`average(column) >= value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the average is computed over the whole
table, so filter the table before passing it to `Validate` if you need the average of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the average to be computed.
value
    The target to compare the column average against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    average of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the average of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_avg_ge()`, the step passes when `average(column) >= target - lower`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the average, so a column with no non-missing values has no average. When the average is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The average of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the average of column `a` is at least `3`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_ge(columns="a", value=3)
    .interrogate()
)

validation
```

The average of `a` is `3`, and `3 >= 3` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_ge(columns=["a", "b"], value=3)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The average of `b` is `2`, so step 2 fails. The report doesn't show the computed average, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the average computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-1`, with positive values above the
boundary and negative values below it.

Next, a target that the average of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_ge(columns="a", value=3.1)
    .col_avg_ge(columns="a", value=3.1, tol=0.05)
    .interrogate()
)

validation
```

The average of `a` is `3`, so `3 >= 3.1` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.155`), so the average only has to be at least `2.945`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_ge(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the average of `b` (`2`) rather than a fixed number. The average of `a` is `3`, so `3 >= 2` holds and the step passes. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_avg_ge(columns="revenue")
    .col_avg_ge(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `111.675 >= 112.75` (the average of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_avg_le(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column average satisfy a less than or equal to comparison?

The `col_avg_le()` validation method checks whether the average of a column is at most a
target `value=`. The whole column is reduced to a single number and compared with the target
(`average(column) <= value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the average is computed over the whole
table, so filter the table before passing it to `Validate` if you need the average of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the average to be computed.
value
    The target to compare the column average against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    average of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the average of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_avg_le()`, the step passes when `average(column) <= target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the average, so a column with no non-missing values has no average. When the average is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The average of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the average of column `a` is at most `3`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_le(columns="a", value=3)
    .interrogate()
)

validation
```

The average of `a` is `3`, and `3 <= 3` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_le(columns=["a", "b"], value=3)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The average of `b` is `2`, so step 2 passes. The report doesn't show the computed average, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the average computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-1`, with positive values above the
boundary and negative values below it.

Next, a target that the average of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_le(columns="a", value=2.9)
    .col_avg_le(columns="a", value=2.9, tol=0.05)
    .interrogate()
)

validation
```

The average of `a` is `3`, so `3 <= 2.9` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.145`), so the average only has to be at most `3.045`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_le(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the average of `b` (`2`) rather than a fixed number. The average of `a` is `3`, so `3 <= 2` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_avg_le(columns="revenue")
    .col_avg_le(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `111.675 <= 112.75` (the average of `revenue` in `current`, then in `baseline`), so it passes. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_avg_eq(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column average satisfy an equal to comparison?

The `col_avg_eq()` validation method checks whether the average of a column equals a
target `value=`. The whole column is reduced to a single number and compared with the target
(`average(column) == value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation. The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the average is computed over the whole
table, so filter the table before passing it to `Validate` if you need the average of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the average to be computed.
value
    The target to compare the column average against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    average of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the average of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_avg_eq()`, the step passes when `target - lower <= average(column) <= target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the average, so a column with no non-missing values has no average. When the average is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The average of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the average of column `a` equals `3`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_eq(columns="a", value=3)
    .interrogate()
)

validation
```

The average of `a` is `3`, and `3 == 3` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_eq(columns=["a", "b"], value=3)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The average of `b` is `2`, so step 2 fails. The report doesn't show the computed average, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the average computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-1`, with positive values above the
boundary and negative values below it.

Next, a target that the average of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_eq(columns="a", value=3.1)
    .col_avg_eq(columns="a", value=3.1, tol=0.05)
    .interrogate()
)

validation
```

The average of `a` is `3`, so `3 == 3.1` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.155`), so any average from `2.945` to `3.255` is accepted, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_avg_eq(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the average of `b` (`2`) rather than a fixed number. The average of `a` is `3`, so `3 == 2` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_avg_eq(columns="revenue")
    .col_avg_eq(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `111.675 == 112.75` (the average of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sd_gt(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column standard deviation satisfy a greater than comparison?

The `col_sd_gt()` validation method checks whether the standard deviation of a column is greater than a
target `value=`. The whole column is reduced to a single number and compared with the target
(`standard deviation(column) > value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation, and the standard deviation is the sample standard deviation (dividing by `n - 1`). The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the standard deviation is computed over the whole
table, so filter the table before passing it to `Validate` if you need the standard deviation of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the standard deviation to be computed.
value
    The target to compare the column standard deviation against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    standard deviation of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the standard deviation of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sd_gt()`, the step passes when `standard deviation(column) > target - lower`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the standard deviation, so a column with fewer than two non-missing values has none. When the standard deviation is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The standard deviation of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the standard deviation of column `a` is greater than `1`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_gt(columns="a", value=1)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, and `1.58114 > 1` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_gt(columns=["a", "b"], value=1)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The standard deviation of `b` is `0`, so step 2 fails. The report doesn't show the computed standard deviation, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the standard deviation computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-1`, with positive values above the
boundary and negative values below it.

Next, a target that the standard deviation of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_gt(columns="a", value=1.65)
    .col_sd_gt(columns="a", value=1.65, tol=0.05)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, so `1.58114 > 1.65` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.0825`), so the standard deviation only has to be greater than `1.5675`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_gt(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the standard deviation of `b` (`0`) rather than a fixed number. The standard deviation of `a` is `1.58114`, so `1.58114 > 0` holds and the step passes. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sd_gt(columns="revenue")
    .col_sd_gt(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `25.0684 > 26.1454` (the standard deviation of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sd_lt(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column standard deviation satisfy a less than comparison?

The `col_sd_lt()` validation method checks whether the standard deviation of a column is less than a
target `value=`. The whole column is reduced to a single number and compared with the target
(`standard deviation(column) < value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation, and the standard deviation is the sample standard deviation (dividing by `n - 1`). The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the standard deviation is computed over the whole
table, so filter the table before passing it to `Validate` if you need the standard deviation of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the standard deviation to be computed.
value
    The target to compare the column standard deviation against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    standard deviation of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the standard deviation of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sd_lt()`, the step passes when `standard deviation(column) < target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the standard deviation, so a column with fewer than two non-missing values has none. When the standard deviation is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The standard deviation of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the standard deviation of column `a` is less than `2`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_lt(columns="a", value=2)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, and `1.58114 < 2` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_lt(columns=["a", "b"], value=2)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The standard deviation of `b` is `0`, so step 2 passes. The report doesn't show the computed standard deviation, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the standard deviation computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-2`, with positive values above the
boundary and negative values below it.

Next, a target that the standard deviation of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_lt(columns="a", value=1.52)
    .col_sd_lt(columns="a", value=1.52, tol=0.05)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, so `1.58114 < 1.52` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.076`), so the standard deviation only has to be less than `1.596`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_lt(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the standard deviation of `b` (`0`) rather than a fixed number. The standard deviation of `a` is `1.58114`, so `1.58114 < 0` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sd_lt(columns="revenue")
    .col_sd_lt(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `25.0684 < 26.1454` (the standard deviation of `revenue` in `current`, then in `baseline`), so it passes. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sd_ge(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column standard deviation satisfy a greater than or equal to comparison?

The `col_sd_ge()` validation method checks whether the standard deviation of a column is at least a
target `value=`. The whole column is reduced to a single number and compared with the target
(`standard deviation(column) >= value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation, and the standard deviation is the sample standard deviation (dividing by `n - 1`). The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the standard deviation is computed over the whole
table, so filter the table before passing it to `Validate` if you need the standard deviation of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the standard deviation to be computed.
value
    The target to compare the column standard deviation against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    standard deviation of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the standard deviation of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sd_ge()`, the step passes when `standard deviation(column) >= target - lower`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the standard deviation, so a column with fewer than two non-missing values has none. When the standard deviation is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The standard deviation of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the standard deviation of column `a` is at least `1`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_ge(columns="a", value=1)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, and `1.58114 >= 1` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_ge(columns=["a", "b"], value=1)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The standard deviation of `b` is `0`, so step 2 fails. The report doesn't show the computed standard deviation, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the standard deviation computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-1`, with positive values above the
boundary and negative values below it.

Next, a target that the standard deviation of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_ge(columns="a", value=1.65)
    .col_sd_ge(columns="a", value=1.65, tol=0.05)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, so `1.58114 >= 1.65` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.0825`), so the standard deviation only has to be at least `1.5675`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_ge(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the standard deviation of `b` (`0`) rather than a fixed number. The standard deviation of `a` is `1.58114`, so `1.58114 >= 0` holds and the step passes. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sd_ge(columns="revenue")
    .col_sd_ge(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `25.0684 >= 26.1454` (the standard deviation of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sd_le(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column standard deviation satisfy a less than or equal to comparison?

The `col_sd_le()` validation method checks whether the standard deviation of a column is at most a
target `value=`. The whole column is reduced to a single number and compared with the target
(`standard deviation(column) <= value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation, and the standard deviation is the sample standard deviation (dividing by `n - 1`). The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the standard deviation is computed over the whole
table, so filter the table before passing it to `Validate` if you need the standard deviation of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the standard deviation to be computed.
value
    The target to compare the column standard deviation against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    standard deviation of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the standard deviation of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sd_le()`, the step passes when `standard deviation(column) <= target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the standard deviation, so a column with fewer than two non-missing values has none. When the standard deviation is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The standard deviation of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the standard deviation of column `a` is at most `2`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_le(columns="a", value=2)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, and `1.58114 <= 2` holds, so the step's one test unit passes.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_le(columns=["a", "b"], value=2)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The standard deviation of `b` is `0`, so step 2 passes. The report doesn't show the computed standard deviation, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the standard deviation computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-2`, with positive values above the
boundary and negative values below it.

Next, a target that the standard deviation of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_le(columns="a", value=1.52)
    .col_sd_le(columns="a", value=1.52, tol=0.05)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, so `1.58114 <= 1.52` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.076`), so the standard deviation only has to be at most `1.596`, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_le(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the standard deviation of `b` (`0`) rather than a fixed number. The standard deviation of `a` is `1.58114`, so `1.58114 <= 0` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sd_le(columns="revenue")
    .col_sd_le(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `25.0684 <= 26.1454` (the standard deviation of `revenue` in `current`, then in `baseline`), so it passes. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.

col_sd_eq(self: 'Validate', columns: 'str | Collection[str]', value: 'float | int | Column | ReferenceColumn | None' = None, tol: 'float' = 0, thresholds: 'int | float | bool | tuple | dict | Thresholds | None' = None, brief: 'str | bool | None' = None, actions: 'Actions | None' = None, active: 'bool | Callable' = True, dimension: 'str | None' = None) -> 'Validate'

Does the column standard deviation satisfy an equal to comparison?

The `col_sd_eq()` validation method checks whether the standard deviation of a column equals a
target `value=`. The whole column is reduced to a single number and compared with the target
(`standard deviation(column) == value`), so the step has exactly one test unit, which
either passes or fails.

Aggregate checks catch problems that row-level checks can't see. Every value in a column can be
within its allowed range while the column as a whole is wrong: a daily total that doesn't match
the control figure from the source system, an average that has shifted after a change upstream,
or a sensor whose readings have stopped varying (a standard deviation near zero). Use row-level
methods like [`col_vals_between()`](`pointblank.Validate.col_vals_between`) to check individual
values, and an aggregate method to check what they add up to. The target can be a fixed number,
the same aggregate of another column, or the same aggregate of a column in reference data. The
last option makes these methods a good fit for comparing a new load of a table with the last
one that was known to be good.

Null values are left out of the computation, and the standard deviation is the sample standard deviation (dividing by `n - 1`). The column must be numeric, and a
non-numeric column raises an error during interrogation. Unlike the row-level methods, the
aggregate methods don't take `pre=` or `segments=`: the standard deviation is computed over the whole
table, so filter the table before passing it to `Validate` if you need the standard deviation of a
subset.

Parameters
----------
columns
    A single column or a list of columns to validate. If multiple columns are supplied,
    there will be a separate validation step generated for each column. The columns must
    contain numeric data for the standard deviation to be computed.
value
    The target to compare the column standard deviation against. This can be: (1) a number, (2) a
    [`col()`](`pointblank.col`) object naming another column, in which case the target is the
    standard deviation of that column in the same table, (3) a [`ref()`](`pointblank.ref`) object naming
    a column in the reference data (set with `Validate(reference=)`), in which case the target
    is the standard deviation of that column there, or (4) `None` (the default), which is shorthand for
    `ref()` of the same column. Using `None` without reference data raises a `ValueError`.
    `col()` takes a single column name here, not a column selector.
tol
    The tolerance around the target. The default is `0`, an exact comparison. A value below
    `1` is a fraction of the target's size (`tol=0.05` allows 5%), and a value of `1` or more
    is an absolute amount (`tol=2` allows `2` either way). A tuple of two values,
    `(lower, upper)`, sets the two sides separately, with each value read the same way. See
    the *Understanding Tolerance* section for how the tolerance applies to each comparison.
thresholds
    Failure threshold levels for the step. They override any thresholds set in
    `Validate(thresholds=)`. Because the step has a single test unit, every level that's set
    is reached when the step fails. See the *Thresholds* section for details.
brief
    An optional brief description of the validation step that will be displayed in the
    reporting table. You can use the templating elements like `"{step}"` to insert
    the step number, or `"{auto}"` to include an automatically generated brief. If `True`
    the entire brief will be automatically generated. If `None` (the default) then there
    won't be a brief.
actions
    Optional actions to take when the validation step meets or exceeds any set threshold
    levels. If provided, the [`Actions`](`pointblank.Actions`) class should be used to
    define the actions.
active
    A boolean value or callable that determines whether the validation step should be
    active. Using `False` will make the validation step inactive (still reporting its
    presence and keeping indexes for the steps unchanged). A callable can also be
    provided; it will receive the data table as its single argument and must return a
    boolean value.
    Inspection functions like [`has_columns()`](`pointblank.has_columns`) and
    [`has_rows()`](`pointblank.has_rows`) can be used here to conditionally activate
    a step based on properties of the target table.
dimension
    An optional data quality dimension to categorize this validation step for health
    scoring. One of `"completeness"`, `"validity"`, `"uniqueness"`, `"consistency"`,
    `"timeliness"`, or `"volume"` (or any custom string). If `None` (the default), the step is
    assigned to `"validity"`, like the other aggregate checks. A check that's really about how
    much data there is (such as a sum of quantities against an expected total) might be better
    labeled `"volume"` or `"consistency"`. This label appears in the validation report and
    feeds the overall and per-dimension health scores.

Returns
-------
Validate
    The `Validate` object with the added validation step.

Understanding Tolerance
-----------------------
Exact comparisons are often too strict for aggregates. A sum of floating-point values can
differ from the expected total in the last decimal places, and an average or standard deviation
computed from real data is rarely a round number. The `tol=` parameter turns the target into a
range:

- with `tol=0.05`, the allowance is 5% of the target (`0.05 * abs(target)`)
- with `tol=2`, the allowance is `2`
- with `tol=(0.1, 0)`, the allowance is 10% below the target and nothing above it

Because values below `1` are relative, an absolute allowance smaller than `1` can't be given
directly. Express it as a fraction of the target instead (for a target of `50`, `tol=0.01`
allows `0.5`). A relative tolerance has no effect when the target is `0`.

How the allowance is used depends on the comparison. For `eq`, the aggregate passes if it's
anywhere from `target - lower` to `target + upper`. For `gt` and `ge`, it's compared with
`target - lower`, so the tolerance lowers the bar. For `lt` and `le`, it's compared with
`target + upper`, which raises the ceiling. In those four cases the other side of the tolerance
isn't used. For `col_sd_eq()`, the step passes when `target - lower <= standard deviation(column) <= target + upper`.

Missing Values and Decimal Columns
----------------------------------
Missing values are left out of the standard deviation, so a column with fewer than two non-missing values has none. When the standard deviation is missing or `NaN` (on Polars, for instance, `NaN`
values make it `NaN`), or the column isn't numeric, the step can't be evaluated. It's reported
as an evaluation error, with a note saying why, rather than as passing or failing.

The standard deviation of a `Decimal` column is compared as a `float`, and `value=` can be given as a
`Decimal` too (e.g., `value=Decimal("6.60")`).

Thresholds
----------
The `thresholds=` parameter is used to set the failure-condition levels for the validation
step. If they are set here at the step level, these thresholds will override any thresholds
set at the global level in `Validate(thresholds=...)`.

An aggregate step has a single test unit, so the fraction of failing test units is either `0`
or `1`. Any level that's set, whether as a count (such as `1`) or as a fraction (such as
`0.1`), is reached when the step fails, so the levels can't be graded within one step. Set
only the level that a failure should reach: `thresholds=pb.Thresholds(error=1)` makes a failure
an 'error' without a 'warning', and `thresholds=(1, 1, 1)` makes it reach all three levels.

Thresholds can be defined using one of these input schemes:

1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create
thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is
the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and
'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units
for the 'warning' level only

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples, we'll use a simple Polars DataFrame with two numeric columns. The table is
shown below:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [2, 2, 2, 2, 2],
    }
)

pb.preview(tbl)
```

Let's validate that the standard deviation of column `a` equals `1.58`:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_eq(columns="a", value=1.58)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, so `1.58114 == 1.58` doesn't hold and the step's one test unit fails. Exact equality is rarely a good test for a computed value like this one, and the tolerance example below shows how to allow some leeway.

When validating multiple columns, each column gets its own validation step:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_eq(columns=["a", "b"], value=1.58)
    .interrogate()
)

validation
```

Each column gets its own step: step 1 is for `a` and step 2 is for `b`. The standard deviation of `b` is `0`, so step 2 fails. The report doesn't show the computed standard deviation, but the step report does:

```{python}
validation.get_step_report(i=2)
```

The step report shows the standard deviation computed for `b` (*ACTUAL*), the target (*EXPECTED*),
the tolerance (*TOL*, none here), and *DIFFERENCE*, which is how far the computed value is from
the boundary it had to meet. Here that's `-1.58`, with positive values above the
boundary and negative values below it.

Next, a target that the standard deviation of `a` just misses, first with an exact comparison and then
with a 5% tolerance:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_eq(columns="a", value=1.65)
    .col_sd_eq(columns="a", value=1.65, tol=0.05)
    .interrogate()
)

validation
```

The standard deviation of `a` is `1.58114`, so `1.58114 == 1.65` doesn't hold, if only just, and step 1 fails. In step 2, `tol=0.05` allows 5% of the target (`0.0825`), so any standard deviation from `1.5675` to `1.7325` is accepted, and the step passes.

The target can also be the same aggregate of another column, given with
[`col()`](`pointblank.col`):

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_sd_eq(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

Here the target is the standard deviation of `b` (`0`) rather than a fixed number. The standard deviation of `a` is `1.58114`, so `1.58114 == 0` does not hold and the step fails. Both aggregates are computed from the same table when the validation is interrogated.

Finally, the target can come from reference data, such as an earlier load of the same table
that passed its checks:

```{python}
current = pl.DataFrame({"revenue": [120.0, 95.5, 143.2, 88.0]})
baseline = pl.DataFrame({"revenue": [110.0, 101.0, 150.0, 90.0]})

validation = (
    pb.Validate(data=current, reference=baseline)
    .col_sd_eq(columns="revenue")
    .col_sd_eq(columns="revenue", value=pb.ref("revenue"), tol=0.05)
    .interrogate()
)

validation
```

With `reference=` set, leaving out `value=` compares against the same column in the reference table. Step 1 checks `25.0684 == 26.1454` (the standard deviation of `revenue` in `current`, then in `baseline`), so it fails. Step 2 names the target column with [`ref()`](`pointblank.ref`), which you'd need if the column had a different name in the reference table, and allows a 5% tolerance, so it passes. Comparing each new load of a table with the last one known to be good like this catches drift that no single row would reveal.


## Column Selection

Use the `col()` function along with column selection helpers to flexibly select columns for validation. Combine `col()` with `starts_with()`, `matches()`, etc. for selecting multiple target columns.



col(exprs: 'str | ColumnSelector | ColumnSelectorNarwhals | nw.selectors.Selector') -> 'Column | ColumnLiteral | ColumnSelectorNarwhals'

Helper function for referencing a column in the input table.

Many of the validation methods (i.e., `col_vals_*()` methods) in Pointblank have a `value=`
argument. These validations are comparisons between column values and a literal value, or,
between column values and adjacent values in another column. The `col()` helper function is used
to specify that it is a column being referenced, not a literal value.

The `col()` doesn't check that the column exists in the input table. It acts to signal that the
value being compared is a column value. During validation (i.e., when
[`interrogate()`](`pointblank.Validate.interrogate`) is called), Pointblank will then check that
the column exists in the input table.

For creating expressions to use with the `conjointly()` validation method, use the
[`expr_col()`](`pointblank.expr_col`) function instead.

Parameters
----------
exprs
    Either the name of a single column in the target table, provided as a string, or, an
    expression involving column selector functions (e.g., `starts_with("a")`,
    `ends_with("e") | starts_with("a")`, etc.).

Returns
-------
Column | ColumnLiteral | ColumnSelectorNarwhals:
    A column object or expression representing the column reference.

Usage with the `columns=` Argument
-----------------------------------
The `col()` function can be used in the `columns=` argument of the following validation methods:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_exists()`](`pointblank.Validate.col_exists`)

If specifying a single column with certainty (you have the exact name), `col()` is not necessary
since you can just pass the column name as a string (though it is still valid to use
`col("column_name")`, if preferred). However, if you want to select columns based on complex
logic involving multiple column selector functions (e.g., columns that start with `"a"` but
don't end with `"e"`), you need to use `col()` to wrap expressions involving column selector
functions and logical operators such as `&`, `|`, `-`, and `~`.

Here is an example of such usage with the [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
validation method:

```python
col_vals_gt(columns=col(starts_with("a") & ~ends_with("e")), value=10)
```

If using only a single column selector function, you can pass the function directly to the
`columns=` argument of the validation method, or, you can use `col()` to wrap the function
(either is valid though the first is more concise). Here is an example of that simpler usage:

```python
col_vals_gt(columns=starts_with("a"), value=10)
```

Usage with the `value=`, `left=`, and `right=` Arguments
--------------------------------------------------------
The `col()` function can be used in the `value=` argument of the following validation methods

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)

and in the `left=` and `right=` arguments (either or both) of these two validation methods

- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)

You cannot use column selector functions such as [`starts_with()`](`pointblank.starts_with`)
in either of the `value=`, `left=`, or `right=` arguments since there would be no guarantee that
a single column will be resolved from the target table with this approach. The `col()` function
is used to signal that the value being compared is a column value and not a literal value.

Available Selectors
-------------------
There is a collection of selectors available in pointblank, allowing you to select columns based
on attributes of column names and positions. The selectors are:

- [`starts_with()`](`pointblank.starts_with`)
- [`ends_with()`](`pointblank.ends_with`)
- [`contains()`](`pointblank.contains`)
- [`matches()`](`pointblank.matches`)
- [`everything()`](`pointblank.everything`)
- [`first_n()`](`pointblank.first_n`)
- [`last_n()`](`pointblank.last_n`)

Alternatively, we support selectors from the Narwhals library! Those selectors can additionally
take advantage of the data types of the columns. The selectors are:

- `boolean()`
- `by_dtype()`
- `categorical()`
- `matches()`
- `numeric()`
- `string()`

Have a look at the [Narwhals API documentation on selectors](https://narwhals-dev.github.io/narwhals/api-reference/selectors/)
for more information.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have a table with columns `a` and `b` and we'd like to validate that the values in
column `a` are greater than the values in column `b`. We can use the `col()` helper function to
reference the comparison column when creating the validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 6, 5, 7, 6, 5],
        "b": [4, 2, 3, 3, 4, 3],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=pb.col("b"))
    .interrogate()
)

validation
```

From results of the validation table it can be seen that values in `a` were greater than values
in `b` for every row (or test unit). Using `value=pb.col("b")` specified that the greater-than
comparison is across columns, not with a fixed literal value.

If you want to select an arbitrary set of columns upon which to base a validation, you can use
column selector functions (e.g., [`starts_with()`](`pointblank.starts_with`),
[`ends_with()`](`pointblank.ends_with`), etc.) to specify columns in the `columns=` argument of
a validation method. Let's use the [`starts_with()`](`pointblank.starts_with`) column selector
function to select columns that start with `"paid"` and validate that the values in those
columns are greater than `10`.

```{python}
tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "paid_2021": [16.32, 16.25, 15.75],
        "paid_2022": [18.62, 16.95, 18.25],
        "person_id": ["A123", "B456", "C789"],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns=pb.col(pb.starts_with("paid")), value=10)
    .interrogate()
)

validation
```

In the above example the `col()` function contains the invocation of the
[`starts_with()`](`pointblank.starts_with`) column selector function. This is not strictly
necessary when using a single column selector function, so `columns=pb.starts_with("paid")`
would be equivalent usage here. However, the use of `col()` is required when using multiple
column selector functions with logical operators. Here is an example of that more complex usage:

```{python}
tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "hours_2022": [160, 180, 160],
        "hours_2023": [182, 168, 175],
        "hours_2024": [200, 165, 190],
        "paid_2022": [18.62, 16.95, 18.25],
        "paid_2023": [19.29, 17.75, 18.35],
        "paid_2024": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(
        columns=pb.col(pb.starts_with("paid") & pb.matches("2023|2024")),
        value=10
    )
    .interrogate()
)

validation
```

In the above example the `col()` function contains the invocation of the
[`starts_with()`](`pointblank.starts_with`) and [`matches()`](`pointblank.matches`) column
selector functions, combined with the `&` operator. This is necessary to specify the set of
columns that start with `"paid"` *and* match the text `"2023"` or `"2024"`.

If you'd like to take advantage of Narwhals selectors, that's also possible. Here is an example
of using the `numeric()` column selector function to select all numeric columns for validation,
checking that their values are greater than `0`.

```{python}
import narwhals.selectors as ncs

tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "hours_2022": [160, 180, 160],
        "hours_2023": [182, 168, 175],
        "hours_2024": [200, 165, 190],
        "paid_2022": [18.62, 16.95, 18.25],
        "paid_2023": [19.29, 17.75, 18.35],
        "paid_2024": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_ge(columns=pb.col(ncs.numeric()), value=0)
    .interrogate()
)

validation
```

In the above example the `col()` function contains the invocation of the `numeric()` column
selector function from Narwhals. As with the other selectors, this is not strictly necessary
when using a single column selector, so `columns=ncs.numeric()` would also be fine here.

Narwhals selectors can also use operators to combine multiple selectors. Here is an example of
using the `numeric()` and [`matches()`](`pointblank.matches`) selectors together to select all
numeric columns that fit a specific pattern.

```{python}
tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "2022_status": ["ft", "ft", "pt"],
        "2023_status": ["ft", "pt", "ft"],
        "2024_status": ["ft", "pt", "ft"],
        "2022_pay_total": [18.62, 16.95, 18.25],
        "2023_pay_total": [19.29, 17.75, 18.35],
        "2024_pay_total": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_lt(columns=pb.col(ncs.numeric() & ncs.matches("2023|2024")), value=30)
    .interrogate()
)

validation
```

In the above example the `col()` function contains the invocation of the `numeric()` and
[`matches()`](`pointblank.matches`) column selector functions from Narwhals, combined with the
`&` operator. This is necessary to specify the set of columns that are numeric *and* match the
text `"2023"` or `"2024"`.

See Also
--------
expr_col : Create a column expression for use in `conjointly()`.

starts_with(text: 'str', case_sensitive: 'bool' = False) -> 'StartsWith'

Select columns that start with specified text.

Many validation methods have a `columns=` argument that can be used to specify the columns for
validation (e.g., [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`), etc.). The `starts_with()` selector
function can be used to select one or more columns that start with some specified text. So if
the set of table columns consists of

`[name_first, name_last, age, address]`

and you want to validate columns that start with `"name"`, you can use
`columns=starts_with("name")`. This will select the `name_first` and `name_last` columns.

There will be a validation step created for every resolved column. Note that if there aren't any
columns resolved from using `starts_with()` (or any other expression using selector functions),
the validation step will fail to be evaluated during the interrogation process. Such a failure
to evaluate will be reported in the validation results but it won't affect the interrogation
process overall (i.e., the process won't be halted).

Parameters
----------
text
    The text that the column name should start with.
case_sensitive
    Whether column names should be treated as case-sensitive. The default is `False`.

Returns
-------
StartsWith
    A `StartsWith` object, which can be used to select columns that start with the specified
    text.

Relevant Validation Methods where `starts_with()` can be Used
-------------------------------------------------------------
This selector function can be used in the `columns=` argument of the following validation
methods:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_exists()`](`pointblank.Validate.col_exists`)

The `starts_with()` selector function doesn't need to be used in isolation. Read the next
section for information on how to compose it with other column selectors for more refined ways
to select columns.

Additional Flexibilty through Composition with Other Column Selectors
---------------------------------------------------------------------
The `starts_with()` function can be composed with other column selectors to create fine-grained
column selections. For example, to select columns that start with `"a"` and end with `"e"`, you
can use the `starts_with()` and [`ends_with()`](`pointblank.ends_with`) functions together. The
only condition is that the expressions are wrapped in the [`col()`](`pointblank.col`) function,
like this:

```python
col(starts_with("a") & ends_with("e"))
```

There are four operators that can be used to compose column selectors:

- `&` (*and*)
- `|` (*or*)
- `-` (*difference*)
- `~` (*not*)

The `&` operator is used to select columns that satisfy both conditions. The `|` operator is
used to select columns that satisfy either condition. The `-` operator is used to select columns
that satisfy the first condition but not the second. The `~` operator is used to select columns
that don't satisfy the condition. As many selector functions can be used as needed and the
operators can be combined to create complex column selection criteria (parentheses can be used
to group conditions and control the order of evaluation).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have a table with columns `name`, `paid_2021`, `paid_2022`, and `person_id` and
we'd like to validate that the values in columns that start with `"paid"` are greater than `10`.
We can use the `starts_with()` column selector function to specify the columns that start with
`"paid"` as the columns to validate.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "paid_2021": [16.32, 16.25, 15.75],
        "paid_2022": [18.62, 16.95, 18.25],
        "person_id": ["A123", "B456", "C789"],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns=pb.starts_with("paid"), value=10)
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `paid_2021` and
one for `paid_2022`. The values in both columns were all greater than `10`.

We can also use the `starts_with()` function in combination with other column selectors (within
[`col()`](`pointblank.col`)) to create more complex column selection criteria (i.e., to select
columns that satisfy multiple conditions). For example, to select columns that start with
`"paid"` and match the text `"2023"` or `"2024"`, we can use the `&` operator to combine column
selectors.

```{python}
tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "hours_2022": [160, 180, 160],
        "hours_2023": [182, 168, 175],
        "hours_2024": [200, 165, 190],
        "paid_2022": [18.62, 16.95, 18.25],
        "paid_2023": [19.29, 17.75, 18.35],
        "paid_2024": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(
        columns=pb.col(pb.starts_with("paid") & pb.matches("23|24")),
        value=10
    )
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `paid_2023` and
one for `paid_2024`.

ends_with(text: 'str', case_sensitive: 'bool' = False) -> 'EndsWith'

Select columns that end with specified text.

Many validation methods have a `columns=` argument that can be used to specify the columns for
validation (e.g., [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`), etc.). The `ends_with()` selector
function can be used to select one or more columns that end with some specified text. So if the
set of table columns consists of

`[first_name, last_name, age, address]`

and you want to validate columns that end with `"name"`, you can use
`columns=ends_with("name")`. This will select the `first_name` and `last_name` columns.

There will be a validation step created for every resolved column. Note that if there aren't any
columns resolved from using `ends_with()` (or any other expression using selector functions),
the validation step will fail to be evaluated during the interrogation process. Such a failure
to evaluate will be reported in the validation results but it won't affect the interrogation
process overall (i.e., the process won't be halted).

Parameters
----------
text
    The text that the column name should end with.
case_sensitive
    Whether column names should be treated as case-sensitive. The default is `False`.

Returns
-------
EndsWith
    An `EndsWith` object, which can be used to select columns that end with the specified text.

Relevant Validation Methods where `ends_with()` can be Used
-----------------------------------------------------------
This selector function can be used in the `columns=` argument of the following validation
methods:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_exists()`](`pointblank.Validate.col_exists`)

The `ends_with()` selector function doesn't need to be used in isolation. Read the next section
for information on how to compose it with other column selectors for more refined ways to select
columns.

Additional Flexibilty through Composition with Other Column Selectors
---------------------------------------------------------------------
The `ends_with()` function can be composed with other column selectors to create fine-grained
column selections. For example, to select columns that end with `"e"` and start with `"a"`, you
can use the `ends_with()` and [`starts_with()`](`pointblank.starts_with`) functions together.
The only condition is that the expressions are wrapped in the [`col()`](`pointblank.col`)
function, like this:

```python
col(ends_with("e") & starts_with("a"))
```

There are four operators that can be used to compose column selectors:

- `&` (*and*)
- `|` (*or*)
- `-` (*difference*)
- `~` (*not*)

The `&` operator is used to select columns that satisfy both conditions. The `|` operator is
used to select columns that satisfy either condition. The `-` operator is used to select columns
that satisfy the first condition but not the second. The `~` operator is used to select columns
that don't satisfy the condition. As many selector functions can be used as needed and the
operators can be combined to create complex column selection criteria (parentheses can be used
to group conditions and control the order of evaluation).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have a table with columns `name`, `2021_pay`, `2022_pay`, and `person_id` and
we'd like to validate that the values in columns that end with `"pay"` are greater than `10`.
We can use the `ends_with()` column selector function to specify the columns that end with
`"pay"` as the columns to validate.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "2021_pay": [16.32, 16.25, 15.75],
        "2022_pay": [18.62, 16.95, 18.25],
        "person_id": ["A123", "B456", "C789"],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns=pb.ends_with("pay"), value=10)
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `2021_pay` and one
for `2022_pay`. The values in both columns were all greater than `10`.

We can also use the `ends_with()` function in combination with other column selectors (within
[`col()`](`pointblank.col`)) to create more complex column selection criteria (i.e., to select
columns that satisfy multiple conditions). For example, to select columns that end with `"pay"`
and match the text `"2023"` or `"2024"`, we can use the `&` operator to combine column
selectors.

```{python}
tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "2022_hours": [160, 180, 160],
        "2023_hours": [182, 168, 175],
        "2024_hours": [200, 165, 190],
        "2022_pay": [18.62, 16.95, 18.25],
        "2023_pay": [19.29, 17.75, 18.35],
        "2024_pay": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(
        columns=pb.col(pb.ends_with("pay") & pb.matches("2023|2024")),
        value=10
    )
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `2023_pay` and one
for `2024_pay`.

contains(text: 'str', case_sensitive: 'bool' = False) -> 'Contains'

Select columns that contain specified text.

Many validation methods have a `columns=` argument that can be used to specify the columns for
validation (e.g., [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`), etc.). The `contains()` selector
function can be used to select one or more columns that contain some specified text. So if the
set of table columns consists of

`[profit, conv_first, conv_last, highest_conv, age]`

and you want to validate columns that have `"conv"` in the name, you can use
`columns=contains("conv")`. This will select the `conv_first`, `conv_last`, and `highest_conv`
columns.

There will be a validation step created for every resolved column. Note that if there aren't any
columns resolved from using `contains()` (or any other expression using selector functions), the
validation step will fail to be evaluated during the interrogation process. Such a failure to
evaluate will be reported in the validation results but it won't affect the interrogation
process overall (i.e., the process won't be halted).

Parameters
----------
text
    The text that the column name should contain.
case_sensitive
    Whether column names should be treated as case-sensitive. The default is `False`.

Returns
-------
Contains
    A `Contains` object, which can be used to select columns that contain the specified text.

Relevant Validation Methods where `contains()` can be Used
----------------------------------------------------------
This selector function can be used in the `columns=` argument of the following validation
methods:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_exists()`](`pointblank.Validate.col_exists`)

The `contains()` selector function doesn't need to be used in isolation. Read the next section
for information on how to compose it with other column selectors for more refined ways to select
columns.

Additional Flexibilty through Composition with Other Column Selectors
---------------------------------------------------------------------
The `contains()` function can be composed with other column selectors to create fine-grained
column selections. For example, to select columns that have the text `"_n"` and start with
`"item"`, you can use the `contains()` and [`starts_with()`](`pointblank.starts_with`) functions
together. The only condition is that the expressions are wrapped in the
[`col()`](`pointblank.col`) function, like this:

```python
col(contains("_n") & starts_with("item"))
```

There are four operators that can be used to compose column selectors:

- `&` (*and*)
- `|` (*or*)
- `-` (*difference*)
- `~` (*not*)

The `&` operator is used to select columns that satisfy both conditions. The `|` operator is
used to select columns that satisfy either condition. The `-` operator is used to select columns
that satisfy the first condition but not the second. The `~` operator is used to select columns
that don't satisfy the condition. As many selector functions can be used as needed and the
operators can be combined to create complex column selection criteria (parentheses can be used
to group conditions and control the order of evaluation).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have a table with columns `name`, `2021_pay_total`, `2022_pay_total`, and `person_id`
and we'd like to validate that the values in columns having `"pay"` in the name are greater than
`10`. We can use the `contains()` column selector function to specify the column names that
contain `"pay"` as the columns to validate.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "2021_pay_total": [16.32, 16.25, 15.75],
        "2022_pay_total": [18.62, 16.95, 18.25],
        "person_id": ["A123", "B456", "C789"],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns=pb.contains("pay"), value=10)
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `2021_pay_total`
and one for `2022_pay_total`. The values in both columns were all greater than `10`.

We can also use the `contains()` function in combination with other column selectors (within
[`col()`](`pointblank.col`)) to create more complex column selection criteria (i.e., to select
columns that satisfy multiple conditions). For example, to select columns that contain `"pay"`
and match the text `"2023"` or `"2024"`, we can use the `&` operator to combine column
selectors.

```{python}
tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "2022_hours": [160, 180, 160],
        "2023_hours": [182, 168, 175],
        "2024_hours": [200, 165, 190],
        "2022_pay_total": [18.62, 16.95, 18.25],
        "2023_pay_total": [19.29, 17.75, 18.35],
        "2024_pay_total": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(
        columns=pb.col(pb.contains("pay") & pb.matches("2023|2024")),
        value=10
    )
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `2023_pay_total`
and one for `2024_pay_total`.

matches(pattern: 'str', case_sensitive: 'bool' = False) -> 'Matches'

Select columns that match a specified regular expression pattern.

Many validation methods have a `columns=` argument that can be used to specify the columns for
validation (e.g., [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`), etc.). The `matches()` selector
function can be used to select one or more columns matching a provided regular expression
pattern. So if the set of table columns consists of

`[rev_01, rev_02, profit_01, profit_02, age]`

and you want to validate columns that have two digits at the end of the name, you can use
`columns=matches(r"[0-9]{2}$")`. This will select the `rev_01`, `rev_02`, `profit_01`, and
`profit_02` columns.

There will be a validation step created for every resolved column. Note that if there aren't any
columns resolved from using `matches()` (or any other expression using selector functions), the
validation step will fail to be evaluated during the interrogation process. Such a failure to
evaluate will be reported in the validation results but it won't affect the interrogation
process overall (i.e., the process won't be halted).

Parameters
----------
pattern
    The regular expression pattern that the column name should match.
case_sensitive
    Whether column names should be treated as case-sensitive. The default is `False`.

Returns
-------
Matches
    A `Matches` object, which can be used to select columns that match the specified pattern.

Relevant Validation Methods where `matches()` can be Used
---------------------------------------------------------
This selector function can be used in the `columns=` argument of the following validation
methods:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_exists()`](`pointblank.Validate.col_exists`)

The `matches()` selector function doesn't need to be used in isolation. Read the next section
for information on how to compose it with other column selectors for more refined ways to select
columns.

Additional Flexibilty through Composition with Other Column Selectors
---------------------------------------------------------------------
The `matches()` function can be composed with other column selectors to create fine-grained
column selections. For example, to select columns that have the text starting with five digits
and end with `"_id"`, you can use the `matches()` and [`ends_with()`](`pointblank.ends_with`)
functions together. The only condition is that the expressions are wrapped in the
[`col()`](`pointblank.col`) function, like this:

```python
col(matches(r"^[0-9]{5}") & ends_with("_id"))
```

There are four operators that can be used to compose column selectors:

- `&` (*and*)
- `|` (*or*)
- `-` (*difference*)
- `~` (*not*)

The `&` operator is used to select columns that satisfy both conditions. The `|` operator is
used to select columns that satisfy either condition. The `-` operator is used to select columns
that satisfy the first condition but not the second. The `~` operator is used to select columns
that don't satisfy the condition. As many selector functions can be used as needed and the
operators can be combined to create complex column selection criteria (parentheses can be used
to group conditions and control the order of evaluation).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have a table with columns `name`, `id_old`, `new_identifier`, and `pay_2021` and we'd
like to validate that text values in columns having `"id"` or `"identifier"` in the name have a
specific syntax. We can use the `matches()` column selector function to specify the columns that
match the pattern.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "id_old": ["ID0021", "ID0032", "ID0043"],
        "new_identifier": ["ID9054", "ID9065", "ID9076"],
        "pay_2021": [16.32, 16.25, 15.75],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_regex(columns=pb.matches("id|identifier"), pattern=r"ID[0-9]{4}")
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `id_old` and one
for `new_identifier`. The values in both columns all match the pattern `"ID[0-9]{4}"`.

We can also use the `matches()` function in combination with other column selectors (within
[`col()`](`pointblank.col`)) to create more complex column selection criteria (i.e., to select
columns that satisfy multiple conditions). For example, to select columns that contain `"pay"`
and match the text `"2023"` or `"2024"`, we can use the `&` operator to combine column
selectors.

```{python}
tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "2022_hours": [160, 180, 160],
        "2023_hours": [182, 168, 175],
        "2024_hours": [200, 165, 190],
        "2022_pay_total": [18.62, 16.95, 18.25],
        "2023_pay_total": [19.29, 17.75, 18.35],
        "2024_pay_total": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(
        columns=pb.col(pb.contains("pay") & pb.matches("2023|2024")),
        value=10
    )
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `2023_pay_total`
and one for `2024_pay_total`.

everything() -> 'Everything'

Select all columns.

Many validation methods have a `columns=` argument that can be used to specify the columns for
validation (e.g., [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`), etc.). The `everything()` selector
function can be used to select every column in the table. If you have a table with six columns
and they're all suitable for a specific type of validation, you can use `columns=everything())`
and all six columns will be selected for validation.

Returns
-------
Everything
    An `Everything` object, which can be used to select all columns.

Relevant Validation Methods where `everything()` can be Used
------------------------------------------------------------
This selector function can be used in the `columns=` argument of the following validation
methods:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_exists()`](`pointblank.Validate.col_exists`)

The `everything()` selector function doesn't need to be used in isolation. Read the next section
for information on how to compose it with other column selectors for more refined ways to select
columns.

Additional Flexibilty through Composition with Other Column Selectors
---------------------------------------------------------------------
The `everything()` function can be composed with other column selectors to create fine-grained
column selections. For example, to select all column names except those having starting with
"id_", you can use the `everything()` and [`starts_with()`](`pointblank.starts_with`)
functions together. The only condition is that the expressions are wrapped in the
[`col()`](`pointblank.col`) function, like this:

```python
col(everything() - starts_with("id_"))
```

There are four operators that can be used to compose column selectors:

- `&` (*and*)
- `|` (*or*)
- `-` (*difference*)
- `~` (*not*)

The `&` operator is used to select columns that satisfy both conditions. The `|` operator is
used to select columns that satisfy either condition. The `-` operator is used to select columns
that satisfy the first condition but not the second. The `~` operator is used to select columns
that don't satisfy the condition. As many selector functions can be used as needed and the
operators can be combined to create complex column selection criteria (parentheses can be used
to group conditions and control the order of evaluation).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have a table with several numeric columns and we'd like to validate that all these
columns have less than `1000`. We can use the `everything()` column selector function to select
all columns for validation.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "2023_hours": [182, 168, 175],
        "2024_hours": [200, 165, 190],
        "2023_pay_total": [19.29, 17.75, 18.35],
        "2024_pay_total": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_lt(columns=pb.everything(), value=1000)
    .interrogate()
)

validation
```

From the results of the validation table we get four validation steps, one each column in the
table. The values in every column were all lower than `1000`.

We can also use the `everything()` function in combination with other column selectors (within
[`col()`](`pointblank.col`)) to create more complex column selection criteria (i.e., to select
columns that satisfy multiple conditions). For example, to select every column except those that
begin with `"2023"` we can use the `-` operator to combine column selectors.

```{python}
tbl = pl.DataFrame(
    {
        "2023_hours": [182, 168, 175],
        "2024_hours": [200, 165, 190],
        "2023_pay_total": [19.29, 17.75, 18.35],
        "2024_pay_total": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_lt(columns=pb.col(pb.everything() - pb.starts_with("2023")), value=1000)
    .interrogate()
)

validation
```

From the results of the validation table we get two validation steps, one for `2024_hours` and
one for `2024_pay_total`.

first_n(n: 'int', offset: 'int' = 0) -> 'FirstN'

Select the first `n` columns in the column list.

Many validation methods have a `columns=` argument that can be used to specify the columns for
validation (e.g., [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`), etc.). The `first_n()` selector
function can be used to select *n* columns positioned at the start of the column list. So if the
set of table columns consists of

`[rev_01, rev_02, profit_01, profit_02, age]`

and you want to validate the first two columns, you can use `columns=first_n(2)`. This will
select the `rev_01` and `rev_02` columns and a validation step will be created for each.

The `offset=` parameter can be used to skip a certain number of columns from the start of the
column list. So if you want to select the third and fourth columns, you can use
`columns=first_n(2, offset=2)`.

Parameters
----------
n
    The number of columns to select from the start of the column list. Should be a positive
    integer value. If `n` is greater than the number of columns in the table, all columns will
    be selected.
offset
    The offset from the start of the column list. The default is `0`. If `offset` is greater
    than the number of columns in the table, no columns will be selected.

Returns
-------
FirstN
    A `FirstN` object, which can be used to select the first `n` columns.

Relevant Validation Methods where `first_n()` can be Used
---------------------------------------------------------
This selector function can be used in the `columns=` argument of the following validation
methods:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_exists()`](`pointblank.Validate.col_exists`)

The `first_n()` selector function doesn't need to be used in isolation. Read the next section
for information on how to compose it with other column selectors for more refined ways to select
columns.

Additional Flexibilty through Composition with Other Column Selectors
---------------------------------------------------------------------
The `first_n()` function can be composed with other column selectors to create fine-grained
column selections. For example, to select all column names starting with "rev" along with the
first two columns, you can use the `first_n()` and [`starts_with()`](`pointblank.starts_with`)
functions together. The only condition is that the expressions are wrapped in the
[`col()`](`pointblank.col`) function, like this:

```python
col(first_n(2) | starts_with("rev"))
```

There are four operators that can be used to compose column selectors:

- `&` (*and*)
- `|` (*or*)
- `-` (*difference*)
- `~` (*not*)

The `&` operator is used to select columns that satisfy both conditions. The `|` operator is
used to select columns that satisfy either condition. The `-` operator is used to select columns
that satisfy the first condition but not the second. The `~` operator is used to select columns
that don't satisfy the condition. As many selector functions can be used as needed and the
operators can be combined to create complex column selection criteria (parentheses can be used
to group conditions and control the order of evaluation).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have a table with columns `paid_2021`, `paid_2022`, `paid_2023`, `paid_2024`, and
`name` and we'd like to validate that the values in the first four columns are greater than
`10`. We can use the `first_n()` column selector function to specify that the first four columns
in the table are the columns to validate.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "paid_2021": [17.94, 16.55, 17.85],
        "paid_2022": [18.62, 16.95, 18.25],
        "paid_2023": [19.29, 17.75, 18.35],
        "paid_2024": [20.73, 18.35, 20.10],
        "name": ["Alice", "Bob", "Charlie"],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns=pb.first_n(4), value=10)
    .interrogate()
)

validation
```

From the results of the validation table we get four validation steps. The values in all those
columns were all greater than `10`.

We can also use the `first_n()` function in combination with other column selectors (within
[`col()`](`pointblank.col`)) to create more complex column selection criteria (i.e., to select
columns that satisfy multiple conditions). For example, to select the first four columns but
also omit those columns that end with `"2023"`, we can use the `-` operator to combine column
selectors.

```{python}
tbl = pl.DataFrame(
    {
        "paid_2021": [17.94, 16.55, 17.85],
        "paid_2022": [18.62, 16.95, 18.25],
        "paid_2023": [19.29, 17.75, 18.35],
        "paid_2024": [20.73, 18.35, 20.10],
        "name": ["Alice", "Bob", "Charlie"],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns=pb.col(pb.first_n(4) - pb.ends_with("2023")), value=10)
    .interrogate()
)

validation
```

From the results of the validation table we get three validation steps, one for `paid_2021`,
`paid_2022`, and `paid_2024`.

last_n(n: 'int', offset: 'int' = 0) -> 'LastN'

Select the last `n` columns in the column list.

Many validation methods have a `columns=` argument that can be used to specify the columns for
validation (e.g., [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`), etc.). The `last_n()` selector
function can be used to select *n* columns positioned at the end of the column list. So if the
set of table columns consists of

`[age, rev_01, rev_02, profit_01, profit_02]`

and you want to validate the last two columns, you can use `columns=last_n(2)`. This will select
the `profit_01` and `profit_02` columns and a validation step will be created for each.

The `offset=` parameter can be used to skip a certain number of columns from the end of the
column list. So if you want to select the third and fourth columns from the end, you can use
`columns=last_n(2, offset=2)`.

Parameters
----------
n
    The number of columns to select from the end of the column list. Should be a positive
    integer value. If `n` is greater than the number of columns in the table, all columns will
    be selected.
offset
    The offset from the end of the column list. The default is `0`. If `offset` is greater than
    the number of columns in the table, no columns will be selected.

Returns
-------
LastN
    A `LastN` object, which can be used to select the last `n` columns.

Relevant Validation Methods where `last_n()` can be Used
--------------------------------------------------------
This selector function can be used in the `columns=` argument of the following validation
methods:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_exists()`](`pointblank.Validate.col_exists`)

The `last_n()` selector function doesn't need to be used in isolation. Read the next section for
information on how to compose it with other column selectors for more refined ways to select
columns.

Additional Flexibilty through Composition with Other Column Selectors
---------------------------------------------------------------------
The `last_n()` function can be composed with other column selectors to create fine-grained
column selections. For example, to select all column names starting with "rev" along with the
last two columns, you can use the `last_n()` and [`starts_with()`](`pointblank.starts_with`)
functions together. The only condition is that the expressions are wrapped in the
[`col()`](`pointblank.col`) function, like this:

```python
col(last_n(2) | starts_with("rev"))
```

There are four operators that can be used to compose column selectors:

- `&` (*and*)
- `|` (*or*)
- `-` (*difference*)
- `~` (*not*)

The `&` operator is used to select columns that satisfy both conditions. The `|` operator is
used to select columns that satisfy either condition. The `-` operator is used to select columns
that satisfy the first condition but not the second. The `~` operator is used to select columns
that don't satisfy the condition. As many selector functions can be used as needed and the
operators can be combined to create complex column selection criteria (parentheses can be used
to group conditions and control the order of evaluation).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have a table with columns `name`, `paid_2021`, `paid_2022`, `paid_2023`, and
`paid_2024` and we'd like to validate that the values in the last four columns are greater than
`10`. We can use the `last_n()` column selector function to specify that the last four columns
in the table are the columns to validate.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "paid_2021": [17.94, 16.55, 17.85],
        "paid_2022": [18.62, 16.95, 18.25],
        "paid_2023": [19.29, 17.75, 18.35],
        "paid_2024": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns=pb.last_n(4), value=10)
    .interrogate()
)

validation
```

From the results of the validation table we get four validation steps. The values in all those
columns were all greater than `10`.

We can also use the `last_n()` function in combination with other column selectors (within
[`col()`](`pointblank.col`)) to create more complex column selection criteria (i.e., to select
columns that satisfy multiple conditions). For example, to select the last four columns but also
omit those columns that end with `"2023"`, we can use the `-` operator to combine column
selectors.

```{python}
tbl = pl.DataFrame(
    {
        "name": ["Alice", "Bob", "Charlie"],
        "paid_2021": [17.94, 16.55, 17.85],
        "paid_2022": [18.62, 16.95, 18.25],
        "paid_2023": [19.29, 17.75, 18.35],
        "paid_2024": [20.73, 18.35, 20.10],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns=pb.col(pb.last_n(4) - pb.ends_with("2023")), value=10)
    .interrogate()
)

validation
```

From the results of the validation table we get three validation steps, one for `paid_2021`,
`paid_2022`, and `paid_2024`.

expr_col(column_name: 'str') -> 'ColumnExpression'

Create a column expression for use in `conjointly()` validation.

This function returns a ColumnExpression object that supports operations like `>`, `<`, `+`,
etc. for use in [`conjointly()`](`pointblank.Validate.conjointly`) validation expressions.

Parameters
----------
column_name
    The name of the column to reference.

Returns
-------
ColumnExpression
    A column expression that can be used in comparisons and operations.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Let's say we have a table with three columns: `a`, `b`, and `c`. We want to validate that:

- The values in column `a` are greater than `2`.
- The values in column `b` are less than `7`.
- The sum of columns `a` and `b` is less than the values in column `c`.

We can use the `expr_col()` function to create a column expression for each of these conditions.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 7, 1, 3, 9, 4],
        "b": [6, 3, 0, 5, 8, 2],
        "c": [10, 4, 8, 9, 10, 5],
    }
)

# Using expr_col() to create backend-agnostic validation expressions
validation = (
    pb.Validate(data=tbl)
    .conjointly(
        lambda df: pb.expr_col("a") > 2,
        lambda df: pb.expr_col("b") < 7,
        lambda df: pb.expr_col("a") + pb.expr_col("b") < pb.expr_col("c")
    )
    .interrogate()
)

validation
```

The above code creates a validation object that checks the specified conditions using the
`expr_col()` function. The resulting validation table will show whether each condition was
satisfied for each row in the table.

See Also
--------
Validate.conjointly : The validation step where these expressions are used.


## Segment Groups

Combine multiple values into a single segment using `seg_*()` helper functions.



seg_group(values: 'list[Any]') -> 'Segment'

Group together values for segmentation.

Many validation methods have a `segments=` argument that can be used to specify one or more
columns, or certain values within a column, to create segments for validation (e.g.,
[`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`col_vals_regex()`](`pointblank.Validate.col_vals_regex`), etc.). When passing in a column, or
a tuple with a column and certain values, a segment will be created for each individual value
within the column or given values. The `seg_group()` selector enables values to be grouped
together into a segment. For example, if you were to create a segment for a column "region",
investigating just "North" and "South" regions, a typical segment would look like:

`segments=("region", ["North", "South"])`

This would create two validation steps, one for each of the regions. If you wanted to group
these two regions into a single segment, you could use the `seg_group()` function like this:

`segments=("region", pb.seg_group(["North", "South"]))`

You could create a second segment for "East" and "West" regions like this:

`segments=("region", pb.seg_group([["North", "South"], ["East", "West"]]))`

There will be a validation step created for every segment. Note that if there aren't any
segments created using `seg_group()` (or any other segment expression), the validation step will
fail to be evaluated during the interrogation process. Such a failure to evaluate will be
reported in the validation results but it won't affect the interrogation process overall
(i.e., the process won't be halted).

Parameters
----------
values
    A list of values to be grouped into a segment. This can be a single list or a list of lists.

Returns
-------
Segment
    A `Segment` object, which can be used to combine values into a segment.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Let's say we're analyzing sales from our local bookstore, and want to check the number of books
sold for the month exceeds a certain threshold. We could pass in the argument
`segments="genre"`, which would return a segment for each unique genre in the datasets. We could
also pass in `segments=("genre", ["Fantasy", "Science Fiction"])`, to only create segments for
those two genres. However, if we wanted to group these two genres into a single segment, we
could use the `seg_group()` function.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "title": [
            "The Hobbit",
            "Harry Potter and the Sorcerer's Stone",
            "The Lord of the Rings",
            "A Game of Thrones",
            "The Name of the Wind",
            "The Girl with the Dragon Tattoo",
            "The Da Vinci Code",
            "The Hitchhiker's Guide to the Galaxy",
            "The Martian",
            "Brave New World"
        ],
        "genre": [
            "Fantasy",
            "Fantasy",
            "Fantasy",
            "Fantasy",
            "Fantasy",
            "Mystery",
            "Mystery",
            "Science Fiction",
            "Science Fiction",
            "Science Fiction",
        ],
        "units_sold": [875, 932, 756, 623, 445, 389, 678, 534, 712, 598],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(
        columns="units_sold",
        value=500,
        segments=("genre", pb.seg_group(["Fantasy", "Science Fiction"]))
    )
    .interrogate()
)

validation
```

The Fantasy and Science Fiction books form a single segment, so the report has one step with
eight test units. Only *The Name of the Wind* (445 units sold) falls short of the threshold,
and the Mystery books aren't checked at all.

What's more, we can create multiple segments, combining the genres in different ways.

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(
        columns="units_sold",
        value=500,
        segments=("genre", pb.seg_group([
            ["Fantasy", "Science Fiction"],
            ["Fantasy", "Mystery"],
            ["Mystery", "Science Fiction"]
        ]))
    )
    .interrogate()
)

validation
```

Each group becomes its own step in the report. The Fantasy and Science Fiction segment has one
failing test unit (445 units sold), the Fantasy and Mystery segment has two (445 and 389), and
the Mystery and Science Fiction segment has one (389). A genre can appear in more than one
group, so the same row may be checked in several steps.


## Interrogation and Reporting

The validation plan is executed when `interrogate()` is called. After interrogation, view validation reports, extract metrics, or split data based on results.



interrogate(self, collect_extracts: 'bool' = True, collect_tbl_checked: 'bool' = True, get_first_n: 'int | None' = None, sample_n: 'int | None' = None, sample_frac: 'int | float | None' = None, extract_limit: 'int' = 500) -> 'Validate'

Execute each validation step against the table and store the results.

When a validation plan has been set with a series of validation steps, the interrogation
process through `interrogate()` should then be invoked. Interrogation will evaluate each
validation step against the table and store the results.

The interrogation process will collect extracts of failing rows if the `collect_extracts=`
option is set to `True` (the default). We can control the number of rows collected using the
`get_first_n=`, `sample_n=`, and `sample_frac=` options. The `extract_limit=` option will
enforce a hard limit on the number of rows collected when `collect_extracts=True`.

After interrogation is complete, the `Validate` object will have gathered information, and
we can use methods like [`n_passed()`](`pointblank.Validate.n_passed`),
[`f_failed()`](`pointblank.Validate.f_failed`), etc., to understand how the table performed
against the validation plan. A visual representation of the validation results can be viewed
by printing the `Validate` object; this will display the validation table in an HTML viewing
environment.

Calling `interrogate()` again (e.g., after the table was fixed or replaced, or with
different extract options) replaces the results of the earlier interrogation. Each
interrogation starts from the validation steps as they were defined, so callable `active=`
values are evaluated again and steps that use column selectors or `segments=` are split by
column and segment for the table as it is now.

Parameters
----------
collect_extracts
    An option to collect rows of the input table that didn't pass a particular validation
    step. The default is `True` and further options (i.e., `get_first_n=`, `sample_*=`)
    allow for fine control of how these rows are collected.
collect_tbl_checked
    The processed data frames produced by executing the validation steps is collected and
    stored in the `Validate` object if `collect_tbl_checked=True`. This information is
    necessary for some methods (e.g.,
    [`get_sundered_data()`](`pointblank.Validate.get_sundered_data`)), but it can
    potentially make the object grow to a large size. To opt out of attaching this data, set
    this to `False`.
get_first_n
    If the option to collect rows where test units is chosen, there is the option here to
    collect the first `n` rows. Supply an integer number of rows to extract from the top of
    subset table containing non-passing rows (the ordering of data from the original table
    is retained).
sample_n
    If the option to collect non-passing rows is chosen, this option allows for the
    sampling of `n` rows. Supply an integer number of rows to sample from the subset table.
    If `n` happens to be greater than the number of non-passing rows, then all such rows
    will be returned.
sample_frac
    If the option to collect non-passing rows is chosen, this option allows for the sampling
    of a fraction of those rows. Provide a number in the range of `0` and `1`. The number of
    rows to return could be very large, however, the `extract_limit=` option will apply a
    hard limit to the returned rows.
extract_limit
    A value that limits the possible number of rows returned when extracting non-passing
    rows. The default is `500` rows. This limit is applied after any sampling or limiting
    options are applied. If the number of rows to be returned is greater than this limit,
    then the number of rows returned will be limited to this value. This is useful for
    preventing the collection of too many rows when the number of non-passing rows is very
    large.

Returns
-------
Validate
    The `Validate` object with the results of the interrogation.

Examples
--------
Let's use a built-in dataset (`"game_revenue"`) to demonstrate some of the options of the
interrogation process. A series of validation steps will populate our validation plan. After
setting up the plan, the next step is to interrogate the table and see how well it aligns
with our expectations. We'll use the `get_first_n=` option so that any extracts of failing
rows are limited to the first `n` rows.

```{python}
import pointblank as pb
import polars as pl

validation = (
    pb.Validate(data=pb.load_dataset(dataset="game_revenue"))
    .col_vals_lt(columns="item_revenue", value=200)
    .col_vals_gt(columns="item_revenue", value=0)
    .col_vals_gt(columns="session_duration", value=5)
    .col_vals_in_set(columns="item_type", set=["iap", "ad"])
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}[0-9]{3}")
)

validation.interrogate(get_first_n=10)
```

The validation table shows that step 3 (checking for `session_duration` greater than `5`)
has 18 failing test units. This means that 18 rows in the table are problematic. We'd like
to see the rows that failed this validation step and we can do that with the
[`get_data_extracts()`](`pointblank.Validate.get_data_extracts`) method.

```{python}
pb.preview(validation.get_data_extracts(i=3, frame=True))
```

The [`get_data_extracts()`](`pointblank.Validate.get_data_extracts`) method will return a
Polars DataFrame here with the first 10 rows that failed the validation step (we passed that
into the [`preview()`](`pointblank.preview`) function for a better display). There are
actually 18 rows that failed but we limited the collection of extracts with
`get_first_n=10`.

set_tbl(self, tbl: 'Any', tbl_name: 'str | None' = None, label: 'str | None' = None) -> 'Validate'

Set or replace the table associated with the Validate object.

This method allows you to replace the table associated with a Validate object with a
different (but presumably similar) table. This is useful when you want to apply the same
validation plan to multiple tables or when you have a validation workflow defined but want
to swap in a different data source.

Parameters
----------
tbl
    The table to replace the existing table with. This can be any supported table type
    including DataFrame objects, Ibis table objects, CSV file paths, Parquet file paths,
    GitHub URLs, or database connection strings. The same table type constraints apply as in
    the `Validate` constructor.
tbl_name
    An optional name to assign to the new input table object. If no value is provided, the
    existing table name will be retained.
label
    An optional label for the validation plan. If no value is provided, the existing label
    will be retained.

Returns
-------
Validate
    A new `Validate` object with the replacement table.

When to Use
-----------
The `set_tbl()` method is particularly useful in scenarios where you have:

- multiple similar tables that need the same validation checks
- a template validation workflow that should be applied to different data sources
- YAML-defined validations where you want to override the table specified in the YAML

The `set_tbl()` method creates a copy of the validation object with the new table, so the
original validation object remains unchanged. This allows you to reuse validation plans
across multiple tables without interference. The validation object can already have been
interrogated: interrogating the copy starts over from the steps as they were defined, so
the new table is checked from scratch (e.g., a step using a column selector such as
`starts_with()` gets one step for each matching column of the new table).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
We will first create two similar tables for our future validation plans.

```{python}
import pointblank as pb
import polars as pl

# Create two similar tables
table_1 = pl.DataFrame({
    "x": [1, 2, 3, 4, 5],
    "y": [5, 4, 3, 2, 1],
    "z": ["a", "b", "c", "d", "e"]
})

table_2 = pl.DataFrame({
    "x": [2, 4, 6, 8, 10],
    "y": [10, 8, 6, 4, 2],
    "z": ["f", "g", "h", "i", "j"]
})
```

Create a validation plan with the first table.

```{python}
validation_table_1 = (
    pb.Validate(
        data=table_1,
        tbl_name="Table 1",
        label="Validation applied to the first table"
    )
    .col_vals_gt(columns="x", value=0)
    .col_vals_lt(columns="y", value=10)
)
```

Now apply the same validation plan to the second table.

```{python}
validation_table_2 = (
    validation_table_1
    .set_tbl(
        tbl=table_2,
        tbl_name="Table 2",
        label="Validation applied to the second table"
    )
)
```

Here is the interrogation of the first table:

```{python}
validation_table_1.interrogate()
```

All test units pass for the first table. And here is the second table:

```{python}
validation_table_2.interrogate()
```

Every value of `y` in the first table is below `10`, but the second table has a `10`, so
the second step has one failing test unit there. The plan was written once and applied to
both tables, each with its own name and label. This is handy for tables that share a
structure, such as monthly extracts or one table per region.

get_tabular_report(self, title: 'str | None' = ':default:', incl_header: 'bool | None' = None, incl_footer: 'bool | None' = None, incl_footer_timings: 'bool | None' = None, incl_footer_notes: 'bool | None' = None, incl_dimensions: 'bool | None' = None) -> 'GT'

Validation report as a GT table.

The `get_tabular_report()` method returns a GT table object that represents the validation
report. This validation table provides a summary of the validation results, including the
validation steps, the number of test units, the number of failing test units, and the
fraction of failing test units. The table also includes status indicators for the 'warning',
'error', and 'critical' levels.

You could simply display the validation table without the use of the `get_tabular_report()`
method. However, the method provides a way to customize the title of the report and to
control which header and footer elements are shown.

Because the returned object is a GT table, it can be further customized with Great Tables
methods such as `cols_hide()`, `cols_label()`, `cols_width()`, and `tab_style()`. These
methods need the table's internal column names rather than the labels shown in the report:
`"status_color"` (status indicator), `"i"` (step number), `"type_upd"` (STEP),
`"columns_upd"` (COLUMNS), `"values_upd"` (VALUES), `"tbl"` (TBL), `"eval"` (EVAL),
`"test_units"` (UNITS), `"pass"` (PASS), `"fail"` (FAIL), `"w_upd"` (W), `"e_upd"` (E),
`"c_upd"` (C), and `"extract_upd"` (EXT). For programmatic access to the validation results
themselves, use [`get_dataframe_report()`](`pointblank.Validate.get_dataframe_report`) or
[`get_json_report()`](`pointblank.Validate.get_json_report`) instead.

Parameters
----------
title
    Options for customizing the title of the report. The default is the `":default:"` value
    which produces a generic title. Another option is `":tbl_name:"`, and that presents the
    name of the table as the title for the report. If no title is wanted, then `":none:"`
    can be used. Aside from keyword options, text can be provided for the title. This will
    be interpreted as Markdown text and transformed internally to HTML.
incl_header
    Controls whether the header section should be displayed. If `None`, uses the global
    configuration setting. The header contains the table name, label, and threshold
    information.
incl_footer
    Controls whether the footer section should be displayed. If `None`, uses the global
    configuration setting. The footer can contain validation timing information and notes.
incl_footer_timings
    Controls whether validation timing information (start time, duration, end time) should
    be displayed in the footer. If `None`, uses the global configuration setting. Only
    applies when `incl_footer=True`.
incl_footer_notes
    Controls whether notes from validation steps should be displayed in the footer. If
    `None`, uses the global configuration setting. Only applies when `incl_footer=True`.
incl_dimensions
    Controls whether the data quality dimension display is shown: a color-coded dimension
    badge on each step number and a health-score summary block in the footer. If `None`,
    uses the global configuration setting (which defaults to `False`, i.e., opt-in). Set to
    `True` to include it for this report.

Returns
-------
GT
    A GT table object that represents the validation report.

Examples
--------
Let's create a `Validate` object with a few validation steps and then interrogate the data
table to see how it performs against the validation plan. We can then generate a tabular
report to get a summary of the results.

```{python}
import pointblank as pb
import polars as pl

# Create a Polars DataFrame
tbl_pl = pl.DataFrame({"x": [1, 2, 3, 4], "y": [4, 5, 6, 7]})

# Validate data using Polars DataFrame
validation = (
    pb.Validate(data=tbl_pl, tbl_name="tbl_xy", thresholds=(2, 3, 4))
    .col_vals_gt(columns="x", value=1)
    .col_vals_lt(columns="x", value=3)
    .col_vals_le(columns="y", value=7)
    .interrogate()
)

# Look at the validation table
validation
```

The validation table is displayed with a default title ('Validation Report'). We can use the
`get_tabular_report()` method to customize the title of the report. For example, we can set
the title to the name of the table by using the `title=":tbl_name:"` option. This will use
the string provided in the `tbl_name=` argument of the `Validate` object.

```{python}
validation.get_tabular_report(title=":tbl_name:")
```

The title of the report is now set to the name of the table, which is 'tbl_xy'. This can be
useful if you have multiple tables and want to keep track of which table the validation
report is for.

Alternatively, you can provide your own title for the report.

```{python}
validation.get_tabular_report(title="Report for Table XY")
```

The title of the report is now set to 'Report for Table XY'. This can be useful if you want
to provide a more descriptive title for the report.

get_step_report(self, i: 'int', columns_subset: 'str | list[str] | Column | None' = None, header: 'str' = ':default:', limit: 'int | None' = 10) -> 'GT'

Get a detailed report for a single validation step.

The `get_step_report()` method returns a report of what went well---or what failed
spectacularly---for a given validation step. The report includes a summary of the validation
step and a detailed breakdown of the interrogation results. The report is presented as a GT
table object, which can be displayed in a notebook or exported to an HTML file.

:::{.callout-warning}
The `get_step_report()` method is still experimental. Please report any issues you encounter
in the [Pointblank issue tracker](https://github.com/posit-dev/pointblank/issues).
:::

Parameters
----------
i
    The step number for which to get the report.
columns_subset
    The columns to display in a step report that shows errors in the input table. By default
    all columns are shown (`None`). If a subset of columns is desired, we can provide a list
    of column names, a string with a single column name, a `Column` object, or a
    `ColumnSelector` object. The last two options allow for more flexible column selection
    using column selector functions. Errors are raised if the column names provided don't
    match any columns in the table (when provided as a string or list of strings) or if
    column selector expressions don't resolve to any columns.
header
    Options for customizing the header of the step report. The default is the `":default:"`
    value which produces a header with a standard title and set of details underneath. Aside
    from this default, free text can be provided for the header. This will be interpreted as
    Markdown text and transformed internally to HTML. You can provide one of two templating
    elements: `{title}` and `{details}`. The default header has the template
    `"{title}{details}"` so you can easily start from that and modify as you see fit. If you
    don't want a header at all, you can set `header=None` to remove it entirely.
limit
    The number of rows to display for those validation steps that check values in rows (the
    `col_vals_*()` validation steps). The default is `10` rows and the limit can be removed
    entirely by setting `limit=None`.

Returns
-------
GT
    A GT table object that represents the detailed report for the validation step.

Types of Step Reports
---------------------
The `get_step_report()` method produces a report based on the *type* of validation step.
The following column-value or row-based validation step validation methods will produce a
report that shows the rows of the data that failed:

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_vals_str_len()`](`pointblank.Validate.col_vals_str_len`)
- [`col_vals_in_table()`](`pointblank.Validate.col_vals_in_table`)
- [`col_vals_expr()`](`pointblank.Validate.col_vals_expr`)
- [`col_missing_coded()`](`pointblank.Validate.col_missing_coded`)
- [`col_missing_only_coded()`](`pointblank.Validate.col_missing_only_coded`)
- [`col_missing_consistent()`](`pointblank.Validate.col_missing_consistent`)
- [`conjointly()`](`pointblank.Validate.conjointly`)
- [`prompt()`](`pointblank.Validate.prompt`)
- [`rows_complete()`](`pointblank.Validate.rows_complete`)

The [`rows_distinct()`](`pointblank.Validate.rows_distinct`) validation step will produce a
report that shows duplicate rows (or duplicate values in one or a set of columns as defined
in that method's `columns_subset=` parameter.

The [`col_schema_match()`](`pointblank.Validate.col_schema_match`) validation step will
produce a report that shows the schema of the data table and the schema of the validation
step. The report will indicate whether the schemas match or not.

The [`tbl_match()`](`pointblank.Validate.tbl_match`) and
[`tbl_reconcile()`](`pointblank.Validate.tbl_reconcile`) validation steps will produce a
report comparing the target table to the comparison table, with samples of the rows (or key
groups) that differ.

The remaining validation steps check the table, or a column, as a whole. Each produces a
summary report with the measured value(s) beside the expected value(s):

- the aggregate methods (e.g., [`col_sum_gt()`](`pointblank.Validate.col_sum_gt`),
  [`col_avg_eq()`](`pointblank.Validate.col_avg_eq`),
  [`col_sd_le()`](`pointblank.Validate.col_sd_le`)): the computed aggregate value
- [`col_exists()`](`pointblank.Validate.col_exists`): the column's position in the table
  (and, if it's missing, the closest-matching column name)
- [`row_count_match()`](`pointblank.Validate.row_count_match`) and
  [`col_count_match()`](`pointblank.Validate.col_count_match`): the row or column count
- [`col_pct_null()`](`pointblank.Validate.col_pct_null`) and
  [`col_pct_missing()`](`pointblank.Validate.col_pct_missing`): the number and percentage
  of null or missing values
- [`data_freshness()`](`pointblank.Validate.data_freshness`): the most recent datetime
  value and its age
- [`specially()`](`pointblank.Validate.specially`): the number of passing and failing test
  units

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Let's create a validation plan with a few validation steps and interrogate the data. With
that, we'll have a look at the validation reporting table for the entire collection of
steps and what went well or what failed.

```{python}
import pointblank as pb

validation = (
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="pandas"),
        tbl_name="small_table",
        label="Example for the get_step_report() method",
        thresholds=(1, 0.20, 0.40)
    )
    .col_vals_lt(columns="d", value=3500)
    .col_vals_between(columns="c", left=1, right=8)
    .col_vals_gt(columns="a", value=3)
    .col_vals_regex(columns="b", pattern=r"[0-9]-[a-z]{3}-[0-9]{3}")
    .interrogate()
)

validation
```

There were four validation steps performed, where the first three steps had failing test
units and the last step had no failures. Let's get a detailed report for the first step by
using the `get_step_report()` method.

```{python}
validation.get_step_report(i=1)
```

The report for the first step is displayed. The report includes a summary of the validation
step and a detailed breakdown of the interrogation results. The report provides details on
what the validation step was checking, the extent to which the test units failed, and a
table that shows the failing rows of the data with the column of interest highlighted.

The second and third steps also had failing test units. Reports for those steps can be
viewed by using `get_step_report(i=2)` and `get_step_report(i=3)` respectively.

The final step did not have any failing test units. A report for the final step can still be
viewed by using `get_step_report(i=4)`. The report will indicate that every test unit passed
and a prview of the target table will be provided.

```{python}
validation.get_step_report(i=4)
```

If you'd like to trim down the number of columns shown in the report, you can provide a
subset of columns to display. For example, if you only want to see the columns `a`, `b`, and
`c`, you can provide those column names as a list.

```{python}
validation.get_step_report(i=1, columns_subset=["a", "b", "c"])
```

If you'd like to increase or reduce the maximum number of rows shown in the report, you can
provide a different value for the `limit` parameter. For example, if you'd like to see only
up to 5 rows, you can set `limit=5`.

```{python}
validation.get_step_report(i=3, limit=5)
```

Step 3 actually had 7 failing test units, but only the first 5 rows are shown in the step
report because of the `limit=5` parameter.

get_json_report(self, use_fields: 'list[str] | None' = None, exclude_fields: 'list[str] | None' = None) -> 'str'

Get a report of the validation results as a JSON-formatted string.

The `get_json_report()` method provides a machine-readable report of validation results in
JSON format. This is particularly useful for programmatic processing, storing validation
results, or integrating with other systems. The report includes detailed information about
each validation step, such as assertion type, columns validated, threshold values, test
results, and more.

By default, all available validation information fields are included in the report. However,
you can customize the fields to include or exclude using the `use_fields=` and
`exclude_fields=` parameters.

Parameters
----------
use_fields
    An optional list of specific fields to include in the report. If provided, only these
    fields will be included in the JSON output. If `None` (the default), all standard
    validation report fields are included. Have a look at the *Available Report Fields*
    section below for a list of fields that can be included in the report.
exclude_fields
    An optional list of fields to exclude from the report. If provided, these fields will
    be omitted from the JSON output. If `None` (the default), no fields are excluded.
    This parameter cannot be used together with `use_fields=`. The *Available Report Fields*
    provides a listing of fields that can be excluded from the report.

Returns
-------
str
    A JSON-formatted string representing the validation report, with each validation step
    as an object in the report array.

Available Report Fields
-----------------------
The JSON report can include any of the standard validation report fields, including:

- `i`: the step number (1-indexed)
- `i_o`: the original step index from the validation plan (pre-expansion)
- `assertion_type`: the type of validation assertion (e.g., `"col_vals_gt"`, etc.)
- `column`: the column being validated (or columns used in certain validations)
- `values`: the comparison values or parameters used in the validation
- `inclusive`: whether the comparison is inclusive (for range-based validations)
- `na_pass`: whether `NA`/`Null` values are considered passing (for certain validations)
- `pre`: preprocessing function applied before validation
- `segments`: data segments to which the validation was applied
- `thresholds`: threshold level statement that was used for the validation step
- `label`: custom label for the validation step
- `brief`: a brief description of the validation step
- `active`: whether the validation step is active
- `all_passed`: whether all test units passed in the step
- `n`: total number of test units
- `n_passed`, `n_failed`: number of test units that passed and failed
- `f_passed`, `f_failed`: Fraction of test units that passed and failed
- `warning`, `error`, `critical`: whether the namesake threshold level was exceeded (is
`null` if threshold not set)
- `time_processed`: when the validation step was processed (ISO 8601 format)
- `proc_duration_s`: the processing duration in seconds

Examples
--------
Let's create a validation plan with a few validation steps and generate a JSON report of the
results:

```{python}
import pointblank as pb
import polars as pl

# Create a sample DataFrame
tbl = pl.DataFrame({
    "a": [5, 7, 8, 9],
    "b": [3, 4, 2, 1]
})

# Create and execute a validation plan
validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=6)
    .col_vals_lt(columns="b", value=4)
    .interrogate()
)

# Get the full JSON report
json_report = validation.get_json_report()

print(json_report)
```

The report is a JSON array with one object per validation step. Each object holds the
step's settings (such as the column, the comparison value and any thresholds) along with
its results (the test unit counts and fractions, and whether each threshold was reached).

You can also customize which fields to include:

```{python}
json_report = validation.get_json_report(
    use_fields=["i", "assertion_type", "column", "n_passed", "n_failed"]
)

print(json_report)
```

Now each step object has just the five requested fields. Or we can choose which fields to
exclude:

```{python}
json_report = validation.get_json_report(
    exclude_fields=[
        "i_o", "thresholds", "pre", "segments", "values",
        "na_pass", "inclusive", "label", "brief", "active",
        "time_processed", "proc_duration_s"
    ]
)

print(json_report)
```

The JSON output can be further processed or analyzed programmatically:

```{python}
import json

# Parse the JSON report
report_data = json.loads(validation.get_json_report())

# Extract and analyze validation results
failing_steps = [step for step in report_data if step["n_failed"] > 0]
print(f"Number of failing validation steps: {len(failing_steps)}")
```

Both steps have one failing test unit: the `5` in column `a` isn't greater than `6`, and
the `4` in column `b` isn't less than `4`. Because the report is plain JSON, it can be
sent to a monitoring service or stored with each run, and read back without Pointblank.

See Also
--------
Validate.get_tabular_report : Get a formatted HTML report as a GT table.
Validate.get_data_extracts : Get the rows that failed validation.

get_dataframe_report(self, tbl_type: "Literal['polars', 'pandas', 'duckdb']" = 'polars') -> 'Any'

Get a report of the validation results as a DataFrame.

The `get_dataframe_report()` method returns a compact, row-wise summary of validation step
results as a DataFrame. Each row corresponds to a single validation step. This is useful for
logging, exporting (e.g., writing to CSV or Parquet), and programmatic analysis of
validation outcomes.

The output format depends on `tbl_type=`: Polars DataFrame (default), Pandas DataFrame, or
DuckDB via an Ibis memtable.

Parameters
----------
tbl_type
    The output backend. One of `"polars"` (default), `"pandas"`, or `"duckdb"`.

Returns
-------
polars.DataFrame | pandas.DataFrame | ibis.expr.types.relations.Table
    A tabular summary of validation results. When `tbl_type="duckdb"`, the return value is
    an Ibis memtable (a `Table` expression).

Raises
------
ValueError
    If `tbl_type=` is not one of `"polars"`, `"pandas"`, or `"duckdb"`.
ImportError
    If the required library for the chosen `tbl_type=` is not installed.

Output Columns
--------------
The returned DataFrame contains the following columns:

- `active`: Whether the validation step was active (`True`/`False`).
- `step_number`: The 1-indexed step number.
- `step_description`: The assertion type (e.g., `"col_vals_gt"`).
- `columns`: The column name validated. For a step on several columns (e.g., a composite
  key), the names are joined with `", "`.
- `values`: The comparison value used in the validation. For regex validations, just the
  pattern string is included. With `tbl_type="duckdb"`, each value is stored as JSON text
  in a string column (e.g., `"[a-z]"` for a pattern, `[1, 2]` for a set), which can be
  decoded with `json.loads()` or cast with `.cast("json")`.
- `step_evaluated`: Whether the step was evaluated without error (`False` if it couldn't
  be evaluated, e.g., because its column doesn't exist).
- `units`: Total number of test units.
- `all_units_passed`: Whether every test unit passed.
- `pass_n`: Number of passing test units.
- `pass_pct`: Fraction of test units that passed (`0.0`-`1.0`).
- `failed_n`: Number of failing test units.
- `failed_pct`: Fraction of test units that failed (`0.0`-`1.0`).
- `warning`, `error`, `critical`: Whether the respective threshold was exceeded.
- `brief`: A coalesced description of the step (from manual brief or auto-generated brief).
- `preprocessed`: Whether a preprocessing function was applied.
- `segmented`: Whether the step used segmented validation.

For inactive steps (`active=False`), the result columns (`step_evaluated` through
`critical`) are set to `None`/`null`; for steps that couldn't be evaluated,
`step_evaluated` is `False` and the other result columns are `None`/`null`.

Examples
--------
Create a validation, interrogate, and get the results as a Polars DataFrame:

```{python}
import pointblank as pb

validation = (
    pb.Validate(
        data=pb.load_dataset("small_table", tbl_type="polars"),
        label="My validation",
    )
    .col_vals_gt(columns="d", value=100)
    .col_vals_regex(columns="b", pattern=r"[0-9]-[a-z]{3}-[0-9]{3}")
    .interrogate()
)

validation.get_dataframe_report()
```

Get the results as a Pandas DataFrame instead:

```{python}
validation.get_dataframe_report(tbl_type="pandas")
```

Both have one row per validation step and the same columns, whatever the type of the
validated table. As a DataFrame, the results can be filtered (to find the steps that
failed), appended to a table of earlier runs to track quality over time, or written to a
database alongside the data.

to_code(self) -> 'str'

Render this validation plan as canonical Pointblank Python code.

The `to_code()` method walks the validation plan and reconstructs the equivalent
`pb.Validate(...).col_vals_*()...` method chain as a string. This is the inverse of
writing the plan by hand: it takes an in-memory `Validate` object and produces source
code that, when executed, recreates the same plan.

This is useful for sharing a plan, reviewing it in a diff, persisting it alongside
results, and it is the foundation for the AI edit/iterate flow (see
[`EditValidation`](`pointblank.EditValidation`)), which sends the current plan to a model
as editable code.

Returns
-------
str
    The validation plan as a block of Python code. The data source is rendered as the
    placeholder `your_data`, since the original data variable name is not known to the
    plan. Reference data given as a file path or connection string is written out as-is;
    an in-memory reference table becomes the placeholder `your_reference_data` (with a
    warning). Values such as `Decimal("2.2")` or `datetime.time(9, 30)` are written as
    Python expressions, and the code starts with the imports they need.

Notes on Fidelity
-----------------
Most validation steps round-trip exactly. Steps that carry non-serializable Python
objects, such as `pre=` preprocessing callables, `actions=`, callable `active=`
conditions, or the expressions used by
[`col_vals_expr()`](`pointblank.Validate.col_vals_expr`),
[`conjointly()`](`pointblank.Validate.conjointly`), and
[`specially()`](`pointblank.Validate.specially`), cannot be reproduced from memory. For
those, a syntactically valid placeholder is emitted and a warning is raised so the code
still parses and the loss is visible.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's create a validation plan for the `small_table` dataset, with thresholds and four
steps, and render it as code.

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

validation = (
    pb.Validate(data=small_table, tbl_name="small_table", thresholds=(0.1, 0.2))
    .col_vals_gt(columns="d", value=100)
    .col_vals_not_null(columns="c")
    .col_vals_in_set(columns="f", set=["low", "mid", "high"])
    .rows_distinct()
)

print(validation.to_code())
```

The code has every step with its arguments, and the thresholds written out as a
[`Thresholds`](`pointblank.Thresholds`) object. The table is the placeholder `your_data`,
and the code doesn't call [`interrogate()`](`pointblank.Validate.interrogate`), since it
describes the plan rather than a run of it. Running the code with `your_data` defined
recreates the plan, which can then be interrogated:

```{python}
namespace = {"your_data": small_table}

exec(validation.to_code(), namespace)

namespace["validation"].interrogate()
```

The recreated plan finds the same failures as the original would (the two missing values
in `c` and the duplicated rows). When a step has something that can't be written as code,
such as a `pre=` function, the code still parses but a warning says what was lost:

```{python}
validation_pre = pb.Validate(data=small_table).col_vals_gt(
    columns="d", value=100, pre=lambda df: df.head(5)
)

print(validation_pre.to_code())
```

The `pre=` function is written as `pre=None`, so the generated step would check every
row rather than the first five. Fix up placeholders like this one before using the code.

to_yaml(self, path: 'str | Path | None' = None) -> 'str'

Serialize this validation plan to a `yaml_interrogate()`-compatible YAML config.

The `to_yaml()` method renders the validation plan as a YAML document using the same
schema consumed by [`yaml_interrogate()`](`pointblank.yaml_interrogate`). This enables
storing plans as configuration, sharing them across projects, and round-tripping a plan
through YAML.

Parameters
----------
path
    An optional file path. If provided, the YAML is written to this file (parent
    directories are created as needed) in addition to being returned.

Returns
-------
str
    The validation plan as a YAML string. The `tbl` field is set from `tbl_name` when
    available, otherwise to the placeholder `your_data`; set it to a loadable data
    source before passing the YAML to `yaml_interrogate()`. The `reference` field is
    written the same way as in [`to_code()`](`pointblank.Validate.to_code`): a file path
    or connection string as-is, and the placeholder `your_reference_data` (with a
    warning) for an in-memory table.

Notes on Fidelity
-----------------
As with [`to_code()`](`pointblank.Validate.to_code`), steps carrying non-serializable
Python objects (`pre=` callables, `actions=`, callable `active=`, and the expressions of
`col_vals_expr()`/`conjointly()`/`specially()`) cannot be represented in YAML; a
placeholder is emitted and a warning is raised.

Values that YAML has no literal for, such as `Decimal("2.2")`, `datetime.time(9, 30)`, or
a datetime in a named time zone, are written as `python:` blocks. These are among the few
calls that [`yaml_interrogate()`](`pointblank.yaml_interrogate`) runs without
`allow_python=True`, so the YAML still loads as is.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's create a validation plan for the `small_table` dataset and write it as YAML.

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

validation = (
    pb.Validate(data=small_table, tbl_name="small_table", thresholds=(0.1, 0.2))
    .col_vals_gt(columns="d", value=100)
    .col_vals_not_null(columns="c")
    .col_vals_in_set(columns="f", set=["low", "mid", "high"])
)

yaml_config = validation.to_yaml()

print(yaml_config)
```

Each step becomes an entry under `steps`, named after its method and with its arguments
beneath. The `tbl` field was set from `tbl_name=`. Here, that name happens to be a dataset
that [`yaml_interrogate()`](`pointblank.yaml_interrogate`) can load, so the YAML runs
as is:

```{python}
pb.yaml_interrogate(yaml_config)
```

For a table of your own, set `tbl` to a file path or connection string before running the
YAML. To keep the configuration (for instance, in version control next to the pipeline it
checks), give `to_yaml()` a `path=`, which writes the file and still returns the string.

to_json_schema(self, path: 'str | Path | None' = None) -> 'dict[str, Any]'

Export this validation plan as a JSON Schema document.

The `to_json_schema()` method walks the validation steps and maps each one to its closest
JSON Schema equivalent. This lets you share your validation rules as a portable,
language-neutral schema that tools across many ecosystems can consume.

When the `Validate` object has data attached, the schema is enriched with column type
information inferred from the data (e.g., `"type": "integer"` for Int64 columns).

Parameters
----------
path
    An optional file path. If provided, the JSON Schema is written to this file (parent
    directories are created as needed) in addition to being returned.

Returns
-------
dict[str, Any]
    The JSON Schema document as a dictionary. This is a valid JSON Schema that can be
    serialized with `json.dumps()` or consumed by any JSON Schema validator.

Supported Mappings
------------------
The following validation methods have direct JSON Schema equivalents:

| Pointblank method | JSON Schema keyword |
|---|---|
| `col_vals_not_null()` | `required` |
| `col_vals_gt()` | `exclusiveMinimum` |
| `col_vals_ge()` | `minimum` |
| `col_vals_lt()` | `exclusiveMaximum` |
| `col_vals_le()` | `maximum` |
| `col_vals_eq()` | `const` |
| `col_vals_in_set()` | `enum` |
| `col_vals_between()` | `minimum` + `maximum` |
| `col_vals_regex()` | `pattern`  |
| `col_vals_within_spec()` | `format` |

Steps with no JSON Schema equivalent (e.g., `rows_distinct()`, `tbl_match()`,
`col_vals_outside()`) are silently skipped.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's write a validation plan for a table of user accounts and export it as JSON Schema.

```{python}
import pointblank as pb
import polars as pl
import json

accounts = pl.DataFrame(
    {
        "name": ["Ana", "Ben", "Chloé"],
        "age": [34, 27, 45],
        "status": ["active", "inactive", "active"],
    }
)

validation = (
    pb.Validate(data=accounts)
    .col_vals_gt(columns="age", value=0)
    .col_vals_not_null(columns="name")
    .col_vals_in_set(columns="status", set=["active", "inactive"])
    .rows_distinct()
)

schema = validation.to_json_schema()

print(json.dumps(schema, indent=2))
```

Each column has a property: `age > 0` became `exclusiveMinimum`, the set of statuses
became `enum`, and the not-null check on `name` put it in `required`. The `type` of each
property comes from the data attached to the plan. The
[`rows_distinct()`](`pointblank.Validate.rows_distinct`) step has no JSON Schema
equivalent (a schema describes one record at a time), so it isn't in the output. Since the
result is a standard schema, it can be given to any JSON Schema validator, for example to
check records in an API before they're loaded into the table. With `path=`, the schema is
also written to a file.

suggest_improvements(self, model: 'str', api_key: 'str | None' = None, verify_ssl: 'bool' = True, max_reprompts: 'int' = 1) -> 'Any'

Propose AI-generated improvements to this validation plan.

This is a thin, convenience wrapper over
[`EditValidation`](`pointblank.EditValidation`): it profiles the table with
[`DataScan`](`pointblank.DataScan`), derives an instruction that targets gaps in the
current plan (columns with no coverage, missing thresholds), and asks the model to extend
the plan accordingly. As with `EditValidation`, you review the proposed change as a diff
and explicitly accept it.

Parameters
----------
model
    The model to use, in `provider:model` form (e.g., `"anthropic:claude-opus-4-8"`).
api_key
    The API key for the model provider.
verify_ssl
    Whether to verify SSL certificates for provider requests. Defaults to `True`.
max_reprompts
    Maximum automatic re-prompts if the returned plan fails the syntax check.

Returns
-------
EditValidation
    An [`EditValidation`](`pointblank.EditValidation`) with the proposed improvements,
    ready to inspect via `.diff()`/`.changed_steps()` and finalize via `.accept()`.

Examples
--------
```python
import pointblank as pb

validation = pb.Validate(data=pb.load_dataset("small_table")).col_vals_gt(
    columns="d", value=100
)

proposal = validation.suggest_improvements(model="anthropic:claude-opus-4-8")
print(proposal.diff())
improved = proposal.accept()
```

This example isn't executed here because it needs an API key and a call to the model
provider. The plan above only checks column `d`, so the instruction sent to the model asks
it to cover the other columns and to add thresholds. Printing `proposal.diff()` shows the
proposed steps as a diff against the current plan, so you can review them before anything
runs. If they look right, `accept()` returns the revised plan as a new `Validate` object.
That object hasn't been interrogated yet, so call
[`interrogate()`](`pointblank.Validate.interrogate`) on it to run the checks. The original
`validation` object is left unchanged.

from_prompt(self, prompt: 'str', model: 'str', api_key: 'str | None' = None, verify_ssl: 'bool' = True, max_reprompts: 'int' = 1) -> 'Any'

Build a validation plan for this table from a natural-language prompt.

This is the same AI edit flow as [`EditValidation`](`pointblank.EditValidation`) but
starting from an *empty* plan: the model is given a bare `pb.Validate(...)` (carrying this
object's table name, label, and thresholds) plus a `DataScan` profile of the data, and is
asked to author steps that satisfy the prompt.

Parameters
----------
prompt
    A plain-English description of the checks the plan should perform (e.g., "ensure no
    nulls in any id column and that ids are unique").
model
    The model to use, in `provider:model` form (e.g., `"anthropic:claude-opus-4-8"`).
api_key
    The API key for the model provider.
verify_ssl
    Whether to verify SSL certificates for provider requests. Defaults to `True`.
max_reprompts
    Maximum automatic re-prompts if the returned plan fails the syntax check.

Returns
-------
EditValidation
    An [`EditValidation`](`pointblank.EditValidation`) whose revised plan realizes the
    prompt; inspect it with `.diff()`/`.to_code()` and finalize with `.accept()`.

Examples
--------
```python
import pointblank as pb

base = pb.Validate(data=pb.load_dataset("small_table"), tbl_name="small_table")
proposal = base.from_prompt(
    "ensure column a has no nulls and column d is always positive",
    model="anthropic:claude-opus-4-8",
)
plan = proposal.accept()
```

This example isn't executed here because it needs an API key and a call to the model
provider. The `base` object contributes its table name (and any label and thresholds) but
no steps. The model writes steps that satisfy the prompt, so here you'd expect a
not-null check on `a` and a greater-than-zero check on `d`. Before accepting, you can
review the generated plan with `proposal.to_code()`. Then `accept()` returns it as a
`Validate` object that hasn't been interrogated yet. Call
[`interrogate()`](`pointblank.Validate.interrogate`) on it to run the checks. To extend a
plan that already has steps, use
[`suggest_improvements()`](`pointblank.Validate.suggest_improvements`) or
[`EditValidation`](`pointblank.EditValidation`) instead.

get_sundered_data(self, type: 'str' = 'pass', duplicates: 'str | None' = None, duplicates_order_by: 'str | list[str] | None' = None) -> 'Any'

Get the data that passed or failed the validation steps.

Validation of the data is one thing but, sometimes, you want to use the best part of the
input dataset for something else. The `get_sundered_data()` method works with a `Validate`
object that has been interrogated (i.e., the
[`interrogate()`](`pointblank.Validate.interrogate`) method was used). We can get either the
'pass' data piece (rows with no failing test units across all column-value based validation
functions), or, the 'fail' data piece (rows with at least one failing test unit across the
same series of validations).

Details
-------
There are some caveats to sundering. The validation steps considered for this splitting will
only involve steps where:

- of certain check types, where test units are rows or cells checked down a column (e.g.,
the `col_vals_*()` methods and `rows_complete()`)
- `active=` is not set to `False`
- `pre=` has not been given an expression for modifying the input table

So long as these conditions are met, the data will be split into two constituent tables: one
with the rows that passed all validation steps and another with the rows that failed at
least one validation step. Steps that use `segments=` are included, with each segment's
results counting only for the rows in that segment.

Steps that use [`rows_distinct()`](`pointblank.Validate.rows_distinct`) are a special case.
That step counts every copy of a duplicated row as failing, since it can't know which of
them is the 'wrong' one. Such steps are only used for sundering if you say which copy to
keep with `duplicates=`: the kept copy goes to the 'pass' piece (if it passes the other
steps) and the other copies go to the 'fail' piece. With `duplicates="flag_all"`, every
copy goes to the 'fail' piece.

Parameters
----------
type
    The type of data to return. Options are `"pass"` or `"fail"`, where the former returns
    a table only containing rows where test units always passed validation steps, and the
    latter returns a table only containing rows had test units that failed in at least one
    validation step.
duplicates
    How to treat the duplicate rows found by `rows_distinct()` steps. The default `None`
    leaves those steps out of the sundering. Use `"keep_first"` or `"keep_last"` to keep one
    copy of each group of duplicates (the first or last one, in row order or in the order
    given by `duplicates_order_by=`), or `"flag_all"` to treat every copy as failing.
duplicates_order_by
    One or more columns that order the copies of a duplicated row, to decide which copy is
    kept with `duplicates="keep_first"` or `duplicates="keep_last"`. For example, use
    `duplicates="keep_last", duplicates_order_by="updated_at"` to keep the newest record.
    Ties are broken by row order, and a copy with a missing value in one of these columns is
    kept only if every copy is missing that value. Without this, the copies are taken in row
    order (for a database table, which has no row order of its own, rows are ordered by
    their values, so the choice is arbitrary but the same from run to run).

Returns
-------
Any
    A table containing the data that passed or failed the validation steps.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(preview_incl_header=False)
```
Let's create a `Validate` object with three validation steps and then interrogate the data.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [7, 6, 9, 7, 3, 2],
        "b": [9, 8, 10, 5, 10, 6],
        "c": ["c", "d", "a", "b", "a", "b"]
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=5)
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation
```

From the validation table, we can see that the first and second steps each had 4 passing
test units. A failing test unit will mark the entire row as failing in the context of the
`get_sundered_data()` method. We can use this method to get the rows of data that passed the
during interrogation.

```{python}
pb.preview(validation.get_sundered_data())
```

The returned DataFrame contains the rows that passed all validation steps (we passed this
object to [`preview()`](`pointblank.preview`) to show it in an HTML view). From the six-row
input DataFrame, the first two rows and the last two rows had test units that failed
validation. Thus the middle two rows are the only ones that passed all validation steps and
that's what we see in the returned DataFrame.

A [`rows_distinct()`](`pointblank.Validate.rows_distinct`) step only takes part in the
sundering if we say which copy of a duplicated row to keep. Here, the record with an `id` of
`2` appears twice, and we keep the copy with the latest `version`:

```{python}
tbl_versions = pl.DataFrame(
    {
        "id": [1, 2, 2, 3],
        "version": [1, 1, 2, 1],
    }
)

validation_versions = (
    pb.Validate(data=tbl_versions)
    .rows_distinct(columns_subset="id")
    .interrogate()
)

pb.preview(
    validation_versions.get_sundered_data(
        duplicates="keep_last", duplicates_order_by="version"
    )
)
```

The validation report would count both copies of `id` `2` as failing, but with
`duplicates="keep_last"` the copy with `version` `2` is kept in the 'pass' piece, and only
the older copy would be in the 'fail' piece. Without `duplicates=`, the
`rows_distinct()` step would be left out and all four rows would be returned.

See Also
--------
Validate.get_row_flags : Get which steps each row failed, as columns added to the table.
Validate.get_data_extracts : Get the rows that failed each validation step.

get_row_flags(self, duplicates: 'str | None' = 'keep_first', duplicates_order_by: 'str | list[str] | None' = None) -> 'Any'

Get the target table with columns flagging the validation steps each row failed.

Where [`get_sundered_data()`](`pointblank.Validate.get_sundered_data`) splits the table into
the rows that passed every step and those that failed any of them, `get_row_flags()` keeps
every row and says *which* steps each one failed. The returned table is the target table
with these columns added:

- `pb_failed_<i>_`, one for each row-level step (where `<i>` is the step number): `True`
  if the row failed step `<i>`
- `pb_n_failed_`: the number of steps the row failed
- `pb_row_score_`: the share of the row's checks that passed (from `0` to `1`), weighted
  by the dimension weights set with [`config(dimension_weights=...)`](`pointblank.config`)
  just as in [`get_health_score()`](`pointblank.Validate.get_health_score`)

This is useful when removing every row that failed a check is too blunt. You can use the
flags to see which problems occur together, filter on just the checks you care about,
stratify a sample on them, or use `pb_row_score_` to weight rows (e.g., as sample weights
when training a model) instead of dropping them.

Details
-------
The steps that get flag columns are the same ones used by `get_sundered_data()`: steps that
check rows or cells down a column, that are active, and that don't use `pre=`. A row that a
step didn't check (e.g., a row outside the segment of a step that uses `segments=`) isn't
flagged by it, and isn't counted in its `pb_row_score_`. A row that no step checked has a
missing `pb_row_score_`.

Unlike `get_sundered_data()`, steps that use
[`rows_distinct()`](`pointblank.Validate.rows_distinct`) are included by default: one copy
of each group of duplicate rows (the first one) isn't flagged, and the other copies are. So
a `rows_distinct()` flag marks the rows that removing duplicates would remove. Use
`duplicates=` and `duplicates_order_by=` to choose which copy is kept.

A row-level step that can't be used is reported in a warning: that's a step that uses
`pre=` (it can change the rows, so its results may not line up with the table's rows), a
step that couldn't be evaluated, or a `rows_distinct()` step when `duplicates=None`.

Parameters
----------
duplicates
    How to flag the duplicate rows found by `rows_distinct()` steps. With `"keep_first"`
    (the default) or `"keep_last"`, one copy of each group of duplicates (the first or last
    one, in row order or in the order given by `duplicates_order_by=`) isn't flagged.
    `"flag_all"` flags every copy, as the step itself counts them, and `None` leaves
    `rows_distinct()` steps out.
duplicates_order_by
    One or more columns that order the copies of a duplicated row, to decide which copy
    isn't flagged. For example, use `duplicates="keep_last",
    duplicates_order_by="updated_at"` to keep the newest record. Ties are broken by row
    order, and a copy with a missing value in one of these columns is kept only if every
    copy is missing that value. Without this, the copies are taken in row order (for a
    database table, which has no row order of its own, rows are ordered by their values, so
    the choice is arbitrary but the same from run to run).

Returns
-------
Any
    The target table, of the same type, with the flag and summary columns added.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(preview_incl_header=False)
```

Let's validate a table of orders with a check on the amounts, a check for missing values,
and a check for duplicated order IDs.

```{python}
import pointblank as pb
import polars as pl

orders = pl.DataFrame(
    {
        "order_id": [1, 2, 2, 3, 4],
        "amount": [25.0, -5.0, -5.0, None, 40.0],
        "status": ["paid", "paid", "paid", "open", None],
    }
)

validation = (
    pb.Validate(data=orders)
    .col_vals_gt(columns="amount", value=0)
    .rows_complete()
    .rows_distinct(columns_subset="order_id")
    .interrogate()
)

pb.preview(validation.get_row_flags())
```

Each step has a flag column. Both copies of order `2` have a negative amount, so both fail
step 1, but only the second copy is flagged by step 3, since removing duplicates would keep
the first one. Order `3` fails step 1 because its amount is missing (missing values fail
`col_vals_gt()` unless `na_pass=True`) and it also fails step 2. Order `1` passed every
check, so it has a `pb_row_score_` of `1`.

See Also
--------
Validate.get_sundered_data : Split the table into the rows that passed and failed the steps.
Validate.get_health_score : Get a single score for the whole table.

get_data_extracts(self, i: 'int | list[int] | None' = None, frame: 'bool' = False) -> 'dict[int, Any] | Any'

Get the rows that failed for each validation step.

After the [`interrogate()`](`pointblank.Validate.interrogate`) method has been called, the
`get_data_extracts()` method can be used to extract the rows that failed in each
column-value or row-based validation step (e.g.,
[`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),
[`rows_distinct()`](`pointblank.Validate.rows_distinct`), etc.). The method returns a
dictionary of tables containing the rows that failed in every validation step. If
`frame=True` and `i=` is a scalar, the value is conveniently returned as a table (forgoing
the dictionary structure).

Parameters
----------
i
    The validation step number(s) from which the failed rows are obtained. Can be provided
    as a list of integers or a single integer. If `None`, all steps are included.
frame
    If `True` and `i=` is a scalar, return the value as a DataFrame instead of a dictionary.

Returns
-------
dict[int, Any] | Any
    A dictionary of tables containing the rows that failed in every compatible validation
    step. Alternatively, it can be a DataFrame if `frame=True` and `i=` is a scalar.

Compatible Validation Methods for Yielding Extracted Rows
---------------------------------------------------------
The following validation methods operate on column values and will have rows extracted when
there are failing test units.

- [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`)
- [`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)
- [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`)
- [`col_vals_le()`](`pointblank.Validate.col_vals_le`)
- [`col_vals_eq()`](`pointblank.Validate.col_vals_eq`)
- [`col_vals_ne()`](`pointblank.Validate.col_vals_ne`)
- [`col_vals_between()`](`pointblank.Validate.col_vals_between`)
- [`col_vals_outside()`](`pointblank.Validate.col_vals_outside`)
- [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
- [`col_vals_not_in_set()`](`pointblank.Validate.col_vals_not_in_set`)
- [`col_vals_increasing()`](`pointblank.Validate.col_vals_increasing`)
- [`col_vals_decreasing()`](`pointblank.Validate.col_vals_decreasing`)
- [`col_vals_null()`](`pointblank.Validate.col_vals_null`)
- [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`)
- [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`)
- [`col_vals_within_spec()`](`pointblank.Validate.col_vals_within_spec`)
- [`col_vals_expr()`](`pointblank.Validate.col_vals_expr`)
- [`conjointly()`](`pointblank.Validate.conjointly`)
- [`prompt()`](`pointblank.Validate.prompt`)

An extracted row for these validation methods means that a test unit failed for that row in
the validation step.

These row-based validation methods will also have rows extracted should there be failing
rows:

- [`rows_distinct()`](`pointblank.Validate.rows_distinct`)
- [`rows_complete()`](`pointblank.Validate.rows_complete`)

The extracted rows are a subset of the original table and are useful for further analysis
or for understanding the nature of the failing test units.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(preview_incl_header=False)
```
Let's perform a series of validation steps on a Polars DataFrame. We'll use the
[`col_vals_gt()`](`pointblank.Validate.col_vals_gt`) in the first step,
[`col_vals_lt()`](`pointblank.Validate.col_vals_lt`) in the second step, and
[`col_vals_ge()`](`pointblank.Validate.col_vals_ge`) in the third step. The
[`interrogate()`](`pointblank.Validate.interrogate`) method executes the validation; then,
we can extract the rows that failed for each validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [5, 6, 5, 3, 6, 1],
        "b": [1, 2, 1, 5, 2, 6],
        "c": [3, 7, 2, 6, 3, 1],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=4)
    .col_vals_lt(columns="c", value=5)
    .col_vals_ge(columns="b", value=1)
    .interrogate()
)

validation.get_data_extracts()
```

The `get_data_extracts()` method returns a dictionary of tables, where each table contains
a subset of rows from the table. These are the rows that failed for each validation step.

In the first step, the[`col_vals_gt()`](`pointblank.Validate.col_vals_gt`) method was used
to check if the values in column `a` were greater than `4`. The extracted table shows the
rows where this condition was not met; look at the `a` column: all values are less than `4`.

In the second step, the [`col_vals_lt()`](`pointblank.Validate.col_vals_lt`) method was
used to check if the values in column `c` were less than `5`. In the extracted two-row
table, we see that the values in column `c` are greater than `5`.

The third step ([`col_vals_ge()`](`pointblank.Validate.col_vals_ge`)) checked if the values
in column `b` were greater than or equal to `1`. There were no failing test units, so the
extracted table is empty (i.e., has columns but no rows).

The `i=` argument can be used to narrow down the extraction to one or more steps. For
example, to extract the rows that failed in the first step only:

```{python}
validation.get_data_extracts(i=1)
```

Note that the first validation step is indexed at `1` (not `0`). This 1-based indexing is
in place here to match the step numbers reported in the validation table. What we get back
is still a dictionary, but it only contains one table (the one for the first step).

If you want to get the extracted table as a DataFrame, set `frame=True` and provide a scalar
value for `i`. For example, to get the extracted table for the second step as a DataFrame:

```{python}
pb.preview(validation.get_data_extracts(i=2, frame=True))
```

The extracted table is now a DataFrame, which can serve as a more convenient format for
further analysis or visualization. We further used the [`preview()`](`pointblank.preview`)
function to show the DataFrame in an HTML view.

all_passed(self) -> 'bool'

Determine if every validation step passed perfectly, with no failing test units.

The `all_passed()` method determines if every validation step passed perfectly, with no
failing test units. This method is useful for quickly checking if the table passed all
validation steps with flying colors. If there's even a single failing test unit in any
validation step, this method will return `False`.

This validation metric might be overly stringent for some validation plans where failing
test units are generally expected (and the strategy is to monitor data quality over time).
However, the value of `all_passed()` could be suitable for validation plans designed to
ensure that every test unit passes perfectly (e.g., checks for column presence,
null-checking tests, etc.).

Returns
-------
bool
    `True` if all validation steps had no failing test units, `False` otherwise.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps, and the second step will have a failing test
unit (the value `10` isn't less than `9`). After interrogation, the `all_passed()` method is
used to determine if all validation steps passed perfectly.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 9, 5],
        "b": [5, 6, 10, 3],
        "c": ["a", "b", "a", "a"],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=0)
    .col_vals_lt(columns="b", value=9)
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation.all_passed()
```

The returned value is `False` since the second validation step had a failing test unit. If
it weren't for that one failing test unit, the return value would have been `True`.

assert_passing(self) -> 'None'

Raise an `AssertionError` if all tests are not passing.

The `assert_passing()` method will raise an `AssertionError` if a test does not pass. This
method simply wraps [`all_passed()`](`pointblank.Validate.all_passed`) for more ready use
in test suites. The step number and assertion made is printed in the `AssertionError`
message if a failure occurs, ensuring some details are preserved.

If the validation has not yet been interrogated, this method will automatically call
[`interrogate()`](`pointblank.Validate.interrogate`) with default parameters before checking
for passing tests.

Raises
-------
AssertionError
    If any validation step has failing test units.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps, and the second step will have a failing test
unit (the value `10` isn't less than `9`). The `assert_passing()` method is used to assert
that all validation steps passed perfectly, automatically performing the interrogation if
needed.

```{python}
#| error: True

import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
    "a": [1, 2, 9, 5],
    "b": [5, 6, 10, 3],
    "c": ["a", "b", "a", "a"],
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=0)
    .col_vals_lt(columns="b", value=9) # this assertion is false
    .col_vals_in_set(columns="c", set=["a", "b"])
)

# No need to call `interrogate()` explicitly
validation.assert_passing()
```

The `AssertionError` lists each step that had failing test units, here only step 2, with
a description of what it expected. In a test suite (with pytest, for instance) or a CI job,
that error fails the run, which is the point: the data can't move on until it passes every
check. When a few failures are acceptable, set thresholds on the plan and use
[`assert_below_threshold()`](`pointblank.Validate.assert_below_threshold`) instead.

assert_below_threshold(self, level: 'str' = 'warning', i: 'int | None' = None, message: 'str | None' = None) -> 'None'

Raise an `AssertionError` if validation steps exceed a specified threshold level.

The `assert_below_threshold()` method checks whether validation steps' failure rates are
below a given threshold level (`"warning"`, `"error"`, or `"critical"`). This is
particularly useful in automated testing environments where you want to ensure your data
quality meets minimum standards before proceeding.

If any validation step exceeds the specified threshold level, an `AssertionError` will be
raised with details about which steps failed. If the validation has not yet been
interrogated, this method will automatically call
[`interrogate()`](`pointblank.Validate.interrogate`) with default parameters.

Parameters
----------
level
    The threshold level to check against, which could be any of `"warning"` (the default),
    `"error"`, or `"critical"`. An `AssertionError` will be raised if any validation step
    exceeds this level.
i
    Specific validation step number(s) to check. Can be provided as a single integer or a
    list of integers. If `None` (the default), all steps are checked.
message
    Custom error message to use if assertion fails. If `None`, a default message will be
    generated that lists the specific steps that exceeded the threshold.

Returns
-------
None

Raises
------
AssertionError
    If any specified validation step exceeds the given threshold level.
ValueError
    If an invalid threshold level is provided.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Below are some examples of how to use the `assert_below_threshold()` method. First, we'll
create a simple Polars DataFrame with two columns (`a` and `b`).

```{python}
import polars as pl

tbl = pl.DataFrame({
    "a": [7, 4, 9, 7, 12],
    "b": [9, 8, 10, 5, 10]
})
```

Then a validation plan will be created with thresholds (`warning=0.1`, `error=0.2`,
`critical=0.3`). After interrogating, we display the validation report table:

```{python}
import pointblank as pb

validation = (
    pb.Validate(data=tbl, thresholds=(0.1, 0.2, 0.3))
    .col_vals_gt(columns="a", value=5)   # 1 failing test unit
    .col_vals_lt(columns="b", value=10)  # 2 failing test units
    .interrogate()
)

validation
```

Using `assert_below_threshold(level="warning")` will raise an `AssertionError` if any step
exceeds the 'warning' threshold:

```{python}
try:
    validation.assert_below_threshold(level="warning")
except AssertionError as e:
    print(f"Assertion failed: {e}")
```

Check a specific step against the 'critical' threshold using the `i=` parameter:

```{python}
validation.assert_below_threshold(level="critical", i=1)  # Won't raise an error
```

As the first step is below the 'critical' threshold (it exceeds the 'warning' and 'error'
thresholds), no error is raised and nothing is printed.

We can also provide a custom error message with the `message=` parameter. Let's try that
here:

```{python}
try:
    validation.assert_below_threshold(
        level="error",
        message="Data quality too low for processing!"
    )
except AssertionError as e:
    print(f"Custom error: {e}")
```

The custom message replaces the default one, which helps when the error ends up in a log
or a notification read by someone who doesn't know the validation plan. Like
[`assert_passing()`](`pointblank.Validate.assert_passing`), this method interrogates the
plan first if that hasn't been done yet.

See Also
--------
Validate.warning : Get the 'warning' status for each validation step.
Validate.error : Get the 'error' status for each validation step.
Validate.critical : Get the 'critical' status for each validation step.
Validate.assert_passing : Assert that all validations pass completely.

assert_dimension_scores(self, thresholds: 'dict[str, float] | None' = None, message: 'str | None' = None) -> 'None'

Raise an `AssertionError` if any dimension's health score falls below a minimum.

The `assert_dimension_scores()` method checks each data quality dimension's score (from
[`get_dimension_scores()`](`pointblank.Validate.get_dimension_scores`)) against a minimum
acceptable value. This is useful in automated testing and CI environments where you want to
fail the run when, say, the completeness score drops below `95`.

Parameters
----------
thresholds
    A mapping of dimension name to a minimum acceptable score (`0`-`100`). If `None`, the
    minimums set via [`config(dimension_thresholds=...)`](`pointblank.config`) are used. A
    dimension present in the thresholds but absent from the validation is ignored.
message
    Custom error message to use if the assertion fails. If `None`, a default message that
    lists the offending dimensions (with actual vs. required scores) is generated.

Returns
-------
None

Raises
------
AssertionError
    If any dimension's score is below its specified minimum.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's validate the `small_table` dataset with a not-null check (a completeness step) and
a range check (a validity step).

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

validation = (
    pb.Validate(data=small_table)
    .col_vals_not_null(columns="c")
    .col_vals_gt(columns="d", value=1000)
    .interrogate()
)

validation.get_dimension_scores()
```

Requiring completeness of at least 80 and validity of at least 50 passes, so nothing
happens:

```{python}
validation.assert_dimension_scores(thresholds={"completeness": 80, "validity": 50})
```

Raising the completeness minimum to 95 makes the assertion fail. The `AssertionError` is
caught here so that its message can be shown:

```{python}
try:
    validation.assert_dimension_scores(thresholds={"completeness": 95, "validity": 50})
except AssertionError as e:
    print(e)
```

The message lists each dimension below its minimum, with the actual and required scores.
In a test suite or a CI job, leave the error uncaught so that it fails the run. To use the
same minimums everywhere, set them once with `config(dimension_thresholds=...)` and call
`assert_dimension_scores()` without `thresholds=`.

See Also
--------
Validate.get_dimension_scores : Get the scores without raising.
config : Set default per-dimension thresholds globally.

above_threshold(self, level: 'str' = 'warning', i: 'int | None' = None) -> 'bool'

Check if any validation steps exceed a specified threshold level.

The `above_threshold()` method checks whether validation steps exceed a given threshold
level. This provides a non-exception-based alternative to
[`assert_below_threshold()`](`pointblank.Validate.assert_below_threshold`) for conditional
workflow control based on validation results.

This method is useful in scenarios where you want to check if any validation steps failed
beyond a certain threshold without raising an exception, allowing for more flexible
programmatic responses to validation issues.

Parameters
----------
level
    The threshold level to check against. Valid options are: `"warning"` (the least severe
    threshold level), `"error"` (the middle severity threshold level), and `"critical"` (the
    most severe threshold level). The default is `"warning"`.
i
    Specific validation step number(s) to check. If a single integer, checks only that step.
    If a list of integers, checks all specified steps. If `None` (the default), checks all
    validation steps. Step numbers are 1-based (first step is `1`, not `0`).

Returns
-------
bool
    `True` if any of the specified validation steps exceed the given threshold level,
    `False` otherwise.

Raises
------
ValueError
    If an invalid threshold level is provided.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Below are some examples of how to use the `above_threshold()` method. First, we'll create a
simple Polars DataFrame with a single column (`values`).

```{python}
import polars as pl

tbl = pl.DataFrame({
    "values": [1, 2, 3, 4, 5, 0, -1]
})
```

Then a validation plan will be created with thresholds (`warning=0.1`, `error=0.2`,
`critical=0.3`). After interrogating, we display the validation report table:

```{python}
import pointblank as pb

validation = (
    pb.Validate(data=tbl, thresholds=(0.1, 0.2, 0.3))
    .col_vals_gt(columns="values", value=0)
    .col_vals_lt(columns="values", value=10)
    .col_vals_between(columns="values", left=0, right=5)
    .interrogate()
)

validation
```

Let's check if any steps exceed the 'warning' threshold with the `above_threshold()` method.
A message will be printed if that's the case:

```{python}
if validation.above_threshold(level="warning"):
    print("Some steps have exceeded the warning threshold")
```

Check if only steps 2 and 3 exceed the 'error' threshold through use of the `i=` argument:

```{python}
if validation.above_threshold(level="error", i=[2, 3]):
    print("Steps 2 and/or 3 have exceeded the error threshold")
```

You can use this in a workflow to conditionally trigger processes. Here's a snippet of how
you might use this in a function:

```python
def process_data(validation_obj):
    # Only continue processing if validation passes critical thresholds
    if not validation_obj.above_threshold(level="critical"):
        # Continue with processing
        print("Data meets critical quality thresholds, proceeding...")
        return True
    else:
        # Log failure and stop processing
        print("Data fails critical quality checks, aborting...")
        return False
```

Note that this is just a suggestion for how to implement conditional workflow processes. You
should adapt this pattern to your specific requirements, which might include  different
threshold levels, custom logging mechanisms, or integration with your organization's data
pipelines and notification systems.

See Also
--------
Validate.assert_below_threshold : Raise an exception instead if thresholds are exceeded.
Validate.warning : Get the 'warning' status for each validation step.
Validate.error : Get the 'error' status for each validation step.
Validate.critical : Get the 'critical' status for each validation step.

get_dimension_scores(self) -> 'dict[str, float]'

Get per-dimension health scores from the validation results.

Each validation step is associated with a data quality dimension (e.g., `"completeness"`,
`"validity"`, `"uniqueness"`, `"consistency"`, `"timeliness"`, or `"volume"`), either
inferred automatically from the assertion type or set explicitly via the `dimension=`
parameter on a validation method. This method rolls the per-step results up into a
test-unit-weighted pass rate (`0`-`100`) for each dimension present in the plan.

Scores are weighted by test units, so larger tables and steps with more test units
contribute proportionally more to each dimension's score. Only steps that have been
interrogated contribute; inactive steps are excluded.

Returns
-------
dict[str, float]
    A dictionary mapping each dimension name to its score (a percentage from `0` to `100`).
    Returns an empty dictionary if the validation has not been interrogated.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's validate the `small_table` dataset with three steps, each of which falls into a
different dimension: a not-null check (completeness), a range check (validity), and a
check for duplicate rows (uniqueness).

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

validation = (
    pb.Validate(data=small_table)
    .col_vals_not_null(columns="c")
    .col_vals_gt(columns="d", value=1000)
    .rows_distinct()
    .interrogate()
)

validation.get_dimension_scores()
```

Each step has 13 test units (one per row). Completeness scores 84.62 because 11 of the
13 values in `c` are present, and uniqueness gets the same score from the 2 duplicated
rows. Validity is lowest: only 7 of the 13 values in `d` are above 1000. A breakdown like
this points to the kind of problem to fix, which the single number from
[`get_health_score()`](`pointblank.Validate.get_health_score`) doesn't.

See Also
--------
Validate.get_health_score : Get a single overall score across all dimensions.

get_health_score(self) -> 'float'

Get the overall data quality health score from the validation results.

The health score is a single number (a percentage from `0` to `100`) that summarizes the
overall quality of the data across all validation steps. It is computed as a test-unit
weighted pass rate: the total number of passing test units divided by the total number of
test units. This means larger tables and steps operating over more rows contribute
proportionally more to the score, so it tracks data volume rather than mere step count.

Per-dimension weights can be set globally via
[`config(dimension_weights=...)`](`pointblank.config`) for organizations that consider some
dimensions (e.g., completeness) more critical than others. A weight scales that dimension's
test-unit contribution to the score (so a dimension's influence is its weight times its
test-unit count); the returned score is always within `[0, 100]`.

Returns
-------
float
    The overall health score as a percentage from `0` to `100`. Returns `100.0` if the
    validation has not been interrogated or contains no scorable steps.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's validate the `small_table` dataset with a not-null check, a range check, and a check
for duplicate rows, then get the overall score.

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

validation = (
    pb.Validate(data=small_table)
    .col_vals_not_null(columns="c")
    .col_vals_gt(columns="d", value=1000)
    .rows_distinct()
    .interrogate()
)

validation.get_health_score()
```

The three steps have 39 test units in all, and 29 of them passed, for a score of 74.36.
If missing values matter more to you than the other problems, give completeness a larger
weight:

```{python}
pb.config(dimension_weights={"completeness": 3})

validation.get_health_score()
```

The completeness step now counts three times, so its 84.62 pass rate pulls the score up
to 78.46. Note that each call to [`config()`](`pointblank.config`) sets every option,
so any other settings you rely on (such as those for the report) need to be included in
the same call. Calling `pb.config()` with no arguments goes back to the defaults, with
all dimensions weighted equally.

See Also
--------
Validate.get_dimension_scores : Get a per-dimension breakdown of the score.

get_scorecard(self, title: 'str | None' = ':default:') -> 'GT'

Get a data quality scorecard as a GT table.

The `get_scorecard()` method produces a compact, standalone scorecard that summarizes data
quality across dimensions. It shows the overall health score prominently, along with a
per-dimension breakdown (a color-coded bar, the dimension's score, and its passing/total
test units). Unlike the full validation report, the scorecard focuses purely on the
aggregate health picture, making it well-suited for dashboards and executive summaries.

The returned object is a Great Tables `GT` object, so it can be displayed directly, exported
to HTML (via `.as_raw_html()`), or saved to an image file (via `.save()`).

Parameters
----------
title
    Options for customizing the title of the scorecard. The default `":default:"` produces
    a generic title (optionally including the table name). Use `":tbl_name:"` to show just
    the table name, `":none:"` for no title, or provide your own Markdown text.

Returns
-------
GT
    A `GT` object representing the scorecard.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's validate the `small_table` dataset with steps in three dimensions and get the
scorecard.

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

validation = (
    pb.Validate(data=small_table, tbl_name="small_table")
    .col_vals_not_null(columns="c")
    .col_vals_gt(columns="d", value=1000)
    .rows_distinct()
    .interrogate()
)

validation.get_scorecard()
```

The overall health score (74%) is at the top, and each dimension has its own row with a
bar, its score, and the passing and total test units. Validity has the shortest bar, since
only 7 of the 13 values in `d` are above 1000. The scorecard leaves out the details of
individual steps, which makes it a good summary to share, while
[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) gives the full picture
for whoever has to fix the data.

See Also
--------
Validate.get_dimension_scores : Get the per-dimension scores behind the scorecard.
Validate.get_health_score : Get the overall score behind the scorecard.
Validate.get_tabular_report : Get the full per-step validation report.

n(self, i: 'int | list[int] | None' = None, scalar: 'bool' = False) -> 'dict[int, int] | int'

Provides a dictionary of the number of test units for each validation step.

The `n()` method provides the number of test units for each validation step. This is the
total number of test units that were evaluated in the validation step. It is always an
integer value.

Test units are the atomic units of the validation process. Different validations can have
different numbers of test units. For example, a validation that checks for the presence of
a column in a table will have a single test unit. A validation that checks for the presence
of a value in a column will have as many test units as there are rows in the table.

The method provides a dictionary of the number of test units for each validation step. If
the `scalar=True` argument is provided and `i=` is a scalar, the value is returned as a
scalar instead of a dictionary. The total number of test units for a validation step is the
sum of the number of passing and failing test units (i.e., `n = n_passed + n_failed`).

Parameters
----------
i
    The validation step number(s) from which the number of test units is obtained.
    Can be provided as a list of integers or a single integer. If `None`, all steps are
    included.
scalar
    If `True` and `i=` is a scalar, return the value as a scalar instead of a dictionary.

Returns
-------
dict[int, int] | int
    A dictionary of the number of test units for each validation step or a scalar value.

Examples
--------
Different types of validation steps can have different numbers of test units. In the example
below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and `c`). There
will be three validation steps, and the number of test units for each step will be a little
bit different.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [1, 2, 9, 5],
        "b": [5, 6, 10, 3],
        "c": ["a", "b", "a", "a"],
    }
)

# Define a preprocessing function
def filter_by_a_gt_1(df):
    return df.filter(pl.col("a") > 1)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=0)
    .col_exists(columns="b")
    .col_vals_lt(columns="b", value=9, pre=filter_by_a_gt_1)
    .interrogate()
)
```

The first validation step checks that all values in column `a` are greater than `0`. Let's
use the `n()` method to determine the number of test units this validation step.

```{python}
validation.n(i=1, scalar=True)
```

The returned value of `4` is the number of test units for the first validation step. This
value is the same as the number of rows in the table.

The second validation step checks for the existence of column `b`. Using the `n()` method
we can get the number of test units for this the second step.

```{python}
validation.n(i=2, scalar=True)
```

There's a single test unit here because the validation step is checking for the presence of
a single column.

The third validation step checks that all values in column `b` are less than `9` after
filtering the table to only include rows where the value in column `a` is greater than `1`.
Because the table is filtered, the number of test units will be less than the total number
of rows in the input table. Let's prove this by using the `n()` method.

```{python}
validation.n(i=3, scalar=True)
```

The returned value of `3` is the number of test units for the third validation step. When
using the `pre=` argument, the input table can be mutated before performing the validation.
The `n()` method is a good way to determine whether the mutation performed as expected.

In all of these examples, the `scalar=True` argument was used to return the value as a
scalar integer value. If `scalar=False`, the method will return a dictionary with an entry
for the validation step number (from the `i=` argument) and the number of test units.
Futhermore, leaving out the `i=` argument altogether will return a dictionary with filled
with the number of test units for each validation step. Here's what that looks like:

```{python}
validation.n()
```

The dictionary is keyed by step number, so it lines up with the step numbers shown in the
validation report. A list such as `i=[1, 3]` returns entries for just those steps. To get
how many of these test units passed or failed, use
[`n_passed()`](`pointblank.Validate.n_passed`) and
[`n_failed()`](`pointblank.Validate.n_failed`), which take the same `i=` and `scalar=`
arguments.

n_passed(self, i: 'int | list[int] | None' = None, scalar: 'bool' = False) -> 'dict[int, int] | int'

Provides a dictionary of the number of test units that passed for each validation step.

The `n_passed()` method provides the number of test units that passed for each validation
step. This is the number of test units that passed in the the validation step. It is always
some integer value between `0` and the total number of test units.

Test units are the atomic units of the validation process. Different validations can have
different numbers of test units. For example, a validation that checks for the presence of
a column in a table will have a single test unit. A validation that checks for the presence
of a value in a column will have as many test units as there are rows in the table.

The method provides a dictionary of the number of passing test units for each validation
step. If the `scalar=True` argument is provided and `i=` is a scalar, the value is returned
as a scalar instead of a dictionary. Furthermore, a value obtained here will be the
complement to the analogous value returned by the
[`n_passed()`](`pointblank.Validate.n_passed`) method (i.e., `n - n_failed`).

Parameters
----------
i
    The validation step number(s) from which the number of passing test units is obtained.
    Can be provided as a list of integers or a single integer. If `None`, all steps are
    included.
scalar
    If `True` and `i=` is a scalar, return the value as a scalar instead of a dictionary.

Returns
-------
dict[int, int] | int
    A dictionary of the number of passing test units for each validation step or a scalar
    value.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps and, as it turns out, all of them will have
failing test units. After interrogation, the `n_passed()` method is used to determine the
number of passing test units for each validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [7, 4, 9, 7, 12],
        "b": [9, 8, 10, 5, 10],
        "c": ["a", "b", "c", "a", "b"]
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=5)
    .col_vals_gt(columns="b", value=pb.col("a"))
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation.n_passed()
```

The returned dictionary shows that all validation steps had no passing test units (each
value was less than `5`, which is the total number of test units for each step).

If we wanted to check the number of passing test units for a single validation step, we can
provide the step number. Also, we could forego the dictionary and get a scalar value by
setting `scalar=True` (ensuring that `i=` is a scalar).

```{python}
validation.n_passed(i=1)
```

The returned value of `4` is the number of passing test units for the first validation step.

n_failed(self, i: 'int | list[int] | None' = None, scalar: 'bool' = False) -> 'dict[int, int] | int'

Provides a dictionary of the number of test units that failed for each validation step.

The `n_failed()` method provides the number of test units that failed for each validation
step. This is the number of test units that did not pass in the the validation step. It is
always some integer value between `0` and the total number of test units.

Test units are the atomic units of the validation process. Different validations can have
different numbers of test units. For example, a validation that checks for the presence of
a column in a table will have a single test unit. A validation that checks for the presence
of a value in a column will have as many test units as there are rows in the table.

The method provides a dictionary of the number of failing test units for each validation
step. If the `scalar=True` argument is provided and `i=` is a scalar, the value is returned
as a scalar instead of a dictionary. Furthermore, a value obtained here will be the
complement to the analogous value returned by the
[`n_passed()`](`pointblank.Validate.n_passed`) method (i.e., `n - n_passed`).

Parameters
----------
i
    The validation step number(s) from which the number of failing test units is obtained.
    Can be provided as a list of integers or a single integer. If `None`, all steps are
    included.
scalar
    If `True` and `i=` is a scalar, return the value as a scalar instead of a dictionary.

Returns
-------
dict[int, int] | int
    A dictionary of the number of failing test units for each validation step or a scalar
    value.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps and, as it turns out, all of them will have
failing test units. After interrogation, the `n_failed()` method is used to determine the
number of failing test units for each validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [7, 4, 9, 7, 12],
        "b": [9, 8, 10, 5, 10],
        "c": ["a", "b", "c", "a", "b"]
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=5)
    .col_vals_gt(columns="b", value=pb.col("a"))
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation.n_failed()
```

The returned dictionary shows that all validation steps had failing test units.

If we wanted to check the number of failing test units for a single validation step, we can
provide the step number. Also, we could forego the dictionary and get a scalar value by
setting `scalar=True` (ensuring that `i=` is a scalar).

```{python}
validation.n_failed(i=1)
```

The returned value of `1` is the number of failing test units for the first validation step.

f_passed(self, i: 'int | list[int] | None' = None, scalar: 'bool' = False) -> 'dict[int, float] | float'

Provides a dictionary of the fraction of test units that passed for each validation step.

A measure of the fraction of test units that passed is provided by the `f_passed` attribute.
This is the fraction of test units that passed the validation step over the total number of
test units. Given this is a fractional value, it will always be in the range of `0` to `1`.

Test units are the atomic units of the validation process. Different validations can have
different numbers of test units. For example, a validation that checks for the presence of
a column in a table will have a single test unit. A validation that checks for the presence
of a value in a column will have as many test units as there are rows in the table.

This method provides a dictionary of the fraction of passing test units for each validation
step. If the `scalar=True` argument is provided and `i=` is a scalar, the value is returned
as a scalar instead of a dictionary. Furthermore, a value obtained here will be the
complement to the analogous value returned by the
[`f_failed()`](`pointblank.Validate.f_failed`) method (i.e., `1 - f_failed()`).

Parameters
----------
i
    The validation step number(s) from which the fraction of passing test units is obtained.
    Can be provided as a list of integers or a single integer. If `None`, all steps are
    included.
scalar
    If `True` and `i=` is a scalar, return the value as a scalar instead of a dictionary.

Returns
-------
dict[int, float] | float
    A dictionary of the fraction of passing test units for each validation step or a scalar
    value.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps, all having some failing test units. After
interrogation, the `f_passed()` method is used to determine the fraction of passing test
units for each validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [7, 4, 9, 7, 12, 3, 10],
        "b": [9, 8, 10, 5, 10, 6, 2],
        "c": ["a", "b", "c", "a", "b", "d", "c"]
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=5)
    .col_vals_gt(columns="b", value=pb.col("a"))
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation.f_passed()
```

The returned dictionary shows the fraction of passing test units for each validation step.
The values are all less than `1` since there were failing test units in each step.

If we wanted to check the fraction of passing test units for a single validation step, we
can provide the step number. Also, we could have the value returned as a scalar by setting
`scalar=True` (ensuring that `i=` is a scalar).

```{python}
validation.f_passed(i=1)
```

The returned value is the proportion of passing test units for the first validation step
(5 passing test units out of 7 total test units).

f_failed(self, i: 'int | list[int] | None' = None, scalar: 'bool' = False) -> 'dict[int, float] | float'

Provides a dictionary of the fraction of test units that failed for each validation step.

A measure of the fraction of test units that failed is provided by the `f_failed` attribute.
This is the fraction of test units that failed the validation step over the total number of
test units. Given this is a fractional value, it will always be in the range of `0` to `1`.

Test units are the atomic units of the validation process. Different validations can have
different numbers of test units. For example, a validation that checks for the presence of
a column in a table will have a single test unit. A validation that checks for the presence
of a value in a column will have as many test units as there are rows in the table.

This method provides a dictionary of the fraction of failing test units for each validation
step. If the `scalar=True` argument is provided and `i=` is a scalar, the value is returned
as a scalar instead of a dictionary. Furthermore, a value obtained here will be the
complement to the analogous value returned by the
[`f_passed()`](`pointblank.Validate.f_passed`) method (i.e., `1 - f_passed()`).

Parameters
----------
i
    The validation step number(s) from which the fraction of failing test units is obtained.
    Can be provided as a list of integers or a single integer. If `None`, all steps are
    included.
scalar
    If `True` and `i=` is a scalar, return the value as a scalar instead of a dictionary.

Returns
-------
dict[int, float] | float
    A dictionary of the fraction of failing test units for each validation step or a scalar
    value.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps, all having some failing test units. After
interrogation, the `f_failed()` method is used to determine the fraction of failing test
units for each validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [7, 4, 9, 7, 12, 3, 10],
        "b": [9, 8, 10, 5, 10, 6, 2],
        "c": ["a", "b", "c", "a", "b", "d", "c"]
    }
)

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=5)
    .col_vals_gt(columns="b", value=pb.col("a"))
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation.f_failed()
```

The returned dictionary shows the fraction of failing test units for each validation step.
The values are all greater than `0` since there were failing test units in each step.

If we wanted to check the fraction of failing test units for a single validation step, we
can provide the step number. Also, we could have the value returned as a scalar by setting
`scalar=True` (ensuring that `i=` is a scalar).

```{python}
validation.f_failed(i=1)
```

The returned value is the proportion of failing test units for the first validation step
(2 failing test units out of 7 total test units).

warning(self, i: 'int | list[int] | None' = None, scalar: 'bool' = False) -> 'dict[int, bool] | bool'

Get the 'warning' level status for each validation step.

The 'warning' status for a validation step is `True` if the fraction of failing test units
meets or exceeds the threshold for the 'warning' level. Otherwise, the status is `False`.

The ascribed name of 'warning' is semantic and does not imply that a warning message is
generated, it is simply a status indicator that could be used to trigger some action to be
taken. Here's how it fits in with other status indicators:

- 'warning': the status obtained by calling 'warning()', least severe
- 'error': the status obtained by calling [`error()`](`pointblank.Validate.error`), middle
severity
- 'critical': the status obtained by calling [`critical()`](`pointblank.Validate.critical`),
most severe

This method provides a dictionary of the 'warning' status for each validation step. If the
`scalar=True` argument is provided and `i=` is a scalar, the value is returned as a scalar
instead of a dictionary.

Parameters
----------
i
    The validation step number(s) from which the 'warning' status is obtained. Can be
    provided as a list of integers or a single integer. If `None`, all steps are included.
scalar
    If `True` and `i=` is a scalar, return the value as a scalar instead of a dictionary.

Returns
-------
dict[int, bool] | bool
    A dictionary of the 'warning' status for each validation step or a scalar value.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps, and the first step will have some failing test
units, the rest will be completely passing. We've set thresholds here for each of the steps
by using `thresholds=(2, 4, 5)`, which means:

- the 'warning' threshold is `2` failing test units
- the 'error' threshold is `4` failing test units
- the 'critical' threshold is `5` failing test units

After interrogation, the `warning()` method is used to determine the 'warning' status for
each validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [7, 4, 9, 7, 12, 3, 10],
        "b": [9, 8, 10, 5, 10, 6, 2],
        "c": ["a", "b", "a", "a", "b", "b", "a"]
    }
)

validation = (
    pb.Validate(data=tbl, thresholds=(2, 4, 5))
    .col_vals_gt(columns="a", value=5)
    .col_vals_lt(columns="b", value=15)
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation.warning()
```

The returned dictionary provides the 'warning' status for each validation step. The first
step has a `True` value since the number of failing test units meets the threshold for the
'warning' level. The second and third steps have `False` values since the number of failing
test units was `0`, which is below the threshold for the 'warning' level.

We can also visually inspect the 'warning' status across all steps by viewing the validation
table:

```{python}
validation
```

We can see that there's a filled gray circle in the first step (look to the far right side,
in the `W` column) indicating that the 'warning' threshold was met. The other steps have
empty gray circles. This means that thresholds were 'set but not met' in those steps.

If we wanted to check the 'warning' status for a single validation step, we can provide the
step number. Also, we could have the value returned as a scalar by setting `scalar=True`
(ensuring that `i=` is a scalar).

```{python}
validation.warning(i=1)
```

The returned value is `True`, indicating that the first validation step had met the
'warning' threshold.

error(self, i: 'int | list[int] | None' = None, scalar: 'bool' = False) -> 'dict[int, bool] | bool'

Get the 'error' level status for each validation step.

The 'error' status for a validation step is `True` if the fraction of failing test units
meets or exceeds the threshold for the 'error' level. Otherwise, the status is `False`.

The ascribed name of 'error' is semantic and does not imply that the validation process
is halted, it is simply a status indicator that could be used to trigger some action to be
taken. Here's how it fits in with other status indicators:

- 'warning': the status obtained by calling [`warning()`](`pointblank.Validate.warning`),
least severe
- 'error': the status obtained by calling `error()`, middle severity
- 'critical': the status obtained by calling [`critical()`](`pointblank.Validate.critical`),
most severe

This method provides a dictionary of the 'error' status for each validation step. If the
`scalar=True` argument is provided and `i=` is a scalar, the value is returned as a scalar
instead of a dictionary.

Parameters
----------
i
    The validation step number(s) from which the 'error' status is obtained. Can be
    provided as a list of integers or a single integer. If `None`, all steps are included.
scalar
    If `True` and `i=` is a scalar, return the value as a scalar instead of a dictionary.

Returns
-------
dict[int, bool] | bool
    A dictionary of the 'error' status for each validation step or a scalar value.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps, and the first step will have some failing test
units, the rest will be completely passing. We've set thresholds here for each of the steps
by using `thresholds=(2, 4, 5)`, which means:

- the 'warning' threshold is `2` failing test units
- the 'error' threshold is `4` failing test units
- the 'critical' threshold is `5` failing test units

After interrogation, the `error()` method is used to determine the 'error' status for each
validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [3, 4, 9, 7, 2, 3, 8],
        "b": [9, 8, 10, 5, 10, 6, 2],
        "c": ["a", "b", "a", "a", "b", "b", "a"]
    }
)

validation = (
    pb.Validate(data=tbl, thresholds=(2, 4, 5))
    .col_vals_gt(columns="a", value=5)
    .col_vals_lt(columns="b", value=15)
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation.error()
```

The returned dictionary provides the 'error' status for each validation step. The first step
has a `True` value since the number of failing test units meets the threshold for the
'error' level. The second and third steps have `False` values since the number of failing
test units was `0`, which is below the threshold for the 'error' level.

We can also visually inspect the 'error' status across all steps by viewing the validation
table:

```{python}
validation
```

We can see that there are filled gray and yellow circles in the first step (far right side,
in the `W` and `E` columns) indicating that the 'warning' and 'error' thresholds were met.
The other steps have empty gray and yellow circles. This means that thresholds were 'set but
not met' in those steps.

If we wanted to check the 'error' status for a single validation step, we can provide the
step number. Also, we could have the value returned as a scalar by setting `scalar=True`
(ensuring that `i=` is a scalar).

```{python}
validation.error(i=1)
```

The returned value is `True`, indicating that the first validation step had the 'error'
threshold met.

critical(self, i: 'int | list[int] | None' = None, scalar: 'bool' = False) -> 'dict[int, bool] | bool'

Get the 'critical' level status for each validation step.

The 'critical' status for a validation step is `True` if the fraction of failing test units
meets or exceeds the threshold for the 'critical' level. Otherwise, the status is `False`.

The ascribed name of 'critical' is semantic and is thus simply a status indicator that could
be used to trigger some action to be take. Here's how it fits in with other status
indicators:

- 'warning': the status obtained by calling [`warning()`](`pointblank.Validate.warning`),
least severe
- 'error': the status obtained by calling [`error()`](`pointblank.Validate.error`), middle
severity
- 'critical': the status obtained by calling `critical()`, most severe

This method provides a dictionary of the 'critical' status for each validation step. If the
`scalar=True` argument is provided and `i=` is a scalar, the value is returned as a scalar
instead of a dictionary.

Parameters
----------
i
    The validation step number(s) from which the 'critical' status is obtained. Can be
    provided as a list of integers or a single integer. If `None`, all steps are included.
scalar
    If `True` and `i=` is a scalar, return the value as a scalar instead of a dictionary.

Returns
-------
dict[int, bool] | bool
    A dictionary of the 'critical' status for each validation step or a scalar value.

Examples
--------
In the example below, we'll use a simple Polars DataFrame with three columns (`a`, `b`, and
`c`). There will be three validation steps, and the first step will have many failing test
units, the rest will be completely passing. We've set thresholds here for each of the steps
by using `thresholds=(2, 4, 5)`, which means:

- the 'warning' threshold is `2` failing test units
- the 'error' threshold is `4` failing test units
- the 'critical' threshold is `5` failing test units

After interrogation, the `critical()` method is used to determine the 'critical' status for
each validation step.

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "a": [2, 4, 4, 7, 2, 3, 8],
        "b": [9, 8, 10, 5, 10, 6, 2],
        "c": ["a", "b", "a", "a", "b", "b", "a"]
    }
)

validation = (
    pb.Validate(data=tbl, thresholds=(2, 4, 5))
    .col_vals_gt(columns="a", value=5)
    .col_vals_lt(columns="b", value=15)
    .col_vals_in_set(columns="c", set=["a", "b"])
    .interrogate()
)

validation.critical()
```

The returned dictionary provides the 'critical' status for each validation step. The first
step has a `True` value since the number of failing test units meets the threshold for the
'critical' level. The second and third steps have `False` values since the number of failing
test units was `0`, which is below the threshold for the 'critical' level.

We can also visually inspect the 'critical' status across all steps by viewing the
validation table:

```{python}
validation
```

We can see that there are filled gray, yellow, and red circles in the first step (far right
side, in the `W`, `E`, and `C` columns) indicating that the 'warning', 'error', and
'critical' thresholds were met. The other steps have empty gray, yellow, and red circles.
This means that thresholds were 'set but not met' in those steps.

If we wanted to check the 'critical' status for a single validation step, we can provide the
step number. Also, we could have the value returned as a scalar by setting `scalar=True`
(ensuring that `i=` is a scalar).

```{python}
validation.critical(i=1)
```

The returned value is `True`, indicating that the first validation step had the 'critical'
threshold met.


## Inspection and Assistance

Functions for getting to grips with a new data table. Use `DataScan` for a quick overview, `preview()` for first/last rows, `col_summary_tbl()` for column summaries, and `missing_vals_tbl()` for missing value analysis.



DataScan(data: 'Any', tbl_name: 'str | None' = None) -> 'None'

Get a summary of a dataset.

Before writing validation rules for a table, it helps to know what's in it: which columns
have missing values, how many distinct values each column has, and what ranges the numbers
and dates fall in. A `DataScan` collects these facts for every column in one pass. You can
look at the result as a report, work with it as a table, or save it to compare against later.

Once a scan is made, it can be used in several ways:

- [`get_tabular_report()`](`pointblank.DataScan.get_tabular_report`) shows the summary as a
table with one row per column (this is also what
[`col_summary_tbl()`](`pointblank.col_summary_tbl`) returns)
- [`summary_data`](`pointblank.DataScan.summary_data`) gives the statistics as a DataFrame, for
filtering, sorting, or building checks on them
- [`to_dict()`](`pointblank.DataScan.to_dict`), [`to_json()`](`pointblank.DataScan.to_json`),
and [`save_to_json()`](`pointblank.DataScan.save_to_json`) store the profile, which
[`from_dict()`](`pointblank.DataScan.from_dict`),
[`from_json()`](`pointblank.DataScan.from_json`), and
[`load_from_json()`](`pointblank.DataScan.load_from_json`) restore without the original data
- [`compare()`](`pointblank.DataScan.compare`) compares two scans and reports schema changes
and drift in each column's statistics, as a [`DataScanDiff`](`pointblank.DataScanDiff`)

The scan stores the profile, not the data. To compare the rows of two tables rather than
their profiles, use [`compare()`](`pointblank.compare`).

:::{.callout-warning}
The `DataScan()` class is still experimental. Please report any issues you encounter in the
[Pointblank issue tracker](https://github.com/posit-dev/pointblank/issues).
:::

Parameters
----------
data
    The data to scan and summarize. This could be a DataFrame object, an Ibis table object,
    a CSV file path, a Parquet file path, a GitHub URL pointing to a CSV or Parquet file,
    or a database connection string.
tbl_name
    Optionally, the name of the table could be provided as `tbl_name`. It's shown in the
    header of the report, stored with the profile, and used to label the two scans in a
    [`DataScanDiff`](`pointblank.DataScanDiff`) report.

Measures of Missingness and Distinctness
----------------------------------------
For each column, the following measures are provided:

- `n_missing`: the number of missing values in the column
- `n_unique`: the number of distinct values in the column (if there are missing values, they
count as one more distinct value)

The report also shows each count as a fraction of the number of rows in the table.

Statistics for Numeric and String Columns
-----------------------------------------
For numeric and string columns, the following statistics are provided. For string columns,
they describe the lengths of the strings (in characters), not the strings themselves, which
makes it easy to spot truncated values or codes of an unexpected length.

- `mean`: the mean
- `std`: the standard deviation
- `min`: the minimum
- `p05`: the 5th percentile
- `q_1`: the first quartile
- `median`: the median
- `q_3`: the third quartile
- `p95`: the 95th percentile
- `max`: the maximum
- `iqr`: the interquartile range (`q_3 - q_1`)

Statistics for Date, Datetime, and Time Columns
-----------------------------------------------
For date, datetime, and time columns, the following statistics are provided:

- `min`: the earliest value in the column
- `max`: the latest value in the column

Statistics for Boolean Columns
------------------------------
For Boolean columns, `freqs` gives the number of `True` and `False` values (as a dictionary,
e.g., `{"True": 9, "False": 4}`). The report shows these as percentages.

Supported Column Types
----------------------
Numeric columns (integers, floats and decimals), string columns (including categorical and
enum columns), date, datetime and time columns, and Boolean columns are summarized. Columns of
other types (e.g., lists, arrays, structs, durations and binary data) can't be summarized, so
they're listed in the report with no statistics, and in the JSON under `skipped_columns`.
pandas stores decimals, times, and Booleans with missing values as `object` columns, which
can't be summarized either.

Returns
-------
DataScan
    A DataScan object.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's scan the `small_table` dataset, which has 13 rows and 8 columns of several types.

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

scan = pb.DataScan(small_table, tbl_name="small_table")

scan.get_tabular_report()
```

The report has one row per column. Column `c` has 2 missing values, `d` spans a wide range
(from about 108 to 9,999.99), and the string column `b` always has 9 characters, which suggests
a fixed-format code. Column `e` is Boolean, so it shows the share of `True` and `False` values
instead of statistics.

The same statistics are available as a DataFrame through `summary_data`, so they can be
filtered like any other table. Here are the columns with missing values:

```{python}
import polars as pl

(
    scan.summary_data
    .filter(pl.col("n_missing") > 0)
    .select("colname", "coltype", "n_missing", "n_unique")
)
```

A scan can also be kept as a baseline. If its profile is saved now with
[`save_to_json()`](`pointblank.DataScan.save_to_json`), a later scan of a new version of the
table can be compared against it with [`compare()`](`pointblank.DataScan.compare`), which
reports added and removed columns, type changes, and changes in each statistic.

See Also
--------
col_summary_tbl : Get the same report directly from a table.
missing_vals_tbl : See where in a table the missing values are.
preview : Show the first and last rows of a table.
DataScanDiff : The result of comparing two scans.

DataScanDiff(current: 'DataScan', baseline: 'DataScan') -> 'None'

The result of comparing two `DataScan` profiles.

A `DataScanDiff` is what [`DataScan.compare()`](`pointblank.DataScan.compare`) returns, and
it isn't meant to be created directly. It records how a table's profile changed between a
baseline scan and a current one: which columns were added, removed, or changed type, which
statistics changed and by how much, and how far each column's distribution drifted (see
[`DataScan.compare()`](`pointblank.DataScan.compare`) for how the drift scores are computed).

The results can be used in three ways. The attributes and the
[`has_changes`](`pointblank.DataScanDiff.has_changes`) property suit checks in code, such as
stopping a pipeline when a column disappears.
[`to_dict()`](`pointblank.DataScanDiff.to_dict`) gives every difference as plain data, for
logging or alerting. And [`get_tabular_report()`](`pointblank.DataScanDiff.get_tabular_report`)
shows the differences as a table, for a person to review.

Parameters
----------
current
    The newer `DataScan` (the one whose `compare()` method was called).
baseline
    The older `DataScan` that `current` is compared against.

Attributes
----------
columns_added
    Column names present in the current scan but not the baseline, sorted alphabetically.
columns_removed
    Column names present in the baseline but not the current scan, sorted alphabetically.
columns_type_changed
    Column names whose data type changed between baseline and current.
column_diffs
    Per-column diff details for all columns that appear in either scan, with the baseline's
    columns first. Each entry has the attributes `colname`, `coltype_baseline` and
    `coltype_current` (`None` for a column missing from that scan), `status` (one of
    `"added"`, `"removed"`, `"type_changed"`, `"changed"`, or `"unchanged"`), `stat_diffs` (a
    dictionary mapping each changed statistic to a `(baseline, current)` tuple), and
    `drift_scores` (a dictionary of drift scores, which might be empty).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

In this example, a table gains a row and a `note` column, and its `score` column changes from
integers to strings.

```{python}
import pointblank as pb
import polars as pl

old = pl.DataFrame({"id": [1, 2, 3], "score": [7, 9, 8]})

new = pl.DataFrame(
    {"id": [1, 2, 3, 4], "score": ["7", "9", "8", "6"], "note": ["a", None, "b", None]}
)

diff = pb.DataScan(new).compare(pb.DataScan(old))

diff.columns_added, diff.columns_removed, diff.columns_type_changed
```

The schema changes are available directly as lists. For a look at each column, go through
`column_diffs`:

```{python}
for col in diff.column_diffs:
    print(col.colname, col.status, col.coltype_baseline, "->", col.coltype_current)
```

The `id` column kept its type, but its statistics changed because of the new row. A
pipeline could stop whenever `columns_removed` or `columns_type_changed` isn't empty, since
those changes tend to break code that reads the table.

See Also
--------
DataScan : Profile a table, column by column.
pointblank.compare : Compare two tables row by row.

preview(data: 'Any', columns_subset: 'str | list[str] | Column | None' = None, n_head: 'int' = 5, n_tail: 'int' = 5, limit: 'int' = 50, show_row_numbers: 'bool' = True, max_col_width: 'int' = 250, min_tbl_width: 'int' = 500, incl_header: 'bool | None' = None) -> 'GT'

Display a table preview that shows some rows from the top, some from the bottom.

To get a quick look at the data in a table, we can use the `preview()` function to display a
preview of the table. The function shows a subset of the rows from the start and end of the
table, with the number of rows from the start and end determined by the `n_head=` and `n_tail=`
parameters (set to `5` by default). This function works with any table that is supported by the
`pointblank` library, including Pandas, Polars, and Ibis backend tables (e.g., DuckDB, MySQL,
PostgreSQL, SQLite, Parquet, etc.).

The view is optimized for readability, with column names and data types displayed in a compact
format. The column widths are sized to fit the column names, dtypes, and column content up to
a configurable maximum width of `max_col_width=` pixels. The table can be scrolled horizontally
to view even very large datasets. Since the output is a Great Tables (`GT`) object, it can be
further customized using the `great_tables` API.

Parameters
----------
data
    The table to preview, which could be a DataFrame object, an Ibis table object, a CSV
    file path, a Parquet file path, or a database connection string. When providing a CSV or
    Parquet file path (as a string or `pathlib.Path` object), the file will be automatically
    loaded using an available DataFrame library (Polars or Pandas). Parquet input also supports
    glob patterns, directories containing .parquet files, and Spark-style partitioned datasets.
    Connection strings enable direct database access via Ibis with optional table specification
    using the `::table_name` suffix. Read the *Supported Input Table Types* section for details
    on the supported table types.
columns_subset
    The columns to display in the table, by default `None` (all columns are shown). This can
    be a string, a list of strings, a `Column` object, or a `ColumnSelector` object. The latter
    two options allow for more flexible column selection using column selector functions. Errors
    are raised if the column names provided don't match any columns in the table (when provided
    as a string or list of strings) or if column selector expressions don't resolve to any
    columns.
n_head
    The number of rows to show from the start of the table. Set to `5` by default.
n_tail
    The number of rows to show from the end of the table. Set to `5` by default.
limit
    The limit value for the sum of `n_head=` and `n_tail=` (the total number of rows shown).
    If the sum of `n_head=` and `n_tail=` exceeds the limit, an error is raised. The default
    value is `50`.
show_row_numbers
    Should row numbers be shown? The numbers shown reflect the row numbers of the head and tail
    in the input `data=` table. By default, this is set to `True`.
max_col_width
    The maximum width of the columns (in pixels) before the text is truncated. The default value
    is `250` (`"250px"`).
min_tbl_width
    The minimum width of the table in pixels. If the sum of the column widths is less than this
    value, the all columns are sized up to reach this minimum width value. The default value is
    `500` (`"500px"`).
incl_header
    Should the table include a header with the table type and table dimensions? Set to `True` by
    default.

Returns
-------
GT
    A GT object that displays the preview of the table.

Supported Input Table Types
---------------------------
The `data=` parameter can be given any of the following table types:

- Polars DataFrame (`"polars"`)
- Pandas DataFrame (`"pandas"`)
- PySpark table (`"pyspark"`)
- DuckDB table (`"duckdb"`)*
- MySQL table (`"mysql"`)*
- PostgreSQL table (`"postgresql"`)*
- SQLite table (`"sqlite"`)*
- Microsoft SQL Server table (`"mssql"`)*
- Snowflake table (`"snowflake"`)*
- Databricks table (`"databricks"`)*
- BigQuery table (`"bigquery"`)*
- Parquet table (`"parquet"`)*
- CSV files (string path or `pathlib.Path` object with `.csv` extension)
- Parquet files (string path, `pathlib.Path` object, glob pattern, directory with `.parquet`
extension, or partitioned dataset)
- Database connection strings (URI format with optional table specification)

The table types marked with an asterisk need to be prepared as Ibis tables (with type of
`ibis.expr.types.relations.Table`). Furthermore, using `preview()` with these types of tables
requires the Ibis library (`v9.5.0` or above) to be installed. If the input table is a Polars or
Pandas DataFrame, the availability of Ibis is not needed.

To use a CSV file, ensure that a string or `pathlib.Path` object with a `.csv` extension is
provided. The file will be automatically detected and loaded using the best available DataFrame
library. The loading preference is Polars first, then Pandas as a fallback.

Connection strings follow database URL formats and must also specify a table using the
`::table_name` suffix. Examples include:

```
"duckdb:///path/to/database.ddb::table_name"
"sqlite:///path/to/database.db::table_name"
"postgresql://user:password@localhost:5432/database::table_name"
"mysql://user:password@localhost:3306/database::table_name"
"bigquery://project/dataset::table_name"
"snowflake://user:password@account/database/schema::table_name"
```

When using connection strings, the Ibis library with the appropriate backend driver is required.

Examples
--------
It's easy to preview a table using the `preview()` function. Here's an example using the
`small_table` dataset (itself loaded using the [`load_dataset()`](`pointblank.load_dataset`)
function):

```{python}
import pointblank as pb

small_table_polars = pb.load_dataset("small_table")

pb.preview(small_table_polars)
```

This table is a Polars DataFrame, but the `preview()` function works with any table supported
by `pointblank`, including Pandas DataFrames and Ibis backend tables. Here's an example using
a DuckDB table handled by Ibis:

```{python}
small_table_duckdb = pb.load_dataset("small_table", tbl_type="duckdb")

pb.preview(small_table_duckdb)
```

The blue dividing line marks the end of the first `n_head=` rows and the start of the last
`n_tail=` rows.

We can adjust the number of rows shown from the start and end of the table by setting the
`n_head=` and `n_tail=` parameters. Let's enlarge each of these to `10`:

```{python}
pb.preview(small_table_polars, n_head=10, n_tail=10)
```

In the above case, the entire dataset is shown since the sum of `n_head=` and `n_tail=` is
greater than the number of rows in the table (which is 13).

The `columns_subset=` parameter can be used to show only specific columns in the table. You can
provide a list of column names to make the selection. Let's try that with the `"game_revenue"`
dataset as a Pandas DataFrame:

```{python}
game_revenue_pandas = pb.load_dataset("game_revenue", tbl_type="pandas")

pb.preview(game_revenue_pandas, columns_subset=["player_id", "item_name", "item_revenue"])
```

Alternatively, we can use column selector functions like
[`starts_with()`](`pointblank.starts_with`) and [`matches()`](`pointblank.matches`) to select
columns based on text or patterns:

```{python}
pb.preview(game_revenue_pandas, n_head=2, n_tail=2, columns_subset=pb.starts_with("session"))
```

Multiple column selector functions can be combined within [`col()`](`pointblank.col`) using
operators like `|` and `&`:

```{python}
pb.preview(
  game_revenue_pandas,
  n_head=2,
  n_tail=2,
  columns_subset=pb.col(pb.starts_with("item") | pb.matches("player"))
)
```

### Working with CSV Files

The `preview()` function can directly accept CSV file paths, making it easy to preview data
stored in CSV files without manual loading:

```{python}
# Get a path to a CSV file from the package data
csv_path = pb.get_data_path("global_sales", "csv")

pb.preview(csv_path)
```

You can also use a Path object to specify the CSV file:

```{python}
from pathlib import Path

csv_file = Path(pb.get_data_path("game_revenue", "csv"))

pb.preview(csv_file, n_head=3, n_tail=3)
```

### Working with Parquet Files

The `preview()` function can directly accept Parquet files and datasets in various formats:

```{python}
# Single Parquet file from package data
parquet_path = pb.get_data_path("nycflights", "parquet")

pb.preview(parquet_path)
```

You can also use glob patterns and directories:

```python
# Multiple Parquet files with glob patterns
pb.preview("data/sales_*.parquet")

# Directory containing Parquet files
pb.preview("parquet_data/")

# Partitioned Parquet dataset
pb.preview("sales_data/")  # Auto-discovers partition columns
```

### Working with Database Connection Strings

The `preview()` function supports database connection strings for direct preview of database
tables. Connection strings must specify a table using the `::table_name` suffix:

```{python}
# Get path to a DuckDB database file from package data
duckdb_path = pb.get_data_path("game_revenue", "duckdb")

pb.preview(f"duckdb:///{duckdb_path}::game_revenue")
```

For comprehensive documentation on supported connection string formats, error handling, and
installation requirements, see the [`connect_to_table()`](`pointblank.connect_to_table`)
function.

col_summary_tbl(data: 'Any', tbl_name: 'str | None' = None) -> 'GT'

Generate a column-level summary table of a dataset.

The `col_summary_tbl()` function generates a summary table of a dataset, focusing on providing
column-level information about the dataset. The summary includes the following information:

- the type of the table (e.g., `"polars"`, `"pandas"`, etc.)
- the number of rows and columns in the table
- column-level information, including:
    - the column name
    - the column type
    - measures of missingness and distinctness
    - descriptive stats and quantiles
    - statistics for datetime columns

The summary table is returned as a GT object, which can be displayed in a notebook or saved to
an HTML file.

:::{.callout-warning}
The `col_summary_tbl()` function is still experimental. Please report any issues you encounter
in the [Pointblank issue tracker](https://github.com/posit-dev/pointblank/issues).
:::

Parameters
----------
data
    The table to summarize, which could be a DataFrame object, an Ibis table object, a CSV
    file path, a Parquet file path, or a database connection string. Read the *Supported Input
    Table Types* section for details on the supported table types.
tbl_name
    Optionally, the name of the table could be provided as `tbl_name=`.

Returns
-------
GT
    A GT object that displays the column-level summaries of the table.

Supported Column Types
----------------------
Numeric columns (integers, floats and decimals), string columns (including categorical and
enum columns), date, datetime and time columns, and Boolean columns are summarized. Columns of
other types (e.g., lists, arrays, structs, durations and binary data) can't be summarized, so
they get a row marked *unsupported type* with no statistics. pandas stores decimals, times,
and Booleans with missing values as `object` columns, which can't be summarized either.

Supported Input Table Types
---------------------------
The `data=` parameter can be given any of the following table types:

- Polars DataFrame (`"polars"`)
- Pandas DataFrame (`"pandas"`)
- DuckDB table (`"duckdb"`)*
- MySQL table (`"mysql"`)*
- PostgreSQL table (`"postgresql"`)*
- SQLite table (`"sqlite"`)*
- Parquet table (`"parquet"`)*
- CSV files (string path or `pathlib.Path` object with `.csv` extension)
- Parquet files (string path, `pathlib.Path` object, glob pattern, directory with `.parquet`
extension, or partitioned dataset)
- GitHub URLs (direct links to CSV or Parquet files on GitHub)
- Database connection strings (URI format with optional table specification)

The table types marked with an asterisk need to be prepared as Ibis tables (with type of
`ibis.expr.types.relations.Table`). Furthermore, using `col_summary_tbl()` with these types of
tables requires the Ibis library (`v9.5.0` or above) to be installed. If the input table is a
Polars or Pandas DataFrame, the availability of Ibis is not needed.

Examples
--------
It's easy to get a column-level summary of a table using the `col_summary_tbl()` function.
Here's an example using the `small_table` dataset (itself loaded using the
[`load_dataset()`](`pointblank.load_dataset`) function):

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

pb.col_summary_tbl(data=small_table)
```

This table used above was a Polars DataFrame, but the `col_summary_tbl()` function works with
any table supported by `pointblank`, including Pandas DataFrames and Ibis backend tables.
Here's an example using a DuckDB table handled by Ibis:

```{python}
nycflights = pb.load_dataset(dataset="nycflights", tbl_type="duckdb")

pb.col_summary_tbl(data=nycflights, tbl_name="nycflights")
```

The `tbl_name=` value appears in the report's title, which helps tell reports apart when
several tables are summarized. An Ibis table is read into memory before its statistics are
computed, so a very large table might be better summarized from a sample. To keep the
summary rather than just display it (to query it, save it, or compare it with a later version
of the table), create a [`DataScan`](`pointblank.DataScan`) instead, which produces this same
report through its [`get_tabular_report()`](`pointblank.DataScan.get_tabular_report`) method.

missing_vals_tbl(data: 'Any', missing: 'dict[str, MissingSpec] | None' = None, as_heatmap: 'bool' = False) -> 'GT'

Display a table that shows the missing values in the input table.

The `missing_vals_tbl()` function generates a table that shows the missing values in the input
table. The table is displayed using the Great Tables API, which allows for further customization
of the table's appearance if so desired.

By default, missingness is treated as binary (a value is either Null or it isn't) and the
function renders a sector-based heatmap of the proportion of Null values across the rows of each
column. When a `missing=` mapping of columns to [`MissingSpec`](`pointblank.MissingSpec`) objects
is supplied, the function instead renders a *structured missingness* breakdown: one row per
column with the count and percentage of complete values and of each missing *reason* (e.g.,
`refused`, `not_asked`). Declared (coded) reasons are grouped under a "Missing Reasons" spanner
and keep their raw input form as labels; actual `Null`/`None`/`NA` values (which are not part of
the spec) are tallied in a fixed "Null" column at the far right (styled like "Complete"), so
they aren't mistaken for declared reasons.

Note that supplying `missing=` produces a *different report* than the default view: it is a
distinct visualization (a per-reason breakdown table, or a per-reason heatmap with
`as_heatmap=True`), not an annotated version of the default sector heatmap. The report titles
differ accordingly ("Missing Values" for the default, "Missing Values by Reason" or "Missing
Pattern Heatmap" for the structured views), and the shared header/title styling makes the family
resemblance clear.

Parameters
----------
data
    The table for which to display the missing values. This could be a DataFrame object, an
    Ibis table object, a CSV file path, a Parquet file path, or a database connection string.
    Read the *Supported Input Table Types* section for details on the supported table types.
missing
    An optional dictionary mapping column names to [`MissingSpec`](`pointblank.MissingSpec`)
    objects. When provided, the function renders a structured breakdown of missingness by
    reason for the specified columns (rather than the default sector heatmap). The reason
    columns are the union of reasons across the supplied specs; a reason that isn't defined for
    a given column is shown as an em dash (not applicable), as distinct from a defined-but-unobserved
    reason (shown as `0 (0%)`).
as_heatmap
    Only applies when `missing=` is provided. When `True`, render the per-reason proportions as
    a color-coded heatmap (cells shaded from light to dark by the proportion missing) instead of
    the count/percentage text breakdown. Default is `False`.

Returns
-------
GT
    A GT object that displays the table of missing values in the input table.

Supported Input Table Types
---------------------------
The `data=` parameter can be given any of the following table types:

- Polars DataFrame (`"polars"`)
- Pandas DataFrame (`"pandas"`)
- PySpark table (`"pyspark"`)
- DuckDB table (`"duckdb"`)*
- MySQL table (`"mysql"`)*
- PostgreSQL table (`"postgresql"`)*
- SQLite table (`"sqlite"`)*
- Microsoft SQL Server table (`"mssql"`)*
- Snowflake table (`"snowflake"`)*
- Databricks table (`"databricks"`)*
- BigQuery table (`"bigquery"`)*
- Parquet table (`"parquet"`)*
- CSV files (string path or `pathlib.Path` object with `.csv` extension)
- Parquet files (string path, `pathlib.Path` object, glob pattern, directory with `.parquet`
extension, or partitioned dataset)
- Database connection strings (URI format with optional table specification)

The table types marked with an asterisk need to be prepared as Ibis tables (with type of
`ibis.expr.types.relations.Table`). Furthermore, using `missing_vals_tbl()` with these types of
tables requires the Ibis library (`v9.5.0` or above) to be installed. If the input table is a
Polars or Pandas DataFrame, the availability of Ibis is not needed.

The Missing Values Table
------------------------
The missing values table shows the proportion of missing values in each column of the input
table. The table is divided into sectors, with each sector representing a range of rows in the
table. The proportion of missing values in each sector is calculated for each column. The table
is displayed using the Great Tables API, which allows for further customization of the table's
appearance.

To ensure that the table can scale to tables with many columns, each row in the reporting table
represents a column in the input table. There are 10 sectors shown in the table, where the first
sector represents the first 10% of the rows, the second sector represents the next 10% of the
rows, and so on. Any sectors that are light blue indicate that there are no missing values in
that sector. If there are missing values, the proportion of missing values is shown by a gray
color (light gray for low proportions, dark gray to black for very high proportions).

Examples
--------
The `missing_vals_tbl()` function is useful for quickly identifying columns with missing values
in a table. Here's an example using the `nycflights` dataset (loaded as a Polars DataFrame using
the [`load_dataset()`](`pointblank.load_dataset`) function):

```{python}
import pointblank as pb

nycflights = pb.load_dataset("nycflights", tbl_type="polars")

pb.missing_vals_tbl(nycflights)
```

The table shows the proportion of missing values in each column of the `nycflights` dataset. The
table is divided into sectors, with each sector representing a range of rows in the table (with
around 34,000 rows per sector). The proportion of missing values in each sector is calculated
for each column. The various shades of gray indicate the proportion of missing values in each
sector. Many columns have no missing values at all, and those sectors are colored light blue.

load_dataset(dataset: "Literal['small_table', 'game_revenue', 'nycflights', 'global_sales']" = 'small_table', tbl_type: "Literal['polars', 'pandas', 'duckdb']" = 'polars') -> 'Any'

Load a dataset hosted in the library as specified table type.

The Pointblank library includes several datasets that can be loaded using the `load_dataset()`
function. The datasets can be loaded as a Polars DataFrame, a Pandas DataFrame, or as a DuckDB
table (which uses the Ibis library backend). These datasets are used throughout the
documentation's examples to demonstrate the functionality of the library. They're also useful
for experimenting with the library and trying out different validation scenarios.

Parameters
----------
dataset
    The name of the dataset to load. Current options are `"small_table"`, `"game_revenue"`,
    `"nycflights"`, and `"global_sales"`.
tbl_type
    The type of table to generate from the dataset. The named options are `"polars"`,
    `"pandas"`, and `"duckdb"`.

Returns
-------
Any
    The dataset for the `Validate` object. This could be a Polars DataFrame, a Pandas DataFrame,
    or a DuckDB table as an Ibis table.

Included Datasets
-----------------
There are three included datasets that can be loaded using the `load_dataset()` function:

- `"small_table"`: A small dataset with 13 rows and 8 columns. This dataset is useful for
testing and demonstration purposes.
- `"game_revenue"`: A dataset with 2000 rows and 11 columns. Provides revenue data for a game
development company. For the particular game, there are records of player sessions, the items
they purchased, ads viewed, and the revenue generated.
- `"nycflights"`: A dataset with 336,776 rows and 18 columns. This dataset provides information
about flights departing from New York City airports (JFK, LGA, or EWR) in 2013.
- `"global_sales"`: A dataset with 50,000 rows and 20 columns. Provides information about
global sales of products across different regions and countries.

Supported DataFrame Types
-------------------------
The `tbl_type=` parameter can be set to one of the following:

- `"polars"`: A Polars DataFrame.
- `"pandas"`: A Pandas DataFrame.
- `"duckdb"`: An Ibis table for a DuckDB database.

Examples
--------
Load the `"small_table"` dataset as a Polars DataFrame by calling `load_dataset()` with
`dataset="small_table"` and `tbl_type="polars"`:

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

pb.preview(small_table)
```

Note that the `"small_table"` dataset is a Polars DataFrame and using the
[`preview()`](`pointblank.preview`) function will display the table in an HTML viewing
environment.

The `"game_revenue"` dataset can be loaded as a Pandas DataFrame by specifying the dataset name
and setting `tbl_type="pandas"`:

```{python}
game_revenue = pb.load_dataset(dataset="game_revenue", tbl_type="pandas")

pb.preview(game_revenue)
```

The `"game_revenue"` dataset is a more real-world dataset with a mix of data types, and it's
significantly larger than the `small_table` dataset at 2000 rows and 11 columns.

The `"nycflights"` dataset can be loaded as a DuckDB table by specifying the dataset name and
setting `tbl_type="duckdb"`:

```{python}
nycflights = pb.load_dataset(dataset="nycflights", tbl_type="duckdb")

pb.preview(nycflights)
```

The `"nycflights"` dataset is a large dataset with 336,776 rows and 18 columns. This dataset is
truly a real-world dataset and provides information about flights originating from New York City
airports in 2013.

Finally, the `"global_sales"` dataset can be loaded as a Polars table by specifying the dataset
name. Since `tbl_type=` is set to `"polars"` by default, we don't need to specify it:

```{python}
global_sales = pb.load_dataset(dataset="global_sales")

pb.preview(global_sales)
```

The `"global_sales"` dataset is a large dataset with 50,000 rows and 20 columns. Each record
describes the sales of a particular product to a customer located in one of three global
regions: North America, Europe, or Asia.

get_data_path(dataset: "Literal['small_table', 'game_revenue', 'nycflights', 'global_sales']" = 'small_table', file_type: "Literal['csv', 'parquet', 'duckdb']" = 'csv') -> 'str'

Get the file path to a dataset included with the Pointblank package.

This function provides direct access to the file paths of datasets included with Pointblank.
These paths can be used in examples and documentation to demonstrate file-based data loading
without requiring the actual data files. The returned paths can be used with
`Validate(data=path)` to demonstrate CSV and Parquet file loading capabilities.

The first call for a dataset and file type writes the dataset to a temporary file and returns
its path. Later calls with the same arguments return the same path, unless the file has been
changed or deleted since, in which case a fresh copy is written. All of these files are deleted
when the Python session ends, so copy a file elsewhere if you need to keep it.

Parameters
----------
dataset
    The name of the dataset to get the path for. Current options are `"small_table"`,
    `"game_revenue"`, `"nycflights"`, and `"global_sales"`.
file_type
    The file format to get the path for. Options are `"csv"`, `"parquet"`, or `"duckdb"`.

Returns
-------
str
    The path to a temporary file holding the requested dataset in the requested format. The
    file is deleted when the Python session ends.

Included Datasets
-----------------
The available datasets are the same as those in [`load_dataset()`](`pointblank.load_dataset`):

- `"small_table"`: A small dataset with 13 rows and 8 columns. Ideal for testing and examples.
- `"game_revenue"`: A dataset with 2000 rows and 11 columns. Revenue data for a game company.
- `"nycflights"`: A dataset with 336,776 rows and 18 columns. Flight data from NYC airports.
- `"global_sales"`: A dataset with 50,000 rows and 20 columns. Global sales data across regions.

File Types
----------
Each dataset is available in multiple formats:

- `"csv"`: Comma-separated values file (`.csv`)
- `"parquet"`: Parquet file (`.parquet`)
- `"duckdb"`: DuckDB database file (`.ddb`)

Examples
--------
Get the path to a CSV file and use it with `Validate`:

```{python}
import pointblank as pb

# Get path to the small_table CSV file
csv_path = pb.get_data_path("small_table", "csv")

# Use the path directly with Validate
validation = (
    pb.Validate(data=csv_path)
    .col_exists(["a", "b", "c"])
    .col_vals_gt(columns="d", value=0)
    .interrogate()
)

validation
```

The CSV file is read into a table during interrogation. The three `col_exists()` steps and the
`col_vals_gt()` step all pass.

We can also get a Parquet file path for validation examples:

```{python}
# Get path to the game_revenue Parquet file
parquet_path = pb.get_data_path(dataset="game_revenue", file_type="parquet")

# Validate the Parquet file directly
validation = (
    pb.Validate(data=parquet_path, label="Game Revenue Data Validation")
    .col_vals_not_null(columns=["player_id", "session_id"])
    .col_vals_gt(columns="item_revenue", value=0)
    .interrogate()
)

validation
```

All 2,000 rows have player and session IDs and positive revenue values, so every step passes.

This is particularly useful for documentation examples where you want to demonstrate
file-based workflows without requiring users to have specific data files:

```{python}
# Example showing CSV file validation
sales_csv = pb.get_data_path(dataset="global_sales", file_type="csv")

validation = (
    pb.Validate(data=sales_csv, label="Sales Data Validation")
    .col_exists(["customer_id", "product_id", "revenue"])
    .col_vals_regex(columns="customer_id", pattern=r"[0-9a-f]{8}")
    .interrogate()
)

validation
```

All three columns exist, and every customer ID that's present is an eight-character
hexadecimal string. The 334 failing test units in the last step are rows where `customer_id`
is missing; setting `na_pass=True` would count those rows as passing.

See Also
--------
load_dataset : Load a dataset directly as a table object.

connect_to_table(connection_string: 'str') -> 'Any'

Connect to a database table using a connection string.

This utility function tests whether a connection string leads to a valid table and returns
the table object if successful. It provides helpful error messages when no table is specified
or when backend dependencies are missing.

Parameters
----------
connection_string
    A database connection string with a required table specification using the `::table_name`
    suffix. Supported formats are outlined in the *Supported Connection String Formats* section.

Returns
-------
Any
    An Ibis table object for the specified database table.

Supported Connection String Formats
-----------------------------------
The `connection_string` parameter must include a valid connection string with a table name
specified using the `::` syntax. Here are some examples on how to format connection strings
for various backends:

```
DuckDB:     "duckdb:///path/to/database.ddb::table_name"
SQLite:     "sqlite:///path/to/database.db::table_name"
PostgreSQL: "postgresql://user:password@localhost:5432/database::table_name"
MySQL:      "mysql://user:password@localhost:3306/database::table_name"
BigQuery:   "bigquery://project/dataset::table_name"
Snowflake:  "snowflake://user:password@account/database/schema::table_name"
```

If the connection string does not include a table name, the function will attempt to connect to
the database and list available tables, providing guidance on how to specify a table.

Examples
--------
Connect to a DuckDB table:

```{python}
import pointblank as pb

# Get path to a DuckDB database file from package data
duckdb_path = pb.get_data_path("game_revenue", "duckdb")

# Connect to the `game_revenue` table in the DuckDB database
game_revenue = pb.connect_to_table(f"duckdb:///{duckdb_path}::game_revenue")

# Use with the `preview()` function
pb.preview(game_revenue)
```

The result is an Ibis table that stays in the database, and `preview()` shows the first and
last five of its 2,000 rows. The same table object can be passed to `Validate(data=)`, and
the validation then runs inside DuckDB.

Here are some backend-specific connection examples:

```python
# PostgreSQL
pg_table = pb.connect_to_table(
    "postgresql://user:password@localhost:5432/warehouse::customer_data"
)

# SQLite
sqlite_table = pb.connect_to_table("sqlite:///local_data.db::products")

# BigQuery
bq_table = pb.connect_to_table("bigquery://my-project/analytics::daily_metrics")
```

This function requires the Ibis library with appropriate backend drivers:

```bash
# You can install a set of common backends:
pip install 'ibis-framework[duckdb,postgres,mysql,sqlite]'

# ...or specific backends as needed:
pip install 'ibis-framework[duckdb]'    # for DuckDB
pip install 'ibis-framework[postgres]'  # for PostgreSQL
```

The connection string's backend name (e.g., `postgresql://`) determines which Ibis backend is
needed. If it isn't installed, `connect_to_table()` raises a `ConnectionError` that shows the
`pip install` command for it.
See Also
--------
print_database_tables : List all available tables in a database for discovery

print_database_tables(connection_string: 'str') -> 'list[str]'

List all tables in a database from a connection string.

The `print_database_tables()` function connects to a database and returns a list of all
available tables. This is particularly useful for discovering what tables exist in a database
before connecting to a specific table with `connect_to_table(). The function automatically
filters out temporary Ibis tables (memtables) to show only user tables. It supports all database
backends available through Ibis, including DuckDB, SQLite, PostgreSQL, MySQL, BigQuery, and
Snowflake.

Parameters
----------
connection_string
    A database connection string *without* the `::table_name` suffix. Example:
    `"duckdb:///path/to/database.ddb"`.

Returns
-------
list[str]
    List of table names, excluding temporary Ibis tables.

See Also
--------
connect_to_table : Connect to a database table with full connection string documentation


## DataScan Methods

Methods for the DataScan class


get_tabular_report(self, *, show_sample_data: 'bool' = False) -> 'GT'

Get a tabular report of the scan.

The report presents the profile of each column as one row of a table: the column's name
and type, the number and fraction of missing and distinct values, and its statistics. It's
the quickest way to get an overview of a table you haven't seen before, and it's what
[`col_summary_tbl()`](`pointblank.col_summary_tbl`) returns.

Each row starts with an icon for the column's type. Statistics that don't apply to any
column are left out, and those that don't apply to a particular column are shown as `-`.
Statistics for string columns describe the lengths of the strings (as the note below the
table says). Boolean columns show the percentages of `True` and `False` values in place of
the number of distinct values. Columns of unsupported types are listed with an
*unsupported type* badge and no statistics.

Parameters
----------
show_sample_data
    Should a column with the first five non-missing values of each column be added? This
    is off by default, since the values can be long, and the table may hold sensitive data.

Returns
-------
GT
    A Great Tables object.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's scan the `game_revenue` dataset and look at the report.

```{python}
import pointblank as pb

game_revenue = pb.load_dataset(dataset="game_revenue", tbl_type="polars")

scan = pb.DataScan(game_revenue, tbl_name="game_revenue")

scan.get_tabular_report()
```

The header shows the table type and its dimensions. In the rows, `item_revenue` has a
long right tail (its 95th percentile is far below its maximum), and the string columns
`player_id` and `session_id` have fixed lengths. To see what the values look like, add
the sample data:

```{python}
scan.get_tabular_report(show_sample_data=True)
```

The samples make it clear that `player_id` and `session_id` are codes, and they show the
format of each column's values. The report is a Great Tables object, so it can be saved
to a standalone HTML file with its `write_raw_html()` method.

DataScan.summary_data

The column statistics as a DataFrame.

The report from [`get_tabular_report()`](`pointblank.DataScan.get_tabular_report`) is for
reading, while `summary_data` holds the same statistics as data. That's useful for finding
columns that need attention (the ones with missing values, or with a single distinct
value), for building validation steps from what was found, or for passing the profile to
another tool.

The statistics are described in the [`DataScan`](`pointblank.DataScan`) documentation. A
statistic that doesn't apply to a column's type is null (for example, `mean` for a date
column). The order of the statistic columns isn't fixed, so select columns by name.

Returns
-------
Any
    A DataFrame of the same type as the scanned table (Polars for a scan restored with
    [`from_dict()`](`pointblank.DataScan.from_dict`) or a related method), with one row per
    summarized column. The `colname` and `coltype` columns give each column's name and
    data type, `sample_data` holds the first five non-missing values (as one string), and
    `icon` is the type icon used in the report. The other columns are the statistics.
    When a statistic holds values of different types across columns (e.g., `min` for a
    number and a date), its values are converted to strings. Columns of unsupported types
    aren't included.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Here's the summary for the numeric columns of the `small_table` dataset.

```{python}
import pointblank as pb
import polars as pl

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

summary = pb.DataScan(small_table).summary_data

(
    summary
    .filter(pl.col("coltype").is_in(["Int64", "Float64"]))
    .select("colname", "n_missing", "n_unique", "mean", "min", "max")
)
```

Column `c` is the only one with missing values. The `min` and `max` columns are strings,
because they also hold the dates from the date columns.

Since this is an ordinary DataFrame, the profile can lead to further checks. This finds
the columns with one distinct value fewer than the table has rows:

```{python}
summary.filter(pl.col("n_unique") == small_table.height - 1).get_column("colname").to_list()
```

Three columns that look like they should identify each row (a timestamp, a code, and an
amount) each have exactly one repeated value. That's a sign of a duplicated row, which
[`Validate.rows_distinct()`](`pointblank.Validate.rows_distinct`) would confirm.

compare(self, baseline: 'DataScan') -> 'DataScanDiff'

Compare this scan against a baseline and return the differences.

Tables that are refreshed regularly (a monthly extract, a nightly load, a model's input
data) can change in ways that no single row reveals: a column is dropped, a type changes,
or the values shift to a different range. Comparing a scan of the current table with a
scan of an earlier version that's known to be good finds those changes. The baseline is
often a profile saved earlier with [`save_to_json()`](`pointblank.DataScan.save_to_json`)
and brought back with [`load_from_json()`](`pointblank.DataScan.load_from_json`).

The scan this method is called on is the *current* one, and `baseline` is the one it's
compared against. The result is a [`DataScanDiff`](`pointblank.DataScanDiff`), which has
the differences as attributes, as a dictionary through
[`to_dict()`](`pointblank.DataScanDiff.to_dict`), and as a report through
[`get_tabular_report()`](`pointblank.DataScanDiff.get_tabular_report`).

This compares profiles, not rows. To find out which rows of two tables differ, use
[`compare()`](`pointblank.compare`) instead.

What Is Compared
----------------
Columns are matched by name. A column that's only in the current scan is *added*, one
that's only in the baseline is *removed*, and one whose data type differs is
*type-changed*. For every column in both scans, each statistic is compared for exact
equality, so any difference at all (even in the last decimal place of a mean) is
reported along with both values. When the type has changed, the statistics might no
longer measure the same thing (a numeric column's mean is of its values, but a string
column's mean is of its lengths).

Drift Scores
------------
Statistics show that something changed but not how much the distribution of values moved.
For that, drift scores are computed for columns whose type hasn't changed:

- numeric columns get the population stability index (`psi`), with the values divided into
ten bins at the baseline's deciles (at least ten values are needed on each side), and the
Kolmogorov–Smirnov statistic (`ks_statistic`, the largest gap between the two cumulative
distributions) with its p-value (`ks_p_value`). The p-value comes from SciPy if it's
installed and from an approximation otherwise
- all other columns get a `psi` computed from how often each distinct value appears

A common rule of thumb reads a PSI below 0.1 as little or no shift, from 0.1 to 0.25 as a
moderate shift, and above 0.25 as a large one. A score of zero is left out.

Drift scores need the values themselves, so they're only computed when both scans were
made from data. A scan restored with [`from_dict()`](`pointblank.DataScan.from_dict`),
[`from_json()`](`pointblank.DataScan.from_json`), or
[`load_from_json()`](`pointblank.DataScan.load_from_json`) only has statistics, and then
only Boolean columns (whose value counts are part of the profile) get a drift score.

Parameters
----------
baseline
    The baseline `DataScan` to compare against (typically the older scan).

Returns
-------
DataScanDiff
    An object describing the differences between the two scans.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's compare two monthly extracts of an orders table. Between January and February, the
`gift` column was dropped, a `coupon` column was added, a new `"app"` channel appeared, and
order amounts went up.

```{python}
import pointblank as pb
import polars as pl

jan = pl.DataFrame(
    {
        "order_id": list(range(1, 21)),
        "amount": [20.0 + i for i in range(20)],
        "channel": ["web"] * 14 + ["store"] * 6,
        "gift": [i % 4 == 0 for i in range(20)],
    }
)

feb = pl.DataFrame(
    {
        "order_id": list(range(21, 45)),
        "amount": [35.0 + 1.5 * i for i in range(24)],
        "channel": ["web"] * 10 + ["store"] * 8 + ["app"] * 6,
        "coupon": ["SPRING"] * 5 + [None] * 19,
    }
)

scan_jan = pb.DataScan(jan, tbl_name="orders_jan")
scan_feb = pb.DataScan(feb, tbl_name="orders_feb")

diff = scan_feb.compare(scan_jan)

diff
```

The printed summary counts one added column, one removed column, and three columns whose
statistics changed. The report shows the details:

```{python}
diff.get_tabular_report()
```

The PSI of `amount` (about 6.7) and its KS statistic (0.875) are far beyond the rule of
thumb, so the amounts really did move. The `channel` column has a PSI of about 2.1 because
of the new `"app"` value. The shift in `order_id` is expected, since each month has its
own IDs, which shows that drift scores still need to be read with the meaning of each
column in mind.

See Also
--------
DataScanDiff : The result of comparing two scans.
pointblank.compare : Compare two tables row by row.

save_to_json(self, output_file: 'str') -> 'None'

Save the profile to a JSON file.

Saving a scan to a file is the simplest way to keep a baseline: scan a table once, save
the profile, and load it later with
[`load_from_json()`](`pointblank.DataScan.load_from_json`) to compare a new scan against
it with [`compare()`](`pointblank.DataScan.compare`). The file holds the output of
[`to_json()`](`pointblank.DataScan.to_json`), so it's small (it stores the statistics, not
the data) and can be kept in version control next to the code that produces the table.

Parameters
----------
output_file
    The path to the output JSON file. An existing file is overwritten.

Returns
-------
None
    Nothing is returned. The profile is written to `output_file`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's save a profile of the `small_table` dataset to a file in a temporary directory.

```{python}
import pointblank as pb
import tempfile
from pathlib import Path

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

profile_path = Path(tempfile.mkdtemp()) / "small_table_profile.json"

pb.DataScan(small_table, tbl_name="small_table").save_to_json(profile_path)

print(profile_path.read_text()[:180])
```

The file starts with the table's metadata, and the column profiles follow. Loading it with
[`load_from_json()`](`pointblank.DataScan.load_from_json`) gives back the same profile,
without needing the table itself.

load_from_json(input_file: 'str') -> 'DataScan'

Load a `DataScan` from a JSON file produced by `save_to_json()`.

This is how a saved baseline is brought back: load the profile from the file, then pass it
to the [`compare()`](`pointblank.DataScan.compare`) method of a new scan to find out what
has changed since. The file is the one written by
[`save_to_json()`](`pointblank.DataScan.save_to_json`) (or any file containing the output
of [`to_json()`](`pointblank.DataScan.to_json`)).

As with [`from_dict()`](`pointblank.DataScan.from_dict`), the loaded scan has the
statistics but not the data, so drift scores are only computed for its Boolean columns.

Parameters
----------
input_file
    The path to the JSON file.

Returns
-------
DataScan
    A restored `DataScan` instance.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's save a profile of a table of orders as a baseline.

```{python}
import pointblank as pb
import polars as pl
import tempfile
from pathlib import Path

orders = pl.DataFrame(
    {"order_id": [1, 2, 3, 4], "amount": [25.0, 40.0, 15.5, 60.0]}
)

baseline_path = Path(tempfile.mkdtemp()) / "orders_profile.json"

pb.DataScan(orders, tbl_name="orders").save_to_json(baseline_path)
```

Later, the table has grown and a refund (a negative amount) has appeared. The baseline is
loaded from the file and compared with a scan of the current table:

```{python}
orders_now = pl.DataFrame(
    {"order_id": [1, 2, 3, 4, 5], "amount": [25.0, 40.0, 15.5, 60.0, -15.5]}
)

baseline = pb.DataScan.load_from_json(baseline_path)

diff = pb.DataScan(orders_now, tbl_name="orders").compare(baseline)

diff.to_dict()["stat_diffs"]["amount"]["min"]
```

The minimum `amount` went from 15.5 to -15.5. A change like this is easy to miss in the
data but stands out in a comparison of profiles, and a check such as
[`Validate.col_vals_ge()`](`pointblank.Validate.col_vals_ge`) can then guard against it.

to_json(self) -> 'str'

Export the profile as a JSON string.

JSON is the format to use when a profile is stored outside of Python or sent to another
system: in a database column, an object store, or an API response. The structure is the
same as that of [`to_dict()`](`pointblank.DataScan.to_dict`), and
[`from_json()`](`pointblank.DataScan.from_json`) restores a `DataScan` from it. Values that
JSON can't represent directly, such as dates, are written as strings. To write the JSON
to a file, use [`save_to_json()`](`pointblank.DataScan.save_to_json`).

Returns
-------
str
    A JSON string representing the profile, indented by four spaces.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Here, a table of shipments with a date column is scanned and converted to JSON.

```{python}
import pointblank as pb
import polars as pl
from datetime import date

shipments = pl.DataFrame(
    {"shipped": [date(2025, 3, 1), date(2025, 3, 4), date(2025, 3, 9)]}
)

profile_json = pb.DataScan(shipments, tbl_name="shipments").to_json()

print(profile_json)
```

The earliest and latest dates are stored as strings. Because the result is text, it can
be stored wherever text can, and [`from_json()`](`pointblank.DataScan.from_json`) turns it
back into a `DataScan`.

from_json(json_string: 'str') -> 'DataScan'

Restore a `DataScan` from a JSON string.

The JSON string is the one produced by [`to_json()`](`pointblank.DataScan.to_json`). It's
the counterpart to that method for profiles stored as text, such as in a database column
or an API response. To read a JSON file, use
[`load_from_json()`](`pointblank.DataScan.load_from_json`).

As with [`from_dict()`](`pointblank.DataScan.from_dict`), the restored scan has the
statistics but not the data. Values that were written as strings in the JSON (such as
dates) stay strings.

Parameters
----------
json_string
    A JSON string as produced by [`to_json()`](`pointblank.DataScan.to_json`).

Returns
-------
DataScan
    A restored `DataScan` instance.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose the profile of the `small_table` dataset was stored as JSON text some time ago.

```{python}
import pointblank as pb
import polars as pl

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

stored_json = pb.DataScan(small_table, tbl_name="small_table").to_json()
```

Restoring it gives a `DataScan` that can be queried like the original one:

```{python}
restored = pb.DataScan.from_json(stored_json)

restored.summary_data.select("colname", "n_missing", "n_unique")
```

The table name, row count, and every column's statistics come from the JSON, so the
original table isn't needed to answer questions about it later.

to_dict(self) -> 'dict[str, Any]'

Export the profile as a structured dictionary.

A profile is worth keeping when it serves as a baseline: a record of what a table looked
like at one point, to compare later scans against, or to store alongside the table in a
catalog. `to_dict()` returns the whole profile as plain Python data, which
[`from_dict()`](`pointblank.DataScan.from_dict`) turns back into a `DataScan`. For a JSON
string or file, use [`to_json()`](`pointblank.DataScan.to_json`) or
[`save_to_json()`](`pointblank.DataScan.save_to_json`).

The profile includes the first five non-missing values of each column (`sample_data`). If
the table holds sensitive values, keep that in mind before storing or sharing the result.

Returns
-------
dict[str, Any]
    A dictionary with two keys. `"metadata"` holds the table name, the row count, the
    list of column names, and `"skipped_columns"` (a dictionary mapping each column of an
    unsupported type to its type). `"columns"` holds a list with one entry per summarized
    column, each a dictionary with `"colname"`, `"coltype"`, `"sample_data"`, and
    `"statistics"` (a dictionary mapping each statistic's name to its value).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's scan a small table of sensor readings, where one reading is missing.

```{python}
import pointblank as pb
import polars as pl

readings = pl.DataFrame(
    {"sensor": ["T1", "T2", "T3", "T4"], "temp_c": [21.4, 22.1, None, 20.8]}
)

profile = pb.DataScan(readings, tbl_name="readings").to_dict()

profile["metadata"]
```

Each column has its own entry, with the statistics in a nested dictionary:

```{python}
profile["columns"][1]["statistics"]
```

The missing reading is counted in `n_missing` and left out of the other statistics. The
statistics keep their Python types (the minimum and maximum of a date column, for example,
are `datetime.date` values), and passing the dictionary to
[`from_dict()`](`pointblank.DataScan.from_dict`) gives back the same profile.

from_dict(d: 'dict[str, Any]') -> 'DataScan'

Restore a `DataScan` from a dictionary.

The dictionary is the one produced by [`to_dict()`](`pointblank.DataScan.to_dict`). This
reconstructs the profile without needing the original data, so a profile that was stored
earlier can be viewed again with
[`get_tabular_report()`](`pointblank.DataScan.get_tabular_report`), queried through
[`summary_data`](`pointblank.DataScan.summary_data`), or used as the baseline in
[`compare()`](`pointblank.DataScan.compare`).

A restored scan has the statistics but not the data. Its `summary_data` is a Polars
DataFrame, whatever the type of the original table. And when it's compared with another
scan, the statistics are compared as usual, but drift scores (which need the values
themselves) are only computed for Boolean columns.

Parameters
----------
d
    A dictionary with `"metadata"` and `"columns"` keys, as produced by
    [`to_dict()`](`pointblank.DataScan.to_dict`).

Returns
-------
DataScan
    A restored `DataScan` instance.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's take a profile of a table of sensor readings as a dictionary, then restore it.

```{python}
import pointblank as pb
import polars as pl

readings = pl.DataFrame(
    {"sensor": ["T1", "T2", "T3", "T4"], "temp_c": [21.4, 22.1, None, 20.8]}
)

profile = pb.DataScan(readings, tbl_name="readings").to_dict()

restored = pb.DataScan.from_dict(profile)

restored.get_tabular_report()
```

The report is the same as the one for the original scan, even though `restored` was made
from the dictionary alone. Converting the restored scan back to a dictionary gives the
same dictionary:

```{python}
restored.to_dict() == profile
```

So a profile can go through any number of save-and-restore cycles without losing
information.


## Table Comparison

Compare two tables row by row. `compare()` aligns the rows of a source and a target table (by key, by position, or as multisets) and reports missing, extra, and changed rows, plus schema differences. The result is a `Comparison` object with counts, row samples, and a tabular report. The same comparison is available as a validation step through `Validate.tbl_match()`, and `Validate.tbl_reconcile()` compares aggregates (e.g., totals per group) instead of rows.



compare(source: 'Any', target: 'Any', keys: 'str | list[str] | None' = None, columns: 'str | list[str] | None' = None, column_map: 'dict[str, str] | None' = None, tolerance: 'Any' = None, normalize: 'dict[str, Any] | None' = None, null_equal: 'bool' = True, schema: "Literal['strict', 'common']" = 'strict', dup_keys: "Literal['flag', 'compare']" = 'flag', order_by: 'str | list[str] | None' = None, partitions: 'int | None' = None, source_name: 'str | None' = None, target_name: 'str | None' = None) -> 'Comparison'

Compare two tables row by row and report how they differ.

A common data question is *how do these two tables differ?* You may have refactored a
pipeline and want to know whether its output changed, migrated a table from one database to
another, or received a new extract of a data feed. Spot checks and row counts can tell you
*that* something changed. The `compare()` function tells you *which* rows changed, how, and in
which columns.

It aligns the rows of a *source* table (the reference, or expected data) with those of a
*target* table (the data being checked) and classifies every aligned row: rows that match,
rows whose values changed, rows missing from the target, and extra rows in the target. It also
reports schema differences (columns added, removed, retyped, or reordered). The result is a
[`Comparison`](`pointblank.Comparison`) object. It holds the counts, gives samples of the
differing rows on request, and displays as a tabular report in a notebook.

Rows are aligned in one of three ways:

- **by key** (`keys=` gives one or more columns): rows are matched on the key values, so row
  order doesn't matter. This is the most informative mode, and the one to use whenever the
  data has an identifier.
- **by position** (`keys=None`, the default): the first row of the source is compared with the
  first row of the target, and so on. This suits outputs that are supposed to be identical row
  for row.
- **as multisets** (`keys="*"`): whole rows are compared regardless of their order, with
  duplicates counted. This is for tables that have no key at all.

The comparison is backend-agnostic and runs inside the table's own engine: tables can be
Polars or Pandas DataFrames, Polars LazyFrames, Ibis tables (e.g., DuckDB, PostgreSQL), or
paths to CSV/Parquet files. File paths are scanned lazily, and only counts and small samples of
differing rows are ever collected, so tables that don't fit in memory can still be compared.

The same comparison is available as a validation step with
[`Validate.tbl_match()`](`pointblank.Validate.tbl_match`), where every aligned row is a test
unit that thresholds and actions can respond to. On the command line, it's available as
`pb compare`. To compare totals per group rather than individual rows, use
[`Validate.tbl_reconcile()`](`pointblank.Validate.tbl_reconcile`).

Parameters
----------
source
    The reference table. Can be any table type supported by Pointblank, a path to a CSV or
    Parquet file, or a callable that returns a table.
target
    The table to check against the source. Same input types as `source=`.
keys
    One or more columns that uniquely identify a row, used to align rows between the tables.
    Key names refer to the source table (use `column_map=` if the target names them
    differently). If `None`, rows are aligned by position, which requires the tables to have
    a defined row order (eager DataFrames, Polars LazyFrames, or any table with `order_by=`).
    Use `"*"` to compare the tables as multisets of whole rows (see the *Test Units and Row
    Statuses* section). `tolerance=` can't be used in that mode.
columns
    The source columns whose values should be compared. By default, all columns present in
    both tables (other than the keys) are compared. Giving `columns=` also limits the schema
    check to the keys and these columns.
column_map
    A mapping of source column names to target column names, for columns that were renamed.
tolerance
    Tolerance for numeric and datetime comparisons. A number is an absolute tolerance applied
    to all numeric columns. A dict with `"atol"` and/or `"rtol"` keys applies to all numeric
    columns, where values are equal if `|source - target| <= atol + rtol * |source|`. A dict
    mapping column names to either of the above sets per-column tolerances. Datetime
    tolerances are given as a `datetime.timedelta` or a string like `"1s"`, `"500ms"`, `"2m"`,
    or `"1d"`, e.g., `tolerance={"updated_at": "1s"}`.
normalize
    Per-column transformations applied to both sides before comparing, as a dict mapping a
    column name to a list of operations: `"strip"`, `"lower"`, `"upper"`,
    `"collapse_whitespace"`, or `{"round": n}`. The string operations only apply to string
    columns and `round` only to numeric ones (other columns are left as they are). Reported
    values are always the original ones.
null_equal
    Should a null in the source and a null in the target count as equal? By default, they do.
    NaN values in floating-point columns are treated as null.
schema
    How schema differences affect the result. With `"strict"` (the default), any schema
    difference (other than renames given in `column_map=`) means the tables don't match, and
    every row counts as failing. With `"common"`, only the shared columns are compared and
    schema differences are reported without failing rows.
dup_keys
    Rows whose key appears more than once in either table can't be aligned unambiguously, so
    they always get the `"dup_key"` status. With `"flag"` (the default), they are only flagged
    and counted. With `"compare"`, the report also compares the duplicated rows pairwise
    (every source row against every target row with the same key) to show which columns
    differ.
order_by
    Column(s) defining row order for positional alignment on backends without an inherent
    row order (e.g., database tables). Ignored when `keys=` is given.
partitions
    Compare the tables in this many passes, each covering a disjoint subset of the key values
    (by key hash). This bounds peak memory for tables too large to compare in one pass (each
    pass joins only about `1/partitions` of the rows), at the cost of reading the tables once
    per pass. Requires `keys=`, and Polars, DuckDB, or Ibis tables (including CSV/Parquet
    paths). The results are identical to a single-pass comparison.
source_name, target_name
    Optional labels for the two tables, used in the report.

Returns
-------
Comparison
    The result of the comparison.

Test Units and Row Statuses
---------------------------
Every aligned row is one *test unit* and gets exactly one status:

- `"match"`: the row is in both tables and all compared values are equal (within tolerance)
- `"changed"`: the row is in both tables but at least one compared value differs
- `"missing"`: the row is in the source but not in the target
- `"extra"`: the row is in the target but not in the source
- `"dup_key"`: the row's key appears more than once in either table. A duplicated key counts
  as many units as the larger of its two occurrence counts.

Rows with a null in a key column can't be matched: they count as `"missing"` (source) or
`"extra"` (target).

In multiset mode (`keys="*"`), a row is either present in the other table or not, so there are
no `"changed"` or `"dup_key"` rows. Each distinct row matches as many times as it occurs in
both tables, and any surplus copies count as `"missing"` or `"extra"`.

Large Tables
------------
The comparison runs inside the tables' own engine and only counts and small samples of rows
are collected, so file paths (scanned lazily) and database tables never have to be loaded into
Python. The engine's join still needs memory for the rows it compares. If that's too much, use
`partitions=` to compare the tables in several smaller passes. For the lowest memory use with
very large files, read them as DuckDB relations (e.g., `duckdb.connect().read_csv(path)`) and
use `partitions=`: in a benchmark with two 10-million-row CSV files (730 MB each), peak memory
was about 0.8 GB with DuckDB and `partitions=8`, versus about 5.7 GB to read both files with
Polars and join them. With 100-million-row files (7.4 GB each), DuckDB with `partitions=8`
peaked at about 2 GB.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's compare two versions of a product catalog. In the newer version, one product was
discontinued, one was added, one had its price lowered, and one went out of stock.

```{python}
import pointblank as pb
import polars as pl

catalog_v1 = pl.DataFrame(
    {
        "sku": ["A-101", "A-102", "A-103", "A-104", "A-105"],
        "name": ["Desk lamp", "Office chair", "Monitor arm", "Keyboard", "Mouse pad"],
        "price": [24.99, 149.00, 59.50, 45.00, 9.99],
        "in_stock": [True, True, False, True, True],
    }
)

catalog_v2 = pl.DataFrame(
    {
        "sku": ["A-101", "A-102", "A-104", "A-105", "A-106"],
        "name": ["Desk lamp", "Office chair", "Keyboard", "Mouse pad", "USB hub"],
        "price": [24.99, 139.00, 45.00, 9.99, 19.99],
        "in_stock": [True, True, True, False, True],
    }
)
```

Every product has a unique `sku`, so we'll use it as the key that lines up the rows. Naming
the two tables makes the report easier to read.

```{python}
comparison = pb.compare(
    catalog_v1, catalog_v2, keys="sku", source_name="catalog_v1", target_name="catalog_v2"
)

comparison
```

The report's header gives the verdict and a breakdown of the six aligned rows. Below it are
the columns with mismatched values, then samples of the changed, missing, and extra rows. For
changed rows, only the differing values are shown.

The same information is available programmatically. The status counts add up to the number
of test units (aligned rows):

```{python}
comparison.status_counts
```

To work with the differing rows, get them by status with
[`rows()`](`pointblank.Comparison.rows`). Changed rows have a `<column>_source` and a
`<column>_target` column for each compared column.

```{python}
comparison.rows("changed")
```

`A-102` changed price and `A-105` changed its stock status. If only prices matter, list the
columns to compare with `columns=`. The stock change is then ignored and `A-105` matches:

```{python}
pb.compare(catalog_v1, catalog_v2, keys="sku", columns="price").status_counts
```

Without `keys=`, rows are compared by position. Since a product was removed from the middle
of the catalog, most rows no longer line up, and four of the five rows count as changed. This
is why a key gives a much more useful answer when the data has one.

```{python}
pb.compare(catalog_v1, catalog_v2).status_counts
```

Schema changes are found too. Suppose the next version renames `price` to `unit_price`. With
the default `schema="strict"`, the comparison fails every row until the schemas agree.
`column_map=` pairs up the renamed column, after which the values are compared as before:

```{python}
catalog_v3 = catalog_v2.rename({"price": "unit_price"})

pb.compare(catalog_v1, catalog_v3, keys="sku", column_map={"price": "unit_price"})
```

The rename is listed as a schema difference, but it no longer causes a failure, and the
result is the same as the comparison of `catalog_v1` with `catalog_v2`. To make a comparison
like this part of a validation, with thresholds, actions, and a step in the validation report,
use [`Validate.tbl_match()`](`pointblank.Validate.tbl_match`) with the same arguments.

See Also
--------
Comparison : The object returned by this function.
SchemaDiff : The schema differences found by a comparison.
Validate.tbl_match : Use a table comparison as a validation step.
Validate.tbl_reconcile : Compare aggregates per group instead of rows.

Comparison() -> 'None'

The result of comparing a source table with a target table.

`Comparison` objects are created by [`compare()`](`pointblank.compare`); they can't be created
directly. A comparison does its counting when it's created, inside the tables' own engine, and
only fetches rows when you ask for them. This keeps it cheap to create and to pass around,
even for very large tables.

What you do with a `Comparison` depends on the question being asked:

- **Do the tables match?** A `Comparison` is truthy only when the tables match (see
  [`identical`](`pointblank.Comparison.identical`)), so `if pb.compare(a, b):` and
  `assert pb.compare(a, b)` read naturally. [`n_failed`](`pointblank.Comparison.n_failed`)
  says by how many rows they don't.
- **How do they differ?** [`status_counts`](`pointblank.Comparison.status_counts`) breaks the
  rows down by status, [`schema_diff`](`pointblank.SchemaDiff`) lists the column differences,
  and [`column_summary()`](`pointblank.Comparison.column_summary`) shows which columns have
  mismatched values.
- **Which rows differ?** [`rows()`](`pointblank.Comparison.rows`) returns a sample of the rows
  with a given status, as a table of the same type as the inputs.
- **How can I show or store the result?** In a notebook, a `Comparison` displays as its
  [tabular report](`pointblank.Comparison.get_tabular_report`). For logs, files, and other
  programs, there's [`to_dict()`](`pointblank.Comparison.to_dict`) and
  [`to_json()`](`pointblank.Comparison.to_json`).

Attributes
----------
source_name, target_name
    Labels for the two tables.
keys
    The key columns used to align rows (empty for positional alignment).
mode
    `"keyed"`, `"positional"`, or `"multiset"` (for `keys="*"`).
compared_columns
    The source columns whose values were compared.
column_map
    The mapping of source to target column names given in `column_map=`.
schema_mode
    How schema differences affect the result: `"strict"` or `"common"`.
schema_diff
    A [`SchemaDiff`](`pointblank.SchemaDiff`) describing schema differences.
n_source_rows, n_target_rows
    Row counts of the two tables.
n_match, n_changed, n_missing, n_extra, n_dup_key
    Number of test units (aligned rows) with each status.
n_dup_key_values
    Number of distinct key values that are duplicated in either table.
column_mismatches
    A dict mapping each compared source column to the number of matched rows where its values
    differ.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's compare the output of a reporting job with the output of a new version of the job. The
new version should produce exactly the same daily totals.

```{python}
import pointblank as pb
import polars as pl

old_totals = pl.DataFrame(
    {
        "day": ["2025-06-01", "2025-06-02", "2025-06-03", "2025-06-04"],
        "orders": [118, 131, 97, 142],
        "revenue": [5902.10, 6611.45, 4875.00, 7120.80],
    }
)

new_totals = pl.DataFrame(
    {
        "day": ["2025-06-01", "2025-06-02", "2025-06-03", "2025-06-04"],
        "orders": [118, 131, 97, 142],
        "revenue": [5902.10, 6611.45, 4857.00, 7120.80],
    }
)

comparison = pb.compare(old_totals, new_totals, keys="day")

print(repr(comparison))
```

The printed form is a one-line summary: one day changed and the schemas agree, so the tables
aren't identical. In a test or a deployment script, the comparison can be used as a condition
directly. Here, when the tables differ, we look at the rows that changed:

```{python}
if comparison:
    print("The new job reproduces the old totals.")
else:
    print(f"{comparison.n_failed} of {comparison.n} days differ:")
    print(comparison.rows("changed"))
```

The revenue for June 3 has two digits swapped. To see where the mismatches are concentrated,
which matters more when there are many columns, use `column_summary()`:

```{python}
comparison.column_summary()
```

Only `revenue` has a mismatch, in one of the four matched rows. From here, the full report is
available with [`get_tabular_report()`](`pointblank.Comparison.get_tabular_report`), and the
result can be saved with `to_json()` to keep a record of the check.

See Also
--------
pointblank.compare : Create a `Comparison` from two tables.
SchemaDiff : The schema differences held in `schema_diff`.
Validate.tbl_match : Run the same comparison as a validation step.

SchemaDiff(only_in_source: 'list[str]' = <factory>, only_in_target: 'list[str]' = <factory>, dtype_changed: 'list[tuple[str, str, str]]' = <factory>, renamed: 'list[tuple[str, str]]' = <factory>, case_only: 'list[tuple[str, str]]' = <factory>, reordered: 'bool' = False) -> None

Differences between the schemas of the source and target tables.

Changes to a table's columns are often more disruptive than changes to its values. A dropped
column breaks the queries that select it, a retyped column changes how values sort and
aggregate, and a renamed column looks like one column removed and another added. Every
[`Comparison`](`pointblank.Comparison`) checks the two schemas before comparing values and
records what it found in its `schema_diff` attribute, which is a `SchemaDiff`.

Whether these differences fail the comparison depends on `schema=` in
[`compare()`](`pointblank.compare`). With the default `"strict"`, any difference other than an
explicit rename means that every row counts as failing. With `"common"`, the differences are
only reported, and the shared columns are compared as usual. Either way, the `SchemaDiff`
lists the differences, and they appear in the *Schema differences* section of the comparison
report.

Column names refer to the source table except in `only_in_target` (and the second element of
`renamed` and `case_only` pairs), which refer to the target table. A `SchemaDiff` is
immutable.

Attributes
----------
only_in_source
    Columns present in the source but not the target.
only_in_target
    Columns present in the target but not the source.
dtype_changed
    `(column, source_dtype, target_dtype)` for shared columns whose data types differ.
renamed
    `(source_column, target_column)` pairs paired up through `column_map=`.
case_only
    `(source_column, target_column)` pairs whose names differ only by letter case. These are
    *not* paired up automatically (they also appear in `only_in_source`/`only_in_target`); use
    `column_map=` to compare them.
reordered
    `True` if the shared columns appear in a different order in the two tables.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's compare a table of customers with a version written by a migrated job. The new version
stores the signup date as a string, capitalizes one column name, and adds a column.

```{python}
import pointblank as pb
import polars as pl
from datetime import date

customers = pl.DataFrame(
    {
        "customer_id": [1, 2, 3],
        "email": ["ana@example.com", "ben@example.com", "cho@example.com"],
        "signup": [date(2024, 1, 5), date(2024, 2, 11), date(2024, 3, 2)],
    }
)

customers_migrated = pl.DataFrame(
    {
        "customer_id": [1, 2, 3],
        "Email": ["ana@example.com", "ben@example.com", "cho@example.com"],
        "signup": ["2024-01-05", "2024-02-11", "2024-03-02"],
        "loaded_at": ["2025-06-01"] * 3,
    }
)

comparison = pb.compare(customers, customers_migrated, keys="customer_id")

comparison.schema_diff
```

Each kind of difference has its own field. `email` and `Email` appear in `only_in_source` and
`only_in_target`, since column names are case-sensitive, and `case_only` points out that the
two names differ only in case. `signup` is in `dtype_changed` with both data types, and
`loaded_at` is new in the target.

Because the schemas differ and `schema="strict"` is the default, no row passes:

```{python}
comparison.schema_diff.matches, comparison.n_passed
```

Pairing up the renamed column with `column_map=` resolves the case change, and
`schema="common"` reports the other two differences without failing any rows:

```{python}
relaxed = pb.compare(
    customers,
    customers_migrated,
    keys="customer_id",
    column_map={"email": "Email"},
    schema="common",
)

relaxed.schema_diff
```

The rename now appears in `renamed`, and `case_only` is empty. The retyped `signup` column is
still listed, but it no longer fails the comparison. When a column's two data types can't be
compared directly (here, a date and a string), both sides are compared as strings, so the
three dates still match:

```{python}
relaxed.status_counts
```

So `schema="common"` answers whether the values survived the migration, while the
`SchemaDiff` records how the columns changed along the way. For a schema check that doesn't
compare any values, see
[`Validate.col_schema_match()`](`pointblank.Validate.col_schema_match`).

See Also
--------
pointblank.compare : Compare two tables, including their schemas.
Comparison : The comparison result that holds a `SchemaDiff`.
Validate.col_schema_match : Check a table against an expected schema.


## Comparison Methods

Methods for the Comparison class


Comparison.identical

Whether the tables match under the comparison settings.

A comparison is identical if the schema check passed (see
[`schema_ok`](`pointblank.Comparison.schema_ok`)) and no test unit failed. This is the
single answer to "are these tables the same?", and it's also the truth value of the
comparison, so `if comparison:` is the same as `if comparison.identical:`.

"Identical" is relative to the settings given to [`compare()`](`pointblank.compare`).
Columns left out with `columns=` aren't checked, values within `tolerance=` count as equal,
values are compared after any `normalize=` operations, and with `schema="common"`, the
tables can have different columns.

Returns
-------
bool
    `True` if the tables match.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Here, a reporting table stores the readings rounded to one decimal place.

```{python}
import pointblank as pb
import polars as pl

raw = pl.DataFrame({"station": ["S1", "S2", "S3"], "temp": [12.14, 14.02, 9.81]})
rounded = pl.DataFrame({"station": ["S1", "S2", "S3"], "temp": [12.1, 14.0, 9.8]})

pb.compare(raw, rounded, keys="station").identical
```

An exact comparison finds every row changed. Allowing for the rounding with `tolerance=`
makes the tables match:

```{python}
comparison = pb.compare(raw, rounded, keys="station", tolerance={"temp": 0.05})

comparison.identical
```

Since a comparison has the same truth value, it can be used directly in a condition:

```{python}
if comparison:
    print("The rounded table matches the raw readings.")
```

A result of `True` here means "the same within 0.05 degrees", not "exactly the same". The
settings belong with the result, and the comparison report lists them in its header.

Comparison.schema_ok

Whether the schema check passed.

A comparison checks the two tables' schemas before it compares their values. How a schema
difference is treated depends on `schema=` in [`compare()`](`pointblank.compare`). With
`schema="strict"` (the default), `schema_ok` is the same as
[`SchemaDiff.matches`](`pointblank.SchemaDiff.matches`), and a `False` value means that
every test unit fails, so [`n_passed`](`pointblank.Comparison.n_passed`) is `0`. With
`schema="common"`, schema differences are only reported, so `schema_ok` is always `True`.

This makes `schema_ok` the property to check when a comparison fails and it's not clear
whether the values or the columns are to blame. The details of the difference are in
`schema_diff`.

Returns
-------
bool
    `True` if the schemas match, or if `schema="common"` was used.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Here, the target table has the same rows as the source, plus a new `humidity` column.

```{python}
import pointblank as pb
import polars as pl

readings = pl.DataFrame({"station": ["S1", "S2", "S3"], "temp": [12.1, 14.0, 9.8]})
readings_new = readings.with_columns(humidity=pl.Series([0.61, 0.58, 0.72]))

comparison = pb.compare(readings, readings_new, keys="station")

comparison.schema_ok, comparison.status_counts, comparison.n_passed
```

Every row has a status of `"match"`, but none of them pass, because the schema check
failed. The cause is in the schema differences:

```{python}
comparison.schema_diff.only_in_target
```

If the new column is expected, `schema="common"` compares the shared columns and lets the
rows pass:

```{python}
relaxed = pb.compare(readings, readings_new, keys="station", schema="common")

relaxed.schema_ok, relaxed.n_passed
```

The new column is still listed in `relaxed.schema_diff`, so the change remains visible in
the report even though it no longer fails the comparison.

Comparison.status_counts

The number of test units with each row status.

Where [`n_passed`](`pointblank.Comparison.n_passed`) and
[`n_failed`](`pointblank.Comparison.n_failed`) say how many rows differ,
`status_counts` says how they differ. The statuses are described in the
*Test Units and Row Statuses* section of [`compare()`](`pointblank.compare`). The counts
are computed when the comparison is made, so reading them doesn't touch the tables again.

The counts are the row statuses alone and aren't affected by the schema check. If the
schema check fails, there can be `"match"` rows even though no test units pass.

Returns
-------
dict[str, int]
    A dictionary with the keys `"match"`, `"changed"`, `"missing"`, `"extra"`, and
    `"dup_key"` (in that order). The values add up to [`n`](`pointblank.Comparison.n`).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

In this example, station `S3` appears twice in the source table.

```{python}
import pointblank as pb
import polars as pl

readings_v1 = pl.DataFrame(
    {"station": ["S1", "S2", "S3", "S3", "S4"], "temp": [12.1, 14.0, 9.8, 9.9, 11.2]}
)
readings_v2 = pl.DataFrame(
    {"station": ["S1", "S2", "S3", "S5"], "temp": [12.1, 14.3, 9.8, 8.0]}
)

comparison = pb.compare(readings_v1, readings_v2, keys="station")

comparison.status_counts
```

Each status has a different cause. `S2` changed, `S4` is missing from the target, and `S5`
is extra. The duplicated `S3` key counts as two `"dup_key"` units, since it occurs twice in
the source. Those rows can't be paired up, so they're not compared. To see the rows behind
any of these counts, pass the status to [`rows()`](`pointblank.Comparison.rows`), as in
`comparison.rows("dup_key")`.

Comparison.n

The number of test units (aligned rows).

A comparison aligns the rows of the two tables and gives each aligned row one status (see
the *Test Units and Row Statuses* section of [`compare()`](`pointblank.compare`)). `n` is
the total, so it's always the sum of the values in
[`status_counts`](`pointblank.Comparison.status_counts`) and of
[`n_passed`](`pointblank.Comparison.n_passed`) and
[`n_failed`](`pointblank.Comparison.n_failed`). It's the denominator to use for any
fraction of rows that pass or fail.

`n` isn't the row count of either table. A row that's in both tables counts once, rows in
only one table count once each, and a duplicated key counts as many units as the larger of
its two occurrence counts. The row counts of the two tables are available as
`n_source_rows` and `n_target_rows`.

Returns
-------
int
    The number of test units.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

These two tables of weather station readings have four rows each, but only three stations
in common.

```{python}
import pointblank as pb
import polars as pl

readings_v1 = pl.DataFrame(
    {"station": ["S1", "S2", "S3", "S4"], "temp": [12.1, 14.0, 9.8, 11.2]}
)
readings_v2 = pl.DataFrame(
    {"station": ["S1", "S2", "S3", "S5"], "temp": [12.1, 14.3, 9.8, 8.0]}
)

comparison = pb.compare(readings_v1, readings_v2, keys="station")

comparison.n, comparison.n_source_rows, comparison.n_target_rows
```

There are five test units: the three shared stations, `S4` (only in the source), and `S5`
(only in the target). Dividing by `n` gives the fraction of rows that failed:

```{python}
comparison.n_failed / comparison.n
```

Three of the five units failed: `S2` changed, `S4` is missing, and `S5` is extra.

Comparison.n_passed

The number of test units that passed.

A test unit passes if its status is `"match"`, which means the row is in both tables with
the same values in every compared column (within any tolerance set in
[`compare()`](`pointblank.compare`)). If the schema check failed (see
[`schema_ok`](`pointblank.Comparison.schema_ok`)), no test unit passes and `n_passed` is
`0`, whatever the row statuses are.

`n_passed` and [`n_failed`](`pointblank.Comparison.n_failed`) are the counts that
[`Validate.tbl_match()`](`pointblank.Validate.tbl_match`) uses for its test units, so they
work with the same thresholds and fractions as any other validation step.

Returns
-------
int
    The number of passing test units.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Two of these four stations have the same reading in both tables.

```{python}
import pointblank as pb
import polars as pl

readings_v1 = pl.DataFrame(
    {"station": ["S1", "S2", "S3", "S4"], "temp": [12.1, 14.0, 9.8, 11.2]}
)
readings_v2 = pl.DataFrame(
    {"station": ["S1", "S2", "S3", "S4"], "temp": [12.1, 14.3, 9.8, 11.5]}
)

comparison = pb.compare(readings_v1, readings_v2, keys="station")

comparison.n_passed, comparison.n_passed / comparison.n
```

Half of the test units passed. The two changed rows are available with
[`rows("changed")`](`pointblank.Comparison.rows`).

Comparison.n_failed

The number of test units that failed.

A test unit fails if its status is `"changed"`, `"missing"`, `"extra"`, or `"dup_key"`. If
the schema check failed (see [`schema_ok`](`pointblank.Comparison.schema_ok`)), every test
unit fails, so `n_failed` equals [`n`](`pointblank.Comparison.n`). It's always
`n - n_passed`.

A count of zero failures, along with a passing schema check, is what makes a comparison
[`identical`](`pointblank.Comparison.identical`). To see how the failures break down by
status, use [`status_counts`](`pointblank.Comparison.status_counts`).

Returns
-------
int
    The number of failing test units.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

One station's reading changed, one station was dropped, and one was added.

```{python}
import pointblank as pb
import polars as pl

readings_v1 = pl.DataFrame(
    {"station": ["S1", "S2", "S3", "S4"], "temp": [12.1, 14.0, 9.8, 11.2]}
)
readings_v2 = pl.DataFrame(
    {"station": ["S1", "S2", "S3", "S5"], "temp": [12.1, 14.3, 9.8, 8.0]}
)

comparison = pb.compare(readings_v1, readings_v2, keys="station")

comparison.n_failed
```

A count is useful for a quick check, for instance to stop a script when too many rows
differ:

```{python}
if comparison.n_failed > 0.25 * comparison.n:
    print(f"{comparison.n_failed} of {comparison.n} rows differ")
```

To act on failures with thresholds and actions instead, compare the tables in a validation
with [`Validate.tbl_match()`](`pointblank.Validate.tbl_match`).

rows(self, status: "Literal['changed', 'missing', 'extra', 'dup_key', 'match']", limit: 'int' = 100) -> 'Any'

Get a sample of rows with a given status.

The counts in [`status_counts`](`pointblank.Comparison.status_counts`) say how many rows
differ, and `rows()` shows which ones. It's the step that turns "12 rows changed" into
something that can be investigated: the keys to look up, the values on each side, and the
rows that were dropped or added.

A comparison collects counts when it's made, but it doesn't collect rows until they're
asked for. `rows()` then fetches up to `limit=` rows with the requested status from the
tables' own engine, so it's safe to use with very large tables and with database tables.
The result is cached, so asking for the same status and limit again returns the same table
without querying the tables a second time.

Parameters
----------
status
    The row status to retrieve:

    - `"changed"`: the keys, then a `<column>_source` and a `<column>_target` column for
      each compared column (including columns whose values are equal in that row)
    - `"missing"`: source rows not in the target (source column names)
    - `"extra"`: target rows not in the source (target column names)
    - `"dup_key"`: duplicated key values with their occurrence counts in each table
      (`n_source`, `n_target`), one row per key
    - `"match"`: the keys of matching rows
limit
    The maximum number of rows to return.

Returns
-------
Any
    A table of the same type as the (collected) input tables. In positional mode, rows
    are identified by a 1-based `_row_num_` column, and a `"changed"` sample has the
    `_source` and `_target` columns for every column. In multiset mode (`keys="*"`),
    `"missing"` and `"extra"` return distinct rows, with the compared values as strings,
    plus `n_source` and `n_target` columns giving how often each row occurs in each table.

Raises
------
ValueError
    If `status=` isn't one of the five row statuses.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's compare two versions of an employee table, where one salary changed, one employee
left, and one joined.

```{python}
import pointblank as pb
import polars as pl

staff_jan = pl.DataFrame(
    {
        "emp_id": [101, 102, 103, 104],
        "dept": ["Sales", "Sales", "Ops", "Ops"],
        "salary": [52000, 61000, 58000, 47000],
    }
)
staff_feb = pl.DataFrame(
    {
        "emp_id": [101, 102, 103, 105],
        "dept": ["Sales", "Sales", "Ops", "Ops"],
        "salary": [52000, 63500, 58000, 49000],
    }
)

comparison = pb.compare(staff_jan, staff_feb, keys="emp_id")

comparison.rows("changed")
```

Each changed row has both versions of every compared column side by side. Here, `dept`
is the same on both sides and `salary` is the column that changed. The departed employee
is a `"missing"` row, and it's shown as it was in the source table:

```{python}
comparison.rows("missing")
```

The new employee is an `"extra"` row, shown as it is in the target table:

```{python}
comparison.rows("extra")
```

In multiset mode, rows have no identity apart from their values. So `"missing"` and
`"extra"` rows are listed once each, with the number of times they occur in each table.
Here, the target table has one fewer copy of a row than the source:

```{python}
scans_source = pl.DataFrame({"badge": [7, 7, 9], "door": ["A", "A", "B"]})
scans_target = pl.DataFrame({"badge": [7, 9], "door": ["A", "B"]})

pb.compare(scans_source, scans_target, keys="*").rows("missing")
```

The row with badge 7 at door A occurs twice in the source and once in the target, so one
copy is missing. For a summary of all the samples in one place, see
[`get_tabular_report()`](`pointblank.Comparison.get_tabular_report`), which shows up to 10
rows for each status by default.

column_summary(self) -> 'Any'

Get a per-column summary of value mismatches.

A row counts as `"changed"` if any compared column differs, so the row statuses don't say
which columns are responsible. `column_summary()` does. It's often the quickest way to find
the cause of a difference. A single column with mismatches in every row suggests a change
in how that column is computed or formatted, while mismatches spread thinly across many
columns suggest changed rows.

The summary also lists each column's data type on both sides and the target column it was
compared against, which shows the effect of `column_map=`. The counts are computed when
the comparison is made, so this doesn't query the tables again.

Returns
-------
Any
    A table (Polars if available, else Pandas) with one row per compared column: the
    source and target column names and data types, the number of matched rows whose
    values differ (`n_mismatch`), and that number as a fraction of matched rows
    (`f_mismatch`). Matched rows are those with a status of `"match"` or `"changed"`. In
    multiset mode, there are no matched pairs, so the last two columns are null.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Here, a rewritten job produces an orders table where the totals are rounded to whole
units, and one order has a different status.

```{python}
import pointblank as pb
import polars as pl

orders_old = pl.DataFrame(
    {
        "order_id": [1, 2, 3, 4],
        "status": ["shipped", "shipped", "pending", "shipped"],
        "total": [19.99, 45.50, 12.00, 80.25],
    }
)
orders_new = pl.DataFrame(
    {
        "order_id": [1, 2, 3, 4],
        "status": ["shipped", "shipped", "cancelled", "shipped"],
        "total": [20.0, 46.0, 12.0, 80.0],
    }
)

comparison = pb.compare(orders_old, orders_new, keys="order_id")

comparison.column_summary()
```

`total` differs in three of the four orders, which points to a systematic change (the
rounding), while `status` differs in a single order. All four rows are `"changed"`, but
the summary separates the two causes. Once the rounding is accepted (for instance, with
`tolerance={"total": 0.5}`), only the status change is left.

get_tabular_report(self, limit: 'int' = 10, title: 'str | None' = None, lang: 'str | None' = None, locale: 'str | None' = None) -> 'GT'

Get a tabular report of the comparison.

The report brings everything about a comparison together in one table, for reading
rather than for further processing. It's what's shown when a comparison is displayed in a
notebook, and calling `get_tabular_report()` directly lets the report be customized and
saved, for example to attach to a pull request or to share with whoever owns the data.

The report has a header with the verdict, the row status counts, and the comparison
settings, followed by sections (as row groups) for schema differences, per-column
mismatches, and samples of changed, missing, extra, and duplicate-key rows. Changed rows
show only the columns that differ, with the source and target values (and the difference,
for numbers). Sections with nothing to show are omitted.

Parameters
----------
limit
    The maximum number of sample rows shown per row status.
title
    An optional title replacing the default.
lang
    The language of the report's labels, as a two-letter language code (e.g., `"fr"`,
    `"de"`) or one of the other Pointblank reporting languages. English by default.
    Values from the tables (keys, column names, and data) are shown as they are.
locale
    The locale used to format counts and percentages (e.g., the thousands separator).
    By default, it's the same as `lang=`.

Returns
-------
GT
    A Great Tables object.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's compare two months of a payroll table. Naming the tables with `source_name=` and
`target_name=` makes the report easier to read.

```{python}
import pointblank as pb
import polars as pl

payroll_jan = pl.DataFrame(
    {
        "emp_id": [101, 102, 103, 104],
        "dept": ["Sales", "Sales", "Ops", "Ops"],
        "salary": [52000, 61000, 58000, 47000],
    }
)
payroll_feb = pl.DataFrame(
    {
        "emp_id": [101, 102, 103, 105],
        "dept": ["Sales", "Sales", "Ops", "Ops"],
        "salary": [52000, 63500, 58000, 49000],
    }
)

comparison = pb.compare(
    payroll_jan,
    payroll_feb,
    keys="emp_id",
    source_name="payroll_jan",
    target_name="payroll_feb",
)

comparison.get_tabular_report()
```

The verdict and counts are at the top, and below them are the sections that have
something to show: the mismatches in `salary`, the changed row for employee 102, and the
employees who left and joined.

The title can be replaced, and the labels can be shown in any of Pointblank's reporting
languages. Here's the same report in French:

```{python}
comparison.get_tabular_report(title="Paie : janvier et février", lang="fr")
```

Numbers are formatted for the French locale too, with a decimal comma (as in `60,0%`). Use
`locale=` to format them differently from the labels. To save a report as a standalone
HTML file, use the `write_raw_html()` method of the returned Great Tables object.

to_dict(self, samples: 'bool' = True, limit: 'int' = 10) -> 'dict[str, Any]'

Get the comparison results as a JSON-serializable dict.

A comparison is a live object that refers to the tables it compared. `to_dict()` turns
its results into plain data that can be stored or sent elsewhere: written to a log or a
database table, attached to a CI run, checked in a test, or passed to a dashboard.
Everything in the dictionary is a string, number, boolean, list, dict, or `None` (dates and
other values in the samples are converted to strings), so it can be serialized as it is.
For a JSON string, use [`to_json()`](`pointblank.Comparison.to_json`).

Parameters
----------
samples
    Should samples of differing rows be included? Set to `False` when the data might
    contain sensitive values (e.g., PII) and only counts and schema details may be stored.
limit
    The maximum number of sample rows per status.

Returns
-------
dict[str, Any]
    A dictionary with the comparison settings (`"source_name"`, `"target_name"`, `"mode"`,
    `"keys"`, `"compared_columns"`, `"column_map"`, `"schema_mode"`, `"null_equal"`, and
    `"partitions"`), the row counts of both tables, the results (`"n"`, `"n_passed"`,
    `"n_failed"`, `"identical"`, `"status_counts"`, and `"column_mismatches"`), and the
    schema differences (`"schema_diff"`, from
    [`SchemaDiff.to_dict()`](`pointblank.SchemaDiff.to_dict`)). If `samples=True`, there's
    also a `"samples"` entry, which maps each failing status that has rows to a list of
    row dictionaries, structured as in [`rows()`](`pointblank.Comparison.rows`).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Let's compare two months of an employee table and turn the result into a dictionary.

```{python}
import pointblank as pb
import polars as pl

staff_jan = pl.DataFrame(
    {"emp_id": [101, 102, 103, 104], "salary": [52000, 61000, 58000, 47000]}
)
staff_feb = pl.DataFrame(
    {"emp_id": [101, 102, 103, 105], "salary": [52000, 63500, 58000, 49000]}
)

comparison = pb.compare(
    staff_jan, staff_feb, keys="emp_id", source_name="staff_jan", target_name="staff_feb"
)

result = comparison.to_dict()

result["status_counts"]
```

The samples hold the actual rows, keyed by status:

```{python}
result["samples"]
```

Salaries are sensitive, so a record that's kept in a shared log might leave the samples
out. With `samples=False`, only counts, settings, and schema details remain:

```{python}
record = comparison.to_dict(samples=False)

{k: record[k] for k in ["source_name", "target_name", "n_failed", "identical"]}
```

Leaving out the samples also means the tables aren't queried again, because the counts
were collected when the comparison was made. To limit the samples instead of dropping them,
set `limit=` to the number of rows to keep for each status.

to_json(self, samples: 'bool' = True, limit: 'int' = 10, indent: 'int | None' = 2) -> 'str'

Get the comparison results as a JSON string.

JSON is the format to use when a comparison result leaves Python: written to a file
alongside a pipeline run, returned from an API, or read by another tool. The structure is
the same as that of [`to_dict()`](`pointblank.Comparison.to_dict`). The `pb compare`
command line tool produces the same JSON with `--format json` (to the terminal) or
`--output-json` (to a file), and its `--no-samples` option corresponds to `samples=False`.

Parameters
----------
samples
    Should samples of differing rows be included? Set to `False` when the data might
    contain sensitive values (e.g., PII) and only counts and schema details may be stored.
limit
    The maximum number of sample rows per status.
indent
    The number of spaces to indent nested levels by. Use `None` for the most compact
    output, on a single line (as for JSON Lines files, with one comparison per line).

Returns
-------
str
    The comparison results as a JSON string.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False, preview_incl_header=False)
```

Here, a new version of a price list has one changed price.

```{python}
import pointblank as pb
import polars as pl

prices_v1 = pl.DataFrame({"sku": ["A-1", "A-2", "A-3"], "price": [9.99, 24.50, 5.00]})
prices_v2 = pl.DataFrame({"sku": ["A-1", "A-2", "A-3"], "price": [9.99, 26.00, 5.00]})

comparison = pb.compare(prices_v1, prices_v2, keys="sku")

print(comparison.to_json(limit=5))
```

For a compact record without row samples, set `samples=False` and `indent=None`:

```{python}
print(comparison.to_json(samples=False, indent=None))
```

Each such record is one line, so a series of comparisons can be appended to a single file
and read back later, one comparison per line.


## Table Pre-checks

Helper functions for use with the `active=` parameter of validation methods. These inspect the target table before a step runs and conditionally skip the step when preconditions are not met.



has_columns(*columns: 'str | list[str]') -> 'Callable[[Any], bool]'

Check whether one or more columns exist in a table.

This function returns a callable that, when given a table, checks whether all specified columns
are present. It is primarily designed for use with the `active=` parameter of validation
methods. When a validation step has `active=has_columns("col_a", "col_b")`, the step will be
skipped (made inactive) if either `col_a` or `col_b` is missing from the target table.

The callable is evaluated against the original table *before* any `pre=` processing is
applied. This means the column check is performed on the raw input data, not on a
pre-processed version of it.

A note is attached to any skipped step in the validation report explaining which columns were
not found.

Parameters
----------
*columns
    One or more column names to check for in the table. Each argument can be a string or a
    list of strings. All specified columns must be present for the callable to return `True`.

Returns
-------
Callable[[Any], bool]
    A callable that accepts a table and returns `True` if every column in `columns` exists
    in the table, `False` otherwise.

Raises
------
ValueError
    If no column names are provided.
TypeError
    If any of the provided column names is not a string or list of strings.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Using `has_columns()` with the `active=` parameter to conditionally run a validation step:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})

validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=0, active=pb.has_columns("a"))
    .col_vals_gt(columns="a", value=0, active=pb.has_columns("z"))
    .interrogate()
)

validation
```

The first step ran because column `a` exists. The second step was skipped because column `z` is
missing, and the report note explains which column was not found.

When checking for multiple columns, the step is only active when *all* columns are present:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=0, active=pb.has_columns("a", "b"))
    .col_vals_gt(columns="a", value=0, active=pb.has_columns("a", "x", "y"))
    .interrogate()
)

validation
```

The first step is active because both `a` and `b` exist. The second step is skipped because `x`
and `y` are missing.

Column names can also be provided as a list:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="a", value=0, active=pb.has_columns(["a", "b"]))
    .interrogate()
)

validation
```

Passing a list is the same as passing the column names as separate arguments, so this step is
active and passes. Lists are handy when the required column names are already stored in a
variable.

has_rows(count: 'int | None' = None, *, min: 'int | None' = None, max: 'int | None' = None) -> 'Callable[[Any], bool]'

Check whether a table has a certain number of rows.

The `has_rows()` function returns a callable that, when given a table, checks whether the row
count satisfies a specified condition. It is designed for use with the `active=` parameter of
validation methods so that a validation step can be conditionally skipped when the target table
is too small, too large, or empty.

The callable supports several modes:

- **exact count**: `has_rows(count=N)` returns `True` only if the table has exactly `N` rows.
- **minimum**: `has_rows(min=N)` returns `True` if the table has at least `N` rows.
- **maximum**: `has_rows(max=N)` returns `True` if the table has at most `N` rows.
- **range**: `has_rows(min=A, max=B)` returns `True` if the row count falls within `[A, B]`.
- **non-empty**: `has_rows()` (no arguments) returns `True` if the table has at least one row.

A note is attached to any skipped step in the validation report explaining the row count
condition that was not met.

The callable is evaluated against the original table *before* any `pre=` processing is
applied. This means the column check is performed on the raw input data, not on a
pre-processed version of it.

Parameters
----------
count
    The exact number of rows the table should have. Cannot be used together with `min=` or
    `max=`.
min
    The minimum number of rows (inclusive) the table should have. Can be used alone or with
    `max=`.
max
    The maximum number of rows (inclusive) the table should have. Can be used alone or with
    `min=`.

Returns
-------
Callable[[Any], bool]
    A callable that accepts a table and returns `True` if the row count satisfies the specified
    condition, `False` otherwise. When the callable returns `False`, it stores diagnostic
    information that is used to generate a descriptive note in the validation report.

How It Works
------------
When [`interrogate()`](`pointblank.Validate.interrogate`) is called, each validation step whose
`active=` parameter is a callable will have that callable evaluated with the target table. If
the callable returns `False`, the step is deactivated and an explanatory note is added to the
validation report. The note is locale-aware: if the
[`Validate`](`pointblank.Validate`) object was created with a non-English `locale=`, the note
will be translated accordingly.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Skip a validation step if the table is empty:

```{python}
import pointblank as pb
import polars as pl

tbl = pl.DataFrame({"x": [1, 2, 3]})
empty_tbl = pl.DataFrame({"x": []})

validation = (
    pb.Validate(data=empty_tbl)
    .col_vals_gt(columns="x", value=0, active=pb.has_rows())
    .interrogate()
)

validation
```

The step was skipped because the table has no rows.

Only run a step when the table has at least a minimum number of rows:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="x", value=0, active=pb.has_rows(min=100))
    .interrogate()
)

validation
```

The step was skipped because the table has only 3 rows, which is fewer than the required
minimum of `100`.

You can also check for an exact count or a range:

```{python}
validation = (
    pb.Validate(data=tbl)
    .col_vals_gt(columns="x", value=0, active=pb.has_rows(count=3))
    .col_vals_gt(columns="x", value=0, active=pb.has_rows(min=2, max=10))
    .col_vals_gt(columns="x", value=0, active=pb.has_rows(count=100))
    .interrogate()
)

validation
```

The first two steps ran because the table has exactly 3 rows (matching `count=3`) and falls
within the range `[2, 10]`. The third step was skipped because `3` does not equal `100`.


## YAML

Functions for using YAML to orchestrate validation workflows.



yaml_interrogate(yaml: 'Union[str, Path]', set_tbl: 'Any' = None, namespaces: 'Optional[Union[Iterable[str], Mapping[str, str]]]' = None, allow_python: 'bool' = False) -> 'Validate'

Execute a YAML-based validation workflow.

This is the main entry point for YAML-based validation workflows. It takes YAML configuration
(as a string or file path) and returns a validated `Validate` object with interrogation results.

The YAML configuration defines the data source, validation steps, and optional settings like
thresholds and labels. This function automatically loads the data, builds the validation plan,
executes all validation steps, and returns the interrogated results.

Parameters
----------
yaml
    YAML configuration as string or file path. Can be: (1) a YAML string containing the
    validation configuration, or (2) a Path object or string path to a YAML file.
set_tbl
    An optional table to override the table specified in the YAML configuration. This allows you
    to apply a YAML-defined validation workflow to a different table than what's specified in
    the configuration. If provided, this table is used in place of the table defined in the
    YAML's `tbl` field (which is then not loaded, so it can be a placeholder or a source
    that isn't available). This can be any supported table type
    including DataFrame objects, Ibis table objects, CSV file paths, Parquet file paths, GitHub
    URLs, or database connection strings.
namespaces
    Optional module namespaces to make available for Python code execution in YAML
    configurations. Can be a dictionary mapping aliases to module names or a list of module
    names. This only has an effect with `allow_python=True`. See the "Using Namespaces"
    section below for detailed examples.
allow_python
    Whether to run Python code in the YAML configuration (`python:` blocks, and the `expr:`,
    `pre:`, and `active:` shortcuts). This is `False` by default, which makes a configuration
    with Python code raise a `YAMLValidationError`. Python code in YAML runs with the full
    access of your Python session (it can read and write files or run commands), so only set
    this to `True` for configurations you trust. A few `pb` functions called with literal
    arguments (e.g., `pb.col("b")` or `pb.has_columns("a")`) don't need it. See the "Python
    Code in YAML" section below.

Returns
-------
Validate
    An instance of the `Validate` class that has been configured based on the YAML input. This
    object contains the results of the validation steps defined in the YAML configuration. It
    includes metadata like table name, label, language, and thresholds if specified.

Raises
------
YAMLValidationError
    If the YAML is invalid, malformed, or execution fails. This includes syntax errors, missing
    required fields, unknown validation methods, data loading failures, or Python code in a
    configuration used without `allow_python=True`.

Python Code in YAML
-------------------
A YAML configuration can include Python code: a `python:` block for the data source, a
parameter value, or an action, and the `expr:`, `pre:`, and `active:` shortcuts for code
written as a plain string. This code isn't run unless you set `allow_python=True`.

Python code in YAML is not sandboxed. It runs with the same access as the rest of your Python
session, so a configuration with Python code should be treated like a Python script: only allow
it for configurations you wrote or otherwise trust, never for ones supplied by other users or
systems.

A small set of `pb` functions can always be used, when called with literal arguments (strings,
numbers, lists, and so on): [`load_dataset()`](`pointblank.load_dataset`),
[`col()`](`pointblank.col`), [`ref()`](`pointblank.ref`),
[`seg_group()`](`pointblank.seg_group`), [`has_columns()`](`pointblank.has_columns`),
[`has_rows()`](`pointblank.has_rows`), and the column selectors (`starts_with()`, `ends_with()`,
`contains()`, `matches()`, `everything()`, `first_n()`, and `last_n()`). So can the
constructors for values that YAML has no literal for: `Decimal()`, `ZoneInfo()`,
`datetime.date()`, `datetime.time()`, `datetime.datetime()`, and `datetime.timedelta()`. These
only load a bundled dataset or build an object, so `value: pb.col("b")`,
`value: {python: 'Decimal("2.2")'}`, or `active: pb.has_columns("a")` work without
`allow_python=True`. In code that is allowed to run, `datetime`, `Decimal`, and `ZoneInfo` are
available along with `pb`, `pl`, `pd`, and `Path`.

Using Namespaces
----------------
The `namespaces=` parameter enables custom Python modules and functions in YAML configurations.
This is particularly useful for custom action functions and advanced Python expressions. Since
these are Python code, both options below need `allow_python=True`.

**Namespace formats:**

- Dictionary format: `{"alias": "module.name"}` maps aliases to module names
- List format: `["module.name", "another.module"]` imports modules directly

**Option 1: Inline expressions (no namespaces needed)**

```{python}
import pointblank as pb

# Simple inline custom action
yaml_config = '''
tbl: small_table
thresholds:
  warning: 0.01
actions:
  warning:
    python: "lambda: print('Custom warning triggered')"
steps:
- col_vals_gt:
    columns: [a]
    value: 1000
'''

result = pb.yaml_interrogate(yaml_config, allow_python=True)
result
```

**Option 2: External functions with namespaces**

```{python}
# Define a custom action function
def my_custom_action():
    print("Data validation failed: please check your data.")

# Add to current module for demo
import sys
sys.modules[__name__].my_custom_action = my_custom_action

# YAML that references the external function
yaml_config = '''
tbl: small_table
thresholds:
  warning: 0.01
actions:
  warning:
    python: actions.my_custom_action
steps:
- col_vals_gt:
    columns: [a]
    value: 1000  # This will fail
'''

# Use namespaces to make the function available
result = pb.yaml_interrogate(
    yaml_config, namespaces={'actions': '__main__'}, allow_python=True
)
result
```

This approach enables modular, reusable validation workflows with custom business logic.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll use YAML configurations to define validation workflows. Let's start
with a basic YAML workflow that validates the built-in `small_table` dataset.

```{python}
import pointblank as pb

# Define a basic YAML validation workflow
yaml_config = '''
tbl: small_table
steps:
- rows_distinct
- col_exists:
    columns: [date, a, b]
'''

# Execute the validation workflow
result = pb.yaml_interrogate(yaml_config)
result
```

The validation table shows the results of our YAML-defined workflow. We can see that the
`rows_distinct()` validation failed (because there are duplicate rows in the table), while the
column existence checks passed.

Now let's create a more comprehensive validation workflow with thresholds and metadata:

```{python}
# Advanced YAML configuration with thresholds and metadata
yaml_config = '''
tbl: small_table
tbl_name: small_table_demo
label: Comprehensive data validation
thresholds:
  warning: 0.1
  error: 0.25
  critical: 0.35
steps:
- col_vals_gt:
    columns: [d]
    value: 100
- col_vals_regex:
    columns: [b]
    pattern: '[0-9]-[a-z]{3}-[0-9]{3}'
- col_vals_not_null:
    columns: [date, a]
'''

# Execute the validation workflow
result = pb.yaml_interrogate(yaml_config)
print(f"Table name: {result.tbl_name}")
print(f"Label: {result.label}")
print(f"Total validation steps: {len(result.validation_info)}")
```

The validation results now include our custom table name and label. The thresholds we defined
will determine when validation steps are marked as warnings, errors, or critical failures.

You can also load YAML configurations from files. Here's how you would work with a YAML file:

```{python}
from pathlib import Path
import tempfile

# Create a temporary YAML file for demonstration
yaml_content = '''
tbl: small_table
tbl_name: File-based Validation
steps:
- col_vals_between:
    columns: [c]
    left: 1
    right: 10
- col_vals_in_set:
    columns: [f]
    set: [low, mid, high]
'''

with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
    f.write(yaml_content)
    yaml_file_path = Path(f.name)

# Load and execute validation from file
result = pb.yaml_interrogate(yaml_file_path)
result
```

This approach is particularly useful for storing validation configurations as part of your data
pipeline or version control system, allowing you to maintain validation rules alongside your
code.

### Governance Metadata

YAML workflows support governance metadata via `owner`, `consumers`, and `version` top-level
keys. These are forwarded to the `Validate` constructor and embedded in the validation report:

```{python}
yaml_config = '''
tbl: small_table
tbl_name: sales_pipeline
owner: Data Engineering
consumers: [Analytics, Finance, Compliance]
version: "2.1.0"
steps:
- col_vals_not_null:
    columns: [a, b]
'''

result = pb.yaml_interrogate(yaml_config)
print(f"Owner: {result.owner}")
print(f"Consumers: {result.consumers}")
print(f"Version: {result.version}")
```

### Aggregate Validations

YAML supports aggregate validation methods for checking column-level statistics. These methods
validate that a column's sum, average, or standard deviation meets a threshold:

```{python}
yaml_config = '''
tbl: small_table
steps:
- col_sum_gt:
    columns: [d]
    value: 0
- col_avg_le:
    columns: [a]
    value: 10
'''

result = pb.yaml_interrogate(yaml_config)
result
```

The 15 available aggregate methods follow the pattern `col_{stat}_{comparator}` where
`{stat}` is `sum`, `avg`, or `sd` and `{comparator}` is `gt`, `lt`, `ge`,
`le`, or `eq`.

### Data Freshness

Check that a date/datetime column has recent data using `data_freshness`:

```yaml
tbl: events.csv
steps:
- data_freshness:
    column: event_date
    max_age: "24h"
```

### Active Parameter Shortcut

The `active=` parameter controls whether a validation step runs. It supports boolean values
and Python expression shortcuts:

```yaml
steps:
- col_vals_gt:
    columns: [d]
    value: 100
    active: false            # Skip this step

- col_vals_not_null:
    columns: [a]
    active: true             # Always run (default)

- col_vals_not_null:
    columns: [e]
    active: pb.has_columns("e")  # Run only if column `e` exists
```

### Null Percentage Check

Use `col_pct_null` to validate that the percentage of null values in a column is within bounds:

```yaml
steps:
- col_pct_null:
    columns: [a, b]
    value: 0.05
```

### Using `set_tbl=` to Override the Table

The `set_tbl=` parameter allows you to override the table specified in the YAML configuration.
This is useful when you have a template validation workflow but want to apply it to different
tables:

```{python}
import polars as pl

# Create a test table with similar structure to small_table
test_table = pl.DataFrame({
    "date": ["2023-01-01", "2023-01-02", "2023-01-03"],
    "a": [1, 2, 3],
    "b": ["1-abc-123", "2-def-456", "3-ghi-789"],
    "d": [150, 200, 250]
})

# Use the same YAML config but apply it to our test table
yaml_config = '''
tbl: small_table  # This will be overridden
tbl_name: Test Table  # This name will be used
steps:
- col_exists:
    columns: [date, a, b, d]
- col_vals_gt:
    columns: [d]
    value: 100
'''

# Execute with table override
result = pb.yaml_interrogate(yaml_config, set_tbl=test_table)
print(f"Validation applied to: {result.tbl_name}")
result
```

This feature makes YAML configurations more reusable and flexible, allowing you to define
validation logic once and apply it to multiple similar tables.

validate_yaml(yaml: 'Union[str, Path]') -> 'None'

Validate YAML configuration against the expected structure.

This function validates that a YAML configuration conforms to the expected structure for
validation workflows. It checks for required fields, proper data types, and valid
validation method names. This is useful for validating configurations before execution or
for building configuration editors and validators.

The function performs comprehensive validation including:

- required fields ('tbl' and 'steps')
- proper data types for all fields
- valid threshold configurations
- known validation method names
- proper step configuration structure

Parameters
----------
yaml
    YAML configuration as string or file path. Can be: (1) a YAML string containing the
    validation configuration, or (2) a Path object or string path to a YAML file.

Raises
------
YAMLValidationError
    If the YAML is invalid, malformed, or execution fails. This includes syntax errors,
    missing required fields, unknown validation methods, or data loading failures.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
For the examples here, we'll demonstrate how to validate YAML configurations before using them
with validation workflows. This is particularly useful for building robust data validation
systems where you want to catch configuration errors early.

Let's start with validating a basic configuration:

```{python}
import pointblank as pb

# Define a basic YAML validation configuration
yaml_config = '''
tbl: small_table
steps:
- rows_distinct
- col_exists:
    columns: [a, b]
'''

# Validate the configuration: no exception means it's valid
pb.validate_yaml(yaml_config)
print("Basic YAML configuration is valid")
```

The function completed without raising an exception, which means our configuration is valid and
follows the expected structure.

Now let's validate a more complex configuration with thresholds and metadata:

```{python}
# Complex YAML configuration with all optional fields
yaml_config = '''
tbl: small_table
tbl_name: My Dataset
label: Quality check
lang: en
locale: en
thresholds:
  warning: 0.1
  error: 0.25
  critical: 0.35
steps:
- rows_distinct
- col_vals_gt:
    columns: [d]
    value: 100
- col_vals_regex:
    columns: [b]
    pattern: '[0-9]-[a-z]{3}-[0-9]{3}'
'''

# Validate the configuration
pb.validate_yaml(yaml_config)
print("Complex YAML configuration is valid")

# Count the validation steps
import pointblank.yaml as pby
config = pby.load_yaml_config(yaml_config)
print(f"Configuration has {len(config['steps'])} validation steps")
```

This configuration includes all the optional metadata fields and complex validation steps,
demonstrating that the validation handles the full range of supported options.

Let's see what happens when we try to validate an invalid configuration:

```{python}
# Invalid YAML configuration: missing required 'tbl' field
invalid_yaml = '''
steps:
- rows_distinct
'''

try:
    pb.validate_yaml(invalid_yaml)
except pb.yaml.YAMLValidationError as e:
    print(f"Validation failed: {e}")
```

The validation correctly identifies that our configuration is missing the required `'tbl'`
field.

Here's a practical example of using validation in a workflow builder:

```{python}
def safe_yaml_interrogate(yaml_config):
    """Safely execute a YAML configuration after validation."""
    try:
        # Validate the YAML configuration first
        pb.validate_yaml(yaml_config)
        print("✓ YAML configuration is valid")

        # Then execute the workflow
        result = pb.yaml_interrogate(yaml_config)
        print(f"Validation completed with {len(result.validation_info)} steps")
        return result

    except pb.yaml.YAMLValidationError as e:
        print(f"Configuration error: {e}")
        return None

# Test with a valid YAML configuration
test_yaml = '''
tbl: small_table
steps:
- col_vals_between:
    columns: [c]
    left: 1
    right: 10
'''

result = safe_yaml_interrogate(test_yaml)
```

This pattern of validating before executing helps build more reliable data validation pipelines
by catching configuration errors early in the process.

Note that this function only validates the structure and does not check if the specified data
source ('tbl') exists or is accessible. Data source validation occurs during execution with
`yaml_interrogate()`.

Supported Top-level Keys
------------------------
The following top-level keys are recognized in the YAML configuration:

- `tbl`: data source specification (required)
- `steps`: list of validation steps (required)
- `tbl_name`: human-readable table name
- `label`: validation description
- `df_library`: DataFrame library (`"polars"`, `"pandas"`, `"duckdb"`)
- `lang`: language code
- `locale`: locale setting
- `brief`: global brief template
- `thresholds`: global failure thresholds
- `actions`: global failure actions
- `final_actions`: actions triggered after all steps complete
- `owner`: data owner (governance metadata)
- `consumers`: data consumers (governance metadata)
- `version`: validation version string (governance metadata)
- `reference`: reference table for comparison-based validations

Unknown top-level keys are rejected, which catches typos like `tbl_nmae` or `step`.

Supported Validation Methods
----------------------------
In addition to all standard validation methods (e.g., `col_vals_gt`, `rows_distinct`,
`col_schema_match`), the following methods are also supported:

- `col_pct_null`: check the percentage of null values in a column
- `data_freshness`: check that data is recent
- aggregate methods: `col_sum_gt`, `col_sum_lt`, `col_sum_ge`, `col_sum_le`,
  `col_sum_eq`, `col_avg_gt`, `col_avg_lt`, `col_avg_ge`, `col_avg_le`,
  `col_avg_eq`, `col_sd_gt`, `col_sd_lt`, `col_sd_ge`, `col_sd_le`, `col_sd_eq`

See Also
--------
yaml_interrogate : execute YAML-based validation workflows

yaml_to_python(yaml: 'Union[str, Path]') -> 'str'

Convert YAML validation configuration to equivalent Python code.

This function takes a YAML validation configuration and generates the equivalent Python code
that would produce the same validation workflow. This is useful for documentation, code
generation, or learning how to translate YAML workflows into programmatic workflows.

The generated Python code includes all necessary imports, data loading, validation steps,
and interrogation execution, formatted as executable Python code.

Parameters
----------
yaml
    YAML configuration as string or file path. Can be: (1) a YAML string containing the
    validation configuration, or (2) a Path object or string path to a YAML file.

Returns
-------
str
    A formatted Python code string enclosed in markdown code blocks that replicates the YAML
    workflow. The code includes import statements, data loading, validation method calls, and
    interrogation execution.

Raises
------
YAMLValidationError
    If the YAML is invalid, malformed, or contains unknown validation methods.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Convert a basic YAML configuration to Python code:

```{python}
import pointblank as pb

# Define a YAML validation workflow
yaml_config = '''
tbl: small_table
tbl_name: Data Quality Check
steps:
- col_vals_not_null:
    columns: [a, b]
- col_vals_gt:
    columns: [c]
    value: 0
'''

# Generate equivalent Python code
python_code = pb.yaml_to_python(yaml_config)
print(python_code)
```

The generated Python code shows exactly how to replicate the YAML workflow programmatically.
This is particularly useful when transitioning from YAML-based workflows to code-based
workflows, or when generating documentation that shows both YAML and Python approaches.

For more complex workflows with thresholds and metadata:

```{python}
# Advanced YAML configuration
yaml_config = '''
tbl: small_table
tbl_name: Advanced Validation
label: Production data check
thresholds:
  warning: 0.1
  error: 0.2
steps:
- col_vals_between:
    columns: [c]
    left: 1
    right: 10
- col_vals_regex:
    columns: [b]
    pattern: '[0-9]-[a-z]{3}-[0-9]{3}'
'''

# Generate the equivalent Python code
python_code = pb.yaml_to_python(yaml_config)
print(python_code)
```

The generated code includes all configuration parameters, thresholds, and maintains the exact
same validation logic as the original YAML workflow.

The `tbl` and `reference` data sources are loaded as `yaml_interrogate()` loads them: a
dataset name becomes a `pb.load_dataset()` call, a CSV file is read with the `df_library`
library (e.g., `pl.read_csv()`), and other file paths and connection strings are passed to
`Validate` as strings.

Python code in the configuration (`python:` blocks, and the `expr:`, `pre:`, and `active:`
shortcuts) is copied into the generated code as written, without being run. So, unlike
[`yaml_interrogate()`](`pointblank.yaml_interrogate`), this function doesn't need
`allow_python=True`, though you should still read the generated code before running it.

Governance metadata (`owner`, `consumers`, `version`) and `reference` are also rendered
in the generated Python code:

```{python}
yaml_config = '''
tbl: small_table
tbl_name: Sales Pipeline
owner: Data Engineering
consumers: [Analytics, Finance]
version: "2.1.0"
steps:
- col_vals_not_null:
    columns: [a]
- col_sum_gt:
    columns: [d]
    value: 0
'''

python_code = pb.yaml_to_python(yaml_config)
print(python_code)
```

This function is also useful for educational purposes, helping users understand how YAML
configurations map to the underlying Python API calls.


## Utility Functions

Functions for accessing metadata about the target data and managing configuration.



get_column_count(data: 'Any') -> 'int'

Get the number of columns in a table.

The `get_column_count()` function returns the number of columns in a table. The function works
with any table that is supported by the `pointblank` library, including Pandas, Polars, and Ibis
backend tables (e.g., DuckDB, MySQL, PostgreSQL, SQLite, Parquet, etc.). It also supports
direct input of CSV files, Parquet files, and database connection strings.

Parameters
----------
data
    The table for which to get the column count, which could be a DataFrame object, an Ibis
    table object, a CSV file path, a Parquet file path, or a database connection string.
    Read the *Supported Input Table Types* section for details on the supported table types.

Returns
-------
int
    The number of columns in the table.

Supported Input Table Types
---------------------------
The `data=` parameter can be given any of the following table types:

- Polars DataFrame (`"polars"`)
- Pandas DataFrame (`"pandas"`)
- PySpark table (`"pyspark"`)
- DuckDB table (`"duckdb"`)*
- MySQL table (`"mysql"`)*
- PostgreSQL table (`"postgresql"`)*
- SQLite table (`"sqlite"`)*
- Microsoft SQL Server table (`"mssql"`)*
- Snowflake table (`"snowflake"`)*
- Databricks table (`"databricks"`)*
- BigQuery table (`"bigquery"`)*
- Parquet table (`"parquet"`)*
- CSV files (string path or `pathlib.Path` object with `.csv` extension)
- Parquet files (string path, `pathlib.Path` object, glob pattern, directory with `.parquet`
extension, or partitioned dataset)
- Database connection strings (URI format with optional table specification)

The table types marked with an asterisk need to be prepared as Ibis tables (with type of
`ibis.expr.types.relations.Table`). Furthermore, using `get_column_count()` with these types of
tables requires the Ibis library (`v9.5.0` or above) to be installed. If the input table is a
Polars or Pandas DataFrame, the availability of Ibis is not needed.

To use a CSV file, ensure that a string or `pathlib.Path` object with a `.csv` extension is
provided. The file will be automatically detected and loaded using the best available DataFrame
library. The loading preference is Polars first, then Pandas as a fallback.

GitHub URLs pointing to CSV or Parquet files are automatically detected and converted to raw
content URLs for downloading. The URL format should be:
`https://github.com/user/repo/blob/branch/path/file.csv` or
`https://github.com/user/repo/blob/branch/path/file.parquet`

Connection strings follow database URL formats and must also specify a table using the
`::table_name` suffix. Examples include:

```
"duckdb:///path/to/database.ddb::table_name"
"sqlite:///path/to/database.db::table_name"
"postgresql://user:password@localhost:5432/database::table_name"
"mysql://user:password@localhost:3306/database::table_name"
"bigquery://project/dataset::table_name"
"snowflake://user:password@account/database/schema::table_name"
```

When using connection strings, the Ibis library with the appropriate backend driver is required.

Examples
--------
To get the number of columns in a table, we can use the `get_column_count()` function. Here's an
example using the `small_table` dataset (itself loaded using the
[`load_dataset()`](`pointblank.load_dataset`) function):

```{python}
import pointblank as pb

small_table_polars = pb.load_dataset("small_table")

pb.get_column_count(small_table_polars)
```

This table is a Polars DataFrame, but the `get_column_count()` function works with any table
supported by `pointblank`, including Pandas DataFrames and Ibis backend tables. Here's an
example using a DuckDB table handled by Ibis:

```{python}
small_table_duckdb = pb.load_dataset("small_table", tbl_type="duckdb")

pb.get_column_count(small_table_duckdb)
```

#### Working with CSV Files

The `get_column_count()` function can directly accept CSV file paths:

```{python}
# Get a path to a CSV file from the package data
csv_path = pb.get_data_path("global_sales", "csv")

pb.get_column_count(csv_path)
```

#### Working with Parquet Files

The function supports various Parquet input formats:

```{python}
# Single Parquet file from package data
parquet_path = pb.get_data_path("nycflights", "parquet")

pb.get_column_count(parquet_path)
```

You can also use glob patterns and directories:

```python
# Multiple Parquet files with glob patterns
pb.get_column_count("data/sales_*.parquet")

# Directory containing Parquet files
pb.get_column_count("parquet_data/")

# Partitioned Parquet dataset
pb.get_column_count("sales_data/")  # Auto-discovers partition columns
```

#### Working with Database Connection Strings

The function supports database connection strings for direct access to database tables:

```{python}
# Get path to a DuckDB database file from package data
duckdb_path = pb.get_data_path("game_revenue", "duckdb")

pb.get_column_count(f"duckdb:///{duckdb_path}::game_revenue")
```

The function always returns the number of columns in the table as an integer value, which is
`8` for the `small_table` dataset.

get_row_count(data: 'Any') -> 'int'

Get the number of rows in a table.

The `get_row_count()` function returns the number of rows in a table. The function works with
any table that is supported by the `pointblank` library, including Pandas, Polars, and Ibis
backend tables (e.g., DuckDB, MySQL, PostgreSQL, SQLite, Parquet, etc.). It also supports
direct input of CSV files, Parquet files, and database connection strings.

Parameters
----------
data
    The table for which to get the row count, which could be a DataFrame object, an Ibis table
    object, a CSV file path, a Parquet file path, or a database connection string.
    Read the *Supported Input Table Types* section for details on the supported table types.

Returns
-------
int
    The number of rows in the table.

Supported Input Table Types
---------------------------
The `data=` parameter can be given any of the following table types:

- Polars DataFrame (`"polars"`)
- Pandas DataFrame (`"pandas"`)
- PySpark table (`"pyspark"`)
- DuckDB table (`"duckdb"`)*
- MySQL table (`"mysql"`)*
- PostgreSQL table (`"postgresql"`)*
- SQLite table (`"sqlite"`)*
- Microsoft SQL Server table (`"mssql"`)*
- Snowflake table (`"snowflake"`)*
- Databricks table (`"databricks"`)*
- BigQuery table (`"bigquery"`)*
- Parquet table (`"parquet"`)*
- CSV files (string path or `pathlib.Path` object with `.csv` extension)
- Parquet files (string path, `pathlib.Path` object, glob pattern, directory with `.parquet`
extension, or partitioned dataset)
- GitHub URLs (direct links to CSV or Parquet files on GitHub)
- Database connection strings (URI format with optional table specification)

The table types marked with an asterisk need to be prepared as Ibis tables (with type of
`ibis.expr.types.relations.Table`). Furthermore, using `get_row_count()` with these types of
tables requires the Ibis library (`v9.5.0` or above) to be installed. If the input table is a
Polars or Pandas DataFrame, the availability of Ibis is not needed.

To use a CSV file, ensure that a string or `pathlib.Path` object with a `.csv` extension is
provided. The file will be automatically detected and loaded using the best available DataFrame
library. The loading preference is Polars first, then Pandas as a fallback.

GitHub URLs pointing to CSV or Parquet files are automatically detected and converted to raw
content URLs for downloading. The URL format should be:
`https://github.com/user/repo/blob/branch/path/file.csv` or
`https://github.com/user/repo/blob/branch/path/file.parquet`

Connection strings follow database URL formats and must also specify a table using the
`::table_name` suffix. Examples include:

```
"duckdb:///path/to/database.ddb::table_name"
"sqlite:///path/to/database.db::table_name"
"postgresql://user:password@localhost:5432/database::table_name"
"mysql://user:password@localhost:3306/database::table_name"
"bigquery://project/dataset::table_name"
"snowflake://user:password@account/database/schema::table_name"
```

When using connection strings, the Ibis library with the appropriate backend driver is required.

Examples
--------
Getting the number of rows in a table is easily done by using the `get_row_count()` function.
Here's an example using the `game_revenue` dataset (itself loaded using the
[`load_dataset()`](`pointblank.load_dataset`) function):

```{python}
import pointblank as pb

game_revenue_polars = pb.load_dataset("game_revenue")

pb.get_row_count(game_revenue_polars)
```

This table is a Polars DataFrame, but the `get_row_count()` function works with any table
supported by `pointblank`, including Pandas DataFrames and Ibis backend tables. Here's an
example using a DuckDB table handled by Ibis:

```{python}
game_revenue_duckdb = pb.load_dataset("game_revenue", tbl_type="duckdb")

pb.get_row_count(game_revenue_duckdb)
```

#### Working with CSV Files

The `get_row_count()` function can directly accept CSV file paths:

```{python}
# Get a path to a CSV file from the package data
csv_path = pb.get_data_path("global_sales", "csv")

pb.get_row_count(csv_path)
```

#### Working with Parquet Files

The function supports various Parquet input formats:

```{python}
# Single Parquet file from package data
parquet_path = pb.get_data_path("nycflights", "parquet")

pb.get_row_count(parquet_path)
```

You can also use glob patterns and directories:

```python
# Multiple Parquet files with glob patterns
pb.get_row_count("data/sales_*.parquet")

# Directory containing Parquet files
pb.get_row_count("parquet_data/")

# Partitioned Parquet dataset
pb.get_row_count("sales_data/")  # Auto-discovers partition columns
```

#### Working with Database Connection Strings

The function supports database connection strings for direct access to database tables:

```{python}
# Get path to a DuckDB database file from package data
duckdb_path = pb.get_data_path("game_revenue", "duckdb")

pb.get_row_count(f"duckdb:///{duckdb_path}::game_revenue")
```

The function always returns the number of rows in the table as an integer value, which is `2000`
for the `game_revenue` dataset.

get_action_metadata() -> 'dict | None'

Access step-level metadata when authoring custom actions.

Get the metadata for the validation step where an action was triggered. This can be called by
user functions to get the metadata for the current action. This function can only be used within
callables crafted for the [`Actions`](`pointblank.Actions`) class.

Returns
-------
dict | None
    A dictionary containing the metadata for the current step. If called outside of an action
    (i.e., when no action is being executed), this function will return `None`.

Description of the Metadata Fields
----------------------------------
The metadata dictionary contains the following fields for a given validation step:

- `step`: The step number.
- `column`: The column name.
- `value`: The value being compared (only available in certain validation steps).
- `type`: The assertion type (e.g., `"col_vals_gt"`, etc.).
- `time`: The time the validation step was executed (in ISO format).
- `level`: The severity level (`"warning"`, `"error"`, or `"critical"`).
- `level_num`: The severity level as a numeric value (`30`, `40`, or `50`).
- `autobrief`: A localized and brief statement of the expectation for the step.
- `failure_text`: Localized text that explains how the validation step failed.

Examples
--------
When creating a custom action, you can access the metadata for the current step using the
`get_action_metadata()` function. Here's an example of a custom action that logs the metadata
for the current step:

```{python}
import pointblank as pb

def log_issue():
    metadata = pb.get_action_metadata()
    print(f"Type: {metadata['type']}, Step: {metadata['step']}")

validation = (
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),
        actions=pb.Actions(warning=log_issue),
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}[0-9]{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .col_vals_gt(
        columns="session_duration",
        value=15,
    )
    .interrogate()
)

validation
```

Key pieces to note in the above example:

- `log_issue()` (the custom action) collects `metadata` by calling `get_action_metadata()`
- the `metadata` is a dictionary that is used to craft the log message
- the action is passed as a bare function to the `Actions` object within the `Validate` object
(placing it within `Validate(actions=)` ensures it's set as an action for every validation step)

See Also
--------
Actions : Create custom actions for steps that exceed a threshold.

get_validation_summary() -> 'dict | None'

Access validation summary information when authoring final actions.

This function provides a convenient way to access summary information about the validation
process within a final action. It returns a dictionary with key metrics from the validation
process. This function can only be used within callables crafted for the
[`FinalActions`](`pointblank.FinalActions`) class.

Returns
-------
dict | None
    A dictionary containing validation metrics. If called outside of an final action context,
    this function will return `None`.

Description of the Summary Fields
--------------------------------
The summary dictionary contains the following fields:

- `n_steps` (`int`): The total number of validation steps.
- `n_passing_steps` (`int`): The number of validation steps where all test units passed.
- `n_failing_steps` (`int`): The number of validation steps that had some failing test units.
- `n_warning_steps` (`int`): The number of steps that exceeded a 'warning' threshold.
- `n_error_steps` (`int`): The number of steps that exceeded an 'error' threshold.
- `n_critical_steps` (`int`): The number of steps that exceeded a 'critical' threshold.
- `list_passing_steps` (`list[int]`): List of step numbers where all test units passed.
- `list_failing_steps` (`list[int]`): List of step numbers for steps having failing test units.
- `dict_n` (`dict`): The number of test units for each validation step.
- `dict_n_passed` (`dict`): The number of test units that passed for each validation step.
- `dict_n_failed` (`dict`): The number of test units that failed for each validation step.
- `dict_f_passed` (`dict`): The fraction of test units that passed for each validation step.
- `dict_f_failed` (`dict`): The fraction of test units that failed for each validation step.
- `dict_warning` (`dict`): The 'warning' level status for each validation step.
- `dict_error` (`dict`): The 'error' level status for each validation step.
- `dict_critical` (`dict`): The 'critical' level status for each validation step.
- `all_passed` (`bool`): Whether or not every validation step had no failing test units.
- `highest_severity` (`str`): The highest severity level encountered during validation. This can
  be one of the following: `"warning"`, `"error"`, or `"critical"`, `"some failing"`, or
  `"all passed"`.
- `tbl_row_count` (`int`): The number of rows in the target table.
- `tbl_column_count` (`int`): The number of columns in the target table.
- `tbl_name` (`str`): The name of the target table.
- `validation_duration` (`float`): The duration of the validation in seconds.
- `dimension_scores` (`dict`): The test-unit-weighted health score (`0`-`100`) for each data
  quality dimension present in the validation (e.g., `{"completeness": 99.2, "validity": 97.8}`).
- `overall_health_score` (`float`): The overall, test-unit-weighted health score (`0`-`100`)
  across all dimensions.

Note that the summary dictionary is only available within the context of a final action. If
called outside of a final action (i.e., when no final action is being executed), this function
will return `None`.

Examples
--------
Final actions are executed after the completion of all validation steps. They provide an
opportunity to take appropriate actions based on the overall validation results. Here's an
example of a final action function (`send_report()`) that sends an alert when critical
validation failures are detected. To keep the example self-contained, the alert is printed;
in practice, this is where you'd send an email or post a message to a chat service:

```{python}
import pointblank as pb

def send_report():
    summary = pb.get_validation_summary()
    if summary["highest_severity"] == "critical":
        print(f"CRITICAL validation failures in {summary['tbl_name']}")
        print(f"Steps with critical severity: {summary['n_critical_steps']}")

validation = (
    pb.Validate(
        data=pb.load_dataset("small_table"),
        tbl_name="small_table",
        thresholds=(1, 2, 3),
        final_actions=pb.FinalActions(send_report),
    )
    .col_vals_gt(columns="d", value=1000)
    .col_vals_not_null(columns="c")
    .interrogate()
)
```

The first step has six failing test units, which reaches the 'critical' threshold of three,
so the alert was printed during `interrogate()`. The second step's two missing values only
reach the 'error' threshold. Outside of a final action, `get_validation_summary()` returns
`None`.

The `get_validation_summary()` function can also be used to create custom reporting for
validation results:

```{python}
def log_validation_results():
    summary = pb.get_validation_summary()

    print(f"Validation completed with status: {summary['highest_severity'].upper()}")
    print(f"Steps: {summary['n_steps']} total")
    print(f"  - {summary['n_passing_steps']} passing, {summary['n_failing_steps']} failing")
    print(
        f"  - Severity: {summary['n_warning_steps']} warnings, "
        f"{summary['n_error_steps']} errors, "
        f"{summary['n_critical_steps']} critical"
    )

    if summary["highest_severity"] in ["error", "critical"]:
        print("Action required: Please review failing validation steps!")

validation = (
    pb.Validate(
        data=pb.load_dataset("small_table"),
        tbl_name="small_table",
        thresholds=(1, 2, 3),
        final_actions=pb.FinalActions(log_validation_results),
    )
    .col_vals_gt(columns="d", value=100)
    .col_vals_not_null(columns="c")
    .col_vals_le(columns="a", value=7)
    .interrogate()
)
```

The first step passes, the second step's two missing values reach the 'error' threshold, and
the third step's single failure reaches the 'warning' threshold. A step that reaches a higher
threshold also counts toward the lower ones, which is why the severity counts can add up to
more than the number of failing steps. Final actions work well with both simple logging and
more complex notification systems, allowing you to integrate validation results into your
broader data quality workflows.

See Also
--------
FinalActions : Create custom actions that run after all validation steps are done.

write_file(validation: 'Validate', filename: 'str', path: 'str | None' = None, keep_tbl: 'bool' = False, keep_extracts: 'bool' = False, quiet: 'bool' = False) -> 'None'

Write a Validate object to disk as a serialized file.

Writing a validation object to disk with `write_file()` can be useful for keeping data
validation results close at hand for later retrieval (with `read_file()`). By default, any data
table that the validation object holds will be removed before writing to disk (not applicable if
no data table is present). This behavior can be changed by setting `keep_tbl=True`, but this
only works when the table is not of a database type (e.g., DuckDB, PostgreSQL, etc.), as
database connections cannot be serialized.

Extract data from failing validation steps can also be preserved by setting
`keep_extracts=True`, which is useful for later analysis of data quality issues.

The serialized file uses Python's pickle format for storage of the validation object state,
including all validation results, metadata, and optionally the source data.

**Important note.** If your validation uses custom preprocessing functions (via the `pre=`
parameter), these functions must be defined at the module level (not interactively or as lambda
functions) to ensure they can be properly restored when loading the validation in a different
Python session. Read the *Creating Serializable Validations* section below for more information.

:::{.callout-warning}
The `write_file()` function is currently experimental. Please report any issues you encounter in
the [Pointblank issue tracker](https://github.com/posit-dev/pointblank/issues).
:::

Parameters
----------
validation
    The `Validate` object to write to disk.
filename
    The filename to create on disk for the validation object. Should not include the file
    extension as `.pkl` will be added automatically.
path
    An optional directory path where the file should be saved. If not provided, the file will be
    saved in the current working directory. The directory will be created if it doesn't exist.
keep_tbl
    An option to keep the data table that is associated with the validation object. The default
    is `False` where the data table is removed before writing to disk. For database tables
    (e.g., Ibis tables with database backends), the table is always removed even if
    `keep_tbl=True`, as database connections cannot be serialized.
keep_extracts
    An option to keep any collected extract data for failing rows from validation steps. By
    default, this is `False` (i.e., extract data is removed to save space).
quiet
    Should the function not inform when the file is written? By default, this is `False`, so a
    message will be printed when the file is successfully written.

Returns
-------
None
    This function doesn't return anything but saves the validation object to disk.

Creating Serializable Validations
---------------------------------
To ensure your validations work reliably across different Python sessions, the recommended
approach is to use module-Level functions. So, create a separate Python file for your
preprocessing functions:

```python
# preprocessing_functions.py
import polars as pl

def multiply_by_100(df):
    return df.with_columns(pl.col("value") * 100)

def add_computed_column(df):
    return df.with_columns(computed=pl.col("value") * 2 + 10)
```

Then import and use them in your validation:

```python
# your_main_script.py
import pointblank as pb
from preprocessing_functions import multiply_by_100, add_computed_column

validation = (
    pb.Validate(data=my_data)
    .col_vals_gt(columns="value", value=500, pre=multiply_by_100)
    .col_vals_between(columns="computed", left=50, right=1000, pre=add_computed_column)
    .interrogate()
)

# Save validation and it will work reliably across sessions
pb.write_file(validation, "my_validation", keep_tbl=True)
```

### Problematic Patterns to Avoid

Don't use lambda functions as they will cause immediate errors.

```python
validation = pb.Validate(data).col_vals_gt(
    columns="value", value=100,
    pre=lambda df: df.with_columns(pl.col("value") * 2)
)
```

Don't use interactive function definitions (as they may fail when loading).

```python
def my_function(df):  # Defined in notebook/REPL
    return df.with_columns(pl.col("value") * 2)

validation = pb.Validate(data).col_vals_gt(
    columns="value", value=100, pre=my_function
)
```

### Automatic Analysis and Guidance

When you call `write_file()`, it automatically analyzes your validation and provides:

- confirmation when all functions will work reliably
- warnings for functions that may cause cross-session issues
- clear errors for unsupported patterns (lambda functions)
- specific recommendations and code examples
- loading instructions tailored to your validation

### Loading Your Validation

To load a saved validation in a new Python session:

```python
# In a new Python session
import pointblank as pb

# Import the same preprocessing functions used when creating the validation
from preprocessing_functions import multiply_by_100, add_computed_column

# Upon loading the validation, functions will be automatically restored
validation = pb.read_file("my_validation.pkl")
```

** Testing Your Validation:**

To verify your validation works across sessions:

1. save your validation in one Python session
2. start a fresh Python session (restart kernel/interpreter)
3. import required preprocessing functions
4. load the validation using `read_file()`
5. test that preprocessing functions work as expected

### Performance and Storage

- use `keep_tbl=False` (default) to reduce file size when you don't need the original data
- use `keep_extracts=False` (default) to save space by excluding extract data
- set `quiet=True` to suppress guidance messages in automated scripts
- files are saved using pickle's highest protocol for optimal performance

Examples
--------
Let's create a simple validation and save it to disk:

```{python}
import pointblank as pb

# Create a validation
validation = (
    pb.Validate(data=pb.load_dataset("small_table"), label="My validation")
    .col_vals_gt(columns="d", value=100)
    .col_vals_regex(columns="b", pattern=r"[0-9]-[a-z]{3}-[0-9]{3}")
    .interrogate()
)

# Save to disk (without the original table data)
pb.write_file(validation, "my_validation")
```

To keep the original table data for later analysis:

```{python}
# Save with the original table data included
pb.write_file(validation, "my_validation_with_data", keep_tbl=True)
```

You can also specify a custom directory and keep extract data:

```python
pb.write_file(
    validation,
    filename="detailed_validation",
    path="/path/to/validations",
    keep_tbl=True,
    keep_extracts=True
)
```

### Working with Preprocessing Functions

For validations that use preprocessing functions to be portable across sessions, define your
functions in a separate `.py` file:

```python
# In `preprocessing_functions.py`

import polars as pl

def multiply_by_100(df):
    return df.with_columns(pl.col("value") * 100)

def add_computed_column(df):
    return df.with_columns(computed=pl.col("value") * 2 + 10)
```

Then import and use them in your validation:

```python
# In your main script

import pointblank as pb
from preprocessing_functions import multiply_by_100, add_computed_column

validation = (
    pb.Validate(data=my_data)
    .col_vals_gt(columns="value", value=500, pre=multiply_by_100)
    .col_vals_between(columns="computed", left=50, right=1000, pre=add_computed_column)
    .interrogate()
)

# This validation can now be saved and loaded reliably
pb.write_file(validation, "my_validation", keep_tbl=True)
```

When you load this validation in a new session, simply import the preprocessing functions
again and they will be automatically restored.

See Also
--------
read_file : Load a validation object saved with `write_file()`.

read_file(filepath: 'str | Path') -> 'Validate'

Read a Validate object from disk that was previously saved with `write_file()`.

This function loads a validation object that was previously serialized to disk using the
`write_file()` function. The validation object will be restored with all its validation results,
metadata, and optionally the source data (if it was saved with `keep_tbl=True`).

:::{.callout-warning}
The `read_file()` function is currently experimental. Please report any issues you encounter in
the [Pointblank issue tracker](https://github.com/posit-dev/pointblank/issues).
:::

Parameters
----------
filepath
    The path to the saved validation file. Can be a string or Path object.

Returns
-------
Validate
    The restored validation object with all its original state, validation results, and
    metadata.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

Let's first save a validation object to disk with `write_file()`. We'll use a temporary
directory here, and `keep_tbl=True` saves the target table along with the validation results:

```{python}
import pointblank as pb
import os
import tempfile

validation = (
    pb.Validate(data=pb.load_dataset("small_table"), tbl_name="small_table")
    .col_vals_gt(columns="d", value=1000)
    .col_vals_not_null(columns="c")
    .interrogate()
)

save_dir = tempfile.mkdtemp()
pb.write_file(validation, "my_validation", path=save_dir, keep_tbl=True, quiet=True)
```

Now we can load the validation object from disk with `read_file()`. The filename can be given
without its extension, in which case `read_file()` looks for `"my_validation.pkl"`:

```{python}
restored = pb.read_file(os.path.join(save_dir, "my_validation"))

restored
```

The restored object has the same results as the original: the first step has 6 failing test
units and the second has 2. The loaded validation object retains all its functionality, so we
can, for example, get the rows that failed any step (since the target table was saved):

```{python}
pb.preview(restored.get_sundered_data(type="fail"))
```

These are the 8 rows of `small_table` that failed at least one of the two steps. Without
`keep_tbl=True` in `write_file()`, the table wouldn't be saved and methods that need the data,
like `get_sundered_data()`, wouldn't be available after loading.

See Also
--------
write_file : Save a validation object to disk for later retrieval with `read_file()`.

ref(column_name: 'str') -> 'ReferenceColumn'

Reference a column from the reference data for aggregate comparisons.

This function is used with aggregate validation methods (like `col_sum_eq`, `col_avg_gt`, etc.)
to compare the aggregate value of a column in the main data against the aggregate value of
a column in the reference data.

To use this function, you must first set the reference data on the `Validate` object using
the `reference=` parameter in the constructor.

Parameters
----------
column_name
    The name of the column in the reference data to compute the aggregate from.

Returns
-------
ReferenceColumn
    A reference column marker that indicates the value should be computed from the
    reference data.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```

Suppose we have two DataFrames: a current data table and a reference (historical) table.
We want to validate that the sum of a column in the current data matches the sum of the
same column in the reference data.

```{python}
import pointblank as pb
import polars as pl

# Current data
current_data = pl.DataFrame({"sales": [100, 200, 300]})

# Reference (historical) data
reference_data = pl.DataFrame({"sales": [100, 200, 300]})

validation = (
    pb.Validate(data=current_data, reference=reference_data)
    .col_sum_eq("sales", pb.ref("sales"))
    .interrogate()
)

validation
```

The `sales` column sums to `600` in both tables, so the single test unit passes.

You can also compare different columns or use tolerance:

```{python}
current_data = pl.DataFrame({"revenue": [105, 205, 305]})
reference_data = pl.DataFrame({"sales": [100, 200, 300]})

# Check if revenue sum is within 10% of sales sum
validation = (
    pb.Validate(data=current_data, reference=reference_data)
    .col_sum_eq("revenue", pb.ref("sales"), tol=0.1)
    .interrogate()
)

validation
```

The `revenue` column sums to `615`, which is `15` more than the `600` of the reference
`sales` column. That's within the relative tolerance of 10% (`60`), so the step passes even
though the sums aren't equal.

See Also
--------
col : Reference columns within the same table.


## Test Data Generation

Generate synthetic test data based on schema definitions. Use `generate_dataset()` to create data from a Schema object, or `schema_from_tbl()` to infer a generation-ready schema from an existing table (Polars, Pandas, or Ibis/DuckDB).



generate_dataset(schema: 'Schema', n: 'int' = 100, seed: 'int | None' = None, output: "Literal['polars', 'pandas', 'dict']" = 'polars', country: 'str | list[str] | dict[str, float]' = 'US', shuffle: 'bool' = True, weighted: 'bool' = True) -> 'Any'

Generate synthetic test data from a schema.

This function generates random data that conforms to a schema's column definitions. When the
schema is defined using `Field` objects with constraints (e.g., `min_val=`, `max_val=`,
`pattern=`, `preset=`), the generated data will respect those constraints.

Synthetic data is useful when the real data can't be shared or doesn't exist yet. It lets you
develop and try out a validation plan before the data arrives, build test fixtures, and create
example tables for documentation and bug reports. Give a `seed=` to get the same data on every
run. The schema's columns usually come from field functions such as
[`int_field()`](`pointblank.int_field`) and [`string_field()`](`pointblank.string_field`), or
from [`profile_fields()`](`pointblank.profile_fields`) for coherent personal details (names,
emails, addresses), and [`schema_from_tbl()`](`pointblank.schema_from_tbl`) can start a schema
from an existing table. In a pytest suite, the `generate_dataset` fixture works the same way
and seeds each test automatically.

Parameters
----------
schema
    The schema object defining the structure and constraints of the data to generate. Each
    column can be specified using a field helper function (e.g., `int_field()`,
    `string_field()`) for fine-grained control, or as a simple dtype string (e.g.,
    `"Int64"`, `"String"`) for unconstrained generation.
n
    Number of rows to generate. The default is `100`.
seed
    Random seed for reproducibility. If provided, the same seed will produce the same data (in
    any Python session). Each column draws from its own random sequence, derived from the seed
    and the column name, so two columns with the same field get different values and have
    their missing values in different rows. Default is `None` (non-deterministic).
output
    Output format for the generated data. Options are: (1) `"polars"` (the default) returns a
    Polars DataFrame, (2) `"pandas"` returns a Pandas DataFrame, and (3) `"dict"` returns
    a dictionary of lists.
country
    Country code(s) for locale-aware generation when using presets. Accepts a single
    ISO 3166-1 alpha-2 or alpha-3 code (e.g., `"US"`, `"DEU"`), a list of codes for
    uniform mixing (e.g., `["US", "DE", "JP"]`), or a dict mapping codes to positive
    weights (e.g., `{"US": 60, "DE": 25, "JP": 15}`). See the *Locale Mixing* section
    below for details. The default is `"US"`.
shuffle
    When `country=` is a list or dict (multi-country mixing), controls whether rows from
    different countries are interleaved randomly (`True`, the default) or grouped by country
    in the order the countries are specified (`False`). Ignored when `country=` is a single
    string.
weighted
    When `True`, names and locations are sampled according to real-world frequency tiers.
    Common names like "James" and "Smith" appear far more often than rare names. Large
    cities like New York and Los Angeles dominate over small towns. Only affects data files
    that have been migrated to the tiered format; flat-list data always uses uniform
    sampling. Default is `True`.

Returns
-------
DataFrame or dict
    Generated data in the requested format. In a DataFrame, each numeric column has its
    field's dtype (e.g., `int_field(dtype="Int32")` gives an `Int32` column), and a
    `time_field()` column holds `datetime.time` values.

Raises
------
ValueError
    If the schema has no columns, if constraints cannot be satisfied, or if a custom
    `generator=` of a numeric field returns a value that doesn't fit the field's dtype (such
    as `1.5` for an integer field, or `300` for a `"UInt8"` one).
ImportError
    If required optional dependencies are not installed.

Presets and the `country=` Parameter
------------------------------------
Several `string_field()` presets produce locale-aware data that varies depending on the
`country=` parameter. The following presets are particularly affected:

- **Address-related presets** (`"address"`, `"city"`, `"state"`, `"postcode"`,
  `"phone_number"`, `"latitude"`, `"longitude"`, `"license_plate"`): produce addresses,
  cities, postal codes, phone numbers, and license plates formatted for the specified
  country. For example, `country="DE"` yields German street names and PLZ postal codes,
  while `country="JP"` yields Japanese addresses. License plates for CA, US, DE, AU, and
  GB use province/state-specific formats when location fields are present.
- **Person-related presets** (`"name"`, `"name_full"`, `"first_name"`, `"last_name"`,
  `"email"`, `"user_name"`) produce culturally appropriate names for the specified country.
  For example, `country="FR"` produces French names, while `country="KR"` produces Korean
  names.
- **Business-related presets** (`"job"`, `"company"`): when both are present, the job and
  company are drawn from the same industry for realism. The `"name_full"` preset will also
  add profession-matched titles (e.g., "Dr." for doctors, "Prof." for professors), and
  integer columns named `age` are automatically constrained to working-age range (22--65).
- **Financial presets** (`"iban"`, `"ssn"`, `"license_plate"`): produce identifiers in the
  format used by the specified country.
- **Locale preset** (`"locale_code"`): returns a locale identifier (e.g., `"en_US"`,
  `"de_DE"`) derived from the country. Multilingual countries randomly select among their
  official locale codes (e.g., `"CH"` yields `"de_CH"`, `"fr_CH"`, or `"it_CH"`).

When multiple columns in the same schema use related presets, the generated data is
automatically coherent across those columns within each row. Person-related presets will share
the same identity (e.g., the email is derived from the name), address-related presets will
share the same location (e.g., the city and postcode match the address), and business-related
presets will share the same industry context. Profiles made by `profile_fields()` with different
`prefix=` values (e.g., `"customer_"` and `"seller_"`) are coherent separately, so each one is a
different person with its own location and employer.

Locale Mixing
-------------
The `country=` parameter accepts three input forms for flexible locale control:

(1) a **single string** (the default), such as `"US"` or `"DEU"`, which generates
all rows from one locale; (2) a **list of strings**, such as `["US", "DE", "JP"]`,
which splits rows equally across the listed countries; and (3) a **dict of weights**,
such as `{"US": 0.6, "DE": 0.3, "FR": 0.1}`, which allocates rows proportionally
(weights are auto-normalized, so `{"US": 6, "DE": 3, "FR": 1}` is equivalent).

Row counts are distributed using largest-remainder apportionment so they always sum
to exactly `n=`. Each country's rows are generated as an independent batch (preserving
all cross-column coherence within each batch), then either interleaved randomly
(`shuffle=True`, the default) or left in contiguous country blocks
(`shuffle=False`).

Supported Countries
-------------------
The `country=` parameter currently supports 100 countries with full locale data:

**Europe (38 countries):** Armenia (`"AM"`), Austria (`"AT"`), Azerbaijan (`"AZ"`),
Belgium (`"BE"`), Bulgaria (`"BG"`), Croatia (`"HR"`), Cyprus (`"CY"`),
Czech Republic (`"CZ"`), Denmark (`"DK"`), Estonia (`"EE"`), Finland (`"FI"`),
France (`"FR"`), Georgia (`"GE"`), Germany (`"DE"`), Greece (`"GR"`),
Hungary (`"HU"`), Iceland (`"IS"`), Ireland (`"IE"`), Italy (`"IT"`),
Latvia (`"LV"`), Lithuania (`"LT"`), Luxembourg (`"LU"`), Malta (`"MT"`),
Moldova (`"MD"`), Netherlands (`"NL"`), Norway (`"NO"`), Poland (`"PL"`),
Portugal (`"PT"`), Romania (`"RO"`), Russia (`"RU"`), Serbia (`"RS"`),
Slovakia (`"SK"`), Slovenia (`"SI"`), Spain (`"ES"`), Sweden (`"SE"`),
Switzerland (`"CH"`), Ukraine (`"UA"`), United Kingdom (`"GB"`)

**Americas (19 countries):** Argentina (`"AR"`), Bolivia (`"BO"`), Brazil (`"BR"`),
Canada (`"CA"`), Chile (`"CL"`), Colombia (`"CO"`), Costa Rica (`"CR"`),
Dominican Republic (`"DO"`), Ecuador (`"EC"`), El Salvador (`"SV"`),
Guatemala (`"GT"`), Honduras (`"HN"`), Jamaica (`"JM"`), Mexico (`"MX"`),
Panama (`"PA"`), Paraguay (`"PY"`), Peru (`"PE"`), United States (`"US"`),
Uruguay (`"UY"`)

**Asia-Pacific (22 countries):** Australia (`"AU"`), Bangladesh (`"BD"`),
Cambodia (`"KH"`), China (`"CN"`), Hong Kong (`"HK"`), India (`"IN"`),
Indonesia (`"ID"`), Japan (`"JP"`), Kazakhstan (`"KZ"`), Malaysia (`"MY"`),
Myanmar (`"MM"`), Nepal (`"NP"`), New Zealand (`"NZ"`), Pakistan (`"PK"`),
Philippines (`"PH"`), Singapore (`"SG"`), South Korea (`"KR"`),
Sri Lanka (`"LK"`), Taiwan (`"TW"`), Thailand (`"TH"`), Uzbekistan (`"UZ"`),
Vietnam (`"VN"`)

**Middle East & Africa (21 countries):** Algeria (`"DZ"`), Cameroon (`"CM"`),
Egypt (`"EG"`), Ethiopia (`"ET"`), Ghana (`"GH"`), Israel (`"IL"`),
Jordan (`"JO"`), Kenya (`"KE"`), Lebanon (`"LB"`), Morocco (`"MA"`),
Mozambique (`"MZ"`), Nigeria (`"NG"`), Rwanda (`"RW"`), Saudi Arabia (`"SA"`),
Senegal (`"SN"`), South Africa (`"ZA"`), Tanzania (`"TZ"`), Tunisia (`"TN"`),
Turkey (`"TR"`), Uganda (`"UG"`), United Arab Emirates (`"AE"`)

Pytest Fixture
--------------
When Pointblank is installed, a `generate_dataset` pytest fixture is automatically
available in all test files: no imports or `conftest.py` setup required. The fixture
behaves identically to this function, but derives a deterministic seed from the test's
fully-qualified name when `seed=` is not provided.

This means:

- the **same test** always produces the **same data**, with no manual seed management.
- **different tests** get different seeds, so they exercise different data.
- **you** can still pass an explicit `seed=` to override the automatic seed.
- **calling** the fixture **multiple times** within one test produces different (but still
deterministic) data on each call.
- the fixture exposes `.default_seed` and `.last_seed` attributes for debugging.

```python
def test_my_pipeline(generate_dataset):
    import pointblank as pb

    schema = pb.Schema(
        user_id=pb.int_field(unique=True),
        email=pb.string_field(preset="email"),
        age=pb.int_field(min_val=18, max_val=100),
    )
    df = generate_dataset(schema, n=500, country="DE")
    # seed is derived from "test_my_pipeline" — same data every run
    result = my_pipeline(df)
    assert result.shape[0] == 500
```

Multiple datasets can be generated within the same test, each with its own
deterministic seed:

```python
def test_merge(generate_dataset):
    customers = generate_dataset(customer_schema, n=1000, country="US")
    orders = generate_dataset(order_schema, n=5000)
    # Both DataFrames are deterministic; each call gets a unique seed
```

When a test fails, include the seed in the assertion message so the failure is easy to
reproduce:

```python
def test_age_range(generate_dataset):
    df = generate_dataset(schema, n=100)
    assert df["age"].min() >= 18, f"Failed with seed {generate_dataset.last_seed}"
```

Seed Stability
--------------
A given seed (whether explicit or auto-derived) is guaranteed to produce identical output
**within the same Pointblank version**. Across versions, changes to country data files or
generator logic may alter the output for a given seed.

For CI pipelines that require bit-exact data across library upgrades, save generated
DataFrames as Parquet or CSV snapshot files rather than relying on cross-version seed
stability. This is the same approach used by snapshot-testing tools like `pytest-snapshot`
and `syrupy`.

Examples
--------
Here we define a schema with field constraints and generate test data from it:

```{python}
import pointblank as pb

schema = pb.Schema(
    user_id=pb.int_field(min_val=1, unique=True),
    email=pb.string_field(preset="email"),
    age=pb.int_field(min_val=18, max_val=100),
    status=pb.string_field(allowed=["active", "pending", "inactive"]),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

It's also possible to generate data from a simple, dtype-only schema. Setting
`output="pandas"` returns a Pandas DataFrame:

```{python}
schema = pb.Schema(name="String", age="Int64", active="Boolean")

pb.preview(pb.generate_dataset(schema, n=50, seed=23, output="pandas"))
```

When using presets, the `country=` parameter controls the locale. Here, `country="DE"`
produces German names and addresses:

```{python}
schema = pb.Schema(
    name=pb.string_field(preset="name"),
    address=pb.string_field(preset="address"),
    city=pb.string_field(preset="city"),
)

pb.preview(pb.generate_dataset(schema, n=20, seed=23, country="DE"))
```

We can combine several field types with nullable columns in a mixed-type dataset:

```{python}
from datetime import date, timedelta

schema = pb.Schema(
    id=pb.int_field(min_val=1, max_val=1000, unique=True),
    name=pb.string_field(preset="name"),
    score=pb.float_field(min_val=0.0, max_val=100.0),
    is_active=pb.bool_field(p_true=0.75),
    joined=pb.date_field(min_date=date(2020, 1, 1), max_date=date(2024, 12, 31)),
    session_time=pb.duration_field(
        min_duration=timedelta(minutes=1),
        max_duration=timedelta(hours=3),
        nullable=True, null_probability=0.2,
    ),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23))
```

Each column gets the dtype of its field: `Int64`, `String`, `Float64`, `Boolean`, `Date` and
`Duration`. The `session_time` column has 12 missing values out of 50, close to the
`null_probability=` of `0.2`, and the same `seed=` gives the same table on every run.

schema_from_tbl(tbl: 'Any', *, infer_constraints: 'bool' = True, categorical_threshold: 'int | float' = 20, detect_presets: 'bool' = True, sample_size: 'int | None' = None) -> 'Schema'

Create a Schema from an existing table with inferred Field constraints.

This is the functional form of `Schema.from_table()`. It inspects the actual values in the
table to infer rich constraints (min/max, uniqueness, null rates, allowed values, presets)
suitable for synthetic data generation via `schema.generate()` or `generate_dataset()`.

Parameters
----------
tbl
    A Polars DataFrame, Pandas DataFrame, or Ibis table (DuckDB, SQLite, etc.).
infer_constraints
    When `True` (default), inspect values to infer min/max, uniqueness, null rates, etc. When
    `False`, behave like `Schema(tbl=df)` (dtype only).
categorical_threshold
    If a column has <= this many unique values (int) or this fraction of total rows (float
    between `0` and `1`), treat it as categorical and populate `allowed=`. Default is `20`.
detect_presets
    Attempt to match string columns to known generation presets (e.g., email, url, phone_number)
    based on column name heuristics and value validation. Default is `True`.
sample_size
    If set, sample this many rows before analysis (useful for very large tables). `None` means
    use all rows.

Returns
-------
Schema
    A Schema populated with Field objects containing inferred constraints, ready for use with
    `schema.generate()` or `generate_dataset()`.

Examples
--------
```{python}
import pointblank as pb
import polars as pl

df = pl.DataFrame({
    "user_id": list(range(1, 51)),
    "email": [f"user{i}@example.com" for i in range(50)],
    "age": [20 + i % 50 for i in range(50)],
    "status": ["active", "pending", "inactive"] * 16 + ["active", "pending"],
})

schema = pb.schema_from_tbl(df)
print(schema)
```

The inferred schema has a field for each column. The `user_id` and `age` fields take their
ranges from the data and are marked unique (every value in those columns is distinct), the
`email` column is matched to the `"email"` preset by its name and values, and `status` has
few enough distinct values to become a categorical field with an `allowed=` list.

Let's generate synthetic data matching the original's characteristics:

```{python}
pb.preview(schema.generate(n=10, seed=23))
```

The new rows follow the inferred constraints: the IDs lie between 1 and 50, the ages between
20 and 69, and every `status` is one of the three original values. The emails are new,
realistic addresses from the preset rather than copies of the original `user{i}@example.com`
values.

int_field(min_val: 'int | None' = None, max_val: 'int | None' = None, allowed: 'list[int] | None' = None, nullable: 'bool' = False, null_probability: 'float' = 0.0, unique: 'bool' = False, generator: 'Callable[[], Any] | None' = None, dtype: 'str' = 'Int64') -> 'IntField'

Create an integer column specification for use in a schema.

The `int_field()` function defines the constraints and behavior for an integer column when
generating synthetic data with `generate_dataset()`. You can control the range of values
with `min_val=` and `max_val=`, restrict values to a specific set with `allowed=`, enforce
uniqueness with `unique=True`, and introduce null values with `nullable=True` and
`null_probability=`. The `dtype=` parameter sets the integer type in the schema (e.g.,
`"Int8"`, `"UInt16"`, `"Int64"`), which also limits the range of values.

When no constraints are specified, values are drawn uniformly from the full range of the
chosen integer dtype. If both `min_val=` and `max_val=` are provided, values are drawn
uniformly from that range. If `allowed=` is provided, values are sampled from that specific
list.

Parameters
----------
min_val
    Minimum value (inclusive). Default is `None` (no minimum, uses dtype lower bound).
max_val
    Maximum value (inclusive). Default is `None` (no maximum, uses dtype upper bound).
allowed
    List of allowed values (categorical constraint). When provided, values are sampled from
    this list. Cannot be combined with `min_val=`/`max_val=`.
nullable
    Whether the column can contain null values. Default is `False`.
null_probability
    Probability of generating a null value for each row when `nullable=True`. Must be
    between `0.0` and `1.0`. Default is `0.0`.
unique
    Whether all values must be unique. Default is `False`. When `True`, the generator will
    retry until it produces `n` distinct values (subject to retry limits).
generator
    Custom callable that generates values. When provided, it's used instead of `min_val=`,
    `max_val=` and `allowed=`, while `unique=` and `nullable=` still apply. The callable should
    take no arguments and return a single integer value. It isn't controlled by the `seed=` of
    `generate_dataset()`, so it needs its own seeded random number generator for reproducible
    data.
dtype
    Integer dtype. Default is `"Int64"`. Options: `"Int8"`, `"Int16"`, `"Int32"`,
    `"Int64"`, `"UInt8"`, `"UInt16"`, `"UInt32"`, `"UInt64"`. The dtype limits the range of
    values (e.g., `0` to `255` for `"UInt8"`) and is the dtype of the column that
    `generate_dataset()` creates. With `output="pandas"`, the column is a NumPy integer (e.g.,
    `int32`), or a pandas nullable integer (e.g., `Int32`) when `nullable=True`.

Returns
-------
IntField
    An integer field specification that can be passed to `Schema()`.

Raises
------
ValueError
    If `min_val` is greater than `max_val`, if `allowed` is an empty list, if
    `null_probability` is not between `0.0` and `1.0`, or if `dtype` is not a valid
    integer type.

Examples
--------
The `min_val=` and `max_val=` parameters constrain generated ranges, while `allowed=`
restricts values to a specific set:

```{python}
import pointblank as pb

schema = pb.Schema(
    user_id=pb.int_field(min_val=1, unique=True),
    age=pb.int_field(min_val=0, max_val=120),
    rating=pb.int_field(allowed=[1, 2, 3, 4, 5]),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

It's possible to introduce missing values with `nullable=True` and `null_probability=`,
and to limit the range of values with a smaller `dtype=`:

```{python}
schema = pb.Schema(
    score=pb.int_field(min_val=0, max_val=255, dtype="UInt8"),
    optional_val=pb.int_field(
        min_val=1, max_val=50,
        nullable=True, null_probability=0.3,
    ),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23))
```

We can also enforce uniqueness with `unique=True` to produce distinct identifiers within
a range:

```{python}
schema = pb.Schema(
    record_id=pb.int_field(min_val=1000, max_val=9999, unique=True),
    priority=pb.int_field(allowed=[1, 2, 3]),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=10))
```

For complete control, a custom `generator=` callable can be provided:

```{python}
import random

rng = random.Random(0)

schema = pb.Schema(
    even_numbers=pb.int_field(generator=lambda: rng.choice(range(0, 100, 2))),
)

pb.preview(pb.generate_dataset(schema, n=20, seed=5))
```

The generator is called once for each row, so every value is an even number below 100. The
`seed=` of `generate_dataset()` doesn't reach the generator, which is why it uses its own
`random.Random(0)`: that makes the data the same on every run.

float_field(min_val: 'float | None' = None, max_val: 'float | None' = None, allowed: 'list[float] | None' = None, precision: 'int | None' = None, nullable: 'bool' = False, null_probability: 'float' = 0.0, unique: 'bool' = False, generator: 'Callable[[], Any] | None' = None, dtype: 'str' = 'Float64') -> 'FloatField'

Create a floating-point column specification for use in a schema.

The `float_field()` function defines the constraints and behavior for a floating-point column
when generating synthetic data with `generate_dataset()`. You can control the range of values
with `min_val=` and `max_val=`, restrict values to a specific set with `allowed=`, enforce
uniqueness with `unique=True`, and introduce null values with `nullable=True` and
`null_probability=`. The `dtype=` parameter sets the float type in the schema (`"Float32"` or
`"Float64"`).

When both `min_val=` and `max_val=` are provided, values are drawn from a uniform
distribution across that range. If neither is specified, values are drawn uniformly from a
large default range. If `allowed=` is provided, values are sampled from that specific list.

Parameters
----------
min_val
    Minimum value (inclusive). Default is `None` (no minimum).
max_val
    Maximum value (inclusive). Default is `None` (no maximum).
allowed
    List of allowed values (categorical constraint). When provided, values are sampled from
    this list. Cannot be combined with `min_val=`/`max_val=`.
precision
    Number of decimal places to round generated values to. Default is `None` (no rounding).
    Must be a non-negative integer. Has no effect when `allowed=` or `generator=` is used.
nullable
    Whether the column can contain null values. Default is `False`.
null_probability
    Probability of generating a null value for each row when `nullable=True`. Must be
    between `0.0` and `1.0`. Default is `0.0`.
unique
    Whether all values must be unique. Default is `False`. When `True`, the generator will
    retry until it produces `n` distinct values.
generator
    Custom callable that generates values. When provided, it's used instead of `min_val=`,
    `max_val=`, `allowed=` and `precision=`, while `unique=` and `nullable=` still apply. The
    callable should take no arguments and return a single float value. It isn't controlled by
    the `seed=` of `generate_dataset()`, so it needs its own seeded random number generator
    for reproducible data.
dtype
    Float dtype. Default is `"Float64"`. Options: `"Float32"`, `"Float64"`. It's the dtype of
    the column that `generate_dataset()` creates.

Returns
-------
FloatField
    A float field specification that can be passed to `Schema()`.

Raises
------
ValueError
    If `min_val` is greater than `max_val`, if `allowed` is an empty list, if
    `null_probability` is not between `0.0` and `1.0`, if `precision` is negative,
    or if `dtype` is not a valid float type.

Examples
--------
The `min_val=` and `max_val=` parameters define the generated value ranges:

```{python}
import pointblank as pb

schema = pb.Schema(
    price=pb.float_field(min_val=0.01, max_val=9999.99),
    probability=pb.float_field(min_val=0.0, max_val=1.0),
    temperature=pb.float_field(min_val=-40.0, max_val=50.0),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

It's also possible to restrict values to a discrete set with `allowed=`, which is useful
for fixed pricing tiers or measurement levels:

```{python}
schema = pb.Schema(
    discount=pb.float_field(allowed=[0.05, 0.10, 0.15, 0.20, 0.25]),
    weight_kg=pb.float_field(min_val=0.5, max_val=100.0),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23))
```

We can simulate missing measurements by introducing null values:

```{python}
schema = pb.Schema(
    reading=pb.float_field(
        min_val=0.0, max_val=500.0,
        nullable=True, null_probability=0.2,
    ),
    calibration=pb.float_field(min_val=0.9, max_val=1.1),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=23))
```

Use `precision=` to round generated values to a fixed number of decimal places. This is useful
for prices, scores, or any measurement where full floating-point precision is unwanted:

```{python}
schema = pb.Schema(
    price=pb.float_field(min_val=1.0, max_val=200.0, precision=2),
    score=pb.float_field(min_val=0.0, max_val=100.0, precision=1),
    probability=pb.float_field(min_val=0.0, max_val=1.0, precision=4),
)

pb.preview(pb.generate_dataset(schema, n=20, seed=23))
```

A custom `generator=` provides full control over value generation, such as values from a
distribution other than the uniform one:

```{python}
import random, math

rng = random.Random(0)

schema = pb.Schema(
    sensor_value=pb.float_field(min_val=-10.0, max_val=10.0),
    log_value=pb.float_field(generator=lambda: math.log(rng.uniform(1, 1000))),
)

pb.preview(pb.generate_dataset(schema, n=20, seed=23))
```

The `log_value` column holds the logarithm of a uniform value between 1 and 1000, so its
values lie between 0 and about 6.9 and are bunched toward the top of that range. The
generator has its own seeded `random.Random(0)`, since the `seed=` of `generate_dataset()`
doesn't reach it.

string_field(min_length: 'int | None' = None, max_length: 'int | None' = None, pattern: 'str | None' = None, preset: 'str | None' = None, allowed: 'list[str] | None' = None, nullable: 'bool' = False, null_probability: 'float' = 0.0, unique: 'bool' = False, generator: 'Callable[[], Any] | None' = None) -> 'StringField'

Create a string column specification for use in a schema.

The `string_field()` function defines the constraints and behavior for a string column when
generating synthetic data with `generate_dataset()`. It provides three main modes of string
generation: (1) controlled random strings with `min_length=`/`max_length=`, (2) strings
matching a regular expression via `pattern=`, or (3) realistic data using `preset=` (e.g.,
`"email"`, `"name"`, `"address"`). You can also restrict values to a fixed set with
`allowed=`. Only one of `preset=`, `pattern=`, or `allowed=` can be specified at a time.

When no special mode is selected, random alphanumeric strings are generated with lengths
between `min_length=` and `max_length=` (defaulting to 1--20 characters).

Parameters
----------
min_length
    Minimum string length (for random string generation). Default is `None` (defaults to
    `1`). Only applies when `preset=`, `pattern=`, and `allowed=` are all `None`.
max_length
    Maximum string length (for random string generation). Default is `None` (defaults to
    `20`). Only applies when `preset=`, `pattern=`, and `allowed=` are all `None`.
pattern
    Regular expression pattern that generated strings must match. Supports character
    classes (e.g., `[A-Z]`, `[0-9]`), quantifiers (e.g., `{3}`, `{2,5}`), alternation,
    and groups. Cannot be combined with `preset=` or `allowed=`.
preset
    Preset name for generating realistic data. When specified, values are produced using
    locale-aware data generation, and the `country=` parameter of `generate_dataset()`
    controls the locale. Cannot be combined with `pattern=` or `allowed=`. See the
    **Available Presets** section below for the full list.
allowed
    List of allowed string values (categorical constraint). Values are sampled uniformly
    from this list. Cannot be combined with `preset=` or `pattern=`.
nullable
    Whether the column can contain null values. Default is `False`.
null_probability
    Probability of generating a null value for each row when `nullable=True`. Must be
    between `0.0` and `1.0`. Default is `0.0`.
unique
    Whether all values must be unique. Default is `False`. When `True`, the generator will
    retry until it produces `n` distinct values.
generator
    Custom callable that generates values. When provided, it's used instead of the other
    constraints, while `unique=` and `nullable=` still apply. The callable should take no
    arguments and return a single string value. It isn't controlled by the `seed=` of
    `generate_dataset()`, so it needs its own seeded random number generator for reproducible
    data.

Returns
-------
StringField
    A string field specification that can be passed to `Schema()`.

Raises
------
ValueError
    If more than one of `preset=`, `pattern=`, or `allowed=` is specified; if `allowed=`
    is an empty list; if `min_length` or `max_length` is negative; if `min_length` exceeds
    `max_length`; or if `preset` is not a recognized preset name.

Available Presets
-----------------
The `preset=` parameter accepts one of the following preset names, organized by category.
When a preset is used, the `country=` parameter of `generate_dataset()` controls the locale
for region-specific formatting (e.g., address formats, phone number patterns).

**Personal:** `"name"` (first + last name), `"name_full"` (full name with possible prefix
or suffix), `"first_name"`, `"last_name"`, `"gender"` (person's gender, coherent with
name), `"email"` (realistic email address), `"phone_number"`, `"address"` (full street
address), `"city"`, `"state"`, `"country"`,
`"country_code_2"` (ISO 3166-1 alpha-2 code, e.g., `"US"`), `"country_code_3"` (ISO
3166-1 alpha-3 code, e.g., `"USA"`), `"postcode"`, `"latitude"`, `"longitude"`

**Business:** `"company"` (company name), `"job"` (job title), `"catch_phrase"`

**Internet:** `"url"`, `"domain_name"`, `"ipv4"`, `"ipv6"`, `"user_name"`, `"password"`

**Text:** `"text"` (paragraph of text), `"sentence"`, `"paragraph"`, `"word"`

**Financial:** `"credit_card_number"`, `"credit_card_provider"` (Visa, Mastercard,
American Express, or Discover), `"iban"`, `"currency_code"`

**Identifiers:** `"uuid4"`, `"md5"` (MD5 hash, 32 hex chars), `"sha1"` (SHA-1 hash,
40 hex chars), `"sha256"` (SHA-256 hash, 64 hex chars), `"ssn"` (social security number),
`"license_plate"`

**Barcodes:** `"ean8"` (EAN-8 barcode with valid check digit), `"ean13"` (EAN-13 barcode
with valid check digit)

**Date/Time (as strings):** `"date_this_year"`, `"date_this_decade"`, `"date_between"`
(random date between 2000–2025), `"date_range"` (two dates joined with an en-dash, e.g.,
`"2012-05-12 – 2015-11-22"`), `"future_date"` (up to 1 year ahead), `"past_date"`
(up to 10 years back), `"time"`

**Miscellaneous:** `"color_name"`, `"file_name"`, `"file_extension"`, `"mime_type"`,
`"user_agent"` (browser user agent string with country-specific browser weighting),
`"locale_code"` (locale identifier like `"en_US"`, `"de_DE"`; multilingual countries
return a random official locale)

Coherent Data Generation
------------------------
When multiple columns in the same schema use related presets, the generated data will be
coherent across those columns within each row. Specifically:

- **Person-related presets** (`"name"`, `"name_full"`, `"first_name"`, `"last_name"`,
  `"gender"`, `"email"`, `"user_name"`): the email and username will be derived from the
  person's name, and `"gender"` will match the person's first name.
- **Address-related presets** (`"address"`, `"city"`, `"state"`, `"postcode"`,
  `"phone_number"`, `"latitude"`, `"longitude"`): the city, state, and postcode will
  correspond to the same location within the address.
- **Credit card presets** (`"credit_card_number"`, `"credit_card_provider"`): the card
  number prefix and provider name will be consistent (e.g., "Visa" with a "4"-prefixed number).

This coherence is automatic and requires no additional configuration.

Examples
--------
The `preset=` parameter generates realistic personal data, while `allowed=` restricts
values to a categorical set:

```{python}
import pointblank as pb

schema = pb.Schema(
    name=pb.string_field(preset="name"),
    email=pb.string_field(preset="email", unique=True),
    status=pb.string_field(allowed=["active", "pending", "inactive"]),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

We can also generate strings that match a regular expression with `pattern=` (e.g.,
product codes, identifiers):

```{python}
schema = pb.Schema(
    product_code=pb.string_field(pattern=r"[A-Z]{3}-[0-9]{4}"),
    batch_id=pb.string_field(pattern=r"BATCH-[A-Z][0-9]{3}"),
    sku=pb.string_field(pattern=r"[A-Z]{2}[0-9]{6}"),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=23))
```

For random alphanumeric strings, `min_length=` and `max_length=` control the length.
Adding `nullable=True` introduces missing values:

```{python}
schema = pb.Schema(
    short_code=pb.string_field(min_length=3, max_length=5),
    notes=pb.string_field(
        min_length=10, max_length=50,
        nullable=True, null_probability=0.4,
    ),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=7))
```

It's possible to combine business and internet presets to build a company directory:

```{python}
schema = pb.Schema(
    company=pb.string_field(preset="company"),
    domain=pb.string_field(preset="domain_name"),
    industry_tag=pb.string_field(allowed=["tech", "finance", "health", "retail"]),
)

pb.preview(pb.generate_dataset(schema, n=20, seed=55))
```

Unlike the person and address presets, `"domain_name"` isn't coherent with `"company"`, so
each domain is unrelated to the company in its row. The `industry_tag` values come from the
`allowed=` list, also independently of the company.

bool_field(p_true: 'float' = 0.5, nullable: 'bool' = False, null_probability: 'float' = 0.0, unique: 'bool' = False, generator: 'Callable[[], Any] | None' = None) -> 'BoolField'

Create a boolean column specification for use in a schema.

The `bool_field()` function defines the constraints and behavior for a boolean column when
generating synthetic data with `generate_dataset()`. The `p_true=` parameter controls the
probability of generating `True` values, which is useful for simulating real-world
distributions where events may be rare or common (e.g., 5% fraud rate, 80% active users).

By default, `True` and `False` are equally likely (`p_true=0.5`). Setting `p_true=0.0`
produces all `False` values, and `p_true=1.0` produces all `True` values.

Parameters
----------
p_true
    Probability of generating `True`. Default is `0.5` (equal probability).
    Must be between `0.0` and `1.0`.
nullable
    Whether the column can contain null values. Default is `False`.
null_probability
    Probability of generating a null value for each row when `nullable=True`. Must be
    between `0.0` and `1.0`. Default is `0.0`.
unique
    Whether all values must be unique. Default is `False`. Note that boolean columns can
    only have 2 unique non-null values, so `n` must be `<= 2` when `unique=True` (or
    `<= 3` with `nullable=True`).
generator
    Custom callable that generates values. When provided, it's used instead of the other
    constraints, while `unique=` and `nullable=` still apply. The callable should take no
    arguments and return a single boolean value. It isn't controlled by the `seed=` of
    `generate_dataset()`, so it needs its own seeded random number generator for reproducible
    data.

Returns
-------
BoolField
    A boolean field specification that can be passed to `Schema()`.

Raises
------
ValueError
    If `p_true` is not between `0.0` and `1.0`, or if `null_probability` is not between
    `0.0` and `1.0`.

Examples
--------
The `p_true=` parameter controls the distribution of `True`/`False` values, allowing
you to simulate different probabilities:

```{python}
import pointblank as pb

schema = pb.Schema(
    is_active=pb.bool_field(p_true=0.8),
    is_premium=pb.bool_field(p_true=0.2),
    is_verified=pb.bool_field(),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

Optional boolean flags can be simulated by combining `nullable=True` with
`null_probability=`:

```{python}
schema = pb.Schema(
    opted_in=pb.bool_field(p_true=0.6),
    has_referral=pb.bool_field(
        p_true=0.3,
        nullable=True, null_probability=0.25,
    ),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23))
```

Boolean fields can be combined with other field types in a realistic schema:

```{python}
schema = pb.Schema(
    user_id=pb.int_field(min_val=1, max_val=9999, unique=True),
    name=pb.string_field(preset="name"),
    email_verified=pb.bool_field(p_true=0.9),
    is_admin=pb.bool_field(p_true=0.05),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=10))
```

Here, `email_verified` is `True` in 24 of the 30 rows and `is_admin` in 1 of them. These
counts are close to, but not exactly, the `p_true=` values of `0.9` and `0.05`, since each
value is drawn independently.

date_field(min_date: 'str | date | None' = None, max_date: 'str | date | None' = None, nullable: 'bool' = False, null_probability: 'float' = 0.0, unique: 'bool' = False, generator: 'Callable[[], Any] | None' = None) -> 'DateField'

Create a date column specification for use in a schema.

The `date_field()` function defines the constraints and behavior for a date column when
generating synthetic data with `generate_dataset()`. You can control the date range with
`min_date=` and `max_date=`, enforce uniqueness with `unique=True`, and introduce null
values with `nullable=True` and `null_probability=`.

Dates are generated uniformly within the specified range. If no range is provided, the
default range is 2000-01-01 to 2030-12-31. Both `min_date=` and `max_date=` accept either
`datetime.date` objects or ISO 8601 date strings (e.g., `"2024-06-15"`).

Parameters
----------
min_date
    Minimum date (inclusive). Can be an ISO format string (e.g., `"2020-01-01"`) or a
    `datetime.date` object. Default is `None` (defaults to `2000-01-01`).
max_date
    Maximum date (inclusive). Can be an ISO format string (e.g., `"2024-12-31"`) or a
    `datetime.date` object. Default is `None` (defaults to `2030-12-31`).
nullable
    Whether the column can contain null values. Default is `False`.
null_probability
    Probability of generating a null value for each row when `nullable=True`. Must be
    between `0.0` and `1.0`. Default is `0.0`.
unique
    Whether all values must be unique. Default is `False`. When `True`, the generator will
    retry until it produces `n` distinct dates. Ensure the date range is large enough to
    accommodate the requested number of unique dates.
generator
    Custom callable that generates values. When provided, it's used instead of the other
    constraints, while `unique=` and `nullable=` still apply. The callable should take no
    arguments and return a single `datetime.date` value. It isn't controlled by the `seed=` of
    `generate_dataset()`, so it needs its own seeded random number generator for reproducible
    data.

Returns
-------
DateField
    A date field specification that can be passed to `Schema()`.

Raises
------
ValueError
    If `min_date` is later than `max_date`, or if a date string cannot be parsed.

Examples
--------
The `min_date=` and `max_date=` parameters accept `datetime.date` objects to define date
ranges:

```{python}
import pointblank as pb
from datetime import date

schema = pb.Schema(
    birth_date=pb.date_field(
        min_date=date(1960, 1, 1),
        max_date=date(2005, 12, 31),
    ),
    hire_date=pb.date_field(
        min_date=date(2020, 1, 1),
        max_date=date(2024, 12, 31),
    ),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

For convenience, ISO format strings can be used instead of `date` objects:

```{python}
schema = pb.Schema(
    event_date=pb.date_field(min_date="2024-01-01", max_date="2024-12-31"),
    signup_date=pb.date_field(min_date="2023-06-01", max_date="2024-06-01"),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23))
```

We can introduce missing dates with `nullable=True` and enforce distinct values using
`unique=True`:

```{python}
schema = pb.Schema(
    order_date=pb.date_field(
        min_date="2024-01-01", max_date="2024-03-31",
        unique=True,
    ),
    cancel_date=pb.date_field(
        min_date="2024-01-01", max_date="2024-12-31",
        nullable=True, null_probability=0.5,
    ),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=7))
```

All 30 `order_date` values are distinct, which is possible because the first quarter of 2024
has 91 days to choose from. About half of the `cancel_date` values are missing (20 of 30 here).
Each column is generated independently, so a few cancellation dates fall before the order
date in the same row.

datetime_field(min_date: 'str | datetime | None' = None, max_date: 'str | datetime | None' = None, nullable: 'bool' = False, null_probability: 'float' = 0.0, unique: 'bool' = False, generator: 'Callable[[], Any] | None' = None) -> 'DatetimeField'

Create a datetime column specification for use in a schema.

The `datetime_field()` function defines the constraints and behavior for a datetime column
when generating synthetic data with `generate_dataset()`. You can control the datetime range
with `min_date=` and `max_date=`, enforce uniqueness with `unique=True`, and introduce null
values with `nullable=True` and `null_probability=`.

Datetime values are generated uniformly (at second-level resolution) within the specified
range. If no range is provided, the default range is 2000-01-01T00:00:00 to
2030-12-31T23:59:59. Both `min_date=` and `max_date=` accept `datetime` objects, `date`
objects (which are converted to datetimes at midnight), or ISO 8601 datetime strings.

Parameters
----------
min_date
    Minimum datetime (inclusive). Can be an ISO format string (e.g.,
    `"2024-01-01T00:00:00"`), a `datetime.datetime` object, or a `datetime.date` object.
    Default is `None` (defaults to `2000-01-01 00:00:00`).
max_date
    Maximum datetime (inclusive). Can be an ISO format string, a `datetime.datetime`
    object, or a `datetime.date` object. Default is `None` (defaults to
    `2030-12-31 23:59:59`).
nullable
    Whether the column can contain null values. Default is `False`.
null_probability
    Probability of generating a null value for each row when `nullable=True`. Must be
    between `0.0` and `1.0`. Default is `0.0`.
unique
    Whether all values must be unique. Default is `False`. With second-level resolution
    over a wide range, collisions are unlikely for moderate dataset sizes.
generator
    Custom callable that generates values. When provided, it's used instead of the other
    constraints, while `unique=` and `nullable=` still apply. The callable should take no
    arguments and return a single `datetime.datetime` value. It isn't controlled by the `seed=`
    of `generate_dataset()`, so it needs its own seeded random number generator for reproducible
    data.

Returns
-------
DatetimeField
    A datetime field specification that can be passed to `Schema()`.

Raises
------
ValueError
    If `min_date` is later than `max_date`, or if a datetime string cannot be parsed.

Examples
--------
The `min_date=` and `max_date=` parameters accept `datetime` objects for precise range
definitions:

```{python}
import pointblank as pb
from datetime import datetime

schema = pb.Schema(
    created_at=pb.datetime_field(
        min_date=datetime(2024, 1, 1),
        max_date=datetime(2024, 12, 31),
    ),
    updated_at=pb.datetime_field(
        min_date=datetime(2024, 6, 1),
        max_date=datetime(2024, 12, 31),
    ),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

For a quick setup, ISO format strings work just as well:

```{python}
schema = pb.Schema(
    event_time=pb.datetime_field(
        min_date="2024-03-01T08:00:00",
        max_date="2024-03-01T18:00:00",
    ),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=23))
```

Optional timestamps can be simulated with `nullable=True`, and datetime fields work
nicely alongside other field types:

```{python}
schema = pb.Schema(
    order_id=pb.int_field(min_val=1000, max_val=9999, unique=True),
    placed_at=pb.datetime_field(
        min_date=datetime(2024, 1, 1),
        max_date=datetime(2024, 12, 31),
    ),
    shipped_at=pb.datetime_field(
        min_date=datetime(2024, 1, 2),
        max_date=datetime(2025, 1, 15),
        nullable=True, null_probability=0.3,
    ),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=7))
```

About 30% of the `shipped_at` values are missing (13 of 30 here). Since each column is
generated independently, some `shipped_at` values come before `placed_at` in the same row, so
these columns aren't suited for checks of date ordering without some extra processing.

time_field(min_time: 'str | time | None' = None, max_time: 'str | time | None' = None, nullable: 'bool' = False, null_probability: 'float' = 0.0, unique: 'bool' = False, generator: 'Callable[[], Any] | None' = None) -> 'TimeField'

Create a time column specification for use in a schema.

The `time_field()` function defines the constraints and behavior for a time-of-day column
when generating synthetic data with `generate_dataset()`. You can control the time range
with `min_time=` and `max_time=`, enforce uniqueness with `unique=True`, and introduce null
values with `nullable=True` and `null_probability=`.

Time values are generated uniformly (at second-level resolution) within the specified range.
If no range is provided, the default range is 00:00:00 to 23:59:59. Both `min_time=` and
`max_time=` accept `datetime.time` objects or ISO format time strings (e.g., `"09:30:00"`).
The column that `generate_dataset()` creates holds `datetime.time` values: a `Time` column in
Polars, and an `object` column in pandas (which has no time dtype).

Parameters
----------
min_time
    Minimum time (inclusive). Can be an ISO format string (e.g., `"08:00:00"`) or a
    `datetime.time` object. Default is `None` (defaults to `00:00:00`).
max_time
    Maximum time (inclusive). Can be an ISO format string (e.g., `"17:30:00"`) or a
    `datetime.time` object. Default is `None` (defaults to `23:59:59`).
nullable
    Whether the column can contain null values. Default is `False`.
null_probability
    Probability of generating a null value for each row when `nullable=True`. Must be
    between `0.0` and `1.0`. Default is `0.0`.
unique
    Whether all values must be unique. Default is `False`. With second-level resolution
    within a time range, uniqueness is feasible for moderate dataset sizes.
generator
    Custom callable that generates values. When provided, it's used instead of the other
    constraints, while `unique=` and `nullable=` still apply. The callable should take no
    arguments and return a single value. It isn't controlled by the `seed=` of
    `generate_dataset()`, so it needs its own seeded random number generator for reproducible
    data.

Returns
-------
TimeField
    A time field specification that can be passed to `Schema()`.

Raises
------
ValueError
    If `min_time` is later than `max_time`, or if a time string cannot be parsed.

Examples
--------
The `min_time=` and `max_time=` parameters accept `datetime.time` objects, making it
easy to define business-hours ranges:

```{python}
import pointblank as pb
from datetime import time

schema = pb.Schema(
    start_time=pb.time_field(
        min_time=time(9, 0, 0),
        max_time=time(12, 0, 0),
    ),
    end_time=pb.time_field(
        min_time=time(13, 0, 0),
        max_time=time(17, 0, 0),
    ),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

ISO format strings can also be used for convenience:

```{python}
schema = pb.Schema(
    login_time=pb.time_field(min_time="06:00:00", max_time="23:59:59"),
    alarm_time=pb.time_field(min_time="05:00:00", max_time="09:00:00"),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=23))
```

It's possible to introduce optional time values with `nullable=True` and combine them
with other field types:

```{python}
schema = pb.Schema(
    employee_id=pb.int_field(min_val=100, max_val=999, unique=True),
    check_in=pb.time_field(min_time="07:00:00", max_time="10:00:00"),
    check_out=pb.time_field(
        min_time="16:00:00", max_time="20:00:00",
        nullable=True, null_probability=0.15,
    ),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=7))
```

Every `check_in` time falls in the morning window and every non-missing `check_out` time in the
late afternoon or evening one, with 7 of the 30 `check_out` values missing. The `employee_id`
values are distinct three-digit numbers.

duration_field(min_duration: 'str | timedelta | None' = None, max_duration: 'str | timedelta | None' = None, nullable: 'bool' = False, null_probability: 'float' = 0.0, unique: 'bool' = False, generator: 'Callable[[], Any] | None' = None) -> 'DurationField'

Create a duration column specification for use in a schema.

The `duration_field()` function defines the constraints and behavior for a duration
(timedelta) column when generating synthetic data with `generate_dataset()`. You can
control the duration range with `min_duration=` and `max_duration=`, enforce uniqueness
with `unique=True`, and introduce null values with `nullable=True` and `null_probability=`.

Duration values are generated uniformly (at second-level resolution) within the specified
range. If no range is provided, the default range is 0 seconds to 30 days. Both
`min_duration=` and `max_duration=` accept `datetime.timedelta` objects or colon-separated
strings in `"HH:MM:SS"` or `"MM:SS"` format.

Parameters
----------
min_duration
    Minimum duration (inclusive). Can be a `"HH:MM:SS"` or `"MM:SS"` string, or a
    `datetime.timedelta` object. Default is `None` (defaults to 0 seconds).
max_duration
    Maximum duration (inclusive). Can be a `"HH:MM:SS"` or `"MM:SS"` string, or a
    `datetime.timedelta` object. Default is `None` (defaults to 30 days).
nullable
    Whether the column can contain null values. Default is `False`.
null_probability
    Probability of generating a null value for each row when `nullable=True`. Must be
    between `0.0` and `1.0`. Default is `0.0`.
unique
    Whether all values must be unique. Default is `False`. With second-level resolution
    within a duration range, uniqueness is feasible for moderate dataset sizes.
generator
    Custom callable that generates values. When provided, it's used instead of the other
    constraints, while `unique=` and `nullable=` still apply. The callable should take no
    arguments and return a single `datetime.timedelta` value. It isn't controlled by the `seed=`
    of `generate_dataset()`, so it needs its own seeded random number generator for reproducible
    data.

Returns
-------
DurationField
    A duration field specification that can be passed to `Schema()`.

Raises
------
ValueError
    If `min_duration` is greater than `max_duration`, or if a duration string cannot be
    parsed.

Examples
--------
The `min_duration=` and `max_duration=` parameters accept `timedelta` objects for
defining duration ranges:

```{python}
import pointblank as pb
from datetime import timedelta

schema = pb.Schema(
    session_length=pb.duration_field(
        min_duration=timedelta(minutes=5),
        max_duration=timedelta(hours=2),
    ),
    wait_time=pb.duration_field(
        min_duration=timedelta(seconds=30),
        max_duration=timedelta(minutes=15),
    ),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

Colon-separated strings can also be used for quick duration definitions:

```{python}
schema = pb.Schema(
    call_duration=pb.duration_field(min_duration="0:01:00", max_duration="1:30:00"),
    break_time=pb.duration_field(min_duration="0:05:00", max_duration="0:30:00"),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=23))
```

Optional durations can be created with `nullable=True`, and duration fields work well
alongside other field types:

```{python}
schema = pb.Schema(
    task_id=pb.int_field(min_val=1, max_val=500, unique=True),
    time_spent=pb.duration_field(
        min_duration=timedelta(minutes=1),
        max_duration=timedelta(hours=8),
    ),
    overtime=pb.duration_field(
        min_duration=timedelta(0),
        max_duration=timedelta(hours=4),
        nullable=True, null_probability=0.6,
    ),
)

pb.preview(pb.generate_dataset(schema, n=30, seed=7))
```

The `time_spent` values range from one minute to eight hours, while the `overtime` column is
missing in most rows (20 of 30 here, in line with the `null_probability=` of `0.6`). Both
columns have a duration dtype, so they can take part in arithmetic with dates and datetimes.

profile_fields(*, set: "Literal['minimal', 'standard', 'full']" = 'standard', split_name: 'bool' = True, include: 'list[str] | None' = None, exclude: 'list[str] | None' = None, prefix: 'str | None' = None) -> 'dict[str, StringField]'

Create a dict of string field specifications representing a person profile.

Returns a dictionary of `StringField` objects suitable for `**`-unpacking into a `Schema()`.
Each field uses a preset that participates in the existing coherence system, so generated
data will have coherent names, emails, addresses, and phone numbers within each row.

Parameters
----------
set
    The base set of profile fields to include. Options are `"minimal"` (name, email, phone;
    3-4 columns depending on `split_name=`), `"standard"` (name, email, city, state,
    postcode, phone; 6-7 columns), and `"full"` (name, email, address, city, state,
    postcode, phone, company, job; 9-10 columns). Default is `"standard"`.
split_name
    Whether to split the name into separate `first_name` and `last_name` columns (`True`,
    the default) or use a single combined `name` column (`False`).
include
    List of additional preset names to add to the base set. For example,
    `include=["company"]` adds a company column to the `"standard"` set. Presets already
    in the base set are silently ignored.
exclude
    List of preset names to remove from the (possibly augmented) set. For example,
    `exclude=["postcode"]` removes the postcode column. Presets not in the set are silently
    ignored.
prefix
    Optional string to prepend to every column name. For example, `prefix="customer_"`
    produces keys like `"customer_first_name"`, `"customer_email"`, etc. Each prefixed
    profile in a schema is a separate person, with its own name, location, and employer in
    each row. Other preset columns whose names start with the same prefix (e.g., a
    hand-written `"customer_city"` column) belong to that profile.

Returns
-------
dict[str, StringField]
    A dictionary mapping column names to `StringField` objects, ordered logically (name fields
    first, then contact, address, phone, business).

Raises
------
ValueError
    If `set=` is not one of `"minimal"`, `"standard"`, or `"full"`; if `include=` or `exclude=`
    contain unknown preset names; if a preset appears in both `include=` and `exclude=`; or if
    `include=` contains name presets incompatible with the `split_name=` setting.

Examples
--------
The default call returns the `"standard"` set of profile columns. The `**` operator unpacks the
returned dictionary directly into `Schema()`, as if each `string_field()` call had been written
by hand. All coherence rules apply automatically: emails are derived from names, and
city/state/postcode/phone are internally consistent.

```{python}
import pointblank as pb

schema = pb.Schema(
    user_id=pb.int_field(unique=True),
    **pb.profile_fields(),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

Use `set=` to control how many columns are generated. The `"minimal"` set includes only `name`,
`email`, and `phone`, while `"full"` adds `address`, `company`, and `job`. Setting
`split_name=False` collapses `first_name` and `last_name` into a single combined `name` column:

```{python}
schema = pb.Schema(
    **pb.profile_fields(set="minimal", split_name=False),
    balance=pb.float_field(min_val=0, max_val=10000),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23))
```

The `include=` and `exclude=` parameters let you customize the column set without switching to a
different base set. Here we start from the `"full"` set but drop the business columns:

```{python}
schema = pb.Schema(
    **pb.profile_fields(set="full", exclude=["company", "job"]),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23, country="DE"))
```

The `prefix=` parameter prepends a string to every column name, which helps when the profile
columns sit alongside other columns in a schema:

```{python}
schema = pb.Schema(
    **pb.profile_fields(set="minimal", prefix="customer_"),
    order_total=pb.float_field(min_val=5.0, max_val=500.0, precision=2),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23))
```

The emails are still derived from the names in each row. A schema can hold several prefixed
profiles, and each one is a different person:

```{python}
schema = pb.Schema(
    **pb.profile_fields(set="minimal", prefix="sender_"),
    **pb.profile_fields(set="minimal", prefix="recipient_"),
)

pb.preview(pb.generate_dataset(schema, n=50, seed=23))
```

Each sender's email matches the sender's name and each recipient's email matches the
recipient's name, but the sender and the recipient in a row are two independent people.


## Prebuilt Actions

Prebuilt action functions for common notification patterns.



send_slack_notification(webhook_url: 'str | None' = None, step_msg: 'str | None' = None, summary_msg: 'str | None' = None, debug: 'bool' = False) -> 'Callable | None'

Create an action that sends a Slack notification about a validation.

Slack is a common place for a team to hear about problems with its data. The
`send_slack_notification()` function creates an action that posts a message to a Slack channel
through an [incoming webhook](https://api.slack.com/messaging/webhooks). The same action works
in two places:

1. In [`Actions`](`pointblank.Actions`), it sends a message about a step that exceeded a
threshold level.
2. In [`FinalActions`](`pointblank.FinalActions`), it sends a summary of the whole validation
after the interrogation.

The action works out from where it runs which kind of message to send. The messages are made
from templates (`step_msg=` and `summary_msg=`), which can use Slack's Markdown formatting.

A notification that can't be sent (because of a wrong webhook URL, a network error or a
template with an unknown variable) doesn't stop the validation. The error is printed as
`Failed to send Slack notification: ...` instead of being raised.

Without a webhook URL, nothing is sent to Slack, which is useful for trying out templates:

- With `webhook_url=None`, the action prints the message that it would send.
- With `webhook_url=None` and `debug=True`, the function prints previews of both templates
filled in with sample values. It then returns `None` instead of an action.

Parameters
----------
webhook_url
    The URL of a Slack incoming webhook (`https://hooks.slack.com/services/...`). Since anyone
    with the URL can post to the channel, it's best read from an environment variable rather
    than written in the code. The default is `None`, which prints the messages instead of
    sending them.
step_msg
    The template for messages about a step, using the variables in *Template Variables for
    Step Messages* below. The default is `None`, which uses a template with the assertion
    type, the threshold level, the step number, the column and the time.
summary_msg
    The template for summary messages, using the variables in *Template Variables for Summary
    Messages* below. The default is `None`, which uses a template with the highest threshold
    level reached, the number of steps, the number of failing steps and the time.
debug
    Whether to print each message, and Slack's response, when it's sent. With
    `webhook_url=None`, the function prints previews of the templates instead (see above). The
    default is `False`.

Returns
-------
Callable or None
    The action, to use in `Actions` or `FinalActions`. With `webhook_url=None` and
    `debug=True`, the function returns `None` after printing the previews.

Template Variables for Step Messages
------------------------------------
A `step_msg=` template can use these variables:

- `"{step}"`: The step number.
- `"{column}"`: The column name.
- `"{value}"`: The value being compared (only available in certain validation steps).
- `"{type}"`: The assertion type (e.g., `"col_vals_gt"`, etc.).
- `"{level}"`: The threshold level (`"warning"`, `"error"`, or `"critical"`).
- `"{level_num}"`: The threshold level as a number (`30`, `40`, or `50`).
- `"{autobrief}"`: A localized and brief statement of the expectation for the step.
- `"{failure_text}"`: Localized text that explains how the validation step failed.
- `"{time}"`: The time of the notification.

These are the keys of [`get_action_metadata()`](`pointblank.get_action_metadata`)'s
dictionary, plus `"{time}"`.

Template Variables for Summary Messages
---------------------------------------
A `summary_msg=` template can use these variables:

- `"{n_steps}"`: The total number of validation steps.
- `"{n_passing_steps}"`: The number of validation steps where all test units passed.
- `"{n_failing_steps}"`: The number of validation steps that had some failing test units.
- `"{n_warning_steps}"`: The number of steps that exceeded a 'warning' threshold.
- `"{n_error_steps}"`: The number of steps that exceeded an 'error' threshold.
- `"{n_critical_steps}"`: The number of steps that exceeded a 'critical' threshold.
- `"{all_passed}"`: Whether or not every validation step had no failing test units.
- `"{highest_severity}"`: The highest threshold level reached in the validation: `"warning"`,
`"error"` or `"critical"`. If no threshold was exceeded, it is `"some failing"` when some
test units failed and `"all passed"` otherwise.
- `"{tbl_row_count}"`: The number of rows in the target table.
- `"{tbl_column_count}"`: The number of columns in the target table.
- `"{tbl_name}"`: The name of the target table (`"Unknown"` without a `tbl_name=`).
- `"{validation_duration}"`: The duration of the validation in seconds.
- `"{time}"`: The time of the notification.

These are keys of [`get_validation_summary()`](`pointblank.get_validation_summary`)'s
dictionary, plus `"{time}"`. Its other keys, such as `"{overall_health_score}"` and
`"{list_failing_steps}"`, can be used too. Since the templates are filled in with Python's
`str.format()`, a literal brace has to be doubled (`"{{"` or `"}}"`).

Examples
--------
Let's validate a small table of orders, which has a missing customer name and two amounts
that aren't positive. The action has no webhook URL, so it prints the message instead of
sending it. In `Actions(critical=)`, it runs for the steps that exceed the critical
threshold:

```{python}
import pointblank as pb
import polars as pl

orders = pl.DataFrame(
    {
        "amount": [29.99, 150.00, -5.00, 75.50, 0.00],
        "customer": ["Alice", "Bob", None, "Dana", "Eve"],
    }
)

notify_slack = pb.send_slack_notification()

validation = (
    pb.Validate(
        data=orders,
        tbl_name="orders",
        thresholds=(0.05, 0.20, 0.40),
        actions=pb.Actions(critical=notify_slack),
    )
    .col_vals_not_null(columns="customer")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)
```

Only the second step, with 40% of its test units failing, reached the critical threshold, so
there is one message. The first step (20% failing) exceeded the error threshold, which has no
action here.

In `FinalActions`, the same action sends a summary of the validation instead:

```{python}
validation = (
    pb.Validate(
        data=orders,
        tbl_name="orders",
        thresholds=(0.05, 0.20, 0.40),
        final_actions=pb.FinalActions(notify_slack),
    )
    .col_vals_not_null(columns="customer")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)
```

The summary is sent once per interrogation, whether or not any step failed. Both steps failed
here, and the highest threshold level reached was critical.

Custom templates can add more detail. With `webhook_url=None` and `debug=True`, the function
shows previews of the templates with sample values, without a validation:

```{python}
pb.send_slack_notification(
    step_msg=(
        "🚨 *Step {step} ({type})* exceeded the {level} threshold\n"
        "• Column: {column}\n"
        "• Details: {failure_text}"
    ),
    summary_msg=(
        "📊 *Validation of {tbl_name}*: {highest_severity}\n"
        "• Failing steps: {n_failing_steps} of {n_steps}\n"
        "• Rows: {tbl_row_count}"
    ),
    debug=True,
)
```

The sample values show where each variable goes. For a real channel, we pass the webhook URL
and use the action for both kinds of message. With `Actions(default=)`, a step message is sent
for every step that exceeds a threshold, at its highest level:

```python
import os

notify_slack = pb.send_slack_notification(webhook_url=os.environ["SLACK_WEBHOOK_URL"])

validation = (
    pb.Validate(
        data=orders,
        tbl_name="orders",
        thresholds=(0.05, 0.20, 0.40),
        actions=pb.Actions(default=notify_slack),
        final_actions=pb.FinalActions(notify_slack),
    )
    .col_vals_not_null(columns="customer")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)
```

This interrogation posts three messages to the channel: one for each failing step (at the
error and critical levels), then the summary. Reading the URL from the `SLACK_WEBHOOK_URL`
environment variable keeps it out of the code.

See Also
--------
Actions : Run actions when a step exceeds a threshold level.
FinalActions : Run actions after an interrogation.
get_action_metadata : Get the information that step messages are made from.
get_validation_summary : Get the information that summary messages are made from.
emit_otel : Send validation results to OpenTelemetry from a final action.

emit_otel(*, meter_name: 'str' = 'pointblank', meter_version: 'str | None' = None, enable_metrics: 'bool' = True, enable_tracing: 'bool' = False, enable_logging: 'bool' = False, meter_provider: 'Any | None' = None, tracer_provider: 'Any | None' = None, logger_provider: 'Any | None' = None, metric_prefix: 'str' = 'pb.validation', log_level: 'str' = 'warning', extra_attributes: 'dict[str, str] | None' = None) -> 'Callable'

Create a final action that sends a validation's results to OpenTelemetry.

The `emit_otel()` function returns an action for [`FinalActions`](`pointblank.FinalActions`)
that sends validation results to OpenTelemetry (OTel) at the end of every
[`interrogate()`](`pointblank.Validate.interrogate`) call. With the action in the validation
plan, every run reports to the monitoring backend without any code after the validation. That
suits validations that run on a schedule, such as checks in a data pipeline.

The action sends the same signals as
[`OTelExporter.export()`](`pointblank.integrations.otel.OTelExporter.export`) called right
after the interrogation: metrics (on by default), trace spans (`enable_tracing=True`) and log
records for the steps that exceeded a threshold (`enable_logging=True`), with the same names
and attributes. See [`OTelExporter`](`pointblank.integrations.otel.OTelExporter`) for the list
of signals. Since the signals are sent during `interrogate()`, the `pb.validate` span is a child
of whatever span is current at that point, such as the span of a pipeline task.

Without `meter_provider=` and `tracer_provider=`, the metrics and traces go to OTel's global
providers, which discard them until the application sets up the OTel SDK. Log records have no
global provider, so logging needs `logger_provider=`. The OTel packages are an optional
dependency, installed with `pip install pointblank[otel]`.

Parameters
----------
meter_name
    The name that identifies Pointblank as the source of the signals (the OTel instrumentation
    scope), used for the meter, the tracer and the logger. The default is `"pointblank"`.
meter_version
    The version of the instrumentation scope. The default is `None`, which uses the installed
    version of Pointblank.
enable_metrics
    Whether to send the metrics. The default is `True`.
enable_tracing
    Whether to send the trace spans. The default is `False`.
enable_logging
    Whether to send log records for the steps that exceeded a threshold. This needs
    `logger_provider=`. The default is `False`.
meter_provider
    An OTel `MeterProvider` for the metrics. The default is `None`, which uses the global meter
    provider.
tracer_provider
    An OTel `TracerProvider` for the trace spans. The default is `None`, which uses the global
    tracer provider.
logger_provider
    An OTel `LoggerProvider` for the log records, needed when `enable_logging=True`. The
    default is `None`.
metric_prefix
    The prefix for the names of the metrics. The default is `"pb.validation"`.
log_level
    The lowest threshold level that produces a log record: `"warning"` (every step that
    exceeded a threshold), `"error"` or `"critical"`. The default is `"warning"`.
extra_attributes
    A dictionary of attributes to add to every signal, such as the environment or the name of
    the pipeline. The default is `None`.

Returns
-------
Callable
    A function to use in `FinalActions`.

Raises
------
ImportError
    If the OTel packages aren't installed.
ValueError
    If `enable_logging=True` is given without `logger_provider=`, or if `log_level=` isn't one
    of the accepted values. Both are raised by `emit_otel()` itself, when the validation plan is
    built, rather than during `interrogate()`.

Examples
--------
Let's add the action to a validation of a small table of orders. The OTel SDK's
`InMemoryMetricReader` keeps the metrics in memory so we can look at them (in production,
the meter provider would send them to a collector instead):

```{python}
import pointblank as pb
import polars as pl
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader

reader = InMemoryMetricReader()
meter_provider = MeterProvider(metric_readers=[reader])

orders = pl.DataFrame(
    {
        "amount": [29.99, 150.00, -5.00, 75.50, 0.00],
        "customer": ["Alice", "Bob", None, "Dana", "Eve"],
    }
)

validation = (
    pb.Validate(
        data=orders,
        tbl_name="orders",
        thresholds=(0.05, 0.20, 0.40),
        final_actions=pb.FinalActions(
            pb.emit_otel(meter_provider=meter_provider, extra_attributes={"env": "dev"})
        ),
    )
    .col_vals_not_null(columns="customer")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)

metrics = reader.get_metrics_data().resource_metrics[0].scope_metrics[0].metrics

for metric in metrics:
    for point in metric.data.data_points:
        value = point.count if hasattr(point, "count") else point.value
        print(metric.name, value)
```

The metrics were sent during `interrogate()`. Both steps failed, 7 of the 10 test units passed,
and the second step exceeded all three thresholds. The step duration histogram has a data point
for each step, and the value printed for it is its count of one observation. Each data point
has the `pb.tbl_name` and `env` attributes:

```{python}
dict(metrics[0].data.data_points[0].attributes)
```

To also send traces, give the action `enable_tracing=True` and a tracer provider. Here, an
in-memory span exporter keeps the spans:

```{python}
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

span_exporter = InMemorySpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))

validation = (
    pb.Validate(
        data=orders,
        tbl_name="orders",
        thresholds=(0.05, 0.20, 0.40),
        final_actions=pb.FinalActions(
            pb.emit_otel(
                enable_metrics=False, enable_tracing=True, tracer_provider=tracer_provider
            )
        ),
    )
    .col_vals_not_null(columns="customer")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)

for span in span_exporter.get_finished_spans():
    print(span.name, span.attributes.get("pb.step.assertion_type"))
```

There's a `pb.validate.step` span for each step, inside the `pb.validate` span of the whole
interrogation, just as with `OTelExporter`.

Since `FinalActions` takes any number of actions, the signals can be sent along with other
notifications. Here, a [`send_slack_notification()`](`pointblank.send_slack_notification`)
action without a webhook URL prints the message it would send:

```{python}
validation = (
    pb.Validate(
        data=orders,
        tbl_name="orders",
        thresholds=(0.05, 0.20, 0.40),
        final_actions=pb.FinalActions(
            pb.emit_otel(meter_provider=meter_provider),
            pb.send_slack_notification(),
        ),
    )
    .col_vals_not_null(columns="customer")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)
```

The actions run in the order they're given, so the metrics are sent before the Slack message.
The message summarizes the same results as the metrics: both steps failed, and the highest
threshold level reached was critical.

See Also
--------
integrations.otel.OTelExporter : Send metrics, traces and log records after an interrogation.
FinalActions : Run actions after an interrogation.
send_slack_notification : Send a Slack notification from a final action.


## Metadata Import/Export

Import variable-level metadata from external data standards files (CDISC Define-XML, Controlled Terminology, SPSS `.sav`, SAS XPORT, Stata `.dta`, and more) and export metadata to various formats. Use `import_metadata()` as the entry point and `export_metadata()` for the reverse.



import_metadata(source: 'str | Path | Any', format: 'str | None' = None, **kwargs: 'Any') -> 'MetadataImport | MetadataPackage'

Import metadata from a statistical-package file or a metadata standard.

Many datasets come with a description of their variables: the variable labels, value labels
and missing-value codes stored in an SPSS, Stata or SAS file, the field types and constraints in
a Frictionless or CSVW schema, or the variable definitions and codelists in a CDISC Define-XML
document. The `import_metadata()` function reads that description (not the data itself) into a
[`MetadataImport`](`pointblank.MetadataImport`) object, with one
[`VariableMetadata`](`pointblank.VariableMetadata`) per variable and any codelists as
[`Codelist`](`pointblank.Codelist`) objects.

The imported metadata is most useful as the starting point for validation. Its
[`to_validate()`](`pointblank.MetadataImport.to_validate`) method turns the declared types and
constraints into a [`Validate`](`pointblank.Validate`) plan,
[`to_schema()`](`pointblank.MetadataImport.to_schema`) gives a
[`Schema`](`pointblank.Schema`) of the column types, and
[`missing_specs()`](`pointblank.MetadataImport.missing_specs`) turns declared missing-value
codes into [`MissingSpec`](`pointblank.MissingSpec`) objects. Use
[`export_metadata()`](`pointblank.export_metadata`) to go the other way and write metadata out
as a Frictionless Table Schema.

A source that describes several datasets (a Define-XML document for a whole study, a Data
Package with several resources, or a Controlled Terminology file with several codelists) gives a
[`MetadataPackage`](`pointblank.MetadataPackage`) instead, which holds one `MetadataImport` per
dataset.

Parameters
----------
source
    The path to the metadata file, as a string or a `pathlib.Path`. Other kinds of objects
    raise a `TypeError`.
format
    The format of the file. If `None` (the default), the format is detected from the file: by
    extension for `.sav`, `.zsav`, `.dta` and `.xpt` files, and by content for `.json` files
    (Frictionless or CSVW) and `.xml` files (Define-XML or Controlled Terminology). Other
    extensions need an explicit format. See the *Supported Formats* section for the values.
**kwargs
    Options for the reader of the chosen format. The Define-XML reader takes `dataset=` (such
    as `"DM"`) to return only that dataset as a `MetadataImport`. The Controlled Terminology
    reader takes `codelist=` to return only one codelist, and the Frictionless reader takes
    `resource=` (a name or an index) to pick one resource from a Data Package. The
    `"cdisc_sdtm"` and `"cdisc_adam"` formats need `domain=` and `dataset=`, respectively, and
    also take `study_id=`.

Returns
-------
MetadataImport | MetadataPackage
    A `MetadataImport` for a source that describes a single dataset, or a `MetadataPackage`
    for a source that describes several.

Raises
------
ValueError
    If the format can't be detected or isn't supported, or if a required option such as
    `domain=` is missing.
FileNotFoundError
    If the file doesn't exist.
ImportError
    If the format needs an optional package that isn't installed.

Supported Formats
-----------------
| `format=` | Reads | Needs |
|-----------|-------|-------|
| `"frictionless"` (or `"datapackage"`, `"table_schema"`) | Frictionless schema (`.json`) | |
| `"csvw"` | W3C CSV on the Web metadata (`.json`) | |
| `"cdisc_define"` (or `"define_xml"`) | CDISC Define-XML 2.0 or 2.1 (`.xml`) | `lxml` |
| `"cdisc_ct"` | CDISC Controlled Terminology in ODM-XML (`.xml`) | `lxml` |
| `"spss"` (or `"sav"`) | SPSS data file (`.sav`, `.zsav`) | `pyreadstat` |
| `"stata"` (or `"dta"`) | Stata data file (`.dta`) | `pyreadstat` |
| `"xpt"` (or `"sas"`) | SAS Transport file (`.xpt`) | `pyreadstat` |
| `"cdisc_sdtm"` | A built-in SDTM domain template | |
| `"cdisc_adam"` | A built-in ADaM dataset template | |

For SPSS, Stata and SAS files only the metadata is read, not the data. The last two formats
don't read a file at all: `source=` isn't used (any string will do), and the template named
by `domain=` or `dataset=` is returned, as with
[`sdtm_to_metadata()`](`pointblank.sdtm_to_metadata`) and
[`adam_to_metadata()`](`pointblank.adam_to_metadata`).

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Pointblank bundles a few metadata files to experiment with, which can be found with
[`load_metadata_example()`](`pointblank.load_metadata_example`). Let's import a Frictionless
Data Package that describes a table of sales transactions:

```{python}
import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("datapackage.json"))

meta
```

The format was detected from the file's contents. The metadata describes eight variables and
one codelist (the allowed values of `category`). Each variable's metadata can be looked up by
name:

```{python}
amount = meta.get_variable("amount")

print(amount.dtype, amount.required, amount.min_val, amount.max_val)
```

The `amount` variable must be a non-null float from `0.01` to `99999.99`. Constraints like these
can be checked against a table with
[`to_validate()`](`pointblank.MetadataImport.to_validate`), which builds a validation plan from
the metadata:

```{python}
import datetime
import polars as pl

transactions = pl.DataFrame(
    {
        "transaction_id": ["TXN-00001", "TXN-00002", "TXN-00003", "TXN-00004"],
        "customer_id": ["CUST-101", "CUST-102", "CUST-101", "CUST-103"],
        "amount": [25.50, 1200.00, 0.00, 89.99],
        "quantity": [1, 3, 2, 1],
        "category": ["food", "electronics", "home", "toys"],
        "sale_date": [datetime.date(2024, 5, d) for d in (1, 1, 2, 3)],
        "discount_pct": [0.0, 10.0, 0.0, 5.0],
        "email": ["a@example.com", "b@example.com", "c@example.com", "not-an-email"],
    }
)

meta.to_validate(data=transactions).interrogate()
```

The table has the declared column types, so the schema check in step 1 passes. Three
later steps find a failing row each: the `amount` of `0.00` is below the minimum, `"toys"`
isn't one of the allowed categories, and `"not-an-email"` doesn't match the pattern given for
`email`.

A Define-XML document usually describes all the datasets in a study, so importing one gives a
[`MetadataPackage`](`pointblank.MetadataPackage`):

```{python}
package = pb.import_metadata(pb.load_metadata_example("define.xml"))

package
```

The package holds the Demographics (`DM`) and Adverse Events (`AE`) datasets, and
`package["DM"]` gives the `MetadataImport` for one of them. If only one dataset is needed, the
`dataset=` option of the Define-XML reader returns it directly:

```{python}
dm = pb.import_metadata(pb.load_metadata_example("define.xml"), dataset="DM")

print(dm.dataset_label)
print(dm.variable_names)
```

Options like `dataset=` are passed through to the reader for the file's format. The
*Supported Formats* section lists the formats, and the `**kwargs` parameter describes the
options that each reader accepts.

See Also
--------
load_metadata_example : Get the path to a bundled metadata file.
export_metadata : Export metadata as a Frictionless Table Schema.
MetadataImport : The metadata for a single dataset.
MetadataPackage : The metadata for several datasets.

export_metadata(source: 'MetadataImport', destination: 'str | Path | None' = None, format: 'str' = 'frictionless', **kwargs: 'Any') -> 'dict[str, Any] | str'

Export metadata as a Frictionless Table Schema.

The `export_metadata()` function is the reverse of
[`import_metadata()`](`pointblank.import_metadata`): it writes a
[`MetadataImport`](`pointblank.MetadataImport`) out in a standard format. The only format
currently supported is the
[Frictionless Table Schema](https://specs.frictionlessdata.io/table-schema/), a JSON
description of a table's fields that many data tools can read. Because the metadata can
come from any of the formats that `import_metadata()` reads, this is also a way to convert
between them, for example to get a Table Schema from a CSVW file or from the variable
definitions in a Define-XML document.

Each variable becomes a field, with its dtype mapped to a Frictionless type (`"integer"`,
`"number"`, `"string"`, `"date"` and so on) and its label and description used as the field's
`title` and `description`. The constraints `required`, `unique`, `minimum`, `maximum`,
`minLength`, `maxLength`, `pattern` and `enum` are written for the variables that have them,
and declared missing values become the field's `missingValues`. Variables that are both
required and unique make up the schema's `primaryKey`, and the dataset label and description
become the schema's `title` and `description`.

Parameters
----------
source
    The metadata to export. To export one dataset from a
    [`MetadataPackage`](`pointblank.MetadataPackage`), select it first (as in
    `package["DM"]`).
destination
    A file path to write the schema to, as indented JSON. Any missing parent directories are
    created. If `None` (the default), nothing is written. The schema is returned in both cases.
format
    The output format. Only `"frictionless"` is supported (`"table_schema"` is accepted as
    another name for it).
**kwargs
    Options for the exporter. `include_constraints=False` leaves out the `constraints` of
    every field, giving a schema with only the field names, types, titles and missing values.

Returns
-------
dict
    The Table Schema as a dictionary.

Raises
------
ValueError
    If the format isn't supported.

Examples
--------
Let's import the bundled CSVW metadata for a table of weather observations and export it as a
Frictionless Table Schema:

```{python}
import json
import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("weather_csvw.json"))

schema = pb.export_metadata(meta)

print(json.dumps(schema["fields"][:3], indent=2))
```

The first three fields show how the CSVW column definitions were carried over: the titles,
the types (a CSVW `decimal` is a Frictionless `number`), and the constraints. The remaining
keys of the schema describe the table as a whole:

```{python}
print(schema["primaryKey"])
print(schema["title"])
```

The two required and unique columns form the primary key, and the title comes from the
dataset label in the CSVW file.

Give a `destination=` to write the schema to a file. Here it's written to a temporary
directory and then imported again:

```{python}
import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "weather_schema.json"

    pb.export_metadata(meta, destination=path, include_constraints=False)

    reimported = pb.import_metadata(path)

print(reimported.variable_names)
print(reimported.get_variable("temperature_c").min_val)
```

The file is a standalone Table Schema, so `import_metadata()` reads it back with the same
variables. With `include_constraints=False` the range of `temperature_c` wasn't written, so
its `min_val` is now `None`. That's useful when you want to share the structure of a table
without the rules for its values.

See Also
--------
import_metadata : Read metadata from a file.
MetadataImport : The metadata that is exported.

load_metadata_example(name: 'str') -> 'Path'

Get the path to a bundled metadata example file.

Pointblank ships a small collection of metadata files in the formats that
[`import_metadata()`](`pointblank.import_metadata`) reads: CDISC Define-XML, CDISC Controlled
Terminology, a Frictionless Data Package and Table Schema, CSVW, and SAS Transport (XPT). This
function returns the path to one of them, so you can try out metadata import (and run the
examples in this documentation) without supplying files of your own.

Parameters
----------
name
    The file name of the example, such as `"define.xml"`. See the *Available Examples* section
    for the full list. An unknown name raises a `ValueError` that lists the valid names.

Returns
-------
Path
    The path to the bundled file, which can be passed directly to `import_metadata()`.

Available Examples
------------------
| Name | Format | Description |
|------|--------|-------------|
| `"define.xml"` | `cdisc_define` | Define-XML 2.0 document with the DM and AE domains |
| `"sdtm_ct.xml"` | `cdisc_ct` | SDTM Controlled Terminology with common codelists |
| `"datapackage.json"` | `frictionless` | Frictionless Data Package for a transactions table |
| `"table_schema.json"` | `table_schema` | Frictionless Table Schema for sensor readings |
| `"weather_csvw.json"` | `csvw` | W3C CSVW metadata for weather observations |
| `"dm.xpt"` | `xpt` | SAS Transport file with a small SDTM Demographics dataset |

The XML formats need the `lxml` package and the XPT file needs `pyreadstat`. The JSON formats
don't need any extra packages.

Examples
--------
Get the path to the bundled Define-XML document:

```{python}
import pointblank as pb

define_path = pb.load_metadata_example("define.xml")

print(define_path.name)
print(define_path.exists())
```

The result is a `pathlib.Path` to a file inside the installed package. Pass it to
[`import_metadata()`](`pointblank.import_metadata`) to read the metadata:

```{python}
package = pb.import_metadata(define_path)

for name in package.keys():
    meta = package[name]
    print(f"{name}: {meta.dataset_label} ({len(meta)} variables)")
```

The format was detected from the file's contents, and because the document describes two
datasets the result is a [`MetadataPackage`](`pointblank.MetadataPackage`) with one entry per
dataset.

See Also
--------
import_metadata : Read metadata from a file, such as one of these examples.

MetadataImport(source_format: 'str', source_path: 'str | None' = None, source_version: 'str | None' = None, dataset_name: 'str | None' = None, dataset_label: 'str | None' = None, dataset_description: 'str | None' = None, creation_date: 'str | None' = None, study_id: 'str | None' = None, domain: 'str | None' = None, variables: 'list[VariableMetadata]' = <factory>, codelists: 'dict[str, Codelist]' = <factory>, missing_value_codes: 'dict[str, list[MissingValueCode]]' = <factory>) -> None

The metadata for a single dataset.

A `MetadataImport` is what [`import_metadata()`](`pointblank.import_metadata`) returns for a
file that describes one dataset. It holds the dataset-level information (name, label, study
and domain), a [`VariableMetadata`](`pointblank.VariableMetadata`) for each variable, the
codelists that the variables use, and any declared missing-value codes, all in the same form
whatever the format of the file.

The methods turn the metadata into things Pointblank can use:
[`to_validate()`](`pointblank.MetadataImport.to_validate`) builds a validation plan from the
declared types and constraints, [`to_schema()`](`pointblank.MetadataImport.to_schema`) gives a
[`Schema`](`pointblank.Schema`) of the column types, and
[`missing_specs()`](`pointblank.MetadataImport.missing_specs`) builds
[`MissingSpec`](`pointblank.MissingSpec`) objects from the missing-value codes. Variables and
codelists can be looked up with [`get_variable()`](`pointblank.MetadataImport.get_variable`)
and [`get_codelist()`](`pointblank.MetadataImport.get_codelist`), and
[`summary()`](`pointblank.MetadataImport.summary`) describes the whole dataset in text.

A `MetadataImport` can also be created directly, which is a way to write down the rules for a
dataset that doesn't come with a metadata file and still use the methods above. Calling
`len()` on it gives the number of variables.

Parameters
----------
source_format
    The format the metadata was read from, such as `"frictionless"`, `"cdisc_define"` or
    `"spss"`. Any string can be used for metadata created directly.
source_path
    The path of the file that was read. The default is `None`.
source_version
    The version of the format or standard, such as `"Define-XML 2.0"`. The default is `None`.
dataset_name
    The name of the dataset, such as `"DM"`. The default is `None`.
dataset_label
    A human-readable label for the dataset, such as `"Demographics"`. The default is `None`.
dataset_description
    A longer description of the dataset. The default is `None`.
creation_date
    When the metadata was created, as given in the file. The default is `None`.
study_id
    The study identifier, for clinical data. The default is `None`.
domain
    The CDISC domain, such as `"DM"` or `"AE"`. The default is `None`.
variables
    The metadata of each variable, as a list of
    [`VariableMetadata`](`pointblank.VariableMetadata`) objects. The default is an empty list.
codelists
    The codelists used by the variables, as a dictionary from codelist name to
    [`Codelist`](`pointblank.Codelist`). The default is an empty dictionary.
missing_value_codes
    The declared missing-value codes, as a dictionary from column name to a list of
    [`MissingValueCode`](`pointblank.MissingValueCode`) objects. The default is an empty
    dictionary.

Examples
--------
Let's import the bundled Frictionless Table Schema, which describes a table of sensor
readings:

```{python}
import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("table_schema.json"))

print(len(meta))
print(meta.variable_names)
print(list(meta.codelists))
```

The schema describes six variables, and the allowed values of `status` were read into a
codelist named `"status_enum"`. Metadata can also be written down directly, here for a small
survey table:

```{python}
survey = pb.MetadataImport(
    source_format="custom",
    dataset_name="survey",
    dataset_label="Household survey",
    variables=[
        pb.VariableMetadata(name="respondent_id", dtype="String", required=True, unique=True),
        pb.VariableMetadata(name="age", dtype="Int64", required=True, min_val=18, max_val=99),
    ],
)

survey
```

Either way, the result can be turned into a validation plan with
[`to_validate()`](`pointblank.MetadataImport.to_validate`), which has an example of checking a
table against its metadata.

See Also
--------
import_metadata : Read a `MetadataImport` from a file.
MetadataPackage : The metadata for several datasets.
VariableMetadata : The metadata of one variable.

MetadataPackage(name: 'str | None' = None, items: 'dict[str, MetadataImport]' = <factory>, description: 'str | None' = None, version: 'str | None' = None) -> None

The metadata for several datasets, read from one source.

When a file describes more than one dataset,
[`import_metadata()`](`pointblank.import_metadata`) returns a `MetadataPackage` with a
[`MetadataImport`](`pointblank.MetadataImport`) for each. This happens with a Define-XML
document for a whole study (one entry per dataset, such as `"DM"` and `"AE"`), a Frictionless
Data Package or CSVW table group with several tables, and a CDISC Controlled Terminology file
(one entry per codelist).

The package works much like a read-only dictionary: `package["DM"]` gets an entry, `"DM" in
package` checks for one, `len()` counts them, iterating gives their names, and
[`keys()`](`pointblank.MetadataPackage.keys`) and
[`values()`](`pointblank.MetadataPackage.values`) give the names and entries.
[`get_domain()`](`pointblank.MetadataPackage.get_domain`) also gets an entry, with an error
message that lists the available names, and
[`summary()`](`pointblank.MetadataPackage.summary`) describes the package in text.

Parameters
----------
name
    The name of the package, such as the study OID of a Define-XML document. The default is
    `None`.
items
    The entries, as a dictionary from dataset name to `MetadataImport`. The default is an
    empty dictionary.
description
    A description of the package. The default is `None`.
version
    The version of the package or of the standard it follows. The default is `None`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
The bundled Define-XML document describes two datasets of a clinical study:

```{python}
import pointblank as pb

package = pb.import_metadata(pb.load_metadata_example("define.xml"))

print(len(package))
print("DM" in package)

for name in package:
    print(name, "-", package[name].dataset_label)
```

Each entry is a `MetadataImport`, so it can be used like one. Here the Demographics metadata
is used to check a small table:

```{python}
import polars as pl

dm_data = pl.DataFrame(
    {
        "STUDYID": ["XYZ789"] * 3,
        "DOMAIN": ["DM"] * 3,
        "USUBJID": ["XYZ789-001", "XYZ789-002", "XYZ789-003"],
        "SUBJID": ["001", "002", "003"],
        "RFSTDTC": ["2024-01-15", "2024-01-20", "2024-02-01"],
        "RFENDTC": ["2024-06-15", "2024-06-20", "2024-07-01"],
        "SITEID": ["SITE01", "SITE01", "SITE02"],
        "AGE": [45, 62, 38],
        "AGEU": ["YEARS"] * 3,
        "SEX": ["M", "F", "M"],
        "RACE": ["WHITE", "ASIAN", "BLACK"],
        "ARMCD": ["DRUG", "PLACEBO", "DRUG"],
        "ARM": ["Active Drug 10mg", "Placebo", "Active Drug 10mg"],
    }
)

package["DM"].to_validate(data=dm_data).interrogate()
```

The table has all the variables of the `DM` dataset with the declared types, and the required
ones have values. The last step fails for one row, since `"BLACK"` isn't one of the values in
the `Race` codelist (which uses `"BLACK OR AFRICAN AMERICAN"`).

See Also
--------
import_metadata : Read metadata from a file.
MetadataImport : The metadata for a single dataset.

VariableMetadata(name: 'str', label: 'str | None' = None, description: 'str | None' = None, dtype: 'str | None' = None, role: 'str | None' = None, required: 'bool' = False, unique: 'bool' = False, min_val: 'float | None' = None, max_val: 'float | None' = None, min_length: 'int | None' = None, max_length: 'int | None' = None, pattern: 'str | None' = None, allowed_values: 'list[Any] | None' = None, codelist_ref: 'str | None' = None, display_format: 'str | None' = None, value_labels: 'dict[Any, str] | None' = None, missing_values: 'list[Any] | None' = None, missing_value_labels: 'dict[Any, str] | None' = None, origin: 'str | None' = None, computational_method: 'str | None' = None, controlled_term: 'str | None' = None, significant_digits: 'int | None' = None, cdisc_domain: 'str | None' = None, cdisc_role: 'str | None' = None, adam_derivation: 'str | None' = None, traceability_ref: 'str | None' = None, unit: 'str | None' = None, unit_system: 'str | None' = None) -> None

The metadata for a single variable (column).

A [`MetadataImport`](`pointblank.MetadataImport`) holds one `VariableMetadata` for each
variable described in the imported file. It gathers everything the file says about the
variable in one place: its name, label and type, the constraints on its values, any value
labels and missing-value codes, and (for clinical data) its CDISC attributes. Each format fills
in the fields it has information for and leaves the rest as `None`.

Some fields are used to generate validation steps in
[`MetadataImport.to_validate()`](`pointblank.MetadataImport.to_validate`): `dtype` (in the
schema check), `required`, `unique`, `min_val` and `max_val`, `allowed_values`, `pattern` and
`missing_values`. All of the constraint fields, including `min_length` and `max_length`, are
written out by [`export_metadata()`](`pointblank.export_metadata`). The other fields are
descriptive, and are kept so that you can use them in your own checks or reports.

Parameters
----------
name
    The name of the variable, which should match the column name in the data.
label
    A short human-readable label, such as `"Age in years"`. The default is `None`.
description
    A longer description of the variable. The default is `None`.
dtype
    The data type, as a Pointblank dtype name such as `"String"`, `"Int64"`, `"Float64"`,
    `"Boolean"`, `"Date"` or `"Datetime"`. The default is `None` (an unknown type, treated as
    `"String"` when a schema is made).
role
    The role of the variable in the dataset, as given by the source. The default is `None`.
required
    Whether the variable must have a value in every row. The default is `False`.
unique
    Whether every value must be different. The default is `False`.
min_val
    The smallest allowed value (inclusive). The default is `None`.
max_val
    The largest allowed value (inclusive). The default is `None`.
min_length
    The shortest allowed string length. The default is `None`.
max_length
    The longest allowed string length, such as the `Length` of a Define-XML variable. The
    default is `None`.
pattern
    A regular expression that values must match. The default is `None`.
allowed_values
    The list of values the variable may take, usually the values of its codelist. The default
    is `None`.
codelist_ref
    The name of the variable's [`Codelist`](`pointblank.Codelist`) in
    `MetadataImport.codelists`. The default is `None`.
display_format
    The display format from the source system, such as the SPSS format `"F8.2"`. The default
    is `None`.
value_labels
    A mapping of values to labels, such as `{1: "Male", 2: "Female"}` for an SPSS variable. The
    default is `None`.
missing_values
    The values that stand for missing data, such as `[-99, -98]` or `["NA"]`. The default is
    `None`.
missing_value_labels
    A mapping of missing values to labels, such as `{-99: "Refused"}`. The default is `None`.
origin
    How the variable's values were obtained, such as `"CRF"`, `"Derived"` or `"Assigned"` in
    Define-XML. The default is `None`.
computational_method
    How a derived variable is computed. The default is `None`.
controlled_term
    The name of the CDISC Controlled Terminology codelist the variable uses. The default is
    `None`.
significant_digits
    The number of significant digits of a numeric variable. The default is `None`.
cdisc_domain
    The CDISC domain or dataset that the variable belongs to, such as `"DM"` or `"AE"`. The
    default is `None`.
cdisc_role
    The CDISC role of the variable, such as `"Identifier"`, `"Topic"` or `"Record Qualifier"`.
    The default is `None`.
adam_derivation
    The source or derivation of an ADaM variable. The default is `None`.
traceability_ref
    A reference from an ADaM variable back to its SDTM source. The default is `None`.
unit
    The unit of measurement, such as `"kg"` or `"years"`. The default is `None`.
unit_system
    The system of units, such as `"SI"`. The default is `None`.

Examples
--------
Let's look up a variable in the bundled Frictionless Data Package with
[`get_variable()`](`pointblank.MetadataImport.get_variable`):

```{python}
import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("datapackage.json"))
var = meta.get_variable("transaction_id")

print(var.dtype, var.required, var.unique, var.min_length, var.max_length)
print(var.missing_values)
```

The `transaction_id` field is a required and unique string of 5 to 20 characters, and the
package declares `"NA"` and `"N/A"` as its missing-value markers. The CDISC fields are filled in
for variables read from a Define-XML document:

```{python}
dm = pb.import_metadata(pb.load_metadata_example("define.xml"), dataset="DM")
sex = dm.get_variable("SEX")

print(sex.label, sex.max_length, sex.codelist_ref)
print(sex.cdisc_domain, sex.cdisc_role)
print(sex.allowed_values)
```

The `SEX` variable uses the `Sex` codelist, whose values were copied into `allowed_values`.
`VariableMetadata` objects can also be created directly, which is how a
[`MetadataImport`](`pointblank.MetadataImport`) can be built for data that doesn't come with a
metadata file:

```{python}
pb.VariableMetadata(name="age", label="Age in years", dtype="Int64", min_val=18, max_val=99)
```

Only the fields that are given get a value, and the rest stay at their defaults.

See Also
--------
MetadataImport : The metadata for a dataset, with one of these per variable.
MetadataImport.get_variable : Get the metadata of a variable by name.

Codelist(name: 'str', codes: 'list[CodelistEntry]' = <factory>, label: 'str | None' = None, version: 'str | None' = None, source: 'str | None' = None, extensible: 'bool' = False) -> None

A set of permitted values, each with a label.

A codelist (also called a value set or controlled terminology) lists the values that a
variable may take and what each one means. When
[`import_metadata()`](`pointblank.import_metadata`) reads a file, the codelists it finds become
`Codelist` objects in [`MetadataImport.codelists`](`pointblank.MetadataImport`): the CodeList
elements of a Define-XML document, the codelists of a CDISC Controlled Terminology file, the
`enum` constraint of a Frictionless field (as a codelist named `"<field>_enum"`), and the value
labels of an SPSS variable (as `"<variable>_values"`). Variables that use a codelist name it in
their [`codelist_ref`](`pointblank.VariableMetadata`) and have its values in `allowed_values`.

A codelist's values are what a [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`)
check needs, and [`to_set()`](`pointblank.Codelist.to_set`) gives them as a list.
[`to_dict()`](`pointblank.Codelist.to_dict`) maps each value to its label, and
[`extended_values()`](`pointblank.Codelist.extended_values`) lists the sponsor extensions of an
extensible codelist.

Parameters
----------
name
    The name of the codelist, which is also its key in `MetadataImport.codelists`.
codes
    The entries of the codelist, as [`CodelistEntry`](`pointblank.CodelistEntry`) objects. The
    default is an empty list.
label
    A human-readable name for the codelist. The default is `None`.
version
    The version of the terminology, such as a CDISC Controlled Terminology release date. The
    default is `None`.
source
    Where the codelist came from, such as `"CDISC Define-XML"` or `"SPSS .sav"`. The default
    is `None`.
extensible
    Whether values that aren't in the codelist may be added, as with CDISC codelists marked as
    extensible. A non-extensible codelist is a closed set. The default is `False`.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
The bundled Define-XML document defines a `Sex` codelist for the Demographics dataset. Let's
get it from the imported metadata:

```{python}
import pointblank as pb

dm = pb.import_metadata(pb.load_metadata_example("define.xml"), dataset="DM")
sex = dm.get_codelist("Sex")

print(sex.label, sex.source, sex.extensible, len(sex))
print(sex.to_dict())
```

The codelist has three entries and isn't extensible, so `M`, `F` and `U` are the only values
allowed. Its values can be used in a validation step directly:

```{python}
import polars as pl

tbl = pl.DataFrame({"SEX": ["M", "F", "F", "X"]})

(
    pb.Validate(data=tbl)
    .col_vals_in_set(columns="SEX", set=sex.to_set())
    .interrogate()
)
```

The `"X"` in the last row isn't in the codelist, so one test unit fails. A codelist can also be
built by hand:

```{python}
route = pb.Codelist(
    name="ROUTE",
    label="Route of Administration",
    extensible=True,
    codes=[
        pb.CodelistEntry(value="ORAL", label="Oral"),
        pb.CodelistEntry(value="INTRAVENOUS", label="Intravenous"),
        pb.CodelistEntry(value="IV", label="Intravenous (old code)", is_deprecated=True),
        pb.CodelistEntry(value="NASOGASTRIC TUBE", label="Nasogastric tube", is_extended=True),
    ],
)

print(len(route))
print(route.to_set())
print(route.extended_values())
```

`len()` counts all four entries, but `to_set()` leaves out the deprecated `"IV"`, and
`extended_values()` gives the one value that was added to the extensible codelist.

See Also
--------
CodelistEntry : A single entry in a codelist.
MetadataImport.get_codelist : Get a codelist from imported metadata.

CodelistEntry(value: 'Any', label: 'str', description: 'str | None' = None, synonyms: 'list[str] | None' = None, is_deprecated: 'bool' = False, is_extended: 'bool' = False) -> None

A single permitted value in a codelist.

A [`Codelist`](`pointblank.Codelist`) is a list of `CodelistEntry` objects, each pairing a coded
value (the value that appears in the data, such as `"F"`) with a label that says what it means
(such as `"Female"`). Codelist entries are created when
[`import_metadata()`](`pointblank.import_metadata`) reads a codelist, and can also be created
directly to build a codelist by hand.

Parameters
----------
value
    The coded value, as it appears in the data.
label
    A human-readable label for the value.
description
    A longer description of the entry. The default is `None`.
synonyms
    Other terms with the same meaning, such as the synonyms listed in CDISC Controlled
    Terminology. The default is `None`.
is_deprecated
    Whether the entry has been retired. A deprecated entry stays in the codelist (and in
    [`to_dict()`](`pointblank.Codelist.to_dict`)) but is left out of the permitted values given
    by [`to_set()`](`pointblank.Codelist.to_set`). The readers don't currently set this flag,
    but you can for codelists you build yourself. The default is `False`.
is_extended
    Whether the entry is a sponsor extension to an extensible codelist: a value that isn't in
    the published terminology but was added for a study. Define-XML marks these with
    `def:ExtendedValue="Yes"`. The default is `False`.

Examples
--------
Each codelist imported from a file holds its entries in `codes`. Here are the entries of the
`Sex` codelist in the bundled CDISC Controlled Terminology file:

```{python}
import pointblank as pb

ct = pb.import_metadata(pb.load_metadata_example("sdtm_ct.xml"), codelist="Sex")
sex = ct.get_codelist("Sex")

for entry in sex.codes:
    print(entry.value, "->", entry.label)
```

The values are the codes used in the data, and the labels say what they mean. An entry can
also be created directly, for example to build a codelist that isn't in any file:

```{python}
entry = pb.CodelistEntry(value="NASOGASTRIC TUBE", label="Nasogastric tube", is_extended=True)

entry
```

Setting `is_extended=True` records that this value was added to an extensible codelist rather
than taken from the published terminology, so
[`extended_values()`](`pointblank.Codelist.extended_values`) will list it.

See Also
--------
Codelist : The list of entries that a codelist is made of.

MissingValueCode(value: 'Any', label: 'str', category: 'str | None' = None, reason: 'str | None' = None) -> None

A declared missing-value code, as read from a metadata file.

Many data formats let a dataset declare the values that stand for missing data: the
user-defined missing values of an SPSS variable (such as `-99` for "Refused"), or the
`missingValues` and `null` markers of a Frictionless or CSVW schema (such as `"NA"`). When
[`import_metadata()`](`pointblank.import_metadata`) reads these declarations, it records each
one as a `MissingValueCode` in
[`MetadataImport.missing_value_codes`](`pointblank.MetadataImport`), a dictionary from column
name to that column's list of codes.

These objects describe the codes. To use the codes in validation, build
[`MissingSpec`](`pointblank.MissingSpec`) objects with
[`MetadataImport.missing_specs()`](`pointblank.MetadataImport.missing_specs`), which works from
each variable's [`missing_values`](`pointblank.VariableMetadata`) and their labels.

Parameters
----------
value
    The value that stands for missing data, such as `-99` or `"NA"`.
label
    What the code means, such as `"Refused"`.
category
    The kind of missingness. The readers use `"user_missing"` for values declared in the
    file. The default is `None`.
reason
    Why the data is missing, if known. The default is `None`.

Examples
--------
The bundled Frictionless Data Package declares `"NA"` and `"N/A"` as missing-value markers for
every field. Let's look at the codes recorded for the `amount` column:

```{python}
import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("datapackage.json"))

for code in meta.missing_value_codes["amount"]:
    print(code.value, "|", code.label, "|", code.category)
```

Each marker is recorded with a generated label and the `"user_missing"` category, since it was
declared in the file. A code can also be created directly:

```{python}
pb.MissingValueCode(value=-99, label="Refused", category="user_missing", reason="refused")
```

Creating codes yourself is mainly useful when you build a
[`MetadataImport`](`pointblank.MetadataImport`) by hand to describe a dataset that doesn't come
with a metadata file.

See Also
--------
MetadataImport.missing_specs : Build missing-value specs from imported metadata.
MissingSpec : Describe missing-value codes for use in validation.


## SDTM Validation

Validate clinical datasets against CDISC SDTM domain templates. Use `validate_sdtm()` to generate a full `Validate` workflow, or `validate_sdtm_structure()` for a quick structural conformance check. Retrieve domain templates with `get_sdtm_domain()` and `list_sdtm_domains()`.



validate_sdtm(data: 'Any', domain: 'str', study_id: 'str | None' = None, check_dates: 'bool' = True, check_lengths: 'bool' = True, label: 'str | None' = None, **kwargs: 'Any')

Build a validation plan for a dataset from an SDTM domain template.

The `validate_sdtm()` function reads the template for an SDTM domain (see
[`get_sdtm_domain()`](`pointblank.get_sdtm_domain`)) and turns it into a
[`Validate`](`pointblank.Validate`) plan for a dataset of that domain. The plan isn't
interrogated, so more steps can be added to it before calling
[`interrogate()`](`pointblank.Validate.interrogate`).

The steps are generated in this order, for the template variables that are columns of the
data:

1. [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`) for each required variable
(one with a core designation of `"Req"`).
2. [`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`) to check that `DOMAIN` holds
only the domain code.
3. [`col_vals_gt()`](`pointblank.Validate.col_vals_gt`) to check that the sequence number (as in
`AESEQ`) is greater than `0`. DM has one record per subject and so has no sequence number.
4. If `check_lengths=True`, a [`col_vals_expr()`](`pointblank.Validate.col_vals_expr`) step for
each character variable, checking that its values are no longer than the template's maximum
length. Null values pass.
5. If `check_dates=True`, a [`col_vals_regex()`](`pointblank.Validate.col_vals_regex`) step for
each `--DTC` timing variable (such as `RFSTDTC` or `AESTDTC`), checking that its values are
ISO 8601 dates or date-times. Null values pass.

Columns that aren't in the template are left alone, and the data types of the columns aren't
checked. A template variable that isn't a column of the data gets no steps, so a dataset that
lacks a required variable isn't reported for it. Use
[`validate_sdtm_structure()`](`pointblank.validate_sdtm_structure`) to find missing variables,
or add a [`col_exists()`](`pointblank.Validate.col_exists`) step to the plan (as in the second
example below).

The date check accepts the right-truncated forms that SDTM uses for partial dates: a year
(`2024`), a year and month (`2024-03`), a full date (`2024-03-15`), and a date with a time to
the hour, minute or second (`2024-03-15T10:30:00`). Dates in other formats (such as
`03/15/2024`), fractional seconds, time zones and dates with missing middle components (such
as `2024---15`) fail. The check is on the format only, so an impossible date such as
`2024-13-45` passes.

Parameters
----------
data
    The dataset to validate, as a Polars or Pandas DataFrame.
domain
    The SDTM domain code, such as `"DM"`, `"AE"` or `"LB"`. This is case-insensitive. See
    [`list_sdtm_domains()`](`pointblank.list_sdtm_domains`) for the domains that have a
    template.
study_id
    A study identifier to use in the default label, which becomes
    `"SDTM {domain} — {study_id}"`. It is ignored if `label=` is given. The default is `None`.
check_dates
    Whether to add the ISO 8601 format checks for the `--DTC` timing variables. The default is
    `True`.
check_lengths
    Whether to add the maximum length checks for the character variables. The default is
    `True`.
label
    A label for the validation. The default is `"SDTM {domain} Validation"`.
**kwargs
    Other arguments for [`Validate`](`pointblank.Validate`), such as `tbl_name=` and
    `thresholds=`.

Returns
-------
Validate
    The validation plan, not yet interrogated.

Raises
------
KeyError
    If there is no template for the domain.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Here is a small Demographics (DM) dataset with a few problems: the third row has the wrong
`DOMAIN` and no `USUBJID`, the second row's `RFSTDTC` isn't an ISO 8601 date, and the fourth
row's `SEX` is longer than the two characters that SDTM allows:

```{python}
import pointblank as pb
import polars as pl

dm = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 4,
        "DOMAIN": ["DM", "DM", "AE", "DM"],
        "USUBJID": ["STUDY01-001", "STUDY01-002", None, "STUDY01-004"],
        "SUBJID": ["001", "002", "003", "004"],
        "RFSTDTC": ["2024-01-15", "03/15/2024", "2024-02", "2024-02-10T09:30"],
        "SEX": ["M", "F", "F", "MALE"],
    }
)

validation = pb.validate_sdtm(dm, domain="DM").interrogate()

validation
```

The plan starts with the null checks for the five required variables in the data, then checks
`DOMAIN`, the lengths of all six columns, and the format of `RFSTDTC`. Four steps have a
failing test unit, one for each problem. The partial date `2024-02` and the date-time
`2024-02-10T09:30` both pass the date check, since SDTM allows dates to be truncated.

The plan doesn't report the required DM variables that the dataset lacks, such as `SITEID` and
`COUNTRY`. Since the plan isn't interrogated yet, a
[`col_exists()`](`pointblank.Validate.col_exists`) step for the template's required variables
can be added to it. This example uses an Adverse Events (AE) dataset and leaves out the length
checks to keep the plan short:

```{python}
ae = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "DOMAIN": ["AE"] * 3,
        "USUBJID": ["STUDY01-001", "STUDY01-001", "STUDY01-002"],
        "AESEQ": [1, 2, 0],
        "AETERM": ["HEADACHE", "NAUSEA", "RASH"],
        "AESTDTC": ["2024-01-20", "2024-02-03", "2024-02"],
    }
)

required = pb.get_sdtm_domain("AE").required_variables

validation = (
    pb.validate_sdtm(ae, domain="AE", check_lengths=False, tbl_name="ae")
    .col_exists(columns=required)
    .interrogate()
)

validation
```

The `col_vals_gt()` step on `AESEQ` fails for the third row, whose sequence number is `0`. The
`col_exists()` steps at the end show that `AEDECOD`, a required AE variable, is missing from
the dataset. The `tbl_name=` argument was passed on to `Validate`.

See Also
--------
validate_sdtm_structure : Check a dataset's variables against an SDTM template.
get_sdtm_domain : Get the template that the plan is built from.
validate_adam : The equivalent for ADaM datasets.
validate_sdtmig : Check SDTM datasets against the SDTMIG conformance rules.

validate_sdtm_structure(data: 'Any', domain: 'str', strict: 'bool' = False) -> 'dict[str, Any]'

Check a dataset's variables against an SDTM domain template.

The `validate_sdtm_structure()` function is a quick check of a dataset's structure, run before
(or instead of) a full validation with [`validate_sdtm()`](`pointblank.validate_sdtm`). It
checks that every required variable of the domain is a column of the dataset, and that the
`DOMAIN` column holds only the domain code. It returns the findings as a dictionary straight
away, without building a [`Validate`](`pointblank.Validate`) plan.

With `strict=True`, two more things are reported: the expected variables (core designation
`"Exp"`) that are missing, and the columns that aren't variables in the template. These don't
affect `"valid"`. SDTM expects an expected variable to be in the dataset even if all of its
values are null, but a missing one is treated here as something to review rather than an
error. A column that isn't in the template may still be a valid SDTM variable, since the
templates only cover the common variables of each domain.

The column order and the data types aren't checked.

Parameters
----------
data
    The dataset to check, as a Polars or Pandas DataFrame.
domain
    The SDTM domain code, such as `"DM"` or `"AE"`. This is case-insensitive.
strict
    Whether to also report missing expected variables and columns that aren't in the template.
    The default is `False`.

Returns
-------
dict
    A dictionary with these keys:

    - `"domain"`: the domain code, in upper case.
    - `"valid"`: `True` if no required variable is missing and `DOMAIN` holds only the domain
    code.
    - `"missing_required"`: the names of the required variables that aren't columns of the
    data.
    - `"missing_expected"`: the names of the expected variables that aren't columns of the data
    (filled in only with `strict=True`).
    - `"unknown_variables"`: the columns that aren't variables of the domain (filled in only
    with `strict=True`).
    - `"domain_mismatch"`: `True` if the `DOMAIN` column has any value (including a null) other
    than the domain code.
    - `"issues"`: a message for each of the problems above.

Raises
------
KeyError
    If there is no template for the domain.

Examples
--------
Let's check a small Adverse Events (AE) dataset. It has all of the required AE variables
except `AEDECOD`, and its `DOMAIN` column has one value from another domain:

```{python}
import pointblank as pb
import polars as pl

ae = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "DOMAIN": ["AE", "AE", "CM"],
        "USUBJID": ["STUDY01-001", "STUDY01-001", "STUDY01-002"],
        "AESEQ": [1, 2, 1],
        "AETERM": ["HEADACHE", "NAUSEA", "RASH"],
        "AESEV": ["MILD", "MILD", "MODERATE"],
        "AEFLAG": ["Y", "N", "Y"],
    }
)

result = pb.validate_sdtm_structure(ae, domain="AE")

print(result["valid"])
for issue in result["issues"]:
    print(issue)
```

The dataset isn't valid, and the issues say why. The extra column `AEFLAG` isn't mentioned,
since that's only reported in strict mode:

```{python}
result = pb.validate_sdtm_structure(ae, domain="AE", strict=True)

print(result["missing_expected"])
print(result["unknown_variables"])
```

Strict mode adds the expected AE variables that the dataset doesn't have, such as `AESTDTC`,
and the column that isn't an AE variable. `AESEV` is a permissible variable of the domain, so
it isn't reported.

See Also
--------
validate_sdtm : Build a validation plan from an SDTM domain template.
get_sdtm_domain : Get the template that the dataset is checked against.
validate_adam_structure : The equivalent for ADaM datasets.

sdtm_to_metadata(domain: 'str', study_id: 'str | None' = None) -> 'MetadataImport'

Convert an SDTM domain template to a `MetadataImport` object.

The SDTM domain templates (see [`get_sdtm_domain()`](`pointblank.get_sdtm_domain`)) have their
own classes, [`SDTMDomainTemplate`](`pointblank.SDTMDomainTemplate`) and
[`SDTMVariableSpec`](`pointblank.SDTMVariableSpec`). The `sdtm_to_metadata()` function converts
a template to a [`MetadataImport`](`pointblank.MetadataImport`), the same object that
[`import_metadata()`](`pointblank.import_metadata`) returns for a metadata file. That gives the
template the methods of a `MetadataImport`, such as
[`to_schema()`](`pointblank.MetadataImport.to_schema`) and
[`get_variable()`](`pointblank.MetadataImport.get_variable`), and lets it be exported with
[`export_metadata()`](`pointblank.export_metadata`).

Each variable in the template becomes a [`VariableMetadata`](`pointblank.VariableMetadata`)
with the same name, label, role, `required` flag, maximum length and controlled terminology
name. SDTM's `"Num"` type becomes the `"Float64"` dtype and `"Char"` becomes `"String"`. The
dataset name and label come from the domain, and the source version is `"IG 3.4"`. The template
names the codelist of a variable (as in `controlled_term="SEX"`) but doesn't contain the
codelist itself, so the result has no codelists. Calling
`import_metadata(format="cdisc_sdtm", domain=...)` gives the same result.

Parameters
----------
domain
    The SDTM domain code, such as `"DM"`, `"AE"` or `"LB"`. This is case-insensitive.
study_id
    A study identifier to record in the metadata's `study_id`. The default is `None`.

Returns
-------
MetadataImport
    The metadata for the domain, with `source_format="cdisc_sdtm"`.

Raises
------
KeyError
    If there is no template for the domain. See
    [`list_sdtm_domains()`](`pointblank.list_sdtm_domains`) for the domains that have one.

Examples
--------
Let's convert the template for the Demographics (DM) domain:

```{python}
import pointblank as pb

dm_meta = pb.sdtm_to_metadata("DM", study_id="STUDY01")

print(dm_meta.dataset_label)
print(dm_meta.source_version)
print(dm_meta.variable_names[:6])
```

The metadata has all 26 variables of the template, in the template's order. Each variable keeps
what the template says about it:

```{python}
sex = dm_meta.get_variable("SEX")

print(sex.dtype, sex.required, sex.max_length, sex.controlled_term)
```

`SEX` is a required character variable of at most two characters, with values from the `SEX`
codelist of CDISC Controlled Terminology. The metadata can be turned into a
[`Schema`](`pointblank.Schema`) for use in
[`Validate.col_schema_match()`](`pointblank.Validate.col_schema_match`):

```{python}
schema = dm_meta.to_schema()

schema.columns[:4]
```

The schema lists every variable of the template, including the expected and permissible ones.
A real DM dataset usually has only some of them, so a schema check against all 26 is stricter
than SDTM requires. For checking a dataset against the template,
[`validate_sdtm()`](`pointblank.validate_sdtm`) (which only checks the variables that are
present) is usually the better fit.

See Also
--------
get_sdtm_domain : Get the template that is converted.
validate_sdtm : Build a validation plan from an SDTM domain template.
adam_to_metadata : The equivalent for ADaM dataset templates.
MetadataImport : The object that is returned.

get_sdtm_domain(domain: 'str') -> 'SDTMDomainTemplate'

Get the template for an SDTM domain.

Pointblank has templates for eight domains of the CDISC Study Data Tabulation Model (SDTM),
based on the SDTM Implementation Guide (SDTMIG) 3.4. Each template is an
[`SDTMDomainTemplate`](`pointblank.SDTMDomainTemplate`): the domain's label and observation
class, the variables that form its natural key, and an
[`SDTMVariableSpec`](`pointblank.SDTMVariableSpec`) for each of its variables, giving the
variable's label, type, role, maximum length, controlled terminology and core designation. The
templates are what [`validate_sdtm()`](`pointblank.validate_sdtm`),
[`validate_sdtm_structure()`](`pointblank.validate_sdtm_structure`) and
[`sdtm_to_metadata()`](`pointblank.sdtm_to_metadata`) work from.

The templates cover the common variables of each domain rather than every variable in the
SDTMIG. Each call returns a new template object, so changing one doesn't affect later calls.

Parameters
----------
domain
    The two-letter domain code, such as `"DM"`, `"AE"`, `"LB"` or `"VS"`. This is
    case-insensitive.

Returns
-------
SDTMDomainTemplate
    The template for the domain.

Raises
------
KeyError
    If there is no template for the domain. The error message lists the domains that have one,
    as does [`list_sdtm_domains()`](`pointblank.list_sdtm_domains`).

Examples
--------
Let's get the template for the Vital Signs (VS) domain:

```{python}
import pointblank as pb

vs = pb.get_sdtm_domain("vs")

print(vs.label, "|", vs.domain_class, "| repeating:", vs.repeating)
print(vs.natural_keys)
```

VS is a Findings domain with many records per subject, one for each test at each time point,
so its natural key includes the test code, the date-time and the time point number. The
template's properties list the variables by their core designation:

```{python}
print(vs.required_variables)
print(vs.expected_variables)
```

The required variables must be in every VS dataset and can't be null. The expected ones should
be in the dataset but may be null, and the domain's other variables are permissible: they're
included only when there is data for them.

See Also
--------
list_sdtm_domains : List the domains that have a template.
SDTMDomainTemplate : The template that is returned.
validate_sdtm : Build a validation plan from an SDTM domain template.
get_adam_dataset : The equivalent for ADaM datasets.

list_sdtm_domains() -> 'list[str]'

List the SDTM domains that have a template.

These are the domain codes that [`get_sdtm_domain()`](`pointblank.get_sdtm_domain`),
[`validate_sdtm()`](`pointblank.validate_sdtm`),
[`validate_sdtm_structure()`](`pointblank.validate_sdtm_structure`) and
[`sdtm_to_metadata()`](`pointblank.sdtm_to_metadata`) accept.

Returns
-------
list[str]
    The domain codes, in alphabetical order.

Examples
--------
Let's list the domains, together with the label and class of each one's template:

```{python}
import pointblank as pb

for code in pb.list_sdtm_domains():
    template = pb.get_sdtm_domain(code)
    print(f"{code}  {template.label:26}{template.domain_class}")
```

The eight domains come from all four of the observation classes that the templates use:
Special Purpose (DM), Events, Interventions and Findings.

See Also
--------
get_sdtm_domain : Get the template for a domain.
list_adam_datasets : The equivalent for ADaM datasets.

SDTMDomainTemplate(domain: 'str', label: 'str', description: 'str', domain_class: 'str', repeating: 'bool', variables: 'list[SDTMVariableSpec]' = <factory>, natural_keys: 'list[str]' = <factory>) -> None

The structural template for an SDTM domain.

An `SDTMDomainTemplate` describes the structure of one domain of the CDISC Study Data
Tabulation Model (SDTM): its label and observation class, whether it has many records per
subject, the variables that form its natural key, and the specifications of its variables (as
[`SDTMVariableSpec`](`pointblank.SDTMVariableSpec`) objects, in the SDTM order). Templates are
obtained with [`get_sdtm_domain()`](`pointblank.get_sdtm_domain`).

Parameters
----------
domain
    The two-letter domain code, such as `"DM"`, `"AE"` or `"LB"`.
label
    The domain label, such as `"Demographics"` or `"Adverse Events"`.
description
    A short description of what the domain holds.
domain_class
    The SDTM observation class: `"Special Purpose"`, `"Events"`, `"Interventions"` or
    `"Findings"`.
repeating
    Whether the domain can have more than one record per subject. Only DM, which has one record
    per subject, isn't repeating.
variables
    The specifications of the domain's variables, in order. The default is an empty list.
natural_keys
    The names of the variables that together identify a record. The default is an empty list.

Examples
--------
Let's look at the template for the Adverse Events (AE) domain:

```{python}
import pointblank as pb

ae = pb.get_sdtm_domain("AE")

print(ae.label, "|", ae.domain_class, "| repeating:", ae.repeating)
print(ae.description)
print(len(ae.variables), "variables")
```

The variables are in the `variables` list. Here are the first eight, with their type, role and
core designation:

```{python}
for var in ae.variables[:8]:
    print(f"{var.name:9}{var.dtype:5}{var.role:12}{var.core}")
```

The template's properties collect the names of the variables with a given core designation or
role, and [`get_variable()`](`pointblank.SDTMDomainTemplate.get_variable`) gets the
specification of one variable.

See Also
--------
get_sdtm_domain : Get the template for a domain.
SDTMVariableSpec : The specification of a variable in the template.
ADaMDatasetTemplate : The equivalent for ADaM datasets.

SDTMVariableSpec(name: 'str', label: 'str', dtype: 'str', role: 'str', required: 'bool' = False, max_length: 'int | None' = None, controlled_term: 'str | None' = None, core: 'str' = 'Perm') -> None

The specification of one variable in an SDTM domain template.

An [`SDTMDomainTemplate`](`pointblank.SDTMDomainTemplate`) holds an `SDTMVariableSpec` for each
variable of its domain, in its `variables` list. The specification gives what the SDTM
Implementation Guide says about the variable: its label, its type, its role in the domain, its
core designation, and the controlled terminology for its values.
[`validate_sdtm()`](`pointblank.validate_sdtm`) uses `required`, `dtype`, `max_length` and
`role` to decide which checks to make.

Parameters
----------
name
    The variable name, such as `"STUDYID"` or `"AESTDTC"`.
label
    The variable label, such as `"Study Identifier"`.
dtype
    The SDTM data type: `"Char"` for character variables or `"Num"` for numeric ones.
role
    The variable's SDTM role. The templates use `"Identifier"` (variables that identify the
    study, subject, domain or record), `"Topic"` (the focus of the observation, such as
    `AETERM`), `"Qualifier"` (variables that describe the observation) and `"Timing"` (dates,
    times and study days).
required
    Whether the variable is required: it must be in the dataset and can't be null. In the
    templates, this is `True` exactly when `core` is `"Req"`. The default is `False`.
max_length
    The maximum length of the values of a character variable. The default is `None` (no
    maximum, as for numeric variables).
controlled_term
    The name of the controlled terminology codelist for the variable's values, such as `"SEX"`
    or `"NY"`. The default is `None`.
core
    The core designation: `"Req"` (required), `"Exp"` (expected: in the dataset, but values may
    be null) or `"Perm"` (permissible: included when there is data for it). The default is
    `"Perm"`.

Examples
--------
Specifications are usually taken from a template. Here is the one for `AESER` in the Adverse
Events (AE) template:

```{python}
import pointblank as pb

aeser = pb.get_sdtm_domain("AE").get_variable("AESER")

aeser
```

`AESER` (whether the adverse event was serious) is an expected character variable of at most
two characters, with values from the `NY` (No Yes Response) codelist.

See Also
--------
SDTMDomainTemplate : The template that holds the specifications of a domain's variables.
ADaMVariableSpec : The equivalent for ADaM dataset templates.


## ADaM Validation

Validate analysis datasets against CDISC ADaM templates. Use `validate_adam()` to generate a full `Validate` workflow, or `validate_adam_structure()` for a quick structural conformance check. Retrieve dataset templates with `get_adam_dataset()` and `list_adam_datasets()`.



validate_adam(data: 'Any', dataset: 'str', study_id: 'str | None' = None, check_population_flags: 'bool' = True, check_bds_structure: 'bool' = True, check_traceability: 'bool' = True, label: 'str | None' = None, **kwargs: 'Any')

Build a validation plan for a dataset from an ADaM dataset template.

The `validate_adam()` function reads an ADaM dataset template (see
[`get_adam_dataset()`](`pointblank.get_adam_dataset`)) and turns it into a
[`Validate`](`pointblank.Validate`) plan for an analysis dataset of that type. The plan isn't
interrogated, so more steps can be added to it before calling
[`interrogate()`](`pointblank.Validate.interrogate`).

The steps are generated in this order, for the template variables that are columns of the
data:

1. [`col_vals_not_null()`](`pointblank.Validate.col_vals_not_null`) for each required variable
(one with a core designation of `"Req"`).
2. If `check_population_flags=True`, a
[`col_vals_in_set()`](`pointblank.Validate.col_vals_in_set`) step for each population flag
(such as `SAFFL` or `ITTFL`), checking that its values are `"Y"` or `"N"`. A null value fails,
since ADaM doesn't allow nulls in population flags.
3. Checks for the type of dataset:
    - BDS (if `check_bds_structure=True`): a
      [`col_vals_expr()`](`pointblank.Validate.col_vals_expr`) step checking that `PARAMCD`
      is no longer than eight characters. Null values pass.
    - ADTTE: `CNSR` must be `0` (an event) or `1` (censored), and `AVAL` (the time to the
      event) must be at least `0`.
    - ADAE: `TRTEMFL` must be `"Y"` or `"N"`, and `AESEQ` must be greater than `0`.
    - ADSL: `TRT01P` must not be null. This is already one of the required-variable checks,
      so it isn't added a second time.
4. If `check_traceability=True`, a null check for each of the traceability variables `SRCDOM`,
`SRCVAR` and `SRCSEQ`, which record the SDTM domain, variable and record that a value came
from.

Columns that aren't in the template are left alone, and the data types of the columns aren't
checked. A template variable that isn't a column of the data gets no steps, so a dataset that
lacks a required variable isn't reported for it. Use
[`validate_adam_structure()`](`pointblank.validate_adam_structure`) to find missing variables,
or add a [`col_exists()`](`pointblank.Validate.col_exists`) step to the plan.

Parameters
----------
data
    The dataset to validate, as a Polars or Pandas DataFrame.
dataset
    The name of the ADaM dataset template: `"ADSL"`, `"BDS"`, `"ADAE"` or `"ADTTE"`. This is
    case-insensitive. Use `"BDS"` for any dataset with the Basic Data Structure, such as ADVS
    or ADLB.
study_id
    A study identifier to use in the default label, which becomes
    `"ADaM {dataset} — {study_id}"`. It is ignored if `label=` is given. The default is `None`.
check_population_flags
    Whether to check that the population flags hold only `"Y"` and `"N"`. The default is `True`.
check_bds_structure
    Whether to add the `PARAMCD` length check for a BDS dataset. It has no effect for the other
    templates. The default is `True`.
check_traceability
    Whether to add the null checks for the traceability variables. The default is `True`.
label
    A label for the validation. The default is `"ADaM {dataset} Validation"`.
**kwargs
    Other arguments for [`Validate`](`pointblank.Validate`), such as `tbl_name=` and
    `thresholds=`.

Returns
-------
Validate
    The validation plan, not yet interrogated.

Raises
------
KeyError
    If there is no template with that name.

Examples
--------
```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False, preview_incl_header=False)
```
Here is a small subject-level analysis dataset (ADSL) with two problems in its population
flags: the fourth subject has no safety flag (`SAFFL`) and an intent-to-treat flag (`ITTFL`)
of `"U"`:

```{python}
import pointblank as pb
import polars as pl

adsl = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 4,
        "USUBJID": ["STUDY01-001", "STUDY01-002", "STUDY01-003", "STUDY01-004"],
        "SUBJID": ["001", "002", "003", "004"],
        "SITEID": ["SITE01", "SITE01", "SITE02", "SITE02"],
        "TRT01P": ["Drug A", "Placebo", "Drug A", "Placebo"],
        "SAFFL": ["Y", "Y", "N", None],
        "ITTFL": ["Y", "Y", "Y", "U"],
    }
)

validation = pb.validate_adam(adsl, dataset="ADSL").interrogate()

validation
```

The five required ADSL variables get null checks, which all pass. The two population flags
are checked for `"Y"` and `"N"`, and each of those steps has one failing test unit.

For a time-to-event dataset (ADTTE), the plan also checks the censoring indicator and the
time to the event. Here the second subject has a negative time, and the third has a `CNSR` of
`2`:

```{python}
adtte = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "USUBJID": ["STUDY01-001", "STUDY01-002", "STUDY01-003"],
        "PARAMCD": ["OS"] * 3,
        "PARAM": ["Overall Survival (days)"] * 3,
        "AVAL": [120.0, -3.0, 365.0],
        "STARTDT": [19723.0, 19730.0, 19741.0],
        "ADT": [19843.0, 19727.0, 20106.0],
        "CNSR": [0, 1, 2],
    }
)

validation = pb.validate_adam(adtte, dataset="ADTTE", tbl_name="adtte").interrogate()

validation
```

The last two steps are the ADTTE checks, and each one finds its problem. The dates
`STARTDT` and `ADT` are numbers of days since 1960-01-01, as is usual in ADaM datasets (which
are often stored as SAS files).

See Also
--------
validate_adam_structure : Check a dataset's variables against an ADaM template.
get_adam_dataset : Get the template that the plan is built from.
validate_sdtm : The equivalent for SDTM datasets.

validate_adam_structure(data: 'Any', dataset: 'str', strict: 'bool' = False) -> 'dict[str, Any]'

Check a dataset's variables against an ADaM dataset template.

The `validate_adam_structure()` function is a quick check of a dataset's structure, run before
(or instead of) a full validation with [`validate_adam()`](`pointblank.validate_adam`). It
checks that every required variable of the template is a column of the dataset, and lists the
population flags that the dataset has. It returns the findings as a dictionary straight away,
without building a [`Validate`](`pointblank.Validate`) plan.

For an ADSL dataset with no population flags at all, an issue is added to `"issues"`, but the
dataset is still `"valid"`. With `strict=True`, two more things are reported: the
conditionally required variables (core designation `"Cond"`) that are missing, and the columns
that aren't variables in the template. These don't affect `"valid"` either, since whether a
conditional variable is needed depends on the study, and the templates only cover the common
variables of each type of dataset.

The column order, the data types and the values aren't checked.

Parameters
----------
data
    The dataset to check, as a Polars or Pandas DataFrame.
dataset
    The name of the ADaM dataset template: `"ADSL"`, `"BDS"`, `"ADAE"` or `"ADTTE"`. This is
    case-insensitive.
strict
    Whether to also report missing conditional variables and columns that aren't in the
    template. The default is `False`.

Returns
-------
dict
    A dictionary with these keys:

    - `"dataset"`: the template name, in upper case.
    - `"dataset_class"`: the ADaM class of the template.
    - `"valid"`: `True` if no required variable is missing.
    - `"missing_required"`: the names of the required variables that aren't columns of the
    data.
    - `"missing_conditional"`: the names of the conditional variables that aren't columns of
    the data (filled in only with `strict=True`).
    - `"unknown_variables"`: the columns that aren't variables in the template, in the order of
    the data's columns (filled in only with `strict=True`).
    - `"population_flags_found"`: the population flags that are columns of the data.
    - `"issues"`: a message for each of the problems above. With `strict=True`, the message for
    a missing conditional variable includes its condition.

Raises
------
KeyError
    If there is no template with that name.

Examples
--------
Let's check a small ADSL dataset that lacks the site identifier, `SITEID`, a required ADSL
variable:

```{python}
import pointblank as pb
import polars as pl

adsl = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "USUBJID": ["STUDY01-001", "STUDY01-002", "STUDY01-003"],
        "SUBJID": ["001", "002", "003"],
        "TRT01P": ["Drug A", "Placebo", "Drug A"],
        "SAFFL": ["Y", "Y", "N"],
        "BMIBL": [24.1, 31.7, 22.9],
    }
)

result = pb.validate_adam_structure(adsl, dataset="ADSL")

print(result["valid"])
print(result["missing_required"])
print(result["population_flags_found"])
```

The dataset isn't valid because of the missing `SITEID`, and it has one population flag,
`SAFFL`. In strict mode, the conditional variables and the unknown columns are reported too:

```{python}
result = pb.validate_adam_structure(adsl, dataset="ADSL", strict=True)

print(result["missing_conditional"])
print(result["unknown_variables"])
```

The baseline BMI column, `BMIBL`, isn't in the template. Since ADaM allows a sponsor to add
variables like this one, it's reported only so that it can be reviewed.

See Also
--------
validate_adam : Build a validation plan from an ADaM dataset template.
get_adam_dataset : Get the template that the dataset is checked against.
validate_sdtm_structure : The equivalent for SDTM datasets.

adam_to_metadata(dataset: 'str', study_id: 'str | None' = None) -> 'MetadataImport'

Convert an ADaM dataset template to a `MetadataImport` object.

The ADaM dataset templates (see [`get_adam_dataset()`](`pointblank.get_adam_dataset`)) have
their own classes, [`ADaMDatasetTemplate`](`pointblank.ADaMDatasetTemplate`) and
[`ADaMVariableSpec`](`pointblank.ADaMVariableSpec`). The `adam_to_metadata()` function converts
a template to a [`MetadataImport`](`pointblank.MetadataImport`), the same object that
[`import_metadata()`](`pointblank.import_metadata`) returns for a metadata file. That gives the
template the methods of a `MetadataImport`, such as
[`to_schema()`](`pointblank.MetadataImport.to_schema`) and
[`get_variable()`](`pointblank.MetadataImport.get_variable`), and lets it be exported with
[`export_metadata()`](`pointblank.export_metadata`).

Each variable in the template becomes a [`VariableMetadata`](`pointblank.VariableMetadata`)
with the same name, label, `required` flag, maximum length and controlled terminology name.
The variable's core designation (`"Req"`, `"Cond"` or `"Perm"`) goes in `cdisc_role`, and its
expected source (such as `"SDTM.DM"`) goes in `adam_derivation`. ADaM's `"Num"` type becomes
the `"Float64"` dtype and `"Char"` becomes `"String"`, so numeric dates such as `TRTSDT` (which
ADaM stores as numbers of days) are `"Float64"`. The source version is `"IG 1.1"`. Calling
`import_metadata(format="cdisc_adam", dataset=...)` gives the same result.

Parameters
----------
dataset
    The name of the ADaM dataset template: `"ADSL"`, `"BDS"`, `"ADAE"` or `"ADTTE"`. This is
    case-insensitive.
study_id
    A study identifier to record in the metadata's `study_id`. The default is `None`.

Returns
-------
MetadataImport
    The metadata for the dataset, with `source_format="cdisc_adam"`.

Raises
------
KeyError
    If there is no template with that name.

Examples
--------
Let's convert the template for the subject-level analysis dataset, ADSL:

```{python}
import pointblank as pb

adsl_meta = pb.adam_to_metadata("ADSL", study_id="STUDY01")

print(adsl_meta.dataset_label)
print(len(adsl_meta.variables), "variables")
```

Each variable keeps what the template says about it, with the ADaM core designation in
`cdisc_role`:

```{python}
for name in ["USUBJID", "TRT01P", "SAFFL", "TRTSDT"]:
    var = adsl_meta.get_variable(name)
    print(f"{name:8}{var.dtype:8}{var.cdisc_role:5}{var.adam_derivation}")
```

`USUBJID` and `TRT01P` are required, while `SAFFL` (the safety population flag) and `TRTSDT`
(the treatment start date) are conditionally required. `USUBJID` comes from the SDTM
Demographics (DM) domain and `TRTSDT` is derived from Exposure (EX). Only the required
variables get `required=True`, so the conditional ones don't get the null checks that
[`to_validate()`](`pointblank.MetadataImport.to_validate`) adds for required variables.

See Also
--------
get_adam_dataset : Get the template that is converted.
validate_adam : Build a validation plan from an ADaM dataset template.
sdtm_to_metadata : The equivalent for SDTM domain templates.
MetadataImport : The object that is returned.

get_adam_dataset(name: 'str') -> 'ADaMDatasetTemplate'

Get the template for a type of ADaM dataset.

Pointblank has templates for four types of CDISC Analysis Data Model (ADaM) dataset: ADSL (the
subject-level analysis dataset), BDS (the Basic Data Structure, used by datasets such as ADVS
and ADLB), ADAE (adverse events) and ADTTE (time to event). Each template is an
[`ADaMDatasetTemplate`](`pointblank.ADaMDatasetTemplate`): the dataset's label and class, the
variables that form its natural key, and an [`ADaMVariableSpec`](`pointblank.ADaMVariableSpec`)
for each of its variables. The templates are what
[`validate_adam()`](`pointblank.validate_adam`),
[`validate_adam_structure()`](`pointblank.validate_adam_structure`) and
[`adam_to_metadata()`](`pointblank.adam_to_metadata`) work from.

The templates cover the common variables of each type of dataset rather than every variable
that the ADaM Implementation Guide describes. Each call returns a new template object, so
changing one doesn't affect later calls.

Parameters
----------
name
    The template name: `"ADSL"`, `"BDS"`, `"ADAE"` or `"ADTTE"`. This is case-insensitive.

Returns
-------
ADaMDatasetTemplate
    The template for the dataset.

Raises
------
KeyError
    If there is no template with that name. The error message lists the templates, as does
    [`list_adam_datasets()`](`pointblank.list_adam_datasets`).

Examples
--------
Let's get the template for the subject-level analysis dataset:

```{python}
import pointblank as pb

adsl = pb.get_adam_dataset("adsl")

print(adsl.label)
print(adsl.natural_keys)
print(adsl.required_variables)
```

ADSL has one record per subject, so the study and subject identifiers are its natural key.
Besides the identifiers, the planned treatment `TRT01P` is required. The template also lists
the population flags:

```{python}
adsl.population_flags
```

These flags (safety, intent-to-treat, efficacy and so on) are conditionally required or
permissible, and [`validate_adam()`](`pointblank.validate_adam`) checks that the ones in a
dataset hold only `"Y"` and `"N"`.

See Also
--------
list_adam_datasets : List the dataset templates.
ADaMDatasetTemplate : The template that is returned.
validate_adam : Build a validation plan from an ADaM dataset template.
get_sdtm_domain : The equivalent for SDTM domains.

list_adam_datasets() -> 'list[str]'

List the ADaM dataset templates.

These are the names that [`get_adam_dataset()`](`pointblank.get_adam_dataset`),
[`validate_adam()`](`pointblank.validate_adam`),
[`validate_adam_structure()`](`pointblank.validate_adam_structure`) and
[`adam_to_metadata()`](`pointblank.adam_to_metadata`) accept.

Returns
-------
list[str]
    The template names, in alphabetical order.

Examples
--------
Let's list the templates, together with each one's label:

```{python}
import pointblank as pb

for name in pb.list_adam_datasets():
    print(f"{name:7}{pb.get_adam_dataset(name).label}")
```

BDS is a structure rather than a single dataset, so its template is used for any BDS dataset
in a study, such as ADVS or ADLB.

See Also
--------
get_adam_dataset : Get a dataset template.
list_sdtm_domains : The equivalent for SDTM domains.

ADaMDatasetTemplate(name: 'str', label: 'str', description: 'str', dataset_class: 'str', variables: 'list[ADaMVariableSpec]' = <factory>, natural_keys: 'list[str]' = <factory>) -> None

The structural template for an ADaM dataset.

An `ADaMDatasetTemplate` describes the structure of one type of CDISC Analysis Data Model
(ADaM) dataset: its label and class, the variables that form its natural key, and the
specifications of its variables (as [`ADaMVariableSpec`](`pointblank.ADaMVariableSpec`)
objects, in order). Templates are obtained with
[`get_adam_dataset()`](`pointblank.get_adam_dataset`).

Parameters
----------
name
    The template name: `"ADSL"`, `"BDS"`, `"ADAE"` or `"ADTTE"`.
label
    The dataset label, such as `"Subject Level Analysis Dataset"`.
description
    A short description of what the dataset holds and what a record is.
dataset_class
    The ADaM dataset class, which [`validate_adam()`](`pointblank.validate_adam`) uses to
    choose the checks for the type of dataset. In the templates, it's the same as `name`.
variables
    The specifications of the dataset's variables, in order. The default is an empty list.
natural_keys
    The names of the variables that together identify a record. The default is an empty list.

Examples
--------
Let's look at the template for the Basic Data Structure (BDS), the structure of analysis
datasets such as ADVS (vital signs) and ADLB (laboratory results):

```{python}
import pointblank as pb

bds = pb.get_adam_dataset("BDS")

print(bds.label, "|", bds.dataset_class)
print(bds.description)
print(bds.natural_keys)
```

A BDS dataset has a record for each analysis parameter (`PARAMCD`) at each visit and date,
for each subject. The template's properties list the required and conditionally required
variables:

```{python}
print(bds.required_variables)
print(bds.conditional_variables)
```

Every BDS dataset needs the identifiers, the parameter and the analysis value. The
conditional variables, such as the baseline value `BASE` and the change from baseline `CHG`,
are needed only for some analyses.

See Also
--------
get_adam_dataset : Get the template for a type of dataset.
ADaMVariableSpec : The specification of a variable in the template.
SDTMDomainTemplate : The equivalent for SDTM domains.

ADaMVariableSpec(name: 'str', label: 'str', dtype: 'str', core: 'str' = 'Perm', required: 'bool' = False, max_length: 'int | None' = None, controlled_term: 'str | None' = None, source: 'str | None' = None, condition: 'str | None' = None, is_population_flag: 'bool' = False) -> None

The specification of one variable in an ADaM dataset template.

An [`ADaMDatasetTemplate`](`pointblank.ADaMDatasetTemplate`) holds an `ADaMVariableSpec` for
each variable of its dataset, in its `variables` list. The specification gives the variable's
label, type and core designation, the condition under which a conditionally required variable
is needed, where its values come from, and whether it is a population flag.
[`validate_adam()`](`pointblank.validate_adam`) uses `required` and `is_population_flag` to
decide which checks to make.

Parameters
----------
name
    The variable name, such as `"USUBJID"`, `"PARAMCD"` or `"AVAL"`.
label
    The variable label, such as `"Unique Subject Identifier"`.
dtype
    The data type: `"Char"` for character variables or `"Num"` for numeric ones (including
    dates, which ADaM stores as numbers).
core
    The core designation: `"Req"` (required), `"Cond"` (conditionally required: needed when
    the condition in `condition` holds) or `"Perm"` (permissible). The default is `"Perm"`.
required
    Whether the variable is required in every dataset of this type. In the templates, this is
    `True` exactly when `core` is `"Req"`. The default is `False`.
max_length
    The maximum length of the values of a character variable. The default is `None` (no
    maximum).
controlled_term
    The name of the controlled terminology codelist for the variable's values, such as `"NY"`
    for the population flags. The default is `None`.
source
    Where the variable's values come from, such as `"SDTM.DM"` (copied from the Demographics
    domain) or `"Derived from SDTM.EX"`. The default is `None` (not specified).
condition
    For a conditionally required variable, a description of when it is required, such as
    `"Required if safety population defined"`. The default is `None`.
is_population_flag
    Whether the variable is a population flag (such as `SAFFL` or `ITTFL`), whose values must
    be `"Y"` or `"N"`. The default is `False`.

Examples
--------
Specifications are usually taken from a template. Here is the one for the safety population
flag, `SAFFL`, in the ADSL template:

```{python}
import pointblank as pb

saffl = pb.get_adam_dataset("ADSL").get_variable("SAFFL")

saffl
```

`SAFFL` is a one-character population flag with values from the `NY` codelist. It is
conditionally required, and `condition` says when: whenever the study defines a safety
population.

See Also
--------
ADaMDatasetTemplate : The template that holds the specifications of a dataset's variables.
SDTMVariableSpec : The equivalent for SDTM domain templates.


## CDISC Submission Conformance

Validate SDTM datasets for CDISC conformance. `validate_sdtmig()` is the primary entry point: pass a dictionary of domain DataFrames and receive a `ConformanceReport` with 362 SDTMIG 3.4 rules evaluated in-process. To work with a whole submission (including cross-dataset referential integrity and SUPP-- linkage, with `validate_conformance(engine="validate")`), use `SubmissionPackage`. For the authoritative CDISC-certified rule set, use `validate_cdisc_submission()` (requires the CORE CLI) or `SubmissionPackage.validate_conformance(engine="core")`. All paths return a `ConformanceReport`.



validate_sdtmig(datasets: 'dict', version: 'str' = '3-4', ct_packages: 'list[str] | None' = None, define_xml: 'Any' = None, study_id: 'str | None' = None) -> 'ConformanceReport'

Check SDTM datasets with Pointblank's bundled SDTMIG conformance rules.

Before SDTM datasets are submitted, they are checked for conformance to the SDTM
Implementation Guide (SDTMIG). The `validate_sdtmig()` function does this with a catalog of
362 rules for SDTMIG 3.4 that comes with Pointblank, so it needs no other software and no
network access. It returns a [`ConformanceReport`](`pointblank.ConformanceReport`) with the
result of each rule. For the published CDISC and agency rules, use
[`validate_cdisc_submission()`](`pointblank.validate_cdisc_submission`), which runs the
CDISC CORE engine.

The rules are of these types:

- Record checks (221 rules) check the values of each record, such as that `SEX` uses the
  terms of its codelist, that dates are in the ISO 8601 format, that `AGE` isn't negative
  and that required variables aren't null. These rules give the failing records as
  findings.
- Variable metadata checks (116 rules) check that the required variables of a domain are
  present, that variables have the right type and that they are in the right order.
- Dataset contents checks (9 rules) check across the records of a dataset, such as that
  `STUDYID` has the same value in every record and that `AESEQ` is unique for each subject.
- Dataset metadata checks (3 rules) check that every dataset has `STUDYID`, `DOMAIN` and
  (for subject-level data) `USUBJID`.
- Domain presence checks (3 rules) check that the DM, TS and TA datasets are present.
- Define-XML checks (12 rules) check the datasets against their Define-XML, such as that
  every variable is declared in it and that values are from the codelists that it declares.
  These rules need `define_xml=`, and are not applicable without it.

Four of the rules are of `"Warning"` sensitivity: the presence of TA and of `USUBJID`, and
the order of the identifier variables and of the sequence variable. All the others are of
`"Error"` sensitivity. A rule for a domain that isn't one of the `datasets` (such as an AE
rule when there's no AE dataset) has nothing to check, so it passes. The codelist checks
ignore case, so `"m"` is accepted for `SEX`. Empty strings (which is how XPT files store
missing character values) are treated as nulls. SUPP-- and RELREC datasets aren't checked
by the rules that apply to all domains, since they have a different structure (with
`RDOMAIN` in place of `DOMAIN`).

The rules check each dataset on its own, so they don't check that the subjects of one
dataset are in DM. For that, use the `"validate"` engine of the `SubmissionPackage` method
[`validate_conformance()`](`pointblank.SubmissionPackage.validate_conformance`).

Parameters
----------
datasets
    The SDTM datasets, as a dict of DataFrames keyed by domain name, such as
    `{"DM": dm, "AE": ae}`. The names are case-insensitive. The DataFrames can be Polars,
    Pandas or any other kind that Narwhals supports.
version
    The version of SDTMIG, as `"3-4"` or `"3.4"`. Only version 3.4 has a bundled catalog. The
    default is `"3-4"`.
ct_packages
    The bundled controlled terminology packages for the codelist checks, as a list of names
    such as `["sdtm-ct-2024-09-27"]`. The default is the latest bundled package, which is
    `"sdtm-ct-2024-09-27"`.
define_xml
    The study's Define-XML, as a path to a `define.xml` file or as a
    [`MetadataPackage`](`pointblank.MetadataPackage`) from
    [`import_metadata()`](`pointblank.import_metadata`). The default is `None`, which makes
    the 12 Define-XML rules not applicable.
study_id
    A study identifier, which is kept in the report's `package`. It isn't used by the rules
    and isn't shown in the report. The default is `None`.

Returns
-------
ConformanceReport
    The report, whose [`is_rules`](`pointblank.ConformanceReport.is_rules`) is `True`. In a
    notebook it is shown as the table of
    [`get_tabular_report()`](`pointblank.ConformanceReport.get_tabular_report`).

Raises
------
ValueError
    If there is no bundled catalog for the `version`. The error message lists the versions
    that have one.
FileNotFoundError
    If one of the `ct_packages` isn't bundled with Pointblank.

Examples
--------
Here is a DM dataset with all the variables that SDTMIG requires for DM, but with three
problems: the second subject's `RFSTDTC` isn't an ISO 8601 date, and the third subject has a
negative `AGE` and a `SEX` of `"FEMALE"` (the SEX codelist has `"F"`):

```{python}
import pointblank as pb
import polars as pl

dm = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "DOMAIN": ["DM"] * 3,
        "USUBJID": ["STUDY01-001", "STUDY01-002", "STUDY01-003"],
        "SUBJID": ["001", "002", "003"],
        "RFSTDTC": ["2024-01-15", "15JAN2024", "2024-01-17"],
        "RFENDTC": ["2024-06-15", "2024-06-16", "2024-06-17"],
        "SITEID": ["S01", "S01", "S02"],
        "AGE": [34, 51, -38],
        "AGEU": ["YEARS"] * 3,
        "SEX": ["M", "F", "FEMALE"],
        "RACE": ["WHITE", "ASIAN", "WHITE"],
        "ARMCD": ["DRUGA", "PBO", "DRUGA"],
        "ARM": ["Drug A", "Placebo", "Drug A"],
        "ACTARMCD": ["DRUGA", "PBO", "DRUGA"],
        "ACTARM": ["Drug A", "Placebo", "Drug A"],
        "COUNTRY": ["USA", "USA", "CAN"],
    }
)

report = pb.validate_sdtmig({"DM": dm})

print(report)
```

Five rules fail, and the 12 Define-XML rules are not applicable. Let's see which rules
failed:

```{python}
for rule in report.rules(status="fail"):
    print(rule.rule_id, rule.sensitivity, rule.message)
```

The three problems are each found by a rule. The other two failures are for the TS and TA
datasets, which every SDTM submission needs. The failing records can be had as a DataFrame
with [`findings_df()`](`pointblank.ConformanceReport.findings_df`) or as a table with
[`get_findings_table()`](`pointblank.ConformanceReport.get_findings_table`).

With a Define-XML, the Define-XML rules are run as well. Pointblank comes with a DM dataset
as an XPT file and a Define-XML for it:

```{python}
import pyreadstat

dm, _ = pyreadstat.read_xport(pb.load_metadata_example("dm.xpt"))

report = pb.validate_sdtmig({"DM": dm}, define_xml=pb.load_metadata_example("define.xml"))

print(report)
print([rule.rule_id for rule in report.rules(status="fail")])
```

No rules are not applicable now. SDTM-057 is one of the Define-XML rules, and it finds a
`RACE` of `"BLACK"`, which isn't in the codelist that the Define-XML declares for `RACE`.
SDTM-008 finds the same value with the CDISC RACE codelist, and the other failures are for
missing variables (such as `COUNTRY`) and the missing TS and TA datasets. The dataset was
read as a Pandas DataFrame, which works as well as a Polars one.

See Also
--------
ConformanceReport : The report that is returned.
validate_cdisc_submission : Check a submission with the CDISC CORE engine.
SubmissionPackage : A set of datasets to check together.
validate_sdtm : Build a validation plan for one dataset from an SDTM template.

validate_cdisc_submission(source: 'str | Path | dict | SubmissionPackage', standard: 'str | None' = None, version: 'str | None' = None, define: 'str | Path | Any | None' = None, controlled_terminology: 'str | Sequence[str] | None' = None, agency: 'str | None' = None, ct_version: 'str | None' = None, study_id: 'str | None' = None, core: 'str | Sequence[str] | None' = None, core_cwd: 'str | Path | None' = None, cache: 'str | Path | None' = None, workdir: 'str | Path | None' = None) -> 'ConformanceReport'

Check a CDISC submission with the CDISC CORE engine.

CDISC CORE is CDISC's open-source engine for its conformance rules, and it has the
published rules of CDISC and of the FDA and PMDA. The `validate_cdisc_submission()` function
runs CORE on a submission in a single call: it makes a
[`SubmissionPackage`](`pointblank.SubmissionPackage`) from the `source` (unless it is one
already), runs CORE on it, and reads CORE's results into a
[`ConformanceReport`](`pointblank.ConformanceReport`). This is the same as calling the
package's [`validate_conformance()`](`pointblank.SubmissionPackage.validate_conformance`)
method with `engine="core"`.

CORE is a separate command-line tool (`cdisc-rules-engine`) that has to be installed. If it
isn't on the `PATH`, give the command with `core=` or in the `POINTBLANK_CDISC_CORE`
environment variable. For checks that need no other software, use
[`validate_sdtmig()`](`pointblank.validate_sdtmig`), which runs Pointblank's bundled SDTMIG
rules.

Parameters
----------
source
    The submission to check. This can be the path of a folder of datasets (`.xpt` or
    Dataset-JSON files, which are read with
    [`SubmissionPackage.from_folder()`](`pointblank.SubmissionPackage.from_folder`)), a dict of
    DataFrames keyed by dataset name, or a `SubmissionPackage`.
standard
    The CDISC standard, such as `"sdtmig"`. The default is `"sdtmig"`, or the package's
    `standard` if `source` is a `SubmissionPackage`.
version
    The version of the standard, such as `"3.4"`. The default is `"3.4"`, or the package's
    `standard_version` if `source` is a `SubmissionPackage`.
define
    The path of the study's Define-XML, which CORE uses in its checks. For a folder, a
    `define*.xml` file in it is found without this. It is ignored if `source` is a
    `SubmissionPackage`, whose own `define` is used. The default is `None`.
controlled_terminology
    The controlled terminology package or packages for CORE's `-ct` option, such as
    `"sdtmct-2024-03-29"`. The default is `None`.
agency
    The regulatory agency, such as `"FDA"` or `"PMDA"`, which is shown in the report. It
    doesn't change the rules that CORE runs. The default is `None`.
ct_version
    The controlled terminology version to record in the package. It is only shown in the
    package's summary and isn't given to CORE. It is ignored if `source` is a
    `SubmissionPackage`. The default is `None`.
study_id
    A study identifier to record in the package. It is ignored if `source` is a
    `SubmissionPackage`. The default is `None`.
core
    How to run CORE. This can be the path or name of the CORE executable, or a command as a
    list (such as `["python", "core.py"]`). With the default of `None`, the command is taken
    from the `POINTBLANK_CDISC_CORE` environment variable, or else CORE is looked for on the
    `PATH`.
core_cwd
    The directory to run CORE from. This is needed when CORE is run from a checkout of its
    repository, since it looks for its `resources/` folder in the current directory. The
    default is `None`.
cache
    The path to CORE's rules cache (its `-ca` option). The default is `None`.
workdir
    A directory for the XPT files that the datasets are written to and for CORE's report.
    With the default of `None`, a temporary directory is used and then deleted.

Returns
-------
ConformanceReport
    A report of CORE's results, whose [`is_core`](`pointblank.ConformanceReport.is_core`) is
    `True`.

Raises
------
NotADirectoryError
    If `source` is a path that isn't a folder.
TypeError
    If `source` isn't a path, a dict or a `SubmissionPackage`.

Examples
--------
CORE has to be installed for this function, so these examples aren't run here. Let's check
a folder of SDTM datasets against SDTMIG 3.4, for a submission to the FDA:

```python
import pointblank as pb

report = pb.validate_cdisc_submission(
    "study_xyz/sdtm/",
    standard="sdtmig",
    version="3.4",
    controlled_terminology="sdtmct-2024-03-29",
    agency="FDA",
)

print(report)
```

Printing the report gives the standard, the version of CORE, the number of rules with each
of CORE's statuses and the total number of issues. The issues and the failing records can
then be looked at, and the report saved for review:

```python
for issue in report.issues(status="ISSUE REPORTED"):
    print(issue["dataset"], issue["rule_id"], issue["issues"], issue["message"])

report.to_excel("study_xyz/conformance/core_report.xlsx")
```

If CORE was installed from a checkout of its repository, it has to be run from that
directory, as in `core=["python", "core.py"], core_cwd="path/to/cdisc-rules-engine"`. A
report that CORE wrote on its own can be read with
[`ConformanceReport.from_core_report()`](`pointblank.ConformanceReport.from_core_report`),
which has an example that runs without CORE.

See Also
--------
validate_sdtmig : Check SDTM datasets with Pointblank's bundled rules.
SubmissionPackage : A set of datasets to check together.
ConformanceReport : The report that is returned.

SubmissionPackage(datasets: 'dict[str, Any]' = <factory>, define: 'Any' = None, ct_version: 'str | None' = None, standard: 'str' = 'sdtmig', standard_version: 'str' = '3.4', study_id: 'str | None' = None) -> None

A study's datasets, gathered for CDISC conformance validation.

A `SubmissionPackage` holds the datasets of a study (SDTM domains, SUPP-- qualifiers, RELREC
and ADaM datasets) together with the study's Define-XML. Where
[`validate_sdtm()`](`pointblank.validate_sdtm`) and
[`validate_adam()`](`pointblank.validate_adam`) look at one dataset at a time, a package
knows about all of them, so it can check how they relate: for example, whether every subject
in AE has a record in DM. It is the data-level counterpart of
[`MetadataPackage`](`pointblank.MetadataPackage`), which holds the metadata of many datasets.

A package can be built from DataFrames, as here, or from a folder of XPT and Dataset-JSON
files with [`from_folder()`](`pointblank.SubmissionPackage.from_folder`). Its
[`validate_conformance()`](`pointblank.SubmissionPackage.validate_conformance`) method runs
the conformance checks and returns a [`ConformanceReport`](`pointblank.ConformanceReport`).
The package also has some helpers for looking across datasets, such as
[`orphan_ids()`](`pointblank.SubmissionPackage.orphan_ids`), which finds the subjects in one
dataset that are missing from another.

Parameters
----------
datasets
    A dictionary that maps each dataset name (a domain code such as `"DM"`, `"AE"`,
    `"SUPPAE"` or `"ADSL"`) to the dataset, as a Polars or Pandas DataFrame. The names are
    case-insensitive and are stored in uppercase. The default is an empty dictionary.
define
    The study's Define-XML, as a path to a `define.xml` file or as a
    [`MetadataPackage`](`pointblank.MetadataPackage`) from
    [`import_metadata()`](`pointblank.import_metadata`). With the built-in rules engine, it
    activates the rules that check the datasets against Define-XML. The CORE engine only
    accepts a path. The default is `None`.
ct_version
    The version of CDISC Controlled Terminology that the study uses (such as
    `"2024-03-29"`). It is shown by [`summary()`](`pointblank.SubmissionPackage.summary`) but
    doesn't affect the checks. To choose the terminology the built-in engine checks against,
    use the `ct_packages=` argument of `validate_conformance()`. The default is `None`.
standard
    The data standard that the datasets follow, `"sdtmig"` or `"adamig"`. The default is
    `"sdtmig"`.
standard_version
    The version of the standard, such as `"3.4"` for SDTMIG 3.4. The default is `"3.4"`.
study_id
    A study identifier. It is shown by `summary()` and used in the labels of the validations
    made by `validate_conformance(engine="validate")`. The default is `None`.

Examples
--------
Let's build a package from a Demographics (DM) and an Adverse Events (AE) dataset. One of the
adverse events belongs to subject `STUDY01-009`, who isn't in DM:

```{python}
import pointblank as pb
import polars as pl

dm = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "DOMAIN": ["DM"] * 3,
        "USUBJID": ["STUDY01-001", "STUDY01-002", "STUDY01-003"],
        "SEX": ["M", "F", "F"],
    }
)
ae = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "DOMAIN": ["AE"] * 3,
        "USUBJID": ["STUDY01-001", "STUDY01-002", "STUDY01-009"],
        "AESEQ": [1, 1, 1],
        "AETERM": ["HEADACHE", "NAUSEA", "RASH"],
    }
)

study = pb.SubmissionPackage(datasets={"dm": dm, "AE": ae}, study_id="STUDY01")

print(study)
```

The dataset names are stored in uppercase, so `"dm"` became `DM`. The package can find the
subject in AE that has no DM record:

```{python}
study.orphan_ids("AE")
```

The validation checks the whole package. The `"validate"` engine builds a
[`Validate`](`pointblank.Validate`) for each dataset, with the structural checks from the SDTM
templates and the checks across datasets:

```{python}
report = study.validate_conformance(engine="validate")

print(report)
```

The AE validation has one failing test unit, from the check that each `USUBJID` in AE exists
in DM. The default engine, the built-in rules engine, checks the package against the SDTMIG
rules instead. See
[`validate_conformance()`](`pointblank.SubmissionPackage.validate_conformance`) for how the
engines differ.

See Also
--------
validate_sdtmig : Check SDTM datasets against the SDTMIG rules, without building a package.
validate_cdisc_submission : Check a submission with the CDISC CORE engine.
ConformanceReport : The report that validate_conformance() returns.
MetadataPackage : The metadata-level counterpart of a submission package.

ConformanceReport(validations: 'dict[str, Validate]' = <factory>, package: 'SubmissionPackage | None' = None, agency: 'str | None' = None, core: 'ParsedCoreReport | None' = None, native_result: 'NativeConformanceResult | None' = None) -> None

The results of a CDISC conformance check.

A `ConformanceReport` holds the results of checking a set of CDISC datasets for conformance
to their standard. It is returned by [`validate_sdtmig()`](`pointblank.validate_sdtmig`),
[`validate_cdisc_submission()`](`pointblank.validate_cdisc_submission`) and the
`SubmissionPackage` method
[`validate_conformance()`](`pointblank.SubmissionPackage.validate_conformance`), and is
rarely made directly. The report has one of three forms, depending on the engine that
did the checks:

- **Built-in rules** ([`is_rules`](`pointblank.ConformanceReport.is_rules`) is `True`): the
  results of Pointblank's bundled catalog of SDTMIG rules. Each rule has a status of
  `"pass"`, `"fail"`, `"error"` (the rule couldn't be evaluated), `"not_applicable"` (an input
  that it needs, such as a Define-XML, wasn't given) or `"not_supported"`. The rules that
  check values give the failing records, which are in
  [`findings_df()`](`pointblank.ConformanceReport.findings_df`) and
  [`get_findings_table()`](`pointblank.ConformanceReport.get_findings_table`).
- **CDISC CORE** ([`is_core`](`pointblank.ConformanceReport.is_core`) is `True`): the results
  of the CDISC CORE engine, with CORE's own rule statuses (such as `"SUCCESS"`,
  `"ISSUE REPORTED"` and `"SKIPPED"`). A report of this form can also be made from a saved
  CORE report with [`from_core_report()`](`pointblank.ConformanceReport.from_core_report`).
- **Validate-based**: a [`Validate`](`pointblank.Validate`) validation for each dataset, from
  `SubmissionPackage.validate_conformance(engine="validate")`. Each validation can be had with
  [`get_validation()`](`pointblank.ConformanceReport.get_validation`).

The methods [`summary()`](`pointblank.ConformanceReport.summary`),
[`issues()`](`pointblank.ConformanceReport.issues`) and
[`all_passed()`](`pointblank.ConformanceReport.all_passed`) work for all three forms. In a
notebook, a report of the built-in rules is shown as the table of
[`get_tabular_report()`](`pointblank.ConformanceReport.get_tabular_report`). A CORE report is
shown as a short summary with a table of its issues, and a Validate-based report as the
validation report of each dataset. Printing a report of any form gives a text summary.

Parameters
----------
validations
    The validation for each dataset, keyed by dataset name, for a Validate-based report. The
    default is an empty dict.
package
    The [`SubmissionPackage`](`pointblank.SubmissionPackage`) that was checked. The default is
    `None`.
agency
    The regulatory agency, such as `"FDA"`, which is only used as a label in the report. The
    default is `None`.
core
    CORE's results, for a CDISC CORE report. The default is `None`.
native_result
    The results of the rules, for a report of the built-in rules. The default is `None`.

Examples
--------
Let's check a small DM dataset with the built-in rules. It has a date that isn't in the
ISO 8601 format, a negative age and a `SEX` value that isn't in its codelist:

```{python}
import pointblank as pb
import polars as pl

dm = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "DOMAIN": ["DM"] * 3,
        "USUBJID": ["STUDY01-001", "STUDY01-002", "STUDY01-003"],
        "RFSTDTC": ["2024-01-15", "15JAN2024", "2024-01-17"],
        "AGE": [34, 51, -38],
        "SEX": ["M", "F", "FEMALE"],
    }
)

report = pb.validate_sdtmig({"DM": dm})

print(report)
```

Ten of the 362 rules fail. Most of them are about what the dataset lacks, such as the DM
variables `SUBJID` and `RACE` and the TS and TA datasets. The failing records are found by
three of the rules:

```{python}
report.findings_df().select("rule_id", "row_index", "checked_column", "checked_value")
```

The `row_index` of a finding is the 0-based position of the record in its dataset. To see the
results of all the rules as a table, use
[`get_tabular_report()`](`pointblank.ConformanceReport.get_tabular_report`).

See Also
--------
validate_sdtmig : Check SDTM datasets with the built-in rules.
validate_cdisc_submission : Check a folder of datasets with CDISC CORE.
SubmissionPackage : A set of datasets to check together.


## Integrations

Classes for integrating Pointblank with external observability and monitoring systems. Use `OTelExporter` to export validation results as OpenTelemetry metrics, traces, and logs.



OTelExporter(*, meter_name: 'str' = 'pointblank', meter_version: 'str | None' = None, enable_metrics: 'bool' = True, enable_tracing: 'bool' = False, enable_logging: 'bool' = False, meter_provider: 'MeterProvider | None' = None, tracer_provider: 'TracerProvider | None' = None, logger_provider: 'Any | None' = None, metric_prefix: 'str' = 'pb.validation', log_level: 'str' = 'warning', extra_attributes: 'dict[str, str] | None' = None) -> 'None'

Export Pointblank validation results as OpenTelemetry signals.

OpenTelemetry (OTel) is a vendor-neutral standard for application telemetry, which backends
such as Grafana, Datadog, New Relic and Prometheus can receive. An `OTelExporter` sends the
results of an interrogated [`Validate`](`pointblank.Validate`) object to OTel, so that data
quality shows up in the same dashboards and alerts as the rest of a pipeline's telemetry. Call
[`export()`](`pointblank.integrations.otel.OTelExporter.export`) after
[`interrogate()`](`pointblank.Validate.interrogate`) to send the results. For an export that
happens at the end of every interrogation, use [`emit_otel()`](`pointblank.emit_otel`) in
[`FinalActions`](`pointblank.FinalActions`) instead (it sends the same signals).

An export can produce three kinds of signal:

- **Metrics** (on by default). There are counters of the steps (`{prefix}.steps.total`,
`{prefix}.steps.passed` and `{prefix}.steps.failed`), of the test units
(`{prefix}.test_units.total`, `{prefix}.test_units.passed` and `{prefix}.test_units.failed`)
and of the steps that exceeded each threshold level (`{prefix}.threshold.warning`,
`{prefix}.threshold.error` and `{prefix}.threshold.critical`). There are gauges of the overall
pass rate (`{prefix}.pass_rate`) and of the interrogation time in seconds (`{prefix}.duration`),
and a histogram of the step processing times (`{prefix}.step.duration`). Here `{prefix}` is the
`metric_prefix=` value. Counters add up over exports, while a gauge keeps the latest value.
- **Traces** (`enable_tracing=True`). A `pb.validate` span covers the interrogation, with a
`pb.validate.step` child span for each step. A step's span has its test unit counts and
fractions and the threshold levels it exceeded. The `pb.validate` span also has a
`threshold_breach` event for each threshold level that each step exceeded.
- **Logs** (`enable_logging=True`). There is one log record for each step that exceeded a
threshold, at the highest level it exceeded (`WARNING`, `ERROR` or `CRITICAL`). The
`log_level=` value sets the lowest level that is logged.

Every signal has the attributes `pb.tbl_name`, `pb.label`, `pb.owner` and `pb.version` (for the
ones that are set in the `Validate` object), plus those in `extra_attributes=`. The signals for
individual steps also have `pb.step.i`, `pb.step.assertion_type` and `pb.step.column`.

The signals go to the providers given as `meter_provider=`, `tracer_provider=` and
`logger_provider=`. Without them, metrics and traces go to OTel's global providers, which
discard everything until the application sets up the OTel SDK (for example, with an OTLP
exporter that sends to a collector). Log records have no global provider to fall back on, so
logging needs `logger_provider=`.

The OTel packages are an optional dependency, installed with `pip install pointblank[otel]`.

Parameters
----------
meter_name
    The name that identifies Pointblank as the source of the signals (the OTel instrumentation
    scope), used for the meter, the tracer and the logger. The default is `"pointblank"`.
meter_version
    The version of the instrumentation scope. The default is `None`, which uses the installed
    version of Pointblank.
enable_metrics
    Whether to send the metrics. The default is `True`.
enable_tracing
    Whether to send the trace spans. The default is `False`.
enable_logging
    Whether to send log records for the steps that exceeded a threshold. This needs
    `logger_provider=`. The default is `False`.
meter_provider
    An OTel `MeterProvider` for the metrics. The default is `None`, which uses the global meter
    provider.
tracer_provider
    An OTel `TracerProvider` for the trace spans. The default is `None`, which uses the global
    tracer provider.
logger_provider
    An OTel `LoggerProvider` for the log records, needed when `enable_logging=True`. The
    default is `None`.
metric_prefix
    The prefix for the names of the metrics. The default is `"pb.validation"`.
log_level
    The lowest threshold level that produces a log record: `"warning"` (every step that
    exceeded a threshold), `"error"` or `"critical"`. The default is `"warning"`.
extra_attributes
    A dictionary of attributes to add to every signal, such as the environment or the name of
    the pipeline. The default is `None`.

Raises
------
ImportError
    If the OTel packages aren't installed.
ValueError
    If `enable_logging=True` is given without `logger_provider=`, or if `log_level=` isn't one
    of the accepted values (they're matched without regard to case).

Examples
--------
Let's validate a small table of orders, which has a missing customer name and two amounts
that aren't positive:

```{python}
import pointblank as pb
import polars as pl

orders = pl.DataFrame(
    {
        "order_id": [101, 102, 103, 104, 105],
        "amount": [29.99, 150.00, -5.00, 75.50, 0.00],
        "customer": ["Alice", "Bob", None, "Dana", "Eve"],
    }
)

validation = (
    pb.Validate(data=orders, tbl_name="orders", thresholds=(0.05, 0.20, 0.40))
    .col_vals_not_null(columns="customer")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)
```

In production, the exporter would use providers that send to a collector. Here, the OTel SDK's
`InMemoryMetricReader` keeps the metrics in memory so we can look at them:

```{python}
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from pointblank.integrations.otel import OTelExporter

reader = InMemoryMetricReader()
meter_provider = MeterProvider(metric_readers=[reader])

exporter = OTelExporter(meter_provider=meter_provider, extra_attributes={"env": "dev"})
exporter.export(validation)

metrics = reader.get_metrics_data().resource_metrics[0].scope_metrics[0].metrics

for metric in metrics:
    for point in metric.data.data_points:
        value = point.count if hasattr(point, "count") else point.value
        print(metric.name, value)
```

Both steps failed, and 7 of the 10 test units passed, for a pass rate of `0.7`. The first step
(20% failing) exceeded the warning and error thresholds, and the second (40% failing) exceeded
all three. The step duration histogram has a data point for each step, and the value printed
for it is its count of one observation. Every data point has the `pb.tbl_name` and `env`
attributes (and the histogram's also have the `pb.step.*` ones):

```{python}
dict(metrics[0].data.data_points[0].attributes)
```

For traces, we give the exporter a `TracerProvider` with an in-memory span exporter. This
exporter sends traces only:

```{python}
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

span_exporter = InMemorySpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))

OTelExporter(
    enable_metrics=False, enable_tracing=True, tracer_provider=tracer_provider
).export(validation)

for span in span_exporter.get_finished_spans():
    attrs = span.attributes
    print(span.name, attrs.get("pb.step.assertion_type"), attrs.get("pb.step.n_failed"))
```

The two `pb.validate.step` spans (one per step, with its number of failing test units) end
before the `pb.validate` span that contains them, so they're listed first. The `pb.validate`
span has five `threshold_breach` events: two for the first step and three for the second. The
[`export()`](`pointblank.integrations.otel.OTelExporter.export`) method has an example of
sending log records.

See Also
--------
emit_otel : Send the metrics automatically at the end of an interrogation.
FinalActions : Run actions after an interrogation.


----------------------------------------------------------------------
This is the CLI documentation for the package.
----------------------------------------------------------------------

## CLI: pb

```
Usage: pb [OPTIONS] COMMAND [ARGS]...

  Pointblank CLI: Data validation and quality tools for data engineers.

  Use this CLI to validate data quality, explore datasets, and generate
  comprehensive reports for CSV, Parquet, and database sources. Suitable for
  data pipelines, ETL validation, and exploratory data analysis from the
  command line.

  Quick Examples:

    pb preview data.csv              Preview your data
    pb scan data.csv                 Generate data profile
    pb validate data.csv             Run basic validation

  Use pb COMMAND --help for detailed help on any command.

Options:
  -v, --version  Show the version and exit.
  -h, --help     Show this message and exit.

Commands:
  info           Display information about a data source.
  preview        Preview a data table showing head and tail rows.
  scan           Generate a data scan profile report.
  missing        Generate a missing values report for a data table.
  validate       Perform single or multiple data validations.
  run            Run a Pointblank validation script or YAML configuration.
  compare        Compare two tables row by row.
  make-template  Create a validation script or YAML configuration template.
  pl             Execute Polars expressions and display results.
  datasets       List available built-in datasets.
  requirements   Check installed dependencies and their availability.
  edit           Edit an existing validation plan with a natural-language...
```

### pb info

```
Usage: pb info [OPTIONS] [DATA_SOURCE]

  Display information about a data source.

  Shows table type, dimensions, column names, and data types.

  DATA_SOURCE can be:

  - CSV file path (e.g., data.csv)
  - Parquet file path or pattern (e.g., data.parquet, data/*.parquet)
  - GitHub URL to CSV/Parquet (e.g., https://github.com/user/repo/blob/main/data.csv)
  - Database connection string (e.g., duckdb:///path/to/db.ddb::table_name)
  - Dataset name from pointblank (small_table, game_revenue, nycflights, global_sales)

Options:
  --help  Show this message and exit.
```

### pb preview

```
Usage: pb preview [OPTIONS] [DATA_SOURCE]

  Preview a data table showing head and tail rows.

  DATA_SOURCE can be:

  - CSV file path (e.g., data.csv)
  - Parquet file path or pattern (e.g., data.parquet, data/*.parquet)
  - GitHub URL to CSV/Parquet (e.g., https://github.com/user/repo/blob/main/data.csv)
  - Database connection string (e.g., duckdb:///path/to/db.ddb::table_name)
  - Dataset name from pointblank (small_table, game_revenue, nycflights, global_sales)
  - Piped data from pb pl command

  COLUMN SELECTION OPTIONS:

  For tables with many columns, use these options to control which columns are
  displayed:

  - --columns: Specify exact columns (--columns "name,age,email")
  - --col-range: Select column range (--col-range "1:10", --col-range "5:", --col-range ":15")
  - --col-first: Show first N columns (--col-first 5)
  - --col-last: Show last N columns (--col-last 3)

  Tables with >15 columns automatically show first 7 and last 7 columns with
  indicators.

Options:
  --columns TEXT             Comma-separated list of columns to display
  --col-range TEXT           Column range like '1:10' or '5:' or ':15'
                             (1-based indexing)
  --col-first INTEGER        Show first N columns
  --col-last INTEGER         Show last N columns
  --head INTEGER             Number of rows from the top (default: 5)
  --tail INTEGER             Number of rows from the bottom (default: 5)
  --limit INTEGER            Maximum total rows to display (default: 50)
  --no-row-numbers           Hide row numbers
  --max-col-width INTEGER    Maximum column width in pixels (default: 250)
  --min-table-width INTEGER  Minimum table width in pixels (default: 500)
  --no-header                Hide table header
  --output-html PATH         Save HTML output to file
  --help                     Show this message and exit.
```

### pb scan

```
Usage: pb scan [OPTIONS] [DATA_SOURCE]

  Generate a data scan profile report.

  Produces a comprehensive data profile including:

  - Column types and distributions
  - Missing value patterns
  - Basic statistics
  - Data quality indicators

  DATA_SOURCE can be:

  - CSV file path (e.g., data.csv)
  - Parquet file path or pattern (e.g., data.parquet, data/*.parquet)
  - GitHub URL to CSV/Parquet (e.g., https://github.com/user/repo/blob/main/data.csv)
  - Database connection string (e.g., duckdb:///path/to/db.ddb::table_name)
  - Dataset name from pointblank (small_table, game_revenue, nycflights, global_sales)
  - Piped data from pb pl command

Options:
  --output-html PATH  Save HTML scan report to file
  -c, --columns TEXT  Comma-separated list of columns to scan
  --help              Show this message and exit.
```

### pb missing

```
Usage: pb missing [OPTIONS] [DATA_SOURCE]

  Generate a missing values report for a data table.

  DATA_SOURCE can be:

  - CSV file path (e.g., data.csv)
  - Parquet file path or pattern (e.g., data.parquet, data/*.parquet)
  - GitHub URL to CSV/Parquet (e.g., https://github.com/user/repo/blob/main/data.csv)
  - Database connection string (e.g., duckdb:///path/to/db.ddb::table_name)
  - Dataset name from pointblank (small_table, game_revenue, nycflights, global_sales)
  - Piped data from pb pl command

Options:
  --output-html PATH  Save HTML output to file
  --help              Show this message and exit.
```

### pb validate

```
Usage: pb validate [OPTIONS] [DATA_SOURCE]

  Perform single or multiple data validations.

  Run one or more validation checks on your data in a single command. Use
  multiple --check options to perform multiple validations.

  DATA_SOURCE can be:

  - CSV file path (e.g., data.csv)
  - Parquet file path or pattern (e.g., data.parquet, data/*.parquet)
  - GitHub URL to CSV/Parquet (e.g., https://github.com/user/repo/blob/main/data.csv)
  - Database connection string (e.g., duckdb:///path/to/db.ddb::table_name)
  - Dataset name from pointblank (small_table, game_revenue, nycflights, global_sales)

  AVAILABLE CHECK_TYPES:

  Require no additional options:

  - rows-distinct: Check if all rows in the dataset are unique (no duplicates)
  - rows-complete: Check if all rows are complete (no missing values in any column)

  Require --column:

  - col-exists: Check if a specific column exists in the dataset
  - col-vals-not-null: Check if all values in a column are not null/missing

  Require --column and --value:

  - col-vals-gt: Check if column values are greater than a fixed value
  - col-vals-ge: Check if column values are greater than or equal to a fixed value
  - col-vals-lt: Check if column values are less than a fixed value
  - col-vals-le: Check if column values are less than or equal to a fixed value

  Require --column and --set:

  - col-vals-in-set: Check if column values are in an allowed set

  Use --list-checks to see all available validation methods with examples. The
  default CHECK_TYPE is 'rows-distinct' which checks for duplicate rows.

  Examples:

  pb validate data.csv                               # Uses default validation (rows-distinct)
  pb validate data.csv --list-checks                 # Show all available checks
  pb validate data.csv --check rows-distinct
  pb validate data.csv --check rows-distinct --show-extract
  pb validate data.csv --check rows-distinct --write-extract failing_rows_folder
  pb validate data.csv --check rows-distinct --exit-code
  pb validate data.csv --check col-exists --column price
  pb validate data.csv --check col-vals-not-null --column email
  pb validate data.csv --check col-vals-gt --column score --value 50
  pb validate data.csv --check col-vals-in-set --column status --set "active,inactive,pending"

  Multiple validations in one command: pb validate data.csv --check rows-
  distinct --check rows-complete

Options:
  --list-checks         List available validation checks and exit
  --check CHECK_TYPE    Type of validation check to perform. Can be used
                        multiple times for multiple checks.
  --column TEXT         Column name or integer position as #N (1-based index)
                        for validation.
  --set TEXT            Comma-separated allowed values for col-vals-in-set
                        checks.
  --value FLOAT         Numeric value for comparison checks.
  --show-extract        Show extract of failing rows if validation fails
  --write-extract TEXT  Save failing rows to folder. Provide base name for
                        folder.
  --limit INTEGER       Maximum number of failing rows to save to CSV
                        (default: 500)
  --exit-code           Exit with non-zero code if validation fails
  --help                Show this message and exit.
```

### pb compare

```
Usage: pb compare [OPTIONS] SOURCE TARGET

  Compare two tables row by row.

  SOURCE is the reference (expected) table and TARGET is the table to check.
  Each can be a CSV or Parquet file (scanned lazily, so large files work), a
  database connection string, or a built-in dataset name. Every row is
  classified as matching, changed, missing from the target, extra in the
  target, or having a duplicated key; schema differences are reported too.

  Exit codes: 0 if the tables match, 1 if they differ, 2 on error.

  Examples:

    pb compare source.csv target.csv --key id
    pb compare old.parquet new.parquet -k id -t amount=rtol:1e-6 --output-html diff.html
    pb compare a.csv b.csv -k region -k date --schema common --format json
    pb compare big_a.csv big_b.csv -k id --engine duckdb --partitions 8

Options:
  -k, --key TEXT                Key column used to align rows (repeat for
                                composite keys). Use '*' to compare as
                                multisets of rows. Without a key, rows are
                                compared by position.
  -c, --columns TEXT            Comma-separated source columns to compare
  --column-map TEXT             Renamed column as SOURCE_COLUMN=TARGET_COLUMN
                                (repeatable)
  -t, --tolerance TEXT          Tolerance as [COLUMN=]SPEC; SPEC is a number
                                (absolute), a duration (1s, 500ms), or
                                atol:X,rtol:Y (repeatable)
  --normalize TEXT              Normalization as COLUMN=op[,op...] with ops
                                strip, lower, upper, collapse_whitespace,
                                round:N (repeatable)
  --null-unequal                Treat two nulls as different values
  --schema [strict|common]      'strict': any schema difference fails all
                                rows; 'common': compare shared columns only
                                [default: strict]
  --dup-keys [flag|compare]     'compare' also compares rows with duplicated
                                keys pairwise in the report  [default: flag]
  --order-by TEXT               Comma-separated columns defining row order
                                (positional mode)
  --partitions INTEGER RANGE    Compare in N passes by key hash to bound
                                memory use for very large tables  [x>=1]
  --engine [polars|duckdb]      Engine for reading CSV/Parquet files: 'polars'
                                is fastest, 'duckdb' uses the least memory
                                (especially with --partitions)  [default:
                                polars]
  --limit INTEGER               Maximum sample rows shown per status
                                [default: 10]
  --output-html PATH            Save the HTML comparison report to file
  --output-json PATH            Save the comparison results as JSON to file
  --format [text|json]          Terminal output: a summary ('text') or the
                                results as JSON ('json')  [default: text]
  --no-samples                  Leave sample row values out of JSON output
                                (e.g., for sensitive data)
  --exit-code / --no-exit-code  Exit with code 1 when the tables differ
                                [default: exit-code]
  --help                        Show this message and exit.
```

### pb datasets

```
Usage: pb datasets [OPTIONS]

  List available built-in datasets.

Options:
  --help  Show this message and exit.
```

### pb requirements

```
Usage: pb requirements [OPTIONS]

  Check installed dependencies and their availability.

Options:
  --help  Show this message and exit.
```

### pb make-template

```
Usage: pb make-template [OPTIONS] [OUTPUT_FILE]

  Create a validation script or YAML configuration template.

  Creates a sample Python script or YAML configuration with examples showing
  how to use Pointblank for data validation. The template type is determined
  by the file extension: - .py files create Python script templates -
  .yaml/.yml files create YAML configuration templates

  Edit the template to add your own data loading and validation rules, then
  run it with 'pb run'.

  OUTPUT_FILE is the path where the template will be created.

  Examples:

  pb make-template my_validation.py        # Creates Python script template
  pb make-template my_validation.yaml      # Creates YAML config template
  pb make-template validation_template.yml # Creates YAML config template

Options:
  --help  Show this message and exit.
```

### pb run

```
Usage: pb run [OPTIONS] [VALIDATION_FILE]

  Run a Pointblank validation script or YAML configuration.

  VALIDATION_FILE can be: - A Python file (.py) that defines validation logic
  - A YAML configuration file (.yaml, .yml) that defines validation steps

  Python scripts should load their own data and create validation objects.
  YAML configurations define data sources and validation steps declaratively.

  If --data is provided, it will automatically replace the data source in your
  validation objects (Python scripts) or override the 'tbl' field (YAML
  configs).

  Python code in a YAML config isn't run unless you use --allow-python. Only
  do that for YAML you trust, since the code runs with full access to your
  system (as a Python script does).

  To get started quickly, use 'pb make-template' to create templates.

  DATA can be:

  - CSV file path (e.g., data.csv)
  - Parquet file path or pattern (e.g., data.parquet, data/*.parquet)
  - GitHub URL to CSV/Parquet (e.g., https://github.com/user/repo/blob/main/data.csv)
  - Database connection string (e.g., duckdb:///path/to/db.ddb::table_name)
  - Dataset name from pointblank (small_table, game_revenue, nycflights, global_sales)

  Examples:

  pb make-template my_validation.py  # Create a Python template
  pb run validation_script.py
  pb run validation_config.yaml
  pb run validation_script.py --data data.csv
  pb run validation_config.yaml --data small_table --output-html report.html
  pb run validation_script.py --show-extract --fail-on error
  pb run validation_config.yaml --write-extract extracts_folder --fail-on critical
  pb run trusted_config.yaml --allow-python  # Run Python code in the YAML

Options:
  --data TEXT                     Data source to replace in validation objects
                                  (Python scripts and YAML configs)
  --output-html PATH              Save HTML validation report to file
  --output-json PATH              Save JSON validation summary to file
  --show-extract                  Show extract of failing rows if validation
                                  fails
  --write-extract TEXT            Save failing rows to folders (one CSV per
                                  step). Provide base name for folder.
  --limit INTEGER                 Maximum number of failing rows to save to
                                  CSV (default: 500)
  --fail-on [critical|error|warning|any]
                                  Exit with non-zero code when validation
                                  reaches this threshold level ('any' also
                                  counts failing test units and steps that
                                  couldn't be evaluated)
  --allow-python                  Run Python code in a YAML config (python:
                                  blocks and the expr/pre/active shortcuts).
                                  The code has full access to your system, so
                                  only use this with YAML you trust
  --help                          Show this message and exit.
```

### pb edit

```
Usage: pb edit [OPTIONS] [VALIDATION_FILE]

  Edit an existing validation plan with a natural-language instruction (AI
  Validation Editor).

  VALIDATION_FILE is a Python (.py) or YAML (.yaml/.yml) file containing a
  validation plan. The instruction is sent, along with the current plan, to
  the specified model; the proposed change is shown as a diff for review and
  can optionally be written to a file.

  Examples:

  pb edit plan.py -i "add a not-null check on user_id" -m anthropic:claude-opus-4-8
  pb edit plan.yaml -i "tighten the price range to 0-1000" -m openai:gpt-6-luna --data sales.csv
  pb edit plan.py -i "drop the email regex check" -m anthropic:claude-opus-4-8 -o plan2.py -y

Options:
  -i, --instruction TEXT  Plain-English description of the change to make to
                          the plan.
  -m, --model TEXT        Model to use, as provider:model (e.g.,
                          anthropic:claude-opus-4-8).
  --data TEXT             Optional data source used for DataScan-informed
                          edits and validation.
  -o, --output PATH       Write the revised plan to this file.
  -y, --yes               Write the output file without asking for
                          confirmation.
  --help                  Show this message and exit.
```

### pb pl

```
Usage: pb pl [OPTIONS] [POLARS_EXPRESSION]

  Execute Polars expressions and display results.

  Execute Polars DataFrame operations from the command line and display the
  results using Pointblank's visualization tools.

  POLARS_EXPRESSION should be a valid Polars expression that returns a
  DataFrame. The 'pl' module is automatically imported and available.

  Examples:

  # Direct expression
  pb pl "pl.read_csv('data.csv')"
  pb pl "pl.read_csv('data.csv').select(['name', 'age'])"
  pb pl "pl.read_csv('data.csv').filter(pl.col('age') > 25)"

  # Multi-line with editor (supports multiple statements)
  pb pl --edit

  # Multi-statement code example in editor:
  # csv = pl.read_csv('data.csv')
  # result = csv.select(['name', 'age']).filter(pl.col('age') > 25)

  # Multi-line with a specific editor
  pb pl --edit --editor nano
  pb pl --edit --editor code
  pb pl --edit --editor micro

  # From file
  pb pl --file query.py

  Piping to other pb commands
  pb pl "pl.read_csv('data.csv').head(20)" --pipe | pb validate --check rows-distinct
  pb pl --edit --pipe | pb preview --head 10
  pb pl --edit --pipe | pb scan --output-html report.html
  pb pl --edit --pipe | pb missing --output-html missing_report.html

  Use --output-format to change how results are displayed:
  pb pl "pl.read_csv('data.csv')" --output-format scan
  pb pl "pl.read_csv('data.csv')" --output-format missing
  pb pl "pl.read_csv('data.csv')" --output-format info

  Note: For multi-statement code, assign your final result to a variable like
  'result', 'df', 'data', or ensure it's the last expression.

Options:
  -e, --edit                      Open editor for multi-line input
  -f, --file PATH                 Read query from file
  --editor TEXT                   Editor to use for --edit mode (overrides
                                  $EDITOR and auto-detection)
  -o, --output-format [preview|scan|missing|info]
                                  Output format for the result
  --preview-head INTEGER          Number of head rows for preview
  --preview-tail INTEGER          Number of tail rows for preview
  --output-html PATH              Save HTML output to file
  --pipe                          Output data in a format suitable for piping
                                  to other pb commands
  --pipe-format [parquet|csv]     Format for piped output (default: parquet)
  --help                          Show this message and exit.
```


----------------------------------------------------------------------
This is the User Guide documentation for the package.
----------------------------------------------------------------------

### Index

<div style="text-align: center;">

![](../assets/pointblank_logo.svg){width=85%}

**Data validation toolkit for assessing and monitoring data quality.**

</div>

Pointblank is a data validation framework for Python that makes data quality checks beautiful,
powerful, and stakeholder-friendly. Instead of cryptic error messages, get stunning interactive
reports that turn data issues into conversations.

Here's what a validation looks like (click "Show the code" to see how it's done):

```{python}
#| code-fold: true
#| code-summary: "Show the code"

import pointblank as pb
import polars as pl

validation = (
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="polars"),
        tbl_name="game_revenue",
        label="Comprehensive validation of game revenue data",
        thresholds=pb.Thresholds(warning=0.10, error=0.25, critical=0.35),
        brief=True
    )
    .col_vals_regex(columns="player_id", pattern=r"^[A-Z]{12}[0-9]{3}$")        # STEP 1
    .col_vals_gt(columns="session_duration", value=20)                          # STEP 2
    .col_vals_ge(columns="item_revenue", value=0.20)                            # STEP 3
    .col_vals_in_set(columns="item_type", set=["iap", "ad"])                    # STEP 4
    .col_vals_in_set(                                                           # STEP 5
        columns="acquisition",
        set=["google", "facebook", "organic", "crosspromo", "other_campaign"]
    )
    .col_vals_not_in_set(columns="country", set=["Mongolia", "Germany"])        # STEP 6
    .col_vals_between(                                                          # STEP 7
        columns="session_duration",
        left=10, right=50,
        pre = lambda df: df.select(pl.median("session_duration")),
        brief="Expect that the median of `session_duration` should be between `10` and `50`."
    )
    .rows_distinct(columns_subset=["player_id", "session_id", "time"])          # STEP 8
    .row_count_match(count=2000)                                                # STEP 9
    .col_count_match(count=11)                                                  # STEP 10
    .col_vals_not_null(columns="item_type")                                     # STEP 11
    .col_exists(columns="start_day")                                            # STEP 12
    .interrogate()
)

validation.get_tabular_report(title="Game Revenue Validation Report")
```

That's the kind of report you get from Pointblank: clear, interactive, and designed for everyone on
your team. And if you need help getting started or want to work faster, Pointblank has built-in AI
support through the [`assistant()`](`assistant`) function to guide you along the way.
You can also use [`DraftValidation`](advanced-validation/draft-validation.qmd) to quickly generate a
validation plan from your existing data (great for getting started fast), or the
[AI Validation Editor](advanced-validation/ai-validation-editor.qmd) to refine an existing plan
with plain-English instructions.

Ready to validate? Start with our [Installation](getting-started/installation.qmd) guide or jump straight
to the [Quickstart](getting-started/quickstart.qmd).

By the way, Pointblank is made with 💙 by [Posit](https://posit.co/).

## What is Data Validation?

Data validation ensures your data meets quality standards before it's used in analysis, reports, or
downstream systems. Pointblank provides a structured way to define validation rules, execute them,
and communicate results to both technical and non-technical stakeholders.

With Pointblank you can:

- **Validate data** through a fluent, chainable API with [25+ validation methods](../reference/index.qmd#validation-steps)
- **Set thresholds** to define acceptable levels of data quality (warning, error, critical)
- **Take actions** when thresholds are exceeded (notifications, logging, custom functions)
- **Generate reports** that make data quality issues immediately understandable
- **Inspect data** with built-in tools for previewing, summarizing, and finding missing values

## Why Pointblank?

Pointblank is designed for the entire data team, not just engineers:

- 🎨 **Beautiful Reports**: Interactive validation reports that stakeholders actually want to read
- 📊 **Threshold Management**: Define quality standards with warning, error, and critical levels
- 🔍 **Error Drill-Down**: Inspect failing data to get to root causes quickly
- 🔗 **Universal Compatibility**: Works with Polars, Pandas, DuckDB, MySQL, PostgreSQL, SQLite, and more
- 🌍 **Multilingual Support**: Reports available in 40 languages for global teams
- 📝 **YAML Support**: Write validations in YAML for version control and team collaboration
- ⚡ **CLI Tools**: Run validations from the command line for CI/CD pipelines or as quick checks
- 📋 **Rich Inspection**: Preview data, analyze columns, and visualize missing values

## Quick Examples

### Threshold-Based Quality

Set expectations and react when data quality degrades (with alerts, logging, or custom functions):

```python
validation = (
    pb.Validate(data=sales_data, thresholds=(0.01, 0.02, 0.05)) # Three threhold levels set
    .col_vals_not_null(columns="customer_id")
    .col_vals_in_set(columns="status", set=["pending", "shipped", "delivered"])
    .interrogate()
)
```

### YAML Workflows

Works wonderfully for CI/CD pipelines and team collaboration:

```yaml
validate:
  data: sales_data
  tbl_name: "sales_data"
  thresholds: [0.01, 0.02, 0.05]

steps:
  - col_vals_not_null:
      columns: "customer_id"
  - col_vals_in_set:
      columns: "status"
      set: ["pending", "shipped", "delivered"]
```

```python
validation = pb.yaml_interrogate("validation.yaml")
```

### Command Line Power

Run validations without writing code:

```bash
# Quick validation
pb validate sales_data.csv --check col-vals-not-null --column customer_id

# Run YAML workflows
pb run validation.yaml --exit-code  # <- Great for CI/CD!

# Explore your data
pb scan sales_data.csv
pb missing sales_data.csv
```

## Installation

Install Pointblank using pip or conda:

```bash
pip install pointblank
# or
conda install conda-forge::pointblank
```

For specific backends:

```bash
pip install "pointblank[pl]"       # Polars support
pip install "pointblank[pd]"       # Pandas support
pip install "pointblank[duckdb]"   # DuckDB support
pip install "pointblank[postgres]" # PostgreSQL support
```

See the [Installation guide](getting-started/installation.qmd) for more details.

## Join the Community

We'd love to hear from you! Connect with us:

- [GitHub Issues](https://github.com/posit-dev/pointblank/issues) for bug reports and feature requests
- [Discord server](https://discord.com/invite/YH7CybCNCQ) for discussions and help
- [Contributing guidelines](https://github.com/posit-dev/pointblank/blob/main/CONTRIBUTING.md) if you'd like to contribute

---

**License**: MIT | **© 2024-2026 Posit Software, PBC**


### Getting Started



### Validation Plan



### Advanced Validation



### Post Interrogation



### Data Inspection



### Table Comparison



### Test Data Generation



### YAML



### Contracts & Pipelines



### The Pointblank CLI



### Integrations



### Metadata Import



### Reference



### Quickstart

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

The Pointblank library is all about assessing the state of data quality for a table. You provide the
validation rules and the library will dutifully interrogate the data and provide useful reporting.
We can use different types of tables like Polars and Pandas DataFrames, Parquet files, or various
database tables. Let's walk through what data validation looks like in Pointblank.

## A Simple Validation Table

This is a validation report table that is produced from a validation of a Polars DataFrame:

```{python}
#| code-fold: true
#| code-summary: "Show the code"
import pointblank as pb

(
    pb.Validate(data=pb.load_dataset(dataset="small_table"), label="Example Validation")
    .col_vals_lt(columns="a", value=10)
    .col_vals_between(columns="d", left=0, right=5000)
    .col_vals_in_set(columns="f", set=["low", "mid", "high"])
    .col_vals_regex(columns="b", pattern=r"^[0-9]-[a-z]{3}-[0-9]{3}$")
    .interrogate()
)
```

Each row in this reporting table constitutes a single validation step. Roughly, the left-hand side
outlines the validation rules and the right-hand side provides the results of each validation step.
While simple in principle, there's a lot of useful information packed into this validation table.

Here's a diagram that describes a few of the important parts of the validation table:

![](/assets/validation-table-diagram.png){width=100%}

There are three things that should be noted here:

- validation steps: each step is a separate test on the table, focused on a certain aspect of the
table
- validation rules: the validation type is provided here along with key constraints
- validation results: interrogation results are provided here, with a breakdown of test units
(*total*, *passing*, and *failing*), threshold flags, and more

The intent is to provide the key information in one place, and have it be interpretable by data
stakeholders. For example, a failure can be seen in the second row (notice there's a CSV button). A
data quality stakeholder could click this to download a CSV of the failing rows for that step.

## Example Code, Step-by-Step

This section will walk you through the example code used above.

```python
import pointblank as pb

(
    pb.Validate(data=pb.load_dataset(dataset="small_table"))
    .col_vals_lt(columns="a", value=10)
    .col_vals_between(columns="d", left=0, right=5000)
    .col_vals_in_set(columns="f", set=["low", "mid", "high"])
    .col_vals_regex(columns="b", pattern=r"^[0-9]-[a-z]{3}-[0-9]{3}$")
    .interrogate()
)
```

Note these three key pieces in the code:

- **data**: the `Validate(data=)` argument takes a DataFrame or database table that you want to validate
- **steps**: the methods starting with `col_vals_` specify validation steps that run on specific columns
- **execution**: the [`Validate.interrogate()`](`Validate.interrogate`) method executes the validation plan on the table

This common pattern is used in a validation workflow, where `Validate` and
[`Validate.interrogate()`](`Validate.interrogate`) bookend a validation plan generated through calling validation methods.

In the next few sections we'll go a bit further by understanding how we can measure data quality and
respond to failures.

## Understanding Test Units

Each validation step will execute a type of validation test on the target table. For example, a
[`Validate.col_vals_lt()`](`Validate.col_vals_lt`) validation step can test that each value in a column is less than a
specified number. And the key finding that's reported in each step is the number of *test units*
that pass or fail.

In the validation report table, test unit metrics are displayed under the `UNITS`, `PASS`, and
`FAIL` columns. This diagram explains what the tabulated values signify:

![](/assets/validation-test-units.png){width=100%}

Test units are dependent on the test being run. Some validation methods might test every value in a
particular column, so each value will be a test unit. Others will only have a single test unit since
they aren't testing individual values but rather if the overall test passes or fails.

## Setting Thresholds for Data Quality Signals

Understanding test units is essential because they form the foundation of Pointblank's threshold
system. Thresholds let you define acceptable levels of data quality, triggering different severity
signals ('warning', 'error', or 'critical') when certain failure conditions are met.

Here's a simple example that uses a single validation step along with thresholds set using the
`Thresholds` class:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table"))
    .col_vals_lt(
        columns="a",
        value=7,

        # Set the 'warning' and 'error' thresholds ---
        thresholds=pb.Thresholds(warning=2, error=4)
    )
    .interrogate()
)
```

If you look at the validation report table, we can see:

- the `FAIL` column shows that 2 tests units have failed
- the `W` column (short for 'warning') shows a filled gray circle indicating those failing test
units reached that threshold value
- the `E` column (short for 'error') shows an open yellow circle indicating that the number of
failing test units is below that threshold

The one final threshold level, `C` (for 'critical'), wasn't set so it appears on the validation
table as a long dash.

## Taking Action on Threshold Exceedances

Pointblank becomes even more powerful when you combine thresholds with actions. The
`Actions` class lets you trigger responses when validation failures exceed threshold levels, turning
passive reporting into active notifications.

Here's a simple example that adds an action to the previous validation:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table"))
    .col_vals_lt(
        columns="a",
        value=7,
        thresholds=pb.Thresholds(warning=2, error=4),

        # Set an action for the 'warning' threshold ---
        actions=pb.Actions(
            warning="WARNING: Column 'a' has values that aren't less than 7."
        )
    )
    .interrogate()
)
```

Notice the printed warning message: `"WARNING: Column 'a' has values that aren't less than
7."`. The warning indicator (filled gray circle) visually confirms this threshold was reached and
the action should trigger.

Actions make your validation workflows more responsive and integrated with your data pipelines. For
example, you can generate console messages, Slack notifications, and more.

## Navigating the User Guide

As you continue exploring Pointblank's capabilities, you'll find the **User Guide** organized into
sections that will help you navigate the various features.

### Getting Started

The *Getting Started* section introduces you to Pointblank:

- [Introduction](index.qmd): Overview of Pointblank and core concepts (**this article**)
- [Installation](installation.qmd): How to install and set up Pointblank

### Validation Plan

The *Validation Plan* section covers everything you need to know about creating robust
validation plans:

- [Overview](../validation-plan/validation-overview.qmd): Survey of validation methods and their shared parameters
- [Validation Methods](../validation-plan/validation-methods.qmd): A closer look at the more common validation methods
- [Column Selection Patterns](../validation-plan/column-selection-patterns.qmd): Techniques for targeting specific columns
- [Preprocessing](../validation-plan/preprocessing.qmd): Transform data before validation
- [Segmentation](../validation-plan/segmentation.qmd): Apply validations to specific segments of your data
- [Thresholds](../validation-plan/thresholds.qmd): Set quality standards and trigger severity levels
- [Actions](../validation-plan/actions.qmd): Respond to threshold exceedances with notifications or custom functions
- [Briefs](../validation-plan/briefs.qmd): Add context to validation steps

### Advanced Validation

The *Advanced Validation* section explores more specialized validation techniques:

- [Expression-Based Validation](../advanced-validation/expressions.qmd): Use column expressions for advanced validation
- [Schema Validation](../advanced-validation/schema-validation.qmd): Enforce table structure and column types
- [Assertions](../advanced-validation/assertions.qmd): Raise exceptions to enforce data quality requirements
- [Draft Validation](../advanced-validation/draft-validation.qmd): Create validation plans from existing data

### Post Interrogation

After validating your data, the *Post Interrogation* section helps you analyze and respond to
results:

- [Validation Reports](../post-interrogation/validation-reports.qmd): Understand and customize the validation report table
- [Step Reports](../post-interrogation/step-reports.qmd): View detailed results for individual validation steps
- [Data Extracts](../post-interrogation/extracts.qmd): Extract and analyze failing data
- [Sundering Validated Data](../post-interrogation/sundering.qmd): Split data based on validation results

### Data Inspection

The *Data Inspection* section provides tools to explore and understand your data:

- [Previewing Data](../data-inspection/preview.qmd): View samples of your data
- [Column Summaries](../data-inspection/col-summary-tbl.qmd): Get statistical summaries of your data
- [Missing Values Reporting](../data-inspection/missing-vals-tbl.qmd): Identify and visualize missing data

By following this guide, you'll gain a comprehensive understanding of how to validate, monitor, and
maintain high-quality data with Pointblank.

::: {.callout-note}
A [PDF version of the User Guide](../user-guide.pdf) is also available for offline reading.
:::


### Installation

Pointblank can be installed using various package managers. The base installation gives you the core
validation functionality, with optional dependencies for working with different data sources.

## Basic Installation

You can install Pointblank using your preferred package manager:

::: {.panel-tabset}
## pip

```bash
pip install pointblank
```

## uv

```bash
uv pip install pointblank
```

## conda

```bash
conda install -c conda-forge pointblank
```

## pixi

```bash
# add pointblank to project
pixi init name-of-project
cd name-of-project
pixi add pointblank
```
:::

## DataFrame Libraries

Pointblank requires a DataFrame library but doesn't include one by default, giving you the
flexibility to choose either [Pandas](https://pandas.pydata.org) or [Polars](https://pola.rs):

::: {.panel-tabset}
## Polars

```bash
# Using pip
pip install pointblank[pl]

# Or manually
pip install polars>=1.24.0
```

## Pandas

```bash
# Using pip
pip install pointblank[pd]

# Or manually
pip install pandas>=2.2.3
```
:::

Pointblank works seamlessly with both libraries, and you can choose the one that best fits your
workflow and performance requirements.

## Optional Dependencies

### Ibis Backends

To work with various database systems through [Ibis](https://ibis-project.org), you can install
additional backends:

::: {.panel-tabset}
## pip

```bash
pip install pointblank[sqlite]      # SQLite
pip install pointblank[duckdb]      # DuckDB
pip install pointblank[postgres]    # PostgreSQL
pip install pointblank[mysql]       # MySQL
pip install pointblank[mssql]       # Microsoft SQL Server
pip install pointblank[bigquery]    # BigQuery
pip install pointblank[pyspark]     # Apache Spark
pip install pointblank[databricks]  # Databricks
pip install pointblank[snowflake]   # Snowflake

# Example of installing multiple backends
pip install pointblank[duckdb,postgres,sqlite]
```

## uv

```bash
uv pip install pointblank[sqlite]      # SQLite
uv pip install pointblank[duckdb]      # DuckDB
uv pip install pointblank[postgres]    # PostgreSQL
uv pip install pointblank[mysql]       # MySQL
uv pip install pointblank[mssql]       # Microsoft SQL Server
uv pip install pointblank[bigquery]    # BigQuery
uv pip install pointblank[pyspark]     # Apache Spark
uv pip install pointblank[databricks]  # Databricks
uv pip install pointblank[snowflake]   # Snowflake

# Example of installing multiple backends
uv pip install pointblank[duckdb,postgres,sqlite]
```

## conda

```bash
conda install -c conda-forge pointblank-sqlite      # SQLite
conda install -c conda-forge pointblank-duckdb      # DuckDB
conda install -c conda-forge pointblank-postgres    # PostgreSQL
conda install -c conda-forge pointblank-mysql       # MySQL
conda install -c conda-forge pointblank-mssql       # Microsoft SQL Server
conda install -c conda-forge pointblank-bigquery    # BigQuery
conda install -c conda-forge pointblank-pyspark     # Apache Spark
conda install -c conda-forge pointblank-databricks  # Databricks
conda install -c conda-forge pointblank-snowflake   # Snowflake

# Example of installing multiple backends
conda install -c conda-forge pointblank-duckdb pointblank-postgres pointblank-sqlite
```

## pixi

```bash
pixi add pointblank-sqlite      # SQLite
pixi add pointblank-duckdb      # DuckDB
pixi add pointblank-postgres    # PostgreSQL
pixi add pointblank-mysql       # MySQL
pixi add pointblank-mssql       # Microsoft SQL Server
pixi add pointblank-bigquery    # BigQuery
pixi add pointblank-pyspark     # Apache Spark
pixi add pointblank-databricks  # Databricks
pixi add pointblank-snowflake   # Snowflake

# Example of installing multiple backends
pixi add pointblank-duckdb pointblank-postgres pointblank-sqlite
```
:::

::: {.callout-note}
Even when using exclusively Ibis backends, you still need either Pandas or Polars installed since
Pointblank's reporting functionality (powered by
[Great Tables](https://posit-dev.github.io/great-tables)) requires a DataFrame library for rendering
tabular reporting results.
:::

### AI-Assisted Validation (Experimental)

Pointblank includes experimental support for AI-assisted validation plan generation:

```bash
pip install pointblank[generate]
```

This installs the necessary dependencies for working with LLM providers to help generate validation
plans. See the [Draft Validation](../advanced-validation/draft-validation.qmd) article for how to create validation plans
from existing data.

### Development Version

If you want the latest development version with the newest features, you can install directly from
GitHub:

```bash
pip install git+https://github.com/posit-dev/pointblank.git
```

## Verifying Your Installation

You can verify your installation by importing Pointblank and checking the version:

```python
import pointblank as pb
print(pb.__version__)
```

## System Requirements

- Python 3.10 or higher
- a supported DataFrame library (Pandas or Polars)
- optional: Ibis (for database connectivity)

## Next Steps

Now that you've installed Pointblank, you're ready to start validating your data. If you haven't
read the [Introduction](index.qmd) yet, consider starting there to learn the basic concepts.

If you encounter any installation issues, please
[open an issue on GitHub](https://github.com/posit-dev/pointblank/issues/new) with details about
your system and the specific error messages you're seeing. The maintainers actively monitor these
issues and can help troubleshoot problems.

For a quick test of your installation, try running a simple validation:

```python
import pointblank as pb

# Load a small dataset
data = pb.load_dataset("small_table")

# Create a simple validation
validation = (
    pb.Validate(data=data)
    .col_exists(columns=["a", "b", "c"])
    .interrogate()
)

# Display the validation results
validation
```

## Command Line Interface

Once installed, Pointblank also provides a powerful command-line interface for quick data validation tasks:

```bash
# Test the CLI with a built-in dataset
pb validate small_table --check rows-distinct

# Check if a column exists
pb validate small_table --check col-exists --column a

# Validate data ranges
pb validate small_table --check col-vals-lt --column a --value 10
```

The CLI is useful for:

- quick data quality checks in CI/CD pipelines
- exploratory data analysis from the terminal
- integration with shell scripts and automation workflows

::: {.callout-tip}
## See the CLI in Action
Watch our [interactive CLI demonstrations](../../examples/05-advanced-topics/cli-interactive.qmd) to see these commands executing in real-time with actual output formatting.
:::

Learn more about the CLI capabilities in the [Command Line Interface](../the-pointblank-cli/index.qmd) guide.


### Overview

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

This article provides a quick overview of the data validation features in Pointblank. It introduces
the key concepts and shows examples of the main functionality, giving you a foundation for using the
library effectively.

Later articles in the **User Guide** will expand on each section covered here, providing more
explanations and examples.

## Validation Methods

Pointblank's core functionality revolves around validation steps, which are individual checks that
verify different aspects of your data. These steps are created by calling validation methods from
the `Validate` class. When combined they create a comprehensive validation plan for your data.

Here's an example of a validation that incorporates three different validation methods:

```{python}
import pointblank as pb
import polars as pl

(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Three different validation methods."
    )
    .col_vals_gt(columns="a", value=0)
    .rows_distinct()
    .col_exists(columns="date")
    .interrogate()
)
```

This example showcases how you can combine different types of validations in a single validation
plan:

- a column value validation with `Validate.col_vals_gt()`
- a row-based validation with `Validate.rows_distinct()`
- a table structure validation with `Validate.col_exists()`

Most validation methods share common parameters that enhance their flexibility and power. These
shared parameters (overviewed in the next few sections) create a consistent interface across all
validation steps while allowing you to customize validation behavior for specific needs.

## Column Selection Patterns

You can apply the same validation logic to multiple columns at once through use of column selection
patterns (used in the `columns=` parameter). This reduces repetitive code and makes your validation
plans more maintainable:

```{python}
import narwhals.selectors as nws

# Map validations across multiple columns
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Applying column mapping in `columns`."
    )

    # Apply validation rules to multiple columns ---
    .col_vals_not_null(
        columns=["a", "b", "c"]
    )

    # Apply to numeric columns only with a Narwhals selector ---
    .col_vals_gt(
        columns=nws.numeric(),
        value=0
    )
    .interrogate()
)
```

This technique is particularly valuable when working with wide datasets containing many
similarly-structured columns or when applying standard quality checks across an entire table. It
also ensures consistency in how validation rules are applied across related data columns.

## Preprocessing

Preprocessing (with the `pre=` parameter) allows you to transform or modify your data before
applying validation checks, enabling you to validate derived or modified data without altering the
original dataset:

```{python}
import polars as pl

# Define preprocessing functions for `pre=` parameters
def double_column_a(df):
    return df.with_columns(pl.col("a") * 2)

def square_column_c(df):
    return df.with_columns(pl.col("c").pow(2))

(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Preprocessing validation steps via `pre=`."
    )
    .col_vals_gt(
        columns="a", value=5,

        # Apply transformation before validation ---
        pre=double_column_a  # Double values before checking
    )
    .col_vals_lt(
        columns="c", value=100,

        # Apply more complex transformation ---
        pre=square_column_c  # Square values before checking
    )
    .interrogate()
)
```

Preprocessing enables validation of transformed data without modifying your original dataset, making
it ideal for checking derived metrics, or validating normalized values. This approach keeps your
validation code clean while allowing for sophisticated data quality checks on calculated results.

## Segmentation

Segmentation (through the `segments=` parameter) allows you to validate data across different
groups, enabling you to identify segment-specific quality issues that might be hidden in aggregate
analyses:

```{python}
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Segmenting validation steps via `segments=`."
    )
    .col_vals_gt(
        columns="c", value=3,

        # Split into steps by categorical values in column 'f' ---
        segments="f"
    )
    .interrogate()
)
```

Segmentation is powerful for detecting patterns of quality issues that may exist only in specific
data subsets, such as certain time periods, categories, or geographical regions. It helps ensure
that all significant segments of your data meet quality standards, not just the data as a whole.

## Thresholds

Thresholds (set through the `thresholds=` parameter) let you set acceptable levels of failure before
triggering warnings, errors, or critical notifications for individual validation steps:

```{python}
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Using thresholds."
    )

    # Add validation steps with different thresholds ---
    .col_vals_gt(
        columns="a", value=1,
        thresholds=pb.Thresholds(warning=0.1, error=0.2, critical=0.3)
    )

    # Add another step with stricter thresholds ---
    .col_vals_lt(
        columns="c", value=10,
        thresholds=pb.Thresholds(warning=0.05, error=0.1)
    )
    .interrogate()
)
```

Thresholds provide a nuanced way to monitor data quality, allowing you to set different severity
levels based on the importance of each validation and your organization's tolerance for specific
types of data issues.

## Actions

Actions (which can be configured in the `actions=` parameter) allow you to define specific responses
when validation thresholds are crossed. You can use simple string messages or custom functions for
more complex behavior:

```{python}
# Example 1: Action with a string message ---

(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Using actions with a string message."
    )
    .col_vals_gt(
        columns="c", value=2,
        thresholds=pb.Thresholds(warning=0.1, error=0.2),

        # Add a print-to-console action for the 'warning' threshold ---
        actions=pb.Actions(
            warning="WARNING: Values below `{value}` detected in column 'c'."
        )
    )
    .interrogate()
)
```

```{python}
# Example 2: Action with a callable function ---

def custom_action():
    from datetime import datetime
    print(f"Data quality issue found ({datetime.now()}).")

(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Using actions with a callable function."
    )
    .col_vals_gt(
        columns="a", value=5,
        thresholds=pb.Thresholds(warning=0.1, error=0.2),

        # Apply the function to the 'error' threshold ---
        actions=pb.Actions(error=custom_action)
    )
    .interrogate()
)
```

With custom action functions, you can implement sophisticated responses like sending notifications
or logging to external systems.

## Briefs

Briefs (which can be set through the `brief=` parameter) allow you to customize descriptions
associated with validation steps, making validation results more understandable to stakeholders.
Briefs can be either automatically generated by setting `brief=True` or defined as custom messages
for more specific explanations:

```{python}
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Using `brief=` for displaying brief messages."
    )
    .col_vals_gt(
        columns="a", value=0,

        # Use `True` for automatic generation of briefs ---
        brief=True
    )
    .col_exists(
        columns=["date", "date_time"],

        # Add a custom brief for this validation step ---
        brief="Verify required date columns exist for time-series analysis"
    )
    .interrogate()
)
```

Briefs make validation results more meaningful by providing context about why each check matters.
They're particularly valuable in shared reports where stakeholders from various disciplines need to
understand validation results in domain-specific terms.

## Getting More Information

Each validation step can be further customized and has additional options. See these pages for more
information:

- [Validation Methods](validation-methods.qmd): A closer look at the more common validation methods
- [Column Selection Patterns](column-selection-patterns.qmd): Techniques for targeting specific columns
- [Preprocessing](preprocessing.qmd): Transform data before validation
- [Segmentation](segmentation.qmd): Apply validations to specific segments of your data
- [Thresholds](thresholds.qmd): Set quality standards and trigger severity levels
- [Actions](actions.qmd): Respond to threshold exceedances with notifications or custom functions
- [Briefs](briefs.qmd): Add context to validation steps

## Conclusion

Validation steps are the building blocks of data validation in Pointblank. By combining steps from
different categories and leveraging common features like thresholds, actions, and preprocessing, you
can create comprehensive data quality checks tailored to your specific needs.

The next sections of this guide will dive deeper into each of these topics, providing detailed
explanations and examples.


### Validation Methods

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

Pointblank provides a comprehensive suite of validation methods to verify different aspects of your
data. Each method creates a validation step that becomes part of your validation plan.

These validation methods cover everything from checking column values against thresholds to
validating the table structure and detecting duplicates. Combined into validation steps, they form
the foundation of your data quality workflow.

Pointblank provides [over 40 validation methods](https://posit-dev.github.io/pointblank/reference/#validation-steps)
to handle diverse data quality requirements. These are grouped into five main categories:

1. Column Value Validations
2. Row-based Validations
3. Table Structure Validations
4. AI-Powered Validations
5. Aggregate Validations

Within each of these categories, we'll walk through several examples showing how each validation
method creates steps in your validation plan.

And we'll use the `small_table` dataset for all of our examples. Here's a preview of it:

```{python}
# | echo: false
pb.preview(pb.load_dataset(dataset="small_table"), n_head=20, n_tail=20)
```

## Validation Methods to Validation Steps

In Pointblank, validation *methods* become validation *steps* when you add them to a validation
plan. Each method creates a distinct step that performs a specific check on your data.

Here's a simple example showing how three validation methods create three validation steps:

```{python}
import pointblank as pb

(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))

    # Step 1: Check that values in column `a` are greater than 2 ---
    .col_vals_gt(columns="a", value=2, brief="Values in 'a' must exceed 2.")

    # Step 2: Check that column 'date' exists in the table ---
    .col_exists(columns="date", brief="Column 'date' must exist.")

    # Step 3: Check that the table has exactly 13 rows ---
    .row_count_match(count=13, brief="Table should have exactly 13 rows.")
    .interrogate()
)
```

Each validation method produces one step in the validation report above. When combined, these steps
form a complete validation plan that systematically checks different aspects of your data quality.

## Common Arguments

Most validation methods in Pointblank share a set of common arguments that provide consistency and
flexibility across different validation types:

- `columns=`: specifies which column(s) to validate (used in column-based validations)
- `pre=`: allows data transformation before validation
- `segments=`: enables validation across different data subsets
- `thresholds=`: sets acceptable failure thresholds
- `actions=`: defines actions to take when validations fail
- `brief=`: provides a description of what the validation is checking
- `active=`: determines if the validation step should be executed (default is `True`)
- `na_pass=`: controls how missing values are handled (only for column value validation methods)

For column validation methods, the `na_pass=` parameter determines whether missing values
(Null/None/NA) should pass validation (this parameter is covered in a later section).

These arguments follow a consistent pattern across validation methods, so you don't need to memorize
different parameter sets for each function. This systematic approach makes Pointblank more intuitive
to work with as you build increasingly complex validation plans.

We'll cover most of these common arguments in their own dedicated sections later in the
**User Guide**, as some of them represent a deeper topic worthy of focused attention.

## 1. Column Value Validations

These methods check individual values within columns against specific criteria:

- **Comparison checks** ([`Validate.col_vals_gt()`](`Validate.col_vals_gt`), [`Validate.col_vals_lt()`](`Validate.col_vals_lt`), etc.) for comparing
values to thresholds or other columns

- **Range checks** ([`Validate.col_vals_between()`](`Validate.col_vals_between`), [`Validate.col_vals_outside()`](`Validate.col_vals_outside`)) for verifying
that values fall within or outside specific ranges

- **Set membership checks** ([`Validate.col_vals_in_set()`](`Validate.col_vals_in_set`), [`Validate.col_vals_not_in_set()`](`Validate.col_vals_not_in_set`)) for
validating values against predefined sets

- **Referential integrity checks** ([`Validate.col_vals_in_table()`](`Validate.col_vals_in_table`)) for verifying that
column values (or composite keys) exist in a reference table

- **Null value checks** ([`Validate.col_vals_null()`](`Validate.col_vals_null`), [`Validate.col_vals_not_null()`](`Validate.col_vals_not_null`)) for testing
presence or absence of null values

- **Pattern matching checks** ([`Validate.col_vals_regex()`](`Validate.col_vals_regex`), [`Validate.col_vals_within_spec()`](`Validate.col_vals_within_spec`))
for validating text patterns with regular expressions or against standard specifications

- **Trending value checks** ([`Validate.col_vals_increasing()`](`Validate.col_vals_increasing`), [`Validate.col_vals_decreasing()`](`Validate.col_vals_decreasing`))
for verifying that values increase or decrease as you move down the rows

- **Custom expression checks** ([`Validate.col_vals_expr()`](`Validate.col_vals_expr`)) for complex validations using custom
expressions

Now let's look at some key examples from select categories of column value validations.

### Comparison Checks

Let's start with a simple example of how [`Validate.col_vals_gt()`](`Validate.col_vals_gt`) might be used to check if the
values in a column are greater than a specified value.

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_gt(columns="a", value=5)
    .interrogate()
)
```

If you're checking data in a column that contains Null/`None`/`NA` values and you'd like to
disregard those values (i.e., let them pass validation), you can use `na_pass=True`. The following
example checks values in column `c` of `small_table`, which contains two `None` values:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_le(columns="c", value=10, na_pass=True)
    .interrogate()
)
```

In the above validation table, we see that all test units passed. If we didn't use `na_pass=True`
there would be 2 failing test units, one for each `None` value in the `c` column.

It's possible to check against column values against values in an adjacent column. To do this,
supply the `value=` argument with the column name within the `col()` helper function. Here's an
example of that:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_lt(columns="a", value=pb.col("c"))
    .interrogate()
)
```

This validation checks that values in column `a` are less than values in column `c`.

### Checking of Missing Values

A very common thing to validate is that there are no Null/NA/missing values in a column. The
[`Validate.col_vals_not_null()`](`Validate.col_vals_not_null`) method checks for the presence of missing values:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_not_null(columns="a")
    .interrogate()
)
```

Column `a` has no missing values and the above validation proves this.

### Checking Percentage of Missing Values

While [`Validate.col_vals_not_null()`](`Validate.col_vals_not_null`) ensures there are no missing values at all, sometimes you
need to validate that missing values match a specific percentage. The [`Validate.col_pct_null()`](`Validate.col_pct_null`)
method checks whether the percentage of missing values in a column matches an expected value:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_pct_null(columns="c", p=0.15, tol=0.05)  # Expect ~15% missing values (±5%)
    .interrogate()
)
```

This validation checks that approximately 15% of values in column `c` are missing, allowing a
tolerance of ±5% (so the acceptable range is 10-20%). The `tol=` parameter can accept various
formats including absolute counts or percentage ranges:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_pct_null(columns="c", p=0.15, tol=(0.05, 0.10))  # Asymmetric tolerance: -5%/+10%
    .interrogate()
)
```

### Checking Strings with Regexes

A regular expression (regex) validation via the [`Validate.col_vals_regex()`](`Validate.col_vals_regex`) validation method
checks if values in a column match a specified pattern. Here's an example with two validation steps,
each checking text values in a column:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_regex(columns="b", pattern=r"^\d-[a-z]{3}-\d{3}$")
    .col_vals_regex(columns="f", pattern=r"high|low|mid")
    .interrogate()
)
```

### Checking Strings Against Specifications

The [`Validate.col_vals_within_spec()`](`Validate.col_vals_within_spec`) method validates column values against common data
specifications like email addresses, URLs, postal codes, credit card numbers, ISBNs, VINs, and
IBANs. This is particularly useful when you need to validate that text data conforms to standard
formats:

```{python}
import polars as pl

# Create a sample table with various data types
sample_data = pl.DataFrame({
    "isbn": ["978-0-306-40615-7", "0-306-40615-2", "invalid"],
    "email": ["test@example.com", "user@domain.co.uk", "not-an-email"],
    "zip": ["12345", "90210", "invalid"]
})

(
    pb.Validate(data=sample_data)
    .col_vals_within_spec(columns="isbn", spec="isbn")
    .col_vals_within_spec(columns="email", spec="email")
    .col_vals_within_spec(columns="zip", spec="postal_code[US]")
    .interrogate()
)
```

### Checking for Trending Values

The [`Validate.col_vals_increasing()`](`Validate.col_vals_increasing`) and [`Validate.col_vals_decreasing()`](`Validate.col_vals_decreasing`) validation methods
check whether column values are increasing or decreasing as you move down the rows. These are useful
for validating time series data, sequential identifiers, or any data where you expect monotonic
trends:

```{python}
import polars as pl

# Create a sample table with increasing and decreasing values
trend_data = pl.DataFrame({
    "id": [1, 2, 3, 4, 5],
    "temperature": [20, 22, 25, 28, 30],
    "countdown": [100, 80, 60, 40, 20]
})

(
    pb.Validate(data=trend_data)
    .col_vals_increasing(columns="id")
    .col_vals_increasing(columns="temperature")
    .col_vals_decreasing(columns="countdown")
    .interrogate()
)
```

The `allow_stationary=` parameter lets you control whether consecutive identical values should pass
validation. By default, stationary values (e.g., `[1, 2, 2, 3]`) will fail the increasing check,
but setting `allow_stationary=True` will allow them to pass.

The rows are compared in the table's own order. Database tables have no inherent row order, so for
them (or to check a different order) use `order_by=` to name the column(s) that put the rows in
order, such as `order_by="date"`. Missing values fail unless `na_pass=True`, and they don't break
the sequence: with `[1, 2, None, 5]`, the `5` is compared with the `2`.

### Handling Missing Values with `na_pass=`

When validating columns containing Null/None/NA values, you can control how these missing values are
treated with the `na_pass=` parameter:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_le(columns="c", value=10, na_pass=True)
    .interrogate()
)
```

In the above example, column `c` contains two `None` values, but all test units pass because we set
`na_pass=True`. Without this setting, those two values would fail the validation.

In summary, `na_pass=` works like this:

- `na_pass=True`: missing values pass validation regardless of the condition being tested
- `na_pass=False` (the default): missing values fail validation

### Structured Missingness with `missing=`

`na_pass=` treats missingness as binary, but real-world data often encodes *why* a value is absent
using sentinel codes (e.g., `-99` for "not asked", `-98` for "refused"). The
[`MissingSpec`](`pointblank.MissingSpec`) class captures these codes and their reasons, and most
validation methods accept a `missing=` argument that uses it.

When you pass `missing=` to a `col_vals_*()` method, declared sentinel values (and, by default,
`Null` values) are *excluded* from the check, so only the "real" values are validated:

```{python}
import polars as pl

tbl = pl.DataFrame({"age": [34, -98, 41, -99, 29, 200, 55, None]})

age_missing = pb.MissingSpec(reasons={-99: "not_asked", -98: "refused"})

validation = (
    pb.Validate(data=tbl)
    .col_vals_between(columns="age", left=0, right=120, missing=age_missing)
    .interrogate()
)

validation
```

Only the real value `200` is out of range; the sentinel codes and the `Null` are excluded and pass.
In the report, such steps are marked with a compact `MISSING-AWARE` badge, and a one-line summary of
the codes appears in the step's notes.

Pointblank also provides dedicated missingness validation methods that use a `MissingSpec`:

- [`Validate.col_pct_missing()`](`Validate.col_pct_missing`): assert the percentage of missing
  values stays within a limit, optionally filtered by a specific `reason=` or `category=`.
- [`Validate.col_missing_coded()`](`Validate.col_missing_coded`): assert every absence is expressed
  as a documented code (no uncoded raw `Null` values).
- [`Validate.col_missing_only_coded()`](`Validate.col_missing_only_coded`): assert a column contains
  only documented codes and legitimate values (catching undocumented codes like a stray `-95`),
  paired with an `allowed=` set or a `min_val`/`max_val` range.
- [`Validate.col_missing_consistent()`](`Validate.col_missing_consistent`): assert related columns
  share a consistent missingness pattern for a given reason (e.g., a survey skip pattern).

```{python}
income_missing = pb.MissingSpec(reasons={-99: "not_asked", -1: "below_threshold"})

survey = pl.DataFrame(
    {
        "income_source": [1, -99, 2, -99],
        "income_amount": [50000, -99, 42000, 38000],
    }
)

validation = (
    pb.Validate(data=survey)
    # No more than 30% of income values may be "not_asked"
    .col_pct_missing(columns="income_amount", missing=income_missing, reason="not_asked", max_pct=0.30)
    # If income wasn't asked, both related columns should be coded together
    .col_missing_consistent(
        columns=["income_source", "income_amount"], missing=income_missing, when_reason="not_asked"
    )
    .interrogate()
)

validation
```

`MissingSpec` also offers pre-built factories for common standards (e.g.,
`pb.MissingSpec.from_cdisc_null_flavors()`, `pb.MissingSpec.from_sas()`,
`pb.MissingSpec.from_spss()`), and importing metadata from SPSS/Stata/SAS files can auto-generate
specs via [`MetadataImport.missing_specs()`](`pointblank.MetadataImport`). For a fuller treatment of
structured-missingness *reporting*, see the *Missing Values Reporting* article.

### Referential Integrity with `col_vals_in_table()`

When your data spans multiple tables, you often need to verify that foreign key values in one table
actually exist in another. The [`col_vals_in_table()`](`Validate.col_vals_in_table`) method handles
this directly, checking each row's value (or composite key) against a reference table.

```{python}
import polars as pl

customers = pl.DataFrame({"id": [1, 2, 3, 4, 5]})

orders = pl.DataFrame({
    "order_id": [101, 102, 103, 104],
    "customer_id": [1, 2, 3, 99],
})

(
    pb.Validate(data=orders)
    .col_vals_in_table(
        columns="customer_id",
        ref_table=customers,
        ref_column="id",
    )
    .interrogate()
)
```

The last row fails because `customer_id=99` does not exist in the customers table. Each row is a
test unit, so failing rows are extractable via `get_data_extracts()` just like any other
column-level validation.

Composite foreign keys are also supported. Supply lists of column names to check multi-column
references:

```{python}
catalog = pl.DataFrame({
    "region": ["US", "US", "EU"],
    "sku": ["A1", "B2", "A1"],
})

line_items = pl.DataFrame({
    "region": ["US", "EU", "US"],
    "sku": ["A1", "A1", "C3"],
})

(
    pb.Validate(data=line_items)
    .col_vals_in_table(
        columns=["region", "sku"],
        ref_table=catalog,
        ref_column=["region", "sku"],
    )
    .interrogate()
)
```

The third row fails because the combination `("US", "C3")` does not appear in the catalog.

Cross-backend matching is fully supported: the reference table can come from a different backend
(e.g., a Polars DataFrame checked against a DuckDB table, or a Pandas DataFrame checked against a
Polars table). The lighter table is automatically materialized to match the heavier table's backend
before comparison, so you can validate referential integrity across heterogeneous data sources
without any manual conversion.

The `ref_table=` parameter also accepts a callable, which is useful when the reference data should
be loaded fresh at interrogation time:

```{python}
def load_valid_skus():
    return pl.DataFrame({"sku": ["A1", "B2", "C3"]})

items = pl.DataFrame({"sku": ["A1", "B2", "D4"]})

(
    pb.Validate(data=items)
    .col_vals_in_table(
        columns="sku",
        ref_table=load_valid_skus,
        ref_column="sku",
    )
    .interrogate()
)
```

By default, null key values are treated as failing (not found in the reference). Set `na_pass=True`
if null foreign keys are acceptable in your schema (e.g., optional relationships).

## 2. Row-based Validations

Row-based validations focus on examining properties that span across entire rows rather than
individual columns. These are essential for detecting issues that can't be found by looking at
columns in isolation:

- [`Validate.rows_distinct()`](`Validate.rows_distinct`): ensures no duplicate rows exist in the table
- [`Validate.rows_complete()`](`Validate.rows_complete`): verifies that no rows contain any missing values

These row-level validations are particularly valuable for ensuring data integrity and completeness
at the record level, which is crucial for many analytical and operational data applications.

### Checking Row Distinctness

Here's an example where we check for duplicate rows with [`Validate.rows_distinct()`](`Validate.rows_distinct`):

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .rows_distinct()
    .interrogate()
)
```

We can also adapt the [`Validate.rows_distinct()`](`Validate.rows_distinct`) check to use a single column or a subset of
columns. To do that, we need to use the `columns_subset=` parameter. Here's an example of that:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .rows_distinct(columns_subset="b")
    .interrogate()
)
```

### Checking Row Completeness

Another important validation is checking for complete rows: rows that have no missing values across
all columns or a specified subset of columns. The [`Validate.rows_complete()`](`Validate.rows_complete`) validation method
performs this check.

Here's an example checking if all rows in the table are complete (have no missing values in any
column):

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .rows_complete()
    .interrogate()
)
```

As the report indicates, there are some incomplete rows in the table.

## 3. Table Structure Validations

Table structure validations ensure that the overall architecture of your data meets expectations.
These structural checks form a foundation for more detailed data quality assessments:

- [`Validate.col_exists()`](`Validate.col_exists`): verifies a column exists in the table
- [`Validate.col_schema_match()`](`Validate.col_schema_match`): ensures table matches a defined schema
- [`Validate.col_count_match()`](`Validate.col_count_match`): confirms the table has the expected number of columns
- [`Validate.row_count_match()`](`Validate.row_count_match`): verifies the table has the expected number of rows
- [`Validate.tbl_match()`](`Validate.tbl_match`): validates that the target table matches a comparison table
- [`Validate.data_freshness()`](`Validate.data_freshness`): checks that data is recent and not stale

These structural validations provide essential checks on the fundamental organization of your data
tables, ensuring they have the expected dimensions and components needed for reliable data analysis.

### Checking Column Presence

If you need to check for the presence of individual columns, the `Validate.col_exists()` validation
method is useful. In this example, we check whether the `date` column is present in the table:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_exists(columns="date")
    .interrogate()
)
```

That column is present, so the single test unit of this validation step is a passing one.

### Checking the Table Schema

For deeper checks of table structure, a schema validation can be performed with the
[`Validate.col_schema_match()`](`Validate.col_schema_match`) validation method, where the goal is to check whether the structure
of a table matches an expected schema. To define an expected table schema, we need to use the
`Schema` class. Here is a simple example that (1) prepares a schema consisting of column names, (2)
uses that `schema` object in a [`Validate.col_schema_match()`](`Validate.col_schema_match`) validation step:

```{python}
schema = pb.Schema(columns=["date_time", "date", "a", "b", "c", "d", "e", "f"])

(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_schema_match(schema=schema)
    .interrogate()
)
```

The [`Validate.col_schema_match()`](`Validate.col_schema_match`) validation step will only have a single test unit (signifying
pass or fail). We can see in the above validation report that the column schema validation passed.

More often, a schema will be defined using column names and column types. We can do that by using a
list of tuples in the `columns=` parameter of `Schema`. Here's an example of that approach in
action:

```{python}
schema = pb.Schema(
    columns=[
        ("date_time", "Datetime(time_unit='us', time_zone=None)"),
        ("date", "Date"),
        ("a", "Int64"),
        ("b", "String"),
        ("c", "Int64"),
        ("d", "Float64"),
        ("e", "Boolean"),
        ("f", "String"),
    ]
)

(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_schema_match(schema=schema)
    .interrogate()
)
```

The [`Validate.col_schema_match()`](`Validate.col_schema_match`) validation method has several boolean parameters for making the
checks less stringent:

- `complete=`: requires exact column matching (all expected columns must exist, no extra columns
allowed)
- `in_order=`: enforces that columns appear in the same order as defined in the schema
- `case_sensitive_colnames=`: column names must match with exact letter case
- `case_sensitive_dtypes=`: data type strings must match with exact letter case

These parameters all default to `True`, providing strict schema validation. Setting any to `False`
relaxes the validation requirements, making the checks more flexible when exact matching isn't
necessary or practical for your use case.

### Comparing Tables with `tbl_match()`

The [`Validate.tbl_match()`](`Validate.tbl_match`) validation method provides a comprehensive way to verify that two
tables are identical. It performs a progressive series of checks, from least to most stringent:

1. Column count match
2. Row count match
3. Schema match (loose - case-insensitive, any order)
4. Schema match (order - columns in correct order)
5. Schema match (exact - case-sensitive, correct order)
6. Data match (cell-by-cell comparison)

This progressive approach helps identify exactly where tables differ. Here's an example comparing
the `small_table` dataset with itself:

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .tbl_match(tbl_compare=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .interrogate()
)
```

This validation method is especially useful for:

- Verifying that data transformations preserve expected properties
- Comparing production data against a golden dataset
- Ensuring data consistency across different environments
- Validating that imported data matches source data

### Checking Counts of Row and Columns

Row and column count validations check the number of rows and columns in a table.

Using [`Validate.row_count_match()`](`Validate.row_count_match`) checks whether the number of rows in a table matches a
specified count.

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .row_count_match(count=13)
    .interrogate()
)
```

The [`Validate.col_count_match()`](`Validate.col_count_match`) validation method checks if the number of columns in a table
matches a specified count.

```{python}
(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_count_match(count=8)
    .interrogate()
)
```

Expectations on column and row counts can be useful in certain situations and they align nicely with
schema checks.

### Validating Data Freshness

Late or missing data is one of the most common (and costly) data quality issues in production
systems. When data pipelines fail silently or experience delays, downstream analytics and ML models
can produce stale or misleading results. The [`Validate.data_freshness()`](`Validate.data_freshness`) validation method helps
catch these issues early by verifying that your data contains recent records.

Data freshness validation works by checking a datetime column against a maximum allowed age. If the
most recent timestamp in that column is older than the specified threshold, the validation fails.
This simple check can prevent major downstream problems caused by stale data.

Here's an example that validates data is no older than 2 days:

```{python}
import polars as pl
from datetime import datetime, timedelta

# Simulate a data feed that should be updated daily
recent_data = pl.DataFrame({
    "event": ["login", "purchase", "logout", "signup"],
    "event_time": [
        datetime.now() - timedelta(hours=1),
        datetime.now() - timedelta(hours=6),
        datetime.now() - timedelta(hours=12),
        datetime.now() - timedelta(hours=18),
    ],
    "user_id": [101, 102, 103, 104]
})

(
    pb.Validate(data=recent_data)
    .data_freshness(column="event_time", max_age="2d")
    .interrogate()
)
```

The `max_age=` parameter accepts a flexible string format: `"30m"` for 30 minutes, `"6h"` for 6
hours, `"2d"` for 2 days, or `"1w"` for 1 week. You can also combine units: `"1d 12h"` for 1.5 days.

When validation succeeds, the report includes details about the data's age in the footer. When it
fails, you'll see exactly how old the most recent data is and what threshold was exceeded. This
context helps quickly diagnose whether you're dealing with a minor delay or a major pipeline
failure.

Data that is dated later than the comparison time also fails, since timestamps from the future
usually mean a clock is set wrong or a timezone was mixed up when the data was loaded. If the system
that writes the data has a clock that may run slightly ahead, allow a small margin with something
like `allow_future="1m"`. To accept future-dated data of any amount, use `allow_future=True`.

Data freshness validation is particularly valuable for:

- monitoring ETL pipelines to catch failures before they cascade to reports and dashboards
- validating data feeds to ensure third-party data sources are delivering as expected
- including freshness checks in automated data quality tests as part of continuous integration
- building alerting systems that trigger notifications when critical data becomes stale

You might wonder why not just use [`Validate.col_vals_gt()`](`Validate.col_vals_gt`) with a datetime threshold. While that
approach works, [`Validate.data_freshness()`](`Validate.data_freshness`) offers several advantages: the method name clearly
communicates your intent, the `max_age=` string format (e.g., `"2d"`) is more readable than datetime
arithmetic, it auto-generates meaningful validation briefs, the report footer shows helpful context
about actual data age and thresholds, and timezone mismatches between your data and comparison time
are handled gracefully with informative warnings.

::: {.callout-note}
When comparing timezone-aware and timezone-naive datetimes, Pointblank will include a warning in the
validation report. For consistent results, ensure your data and comparison times use compatible
timezone settings.
:::

## 4. AI-Powered Validations

AI-powered validations use Large Language Models (LLMs) to validate data based on natural language
criteria. This opens up new possibilities for complex validation rules that are difficult to express
with traditional programmatic methods.

### Validating with Natural Language Prompts

The [`Validate.prompt()`](`Validate.prompt`) validation method allows you to describe validation criteria in plain
language. The LLM interprets your prompt and evaluates each row, producing pass/fail results just
like other Pointblank validation methods.

This is particularly useful for:

- Semantic checks (e.g., "descriptions should mention a product name")
- Context-dependent validation (e.g., "prices should be reasonable for the product category")
- Subjective quality assessments (e.g., "comments should be professional and constructive")
- Complex rules that would require extensive regex patterns or custom functions

Here's a simple example that validates whether text descriptions contain specific information:

```{python}
#| eval: false
import polars as pl

# Create sample data with product descriptions
products = pl.DataFrame({
    "product": ["Widget A", "Gadget B", "Tool C"],
    "description": [
        "High-quality widget made in USA",
        "Innovative gadget with warranty",
        "Professional tool"
    ],
    "price": [29.99, 49.99, 19.99]
})

# Validate that descriptions mention quality or features
(
    pb.Validate(data=products)
    .prompt(
        prompt="Each description should mention either quality, features, or warranty",
        columns_subset=["description"],
        model="anthropic:claude-opus-4-6"
    )
    .interrogate()
)
```

The `columns_subset=` parameter lets you specify which columns to include in the validation,
improving performance and reducing API costs by only sending relevant data to the LLM.

**Note:** To use [`Validate.prompt()`](`Validate.prompt`), you need to have the appropriate API credentials configured
for your chosen LLM provider (Anthropic, OpenAI, Ollama, AWS Bedrock, or Azure OpenAI).

For **Azure OpenAI**, use `model="azure-openai:<deployment_id>"` where `<deployment_id>` is the
name you assigned when deploying the model in your Azure OpenAI resource. In addition to
`AZURE_OPENAI_API_KEY`, you must set `AZURE_OPENAI_ENDPOINT` (e.g.,
`https://<resource>.openai.azure.com`) and `OPENAI_API_VERSION` (e.g., `"2024-06-01"`) in your
environment.

## 5. Aggregate Validations

Aggregate validations operate on column-level statistics rather than individual row values. These
methods compute an aggregate value (such as sum, average, or standard deviation) from a column and
compare it against an expected value. Unlike row-level validations where each row is a test unit,
aggregate validations treat the entire column as a single test unit that either passes or fails.

Pointblank provides three families of aggregate validation methods:

- **Sum validations** ([`Validate.col_sum_eq()`](`Validate.col_sum_eq`), [`Validate.col_sum_gt()`](`Validate.col_sum_gt`), [`Validate.col_sum_lt()`](`Validate.col_sum_lt`),
  [`Validate.col_sum_ge()`](`Validate.col_sum_ge`), [`Validate.col_sum_le()`](`Validate.col_sum_le`)) for validating the sum of column values

- **Average validations** ([`Validate.col_avg_eq()`](`Validate.col_avg_eq`), [`Validate.col_avg_gt()`](`Validate.col_avg_gt`), [`Validate.col_avg_lt()`](`Validate.col_avg_lt`),
  [`Validate.col_avg_ge()`](`Validate.col_avg_ge`), [`Validate.col_avg_le()`](`Validate.col_avg_le`)) for validating the mean of column values

- **Standard deviation validations** ([`Validate.col_sd_eq()`](`Validate.col_sd_eq`), [`Validate.col_sd_gt()`](`Validate.col_sd_gt`),
  [`Validate.col_sd_lt()`](`Validate.col_sd_lt`), [`Validate.col_sd_ge()`](`Validate.col_sd_ge`), [`Validate.col_sd_le()`](`Validate.col_sd_le`)) for validating the
  standard deviation of column values

Each family supports the five comparison operators: equal to (`_eq`), greater than (`_gt`), less
than (`_lt`), greater than or equal to (`_ge`), and less than or equal to (`_le`).

### Validating Column Sums

Here's an example validating that the sum of column `a` equals 55:

```{python}
import polars as pl

agg_data = pl.DataFrame({
    "a": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    "b": [10, 20, 30, 40, 50, 60, 70, 80, 90, 100],
})

(
    pb.Validate(data=agg_data)
    .col_sum_eq(columns="a", value=55)
    .col_sum_gt(columns="b", value=500)
    .interrogate()
)
```

### Validating Column Averages

Average validations are useful for ensuring that typical values remain within expected bounds:

```{python}
(
    pb.Validate(data=agg_data)
    .col_avg_eq(columns="a", value=5.5)
    .col_avg_ge(columns="b", value=50)
    .interrogate()
)
```

### Validating Standard Deviations

Standard deviation validations help ensure data variability is within expected ranges:

```{python}
(
    pb.Validate(data=agg_data)
    .col_sd_gt(columns="a", value=2)
    .col_sd_lt(columns="b", value=35)
    .interrogate()
)
```

### Using Tolerance for Fuzzy Comparisons

Floating-point arithmetic can introduce small precision errors, making exact equality comparisons
unreliable. The `tol=` parameter allows for fuzzy comparisons by specifying an acceptable tolerance:

```{python}
(
    pb.Validate(data=agg_data)
    .col_avg_eq(columns="a", value=5.5, tol=0.01)  # Pass if average is within ±0.01 of 5.5
    .col_sum_eq(columns="b", value=550, tol=1)    # Pass if sum is within ±1 of 550
    .interrogate()
)
```

For equality comparisons, the tolerance creates a range `[value - tol, value + tol]` within which
the aggregate is considered valid.

### Comparing Against Reference Data

Aggregate validations shine when comparing current data against a baseline or reference dataset.
This is invaluable for detecting drift in data properties over time:

```{python}
# Current data
current_data = pl.DataFrame({"revenue": [100, 200, 150, 175, 125]})

# Historical baseline
baseline_data = pl.DataFrame({"revenue": [95, 205, 145, 180, 130]})

(
    pb.Validate(data=current_data, reference=baseline_data)
    .col_sum_eq(columns="revenue", tol=50)   # Compare sums with tolerance
    .col_avg_eq(columns="revenue", tol=5)    # Compare averages with tolerance
    .interrogate()
)
```

When `value=None` (the default) and reference data is set, aggregate methods automatically compare
against the same column in the reference data.

## 6. Custom Validations with `specially()`

While Pointblank provides over 40 built-in validation methods, there are times when you need to
implement custom validation logic that goes beyond these standard checks. The [`Validate.specially()`](`Validate.specially`)
method gives you complete flexibility to create bespoke validations for domain-specific business
rules, complex multi-column checks, or cross-dataset referential integrity constraints.

### Basic Custom Validations

The `specially()` method accepts a callable function that performs your custom validation logic. The
function should return boolean values indicating whether each test unit passes:

```{python}
import polars as pl

simple_tbl = pl.DataFrame({
    "a": [5, 7, 1, 3, 9, 4],
    "b": [6, 3, 0, 5, 8, 2]
})

# Custom validation: sum of two columns must be positive
def validate_sum_positive(data):
    return data.select(pl.col("a") + pl.col("b") > 0)

(
    pb.Validate(data=simple_tbl)
    .specially(
        expr=validate_sum_positive,
        brief="Sum of columns 'a' and 'b' must be positive"
    )
    .interrogate()
)
```

This validation passes because all rows have a positive sum for columns `a` and `b`. The
`specially()` method provides the flexibility to implement any validation logic you can express in
Python, making it a powerful tool for custom data quality checks.

### Cross-Dataset Referential Integrity

:::{.callout-tip}
For straightforward foreign key checks (single-column or composite-key lookups), use the built-in
[`col_vals_in_table()`](`Validate.col_vals_in_table`) method described earlier in this article. It
gives you row-level pass/fail results, data extracts, and cross-backend support out of the box.

The `specially()` approach shown below is still valuable for more advanced cross-dataset logic:
conditional existence rules, cardinality constraints, and composite keys with business-logic
predicates that go beyond simple key matching.
:::

One powerful use case for `specially()` is validating relationships between multiple datasets. This
is particularly valuable for conditional existence rules and cardinality relationships that span
multiple tables.

#### Foreign Key Validation

Verify that all keys in one dataset exist in another (this is equivalent to using
`col_vals_in_table()` but shown here for completeness):

```{python}
# Create related datasets: Orders and OrderDetails
orders = pl.DataFrame({
    "order_id": [1, 2, 3, 4, 5],
    "customer_id": ["A", "B", "A", "C", "B"],
    "status": ["completed", "pending", "completed", "cancelled", "completed"]
})

order_details = pl.DataFrame({
    "detail_id": [101, 102, 103, 104, 105, 106, 107, 108, 109],
    "order_id": [1, 1, 1, 2, 3, 3, 4, 5, 5],
    "product_id": ["P1", "P2", "P3", "P4", "P5", "P6", "P7", "P8", "P9"],
    "quantity": [2, 1, 3, 1, 2, 1, 1, 2, 1]
})

# Validate foreign key constraint
def check_foreign_key(df):
    """Check if all order_ids in order_details exist in orders table"""
    valid_order_ids = orders.select("order_id")
    # Semi join returns only rows with matching keys
    return df.join(valid_order_ids, on="order_id", how="semi").height == df.height

(
    pb.Validate(data=order_details, tbl_name="order_details")
    .specially(
        expr=check_foreign_key,
        brief="All order_ids must exist in orders table"
    )
    .interrogate()
)
```

This validation ensures referential integrity by confirming that every `order_id` in the
`order_details` table has a corresponding record in the `orders` table. The use of a semi-join makes
this check efficient, as it only verifies the existence of matching keys without returning full
joined data.

#### Conditional Existence Checks

Implement "if X then Y must exist" logic across datasets:

```{python}
def check_completed_orders_have_details(df):
    """Completed orders must have at least one detail record"""
    completed_orders = df.filter(pl.col("status") == "completed")
    order_ids_with_details = order_details.select("order_id").unique()

    # Check each completed order has matching details
    return completed_orders.join(
        order_ids_with_details,
        on="order_id",
        how="left"
    ).with_columns(
        pl.col("order_id").is_not_null().alias("has_details")
    ).select("has_details")

(
    pb.Validate(data=orders, tbl_name="orders")
    .specially(
        expr=check_completed_orders_have_details,
        brief="Completed orders must have detail records"
    )
    .interrogate()
)
```

This validation implements conditional business logic: only orders with a `completed` status are
required to have detail records. This pattern is common in real-world scenarios where certain
records trigger mandatory relationships while others don't. The validation returns a boolean for
each completed order, allowing you to see exactly which records pass or fail.

#### Cardinality Constraints

Validate that relationships between datasets follow specific cardinality rules:

```{python}
def check_quantity_ratio(df):
    """Each order should have exactly 3x quantity units in details"""
    order_counts = orders.group_by("order_id").agg(pl.lit(1).alias("order_count"))

    detail_quantities = order_details.group_by("order_id").agg(
        pl.col("quantity").sum().alias("total_quantity")
    )

    joined = order_counts.join(detail_quantities, on="order_id", how="left")

    return joined.with_columns(
        (pl.col("total_quantity") == pl.col("order_count") * 3).alias("valid_ratio")
    ).select("valid_ratio")


(
    pb.Validate(data=orders, tbl_name="orders")
    .specially(
        expr=check_quantity_ratio,
        brief="Each order should have 3x quantity units in details",
        thresholds=(0.4, 0.7),  # Allow some flexibility
    )
    .interrogate()
)
```

Cardinality constraints like this validate that the relationship between datasets follows expected
patterns. In this example, we expect each order to have a specific quantity ratio in the detail
records. Note the use of `thresholds=` to allow some flexibility (not every order needs to meet this
requirement perfectly, but too many violations would indicate a data quality issue).

#### Composite Keys with Business Logic

Validate complex relationships involving multiple columns and conditional logic:

```{python}
# More complex scenario with composite keys
employees = pl.DataFrame({
    "dept_id": ["D1", "D1", "D2", "D2", "D3"],
    "emp_id": ["E001", "E002", "E003", "E004", "E005"],
    "emp_name": ["Alice", "Bob", "Charlie", "Diana", "Eve"],
    "is_manager": [True, False, True, False, False]
})

projects = pl.DataFrame({
    "project_id": ["P1", "P2", "P3", "P4"],
    "dept_id": ["D1", "D2", "D1", "D3"],
    "manager_emp_id": ["E001", "E003", "E001", "E005"]
})

def check_project_manager_validity(df):
    """Project managers must be valid managers in their department"""
    validation_result = df.join(
        employees,
        left_on=["dept_id", "manager_emp_id"],
        right_on=["dept_id", "emp_id"],
        how="left"
    ).with_columns(
        # Manager must exist in dept AND have manager status
        ((pl.col("emp_name").is_not_null()) & (pl.col("is_manager") == True)).alias("valid_manager")
    ).select("valid_manager")

    return validation_result

(
    pb.Validate(data=projects, tbl_name="projects")
    .specially(
        expr=check_project_manager_validity,
        brief="Project managers must be valid managers in their department"
    )
    .interrogate()
)
```

This example demonstrates validation using composite keys (both `dept_id` and `emp_id`) combined
with conditional business logic (checking the `is_manager` flag). Such validations are common in
enterprise systems where relationships must satisfy multiple constraints simultaneously. The
validation reveals that one project (`P4`) fails because employee `E005` is not a manager, even
though they exist in the same department.

### Reusable Validation Factories

For validations you'll use repeatedly, create factory functions that generate customized validators:

```{python}
def make_foreign_key_validator(reference_table, key_columns):
    """Factory function to create reusable foreign key validators"""
    def validate_fk(df):
        if isinstance(key_columns, str):
            keys = [key_columns]
        else:
            keys = key_columns

        ref_keys = reference_table.select(keys).unique()
        matched = df.join(ref_keys, on=keys, how="semi")
        return matched.height == df.height

    return validate_fk

# Use the factory across multiple validations
(
    pb.Validate(data=order_details, tbl_name="order_details")
    .specially(
        expr=make_foreign_key_validator(orders, "order_id"),
        brief="FK constraint: order_id → orders"
    )
    .interrogate()
)
```

Factory functions like `make_foreign_key_validator()` make your validation code more maintainable
and reusable. Once defined, you can use the same factory to validate different foreign key
relationships across your entire data pipeline, ensuring consistency in how these constraints are
checked. This pattern is particularly valuable in production environments where you validate
multiple related tables.

### When to Use `specially()`

The `specially()` method is ideal for:

- advanced cross-dataset validations: conditional existence, cardinality constraints, and business-logic
  predicates that go beyond simple key matching (for standard foreign key checks, use
  [`col_vals_in_table()`](`Validate.col_vals_in_table`))
- complex business rules: multi-column checks, conditional logic, domain-specific constraints
- custom statistical tests: validations requiring calculations not covered by built-in methods
- SQL-style checks: converting complex SQL queries into validation steps
- prototype validations: testing new validation patterns before implementing them as dedicated methods

By combining `specially()` with Pointblank's built-in validation methods, you can create
comprehensive data quality checks that address both standard and highly specific validation
requirements.

## Conclusion

In this article, we've explored the various types of validation methods that Pointblank offers for
ensuring data quality. These methods provide a framework for validating column values, checking
referential integrity across tables, verifying table structures, using AI for complex semantic
validations, and validating aggregate statistics across columns. By combining these validation
methods into comprehensive plans, you can systematically test your data against business rules and
quality expectations. And this all helps to ensure your data remains reliable and trustworthy.


### Column Selection Patterns

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

Data validation often requires working with columns in flexible ways. Pointblank offers two powerful
approaches:

1. Applying validation rules across multiple columns: validate many columns with a single rule
2. Comparing values between columns: create validations that compare values across different columns

This guide covers both approaches in detail with practical examples.

## Part 1: Applying Rules Across Multiple Columns

Many of Pointblank's validation methods perform column-level checks. These methods provide the
`columns=` parameter, which accepts not just a single column name but multiple columns through
various selection methods.

Why is this useful? Often you'll want to perform the same validation check (e.g., checking that
numerical values are all positive) across multiple columns. Rather than defining the same rules
multiple times, you can map the validation across those columns in a single step.

Let's explore this using the `game_revenue` dataset:

```{python}
#| echo: false
pb.preview(pb.load_dataset(dataset="game_revenue"))
```

### Using a List of Column Names

The simplest way to validate multiple columns is to provide a list to the `columns=` parameter. In
the `game_revenue` dataset, we have two columns with numerical data: `item_revenue` and
`session_duration`. If we expect all values in both columns to be greater than `0`, we can write:

```{python}
import pointblank as pb

(
    pb.Validate(data=pb.load_dataset("game_revenue"))
    .col_vals_gt(
        columns=["item_revenue", "session_duration"],
        value=0
    )
    .interrogate()
)
```

The validation report shows two validation steps were created from a single method call! All
validation parameters are shared across all generated steps, including thresholds and briefs:

```{python}
(
    pb.Validate(data=pb.load_dataset("game_revenue"))
    .col_vals_gt(
        columns=["item_revenue", "session_duration"],
        value=0,
        thresholds=(0.1, 0.2, 0.3),
        brief="`{col}` must be greater than zero."
    )
    .interrogate()
)
```

In this example, you can see that the validation report displays customized briefs for each column
("`item_revenue` must be greater than zero." and "`session_duration` must be greater than zero."),
automatically substituting the column name using the `{col}` placeholder in the brief template. This
feature is particularly helpful when reviewing reports, as it provides clear, human-readable
descriptions of what each validation step is checking. When working with multiple columns through a
single validation call, these dynamically generated briefs make your validation reports more
understandable for both technical and non-technical stakeholders.

### Using Pointblank's Column Selectors

For more advanced column selection, Pointblank provides selector functions that resolve columns
based on:

- text patterns in column names
- column position
- column data type

Two common selectors, `starts_with()` and `ends_with()`, resolve columns based on text patterns in
column names.

The `game_revenue` dataset has three columns starting with "item": `item_type`, `item_name`, and
`item_revenue`. Let's check that these columns contain no missing values:

```{python}
(
    pb.Validate(data=pb.load_dataset("game_revenue"))
    .col_vals_not_null(columns=pb.starts_with("item"))
    .interrogate()
)
```

Three validation steps were automatically created because three columns matched the pattern.

The complete list of column selectors includes:

- `starts_with()`
- `ends_with()`
- `contains()`
- [`matches()`](`pointblank.matches`)
- `everything()`
- `first_n()`
- `last_n()`

### Combining Column Selectors

Column selectors can be combined for more powerful selection. To do this, use the `col()` helper
function with logical operators:

- `&` (*and*)
- `|` (*or*)
- `-` (*difference*)
- `~` (*not*)

For example, to select all columns except the first four:

```{python}
col_selection = pb.col(pb.everything() - pb.first_n(4))

(
    pb.Validate(data=pb.load_dataset("game_revenue"))
    .col_vals_not_null(
        columns=col_selection,
        thresholds=(1, 0.05, 0.1)
    )
    .interrogate()
)
```

This selects every column except the first four, resulting in seven validation steps.

### Narwhals Selectors

Pointblank also supports column selectors from the
[Narwhals](https://narwhals-dev.github.io/narwhals/) library, which include:

- [`matches()`](`pointblank.matches`)
- `by_dtype()`
- `boolean()`
- `categorical()`
- `datetime()`
- `numeric()`
- `string()`

Here's an example selecting all numeric columns:

```{python}
import narwhals.selectors as ncs

(
    pb.Validate(data=pb.load_dataset("game_revenue"))
    .col_vals_gt(
        columns=ncs.numeric(),
        value=0
    )
    .interrogate()
)
```

And selecting all string columns matching "item_":

```{python}
(
    pb.Validate(data=pb.load_dataset("game_revenue"))
    .col_vals_not_null(columns=pb.col(ncs.string() & ncs.matches("item_")))
    .interrogate()
)
```

This example demonstrates the power of combining Narwhals selectors with logical operators. By using
`ncs.string()` to select string columns and then filtering with `ncs.matches("item_")`, we can
precisely target text columns with specific naming patterns. This type of targeted selection is
particularly valuable when working with wide datasets that have consistent column naming
conventions, allowing you to apply appropriate validation rules to logically grouped columns without
explicitly listing each one.

### Caveats for Using Column Selectors

While column selectors are powerful, there are some caveats. If a selector doesn't match any
columns, the validation won't fail but will show an 'explosion' in the report:

```{python}
(
    pb.Validate(data=pb.load_dataset("game_revenue"))
    .col_vals_not_null(columns=pb.starts_with("items"))
    .col_vals_gt(columns="item_revenue", value=0)
    .interrogate()
)
```

Notice that although there was a problem with Step 1 (that should be addressed), the interrogation
did move on to Step 2 without complication.

To mitigate uncertainty, include validation steps that check for the existence of key columns with
[`Validate.col_exists()`](`Validate.col_exists`) or verify the schema with [`Validate.col_schema_match()`](`Validate.col_schema_match`).

## Part 2: Comparing Values Between Columns

Sometimes you need to compare values across different columns rather than against fixed values.
Pointblank enables this through the `col()` helper function.

Let's look at examples using the `small_table` dataset:

```{python}
# | echo: false
pb.preview(pb.load_dataset(dataset="small_table"), n_head=20, n_tail=20)
```

### Using `col()`{.qd-no-link} to Specify a Comparison Column

While we typically use validation methods to compare column values against fixed values:

```python
...
.col_vals_gt(columns="a", value=2, ...)
...
```

We can also compare values between columns by using `col()` in the `value=` parameter:

```python
...
.col_vals_gt(columns="a", value=pb.col("x"), ...)
...
```

This checks that each value in column `a` is greater than the corresponding value in column `x`.
Here's a concrete example:

```{python}
(
    pb.Validate(data=pb.load_dataset("small_table"))
    .col_vals_gt(
        columns="d",
        value=pb.col("c")
    )
    .interrogate()
)
```

Notice that the validation report shows both column names (`d` and `c`). There are two failing test
units because of missing values in column `c`. When comparing across columns, missing values in
either column can cause failures.

To handle missing values, use `na_pass=True`:

```{python}
(
    pb.Validate(data=pb.load_dataset("small_table"))
    .col_vals_gt(
        columns="d",
        value=pb.col("c"),
        na_pass=True
    )
    .interrogate()
)
```

Now all tests pass.

The following validation methods accept a `col()` expression in their `value=` parameter:

- [`Validate.col_vals_gt()`](`Validate.col_vals_gt`)
- [`Validate.col_vals_lt()`](`Validate.col_vals_lt`)
- [`Validate.col_vals_ge()`](`Validate.col_vals_ge`)
- [`Validate.col_vals_le()`](`Validate.col_vals_le`)
- [`Validate.col_vals_eq()`](`Validate.col_vals_eq`)
- [`Validate.col_vals_ne()`](`Validate.col_vals_ne`)

### Using `col()` in Range Checks

For range validations via [`Validate.col_vals_between()`](`Validate.col_vals_between`) and [`Validate.col_vals_outside()`](`Validate.col_vals_outside`) you
can use a mix of column references and fixed values:

```{python}
(
    pb.Validate(data=pb.load_dataset("small_table"))
    .col_vals_between(
        columns="d",
        left=pb.col("c"),
        right=10_000,
        na_pass=True
    )
    .interrogate()
)
```

The validation report shows the range as `[c, 10000]`, indicating that the lower bound comes from
column `c` while the upper bound is fixed at `10000`.

## Advanced Examples: Combining Both Approaches

The true power comes from combining both approaches: validating multiple columns and using
cross-column comparisons:

```{python}
validation = (
    pb.Validate(data=pb.load_dataset("small_table"))
    .col_vals_gt(
        columns=["c", "d"],
        value=pb.col("a"),
        na_pass=True
    )
    .interrogate()
)

validation
```

This creates validation steps checking that values in both columns `d` and `e` are greater than
their corresponding values in column `a`.

## Conclusion

Pointblank provides flexible approaches to working with columns:

1. Column selection: validate multiple columns with a single validation rule
2. Cross-column comparison: compare values between columns

These capabilities allow you to:

- write more concise validation code
- apply consistent validation rules across similar columns
- create dynamic validations that check relationships between columns
- build comprehensive data quality checks with minimal code

By getting familiar with these techniques, you can create more elegant and powerful validation plans
while also reducing repetition in your code.


### Preprocessing

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

While the available validation methods can do a lot for you, there's likewise a lot of things you
*can't* easily do with them. What if you wanted to validate that

- string lengths in a column are less than 10 characters?
- the median of values in a column is less than the median of values in another column?
- there are at least three instances of every categorical value in a column?

These constitute more sophisticated validation requirements, yet such examinations are quite
prevalent in practice. Rather than expanding our library to encompass every conceivable validation
scenario (a pursuit that would yield an unwieldy and potentially infinite collection) we instead
employ a more elegant approach. By transforming the table under examination through judicious
preprocessing and exposing key metrics, we may subsequently employ the existing collection of
validation methods. This compositional strategy affords us considerable analytical power while
maintaining conceptual clarity and implementation parsimony.

Central to this approach is the idea of composability. Pointblank makes it easy to safely transform
the target table for a given validation via the `pre=` argument. Any computed columns are available
for the (short) lifetime of the validation step during interrogation. This composability means:

1. we can validate on different forms of the initial dataset (e.g., validating on
aggregate forms, validating on calculated columns, etc.)
2. there's no need to start an entirely new validation process for each transformed version of the
data (i.e., one tabular report could be produced instead of several)

This compositional paradigm allows us to use data transformation effectively within our validation
workflows, maintaining both flexibility and clarity in our data quality assessments.

## Transforming Data with Lambda Functions

Now, through examples, let's look at the process of performing the validations mentioned above.
We'll use the `small_table` dataset for all of the examples. Here it is in its entirety:

```{python}
#| echo: false
pb.preview(pb.load_dataset(dataset="small_table", tbl_type="polars"), n_head=20, n_tail=20)
```

In getting to grips with the basics, we'll try to validate that string lengths in the `b` column are
less than 10 characters. We can't directly use the [`Validate.col_vals_lt()`](`Validate.col_vals_lt`) validation method with
that column because it is meant to be used with a column of numeric values. Let's just give that
method what it needs and create a column with string lengths!

The target table is a Polars DataFrame so we'll provide a lambda function that uses the Polars API
to add in that numeric column:

```{python}
import polars as pl

(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        tbl_name="small_table",
        label="String lengths"
    )
    .col_vals_lt(

        # The generated column, via `pre=` (see below) ---
        columns="string_lengths",

        # The string length value to be less than ---
        value=10,

        # Lambda function that modifies the table ---
        pre=lambda df: df.with_columns(string_lengths=pl.col("b").str.len_chars())
    )
    .interrogate()
)
```

The validation was successfully constructed and we can see from the validation report table that all
strings in `b` had lengths less than 10 characters. Also note that the icon under the `TBL` column
is no longer a rightward-facing arrow, but one that is indicative of a transformation taking place.

Let's examine the transformation approach more closely. In the previous example, we're not directly
testing the `b` column itself. Instead, we're validating the `string_lengths` column that was
generated by the lambda function provided to `pre=`. The Polars API's `with_columns()` method does
the heavy lifting, creating numerical values that represent each string's length in the original
column.

That transformation occurs only during interrogation and only for that validation step. Any prior or
subsequent steps would normally use the as-provided `small_table`. Having the possibility for
data transformation being isolated at the step level means that you don't have to generate separate
validation plans for each form of the data, you're free to fluidly transform the target table as
necessary for perform validations on different representations of the data.

## Using Custom Functions for Preprocessing

While lambda functions work well for simple transformations, custom named functions can make your
validation code more organized and reusable, especially for complex preprocessing logic. Let's
implement the same string length validation using a dedicated function:

```{python}
def add_string_lengths(df):
    # This generates string length from a column `b`; the new column with
    # the values is called `string_lengths` (will be placed as the last column)
    return df.with_columns(string_lengths=pl.col("b").str.len_chars())

(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        tbl_name="small_table",
        label="String lengths for column `b`."
    )
    .col_vals_lt(

        # Use of a column selector function to select the last column ---
        columns=pb.last_n(1),

        # The string length to be less than ---
        value=10,

        # Custom function for generating string lengths in a new column ---
        pre=add_string_lengths
    )
    .interrogate()
)
```

The column-generating logic was placed in the `add_string_lengths()` function, which is then passed
to `pre=`. Notice we're using `pb.last_n(1)` in the `columns` parameter. This is a convenient column
selector that targets the last column in the DataFrame, which in our case is the newly created
`string_lengths` column. This saves us from having to explicitly write out the column name, making
our code more adaptable if column names change. Despite not specifying the name directly, you'll
still see the actual column name (`string_lengths`) displayed in the validation report.

## Creating Parameterized Preprocessing Functions

So far we've used simple functions and lambdas, but sometimes you may want to create more flexible
preprocessing functions that can be configured with parameters. Let's create a reusable function
that can calculate string lengths for any column:

```{python}
def string_length_calculator(column_name):
    """Returns a preprocessing function that calculates string lengths for the specified column."""
    def preprocessor(df):
        return df.with_columns(string_lengths=pl.col(column_name).str.len_chars())
    return preprocessor

# Validate string lengths in column b
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        tbl_name="small_table",
        label="String lengths for column `b`."
    )
    .col_vals_lt(
        columns=pb.last_n(1),
        value=10,
        pre=string_length_calculator(column_name="b")
    )
    .interrogate()
)
```

This pattern is called a *function factory*, which is a function that creates and returns another
function. The outer function (`string_length_calculator()`) accepts parameters that customize the
behavior of the returned preprocessing function. The inner function (`preprocessor()`) is what
actually gets called during validation.

This approach offers several benefits as it:

- creates reusable, configurable preprocessing functions
- keeps your validation code DRY
- allows you to separate configuration from implementation
- enables easy application of the same transformation to different columns

You could extend this pattern to create even more sophisticated preprocessing functions with
multiple parameters, default values, and complex logic.

## Using Narwhals to Preprocess Many Types of DataFrames

In this previous example we used a Polars table. You might have a situation where you perform data
validation variously on Pandas and Polars DataFrames. This is where Narwhals becomes handy: it
provides a single, consistent API that works across multiple DataFrame types, eliminating the need
to learn and switch between different APIs depending on your data source.

Let's obtain `small_table` as a Pandas DataFrame. We'll construct a validation step to verify that
the median of column `c` is greater than the median in column `a`.

```{python}
import narwhals as nw

# Define preprocessing function using Narwhals for cross-backend compatibility
def get_median_columns_c_and_a(df):
    return nw.from_native(df).select(nw.median("c"), nw.median("a"))

(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="pandas"),
        tbl_name="small_table",
        label="Median comparison.",
    )
    .col_vals_gt(
        columns="c",
        value=pb.col("a"),

        # Custom function to modify the table; generates table with columns `c` and `a` ---
        pre=get_median_columns_c_and_a
    )
    .interrogate()
)
```

The goal is to check that the median value of `c` is greater than the corresponding median of
column `a`, which we set up through the `columns=` and `value=` parameters in the
[`Validate.col_vals_gt()`](`Validate.col_vals_gt`) method.

There's a bit to unpack here, so let's look at the preprocessing function first. Narwhals can
translate a Pandas DataFrame to a Narwhals DataFrame with its `from_native()` function. After that
initiating step, you're free to use the Narwhals API (which is modeled on a subset of the Polars
API) to do the necessary data transformation. In this case, we are getting the medians of the `c`
and `a` columns and ending up with a one-row, two-column table.

We should note that the transformed table is, perhaps surprisingly, a Narwhals DataFrame (we didn't
have to go back to a Pandas DataFrame by using `.to_native()`). Pointblank is able to work directly
with the Narwhals DataFrame for validation purposes, which makes the workflow more concise.

One more thing to note: Pointblank provides a convenient syntactic sugar for working with Narwhals.
If you name the parameter of the preprocessing function (a lambda or a regular function) `dfn`
instead of `df`, the system automatically applies `nw.from_native()` to the input DataFrame first,
and converts the result back to the original DataFrame type. This lets you write more concise code
without having to explicitly convert the DataFrame to a Narwhals format. The function above could
be written like this:

```python
def get_median_columns_c_and_a(dfn):
    return dfn.select(nw.median("c"), nw.median("a"))
```

## Swapping in a Totally Different DataFrame

Sometimes data validation requires looking at completely transformed versions of your data (such as
aggregated summaries, pivoted views, or even reference tables). While this approach goes against the
typical paradigm of validating a single *target table*, there are legitimate use cases where you
might need to validate properties that only emerge after significant transformations.

Let's now try to prepare the final validation scenario, checking that there are at least three
instances of every categorical value in column `f` (which contains string values in the set of
`"low"`, `"mid"`, and `"high"`). This time, we'll prepare the transformed table (transformed by
Polars expressions) outside of the Pointblank code.

```{python}
data_original = pb.load_dataset(dataset="small_table", tbl_type="polars")
data_transformed = data_original.group_by("f").len(name="n")

data_transformed
```

Then, we'll plug in the `data_transformed` DataFrame with a preprocessing function:

```{python}
# Define preprocessing function to use the transformed data
def use_transformed_data(df):
    return data_transformed

(
    pb.Validate(
        data=data_original,
        tbl_name="small_table",
        label="Category counts.",
    )
    .col_vals_ge(
        columns="n",
        value=3,
        pre=use_transformed_data
    )
    .interrogate()
)
```

We can see from the validation report table that there are three test units. This corresponds to a
row for each of the categorical value counts. From the report, we find that two of the three test
units are passing test units (turns out there are only two instances of `"mid"` in column `f`).

Note that the swapped-in table can be any table type that Pointblank supports, like a Polars
DataFrame (as shown here), a Pandas DataFrame, a Narwhals DataFrame, or any other compatible format.
This flexibility allows you to validate properties of your data that might only be apparent after
significant reshaping or aggregation.

## Conclusion

The preprocessing capabilities in Pointblank provide the power and flexibility for validating
complex data properties beyond what's directly possible with the standard validation methods.
Through the `pre=` parameter, you can:

- transform your data on-the-fly with computed columns
- generate aggregated metrics to validate statistical properties
- work seamlessly across different DataFrame types using Narwhals
- swap in completely different tables when validating properties that emerge only after
transformation

By combining these preprocessing techniques with Pointblank's validation methods, you can create
comprehensive data quality checks that address virtually any validation scenario without needing an
endless library of specialized validation functions. This composable approach keeps your validation
code concise while allowing you to verify even the most complex data quality requirements.

Remember that preprocessing happens just for the specific validation step, keeping your validation
plan organized and maintaining the integrity of your original data throughout the rest of the
validation process.


### Segmentation

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

When validating data, you often need to analyze specific subsets or segments of your data
separately. Maybe you want to ensure that data quality meets standards in each geographic region,
for each product category, or across different time periods. This is where the `segments=` argument
can be useful.

Data segmentation lets you split a validation step into multiple segments, with each segment
receiving its own validation step. Rather than validating an entire table at once, you could instead
validate different partitions separately and get separate results for each.

The `segments=` argument is available in many validation methods; typically it's in those methods
that check values within rows, and those methods that examine entire rows
([`Validate.rows_distinct()`](`Validate.rows_distinct`), [`Validate.rows_complete()`](`Validate.rows_complete`)). When you use it, Pointblank will:

1. split your data according to your segmentation criteria
2. run the validation separately on each segment
3. report results individually for each segment

Let's explore how to use the `segments=` argument through a few practical examples.

## Basic Segmentation by Column Values

The simplest way to segment data is by the unique values in a column. For the upcoming example,
we'll use the `small_table` dataset, which contains a categorical-value column called `f`.

First, let's preview the dataset:

```{python}
table = pb.load_dataset()

pb.preview(table)
```

Now, let's validate that values in column `d` are greater than `100`, but we'll also segment the
validation by the categorical values in column `f`:

```{python}
validation_1 = (
    pb.Validate(
        data=pb.load_dataset(),
        tbl_name="small_table",
        label="Segmented validation by category"
    )
    .col_vals_gt(
        columns="d", value=100,

        # Segment by unique values in column `f` ---
        segments="f"
    )
    .interrogate()
)

validation_1
```

In the validation report, notice that instead of a single validation step, we have multiple steps:
one for each unique value in the `f` column. The segmentation is clearly indicated in the `STEP`
column with labels like `SEGMENT  f / high`, making it easy to identify which segment each
validation result belongs to. This clear labeling helps when reviewing reports, especially with
complex validations that use multiple segmentation criteria.

## Segmenting on Specific Values

Sometimes you don't want to segment on all unique values in a column, but only on specific ones of
interest. You can do this by providing a tuple with the column name and a list of values:

```{python}
validation_2 = (
    pb.Validate(
        data=pb.load_dataset(),
        tbl_name="small_table",
        label="Segmented validation on specific categories"
    )
    .col_vals_gt(
        columns="d",
        value=100,
        segments=("f", ["low", "high"])  # Only segment on "low" and "high" values in column `f`
    )
    .interrogate()
)

validation_2
```

In this example, we only create validation steps for the `"low"` and `"high"` segments, ignoring any
rows with `f` equal to `"mid"`.

If no segments are found (for example, because the table has no rows or the list of values is
empty), the step is kept as a single step with a note explaining why. That way the step still shows
up in the report and the steps after it keep their numbers.

## Multiple Segmentation Criteria

For more complex segmentation, you can provide a list of columns or column-value tuples. This
creates segments based on combinations of criteria:

```{python}
validation_3 = (
    pb.Validate(
        data=pb.load_dataset(),
        tbl_name="small_table",
        label="Multiple segmentation criteria"
    )
    .col_vals_gt(
        columns="d",
        value=100,

        # Segment by values in `f` AND specific values in `a` ---
        segments=["f", ("a", [1, 2])]
    )
    .interrogate()
)

validation_3
```

This creates validation steps for each combination of values in column `f` and the specified values
in column `a`.

## Segmentation with Preprocessing

You can combine segmentation with preprocessing for powerful and flexible validations. All
preprocessing is applied before segmentation occurs, which means you can create derived columns to
segment on:

```{python}
import polars as pl

# Define preprocessing function for creating a categorical column
def add_d_category_column(df):
    return df.with_columns(
        d_category=pl.when(pl.col("d") > 150).then(pl.lit("high")).otherwise(pl.lit("low"))
    )

validation_4 = (
    pb.Validate(
        data=pb.load_dataset(tbl_type="polars"),
        tbl_name="small_table",
        label="Segmentation with preprocessing",
    )
    .col_vals_gt(
        columns="d", value=100,

        # Create a column containing categorical values ---
        pre=add_d_category_column,

        # Segment by the computed column `d_category` generated via `pre=` ---
        segments="d_category",
    )
    .interrogate()
)

validation_4
```

In this example, we first create a derived column `d_category` based on whether `d` is greater than
`150`. Then, we segment our validation based on this derived column by using
`segments="d_category"`.

## When to Use Segmentation

Segmentation is particularly useful when:

1. Data quality standards vary by group: different regions, product lines, or customer segments
might have different acceptable thresholds
2. Identifying problem areas: segmentation helps pinpoint exactly where data quality issues
exist, rather than just knowing that some issue exists somewhere in the data
3. Generating detailed reports: by segmenting, you get more granular reporting that can be
shared with different stakeholders responsible for different parts of the data
4. Tracking improvements over time: segmented validations make it easier to see if data quality
is improving in specific areas that were previously problematic

By using segmentation strategically in these scenarios, you can transform your data validation from
a simple pass/fail system into a much more nuanced diagnostic tool that provides actionable insights
about data quality across different dimensions. This targeted approach not only helps identify
issues more precisely but also enables more effective communication of data quality metrics to
relevant stakeholders.

## Segmentation vs. Multiple Validation Steps

So why use segmentation instead of just creating separate validation steps for each segment using
filtering in the `pre=` argument? Well, segmentation offers several nice advantages:

1. Conciseness: you define your validation logic once, not repeatedly for each segment
2. Consistency: we can be certain that the same validation is applied uniformly across segments
3. Clarity: the validation report will clearly organize results by segment (with extra labeling)
4. Convenience: there's no need to manually extract and filter subsets of your data

Segmentation can end of simplifying your validation code while also providing more structured and
informative reporting about different portions of your data.

## Practical Example: Validating Sales Data by Region and Product Type

Let's see a more realistic example where we validate sales data segmented by both region and product
type:

```{python}
import pandas as pd
import numpy as np

# Create a sample sales dataset
np.random.seed(123)

# Create a simple sales dataset
sales_data = pd.DataFrame({
    "region": np.random.choice(["North", "South", "East", "West"], 100),
    "product_type": np.random.choice(["Electronics", "Clothing", "Food"], 100),
    "units_sold": np.random.randint(5, 100, 100),
    "revenue": np.random.uniform(100, 10000, 100),
    "cost": np.random.uniform(50, 5000, 100)
})

# Calculate profit
sales_data["profit"] = sales_data["revenue"] - sales_data["cost"]
sales_data["profit_margin"] = sales_data["profit"] / sales_data["revenue"]

# Preview the dataset
pb.preview(sales_data)
```

Now, let's validate that profit margins are above 20% across different regions and product types:

```{python}
validation_5 = (
    pb.Validate(
        data=sales_data,
        tbl_name="sales_data",
        label="Sales data validation by region and product"
    )
    .col_vals_gt(
        columns="profit_margin",
        value=0.2,
        segments=["region", "product_type"],
        brief="Profit margin > 20% check"
    )
    .interrogate()
)

validation_5
```

This validation gives us a detailed breakdown of profit margin performance across the different
regions and product types, making it easy to identify areas that need attention.

## Best Practices for Segmentation

Effective data segmentation requires thoughtful planning about how to divide your data in ways that
make sense for your validation needs. When implementing segmentation in your data validation
workflow, consider these key principles:

1. Choose meaningful segments: select segmentation columns that align with your business logic and
organizational structure

2. Use preprocessing when needed: if your raw data doesn't have good segmentation columns, create
them through preprocessing (with the `pre=` argument)

3. Combine with actions: for critical segments, define segment-specific actions using the `actions=`
parameter to respond to validation failures.

By implementing these best practices, you'll create more targeted, maintainable, and actionable data
validations. Segmentation becomes most powerful when it aligns with natural divisions in your data
and analytical processes, allowing for more precise identification of quality issues while
maintaining a unified validation framework.

## Conclusion

Data segmentation can make your validations more targeted and informative. By dividing your data
into meaningful segments, you can identify quality issues with greater precision, apply appropriate
validation standards to different parts of your data, and generate more actionable reports.

The `segments=` parameter transforms validation from a monolithic process into a granular assessment
of data quality across various dimensions of your dataset. Whether you're dealing with regional
differences, product categories, time periods, or any other meaningful divisions in your data,
segmentation makes it possible to validate each portion according to its specific requirements while
maintaining the simplicity of a unified validation framework.


### Thresholds

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Thresholds are a key concept in Pointblank that allow you to define acceptable limits for failing
validation tests. Rather than a simple pass/fail model, thresholds enable you to signal failure at
different severity levels ('warning', 'error', and 'critical'), giving you fine-grained control over
how data quality issues are reported and handled.

When used with actions (covered in the next section), thresholds create a robust system for
responding to data quality issues based on their severity. This approach allows you to:

- set different tolerance levels for different types of validation checks
- escalate responses based on the severity of data quality issues
- configure different notification strategies for different threshold levels
- create a more nuanced data validation workflow than simple pass/fail tests

## A Simple Example

Let's start with a basic example that demonstrates how thresholds work in practice:

```{python}
import pointblank as pb

(
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_not_null(
        columns="c",

        # Set thresholds for the validation step ---
        thresholds=pb.Thresholds(warning=1, error=0.2)
    )
    .interrogate()
)
```

In this example, we're validating that column `c` contains no Null values. We've set:

- A 'warning' threshold of `1` (triggers when 1 or more values are Null)
- An 'error' threshold of `0.2` (triggers when 20% or more values are Null)

Looking at the results:

- the `FAIL` column shows that 2 test units have failed
- the `W` column (for 'warning') shows a filled gray circle, indicating the warning threshold has
been exceeded
- the `E` column (for 'error') shows an open yellow circle, indicating the error threshold has not
been exceeded
- the `C` column (for 'critical') shows a dash since we didn't set a critical threshold

## Types of Threshold Values

Thresholds in Pointblank can be specified in two different ways:

### Absolute Thresholds

Absolute thresholds are specified as integers and represent a fixed number of failing test units:

```python
# Warning threshold of exactly 5 failing test units
thresholds_absolute = pb.Thresholds(warning=5)
```

With this configuration, the 'warning' threshold would be triggered if 5 or more test units fail.

### Proportional Thresholds

Proportional thresholds are specified as decimals between 0 and 1, representing a percentage of the
total test units:

```python
# Error threshold of 10% of test units failing
thresholds_proportional = pb.Thresholds(error=0.1)
```

With this configuration, the 'error' threshold would be triggered if 10% or more of the test units
fail.

### Boolean Shorthand

For cases where you want to allow exactly 1 failing test unit, you can use `True` as a convenient
shorthand:

```python
# Critical threshold of exactly 1 failing test unit
thresholds_boolean = pb.Thresholds(critical=True)
```

This is equivalent to setting `critical=1` but provides a more intuitive way to express "allow at
most one failure". This shorthand is particularly useful for strict validations where any failure
beyond a single edge case should trigger immediate attention.

## Understanding Severity Levels

The three threshold levels in Pointblank ('warning', 'error', and 'critical') are inspired by
traditional logging levels used in software development. These names suggest a progression of
severity:

- **'warning'** (level `30`): indicates potential issues that don't necessarily prevent normal
operation
- **'error'** (level `40`): suggests more serious problems that might impact data quality
- **'critical'** (level `50`): represents the most severe issues that likely require immediate
attention

These numerical values (`30`, `40`, `50`) are used internally by Pointblank when determining
threshold hierarchy and can be accessed through the `{level_num}` field in action metadata (covered
in the next **User Guide** article).

While these names imply certain severity levels, they're ultimately just convenient labels for
different thresholds. You have complete flexibility in how you use them:

- you could use 'warning' for issues that should block a pipeline
- you might configure 'critical' for minor issues that just need documentation
- the 'error' level could trigger informational emails rather than actual error handling

The naming is primarily a suggestion to help organize your validation strategy. What matters most is
how you configure actions for each threshold level to suit your specific data quality requirements.

## Threshold Behavior

It's important to understand a few key behaviors of thresholds:

- thresholds are **inclusive**: a value equal to or exceeding the threshold will trigger the
associated level
- thresholds can be **mixed**: you can use absolute values for some levels and proportional for
others
- threshold levels are **hierarchical**: 'critical' is more severe than 'error', which is more
severe than 'warning'
- when a test fails, **all** applicable threshold levels are marked in the report (though actions
may only execute for the highest level by default)

## Setting Global Thresholds

You can set thresholds globally for all validation steps in a workflow using the `thresholds=`
parameter in `Validate`:

```{python}
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),

        # Setting thresholds for all validation steps ---
        thresholds=pb.Thresholds(warning=1, error=0.1)
    )
    .col_vals_not_null(columns="a")
    .col_vals_gt(columns="a", value=2)
    .interrogate()
)
```

With this approach, the same thresholds are applied to every validation step in the workflow.

## Overriding Thresholds for Specific Steps

You can override global thresholds for specific validation steps by providing the `thresholds=`
parameter in individual validation methods:

```{python}
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),

        # Setting global thresholds ---
        thresholds=pb.Thresholds(warning=1, error=0.1)
    )
    .col_vals_not_null(columns="a")
    .col_vals_gt(
        columns="a", value=2,

        # Step-specific threshold that overrides global ---
        thresholds=pb.Thresholds(warning=3)
    )
    .interrogate()
)
```

In this example, the second validation step uses its own 'warning' threshold of `3`, overriding the
global setting of `1`.

## Ways to Define Thresholds

Pointblank offers multiple ways to define thresholds to accommodate different coding styles and
requirements.

### 1. Using the `Thresholds` Class (Recommended)

The most explicit and flexible approach is using the `Thresholds` class:

```{python}
# Set individual thresholds for different levels
thresholds_all_levels = pb.Thresholds(warning=0.05, error=0.1, critical=0.25)

# Set only specific levels
thresholds_error_only = pb.Thresholds(error=0.15)
```

This approach allows you to:

- set any combination of threshold levels
- use descriptive parameter names for clarity
- skip levels you don't need to set

### 2. Using a Tuple

For concise code, you can use a tuple where positions represent 'warning', 'error', and 'critical'
levels in that order:

```{python}
# (warning, error, critical)
thresholds_tuple = (1, 0.1, 0.25)

# Shorter tuples are also allowed
thresholds_tuple_warning = (3,)            # Only the 'warning' threshold
thresholds_tuple_warning_error = (3, 0.2)  # Both 'warning' and 'error' thresholds
```

While concise, this approach requires you to start with the 'warning' level and add levels in order.

### 3. Using a Dictionary

You can also use a dictionary with keys that match the threshold level names:

```{python}
# Can use any combination of threshold levels
thresholds_dict = {"warning": 1, "critical": 0.15}
```

The dictionary must use the exact keys `"warning"`, `"error"`, and/or `"critical"`.

### 4. Using a Single Value

The simplest approach is using a single numeric value, which sets just the 'warning' threshold:

```{python}
# Sets 'warning' threshold to `5`
thresholds_single = 5
```

This is equivalent to `pb.Thresholds(warning=5)`.

## Thresholds and Validation Steps

Let's look at a more complete validation workflow that demonstrates different threshold
configurations:

```{python}
# Create a validation workflow with global and step-specific thresholds
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),

        # Global thresholds applied to all steps unless overridden ---
        thresholds=pb.Thresholds(warning=0.05, error=0.1, critical=0.2)
    )

    # Step 1: Uses global thresholds ---
    .col_vals_not_null(columns="b")

    # Step 2: Overrides with step-specific thresholds ---
    .col_vals_gt(
        columns="a", value=2,
        thresholds=pb.Thresholds(warning=1, critical=0.3) # No 'error' threshold
    )

    # Step 3: Uses a simplified tuple notation ---
    .col_vals_not_null(columns="c", thresholds=(2, 0.15))

    .interrogate()
)
```

## Thresholds and Actions

While thresholds by themselves provide visual indicators of validation severity in reports, their
real power emerges when combined with Actions. The Actions system (covered in the next article)
allows you to specify what happens when a threshold is exceeded.

For example, you might configure:

- A 'warning' threshold that logs a message
- An 'error' threshold that sends an email notification
- A 'critical' threshold that blocks a data pipeline

Here's a simple preview of how thresholds and actions work together:

```{python}
(
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),

        # Define thresholds for all three severity levels ---
        thresholds=pb.Thresholds(warning=1, error=2, critical=3),

        # Define actions for different threshold levels ---
        actions=pb.Actions(
            warning="Warning: {step} has {FAIL} failing values",
            error="ERROR: Step {step} exceeded the 'error' threshold",
            critical="CRITICAL: Data quality issue in column {col}"
        )
    )
    .col_vals_not_null(columns="c")
    .interrogate()
)
```

## Conclusion

Thresholds are a powerful feature that transform Pointblank from a simple validation tool into a
sophisticated data quality monitoring system. By setting appropriate thresholds, you can:

1. Define different severity levels for data quality issues
2. Customize tolerance levels for different types of validation checks
3. Create a more nuanced approach to data validation than binary pass/fail
4. Enable targeted actions based on the severity of issues detected

In the next article, we'll explore the Actions system in depth, showing you how to define automatic
responses when thresholds are exceeded.


### Actions

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Actions transform data validation from passive reporting to active response by automatically
executing code when quality issues arise. They bridge the gap between detection and intervention,
enabling immediate notifications and comprehensive logging when thresholds are exceeded.

Whether you need simple console messages for interactive analysis or complex alerting for production
pipelines, Actions provide the framework to make your validation workflows responsive. For example,
when validating revenue values, you can configure immediate alerts if failures exceed acceptable
thresholds, ensuring data issues are addressed promptly rather than discovered later.

In this article, we'll explore how to use Actions to respond to threshold violations during data
validation, and Final Actions to execute code after all validation steps are complete, giving you
powerful tools to monitor, alert, and report on your data's quality.

## How Actions Work

Let's look at an example on how this works in practice. The following validation plan contains a
single step (using [`Validate.col_vals_gt()`](`Validate.col_vals_gt`)) where the `thresholds=` and `actions=` parameters are
set using `Thresholds` and `Actions` calls:

```{python}
import pointblank as pb

(
    pb.Validate(data=pb.load_dataset(dataset="small_table"))
    .col_vals_gt(
        columns="c", value=2,
        thresholds=pb.Thresholds(warning=1, error=5),

        # Emit a console message when the warning threshold is exceeded ---
        actions=pb.Actions(warning="WARNING: failing test found.")
    )
    .interrogate()
)
```

The code uses `thresholds=pb.Thresholds(warning=1, error=5)` to set a 'warning' threshold of `1` and
an 'error' threshold of `5` failing test units. The results part of the validation table shows that:

- The `FAIL` column shows that 3 tests units have failed
- The `W` column (short for 'warning') shows a filled gray circle indicating it's reached its
threshold level
- The `E` ('error') column shows an open yellow circle indicating it's below the threshold
level

More importantly, the text `"WARNING: failing test found."` has been emitted. Here it appears above
the validation table and that's because the action is executed eagerly during interrogation
(before the report has even been generated).

So, an action is executed for a particular condition (e.g., 'warning') within a validation step if
these three things are true:

1. there is a threshold set for that condition (either globally, or as part of that step)
2. there is an associated action set for the condition (again, either set globally or within the
step)
3. during interrogation, the threshold value for the condition was exceeded by the number or
proportion of failing test units

There is a lot of flexibility for setting both thresholds and actions and everything here is
considered optional. Put another way, you can set various thresholds and various actions as needed
and the interrogation phase will determine whether all the requirements are met for executing
an action.

## Defining Actions

Actions can be defined in several ways, providing flexibility for different notification needs.

### Using String Messages

There are a few options in how to define the actions:

1. **String**: a message to be displayed in the console
2. **Callable**: a function to be called
3. **List of Strings/Callables**: for execution of multiple messages or functions

The actions are executed at interrogation time when the threshold level assigned to the action
is exceeded by the number or proportion of failing test units. When providing a string, it will
simply be printed to the console. A callable will also be executed at the time of interrogation.
If providing a list of strings or callables, each item in the list will be executed in order.
Such a list can contain a mix of strings and callables.

Displaying console messages may be a simple approach, but it is effective. And the strings don't
have to be static, there are templating features that can be useful for constructing strings for a
variety of situations. The following placeholders are available for use:

- `{type}`: The validation step type where the action is executed (e.g., ‘col_vals_gt’, etc.)
- `{level}`: The threshold level where the action is executed (‘warning’, ‘error’, or ‘critical’)
- `{step}` or `{i}`: The step number in the validation workflow where the action is executed
- `{col}` or `{column}`: The column name where the action is executed
- `{val}` or `{value}`: An associated value for the validation method
- `{time}`: A datetime value for when the action was executed

Here's an example where we prepare a console message with a number of value placeholders
(`action_str`) and use it globally at `Actions(critical=)`:

```{python}
action_str = "[{LEVEL}: {TYPE}]: Step {step} has failed validation. ({time})"

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),

        # Use `action_str` for any critical thresholds exceeded ---
        actions=pb.Actions(critical=action_str),
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
    .col_vals_gt(columns="item_revenue", value=0.10)
    .col_vals_ge(columns="session_duration", value=15)
    .interrogate()
)
```

What we get here are two messages in the console, corresponding to critical failures in steps 2 and
3. The placeholders were replaced with the correct text for the context. Note that some of the
resulting text is capitalized (e.g., `"CRITICAL"`, `"COL_VALS_GT"`, etc.) and this is because we
capitalized the placeholder text itself. Have a look at the documentation article of `Actions` for
more details on this.

### Using Callable Functions

Aside from strings, any callable can be used as an action value. Here's an example where we use a
custom function as part of an action:

```{python}
def duration_issue():
    from datetime import datetime
    print(f"Data quality issue found ({datetime.now()}).")

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .col_vals_gt(
        columns="session_duration", value=15,

        # Use the `duration_issue()` function as an action for this step ---
        actions=pb.Actions(warning=duration_issue),
    )
    .interrogate()
)
```

In this case, the 'warning' action is set to call the user's `dq_issue()` function. This action is
only executed when the 'warning' threshold is exceeded in step 3. Because all three thresholds are
exceeded in that step, the 'warning' action of executing the function occurs (resulting in a
message being printed to the console).

This is an example where actions can be defined locally for an individual validation step. The
global threshold setting applied to all three validation steps but the step-level action only
applied to step 3. You are free to mix and match both threshold and action settings at the global
level (i.e., set in the `Validate` call) or at the step level. The key thing to be aware of is that
step-level settings of thresholds and actions take precedence.

## Accessing Context in Actions

While string templates provide helpful placeholders to access information about validation steps,
callable functions offer more flexibility through access to detailed metadata. When using functions
as actions, you can retrieve comprehensive information about the validation context, allowing for
complex logic and dynamic responses to validation issues.

### Using `get_action_metadata()`{.qd-no-link} in Callables

To access information about the validation step where an action was triggered, we can call
`get_action_metadata()` in the body of a function to be used within `Actions`. This provides useful
context about the validation step that triggered the action.

```{python}
def print_problem():
    m = pb.get_action_metadata()
    print(f"{m['level']} ({m['level_num']}) for Step {m['step']}: {m['failure_text']}")

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),

        # Use the `print_problem()` function as the action ---
        actions=pb.Actions(default=print_problem),
        brief=True,
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .col_vals_gt(columns="session_duration", value=15)
    .interrogate()
)
```

In this example, we're creating a function called `print_problem()` that prints information about
each validation step that fails. We then apply this function as the default action for all threshold
levels using `actions=pb.Actions(default=print_problem)`. (Note that the `default=` and
`highest_only=` parameters will be covered in more detail in following sections.)

We end up seeing two messages printed for failures in Steps 2 and 3. And though those steps had more
than one threshold exceeded, only the most severe level in each yielded a console message (due to
the default `highest_only=True` behavior).

By setting the action in `Validate(actions=)`, we applied it to all validation steps where
thresholds are exceeded. This eliminates the need to set `actions=` at every validation step (though
you can do this as a local override, even setting `actions=None` to disable globally set actions).

### Available Metadata Fields

The dictionary returned by `get_action_metadata()` contains the following fields:

- `step`: The step number.
- `column`: The column name.
- `value`: The value being compared (only available in certain validation steps).
- `type`: The assertion type (e.g., `"col_vals_gt"`, etc.).
- `time`: The time the validation step was executed (in ISO format).
- `level`: The severity level (`"warning"`, `"error"`, or `"critical"`).
- `level_num`: The severity level as a numeric value (`30`, `40`, or `50`).
- `autobrief`: A localized and brief statement of the expectation for the step.
- `failure_text`: Localized text that explains how the validation step failed.

## Customizing Action Behavior

The `Actions` class has two additional parameters that provide more control over how actions are
executed:

### Setting Default Actions with `default=`

Instead of specifying actions separately for each threshold level, you can use the `default=`
parameter to set a common action for all levels:

```{python}
def log_all_issues():
    m = pb.get_action_metadata()
    print(f"[{m['level'].upper()}] Validation failed in step {m['step']} with level {m['level']}")

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),

        # The `log_all_issues()` callable is set to every threshold ---
        actions=pb.Actions(default=log_all_issues),
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .col_vals_gt(columns="session_duration", value=15)
    .interrogate()
)
```

The `default=` parameter sets the same action for all threshold levels. If you later specify an
action for a specific level, it will override this default for that level only.

When using the `default=` parameter, be aware that your action (whether a string template or
callable function) needs to work across all validation steps where thresholds might be exceeded. Not
all validation methods provide the same context for string templates or in the metadata dictionary
returned by `get_action_metadata()`.

For example, some validation steps like [`Validate.col_vals_gt()`](`Validate.col_vals_gt`) provide a `value` field that can
be accessed with `{value}` in string templates, while others like [`Validate.col_exists()`](`Validate.col_exists`) don't
have this concept. When creating default actions, either use only the universally available
placeholders (`{step}`, `{level}`, `{type}`, and `{time}`), or include conditional logic in your
callable functions to handle different validation types appropriately.

### Controlling Action Execution with `highest_only=`

By default, Pointblank only executes the action for the most severe threshold level that's been
exceeded. If you want actions for all exceeded thresholds to be executed, you can set
`highest_only=False`:

```{python}
(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),
        actions=pb.Actions(
            warning="Warning threshold exceeded in step {step}",
            error="Error threshold exceeded in step {step}",
            critical="Critical threshold exceeded in step {step}",

            # Execute all applicable actions ---
            highest_only=False
        ),
    )
    .col_vals_gt(columns="session_duration", value=15)
    .interrogate()
)
```

In this example, if all three thresholds are exceeded in a step, you'll see all three messages
printed, rather than just the critical one.

The default behavior (`highest_only=True`) helps prevent notification fatigue by limiting the number
of actions executed when multiple thresholds are exceeded in the same validation step. For example,
if a validation step fails with 60% of rows not passing, it would exceed 'warning', 'error', and
'critical' thresholds simultaneously. With `highest_only=True`, only the critical action would
execute.

You might want to set `highest_only=False` when:

- different threshold levels need to trigger different types of notifications (e.g., warnings to
Slack, errors to email, critical to urgent notifications)
- you need comprehensive logging of all severity levels for audit purposes
- you're building a dashboard that displays counts of issues at each severity level

## Using Multiple Actions for a Threshold

You can specify multiple actions to be executed for a single threshold level by providing a list:

```{python}
def send_notification():
    print("📧 Notification sent to data team")

def log_to_system():
    print("📝 Issue logged in system")

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(critical=0.15),

        # Set multiple actions for the critical threshold exceedance ---
        actions=pb.Actions(
            critical=[
                "CRITICAL: Data validation failed",  # First action: display message
                send_notification,                   # Second action: call function
                log_to_system                        # Third action: call another function
            ]
        ),
    )
    .col_vals_gt(columns="session_duration", value=15)
    .interrogate()
)
```

When providing a list of actions, they will be executed in sequence when the threshold is exceeded.
This allows you to combine different types of actions such as displaying messages, sending
notifications, and logging events.

## Final Actions

### Creating Final Actions

When you need to execute actions after all validation steps are complete, Pointblank provides the
`FinalActions` class. Unlike `Actions` which triggers on a per-step basis during the validation
process, `FinalActions` executes after the entire validation is complete, giving you a way to
respond to the overall validation results.

Here's how to use `FinalActions`:

```{python}
def send_alert():
    summary = pb.get_validation_summary()
    if summary["highest_severity"] == "critical":
        print(f"ALERT: Critical validation failures found in `{summary['tbl_name']}`")

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        tbl_name="game_revenue",
        thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),

        # Set final actions to be executed after all interrogations ---
        final_actions=pb.FinalActions(
            "Validation complete.",  # 1. a string message
            send_alert               # 2. a callable function
        )
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
    .col_vals_gt(columns="item_revenue", value=0.10)
    .interrogate()
)
```

In this example:

- We define the function `send_alert()` that checks the validation summary for critical failures
- We provide a simple string message `"Validation complete."` that will print to the console
- Both actions will execute in order after all validation steps have completed

Because the 'critical' threshold was exceeded in Step 2, we see the printed alert of `send_alert()`
after the simple string message.

`FinalActions` accepts any number of actions as positional arguments. Each argument can be:

1. **String**: A message to be displayed in the console
2. **Callable**: A function to be called with no arguments
3. **List of Strings/Callables**: Multiple actions to execute in sequence

All actions will be executed in the order they are provided after all validation steps have
completed.

### Using `get_validation_summary()`{.qd-no-link} in Final Actions

When creating a callable function to use with `FinalActions`, you can access information about the
overall validation results using the `get_validation_summary()` function. This gives you a
dictionary with comprehensive information about the validation:

```python
def comprehensive_report():
    summary = pb.get_validation_summary()
    print(f"Validation Report for {summary['tbl_name']}:")
    print(f"- Steps: {summary['n_steps']}")
    print(f"- Passing steps: {summary['n_passing_steps']}")
    print(f"- Failing steps: {summary['n_failing_steps']}")

    # Take additional actions based on results
    if summary["n_failing_steps"] > 0:

        # Create a Slack notification function ---
        notify = pb.send_slack_notification(
            webhook_url="https://hooks.slack.com/services/your/webhook/url",
            summary_msg="""
            🚨 *Validation Failure Alert*
            • Table: {tbl_name}
            • Failed Steps: {n_failing_steps} of {n_steps}
            • Highest Severity: {highest_severity}
            • Time: {time}
            """,
        )

        # Execute the notification function
        notify()

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        tbl_name="game_revenue",
        final_actions=pb.FinalActions(comprehensive_report),
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .interrogate()
)
```

```{python}
# | echo: false

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        tbl_name="game_revenue",
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .interrogate()
)
```

Here we used the `send_slack_notification()` function, which is available in Pointblank as a
pre-built action. It can be used by itself in `final_actions=` but here it's integrated into the
user's `comprehensive_report()` function to provide finer control with conditional logic.

### Combining Step-level and Final Actions

You can use both `Actions` and `FinalActions` together for comprehensive validation control:

```{python}
def log_step_failure():
    m = pb.get_action_metadata()
    print(f"Step {m['step']} failed with {m['level']}")


def generate_summary():
    summary = pb.get_validation_summary()
    # Sum up total failed test units across all steps
    total_failed = sum(summary["dict_n_failed"].values())
    # Sum up total test units across all steps
    total_units = sum(summary["dict_n"].values())
    print(f"Validation complete: {total_failed} failures out of {total_units} tests")

(
    pb.Validate(
        data=pb.load_dataset(dataset="game_revenue", tbl_type="duckdb"),
        thresholds=pb.Thresholds(warning=0.05, error=0.10),

        # Set an action for each step (highest threshold exceeded) ---
        actions=pb.Actions(default=log_step_failure),

        # Set a final action to get a summary of the validation process ---
        final_actions=pb.FinalActions(generate_summary),
    )
    .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
    .col_vals_gt(columns="item_revenue", value=0.05)
    .interrogate()
)
```

This approach allows you to:

1. log individual step failures during the validation process using `Actions`
2. generate a comprehensive report after all validation steps are complete using `FinalActions`

Using both action types gives you fine-grained control over when and how notifications and other
actions are triggered in your validation workflow.

## Conclusion

Actions provide a powerful mechanism for responding to data validation results in Pointblank. By
combining threshold settings with appropriate actions, you can create sophisticated data quality
workflows that:

- provide immediate feedback through console messages
- execute custom functions when validation thresholds are exceeded
- customize notifications based on severity levels
- generate comprehensive reports after validation is complete
- automate responses to data quality issues

The flexible design of `Actions` and `FinalActions` allows you to start simple with basic console
messages and gradually build up to complex validation workflows with conditional logic, custom
reporting, and integrations with other systems like Slack, email, or logging services.

When designing your validation strategy, consider leveraging both step-level actions for immediate
responses and final actions for holistic reporting. This combination provides comprehensive control
over your data validation process and helps ensure that data quality issues are detected, reported,
and addressed efficiently.


### Briefs

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

When validating data with Pointblank, it's often helpful to have descriptive labels for each
validation step. This is where *briefs* come in. A brief is a short description of what a validation
step is checking and it appears in the `STEP` column of the validation report table. Briefs make
your validation reports more readable and they help others understand what each step is verifying
without needing to look at the code.

Briefs can be set in two ways:

1. Globally: applied to all validation steps via the `brief=` parameter in `Validate`
2. Locally: set for individual validation steps via the `brief=` parameter in each validation method

Understanding these two approaches to adding briefs gives you flexibility in how you document your
validation process. Global briefs provide consistency across all steps and save time when you want
similar descriptions throughout, while step-level briefs allow for precise customization when
specific validations need more detailed or unique explanations. In practice, many validation
workflows will combine both approaches (i.e., setting a useful global brief template while
overriding it for steps that require special attention).

## Global Briefs

To set a global brief that applies to all validation steps, use the `Validate(brief=)` parameter
when creating a `Validate` object:

```{python}
import pointblank as pb
import polars as pl

# Sample data
data = pl.DataFrame({
    "id": [1, 2, 3, 4, 5],
    "value": [10, 20, 30, 40, 50],
    "category": ["A", "B", "C", "A", "B"]
})

# Create a validation with a global brief
(
    pb.Validate(
        data=data,

        # Global brief template ---
        brief="Step {step}: {auto}"
    )
    .col_vals_gt(columns="value", value=5)
    .col_vals_in_set(columns="category", set=["A", "B", "C"])
    .interrogate()
)
```

In this example, every validation step will have a brief description that follows the pattern
`"Step X: [auto-generated description]"`.

This is a simple example of template-based briefs. Later in this guide, we'll explore the full range
of templating elements available for creating custom brief descriptions that precisely communicate
what each validation step is checking.

## Step-level Briefs

You can also set briefs for individual validation steps:

```{python}
(
    pb.Validate(data=data)
    .col_vals_gt(
        columns="value", value=5,
        brief="Check if values exceed minimum threshold of 5"
    )
    .col_vals_in_set(
        columns="category", set=["A", "B", "C"],
        brief="Verify categories are valid"
    )
    .interrogate()
)
```

Local briefs override any global briefs that might be set.

## Brief Templating

Briefs support templating elements that get replaced with specific values:

- `{auto}`: an auto-generated description of the validation
- `{step}`: the step number in the validation plan
- `{col}`: the column name(s) being validated
- `{value}`: the comparison value used in the validation (when applicable)
- `{thresholds}`: a short summary of thresholds levels set (or unset) for the step
- `{segment}`, `{segment_column}`, `{segment_value}`: information on the step's segment

Here's how to use these templates:

```{python}
(
    pb.Validate(data=data)
    .col_vals_gt(
        columns="value", value=5,
        brief="Step {step}: Checking column '{col}' for values `> 5`"
    )
    .col_vals_in_set(
        columns="category", set=["A", "B", "C"],
        brief="{auto} **(Step {step})**"
    )
    .interrogate()
)
```

These template elements make briefs highly flexible and customizable. You can combine multiple
templating elements in a single brief to create descriptive yet concise validation step
descriptions. The templates help maintain consistency across your validation reports while providing
enough detail to understand what each step is checking.

Note that not all templating elements will be relevant for every validation step. For instance,
`{value}` is only applicable to validation functions that hold a comparison value like
[`Validate.col_vals_gt()`](`Validate.col_vals_gt`). If you include a templating element that isn't relevant to a particular
step, it will not be replaced with a corresponding value.

Briefs support the use of Markdown formatting, allowing you to add emphasis with **bold** or
_italic_ text, include `inline code` formatting, or other Markdown elements to make your briefs more
visually distinctive and informative. This can be especially helpful when you want certain parts of
your briefs to stand out in the validation report.

## Automatic Briefs

If you want Pointblank to generate briefs for you automatically, you can set `brief=True`. Here,
we'll make that setting at the global level (by using `Validate(brief=True)`):

```{python}
(
    pb.Validate(
        data=data,

        # Setting for automatically generated briefs ---
        brief=True
    )
    .col_vals_gt(columns="value", value=5)
    .col_vals_in_set(columns="category", set=["A", "B", "C"])
    .interrogate()
)
```

Automatic briefs are descriptive and include information about what's being validated, including the
column names and the validation conditions.

## Briefs Localized to a Specified Language

When using the `lang=` parameter in `Validate`, automatically generated briefs will be created in
the specified language (along with other elements of the validation report table):

```{python}
(
    pb.Validate(
        data=data,

        # Setting the language as Spanish ---
        lang="es",

        # Automatically generate all briefs in Spanish
        brief=True
    )
    .col_vals_gt(columns="value", value=5)
    .col_vals_in_set(columns="category", set=["A", "B", "C"])
    .interrogate()
)
```

When using the `lang=` parameter in combination with the `{auto}` templating element, the
auto-generated portion of the brief will also be translated to the specified language. This makes it
possible to create fully localized validation reports where both custom text and auto-generated
descriptions appear in the same language.

Pointblank supports several languages for localized briefs, including French (`"fr"`), German
(`"de"`), Spanish (`"es"`), Italian (`"it"`), and Portuguese (`"pt"`). For the complete list of
supported languages, refer to the `Validate` documentation.

## Disabling Briefs

If you've set a global brief but want to disable it for specific validation steps, you can set
`brief=False`:

```{python}
(
    pb.Validate(
        data=data,

        # Global brief template ---
        brief="Step {step}: {auto}"
    )
    .col_vals_gt(columns="value", value=5)  # This step uses the global brief setting
    .col_vals_in_set(
        columns="category",
        set=["A", "B", "C"],

        # No brief for this step ---
        brief=False
    )
    .interrogate()
)
```

## Practical Example: Comprehensive Validation with Briefs

In real-world data validation scenarios, you'll likely work with more complex datasets and apply
various types of validation checks. This final example brings together many of the brief-generating
techniques we've covered, showing how you can mix different approaches in a single validation
workflow.

```{python}
# Create a slightly larger dataset
data_2 = pl.DataFrame({
    "id": [1, 2, 3, 4, 5, 6, 7, 8],
    "value": [10, 20, 30, 40, 50, 60, 70, 80],
    "ratio": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
    "category": ["A", "B", "C", "A", "B", "C", "A", "B"],
    "date": ["2023-01-01", "2023-01-02", "2023-01-03", "2023-01-04",
             "2023-01-05", "2023-01-06", "2023-01-07", "2023-01-08"]
})

(
    pb.Validate(data=data_2)
    .col_vals_gt(
        columns="value", value=0,

        # Plaintext brief ---
        brief="All values must be positive."
    )
    .col_vals_between(
        columns="ratio", left=0, right=1,

        # Template-based brief ---
        brief="**Step {step}**: Ratios should be between `0` and `1`."
    )
    .col_vals_in_set(
        columns="category", set=["A", "B", "C"],

        # Automatically generated brief ---
        brief=True
    )
    .interrogate()
)
```

The example above demonstrates:

- plaintext briefs with direct messages
- template-based briefs with Markdown formatting
- automatically generated briefs (`brief=True`)

By combining these different brief styles, you can create validation reports that are informative,
consistent, and tailored to your specific data quality requirements.

## Best Practices for Using Briefs

Well-crafted briefs can significantly enhance the readability and usefulness of your validation
reports. Here are some guidelines to follow:

1. Be concise: briefs should be short and to the point; they're meant to quickly communicate the
purpose of a validation step

2. Be specific: include relevant details or conditions that make the validation meaningful

3. Use templates consistently: if you're using template elements like `"{step}"` or `"{col}"`, try
to use them consistently across all briefs for a cleaner look

4. Use auto-generated briefs as a starting point: you can start with `Validate(brief=True)` to see
what Pointblank generates automatically, then customize as needed

5. Add custom briefs for complex validations: custom briefs are especially useful for complex
validations where the purpose might not be immediately obvious from the code

Following these best practices will help ensure your validation reports are easy to understand for
everyone who needs to review them.

## Conclusion

Briefs help make validation reports more readable and understandable. By using global briefs,
step-level briefs, or a combination of both, you can create validation reports that clearly
communicate what each validation step is checking.

Whether you want automatically generated descriptions or precisely tailored custom messages, the
brief system provides the flexibility to make your data validation work more transparent and easier
to interpret for all stakeholders.


### Expression-Based Validation

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

While Pointblank offers many specialized validation functions for common data quality checks,
sometimes you need more flexibility for complex validation requirements. This is where
expression-based validation with [`Validate.col_vals_expr()`](`Validate.col_vals_expr`) comes in.

The [`Validate.col_vals_expr()`](`Validate.col_vals_expr`) method allows you to:

- combine multiple conditions in a single validation step
- access row-wise values across multiple columns

Now let's explore how to use these capabilities through a collection of examples!

## Basic Usage

At its core, [`Validate.col_vals_expr()`](`Validate.col_vals_expr`) validates whether an expression evaluates to `True` for
each row in your data. Here's a simple example:

```{python}
import pointblank as pb
import polars as pl

# Load small_table dataset as a Polars DataFrame
small_table_pl = pb.load_dataset(dataset="small_table", tbl_type="polars")

(
    pb.Validate(data=small_table_pl)
    .col_vals_expr(

        # Use Polars expression syntax ---
        expr=pl.col("d") > pl.col("a") * 50,
        brief="Column `d` should be at least 50 times larger than `a`."
    )
    .interrogate()
)
```

In this example, we're validating that for each row, the value in column `d` is at least 50 times
larger than the value in column `a`.

## Notes on Expression Syntax

The expression syntax depends on your table type:

- **Polars**: uses Polars expression syntax with `pl.col("column_name")`
- **Pandas**: uses standard Python/NumPy syntax

The expression should:

- evaluate to a boolean result for each row
- reference columns using the appropriate syntax for your table type
- use standard operators (`+`, `-`, `*`, `/`, `>`, `<`, `==`, etc.)
- not include assignments

## Complex Expressions

The real power of [`Validate.col_vals_expr()`](`Validate.col_vals_expr`) comes with complex expressions that would be
difficult to represent using the standard validation functions:

```{python}
# Load game_revenue dataset as a Polars DataFrame
game_revenue_pl = pb.load_dataset(dataset="game_revenue", tbl_type="polars")

(
    pb.Validate(data=game_revenue_pl)
    .col_vals_expr(

        # Use Polars expression syntax ---
        expr=(pl.col("session_duration") > 20) | (pl.col("item_revenue") > 10),
        brief="Sessions should be either long (>20 min) or high-value (>$10)."
    )
    .interrogate()
)
```

This validates that either the session duration is longer than 20 minutes OR the item revenue is
greater than $10.

## Example: Multiple Conditions

You can create sophisticated validations with multiple conditions:

```{python}
# Create a simple Polars DataFrame
employee_df = pl.DataFrame({
    "age": [25, 30, 15, 40, 35],
    "income": [50000, 75000, 0, 100000, 60000],
    "years_experience": [3, 8, 0, 15, 7]
})

(
    pb.Validate(data=employee_df, tbl_name="employee_data")
    .col_vals_expr(

        # Complex condition with multiple comparisons ---
        expr=(
            (pl.col("age") >= 18) &
            (pl.col("income") / (pl.col("years_experience") + 1) <= 25000)
        ),
        brief="Adults should have reasonable income-to-experience ratios."
    )
    .interrogate()
)
```

## Example: Handling Null Values

When working with expressions, consider how to handle null/missing values:

```{python}
(
    pb.Validate(data=small_table_pl)
    .col_vals_expr(

        # Check for nulls before division ---
        expr=(pl.col("c").is_not_null()) & ((pl.col("c") / pl.col("a")) > 1.5),
        brief="Ratio of `c`/`a` should exceed 1.5 (when `c` is not null)."
    )
    .interrogate()
)
```

## Best Practices

Here are some tips and tricks for effectively using expression-based validation with
[`Validate.col_vals_expr()`](`Validate.col_vals_expr`).

### Document Your Expressions

Always provide clear documentation in the `brief=` parameter:

```{python}
(
    pb.Validate(data=small_table_pl)
    .col_vals_expr(
        expr=pl.col("d") > pl.col("a") * 1.5,

        # Document which columns are being compared ---
        brief="Column `d` should be at least 1.5 times larger than column `a`."
    )
    .interrogate()
)
```

### Handle Edge Cases

Consider potential edge cases like division by zero or nulls:

```{python}
(
    pb.Validate(data=small_table_pl)
    .col_vals_expr(

        # Check denominator before division ---
        expr=(pl.col("a") != 0) & (pl.col("d") / pl.col("a") > 1.5),
        brief="Ratio of `d`/`a` should exceed 1.5 (avoiding division by zero)."
    )
    .interrogate()
)
```

### Test on Small Datasets First

When developing complex expressions, test on a small sample of your data first to ensure your logic
is correct before applying it to large datasets.

## Conclusion

The [`Validate.col_vals_expr()`](`Validate.col_vals_expr`) method provides a powerful way to implement complex validation
logic in Pointblank when standard validation methods aren't sufficient. By leveraging expressions,
you can create sophisticated data quality checks tailored to your specific requirements, combining
conditions across multiple columns and applying transformations as needed.

This flexibility makes expression-based validation an essential tool for addressing complex data
quality scenarios in your validation workflows.


### Schema Validation

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Schema validation in Pointblank allows you to verify that your data conforms to an expected
structure and type specification. This is particularly useful when ensuring data consistency across
systems or validating incoming data against predefined requirements.

Let's first look at the dataset we'll use for the first example:

```{python}
import pointblank as pb

# Preview the small_table dataset we'll use throughout this guide
pb.preview(pb.load_dataset(dataset="small_table", tbl_type="polars"))
```

## Schema Definition and Validation

A schema in Pointblank is created using the `Schema` class which defines the expected structure of a
table. Once created, you apply schema validation through the [`Validate.col_schema_match()`](`Validate.col_schema_match`)
validation step.

```{python}
# Create a schema definition matching small_table structure
schema = pb.Schema(
    columns=[
        ("date_time",),   # Only check column name
        ("date",),        # Only check column name
        ("a", "Int64"),   # Check name and type
        ("b", "String"),  # Check name and type
        ("c", "Int64"),   # Check name and type
        ("d", "Float64"), # Check name and type
        ("e", "Boolean"), # Check name and type
        ("f",),           # Only check column name
    ]
)

# Validate the small_table against the schema
small_table_validation = (
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        label="Schema validation of `small_table`.",
    )
    .col_schema_match(schema=schema)
    .interrogate()
)

small_table_validation
```

The output shows the validation passed successfully. When all columns have the correct names and
types as specified in the schema, the validation passes with a single passing test unit. If there
were discrepancies, this would fail, but the basic output wouldn't show specific issues.

For detailed information about validation results, use [`Validate.get_step_report()`](`Validate.get_step_report`):

```{python}
small_table_validation.get_step_report(i=1)
```

The step report provides specific details about which columns were checked and whether they matched
the schema, helping diagnose issues when validation fails.

## Schema Components and Column Types

When defining a schema, you need to specify column names and optionally their data types. By
default, Pointblank enforces strict validation where:

- all columns in your table must match the specified schema
- column order must match the schema
- column types are case-sensitive
- type names must match exactly

The schema definition accepts column types as string representations, which vary depending on your
data source:

- `string`: Character data (may also be `"String"`, `"varchar"`, `"character"`, etc.)
- `integer`: Integer values (may also be `"Int64"`, `"int"`, `"bigint"`, etc.)
- `numeric`: Numeric values including integers and floating-point numbers (may also be `"Float64"`,
`"double"`, `"decimal"`, etc.)
- `boolean`: Logical values (`True`/`False`) (may also be `"Boolean"`, `"bool"`, etc.)
- `datetime`: Date and time values (may also be `"Datetime"`, `"timestamp"`, etc.)
- `date`: Date values (may also be `"Date"`, etc.)
- `time`: Time values

For specific database engines or DataFrame libraries, you may need to use their exact type names
(like `"VARCHAR(255)"` for SQL databases or `"Int64"` for Polars integers).

## Discovering Column Types

To easily determine the correct type string for columns in your data, Pointblank provides two
helpful functions:

```{python}
import polars as pl
from datetime import date

# Define a sample dataframe
sample_df = pl.DataFrame({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Charlie"],
    "join_date": [date(2020, 1, 1), date(2021, 3, 15), date(2022, 7, 10)]
})
```

```{python}
# Method 1: Using `preview()` with `show_types=True` to see column types
pb.preview(sample_df)
```

```{python}
# Method 2: Using `col_summary_tbl()` which shows column types and other details
pb.col_summary_tbl(sample_df)
```

These functions help you identify the exact type strings to use in your schema definitions,
eliminating guesswork and ensuring compatibility with your data source.

## Creating a Schema

You can create a schema in four different ways, each with its own advantages. All schema objects
can be printed to display their column names and data types.

### 1. Using a List of Tuples with `columns=`

This approach allows for mixed validation: some columns checked for both name and type, others only
for name:

```{python}
schema_tuples = pb.Schema(

    # List of tuples approach: flexible for mixed type/name checking ---
    columns=[
        ("name", "String"), # Check name and type
        ("age", "Int64"),   # Check name and type
        ("height",)         # Check name only
    ]
)

print(schema_tuples)
```

This is the only method that allows checking just column names for some columns while checking both
names and types for others.

### 2. Using a Dictionary with `columns=`

This approach is often the most readable when defining a schema manually, especially for larger
schemas:

```{python}
schema_dict = pb.Schema(

    # Dictionary approach (keys are column names, values are data types) ---
    columns={
        "name": "String",
        "age": "Int64",
        "height": "Float64",
        "created_at": "Datetime"
    }
)

print(schema_dict)
```

With this method, you must always provide both column names (as keys) and their types (as values).

### 3. Using Keyword Arguments

For more readable code with a small number of columns:

```{python}
schema_kwargs = pb.Schema(

    # Keyword arguments approach (more readable for simple schemas) ---
    name="String",
    age="Int64",
    height="Float64"
)

print(schema_kwargs)
```

Like the dictionary method, this approach requires both column names and types.

### 4. Extracting from an Existing Table with `tbl=`

You can automatically extract a schema from an existing table:

```{python}
import polars as pl

# Create a sample dataframe
df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "height": [5.6, 6.0, 5.8]
})

# Extract schema from table
schema_from_table = pb.Schema(tbl=df)

print(schema_from_table)
```

This is especially useful when you want to validate that future data matches the structure of a
reference dataset.

## Multiple Data Types for a Column

You can specify multiple acceptable types for a column by providing a list of types:

```{python}
# Schema with multiple possible types for a column
schema_multi_types = pb.Schema(
    columns={
        "name": "String",
        "age": ["Int64", "Float64"],  # Accept either integer or float
        "active": "Boolean"
    }
)

print(schema_multi_types)
```

This is useful when working with data sources that might represent the same information in different
ways (e.g., integers sometimes stored as floats).

## Schema Validation Options

When using `col_schema_match()`, you can customize validation behavior with several important
options:

| Option | Default | Description |
|--------|---------|-------------|
| `complete` | `True` | Require exact column presence (no extra columns allowed) |
| `in_order` | `True` | Enforce column order |
| `case_sensitive_colnames` | `True` | Make column name matching case-sensitive |
| `case_sensitive_dtypes` | `True` | Make data type matching case-sensitive |
| `full_match_dtypes` | `True` | Require exact (not partial) type name matches |

### Controlling Column Presence

By default, [`Validate.col_schema_match()`](`Validate.col_schema_match`) requires a complete match between the schema's columns
and the table's columns. You can make this more flexible:

```{python}
# Create a sample table
users_table_extra = pl.DataFrame({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "extra_col": ["a", "b", "c"]  # Extra column not in schema
})

# Create a schema
schema = pb.Schema(
    columns={"id": "Int64", "name": "String", "age": "Int64"}
)

# Validate without requiring all columns to be present
validation = (
    pb.Validate(data=users_table_extra)
    .col_schema_match(
        schema=schema,

        # Allow schema columns to be a subset ---
        complete=False
    )
    .interrogate()
)

validation.get_step_report(i=1)
```

### Column Order Enforcement

You can control whether column order matters in your validation:

```{python}
# Create a sample table
users_table = pl.DataFrame({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
})

# Create a schema
schema = pb.Schema(
    columns={"name": "String", "age": "Int64", "id": "Int64"}
)

# Validate without enforcing column order
validation = (
    pb.Validate(data=users_table)
    .col_schema_match(
        schema=schema,

        # Don't enforce column order ---
        in_order=False
    )
    .interrogate()
)

validation.get_step_report(i=1)
```

### Case Sensitivity

Control whether column names and data types are case-sensitive:

```{python}
# Create schema with different case charactistics
case_schema = pb.Schema(
    columns={"ID": "int64", "NAME": "string", "AGE": "int64"}
)

# Create validation with case-insensitive column names and types
validation = (
    pb.Validate(data=users_table)
    .col_schema_match(
        schema=case_schema,

        # Ignore case in column names ---
        case_sensitive_colnames=False,

        # Ignore case in data type names ---
        case_sensitive_dtypes=False
    )
    .interrogate()
)

validation.get_step_report(i=1)
```

### Type Matching Precision

Control how strictly data types must match:

```{python}
# Create schema with simplified type names
type_schema = pb.Schema(

    # Using simplified type names ---
    columns={"id": "int", "name": "str", "age": "int"}
)

# Allow partial type matches
validation = (
    pb.Validate(data=users_table)
    .col_schema_match(
        schema=type_schema,

        # Ignore case in data type names ---
        case_sensitive_dtypes=False,

        # Allow partial type name matches ---
        full_match_dtypes=False
    )
    .interrogate()
)

validation.get_step_report(i=1)
```

## Common Schema Validation Patterns

This section explores common patterns for applying schema validation to different scenarios. Each
pattern addresses specific validation needs you might encounter when working with real-world data.
We'll examine the step reports ([`Validate.get_step_report()`](`Validate.get_step_report`)) for these validations since they
provide more detailed information about what was checked and how the validation performed, offering
an intuitive way to understand the results beyond simple pass/fail indicators.

### Structural Validation Only

When you only care about column names but not their types:

```{python}
# Create a schema with only column names
structure_schema = pb.Schema(
    columns=["id", "name", "age", "extra_col"]
)

# Validate structure only
validation = (
    pb.Validate(data=users_table_extra)
    .col_schema_match(schema=structure_schema)
    .interrogate()
)

validation.get_step_report(i=1)
```

### Mixed Validation

Validate types for critical columns but just presence for others:

```{python}
# Mixed validation for different columns
mixed_schema = pb.Schema(
    columns=[
        ("id", "Int64"),     # Check name and type
        ("name", "String"),  # Check name and type
        ("age",),            # Check name only
        ("extra_col",)       # Check name only
    ]
)

# Validate with mixed approach
validation = (
    pb.Validate(data=users_table_extra)
    .col_schema_match(schema=mixed_schema)
    .interrogate()
)

validation.get_step_report(i=1)
```

### Progressive Schema Evolution

As your data evolves, you might need to adapt your validation approach:

```{python}
# Original schema
original_schema = pb.Schema(
    columns={
        "id": "Int64",
        "name": "String"
    }
)

# New data with additional columns
evolved_data = pl.DataFrame({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],           # New column
    "active": [True, False, True]  # New column
})

# Validate with flexible parameters
validation = (
    pb.Validate(evolved_data)
    .col_schema_match(
        schema=original_schema,

        # Allow extra columns ---
        complete=False,

        # Don't enforce order ---
        in_order=False
    )
    .interrogate()
)

validation.get_step_report(i=1)
```

## Integrating with Larger Validation Workflows

Schema validation is often just one part of a comprehensive data validation strategy. You can
combine schema checks with other validation steps:

```{python}
# Define a schema
schema = pb.Schema(
    columns={
        "id": "Int64",
        "name": "String",
        "age": "Int64"
    }
)

# Create a validation plan
validation = (
    pb.Validate(
        users_table,
        label="User data validation",
        thresholds=pb.Thresholds(warning=0.05, error=0.1)
    )

    # Add schema validation ---
    .col_schema_match(schema=schema)

    # Add other validation steps ---
    .col_vals_not_null(columns="id")
    .col_vals_gt(columns="age", value=26)
    .interrogate()
)

validation
```

This approach allows you to first validate the structure of your data and then check specific
business rules or constraints.

## Best Practices

1. Define schemas early: document and define expected data structures early in your data
workflow.

2. Choose the right creation method:
   - use `columns=<dict>` for readability with many columns
   - use `columns=<list of tuples>` for mixed name/type validation
   - use `kwargs` for small schemas with simple column names
   - use `tbl=` to extract schemas from reference datasets

3. Be deliberate about strictness: choose validation parameters based on your specific needs:
   - strict validation (`complete=True`) for critical data interfaces
   - flexible validation (`complete=False`, `in_order=False`) for evolving datasets

4. Reuse schemas: create schema definitions that can be reused across multiple validation
contexts.

5. Version control schemas: as your data evolves, maintain versions of your schemas to track
changes.

6. Extract schemas from reference data: when you have a 'golden' dataset that represents your
ideal structure, use `Schema(tbl=reference_data)` to extract its schema.

7. Consider type flexibility: use multiple types per column (`["Int64", "Float64"]`) when
working with data from diverse sources.

8. Combine with targeted validation: use schema validation for structural checks and add
specific validation steps for business rules.

## Conclusion

Schema validation provides a powerful mechanism for ensuring your data adheres to expected
structural requirements. It serves as an excellent first line of defense in your data validation
strategy, verifying that the data you're working with has the expected shape before applying more
detailed business rule validations.

The `Schema` class offers multiple ways to define schemas, from manual specification with
dictionaries or keyword arguments to automatic extraction from reference tables. When combined with
the flexible options of [`Validate.col_schema_match()`](`Validate.col_schema_match`), you can implement validation approaches
ranging from strict structural enforcement to more flexible evolution-friendly checks.

By understanding the different schema creation methods and validation options, you can efficiently
validate the structure of your data tables and ensure they meet your requirements before processing.


### Assertions

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

In addition to validation steps that create reports, Pointblank provides **assertions**. This is a
lightweight way to confirm data quality by raising exceptions when validation conditions aren't met.
Assertions are particularly useful in:

- data processing pipelines where you need to halt execution if data doesn't meet expectations
- testing environments where you want to verify data properties programmatically
- scripts and functions where you need immediate notification of data problems

## Basic Assertion Workflow

The assertion workflow uses your familiar validation steps with assertion methods to check that
validations meet your requirements:

```{python}
import pointblank as pb
import polars as pl

# Create sample data
sample_data = pl.DataFrame({
    "id": [1, 2, 3, 4, 5],
    "value": [10.5, 8.3, -2.1, 15.7, 7.2]
})

# Create a validation plan and assert that all steps pass
(
    pb.Validate(data=sample_data)
    .col_vals_gt(columns="id", value=0, brief="IDs must be positive")
    .col_vals_gt(columns="value", value=-5, brief="Values should exceed -5")

    # Will automatically `interrogate()` and raise an AssertionError if any validation fails ---
    .assert_passing()
)
```

This simple pattern allows you to integrate data quality checks into your data pipelines. With it,
you can create clear stopping points when data doesn't meet specified criteria.

## Assertion Methods

Pointblank offers two types of assertions:

1. Full Passing Assertions: using [`Validate.assert_passing()`](`Validate.assert_passing`) to verify that every single test
unit passes
2. Threshold-Based Assertions: using [`Validate.assert_below_threshold()`](`Validate.assert_below_threshold`) to verify that failure
rates stay within acceptable thresholds

### `assert_passing()`

The [`Validate.assert_passing()`](`Validate.assert_passing`) method is the strictest form of assertion, requiring every single
validation test unit to pass:

```{python}
try:
    (
        pb.Validate(data=sample_data)
        .col_vals_gt(columns="value", value=0)

        # Direct assertion: automatically interrogates ---
        .assert_passing()
    )
except AssertionError as e:
    print("AssertionError:", str(e))
```

### `assert_below_threshold()`

The [`Validate.assert_below_threshold()`](`Validate.assert_below_threshold`) method is more flexible as it allows some failures as long
as they stay below specified threshold levels. Pointblank uses three severity thresholds that
increase in order of seriousness:

- **'warning'** (least severe): the first threshold that gets triggered when failures exceed this
level
- **'error'** (more severe): the middle threshold indicating more serious data quality issues
- **'critical'** (most severe): the highest threshold indicating critical data quality problems

```{python}
# Create a two-column DataFrame for this example
tbl_pl = pl.DataFrame({
    "a": [4, 6, 9, 7, 12, 8, 7, 12, 10, 7],
    "b": [9, 8, 10, 5, 10, 9, 14, 6, 6, 8],

})

# Set thresholds: warning=0.2 (20%), error=0.3 (30%), critical=0.4 (40%)
validation = (
    pb.Validate(data=tbl_pl, thresholds=(0.2, 0.3, 0.4))
    .col_vals_gt(columns="b", value=5)   # 1/10 failing (10% failure rate)
    .col_vals_lt(columns="a", value=11)  # 2/10 failing (20% failure rate)
    .col_vals_ge(columns="b", value=8)   # 3/10 failing (30% failure rate)
    .interrogate()
)

validation
```

The validation report above visually indicates threshold levels with colored circles:

- gray circles in the `W` column indicate the 'warning' threshold
- yellow circles in the `E` column indicate the 'error' threshold
- red circles in the `C` column indicate the 'critical' threshold

This won't pass the [`Validate.assert_below_threshold()`](`Validate.assert_below_threshold`) assertion for the 'error' level because
step 3 exceeds this threshold (30% failure rate matches the error threshold):

```{python}
try:
    validation.assert_below_threshold(level="error")
except AssertionError as e:
    print("AssertionError:", str(e))
```

We can check against the 'error' threshold for specific steps with the `i=` parameter:

```{python}
validation.assert_below_threshold(level="error", i=[1, 2])
```

This passes because the highest threshold exceeded in steps 1 and 2 is 'warning'.

The [`Validate.assert_below_threshold()`](`Validate.assert_below_threshold`) method takes these parameters:

- `level=`: threshold level to check against (`"warning"`, `"error"`, or `"critical"`)
- `i=`: optional specific step number(s) to check
- `message=`: optional custom error message

This is particularly useful when:

- working with real-world data where some percentage of failures is acceptable
- implementing different severity levels for data quality rules
- gradually improving data quality with stepped thresholds

::: {.callout-note}
Assertion methods like [`Validate.assert_passing()`](`Validate.assert_passing`) and [`Validate.assert_below_threshold()`](`Validate.assert_below_threshold`)
will automatically call [`Validate.interrogate()`](`Validate.interrogate`) if needed, so you don't have to explicitly
include this step when using assertions directly.
:::

## Using Status Check Methods

In addition to assertion methods that raise exceptions, Pointblank provides status check methods
that return boolean values:

### `all_passed()`

The [`Validate.all_passed()`](`Validate.all_passed`) method will return `True` only if every single test unit in every
validation step passed:

```{python}
validation = (
    pb.Validate(data=sample_data)
    .col_vals_gt(columns="value", value=0)
    .interrogate()
)

if not validation.all_passed():
    print("Validation failed: some values are not positive")
```

### `warning()`, `error()`, and `critical()`

The methods [`Validate.warning()`](`Validate.warning`), [`Validate.error()`](`Validate.error`), and [`Validate.critical()`](`Validate.critical`) all return
information about whether validation steps exceeded that specific threshold level.

While assertion methods raise exceptions to halt execution when thresholds are exceeded, these
status methods give you fine-grained control to implement custom logic based on different validation
quality levels.

```{python}
validation = (
    pb.Validate(data=sample_data, thresholds=(0.05, 0.10, 0.20))
    .col_vals_gt(columns="value", value=0)  # Some values are negative
    .interrogate()
)

validation
```

The [`Validate.warning()`](`Validate.warning`) method returns a dictionary mapping step numbers to boolean values. A
`True` value means that step exceeds the warning threshold:

```{python}
# Get dictionary of warning status for each step
warning_status = validation.warning()
print(f"Warning status: {warning_status}")  # {1: True} means step 1 exceeds warning threshold
```

You can check a specific step using the `i=` parameter, and get a single boolean with `scalar=True`:

```{python}
# Check error threshold for specific step
has_errors = validation.error(i=1, scalar=True)

if has_errors:
    print("Step 1 exceeded the error threshold.")
```

Similarly, we can check if any steps exceed the 'critical' threshold:

```{python}
# Check against critical threshold
critical_status = validation.critical()
print(f"Critical status: {critical_status}")
```

These methods are particularly useful for:

1. Conditional logic: taking different actions based on threshold severity
2. Reporting: generating summary reports about validation quality
3. Monitoring: tracking data quality trends over time
4. Graceful degradation: implementing fallback logic when quality decreases

Each method has these options:

- without parameters: returns a dictionary mapping step numbers to boolean status values
- with `i=`: check specific step(s)
- with `scalar=True`: return a single boolean instead of a dictionary (when checking a specific
step)

While assertion methods raise exceptions to halt execution when thresholds are exceeded, these
methods give you fine-grained control to implement custom logic based on different validation
quality levels.

## Customizing Error Messages

You can provide custom error messages when assertions fail to make them more meaningful in your
specific workflow context:

```{python}
# Create a validation with potential failures
validation = (
    pb.Validate(data=sample_data, thresholds=(0.2, 0.3, 0.4))
    .col_vals_gt(columns="value", value=0)
    .interrogate()
)

# Display the validation results
validation
```

When you need to customize the error message that appears when an assertion fails, use the
`message=` parameter:

```{python}
try:
    # Custom message for threshold assertion
    validation.assert_below_threshold(
        level="warning",
        message="Data quality too low for processing!"
    )
except AssertionError as e:
    print(f"Custom handling of failure: {e}")
```

Descriptive error messages are essential in production systems where multiple team members might
need to interpret validation failures. The custom message lets you provide context appropriate to
your specific workflow or data pipeline stage.

## Combining Assertions with Actions

Actions and assertions serve complementary but distinct purposes in data validation workflows:

- Actions trigger during validation but shouldn't raise errors (as this would halt report
generation)
- Assertions are designed to raise errors based on specific conditions, making them ideal for
flow control after validation completes

Here's a simplified example showing how to use them together. The print statements simulate logging
or monitoring that would be valuable in production data pipelines:

```{python}
# Define a simple action function (won't raise errors)
def notify_quality_issue(message="Data quality issue detected"):
    print(f"ACTION TRIGGERED: {message}")

# Create data with known failures
problem_data = pl.DataFrame({
    "id": [1, 2, 3, -4, 5],  # One negative ID
    "value": [10.5, 8.3, -2.1, 15.7, 7.2]  # One negative value
})

# First use actions for automated responses during validation
print("Running validation with actions...")
validation = (
    pb.Validate(data=problem_data, thresholds=(0.1, 0.2, 0.3))
    .col_vals_gt(
        columns="id", value=0,
        brief="IDs must be positive",
        actions=pb.Actions(warning=notify_quality_issue)
    )
    .interrogate()  # Actions trigger here but won't stop report generation
)

# Then use assertions after validation for workflow control
print("\nNow using assertion for flow control...")
try:
    validation.assert_below_threshold(level="warning")
    print("This line won't execute if the assertion fails")
except AssertionError as e:
    print(f"Validation failed threshold check: {e}")
    print("Implementing fallback process...")
```

This approach gives you the best of both worlds:

- Actions provide immediate notification during validation without interrupting the process
- Assertions control workflow execution after validation when important thresholds are exceeded

This pattern works well in data pipelines where you want both: (1) automated responses during
validation and (2) clear decision points after validation is complete.

## Best Practices for Assertions

When using assertions in your data workflows, consider these best practices:

1. **Choose the right assertion type**:
   - use [`Validate.assert_passing()`](`Validate.assert_passing`) for critical validations where any failure is unacceptable
   - use [`Validate.assert_below_threshold()`](`Validate.assert_below_threshold`) for validations where some failure rate is acceptable

2. **Set appropriate thresholds** that match your data quality requirements:
   ```python
   # Example threshold strategy
   validation = pb.Validate(
       data=sample_data,
       # warning at 1%, error at 5%, critical at 10%
       thresholds=pb.Thresholds(warning=0.01, error=0.05, critical=0.10)
   )
   ```

3. **Use a graduated approach** to validation severity:
   ```python
   # Critical validations: must be perfect
   validation_1.assert_passing()

   # Important validations: must be below error threshold
   validation_2.assert_below_threshold(level="error")

   # Monitor-only validations: check warning status
   warning_status = validation_3.warning()
   ```

4. **Placement in pipelines**: place assertions at critical points where data quality is essential

5. **Error handling**: wrap assertions in try-except blocks for better error handling in production
systems

6. **Combine with reporting**: use both assertions and reporting approaches for comprehensive
quality control

## Conclusion

Pointblank's assertion methods give you flexible options for enforcing data quality requirements:

- [`Validate.assert_passing()`](`Validate.assert_passing`) for strict validation where every test unit must pass
- [`Validate.assert_below_threshold()`](`Validate.assert_below_threshold`) for more flexible validation where some failures are
tolerable
- Status methods ([`Validate.warning()`](`Validate.warning`), [`Validate.error()`](`Validate.error`), and [`Validate.critical()`](`Validate.critical`)) for
programmatic threshold checking

By using these assertion methods appropriately, you can build robust data pipelines with different
levels of quality enforcement (from strict validation of critical data properties to more lenient
checks for less critical aspects). This graduated approach to data quality helps create systems that
are both reliable and practical in real-world data environments.


### Draft Validation

Draft validation in Pointblank leverages large language models (LLMs) to automatically generate
validation plans for your data. This feature is especially useful when starting validation on a new
dataset or when you need to quickly establish baseline validation coverage.

The `DraftValidation` class connects to various LLM providers to analyze your data's characteristics
and generate a complete validation plan tailored to its structure and content.

## How `DraftValidation`{.qd-no-link} Works

When you use `DraftValidation`, the process works through these steps:

1. a statistical summary of your data is generated using the `DataScan` class
2. this summary is converted to JSON format and sent to your selected LLM provider
3. the LLM uses the summary along with knowledge about Pointblank's validation capabilities to
generate a validation plan
4. the result is returned as executable Python code that you can use directly or modify as needed

The entire process happens without sending all of the data to the LLM provider, but only a summary
that includes column names, data types, basic statistics, and a small sample of values.

## Requirements and Setup

To use the `DraftValidation` feature, you'll need:

1. an API key from a supported LLM provider
2. the required Python packages installed

You can install all necessary dependencies with:

```bash
pip install pointblank[generate]
```

This will install the `chatlas` package and other dependencies required for `DraftValidation`.

### Supported LLM Providers

The `DraftValidation` class supports multiple LLM providers:

- **Anthropic** (Claude models)
- **OpenAI** (GPT models)
- **Ollama** (local LLMs)
- **Amazon Bedrock** (AWS-hosted models)
- **Azure OpenAI** (OpenAI models deployed on Azure)

Each provider has different capabilities and performance characteristics, but all can be used to
generate validation plans through a consistent interface.

## Basic Usage

The simplest way to use `DraftValidation` is to provide your data and specify an LLM model. Let's
try it out with the `global_sales` dataset.

```python
import pointblank as pb

# Load a dataset
data = pb.load_dataset(dataset="global_sales", tbl_type="polars")

# Generate a validation plan
pb.DraftValidation(
    data=data,
    model="anthropic:claude-opus-4-6",
    api_key="your_api_key_here"  # Replace with your actual API key
)
```

````plaintext
```python
import pointblank as pb

# Define schema based on column names and dtypes
schema = pb.Schema(columns=[
    ("product_id", "String"),
    ("product_category", "String"),
    ("customer_id", "String"),
    ("customer_segment", "String"),
    ("region", "String"),
    ("country", "String"),
    ("city", "String"),
    ("timestamp", "Datetime(time_unit='us', time_zone=None)"),
    ("quarter", "String"),
    ("month", "Int64"),
    ("year", "Int64"),
    ("price", "Float64"),
    ("quantity", "Int64"),
    ("status", "String"),
    ("email", "String"),
    ("revenue", "Float64"),
    ("tax", "Float64"),
    ("total", "Float64"),
    ("payment_method", "String"),
    ("sales_channel", "String")
])

# The validation plan
validation = (
    pb.Validate(
        data=your_data,  # Replace your_data with the actual data variable
        label="Draft Validation",
        thresholds=pb.Thresholds(warning=0.10, error=0.25, critical=0.35)
    )
    .col_schema_match(schema=schema)
    .col_vals_not_null(columns=[
        "product_category", "customer_segment", "region", "country",
        "price", "quantity", "status", "email", "revenue", "tax",
        "total", "payment_method", "sales_channel"
    ])
    .col_vals_between(columns="month", left=1, right=12, na_pass=True)
    .col_vals_between(columns="year", left=2021, right=2023, na_pass=True)
    .col_vals_gt(columns="price", value=0)
    .col_vals_gt(columns="quantity", value=0)
    .col_vals_gt(columns="revenue", value=0)
    .col_vals_gt(columns="tax", value=0)
    .col_vals_gt(columns="total", value=0)
    .col_vals_in_set(columns="product_category", set=[
        "Manufacturing", "Retail", "Healthcare"
    ])
    .col_vals_in_set(columns="customer_segment", set=[
        "Government", "Consumer", "SMB"
    ])
    .col_vals_in_set(columns="region", set=[
        "Asia Pacific", "Europe", "North America"
    ])
    .col_vals_in_set(columns="status", set=[
        "returned", "shipped", "delivered"
    ])
    .col_vals_in_set(columns="payment_method", set=[
        "Apple Pay", "PayPal", "Bank Transfer", "Credit Card"
    ])
    .col_vals_in_set(columns="sales_channel", set=[
        "Partner", "Distributor", "Phone"
    ])
    .row_count_match(count=50000)
    .col_count_match(count=20)
    .rows_distinct()
    .interrogate()
)

validation
```
````

### Managing API Keys

While you can directly provide API keys as shown above, there are more secure approaches:

```python
import os

# Get API key from environment variable
api_key = os.getenv("ANTHROPIC_API_KEY")

draft_validation = pb.DraftValidation(
    data=data,
    model="anthropic:claude-opus-4-6",
    api_key=api_key
)
```

You can also store API keys in a `.env` file in your project's root directory:

```
# Contents of .env file
ANTHROPIC_API_KEY=your_anthropic_api_key_here
OPENAI_API_KEY=your_openai_api_key_here

# For Azure OpenAI, three variables are required:
AZURE_OPENAI_API_KEY=your_azure_openai_api_key_here
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
OPENAI_API_VERSION=2025-03-01-preview
```

If your API keys have standard names (like `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`),
`DraftValidation` will automatically find and use them:

```python
# No API key needed if stored in .env with standard names
draft_validation = pb.DraftValidation(
    data=data,
    model="anthropic:claude-opus-4-6"
)
```

## Example Output for `nycflights`

Here's an example of a validation plan that might be generated by `DraftValidation` for the
`nycflights` dataset:

```python
pb.DraftValidation(
    pb.load_dataset(dataset="nycflights", tbl_type="duckdb",
    model="anthropic:claude-opus-4-6"
)
```

````plaintext
```python
import pointblank as pb

# Define schema based on column names and dtypes
schema = pb.Schema(columns=[
    ("year", "int64"),
    ("month", "int64"),
    ("day", "int64"),
    ("dep_time", "int64"),
    ("sched_dep_time", "int64"),
    ("dep_delay", "int64"),
    ("arr_time", "int64"),
    ("sched_arr_time", "int64"),
    ("arr_delay", "int64"),
    ("carrier", "string"),
    ("flight", "int64"),
    ("tailnum", "string"),
    ("origin", "string"),
    ("dest", "string"),
    ("air_time", "int64"),
    ("distance", "int64"),
    ("hour", "int64"),
    ("minute", "int64")
])

# The validation plan
validation = (
    pb.Validate(
        data=your_data,  # Replace your_data with the actual data variable
        label="Draft Validation",
        thresholds=pb.Thresholds(warning=0.10, error=0.25, critical=0.35)
    )
    .col_schema_match(schema=schema)
    .col_vals_not_null(columns=[
        "year", "month", "day", "sched_dep_time", "carrier", "flight",
        "origin", "dest", "distance", "hour", "minute"
    ])
    .col_vals_between(columns="month", left=1, right=12)
    .col_vals_between(columns="day", left=1, right=31)
    .col_vals_between(columns="sched_dep_time", left=106, right=2359)
    .col_vals_between(columns="dep_delay", left=-43, right=1301, na_pass=True)
    .col_vals_between(columns="air_time", left=20, right=695, na_pass=True)
    .col_vals_between(columns="distance", left=17, right=4983)
    .col_vals_between(columns="hour", left=1, right=23)
    .col_vals_between(columns="minute", left=0, right=59)
    .col_vals_in_set(columns="origin", set=["EWR", "LGA", "JFK"])
    .col_count_match(count=18)
    .row_count_match(count=336776)
    .rows_distinct()
    .interrogate()
)

validation
```
````

Notice how the generated plan includes:

1. A schema validation with appropriate data types
2. Not-null checks for required columns
3. Range validations for numerical data
4. Set membership checks for categorical data
5. Row and column count validations
6. Appropriate handling of missing values with `na_pass=True`

## Working with Model Providers

### Specifying Models

When using `DraftValidation`, you specify the model in the format `"provider:model_name"`:

```python
# Using Anthropic's Claude model
pb.DraftValidation(data=data, model="anthropic:claude-opus-4-6")

# Using OpenAI's GPT model
pb.DraftValidation(data=data, model="openai:gpt-5.6-sol")

# Using a local model with Ollama
pb.DraftValidation(data=data, model="ollama:llama3:latest")

# Using Amazon Bedrock
pb.DraftValidation(data=data, model="bedrock:anthropic.claude-sonnet-5-5")

# Using Azure OpenAI (the value after the colon is the Azure deployment id, not an OpenAI model id;
# requires AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, and OPENAI_API_VERSION env vars)
pb.DraftValidation(data=data, model="azure-openai:my-gpt4-deployment")
```

### Model Performance and Privacy

Different models have different capabilities when it comes to generating validation plans:

- Current Anthropic Claude and OpenAI GPT models generally provide the most comprehensive and
accurate validation plans
- Local models through Ollama can be useful for private data but they currently have reduced
capabilities here

A key advantage of `DraftValidation` is that your actual dataset is not sent to the LLM provider.
Instead, only a summary is transmitted, which includes:

- the number of rows and columns
- column names and data types
- basic statistics (min, max, mean, median, missing values count)
- a small sample of values from each column (usually 5-10 values)

This approach protects your data while still providing enough context for the LLM to generate
relevant validation rules.

## Customizing Generated Plans

The validation plan generated by `DraftValidation` is just a starting point. You'll typically want
to:

1. review the generated code for correctness
2. replace `your_data` with your actual data variable name that exists in your workspace
3. ensure the data object referenced is actually present in your workspace
4. adjust thresholds and validation parameters
5. add domain-specific validation rules
6. remove any unnecessary checks

For example, you might start by capturing the text representation of your draft validation. This
will give you the raw Python code that you can copy into a new code cell in your notebook or script.
From there, you can customize it by modifying thresholds to match your organization's data quality
standards, adding business-specific validation rules that require domain knowledge, or removing
checks that aren't relevant to your use case. Once you've made your modifications, you can execute
the customized validation plan as you would any other Pointblank validation.

## Under the Hood

### The Generated Data Summary

To understand what the LLM works with, here's an example of the data summary format that's sent:

```json
{
  "table_info": {
    "rows": 336776,
    "columns": 18,
    "table_type": "duckdb"
  },
  "column_info": [
    {
      "column_name": "year",
      "column_type": "int64",
      "missing_values": 0,
      "min": 2013,
      "max": 2013,
      "mean": 2013.0,
      "median": 2013,
      "sample_values": [2013, 2013, 2013, 2013, 2013]
    },
    {
      "column_name": "month",
      "column_type": "int64",
      "missing_values": 0,
      "min": 1,
      "max": 12,
      "mean": 6.548819,
      "median": 7,
      "sample_values": [1, 1, 1, 1, 1]
    },
    // Additional columns...
  ]
}
```

### The Prompt Strategy

The `DraftValidation` class uses a carefully crafted prompt that instructs the LLM to:

1. use the schema information to create a `Schema` object
2. include [`Validate.col_vals_not_null()`](`Validate.col_vals_not_null`) for columns with no missing values
3. add appropriate range validations based on min/max values
4. include row and column count validations
5. format the output as clean, executable Python code

The prompt also contains constraints to ensure consistent, high-quality results, such as using line
breaks in long lists for readability, applying `na_pass=True` for columns with missing values, and
avoiding duplicate validations.

## Best Practices and Troubleshooting

### When to Use `DraftValidation`{.qd-no-link}

Drafting a validation is most useful when:

- working with a new dataset for the first time
- needing to quickly establish baseline validation
- exploring potential validation rules before formalizing them
- validating columns with consistent patterns (numeric ranges, categories, etc.)

Consider writing validation plans manually when you need very specific business rules, are working
with sensitive data, need complex validation logic, or need to validate relationships between
columns.

### Recommended Workflow and Common Issues

Here's a recommended workflow incorporating `DraftValidation`:

1. generate an initial plan with `DraftValidation`
2. review the generated validations for relevance
3. adjust thresholds and parameters as needed
4. add specific business logic and cross-column validations
5. store the final validation plan in version control

It's possible that you might bump up against some issues. Here are some common ones and solutions
you might try:

- Authentication Errors: ensure your API key is valid and correctly passed to `DraftValidation`
- Package Not Found: make sure you've installed the required packages with
`pip install pointblank[generate]`
- Unsupported Model: verify you're using the correct `provider:model` format
- Poor Quality Plans: try a more capable model

## Conclusion

`DraftValidation` provides a powerful way to jumpstart your data validation process by leveraging
LLMs to generate context-aware validation plans. By analyzing your data's structure and content,
`DraftValidation` can create comprehensive validation rules that would otherwise take significant
time to develop manually.

The feature balances privacy (by sending only data summaries) with utility (by generating executable
validation code). While the generated plans should always be reviewed and refined, they provide an
excellent starting point for ensuring your data meets your quality requirements.

By understanding how `DraftValidation` works and how to customize its output, you can significantly
accelerate your data validation workflows and improve the quality of your data throughout your
projects.


### Serializing Validation Plans

A validation plan you build in Python isn't locked inside the `Validate` object. Pointblank can
render an in-memory plan back into canonical source, either as a Python method chain, a YAML
configuration, or a language-neutral JSON Schema document. This is the inverse of writing a plan by
hand: you hand it a `Validate` object and it gives you a portable representation.

Three methods do this:

- [`Validate.to_code()`](`Validate.to_code`) produces the
`pb.Validate(...).col_vals_*()...` chain as a string
- [`Validate.to_yaml()`](`Validate.to_yaml`) produces a
[`yaml_interrogate()`](`yaml_interrogate`)-compatible YAML document
- [`Validate.to_json_schema()`](`Validate.to_json_schema`) produces a
[JSON Schema](https://json-schema.org/) document

The first two are ordinary, deterministic utilities that round-trip through Pointblank itself.
`to_json_schema()` produces a standard that tools in many ecosystems can consume. All three are
useful for sharing plans, reviewing changes in a diff, and persisting a plan alongside its results.
[`to_code()`](`Validate.to_code`) and [`to_yaml()`](`Validate.to_yaml`) are also the foundation for
the [AI Validation Editor](ai-validation-editor.qmd), which sends the current plan to a model as
editable code.

## Rendering a Plan as Python Code

Given any validation plan, [`to_code()`](`Validate.to_code`) returns a self-contained
block of Python that reproduces it:

```{python}
import pointblank as pb

validation = (
    pb.Validate(
        data=pb.load_dataset("small_table"),
        tbl_name="small_table",
        label="Small table checks",
        thresholds=pb.Thresholds(warning=0.1, error=0.25),
    )
    .col_vals_gt(columns="d", value=100)
    .col_vals_not_null(columns="a")
    .col_vals_not_null(columns="b")
    .col_vals_in_set(columns="f", set=["low", "mid", "high"])
    .rows_distinct()
    .row_count_match(count=13)
)

print(validation.to_code())
```

A few things to notice in the output:

- The table-level configuration (`tbl_name`, `label`, `thresholds`) is reproduced on the
`pb.Validate(...)` call.
- The data source is rendered as the placeholder `your_data`, since the plan doesn't know the name
of the variable your data lives in. Replace it with your actual data variable before running the
code.
- Adjacent steps that differ only by column are collapsed into a single call: the two
`col_vals_not_null()` steps above become `col_vals_not_null(columns=["a", "b"])`. This keeps the
output compact and diffs minimal.
- Only non-default arguments are emitted. A step-level threshold that matches the table-level default
is omitted, `na_pass=True` appears only where it was set, and so on.

The result is regular source you can paste into a script or notebook, commit to version control, or
share with a colleague.

## Rendering a Plan as YAML

[`to_yaml()`](`Validate.to_yaml`) renders the same plan using the schema consumed by
[`yaml_interrogate()`](`yaml_interrogate`):

```{python}
print(validation.to_yaml())
```

The `tbl` field is set from `tbl_name` when available, otherwise to the placeholder `your_data`. Set
it to a loadable data source (a dataset name, file path, or Python expression) before handing the
YAML to `yaml_interrogate()`. See the [YAML Validation Workflows](../yaml/yaml-validation-workflows.qmd)
page for the full YAML schema.

You can also write the YAML directly to a file by passing `path=`:

```python
validation.to_yaml(path="plans/small_table.yaml")
```

## Round-Tripping a Plan

Because `to_yaml()` emits a `yaml_interrogate()`-compatible document, a plan round-trips through
YAML. Here we serialize a plan, then rebuild and run it with `yaml_interrogate()`:

```{python}
yaml_str = validation.to_yaml()

rebuilt = pb.yaml_interrogate(yaml_str)
rebuilt
```

The rebuilt plan carries the same steps as the original. The Python code from `to_code()`
round-trips in the same way: executing it (after replacing `your_data`) reconstructs an equivalent
plan.

## Fidelity and Limitations

Most validation steps round-trip exactly. The exceptions are steps that carry live Python objects
that can't be reconstructed from source:

- `pre=` preprocessing callables
- step-level `actions=`
- callable `active=` conditions
- the expressions used by [`col_vals_expr()`](`Validate.col_vals_expr`),
[`conjointly()`](`Validate.conjointly`), and
[`specially()`](`Validate.specially`)

For these, Pointblank emits a clearly-marked placeholder so the generated code still parses and runs,
and raises a warning describing what was lost. The following plan uses a `pre=` callable that can't
be serialized:

```{python}
import warnings

flagged = pb.Validate(data=pb.load_dataset("small_table")).col_vals_gt(
    columns="d", value=100, pre=lambda df: df
)

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    code = flagged.to_code()
    for w in caught:
        print("warning:", w.message)
```

When you see such a warning, review the generated plan and restore the affected step by hand. Plans
built entirely from literal arguments (values, ranges, sets, patterns, counts, schemas) serialize
without any loss.

## When to Use Serialization

Rendering a plan back to source is helpful whenever you want the plan to leave the current Python
session:

- **Sharing.** Send a colleague the exact plan as code or YAML rather than describing it.
- **Code review.** Serialize a plan before and after a change and diff the two to see precisely what
moved. You can produce this comparison directly with
[`EditValidation.from_plans()`](`EditValidation.from_plans`) (no LLM required), which is also how
the [AI Validation Editor](ai-validation-editor.qmd) presents proposed edits.
- **Persistence.** Store the *plan* (not just the results) alongside a validation run, so you have a
durable, human-readable record of what was checked.
- **Migrating between formats.** Move a plan from Python to YAML (or, via
[`yaml_to_python()`](`yaml_to_python`), from YAML to Python) to fit it into a
configuration-driven workflow.

## Exporting a Plan as JSON Schema

While `to_code()` and `to_yaml()` reproduce your plan as Pointblank-specific source,
[`to_json_schema()`](`Validate.to_json_schema`) converts your validation rules into a
[JSON Schema](https://json-schema.org/) document. JSON Schema is a widely adopted, language-neutral
standard, so the exported schema can be consumed by validation libraries in JavaScript, Java, Go,
and many other ecosystems.

```{python}
import json

validation = (
    pb.Validate(
        data=pb.load_dataset("small_table"),
        tbl_name="small_table",
    )
    .col_vals_gt(columns="d", value=100)
    .col_vals_not_null(columns="date_time")
    .col_vals_in_set(columns="f", set=["low", "mid", "high"])
    .col_vals_regex(columns="b", pattern="^[0-9]-[a-z]{3}-[0-9]{3}$")
    .col_vals_between(columns="a", left=1, right=8)
)

schema = validation.to_json_schema()

print(json.dumps(schema, indent=2))
```

Each validation method is mapped to its closest JSON Schema keyword:

| Pointblank method | JSON Schema keyword |
|---|---|
| `col_vals_not_null()` | `required` |
| `col_vals_gt()` | `exclusiveMinimum` |
| `col_vals_ge()` | `minimum` |
| `col_vals_lt()` | `exclusiveMaximum` |
| `col_vals_le()` | `maximum` |
| `col_vals_eq()` | `const` |
| `col_vals_between()` | `minimum` + `maximum` |
| `col_vals_in_set()` | `enum` |
| `col_vals_regex()` | `pattern` |
| `col_vals_within_spec()` | `format` |

When the `Validate` object has data attached, the schema is automatically enriched with column type
information (`"type": "integer"`, `"type": "string"`, etc.) inferred from the data. Steps that have
no natural JSON Schema equivalent (e.g., `rows_distinct()`, `tbl_match()`) are silently skipped.

### Writing to a File

Pass a `path=` argument to write the JSON Schema to disk. Parent directories are created
automatically:

```python
validation.to_json_schema(path="schemas/small_table.schema.json")
```

This is useful for checking schemas into version control or feeding them into downstream tooling
that consumes JSON Schema files.

### Relationship to `export_contract()`

Under the hood, `to_json_schema()` delegates to the
[`export_contract()`](`pointblank.export_contract`) function with `format="json_schema"`. The
convenience method adds type enrichment from the data and handles file output. If you need the raw
adapter behavior (for example, to export without type enrichment, or to work with a `Contract`
object rather than a `Validate` object), use `export_contract()` directly. See the [Importing and
Exporting Contracts](../08-contracts-and-pipelines/04-importing-contracts.qmd) guide for the full
story on contract adapters.

## Conclusion

[`to_code()`](`Validate.to_code`), [`to_yaml()`](`Validate.to_yaml`), and
[`to_json_schema()`](`Validate.to_json_schema`) turn a live validation plan back into portable
source. The first two are deterministic, Pointblank-specific formats that round-trip through
execution and [`yaml_interrogate()`](`yaml_interrogate`) respectively. The third produces a
language-neutral JSON Schema that any JSON-Schema-aware tool can consume. Beyond sharing and
persistence, these representations provide the reviewable, diffable record that powers the
[AI Validation Editor](ai-validation-editor.qmd) covered in the next section.


### AI Validation Editor

Where [Draft Validation](draft-validation.qmd) uses a large language model (LLM) to *generate* a
validation plan from scratch, the AI Validation Editor uses one to *edit* a plan you already have.
You give it an existing plan plus a plain-English instruction, such as "add a not-null check on
`user_id`, tighten the email regex, and drop the `price > 0` check", and it returns a revised plan.
You review the change as a diff and explicitly accept it before anything runs, keeping a human in the
loop.

The Editor is built on the [plan serialization](plan-serialization.qmd) primitives described in the
previous section: it renders your current plan to code with
[`to_code()`](`Validate.to_code`), sends that code and your instruction to the model, and
returns the model's revised code for your review.

The main entry point is the `EditValidation` class. Two convenience methods on `Validate`,
[`suggest_improvements()`](`Validate.suggest_improvements`) and
[`from_prompt()`](`Validate.from_prompt`), wrap the same flow, and the `pb edit` CLI
command brings it to the terminal.

::: {.callout-note}
## About the examples on this page

The `EditValidation` calls below need a live LLM, so they're shown as **non-executable code** (what
you'd write once your API key is set up). The output beneath each one is rendered from a
*representative* stand-in model response, so you can see the shape of the results here without a key.
A real model may word things slightly differently, but the mechanics (diffing, reviewing,
accepting) are exactly what you'll experience.
:::

## How the Editor Works

When you edit a plan, the process works through these steps:

1. your current plan is normalized to canonical Pointblank code via
[`to_code()`](`Validate.to_code`)
2. that code, your instruction, and (optionally) a `DataScan` summary of the table are assembled into
a prompt and sent to your selected LLM provider
3. the model returns a complete revised plan as Python code
4. the revised code is checked for syntax and unknown methods; on failure the model is automatically
re-prompted to fix it
5. you inspect the proposed change as a diff and, when satisfied, accept it to obtain a runnable
`Validate` object

As with `DraftValidation`, the entire dataset is never sent to the model, only a summary. See
[Data Privacy](#data-privacy) below.

## Requirements and Setup

The Editor shares its requirements with [Draft Validation](draft-validation.qmd): an API key from
a supported LLM provider and the optional dependencies, installed with:

```bash
pip install pointblank[generate]
```

The same five providers are supported (`anthropic`, `openai`, `ollama`, `bedrock`, and
`azure-openai`), the model is specified in `provider:model` form, and API keys are resolved from the
`api_key=` argument, environment variables, or a `.env` file exactly as described on the Draft
Validation page.

## Editing a Plan with `EditValidation`{.qd-no-link}

Suppose you have an existing plan:

```{python}
import pointblank as pb

validation = (
    pb.Validate(data=pb.load_dataset("small_table"), label="Small table checks")
    .col_vals_gt(columns="d", value=100)
    .col_vals_regex(columns="b", pattern="[0-9]-[a-z]{3}")
)
```

Describe the change you want in plain English and hand both to `EditValidation`:

```{python}
#| eval: false
edited = pb.EditValidation(
    validation=validation,
    instruction="Loosen the d check to 50, drop the b regex, and add a not-null check on a.",
    model="anthropic:claude-opus-4-8",
    data=pb.load_dataset("small_table"),
)
```

```{python}
#| echo: false
# Stand in for a live model so this page renders without an API key. This sets up a fake chat
# client that returns a representative revised plan; everything else on the page is real.
import sys
import types

from unittest.mock import patch

# The optional `chatlas` dependency isn't installed in the docs build; stub it so the
# EditValidation availability check passes (the model call itself is mocked below).
sys.modules.setdefault("chatlas", types.ModuleType("chatlas"))


class _DemoChat:
    def __init__(self, response):
        self._response = response

    def chat(self, prompt, *args, **kwargs):
        return self._response


def _use_model_response(code_block):
    patch("pointblank.edit._create_chat_instance", return_value=_DemoChat(code_block)).start()


_EDITED = """import pointblank as pb

validation = (
    pb.Validate(
        data=your_data,  # Replace your_data with the actual data variable
        label="Small table checks",
    )
    .col_vals_gt(columns="d", value=50)
    .col_vals_not_null(columns="a")
)

validation"""

_IMPROVED = """import pointblank as pb

validation = (
    pb.Validate(
        data=your_data,  # Replace your_data with the actual data variable
        label="Small table checks",
        thresholds=pb.Thresholds(warning=0.1, error=0.25),
    )
    .col_vals_gt(columns="d", value=100)
    .col_vals_regex(columns="b", pattern="[0-9]-[a-z]{3}")
    .col_vals_not_null(columns=["a", "c", "e"])
    .col_vals_in_set(columns="f", set=["low", "mid", "high"])
    .rows_distinct()
)

validation"""

_AUTHORED = """import pointblank as pb

validation = (
    pb.Validate(
        data=your_data,  # Replace your_data with the actual data variable
        tbl_name="small_table",
    )
    .col_vals_not_null(columns="a")
    .col_vals_gt(columns="d", value=0)
)

validation"""

_use_model_response(_EDITED)
edited = pb.EditValidation(
    validation=validation,
    instruction="Loosen the d check to 50, drop the b regex, and add a not-null check on a.",
    model="anthropic:claude-opus-4-8",
    data=pb.load_dataset("small_table"),
)
```

The `validation=` argument is flexible. It accepts a live `Validate` object (as above), a code string
(such as the output of [`to_code()`](`Validate.to_code`)), a YAML configuration string, or
a path to a `.py` or `.yaml`/`.yml` file. All of these are normalized to code before being sent to
the model, so you can edit a plan wherever it happens to live.

The optional `data=` argument is worth providing when you can: it includes a `DataScan` profile of
the table in the prompt so the model can make *informed* edits (for example, choosing a realistic
range from the observed min and max). It's also used as the default data source when you accept the
edit.

## Reviewing the Proposed Change

The Editor proposes; you dispose. Before anything runs, inspect what the model changed using the
`edited` object from above.

### As a Unified Diff

[`diff()`](`EditValidation.diff`) returns a textual, line-level diff of the original plan
versus the revised plan:

```{python}
print(edited.diff())
```

### As a Structured Change List

[`changed_steps()`](`EditValidation.changed_steps`) compares the two plans at the level of
validation steps and returns a structured list. Unlike the textual diff, this is robust to
reformatting (it compares whole steps rather than lines):

```{python}
edited.changed_steps()
```

Each record has an `"action"` of `"add"`, `"remove"`, or `"modify"`, the `"method"` involved, and the
relevant `"old"`/`"new"` step text.

### As a Table

[`review()`](`EditValidation.review`) renders the same change list as a
[Great Tables](https://posit-dev.github.io/great-tables/) object, color-coding additions, removals,
and modifications for quick visual review in a notebook:

```{python}
edited.review()
```

## Accepting the Edit

Once you're satisfied with the proposed change,
[`accept()`](`EditValidation.accept`) executes the revised plan and returns a `Validate`
object, ready to interrogate:

```{python}
plan = edited.accept()          # a Validate (not yet interrogated)
plan.interrogate()              # run it
```

The data bound to the plan is the `data=` you passed to `EditValidation`; you can override it with
`edited.accept(data=other_table)`. Nothing runs until you call `accept()`, and the plan isn't
interrogated until you call `interrogate()`, so there's a clear, deliberate boundary between the
model's proposal and execution.

## Comparing Two Plans Without a Model

You don't need an LLM to review the difference between two plans. When you already have a "before"
and "after" (for example, two versions of a plan from source control),
[`EditValidation.from_plans()`](`EditValidation.from_plans`) builds the same comparison object
directly, with all of `diff()`, `changed_steps()`, and `review()` available and no API key
required. The following example is fully live:

```{python}
before = pb.Validate(data=pb.load_dataset("small_table")).col_vals_gt(columns="d", value=100)
after = (
    pb.Validate(data=pb.load_dataset("small_table"))
    .col_vals_gt(columns="d", value=50)
    .col_vals_not_null(columns="a")
)

pb.EditValidation.from_plans(before, after).review()
```

## Guardrails

Two guardrails keep the flow from handing you broken code.

- **Syntax and lint check.** The revised code is parsed with `ast.parse()` and checked for
validation-method typos before it's returned.
[`validate_syntax()`](`EditValidation.validate_syntax`) reports whether the current
revision passes.
- **Automatic re-prompt.** If the check fails, the model is automatically re-prompted with the error
message to correct itself, up to `max_reprompts` times (default `1`). If the code still doesn't pass,
[`accept()`](`EditValidation.accept`) refuses to run it and points you at
[`to_code()`](`EditValidation.to_code`) so you can fix it by hand.

These same guardrails are also applied to [`DraftValidation`](draft-validation.qmd).

## Natural-Language Refinements

Two methods on `Validate` wrap the edit flow for common cases.

### Suggesting Improvements

[`suggest_improvements()`](`Validate.suggest_improvements`) profiles the table, derives an
instruction targeting gaps in the current plan (columns with no coverage, missing thresholds), and
asks the model to extend the plan. It returns an `EditValidation` you review and accept as usual:

```{python}
#| eval: false
proposal = validation.suggest_improvements(model="anthropic:claude-opus-4-8")
```

```{python}
#| echo: false
_use_model_response(_IMPROVED)
proposal = validation.suggest_improvements(model="anthropic:claude-opus-4-8")
```

```{python}
print(proposal.diff())
```

### Building a Plan from a Prompt

[`from_prompt()`](`Validate.from_prompt`) is the same flow starting from an *empty* plan:
the model authors steps that satisfy your prompt, using a `DataScan` profile of the data and any
table name, label, and thresholds carried by the object you call it on:

```{python}
#| eval: false
base = pb.Validate(data=pb.load_dataset("small_table"), tbl_name="small_table")

authored = base.from_prompt(
    "ensure column a has no nulls and column d is always positive",
    model="anthropic:claude-opus-4-8",
)
```

```{python}
#| echo: false
base = pb.Validate(data=pb.load_dataset("small_table"), tbl_name="small_table")
_use_model_response(_AUTHORED)
authored = base.from_prompt(
    "ensure column a has no nulls and column d is always positive",
    model="anthropic:claude-opus-4-8",
)
```

```{python}
print(authored.to_code())
```

## Editing from the Command Line

The `pb edit` command brings the Editor to the terminal. Point it at a validation file, describe the
change, and choose a model:

```bash
pb edit plan.py -i "add a not-null check on user_id" -m anthropic:claude-opus-4-8
```

The command prints the proposed change as a color diff along with a summary (for example,
`1 added, 1 removed, 1 modified`). Provide a data source for profile-informed edits with `--data`,
and write the revised plan to a file with `--output` (add `--yes` to skip the confirmation prompt):

```bash
pb edit plan.yaml -i "tighten the price range to 0-1000" -m openai:gpt-6-luna --data sales.csv
pb edit plan.py   -i "drop the email regex check"        -m anthropic:claude-opus-4-8 -o plan2.py -y
```

## Data Privacy {#data-privacy}

The Editor follows the same privacy posture as [Draft Validation](draft-validation.qmd): your
actual dataset is never sent to the model. When you provide `data=`, only a `DataScan` summary is
transmitted, which includes column names and types, per-column statistics (min, max, mean, median,
missing counts), and a small sample of values from each column. This is enough context for informed
edits without exposing the full table.

If you prefer to send no data at all, omit `data=` on `EditValidation`; the model then edits the plan
using only the plan code and your instruction. (Because their whole purpose is to profile coverage
gaps, `suggest_improvements()` and `from_prompt()` always include the summary.)

## Best Practices

- **Make small, targeted edits.** The Editor is designed to change only what your instruction
implies and preserve unrelated steps verbatim. Keeping instructions focused produces minimal,
reviewable diffs.
- **Always review before accepting.** Read the diff or the change list; the model proposes, but you
decide what runs.
- **Provide data when you can.** A `DataScan` profile lets the model choose realistic bounds and set
members rather than guessing.
- **Version-control the accepted plan.** Once you accept an edit, serialize the result with
[`to_code()`](`Validate.to_code`) or [`to_yaml()`](`Validate.to_yaml`) and
commit it, so the plan's evolution is tracked alongside your code.

## Conclusion

The AI Validation Editor extends Pointblank's AI capabilities from generating plans to iterating on
them. By combining plan serialization, a natural-language edit prompt, structural diffing, and
syntax guardrails, it lets you refine validation plans conversationally while keeping a human firmly
in control of what actually runs. Together with [Draft Validation](draft-validation.qmd), it
covers both halves of an AI-assisted validation workflow: create a plan, then keep improving it.


### Composable Steps

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

When validation logic grows beyond a handful of checks, you often want to **reuse** groups of steps
across datasets and **compose** them into larger plans. Pointblank provides two complementary tools
for this:

- **`Step`** (singular): a declarative data object used inside `Contract` definitions
- **`Steps`** (plural): a fluent builder that collects validation steps for use in any
  `Validate` plan

This page focuses on `Steps` and the `Validate.add_steps()` method, which together let you define
portable step libraries and compose them into readable, top-to-bottom validation plans.

## The Problem

Without composable steps, reusing validation logic means writing helper functions:

```python
def add_completeness_checks(v):
    return v.col_vals_not_null(columns=["order_id", "email"])

def add_range_checks(v):
    return v.col_vals_ge(columns="amount", value=0)

validation = (
    add_range_checks(add_completeness_checks(
        pb.Validate(data=orders)
    ))
    .interrogate()
)
```

This works but reads *inside-out*. With more than two or three groups, the nesting becomes hard to
follow. The step definitions are also coupled to whichever `Validate` object they happen to be
called on.

## Building Step Libraries with `Steps`

The `Steps` class lets you record validation steps without binding them to data, thresholds, or any
other `Validate` configuration. You use the same method names and parameters you already know from
`Validate` (`col_vals_gt()`, `col_vals_not_null()`, `col_vals_regex()`, etc.), but instead of
running the checks, `Steps` stores them for later use.

The key benefit is **portability**: a `Steps` object can live in a shared Python module and be
imported into any number of validation pipelines. The step definitions stay the same and only the
data and configuration change at the point of use.

```{python}
import pointblank as pb
import polars as pl

completeness = (
    pb.Steps()
    .col_vals_not_null(columns="order_id")
    .col_vals_not_null(columns="email")
)

positive_amounts = (
    pb.Steps()
    .col_vals_ge(columns="amount", value=0)
    .col_vals_gt(columns="total", value=0)
)

format_checks = (
    pb.Steps()
    .col_vals_regex(columns="email", pattern=r".+@.+\..+")
)
```

Each `Steps` object above is a self-contained recipe. You can inspect one to see what it contains:

```{python}
print(completeness)
```

```{python}
print(f"Number of steps: {len(completeness)}")
```

In a Jupyter notebook or other rich-display environment, `Steps` renders as an HTML summary table.

## Composing Plans with `add_steps()`

The `add_steps()` method on `Validate` accepts one or more `Steps` objects and appends their
recorded steps to the validation plan:

```{python}
orders = pl.DataFrame(
    {
        "order_id": ["ORD-001", "ORD-002", "ORD-003"],
        "email": ["alice@example.com", "bob@corp.io", "charlie@mail.org"],
        "amount": [29.99, 149.50, 9.99],
        "total": [34.99, 155.00, 14.99],
    }
)

(
    pb.Validate(data=orders, label="Order quality")
    .add_steps(completeness, positive_amounts, format_checks)
    .interrogate()
)
```

Because `add_steps()` returns the `Validate` object, it chains naturally and reads top-to-bottom.

You can also mix `add_steps()` calls with regular validation method calls. This is useful when a
pipeline needs a shared library of checks plus a few ad-hoc rules specific to that dataset:

```{python}
(
    pb.Validate(data=orders, label="Orders (extended)")
    .add_steps(completeness)
    .col_vals_lt(columns="amount", value=500)
    .add_steps(format_checks)
    .interrogate()
)
```

### Passing Multiple Sources

The `add_steps()` method accepts any number of positional arguments. Each can be a `Steps` object or
even a `Validate` object (more on that [below](#importing-steps-from-a-validate-object)). All steps
are appended in order:

```{python}
all_checks = (
    pb.Validate(data=orders)
    .add_steps(completeness, positive_amounts, format_checks)
)

print(f"Total validation steps: {len(all_checks.validation_info)}")
```

## Selectors and Multi-Column Expansion

One of the strengths of `Steps` is that step definitions are not locked to specific columns at
definition time. When a step uses a column selector like `starts_with()` or a multi-column list,
the columns are resolved when `add_steps()` applies them to the `Validate` plan, not when the
`Steps` object is created. This means the same step library can validate tables with different
schemas as long as they follow a common naming convention.

```{python}
products = pl.DataFrame(
    {
        "product_id": ["P1", "P2", "P3"],
        "amt_price": [19.99, 49.99, 9.99],
        "amt_tax": [1.60, 4.00, 0.80],
        "amt_total": [21.59, 53.99, 10.79],
    }
)

# Column selectors resolve at add_steps() time against the target data
price_checks = (
    pb.Steps()
    .col_vals_gt(columns=pb.starts_with("amt_"), value=0)
)

(
    pb.Validate(data=products)
    .add_steps(price_checks)
    .interrogate()
)
```

The selector `starts_with("amt_")` is recorded verbatim in the `Steps` object and resolved when the
steps are applied to the `Validate` plan. This means the same `Steps` object adapts to tables
with different numbers of matching columns.

## Overrides in `add_steps()`

A shared step library represents a general-purpose set of checks, but individual pipelines often
need to tweak how those checks are applied: tighter thresholds in production, certain checks
disabled during development, or column names that differ across teams. Rather than creating a
separate `Steps` object for every variation, `add_steps()` supports keyword arguments that modify
the imported steps at the point of use, without changing the original `Steps` object.

### Conditional Inclusion with `active=`

Add steps only when a condition is met, keeping the chain flat instead of wrapping in `if` blocks:

```{python}
is_production = True

strict_checks = (
    pb.Steps()
    .col_vals_gt(columns="total", value=0)
    .col_vals_lt(columns="total", value=10_000)
)

(
    pb.Validate(data=orders)
    .add_steps(completeness)
    .add_steps(strict_checks, active=is_production)
    .interrogate()
)
```

When `active=False`, the steps are still added to the plan (so they appear in the report) but they
are not executed during interrogation.

### Threshold Override with `thresholds=`

Different groups of checks often warrant different levels of strictness. Completeness checks might
need a very low failure tolerance (even 1% missing IDs is a problem), while range checks can
tolerate slightly more variance. The `thresholds=` parameter lets you set per-group thresholds
without modifying the shared step library:

```{python}
(
    pb.Validate(data=orders, thresholds=pb.Thresholds(warning=0.05))
    .add_steps(completeness, thresholds=pb.Thresholds(warning=0.01))
    .add_steps(positive_amounts)
    .interrogate()
)
```

Here, the completeness checks use a stricter 1% warning threshold while the amount checks inherit
the 5% global threshold.

### Step Filtering with `exclude=`

Sometimes a shared library is *almost* right for a particular pipeline, but one or two checks don't
apply. Rather than forking the library, you can exclude specific steps at import time. This keeps
the shared definition as the single source of truth while giving individual pipelines the
flexibility to opt out of checks that aren't relevant.

You can exclude by method name (a string, which removes all steps using that method) or by 1-based
step index (an integer, which removes only the step at that position):

```{python}
full_checks = (
    pb.Steps()
    .col_vals_not_null(columns="order_id")
    .col_vals_not_null(columns="email")
    .col_vals_regex(columns="email", pattern=r".+@.+\..+")
)

# Exclude the regex check by method name
(
    pb.Validate(data=orders)
    .add_steps(full_checks, exclude=["col_vals_regex"])
    .interrogate()
)
```

```{python}
# Or exclude the second step by index
(
    pb.Validate(data=orders)
    .add_steps(full_checks, exclude=[2])
    .interrogate()
)
```

Excluding by method name removes *all* steps with that method name from the source. Excluding by
index removes only the step at that position (1-based) within the source.

### Column Remapping with `columns_map=`

In practice, teams often have tables that contain the same kind of data but use different column
names. A finance team might call it `amount` while the warehouse table uses `txn_amount`. Rather
than maintaining separate step libraries for each naming convention, `columns_map=` lets you remap
column names at import time:

```{python}
# Steps written for one naming convention
amount_checks = (
    pb.Steps()
    .col_vals_ge(columns="amount", value=0)
    .col_vals_lt(columns="amount", value=100_000)
)

# Table uses a different column name
transactions = pl.DataFrame(
    {
        "txn_id": ["T1", "T2"],
        "txn_amount": [50.00, 75.00],
    }
)

(
    pb.Validate(data=transactions)
    .add_steps(amount_checks, columns_map={"amount": "txn_amount"})
    .interrogate()
)
```

The remapping applies to the `columns=`, `column=`, and `columns_subset=` parameters. Column
selectors (like `starts_with()`) pass through unchanged since they resolve dynamically against the
target table.

## Importing Steps from a `Validate` Object

You don't always start from a `Steps` object. Sometimes you already have a `Validate` object with
steps defined on it, perhaps loaded from a YAML file with `Validate.from_yaml()`, generated from a
prompt with `Validate.from_prompt()`, or simply built up in another part of your codebase. Rather
than re-expressing those steps from scratch, you can pass the `Validate` object directly to
`add_steps()` and its step definitions will be extracted and applied:

```{python}
existing_plan = (
    pb.Validate(data=orders)
    .col_vals_not_null(columns="order_id")
    .col_vals_gt(columns="amount", value=0)
)

(
    pb.Validate(data=orders)
    .add_steps(existing_plan)
    .col_vals_regex(columns="email", pattern=r".+@.+\..+")
    .interrogate()
)
```

When a `Validate` object is passed as a step source, `add_steps()` extracts its step definitions and
applies them. The data, thresholds, and metadata from the source `Validate` are *not* carried over,
only the step recipes.

::: {.callout-note}
When a `Validate` object used multi-column lists (e.g., `columns=["id", "name"]`), those columns
were already expanded into separate internal entries. The extracted steps will be individual
single-column steps, not the original grouped call.
:::

## `Steps` vs. `Step`: When to Use Which

Pointblank has two similarly named classes that serve different purposes:

| | `Step` | `Steps` |
|---|---|---|
| **What it is** | A single step as a data object | A builder that collects multiple steps |
| **How you create it** | `pb.Step("col_vals_gt", columns="x", value=0)` | `pb.Steps().col_vals_gt(columns="x", value=0)` |
| **Primary use** | Inside `Contract(steps=[...])` | With `Validate.add_steps()` |
| **API style** | Declarative (method name as string) | Fluent (same methods as `Validate`) |
| **Composition** | List concatenation: `steps_a + steps_b` | Chaining: `.add_steps(a).add_steps(b)` |

**Use `Step`** when you're defining data contracts. Contracts are declarative specifications that
serialize to YAML and represent a fixed agreement about data quality.

**Use `Steps`** when you're building reusable validation logic for `Validate` workflows. The fluent
API gives you autocomplete, type checking, and the same familiar syntax as writing steps directly on
`Validate`.

Both ultimately produce the same validation steps when applied to data. You can even initialize a
`Steps` object from a list of `Step` objects:

```{python}
step_list = [
    pb.Step("col_vals_not_null", columns="order_id"),
    pb.Step("col_vals_gt", columns="amount", value=0),
]

s = pb.Steps(steps=step_list)
print(s)
```

## Putting It All Together

Imagine you're a data platform team maintaining validation rules for an order-processing pipeline.
You have shared step libraries for common concerns (completeness, format, range), and each pipeline
applies them with environment-specific configuration. Here's how that looks in practice:

```{python}
# --- Shared step libraries (defined once, imported anywhere) ---

completeness_lib = (
    pb.Steps()
    .col_vals_not_null(columns="order_id")
    .col_vals_not_null(columns="email")
    .col_vals_not_null(columns="amount")
)

format_lib = (
    pb.Steps()
    .col_vals_regex(columns="email", pattern=r".+@.+\..+")
)

range_lib = (
    pb.Steps()
    .col_vals_ge(columns="amount", value=0)
    .col_vals_lt(columns="amount", value=100_000)
)

# --- Pipeline-specific configuration ---

is_production = True
strict_thresholds = pb.Thresholds(warning=0.01, error=0.05)

orders = pl.DataFrame(
    {
        "order_id": ["ORD-001", "ORD-002", "ORD-003", "ORD-004"],
        "email": ["alice@example.com", "bob@corp.io", "charlie@mail.org", "dave@startup.co"],
        "amount": [29.99, 149.50, 9.99, 75.00],
        "total": [34.99, 155.00, 14.99, 80.00],
    }
)

(
    pb.Validate(data=orders, label="Order validation (production)")
    .add_steps(completeness_lib, thresholds=strict_thresholds)
    .add_steps(format_lib)
    .add_steps(range_lib, active=is_production)
    .rows_distinct(columns_subset=["order_id"])
    .interrogate()
)
```

The validation plan reads as a sequence of concerns: completeness first (with strict thresholds),
then format checks, then range checks (only in production), and finally a uniqueness check added
directly. Each step library can be version-controlled, tested independently, and easily shared.

## Conclusion

The [`Steps`](`pointblank.Steps`) class and
[`Validate.add_steps()`](`pointblank.Validate.add_steps`) method bring composability to Pointblank's
validation workflows. By separating step *definitions* from step *execution*, you can build reusable
rule libraries that stay portable across datasets, teams, and environments. The override parameters
(`active=`, `thresholds=`, `exclude=`, and `columns_map=`) give each pipeline the flexibility to
tailor shared checks without forking the underlying library.

For a complementary approach to reusable validation rules, see
[Data Contracts](../08-contracts-and-pipelines/01-contracts.qmd), which define expectations as
declarative, serializable specifications using the [`Step`](`pointblank.Step`) class. Both
approaches produce the same validation steps at interrogation time. Choose `Steps` for fluent,
code-first composition and `Contract` for declarative, YAML-friendly definitions.


### Validation Reports

```{python}
#| echo: false
#| output: false
import pointblank as pb
```

After interrogating your data with a validation plan, Pointblank automatically generates a
*validation report*. That tabular report comprehensively summarizes the results of all validation
steps. It'll be your primary tool for understanding data quality at a glance, identifying issues,
and communicating results to stakeholders.

Validation reports are [Great Tables](https://github.com/posit-dev/great-tables) objects that
provide rich information about each validation step. It includes: identifying information for the
step, pass/fail statistics, threshold exceedances, and visual status indicators. The report makes it
easy to quickly assess overall data quality and pinpoint specific areas that need attention.

## Viewing the Validation Report

The most straightforward way to view a validation report is to simply print the `Validate` object
after calling `interrogate()`:

```{python}
import pointblank as pb
import polars as pl

# Sample data
data = pl.DataFrame({
    "id": range(1, 11),
    "value": [120, 85, 47, 210, 30, 155, 175, 95, 205, 140],
    "category": ["A", "B", "C", "A", "D", "B", "A", "E", "A", "C"],
    "ratio": [0.5, 0.7, 0.3, 1.2, 0.8, 0.9, 0.4, 1.5, 0.6, 0.2],
})

# Create and interrogate a validation
validation = (
    pb.Validate(data=data, tbl_name="sales_data")
    .col_vals_gt(columns="value", value=50, brief=True)
    .col_vals_in_set(columns="category", set=["A", "B", "C"], brief=True)
    .col_exists(columns=["id", "value"], brief=True)
    .interrogate()
)

# Display the validation report
validation
```

In a notebook or interactive environment, simply typing the validation object name displays the
report automatically. In a script or REPL, you might need to explicitly call
`validation.get_tabular_report().show()` to display the table.

::: {.callout-note}
You can display a validation report even before calling `interrogate()`. The report will show your
validation plan with all the steps you've defined, but it won't contain any interrogation results.
Additionally, validation steps that use column selection patterns (like validating multiple columns
at once) won't be expanded into individual rows yet, as that expansion happens during interrogation.
:::

## Understanding Report Components

The validation report table consists of several key components that work together to provide a
complete picture of your data quality:

#### Report Header

The report header (title and subtitle area) contains important metadata about the validation:

- **Title**: by default, shows "Pointblank Validation" but can be customized
- **Label**: your custom label for the validation (if provided via the `label=` parameter)
- **Table Information**: the table name and type (Polars, Pandas, DuckDB, etc.)
- **Thresholds**: the warning, error, and critical threshold values used

This header information provides essential context for interpreting the validation results,
especially when sharing reports with stakeholders or reviewing historical validations.

#### Report Footer

The report footer contains several pieces of information that provide context and traceability:

**Timestamps**: The footer shows when the interrogation was performed, including the start time,
duration, and end time. This helps track when data quality checks were executed, which is especially
useful when archiving reports or monitoring data quality over time.

**Governance Metadata**: When you provide governance parameters to `Validate`, they are displayed
in the footer as well. This metadata helps document data ownership and dependencies:

- **Owner**: who is responsible for the data quality (e.g., `"data-platform-team"`)
- **Consumers**: who depends on this data (e.g., `["ml-team", "analytics"]`)
- **Version**: the version of the validation plan or data contract (e.g., `"2.1.0"`)

Here's an example showing governance metadata in a validation report:

```{python}
# Example with governance metadata
governance_validation = (
    pb.Validate(
        data=data,
        tbl_name="sales_data",
        label="Sales data validation",
        owner="data-platform-team",
        consumers=["ml-team", "analytics", "finance"],
        version="1.2.0",
    )
    .col_vals_gt(columns="value", value=0)
    .col_vals_in_set(columns="category", set=["A", "B", "C", "D", "E"])
    .interrogate()
)

governance_validation
```

The governance metadata appears below the timestamps in the footer, making it easy to identify who
owns the data, who depends on it, and which version of the validation rules are being applied.

::: {.callout-tip}
Governance metadata is particularly useful in enterprise environments where data lineage and
accountability are important. By including `owner`, `consumers`, and `version` in your validations,
you create self-documenting reports that can be easily understood by anyone reviewing them.
:::

::: {.callout-note}
Throughout this documentation, the footer is hidden in example reports for brevity. This is
controlled through a global option (see the section on controlling header and footer display later
in this guide). In practice, including the footer provides valuable timestamp information for
tracking when validations were executed.
:::

### Report Columns

The validation report table includes the following columns, each providing specific information
about the validation steps:

#### Status Indicator (first column, unlabeled)

The first column is an unlabeled vertical colored bar that provides instant visual feedback about
each step's status:

- **Green**: all test units passed the validation
- **Light green (semi-transparent)**: some test units failed but no thresholds were exceeded
- **Gray**: the 'warning' threshold was exceeded
- **Yellow**: the 'error' threshold was exceeded
- **Red**: the 'critical' threshold was exceeded

This visual indicator allows you to quickly scan the report and identify problem areas.

#### Step Number (second column, unlabeled)

The second column is unlabeled and contains the sequential step number, starting from 1. This number
is used when referencing specific steps in other methods like `get_step_report(i=2)` or when
extracting data from specific validation steps.

#### STEP

The STEP column displays the validation method name along with an icon that visually represents the
type of validation being performed. The validation method indicates what aspect of data quality is
being checked, such as:

- `col_vals_gt()`: column values greater than
- `col_vals_in_set()`: column values in a set
- `col_exists()`: column existence check
- `rows_distinct()`: row uniqueness check
- and many others...

When you provide a brief message (via `brief=True` for auto-generated briefs or `brief="custom
text"` for custom messages), it appears within the STEP column below the validation method name.
These briefs provide human-readable explanations of what each validation step is checking, making
the report more accessible to non-technical stakeholders.

```{python}
# Example showing brief messages in the STEP column
validation_with_briefs = (
    pb.Validate(data=data, tbl_name="sales_data")
    .col_vals_gt(
        columns="value",
        value=50,
        brief="Sales values should always exceed the $50 threshold"
    )
    .col_vals_in_set(
        columns="category",
        set=["A", "B", "C"],
        brief=True  # Auto-generated brief
    )
    .interrogate()
)

validation_with_briefs
```

In the above report, you'll see the custom brief message appear below the `col_vals_gt` method name
in the first step, and an automatically generated brief below `col_vals_in_set` in the second step.

#### COLUMNS

The column(s) being validated in this step. For validation methods that don't target specific
columns (like `row_count_match`), this will show an em dash (—).

#### VALUES

The comparison value(s) or criteria used in the validation. For example:

- for `col_vals_gt(value=100)`, this shows `100`
- for `col_vals_in_set(set=["A", "B", "C"])`, this shows `A | B | C`
- for existence checks, this shows an em dash (—)

#### TBL

Icons indicating whether any preprocessing or segmentation was applied:

- **Table icon**: standard validation on the original data
- **Transformation icon**: preprocessing function was applied via `pre=`
- **Segmentation icon**: data was segmented via `segments=`

These icons help you understand if you're validating transformed or segmented data.

#### EVAL

Indicates whether the validation step was evaluated:

- **Checkmark**: step was successfully evaluated
- **Error icon**: an evaluation error occurred (e.g., column not found)
- **Inactive icon**: step was marked as inactive

This column is crucial for identifying validation steps that couldn't be executed properly.

#### UNITS

The number of units tested in this validation step. A 'test unit' is the atomic unit being
validated, which varies by validation type:

- for column value checks: each cell in the target column(s)
- for row checks: each row
- for table checks: typically 1 (the table itself)

This number is formatted with locale-appropriate thousand separators for readability. Also, since
space is limited, values are often abbreviated so a figure like 43,534 will appear as `43.5K`.

#### PASS

The number and fraction of test units that passed the validation, displayed as:

```
n_passed
f_passed
```

For example, the cell with

```
8
0.80
```

means 8 test units passed out of the total, representing an 80% success rate (though `f_passed` is
always expressed as a fractional value from `0` to `1`).

#### FAIL

The number and fraction of test units that failed the validation, displayed similarly to PASS:

```
n_failed
f_failed
```

For example, the cell with

```
2
0.20
```

means 2 test units failed, representing a 20% failure rate from a fractional value of `0.20`. Note
that this fractional `f_failed` value is what's used to set failure thresholds for 'warning',
'error', and 'critical' states.

#### W, E, C (Warning, Error, Critical)

Three columns showing whether each threshold level was exceeded for the three different states.

- **Long dash**: threshold wasn't set for a state
- **Empty colored circle**: threshold was set but wasn't exceeded for a given state
- **Filled colored circle**: threshold was set and exceeded

In terms of colors, the 'warning' state is gray, the 'error' state is yellow, and the 'critical'
state is red.

Having visual indicators makes it easy to identify which validation steps have crossed into warning,
error, or critical territory.

#### EXT

Indicates whether failing row data was extracted for this step:

- **Em dash (—)**: no extract available
- **Download button**: click to download failing rows as CSV

When extracts are available, you can download them directly from the report for further analysis or
to share with data stewards who need to fix the issues.

## Understanding Validation Status

The validation report helps you quickly understand the overall status of your data:

- **All green status indicators**: all validations passed completely
- **Light green indicators**: minor failures below warning threshold
- **Gray, yellow, or red indicators**: threshold exceedances requiring attention
- **Error icons in EVAL column**: validation steps that couldn't be evaluated

By scanning the status indicators column, you can immediately identify which validation steps need
attention and prioritize your data quality efforts accordingly.

## Customizing the Report Title

You can customize the validation report's title using the `title=` parameter in
[`get_tabular_report()`](`Validate.get_tabular_report`). This is particularly useful when generating
multiple reports or when you want to provide more context:

```{python}
# Default title
validation.get_tabular_report()
```

```{python}
# Use the table name as the title
validation.get_tabular_report(title=":tbl_name:")
```

```{python}
# Provide a custom title (supports Markdown)
validation.get_tabular_report(title="**Sales Data** Quality Report")
```

```{python}
# No title
validation.get_tabular_report(title=":none:")
```

The title customization options are:

- `":default:"` (default): shows `"Pointblank Validation"`
- `":tbl_name:"`: uses the table name from `tbl_name=` parameter
- `":none:"`: hides the title completely
- Any string: custom title text (Markdown is supported)

## Customizing with Great Tables

Since the validation report is a Great Tables object, you can leverage the full power of Great
Tables to customize its appearance. This allows you to match your organization's branding, highlight
specific information, or adjust the presentation for different audiences.

### Guide to Internal Column Names

When working with Great Tables methods to customize the validation report, you'll need to use the
*internal column names* rather than the display labels you see in the rendered table. This is
because Great Tables operates on the underlying data table structure, where columns have technical
names that differ from their user-facing labels.

For example, the column labeled `"STEP"` in the report is actually stored internally as
`"type_upd"`, and the unlabeled step-number column is internally named `"i"`. Most Great Tables
methods that target specific columns (like `tab_style()`, `cols_width()`, `cols_hide()`, etc.)
require these internal names.

Here's the complete mapping from display labels to internal column names:

1. Status indicator (no label): `"status_color"`
2. Step number (no label): `"i"`
3. `STEP`: `"type_upd"`
4. `COLUMNS`: `"columns_upd"`
5. `VALUES`: `"values_upd"`
6. `TBL`: `"tbl"`
7. `EVAL`: `"eval"`
8. `UNITS`: `"test_units"`
9. `PASS`: `"pass"`
10. `FAIL`: `"fail"`
11. `W`: `"w_upd"`
12. `E`: `"e_upd"`
13. `C`: `"c_upd"`
14. `EXT`: `"extract_upd"`

Always use these internal names when calling Great Tables methods. Using the display labels (like
`"STEP"` or `"UNITS"`) will result in errors since these labels only exist in the rendered output,
not in the underlying data structure.

In the examples that follow, you'll see how to use these internal column names to customize various
aspects of the validation report.

### Adding Custom Styling

You can apply custom styles to the report table:

```{python}
from great_tables import style, loc

# Get the report as a Great Tables object
report = validation.get_tabular_report()

# Add custom styling using internal column names
report = (
    report
    .tab_style(
        style=style.fill(color="#F0F8FF"),
        locations=loc.body(columns="i")  # Internal name for step number
    )
    .tab_style(
        style=style.text(weight="bold"),
        locations=loc.body(columns="type_upd")  # Internal name for STEP
    )
)

report
```

### Modifying Column Widths

Adjust column widths to optimize the layout:

```{python}
report = (
    validation
    .get_tabular_report()
    .cols_width(
        cases={
            "status_color": "20px", # Status indicator column
            "i": "40px",            # Step number column
            "type_upd": "170px",    # STEP column
            "columns_upd": "100px", # COLUMNS column
        }
    )
)

report
```

### Hiding Columns

Hide specific columns that aren't relevant for your audience:

```{python}
# Hide the TBL and EVAL columns for a cleaner presentation (using internal names)
report = (
    validation
    .get_tabular_report()
    .cols_hide(columns=["tbl", "eval"])  # Use internal column names
)

report
```

### Adding a Source Note

Add information about data source or validation context:

```{python}
report = (
    validation
    .get_tabular_report()
    .tab_source_note(
        source_note="Data validated on 2025-10-10 | Production database snapshot"
    )
)

report
```

## Exporting the Report

Great Tables provides multiple export options for sharing validation reports:

```python
# Save as a standalone HTML file
validation.get_tabular_report().write_raw_html("validation_report.html")

# Save as a PNG image
validation.get_tabular_report().save("validation_report.png")

# Open in browser
validation.get_tabular_report().show("browser")
```

## Controlling Header and Footer Display

You can control whether the header and footer appear in the validation report:

```{python}
# Hide the footer
validation.get_tabular_report(incl_footer=False)
```

```{python}
# Hide the header
validation.get_tabular_report(incl_header=False)
```

```{python}
# Hide both
validation.get_tabular_report(incl_header=False, incl_footer=False)
```

You can also set these preferences globally using `pb.config()`:

```python
# Set global preferences
pb.config(report_incl_header=True, report_incl_footer_timings=False)
```

## Best Practices for Validation Reports

Here are some guidelines for creating effective validation reports:

#### 1. Use Descriptive Table Names and Labels

Provide meaningful names and labels to make reports self-documenting:

```python
validation = pb.Validate(
    data=sales_df,
    tbl_name="Q3_2025_sales",
    label="Quarterly sales data validation for financial reporting"
)
```

#### 2. Include Governance Metadata for Accountability

Add ownership and dependency information for enterprise data governance:

```python
validation = pb.Validate(
    data=sales_df,
    tbl_name="Q3_2025_sales",
    label="Quarterly sales data validation",
    owner="data-platform-team",
    consumers=["ml-team", "analytics", "finance"],
    version="2.1.0"
)
```

This creates a clear record of who is responsible for the data, who depends on it, and which version
of the validation rules are being applied.

#### 3. Add Brief Messages for Stakeholder Reports

When sharing reports with non-technical stakeholders, always include briefs:

```python
.col_vals_between(
    columns="price",
    left=0, right=10000,
    brief="Product prices must be between $0 and $10,000"
)
```

#### 4. Set Appropriate Thresholds

Configure thresholds that align with your data quality requirements:

```python
validation = pb.Validate(
    data=data,
    tbl_name="customer_data",
    thresholds=pb.Thresholds(
        warning=0.01,  # 1% failure triggers warning
        error=0.05,    # 5% failure triggers error
        critical=0.10  # 10% failure triggers critical
    )
)
```

#### 5. Customize for Your Audience

Tailor the report presentation to your audience:

- **Technical teams**: include all columns, show preprocessing indicators
- **Management**: hide technical columns, emphasize status indicators
- **Data stewards**: include extract download buttons, detailed briefs

#### 6. Combine with Other Reporting Tools

Use validation reports alongside other Pointblank features:

- **Step reports**: drill down into specific failing steps with `get_step_report()`
- **Extracts**: use `get_data_extracts()` to get all failing data for analysis
- **Sundered data**: use `get_sundered_data()` to split data into passing/failing sets

#### 7. Archive Reports for Trend Analysis

Save validation reports over time to track data quality trends:

```python
from datetime import datetime

# Save with timestamp
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
validation.get_tabular_report().write_raw_html(f"validation_report_{timestamp}.html")
```

## Conclusion

The validation report is your primary interface for understanding data quality after running a
validation. By providing a comprehensive overview of all validation steps, visual status indicators,
and detailed statistics, it enables you to:

- quickly assess overall data quality across multiple dimensions
- identify specific validation steps that need attention
- communicate data quality status to technical and non-technical stakeholders
- track threshold exceedances and their severity levels
- access failing data through extract downloads

Combined with customization options from Great Tables, you can create reports that perfectly match
your organization's needs and workflows. Whether you're validating data in an interactive notebook,
generating automated quality reports, or presenting findings to stakeholders, the validation report
provides the clarity and detail you need to maintain high data quality standards.


### Step Reports

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

While validation reports provide a comprehensive overview of all validation steps, sometimes you
need to focus on a specific validation step in greater detail. This is where *step reports* come in.
A step report is a detailed examination of a single validation step, providing in-depth information
about the test units that were validated and their pass/fail status.

Step reports are especially useful when debugging validation failures, investigating problematic
data, or communicating detailed findings to colleagues who are responsible for specific data quality
issues.

## Creating a Step Report

To create a step report, you first need to run a validation and then use the
[`Validate.get_step_report()`](`Validate.get_step_report`) method, specifying which validation step you want to examine:

```{python}
import pointblank as pb
import polars as pl

# Sample data as a Polars DataFrame
data = pl.DataFrame({
    "id": range(1, 11),
    "value": [10, 20, 3, 35, 50, 2, 70, 8, 20, 4],
    "category": ["A", "B", "C", "A", "D", "F", "A", "E", "H", "G"],
    "ratio": [0.5, 0.7, 0.3, 1.2, 0.8, 0.9, 0.4, 1.5, 0.6, 0.2],
    "status": ["active", "active", "inactive", "active", "inactive",
               "active", "inactive", "active", "active", "inactive"]
})

# Create a validation
validation = (
    pb.Validate(data=data, tbl_name="example_data")
    .col_vals_gt(
        columns="value",
        value=10
    )
    .col_vals_in_set(
        columns="category",
        set=["A", "B", "C"]
    )
    .interrogate()
)

# Get step report for the second validation step (i=2)
step_report = validation.get_step_report(i=2)

step_report
```

In this example, we first create and interrogate a validation object with two steps. We then
generate a step report for the second validation step (`i=2`), which checks if the values in the
`category` column are in the set `["A", "B", "C"]`.

Note that step numbers in Pointblank start at `1`, matching what you see in the validation report's
`STEP` column (i.e., not 0-based indexing). So the first step is referred to with `i=1`, the second
step with `i=2`, and so on.

## Understanding Step Report Components

A step report consists of several key components that provide detailed information about the
validation step:

1. Header: displays the validation step number, type of validation, and a brief description
2. Table Body: presents either the failing rows, a sample of completely passing data, or an
expected/actual comparison (for a [`Validate.col_schema_match()`](`Validate.col_schema_match`) step)

The step report table highlights passing and failing rows, making it easy to identify problematic
data points. This is especially useful for diagnosing issues when dealing with large datasets.

## Different Types of Step Reports

It's important to note that step reports vary in appearance and structure depending on the type of
validation method used:

- Value-based validations (like [`Validate.col_vals_gt()`](`Validate.col_vals_gt`), [`Validate.col_vals_in_set()`](`Validate.col_vals_in_set`)): show
individual rows that failed validation
- Uniqueness checks ([`Validate.rows_distinct()`](`Validate.rows_distinct`)): group together the duplicate records in order of
appearance
- Schema validations ([`Validate.col_schema_match()`](`Validate.col_schema_match`)): display column-level information about
expected vs. actual data types
- Aggregate and table-level validations (like [`Validate.col_sum_gt()`](`Validate.col_sum_gt`),
[`Validate.row_count_match()`](`Validate.row_count_match`), [`Validate.col_exists()`](`Validate.col_exists`),
[`Validate.col_pct_null()`](`Validate.col_pct_null`), and [`Validate.data_freshness()`](`Validate.data_freshness`)):
show a one-row summary of the measured value next to the expected one

Additionally, step reports for value-based validations and uniqueness checks operate in two distinct
modes:

1. When errors are present: The report shows only the failing rows and, for value-based validations,
clearly highlights the column under study
2. When no errors exist: The report header clearly indicates success, and a sample of the data is
shown (along with the studied column highlighted, for value-based validations)

This variation in reporting style allows step reports to effectively communicate the specific type
of validation being performed and display relevant information in the most appropriate format. When
you're working with different validation types, expect to see different step report layouts
optimized for each context.

### Value-Based Validation Step Reports

Value-based step reports focus on showing individual rows where values in the target column failed
the validation check. These reports highlight the specific column being validated and clearly
display which values violated the condition.

```{python}
# Create sample data with some validation failures
data = pl.DataFrame({
    "id": range(1, 8),
    "value": [120, 85, 47, 210, 30, 10, 5],
    "category": ["A", "B", "C", "A", "D", "B", "E"]
})

# Create a validation with a value-based check
validation_values = (
    pb.Validate(data=data, tbl_name="sales_data")
    .col_vals_gt(
        columns="value",
        value=50,
        brief="Sales values should exceed $50"
    )
    .interrogate()
)

# Display the step report for the value-based validation
validation_values.get_step_report(i=1)
```

This report clearly identifies which rows contain values that don't meet our threshold, making it
easy to investigate these specific data points.

### Uniqueness Validation Step Reports

Uniqueness checks produce a different type of step report that groups duplicate records together.
This format makes it easy to identify patterns in duplicate data.

```{python}
# Create sample data with some duplicate rows based on the combination of columns
data = pl.DataFrame({
    "customer_id": [101, 102, 103, 101, 104, 105, 102],
    "order_date": ["2023-01-15", "2023-01-16", "2023-01-16",
                   "2023-01-15", "2023-01-17", "2023-01-18", "2023-01-19"],
    "product": ["Laptop", "Phone", "Tablet", "Laptop",
                "Monitor", "Keyboard", "Headphones"]
})

# Create a validation checking for unique customer-product combinations
validation_duplicates = (
    pb.Validate(data=data, tbl_name="order_data")
    .rows_distinct(
        columns_subset=["customer_id", "product"],
        brief="Customer should not order the same product twice"
    )
    .interrogate()
)

# Display the step report for the uniqueness validation
validation_duplicates.get_step_report(i=1)
```

The report organizes duplicate records together, making it easy to see which combinations are
repeated and how many times they appear.

### Schema Validation Step Reports

Schema validation step reports have a completely different structure, comparing expected versus
actual column data types and presence.

```{python}
schema = pb.Schema(
    columns=[
        ("date_time", "timestamp"),
        ("dates", "date"),
        ("a", "int64"),
        ("b",),
        ("c",),
        ("d", "float64"),
        ("e", ["bool", "boolean"]),
        ("f", "str"),
    ]
)

validation_schema = (
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="duckdb"),
        tbl_name="small_table",
        label="Step report for a schema check"
    )
    .col_schema_match(schema=schema)
    .interrogate()
)

# Display the step report for the schema validation
validation_schema.get_step_report(i=1)
```

This report style focuses on comparing the expected schema against the actual table structure,
highlighting mismatches in data types or missing/extra columns. The table format makes it easy to
see exactly where the schema expectations differ from reality.

### Summary Step Reports

Validations that check the table (or a column) as a whole, rather than row by row, produce a compact
summary step report. It's a single-row table that puts the measured value (e.g., the row count, the
percentage of null values, or the age of the most recent data) beside the expected value, along
with a pass/fail indicator.

```{python}
validation_summary = (
    pb.Validate(
        data=pb.load_dataset(dataset="small_table", tbl_type="polars"),
        tbl_name="small_table",
    )
    .row_count_match(count=15, tol=1)
    .col_pct_null(columns="c", p=0.1)
    .interrogate()
)

# Display the step report for the row count check
validation_summary.get_step_report(i=1)
```

The row count is outside of the tolerated range, so the report shows both counts and how far apart
they are. The second step's report gives the number and percentage of null values in column `c`:

```{python}
validation_summary.get_step_report(i=2)
```

The same kind of report is available for [`Validate.col_exists()`](`Validate.col_exists`) (where a
missing column's closest match in the table is suggested),
[`Validate.col_count_match()`](`Validate.col_count_match`),
[`Validate.col_pct_missing()`](`Validate.col_pct_missing`),
[`Validate.data_freshness()`](`Validate.data_freshness`), and
[`Validate.specially()`](`Validate.specially`) (which summarizes the passing and failing test
units).

## Customizing Step Reports

Step reports can be customized with several parameters to better focus your analysis and tailor the
output to your specific needs. The [`Validate.get_step_report()`](`Validate.get_step_report`) method offers multiple
customization options to help you create more effective reports.

When a dataset has many columns, you might want to focus on just those relevant to your analysis.
You can create a step report containing only a subset of the columns in the target table:

```{python}
validation.get_step_report(
    i=2,

    # Only show these columns ---
    columns_subset=["id", "category", "status"]
)
```

This approach makes step reports much easier to interpret by highlighting just the essential columns
that help understand the validation failures.

For large datasets with many failing rows, you might want to use `limit=` to set a cap on the number
of rows shown in the report:

```{python}
validation.get_step_report(
    i=2,

    # Only show up to 2 failing rows ---
    limit=2
)
```

The report header can also be extensively customized to provide more specific context. You can
replace the default header with plain text or Markdown formatting:

```{python}
validation.get_step_report(
    i=2,
    header="Category Values Validation: *Critical Analysis*"
)
```

For more advanced header customization, you can use the templating system with the `{title}` and
`{details}` elements to retain parts of the default header while adding your own content. The
`{title}` template is the default title whereas `{details}` provides information on the assertion,
number of failures, etc. Let's move away from the default template of `{title}{details}` and provide
a custom title to go with the details text:

```{python}
validation.get_step_report(
    i=2,
    header="Custom Category Validation Report {details}"
)
```

We can keep `{title}` and `{details}` and add some more context in between the two:

```{python}
validation.get_step_report(
    i=2,
    header=(
        "{title}<br>"
        "<span style='font-size: 0.75em;'>"
        "This validation is critical for our data quality standards."
        "</span><br>"
        "{details}"
    )
)
```

You could always use more HTML and CSS to do *a lot* of customization:

```{python}
validation.get_step_report(
    i=2,
    header=(
        "VALIDATION SUMMARY\n\n{details}\n\n"
        "<hr style='color: lightblue;'>"
        "<div style='font-size: smaller; padding-bottom: 5px; text-transform: uppercase'>"
        "{title}"
        "</div>"
    )
)
```

If you prefer no header at all, simply set `header=None`:

```{python}
validation.get_step_report(
    i=2,
    header=None
)
```

These customization options can be combined to create highly focused reports tailored to specific
needs:

```{python}
validation.get_step_report(
    i=2,
    columns_subset=["id", "category"],
    header="*Category Validation:* Top Issues",
    limit=2
)
```

Through these customization options, you can craft step reports that effectively communicate the
most important information to different audiences. Technical teams might benefit from seeing all
columns but with a limited number of examples. Business stakeholders might prefer a focused view
with only the most relevant columns. For documentation purposes, custom headers provide important
context about what's being validated.

Remember that customizing your step reports is about more than aesthetics: it's about making complex
validation information more accessible and actionable for all stakeholders involved in data quality.

## Using Step Reports for Data Investigation

Step reports can be powerful tools for investigating data quality issues. Let's look at a more
complex example:

```{python}
# Create a more complex dataset with multiple issues
complex_data = pl.DataFrame({
    "id": range(1, 11),
    "value": [10, 20, 3, 40, 50, 2, 70, 80, 90, 7],
    "ratio": [0.1, 0.2, 0.3, 1.4, 0.5, 0.6, 0.7, 0.8, 1.2, 0.9],
    "category": ["A", "B", "C", "A", "D", "B", "A", "C", "B", "E"]
})

# Create a validation with multiple steps
validation_complex = (
    pb.Validate(data=complex_data, tbl_name="complex_data")
    .col_vals_gt(columns="value", value=10)
    .col_vals_le(columns="ratio", value=1.0)
    .col_vals_in_set(columns="category", set=["A", "B", "C"])
    .interrogate()
)

# Get step report for the ratio validation (step 2)
ratio_report = validation_complex.get_step_report(i=2)

ratio_report
```

In this example, we're investigating issues with the `ratio` column by generating a step report
specifically for that validation step. The step report shows exactly which rows have values that
exceed our maximum threshold of `1.0`.

## Combining Step Reports with Extracts

For more advanced analysis, you can extract the actual data from a step report into a DataFrame:

```{python}
# Extract the data from the step report
failing_ratios = validation_complex.get_data_extracts(i=2)

failing_ratios
```

This extracts the failing rows from the validation step, which you can then further analyze or fix
as needed. Note that the parameter `i=2` corresponds directly to the step number shown in the
validation report; it's the same numbering system used for [`Validate.get_step_report()`](`Validate.get_step_report`).

These extracts are particularly valuable for analysts who need to:

- perform additional calculations on problematic data
- feed failing records into correction pipelines
- create visualizations of data patterns that led to validation failures
- export problem records to share with data owners

It's worth noting that the validation report itself includes export buttons on the far right of each
row that allow you to download CSV files of the failing data directly. This serves as a convenient
delivery mechanism for sharing extracts with colleagues who may not be working in Python, making the
validation report not just a visual tool but also a practical means of distributing problematic data
for further investigation.

## Step Reports with Segmented Data

When working with segmented validation, step reports become even more valuable as they allow you to
investigate issues within specific segments:

```{python}
# Create data with different regions
segmented_data = pl.DataFrame({
    "id": range(1, 10),
    "value": [10, 20, 3, 40, 50, 2, 6, 8, 60],
    "region": ["North", "North", "South", "South", "East", "East", "West", "West", "West"]
})

# Create a validation with segments
segmented_validation = (
    pb.Validate(data=segmented_data, tbl_name="regional_data")
    .col_vals_gt(
        columns="value",
        value=10,
        segments="region"  # Segment by region
    )
    .interrogate()
)

# Get step report for a specific segment (the 'West' region)
# For segmented validations, each segment gets its own step number
north_report = segmented_validation.get_step_report(i=4)

north_report
```

For segmented validations, each segment is treated as a separate validation step with its own step
number. This allows you to investigate issues specific to each data segment using the appropriate
step number from the validation report.

## Best Practices for Using Step Reports

Here are some guidelines for effectively using step reports in your data validation workflow:

1. Generate step reports selectively: create reports only for steps that require detailed
investigation rather than for all steps

2. Use the `limit=` parameter for large datasets: when working with large datasets, focus only
on a subset of failing rows to avoid information overload

3. Share specific step reports with stakeholders: when collaborating with domain experts, share
relevant step reports to help them understand and address specific data quality issues (and
customize the header to improve clarity)

4. Combine with extracts for deeper analysis: use the [`Validate.get_data_extracts()`](`Validate.get_data_extracts`) method to
extract the failing rows for further analysis or correction

5. Document findings from step reports: when you discover patterns or insights from step reports,
document them to inform future data quality improvements

Remember that step reports are most valuable when used strategically as part of a broader data
quality framework. By following these best practices, you can use step reports not just for
troubleshooting, but to develop a deeper understanding of your data's characteristics and quality
patterns over time. This approach transforms step reports from simple debugging tools into strategic
assets for continuous data quality improvement.

## Conclusion

Step reports provide a focused lens into specific validation steps, allowing you to investigate data
quality issues in detail. By generating targeted reports for specific validation steps, you can:

- pinpoint exactly which data points are causing validation failures
- communicate specific issues to relevant stakeholders
- gather insights that might be missed in the aggregate validation report
- track improvements in specific aspects of data quality over time

Whether you're debugging validation failures, investigating edge cases, or communicating specific
data quality issues to colleagues, step reports can give you the detailed information you need to
understand and resolve data quality problems effectively.


### Data Extracts

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

When validating data, identifying exactly which rows failed is critical for diagnosing and resolving
data quality issues. This is where *data extracts* come in. Data extracts consist of target table
rows containing at least one cell that failed validation. While the validation report provides an
overview of pass/fail statistics, data extracts give you the actual problematic records for deeper
investigation.

This article will cover:

- which validation methods collect data extracts
- multiple ways to access and work with data extracts
- practical examples of using extracts for data quality improvement
- advanced techniques for analyzing extract patterns

## The Validation Methods that Work with Data Extracts

The following validation methods operate on column values and will have rows extracted when there
are failing test units in those rows:

- [`Validate.col_vals_gt()`](`Validate.col_vals_gt`)
- [`Validate.col_vals_lt()`](`Validate.col_vals_lt`)
- [`Validate.col_vals_ge()`](`Validate.col_vals_ge`)
- [`Validate.col_vals_le()`](`Validate.col_vals_le`)
- [`Validate.col_vals_eq()`](`Validate.col_vals_eq`)
- [`Validate.col_vals_ne()`](`Validate.col_vals_ne`)
- [`Validate.col_vals_between()`](`Validate.col_vals_between`)
- [`Validate.col_vals_outside()`](`Validate.col_vals_outside`)
- [`Validate.col_vals_in_set()`](`Validate.col_vals_in_set`)
- [`Validate.col_vals_not_in_set()`](`Validate.col_vals_not_in_set`)
- [`Validate.col_vals_in_table()`](`Validate.col_vals_in_table`)
- [`Validate.col_vals_null()`](`Validate.col_vals_null`)
- [`Validate.col_vals_not_null()`](`Validate.col_vals_not_null`)
- [`Validate.col_vals_regex()`](`Validate.col_vals_regex`)
- [`Validate.col_vals_expr()`](`Validate.col_vals_expr`)
- [`Validate.conjointly()`](`Validate.conjointly`)

These row-based validation methods will also have rows extracted should there be failing rows:

- [`Validate.rows_distinct()`](`Validate.rows_distinct`)
- [`Validate.rows_complete()`](`Validate.rows_complete`)

Note that some validation methods like [`Validate.col_exists()`](`Validate.col_exists`) and [`Validate.col_schema_match()`](`Validate.col_schema_match`)
don't generate data extracts because they validate structural aspects of the table rather than
checking column values.

## Accessing Data Extracts

There are three primary ways to access data extracts in Pointblank:

1. the **CSV** buttons in validation reports
2. through the [`Validate.get_data_extracts()`](`Validate.get_data_extracts`) method
3. inspecting a subset of failed rows in step reports

Let's explore each approach using examples.

### CSV Data from Validation Reports

Data extracts are embedded within validation report tables. Let's look at an example, using the
`small_table` dataset, where data extracts are collected in a single validation step due to failing
test units:

```{python}
import pointblank as pb

validation = (
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_lt( columns="d", value=3000)
    .interrogate()
)

validation
```

The single validation step checks whether values in `d` are less than `3000`. Within that column,
values range from `108.34` to `9999.99` so it makes sense that we can see 4 failing test units in
the `FAIL` column.

If you look at the far right of the validation report you'll find there's a `CSV` button. Pressing
it initiates the download of a CSV file, and that file contains the data extract for this validation
step. The `CSV` button only appears when:

1. there is a non-zero number of failing test units
2. the validation step is based on the use of a column-value or a row-based validation method (the
methods outlined in the section entitled *The Validation Methods that Work with Data Extracts*)

Access to CSV data for the test unit errors is useful when the validation report is shared with
other data quality stakeholders, since it is easily accessible and doesn't require further use of
Pointblank. The stakeholder can simply open the downloaded CSV in their preferred spreadsheet
software, import it into a different analysis environment like R or Julia, or process it with any
tool that supports CSV files. This cross-platform compatibility makes the CSV export particularly
valuable in mixed-language data teams where different members might be working with different tools.

### `get_data_extracts()`

For programmatic access to data extracts, Pointblank provides the [`Validate.get_data_extracts()`](`Validate.get_data_extracts`)
method. This allows you to work with extract data directly in your Python workflow:

```{python}
# Get data extracts from step 1
extract_1 = validation.get_data_extracts(i=1, frame=True)

extract_1
```

The extracted table is of the same type (a Polars DataFrame) as the target table. Previously we used
`load_dataset()` with the `tbl_type="polars"` option to fetch the dataset in that form.

Note these important details about using [`Validate.get_data_extracts()`](`Validate.get_data_extracts`):

- the parameter `i=1` corresponds to the step number shown in the validation report (1-indexed, not
0-indexed)
- setting `frame=True` returns the data as a DataFrame rather than a dictionary (only works when `i`
is a single integer)
- the extract includes all columns from the original data, not just the column being validated
- an additional `_row_num_` column is added to identify the original row positions

### Step Reports

Step reports provide another way to access and visualize failing data. When you generate a step
report for a validation step that has failing rows, those failing rows are displayed directly in the
report:

```{python}
# Get a step report for the first validation step
step_report = validation.get_step_report(i=1)

step_report
```

Step reports offer several advantages for working with data extracts as they:

1. provide immediate visual context by highlighting the specific column being validated
2. format the data for better readability, especially useful when sharing results with colleagues
3. include additional metadata about the validation step and failure statistics

For steps with many failures, you can customize how many rows to display:

```{python}
# Limit to just 2 rows of failing data
limited_report = validation.get_step_report(i=1, limit=2)

limited_report
```

Step reports are particularly valuable when you want to quickly inspect the failing data without
extracting it into a separate DataFrame. They provide a bridge between the high-level validation
report and the detailed data extracts.

## Viewing Data Extracts with `preview()`{.qd-no-link}

To get a consistent HTML representation of any data extract (regardless of the table type), we can
use the `preview()` function:

```{python}
pb.preview(data=extract_1)
```

The view is optimized for readability, with column names and data types displayed in a compact
format. Notice that the `_row_num_` column is now part of the table stub and doesn't steal focus
from the table's original columns.

The `preview()` function is designed to provide the head and tail (5 rows each) of the table so very
large extracts won't overflow the display.

## Working with Multiple Validation Steps

When validating data with multiple steps, you can extract failing rows from any step or combine
extracts from multiple steps:

```{python}
# Create a validation with multiple steps
multi_validation = (
    pb.Validate(data=pb.load_dataset(dataset="small_table", tbl_type="polars"))
    .col_vals_gt(columns="a", value=3)                                  # Step 1
    .col_vals_lt(columns="d", value=3000)                               # Step 2
    .col_vals_regex(columns="b", pattern="^[0-9]-[a-z]{3}-[0-9]{3}$")   # Step 3
    .interrogate()
)

multi_validation
```

### Extracting Data from a Specific Step

You can access extracts from any specific validation step:

```{python}
# Get extracts from step 2 (`d < 3000` validation)
less_than_failures = multi_validation.get_data_extracts(i=2, frame=True)

less_than_failures
```

Using `frame=True` means that returned value will be a DataFrame (not a dictionary that contains a
single DataFrame).

If a step has no failing rows, an empty DataFrame will be returned:

```{python}
# Get extracts from step 3 (regex check)
regex_failures = multi_validation.get_data_extracts(i=3, frame=True)

regex_failures
```

### Getting All Extracts at Once

To retrieve extracts from all steps with failures in one command:

```{python}
# Get all extracts ()
all_extracts = multi_validation.get_data_extracts()

# Display the step numbers that have extracts
print(f"Steps with data extracts: {list(all_extracts.keys())}")
```

A dictionary of DataFrames is returned and only steps with failures will appear in this dictionary.

### Getting Specific Extracts

You can also retrieve data extracts from several specified steps as a dictionary:

```{python}
# Get extracts from steps 1 and 2 as a dictionary
extract_dict = multi_validation.get_data_extracts(i=[1, 2])

# The keys are the step numbers
print(f"Dictionary keys: {list(extract_dict.keys())}")

# Get the number of failing rows in each extract
for step, extract in extract_dict.items():
    print(f"Step {step}: {len(extract)} failing rows")
```

Note that `frame=True` cannot be used when retrieving multiple extracts.

## Applications of Data Extracts

Once you have extracted the failing data, there are numerous ways to analyze and use this
information to improve data quality. Let's explore some practical applications.

### Finding Patterns Across Validation Steps

You can analyze patterns across different validation steps by combining extracts:

```{python}
# Get a consolidated view of all rows that failed any validation
all_failure_rows = set()
for step, extract in all_extracts.items():
    if len(extract) > 0:
        all_failure_rows.update(extract["_row_num_"])

print(f"Total unique rows with failures: {len(all_failure_rows)}")
print(f"Row numbers with failures: {sorted(all_failure_rows)}")
```

### Identifying Rows with Multiple Failures

You might want to find rows that failed multiple validation checks, as these often represent more
serious data quality issues:

```{python}
# Get row numbers from each extract
step1_rows = set(multi_validation.get_data_extracts(i=1, frame=True)["_row_num_"])
step2_rows = set(multi_validation.get_data_extracts(i=2, frame=True)["_row_num_"])

# Find rows that failed both validations
common_failures = step1_rows.intersection(step2_rows)
print(f"Rows failing both step 1 and step 2: {common_failures}")
```

### Statistical Analysis of Failing Values

Once you have data extracts, you can perform statistical analysis to identify patterns in the
failing data:

```{python}
# Get extracts from step 2
d_value_failures = multi_validation.get_data_extracts(i=2, frame=True)

# Basic statistical analysis of the failing values
if len(d_value_failures) > 0:
    print(f"Min failing value: {d_value_failures['d'].min()}")
    print(f"Max failing value: {d_value_failures['d'].max()}")
    print(f"Mean failing value: {d_value_failures['d'].mean()}")
```

These analysis techniques help you thoroughly investigate data quality issues by examining failing
data from multiple perspectives. Rather than treating failures as isolated incidents, you can
identify patterns that might indicate systematic problems in your data pipeline.

### Detailed Analysis with `col_summary_tbl()`{.qd-no-link}

For a more comprehensive view of the statistical properties of your extract data, you can use the
`col_summary_tbl()` function:

```{python}
# Get extracts from step 2
d_value_failures = multi_validation.get_data_extracts(i=2, frame=True)

# Generate a comprehensive statistical summary of the failing data
pb.col_summary_tbl(d_value_failures)
```

This statistical overview provides:

1. a count of values (including missing values)
2. type information for each column
3. distribution metrics like min, max, mean, and quartiles for numeric columns
4. frequency of common values for categorical columns
5. missing value counts and proportions

Using `col_summary_tbl()` on data extracts lets you quickly understand the characteristics of
failing data without writing custom analysis code. This approach is particularly valuable when:

- You need to understand the statistical properties of failing records
- You want to compare distributions of failing vs passing data
- You're looking for anomalies or unexpected patterns within the failing rows

For example, if values failing a validation check are concentrated at certain quantiles or have an
unusual distribution shape, this might indicate a systematic data collection or processing issue
rather than random errors.

## Using Extracts for Data Quality Improvement

Data extracts are especially valuable for:

1. **Root Cause Analysis**: examining the full context of failing rows to understand why they failed
2. **Data Cleaning**: creating targeted cleanup scripts that focus only on problematic records
3. **Feedback Loops**: sharing specific examples with data providers to improve upstream quality
4. **Pattern Recognition**: identifying systemic issues by analyzing groups of failing records

Here's an example of using extracts to create a corrective action plan:

```{python}
import polars as pl

# Create a new sample of an extract DF
sample_extract = pl.DataFrame({
    "id": range(1, 11),
    "value": [3500, 4200, 3800, 9800, 5500, 7200, 8300, 4100, 7600, 3200],
    "category": ["A", "B", "A", "C", "B", "A", "C", "B", "A", "B"],
    "region": [
        "South", "South", "North", "East", "South",
        "South", "East", "South", "West", "South"
    ]
})

# Identify which regions have the most failures
region_counts = (
    sample_extract
    .group_by("region")
    .agg(pl.len().alias("failure_count"))
    .sort("failure_count", descending=True)
)

region_counts
```

Analysis shows that 6 out of 10 failing records (60%) are from the `"South"` region, making it the
highest priority area for data quality investigation. This suggests a potential systemic issue with
data collection or processing in that specific region.

## Best Practices for Working with Data Extracts

When incorporating data extracts into your data quality workflow:

1. Use extracts for investigation, not just reporting: the real value is in the insights you gain
from analyzing the problematic data

2. Combine with other Pointblank features: data extracts work well with step reports and can inform
threshold settings for future validations

3. Consider sampling for very large datasets: if your extracts contain thousands of rows, focus your
investigation on a representative sample

4. Look beyond individual validation steps: cross-reference extracts from different steps to
identify complex issues that span multiple validation rules

5. Document patterns in failing data: record and share insights about common failure modes to build
organizational knowledge about data quality issues.

By integrating these practices into your data validation workflow, you'll transform data extracts
from simple error lists into powerful diagnostic tools. The most successful data quality initiatives
treat extracts as the starting point for investigation rather than the end result of validation.
When systematically analyzed and documented, patterns in failing data can reveal underlying issues
in data systems, collection methods, or business processes that might otherwise remain hidden.
Remember that the ultimate goal isn't just to identify problematic records, but to use that
information to implement targeted improvements that prevent similar issues from occurring in the
future.

## Conclusion

Data extracts bridge the gap between high-level validation statistics and the detailed context
needed to fix data quality issues. By providing access to the actual failing records, Pointblank
enables you to:

- pinpoint exactly which data points caused validation failures
- understand the full context around problematic values
- develop targeted strategies for data cleanup and quality improvement
- communicate specific examples to stakeholders

Whether you're accessing extracts through CSV downloads, the [`Validate.get_data_extracts()`](`Validate.get_data_extracts`)
method, or step reports, this feature provides the detail needed to move from identifying problems
to implementing solutions.


### Sundering Validated Data

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

Sundering data? First off, let's get the correct meaning across here. Sundering is really just
splitting, dividing, cutting into two pieces. And it's a useful thing we can do in Pointblank to any
data that we are validating. When you interrogate the data, you learn about which rows have test
failures within them. With more validation steps, we get an even better picture of this simply by
virtue of more testing.

The power of sundering lies in its ability to separate your data into two distinct categories:

1. rows that pass all validation checks (clean data)
2. rows that fail one or more validation checks (problematic data)

This approach allows you to:

- focus your analysis on clean, reliable data
- isolate problematic records for investigation or correction
- create pipelines that handle good and bad data differently

Let's use the `small_table` in our examples to show just how sundering is done. Here's that table:

```{python}
# | echo: false
pb.preview(pb.load_dataset(dataset="small_table"), n_head=20, n_tail=20)
```

## A Simple Example Where Data is Torn Asunder

We'll begin with a very simple validation plan, having only a single step. There *will be* failing
test units here.

```{python}
import pointblank as pb

validation = (
    pb.Validate(data=pb.load_dataset(dataset="small_table"))
    .col_vals_ge(columns="d", value=1000)
    .interrogate()
)

validation
```

We see six failing test units in `FAIL` column of the above validation report table. There is a data
extract (collection of failing rows) available. Let's use the [`Validate.get_data_extracts()`](`Validate.get_data_extracts`)
method to have a look at it.

```{python}
validation.get_data_extracts(i=1, frame=True)
```

This is six rows of data that had failing test units in column `d`. Indeed we can see that all
values in that column are less than `1000` (and we asserted that values should be greater than or
equal to `1000`). This is the 'bad' data, if you will. Using the [`Validate.get_sundered_data()`](`Validate.get_sundered_data`)
method, we get the 'good' part:

```{python}
validation.get_sundered_data()
```

This is a Polars DataFrame of seven rows. All values in `d` were passing test units (i.e., fulfilled
the expectation outlined in the validation step) and, in many ways, this is like a 'good extract'.

You can always collect the failing rows with [`Validate.get_sundered_data()`](`Validate.get_sundered_data`) by using the
`type="fail"` option. Let's try that here:

```{python}
validation.get_sundered_data(type="fail")
```

It gives us the same rows as in the DataFrame obtained from using
`validation.get_data_extracts(i=1, frame=True)`. Two important things to know about
[`Validate.get_sundered_data()`](`Validate.get_sundered_data`) are that the table rows returned from `type=pass` (the default) and
`type=fail` are:

- the sum of rows across these returned tables will be equal to that of the original table
- the rows in each split table are mutually exclusive (i.e., you won't find the same row in both)

You can think of sundered data as a filtered version of the original dataset based on validation
results. While the simple example illustrates how this process works on a basic level, the value of
the method is better seen in a slightly more complex example.

## Using `get_sundered_data()` with a More Comprehensive Validation

The previous example used exactly one validation step. You're likely to use more than that in
standard practice so let's see how [`Validate.get_sundered_data()`](`Validate.get_sundered_data`) works in those common
situations. Here's a validation with three steps:

```{python}
validation_2 = (
    pb.Validate(data=pb.load_dataset(dataset="small_table"))
    .col_vals_ge(
        columns="d",
        value=1000
    )
    .col_vals_not_null(columns="c")
    .col_vals_gt(
        columns="a",
        value=2
    )
    .interrogate()
)

validation_2
```

There are quite a few failures here across the three validation steps. In the `FAIL` column of the
validation report table, there are 12 failing test units if we were to tally them up. So if the
input table has 13 rows in total, does this mean there would be one row in the table returned by
[`Validate.get_sundered_data()`](`Validate.get_sundered_data`)? Not so:

```{python}
validation_2.get_sundered_data()
```

There are four rows. This is because the different validation steps tested values in different
columns of the table. Some of the failing test units had to have occurred in more than once in
certain rows. The rows that didn't have any failing test units across the three different tests
(in three different columns) are the ones seen above. This brings us to the third important thing
about the sundering process:

- the absence of test-unit failures in a row across all validation steps means those rows are
returned as the 'passing' set, all others are placed in the 'failing' set

In validations where many validation steps are used, we can be more confident about the level of
data quality for those rows returned in the passing set. But not every type of validation step is
considered within this splitting procedure. The next section will explain the rules on that.

## The Validation Methods Considered When Sundering

The sundering procedure relies on row-level validation types to be used. This makes sense as it's
impossible to judge the quality of a row when using the
[`col_exists()`](https://posit-dev.github.io/pointblank/reference/Validate.col_exists.html)
validation method, for example. Luckily, we have many row-level validation methods; here's a list:

- [`Validate.col_vals_gt()`](`Validate.col_vals_gt`)
- [`Validate.col_vals_lt()`](`Validate.col_vals_lt`)
- [`Validate.col_vals_ge()`](`Validate.col_vals_ge`)
- [`Validate.col_vals_le()`](`Validate.col_vals_le`)
- [`Validate.col_vals_eq()`](`Validate.col_vals_eq`)
- [`Validate.col_vals_ne()`](`Validate.col_vals_ne`)
- [`Validate.col_vals_between()`](`Validate.col_vals_between`)
- [`Validate.col_vals_outside()`](`Validate.col_vals_outside`)
- [`Validate.col_vals_in_set()`](`Validate.col_vals_in_set`)
- [`Validate.col_vals_not_in_set()`](`Validate.col_vals_not_in_set`)
- [`Validate.col_vals_in_table()`](`Validate.col_vals_in_table`)
- [`Validate.col_vals_null()`](`Validate.col_vals_null`)
- [`Validate.col_vals_not_null()`](`Validate.col_vals_not_null`)
- [`Validate.col_vals_regex()`](`Validate.col_vals_regex`)
- [`Validate.col_vals_within_spec()`](`Validate.col_vals_within_spec`)
- [`Validate.col_vals_str_len()`](`Validate.col_vals_str_len`)
- [`Validate.col_vals_increasing()`](`Validate.col_vals_increasing`)
- [`Validate.col_vals_decreasing()`](`Validate.col_vals_decreasing`)
- [`Validate.col_missing_coded()`](`Validate.col_missing_coded`)
- [`Validate.col_missing_only_coded()`](`Validate.col_missing_only_coded`)
- [`Validate.col_vals_expr()`](`Validate.col_vals_expr`)
- [`Validate.rows_complete()`](`Validate.rows_complete`)
- [`Validate.conjointly()`](`Validate.conjointly`)
- [`Validate.prompt()`](`Validate.prompt`)
- [`Validate.tbl_match()`](`Validate.tbl_match`) (when its results can be lined up with the rows of
the table)
- [`Validate.rows_distinct()`](`Validate.rows_distinct`) (only when you say how to treat duplicate
rows; see the next section)

There are some additional caveats though. Even if using a validation method drawn from the set
above, the validation step won't be used for sundering if:

- the `active=` parameter for that step has been set to `False`
- the `pre=` parameter has been used

The first one makes intuitive sense (you decided to skip this validation step entirely), the second
one requires some explanation. Using `pre=` allows you to modify the target table, there's no easy
or practical way to compare rows in a mutated table compared to the original table (e.g., a
mutation may drastically reduce the number of rows).

Steps that use `segments=` are used for sundering. Each segment's results apply only to the rows in
that segment, so rows outside all of a step's segments aren't affected by that step.

## Sundering with Duplicate Rows

A [`Validate.rows_distinct()`](`Validate.rows_distinct`) step counts *every* copy of a duplicated
row as a failing test unit, since there's no telling which of the copies is the 'wrong' one. That's
the right count for a validation report, but it's rarely what you want when sundering: removing all
copies of a duplicate would remove the record entirely. So by default, `rows_distinct()` steps are
left out of sundering, and the `duplicates=` argument lets you bring them in by saying which copy to
keep. The `small_table` dataset has one pair of duplicate rows, so let's check for that:

```{python}
validation_3 = (
    pb.Validate(data=pb.load_dataset(dataset="small_table"))
    .rows_distinct()
    .interrogate()
)

validation_3.get_sundered_data(duplicates="keep_first")
```

That's 12 of the 13 rows: the first copy of the duplicate stays in the passing set and the second
copy goes to the failing set. The options for `duplicates=` are:

- `"keep_first"`: the first copy of each group of duplicates passes and the others fail
- `"keep_last"`: the last copy passes and the others fail
- `"flag_all"`: every copy fails (just as the validation step counts them)
- `None` (the default): `rows_distinct()` steps aren't used for sundering

The 'first' and 'last' copies are those in row order, unless you use `duplicates_order_by=` to name
one or more columns to order them by. For example, a check for duplicated customer IDs combined
with `duplicates="keep_last", duplicates_order_by="updated_at"` keeps the most recently updated
record for each customer.

## Flagging Rows Instead of Splitting Them

Sundering gives a yes-or-no answer for each row: it passed every step, or it didn't. Sometimes you
want to know *which* steps a row failed, so you can decide what to do with it later. The
[`Validate.get_row_flags()`](`Validate.get_row_flags`) method returns the whole table with a
`True`/`False` column for each row-level step (`pb_failed_1_`, `pb_failed_2_`, and so on), along
with the number of steps each row failed (`pb_n_failed_`) and the share of its checks that it passed
(`pb_row_score_`):

```{python}
validation_2.get_row_flags()
```

The rows with a `pb_n_failed_` of `0` are exactly the passing set from sundering. With the flags in
hand, though, you can filter on just the checks that matter for a particular use, look for problems
that tend to occur together, or use `pb_row_score_` to weight rows rather than drop them. The same
`duplicates=` and `duplicates_order_by=` options are available here, except that
`get_row_flags()` uses `duplicates="keep_first"` by default, so that a `rows_distinct()` flag marks
the copies that removing duplicates would remove.

## Practical Applications of Sundering

### 1. Creating Clean Datasets for Analysis

One of the most common use cases for sundering is preparing validated data for downstream analysis:

```{python}
# Comprehensive validation for analysis-ready data
analysis_validation = (
    pb.Validate(data=pb.load_dataset(dataset="small_table"))
    .col_vals_not_null(columns=["a", "b", "c", "d", "e", "f"])  # No missing values
    .col_vals_gt(columns="a", value=0)                          # Positive values only
    .col_vals_lt(columns="d", value=10000)                      # No extreme outliers
    .interrogate()
)

# Extract only the clean data that passed all checks
clean_data = analysis_validation.get_sundered_data(type="pass")

# Use the clean data for your analysis
pb.preview(clean_data)
```

This approach ensures that any subsequent analysis is based on data that meets your quality
standards, reducing the risk of misleading results or spurious conclusions due to problematic
records. By making validation an explicit step in your analytical workflow, you create a natural
quality gate that prevents invalid data from influencing your findings.

### 2. Creating Parallel Workflows for Clean and Problematic Data

You can use sundering to create parallel processing paths:

```{python}
# Get both clean and problematic data
clean_data = analysis_validation.get_sundered_data(type="pass")
problem_data = analysis_validation.get_sundered_data(type="fail")

# Process clean data (in real applications, you'd do more here)
print(f"Clean data size: {len(clean_data)} rows")

# Log problematic data for investigation
print(f"Problematic data size: {len(problem_data)} rows")
```

This approach enables you to build robust data processing pathways with separate handling for clean
and problematic data. In production environments, you could save problematic records to a separate
location for further investigation, generate detailed logs of validation failures, and trigger
automated notifications to data stewards when issues arise. By establishing clear protocols for
handling both data streams, you create a systematic approach to data quality that balances immediate
analytical needs with longer-term data improvement goals.

### 3. Data Quality Monitoring and Improvement

Tracking the ratio of passing to failing rows over time can help monitor data quality trends:

```{python}
# Calculate data quality metrics
total_rows = len(pb.load_dataset(dataset="small_table"))
passing_rows = len(clean_data)
quality_score = passing_rows / total_rows

print(f"Data quality score: {quality_score:.2%}")
print(f"Passing rows: {passing_rows} out of {total_rows}")
```

By tracking these metrics over time, you can measure the impact of your data quality improvement
efforts and communicate progress to stakeholders. This approach transforms sundering from a one-time
filtering tool into an ongoing data quality management system, where improving the ratio of passing
rows becomes a measurable business objective aligned with broader data governance goals.

## Best Practices for Using Sundered Data

When incorporating data sundering into your workflow, consider these best practices:

1. Be comprehensive in your validation: the more validation steps you include (assuming they're
meaningful), the more confidence you can have in your passing dataset

2. Document your validation criteria: when sharing sundered data with others, always document the
criteria used to determine passing rows

3. Consider traceability: for audit purposes, it may be valuable to add a column indicating whether
a record was originally in the passing or failing set

4. Balance strictness and practicality: if you're too strict with validation rules, you might end up
with very few passing rows; consider the appropriate level of strictness for your use case

5. Use sundering as part of a pipeline: automate the process of validation, sundering, and
subsequent handling of the two resulting datasets

6. Continually refine validation rules: as you learn more about your data and domain, update your
validation rules to improve the accuracy of your sundering process

By following these best practices, data scientists and engineers can transform sundering from a
simple utility into a strategic component of their data quality framework. When implemented
thoughtfully, sundering enables a shift from reactive data cleaning to proactive quality management,
where validation criteria evolve alongside your understanding of the data.

The ultimate goal isn't just to separate good data from bad, but to gradually improve your entire
dataset over time by addressing the root causes of validation failures that appear in the failing
set. This approach turns data validation from a gatekeeper function into a continuous improvement
process.

## Conclusion

Data sundering provides a powerful way to separate your data based on validation results. While
the concept is simple (splitting data into passing and failing sets) the feature can very useful in
many data workflows. By integrating sundering into your data pipeline, you can:

- ensure that downstream analysis only works with validated data
- create focused datasets for different purposes
- improve overall data quality through systematic identification and isolation of problematic
records
- build more robust data pipelines that explicitly handle data quality issues

So long as you're aware of the rules and limitations of sundering, you're likely to find it to be a
simple and useful way to filter your input table on the basis of a validation plan, turning data
validation from a passive reporting tool into an active component of your data processing workflow.


### Quality Dimensions & Scoring

```{python}
#| echo: false
#| output: false
import pointblank as pb
```

Beyond knowing *which* validation steps passed or failed, it's often useful to understand data
quality along broad, well-understood **dimensions** (completeness, validity, uniqueness, and so
on) and to roll everything up into a single **health score** that a governance stakeholder can
track over time.

Pointblank does this automatically. Every validation step is tagged with a data quality dimension
(inferred from what the step checks), and after interrogation you can obtain per-dimension scores
and an overall health score. Nothing about how checks *run* changes; this is a labeling and
aggregation layer over results you already have.

## The Six Dimensions

Each validation step belongs to one of six data quality dimensions:

| Dimension | What it measures | Example methods |
|---|---|---|
| **Completeness** | Presence of required values | `col_vals_not_null()`, `col_pct_null()`, `rows_complete()` |
| **Validity** | Values conform to rules, ranges, formats, or schema | `col_vals_gt()`, `col_vals_regex()`, `col_vals_in_set()`, `col_schema_match()` |
| **Uniqueness** | Absence of duplicate rows | `rows_distinct()` |
| **Consistency** | Internal agreement across columns, rows, or tables | `conjointly()`, `col_missing_consistent()`, `col_vals_in_table()`, `tbl_match()` |
| **Timeliness** | Data recency / freshness | `data_freshness()` |
| **Volume** | Expected row and column counts | `row_count_match()`, `col_count_match()` |

The dimension for each step is inferred from its assertion type, so existing validations gain
dimensions with no changes on your part.

## Dimensions in the Validation Report

The dimension display in the validation report is **opt-in**: pass `incl_dimensions=True` to
[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) (or set it globally with
`pb.config(report_incl_dimensions=True)`). Consider a validation of some sales data that touches
several dimensions:

```{python}
import pointblank as pb
import polars as pl

sales = pl.DataFrame({
    "order_id": [1, 2, 3, 4, 5, 6, 7, 8, 8, 10],   # one duplicate (8)
    "amount":   [120.0, -5.0, 47.5, 0.0, 30.0, 155.0, 175.0, 95.0, 205.0, None],
    "email":    ["a@ex.com", "invalid", "c@ex.com", "d@ex.com", "nope",
                 "f@ex.com", "g@ex.com", "h@ex.com", "i@ex.com", "j@ex.com"],
    "status":   ["paid", "paid", "refund", "paid", "pending",
                 "paid", "paid", "refund", "paid", "paid"],
})

validation = (
    pb.Validate(data=sales, tbl_name="sales", label="Sales data quality")
    .col_vals_not_null(columns="amount")                                # Completeness
    .col_vals_gt(columns="amount", value=0)                             # Validity
    .col_vals_regex(columns="email", pattern=r"^[^@]+@[^@]+\.[^@]+$")   # Validity
    .col_vals_in_set(columns="status", set=["paid", "refund", "pending"])  # Validity
    .rows_distinct(columns_subset=["order_id"])                         # Uniqueness
    .row_count_match(count=10)                                          # Volume
    .interrogate()
)

validation.get_tabular_report(incl_dimensions=True)
```

Two things in the report now relate to dimensions:

- **A dimension badge on each step number.** Each step's number carries a small, color-coded
  two-letter badge in its top-left corner (`CM` completeness, `VA` validity, `UQ` uniqueness,
  `CS` consistency, `TM` timeliness, `VO` volume). Hover over a badge to see the full dimension
  name. The badge is compact by design, so it doesn't widen the report.
- **A health-score summary in the footer.** Below the table you'll find the overall **Health
  Score** followed by a per-dimension breakdown, with each dimension's color reinforcing the badges
  above.

The scores themselves (below) are always available programmatically, whether or not the display is
enabled.

## Overriding a Step's Dimension

Automatic inference covers the common cases, but you can set a dimension explicitly with the
`dimension=` parameter on any validation method. This is useful for multi-faceted checks. For
example, treating a particular range check as a *consistency* rule rather than plain validity:

```{python}
validation_override = (
    pb.Validate(data=sales, tbl_name="sales")
    .col_vals_gt(columns="amount", value=0, dimension="consistency")
    .interrogate()
)

validation_override.get_dimension_scores()
```

You can also remap dimensions globally (for every step of a given type) with
[`config()`](`pointblank.config`):

```{python}
pb.config(dimension_map={"col_vals_gt": "consistency"})

# Now `col_vals_gt` steps are categorized as "consistency" everywhere
remapped = (
    pb.Validate(data=sales)
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)
print(remapped.validation_info[0].dimension)

pb.config()  # reset back to defaults
```

An explicit `dimension=` on a step always takes precedence over the global map.

## Health Scores

After interrogation, two methods surface the scores:

```{python}
validation.get_dimension_scores()
```

```{python}
validation.get_health_score()
```

Scores are **test-unit weighted**: a dimension's score is the total number of passing test units
divided by the total number of test units across its steps, expressed as a percentage. The overall
health score is the same calculation across every step. Because it's weighted by test units (not by
step count), the score reflects data volume. A failing check over a large table pulls the score down
more than one over a small table.

::: {.callout-note}
Only steps that produced a pass/fail result contribute to scoring. Steps that haven't been
interrogated, inactive steps (`active=False`), and steps that could not be evaluated (for example,
a check that references a nonexistent column) are all excluded, so a broken check doesn't distort
the score.
:::

### Weighting Dimensions

Some organizations consider certain dimensions more critical than others. Provide per-dimension
weights with `config(dimension_weights=...)` to scale each dimension's contribution to the overall
score (a dimension not listed keeps a weight of `1.0`):

```{python}
pb.config(dimension_weights={"completeness": 3.0})

validation_weighted = (
    pb.Validate(data=sales)
    .col_vals_not_null(columns="amount")   # Completeness, weighted 3x
    .col_vals_gt(columns="amount", value=0)  # Validity
    .interrogate()
)
print(validation_weighted.get_health_score())

pb.config()  # reset back to defaults
```

## The Scorecard

For a compact, standalone summary (well suited to dashboards or an executive overview) use
[`get_scorecard()`](`pointblank.Validate.get_scorecard`). It shows the overall health score
prominently along with a per-dimension breakdown (a color-coded bar, the score, and passing/total
test units):

```{python}
validation.get_scorecard()
```

The scorecard is a [Great Tables](https://github.com/posit-dev/great-tables) object, so you can
display it directly, export it to HTML with `.as_raw_html()`, or save it to an image file with
`.gtsave()`.

## Enforcing Minimum Scores

In automated pipelines and CI you often want to *fail the run* when a dimension slips below an
acceptable level. Call [`assert_dimension_scores()`](`pointblank.Validate.assert_dimension_scores`)
with per-dimension minimums; it raises an `AssertionError` if any dimension is below its minimum
(here, uniqueness is `80%`):

```{python}
try:
    validation.assert_dimension_scores(thresholds={"uniqueness": 95})
except AssertionError as e:
    print(e)
```

Dimensions present in the thresholds but absent from the validation are ignored, and a call where
every dimension meets its minimum simply returns without raising. You can also set the minimums
globally with `config(dimension_thresholds=...)`, and then call the method without arguments:

```{python}
pb.config(dimension_thresholds={"completeness": 95})

try:
    validation.assert_dimension_scores()
except AssertionError as e:
    print(e)

pb.config()  # reset back to defaults
```

## Accessing Scores Programmatically

The scores are also included in the summary available to
[`FinalActions`](`pointblank.FinalActions`) via
[`get_validation_summary()`](`pointblank.get_validation_summary`), under the `dimension_scores` and
`overall_health_score` keys. This makes it easy to log or trend the health score over time:

```{python}
def log_health():
    summary = pb.get_validation_summary()
    print(f"Overall health score: {summary['overall_health_score']}")
    print(f"By dimension: {summary['dimension_scores']}")

(
    pb.Validate(data=sales, final_actions=pb.FinalActions(log_health))
    .col_vals_not_null(columns="amount")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)
```

## Localized Dimensions

Like the rest of the validation report, dimension labels and the health-score summary are
localized. When you set a reporting language (e.g., `Validate(..., lang="fr")`), the dimension
names in badge tooltips, the footer summary, and the scorecard are translated automatically. The
two-letter badge codes stay language-neutral (matching the report's other short codes such as
`TBL`, `EVAL`, and `W`/`E`/`C`), with the full localized name available on hover.


### Previewing Data

```{python}
#| echo: false
#| output: false
import pointblank as pb
```

In many cases, it's *good* to look at your data tables. Before validating a table, you'll likely
want to inspect a portion of it before diving into the creation of data-quality rules. This is
pretty easily done with Polars and Pandas DataFrames, however, it's not as easy with database tables
and each table backend displays things differently.

To make this common task a little better, you can use the `preview()` function in Pointblank. It has
been designed to work with every table that the package supports (i.e., DataFrames and Ibis-backend
tables, the latter of which are largely database tables). Plus, what's shown in the output is
consistent, no matter what type of data you're looking at.

## Viewing a Table with `preview()`{.qd-no-link}

Let's look at how `preview()` works. It requires only a table and, for this first example, let's use
the `nycflights` dataset:

```{python}
import pointblank as pb

nycflights = pb.load_dataset(dataset="nycflights", tbl_type="polars")

pb.preview(nycflights)
```

This is an HTML table using the style of the other reporting tables in the library. The header is
more minimal here, only showing the type of table we're looking at (`POLARS` in this case) along
with the table dimensions. The column headers provide both the column names and the column data
types.

By default, we're getting the first five rows and the last five rows. Row numbers (from the original
dataset) provide an indication of which rows are the head and tail rows. The blue lines provide
additional demarcation of the column containing the row numbers and the head and tail row groups.
Finally, any cells with missing values are prominently styled with red lettering and a lighter red
background.

If you'd rather not see the row numbers in the table, you can use the `show_row_numbers=False`
option. Let's try that with the `game_revenue` dataset as a DuckDB table:

```{python}
game_revenue = pb.load_dataset(dataset="game_revenue", tbl_type="duckdb")

pb.preview(game_revenue, show_row_numbers=False)
```

With the above preview, the row numbers are gone. The horizontal blue line still serves to divide
the top and bottom rows of the table, however.

## Adjusting the Number of Rows Shown

It could be that displaying the five top and bottom rows is not preferred. This can be changed with
the `n_head=` and `n_tail=`. Maybe, you want three from the top along with the last row? Let's try
that out with the `small_table` dataset as a Pandas DataFrame:

```{python}
small_table = pb.load_dataset(dataset="small_table", tbl_type="pandas")

pb.preview(small_table, n_head=3, n_tail=1)
```

If you're looking at a small table and want to see the entirety of it, you can enlarge the `n_head=`
and `n_tail=` values:

```{python}
small_table = pb.load_dataset(dataset="small_table", tbl_type="pandas")

pb.preview(small_table, n_head=10, n_tail=10)
```

Given that the table has 13 rows, asking for 20 rows to be displayed effectively shows the entire
table.

## Previewing a Subset of Columns

The preview scales well to tables that have many columns by allowing for a horizontal scroll.
However, previewing data from all columns can be impractical if you're only concerned with a key set
of them. To preview only a subset of a table's columns, we can use the `columns_subset=` argument.
Let's do this with the `nycflights` dataset and provide a list of six columns from that table.

```{python}
pb.preview(
    nycflights,
    columns_subset=["hour", "minute", "sched_dep_time", "year", "month", "day"]
)
```

What we see are the six columns we specified from the `nycflights` dataset.

Note that the columns are displayed in the order provided in the `columns_subset=` list. This can be
useful for making quick, side-by-side comparisons. In the example above, we placed `hour` and
`minute` next to the `sched_dep_time` column. In the original dataset, `sched_dep_time` is far
apart from the other two columns, but, it's useful to have them next to each other in the preview
since `hour` and `minute` are derived from `sched_dep_time` (and this lets us spot check any
issues).

We can also use column selectors within `columns_subset=`. Suppose we want to only see those columns
that have `"dep_"` or `"arr_"` in the name. To do that, we use the
[`matches()`](`pointblank.matches`) column selector function:

```{python}
pb.preview(nycflights, columns_subset=pb.matches("dep_|arr_"))
```

Several selectors can be combined together through use of the `col()` function and operators such as
`&` (*and*), `|` (*or*), `-` (*difference*), and `~` (*not*). Let's look at a column selection case
where:

- the first three columns are selected
- all columns containing `"dep_"` or `"arr_"` are selected
- any columns beginning with `"sched"` are omitted

This is how we put that together within `col()`:

```{python}
pb.preview(
    nycflights,
    columns_subset=pb.col((pb.first_n(3) | pb.matches("dep_|arr_")) & ~ pb.starts_with("sched"))
)
```

This gives us a preview with only the columns that fit the specific selection rules. Incidentally,
using selectors with a dataset through `preview()` is a good way to test out the use of selectors
more generally. Since they are primarily used to select columns for validation, trying them
beforehand with `preview()` can help verify that your selection logic is sound.


### Column Summaries

```{python}
#| echo: false
#| output: false
import os
```

While previewing a table with `preview()` is undoubtedly a good thing to do, sometimes you need
more. This is where summarizing a table comes in. When you view a summary of a table, the
column-by-column info can quickly increase your understanding of a dataset. Plus, it allows you to
quickly catch anomalies in your data (e.g., the maximum value of a column could be far outside the
realm of possibility).

Pointblank provides a function to make it extremely easy to view column-level summaries in a single
table. That function is called `col_summary_tbl()` and, just like `preview()` does, it supports the
use of any table that Pointblank can use for validation. And no matter what the input data is, the
resultant reporting table is consistent in its design and construction.

## Trying out `col_summary_tbl()`{.qd-no-link}

The function only requires a table. Let's use the `small_table` dataset (a very simple table) to
start us off:

```{python}
import pointblank as pb

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

pb.col_summary_tbl(small_table)
```

The header provides the type of table we're looking at (`POLARS`, since this is a Polars DataFrame)
and the table dimensions. The rest of the table focuses on the column-level summaries. As such, each
row represents a summary of a column in the `small_table` dataset. There's a lot of information in
this summary table to digest. Some of it is intuitive since this sort of table summarization isn't
all that uncommon, but other aspects of it could also give some pause. So we'll carefully wade
through how to interpret this report.

## Data Categories in the Column Summary Table

On the left side of the table are icons of different colors. These represent categories that the
columns fall into. There are only five categories and columns can only be of one type. The
categories (and their letter marks) are:

- `N`: numeric
- `S`: string-based
- `D`: date/datetime
- `T/F`: boolean
- `O`: object

The numeric category (`N`) takes data types such as floats and integers. The `S` category is for
string-based columns. Date or datetime values are lumped into the `D` category. Boolean columns
(`T/F`) have their own category and are *not* considered numeric (e.g., `0`/`1`). The `O` category
is a catchall for all other types of columns. Given the disparity of these categories and that we
want them in the same table, some statistical measures will be sensible for certain column
categories but not for others. Given that, we'll explain how each category is represented in the
column summary table.

## Numeric Data

Three columns in `small_table` are numeric: `a` (`Int64`), `c` (`Int64`), and `d` (`Float64`). The
common measures of the missing count/proportion (`NA`) and the unique value count/proportion (`UQ`)
are provided for the numeric data type. For these two measures, the top number is the absolute count
of missing values and the count of unique values. The bottom number is a proportion of the absolute
count divided by the row count; this makes each proportion a value between `0` and `1` (bounds
included).

The next two columns represent the mean (`Mean`) and the standard deviation (`SD`). The minumum
(`Min`), maximum, (`Max`) and a set of quantiles occupy the next few columns (includes `P5`, `Q1`,
`Med` for median, `Q3`, and `P95`). Finally, the interquartile range (`IQR`: `Q3` - `Q1`) is the
last measure provided.

## String Data

String data is present in `small_table`, being in columns `b` and `f`. The missing value (`NA`) and
uniqueness (`UQ`) measures are accounted for here. The statistical measures are all based on string
lengths, so what happens is that all strings in a column are converted to those numeric values and a
subset of stats values is presented. To avoid some understandable confusion when reading the table,
the stats values in each of the cells with values are annotated with the text `"SL"`. It makes less
sense to provide a full suite of quantile values so only the minimum (`Min`), median (`Med`), and
maximum (`Max`) are provided.

## Date/Datetime Data and Boolean Data

We see that in the first two rows of our summary table there are summaries of the `date_time` and
`date` columns. The summaries we provide for a date/datetime category (notice the green `D` to the
left of the column names) are:

1. the missing count/proportion (`NA`)
2. the unique value count/proportion (`UQ`)
3. the minimum and maximum dates/datetimes

One column, `e`, is of the `Boolean` type. Because columns of this type could only have `True`,
`False`, or missing values, we provide summary data for missingness (under `NA`) and proportions of
`True` and `False` values (under `UQ`).

## Using `DataScan` Directly

The `col_summary_tbl()` function is a convenience wrapper around the
[`DataScan`](`pointblank.DataScan`) class. When you need more than a visual report (for example, to
save the profile for later comparison), you can work with `DataScan` directly:

```{python}
small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

scan = pb.DataScan(data=small_table, tbl_name="small_table")
```

The `DataScan` object computes the same column-level statistics shown in the summary table. You can
access the profile as a dictionary with [`to_dict()`](`DataScan.to_dict`), export it as JSON with
[`to_json()`](`DataScan.to_json`), or render the same tabular report with
[`get_tabular_report()`](`DataScan.get_tabular_report`).

## Saving and Loading Profiles

A `DataScan` profile can be saved to disk and loaded back later. This is useful for establishing
baselines that you compare against in the future.

```{python}
# Save the profile to a JSON file
scan.save_to_json("small_table_profile.json")

# Later, load it back without needing the original data
loaded = pb.DataScan.load_from_json("small_table_profile.json")
```

```{python}
#| echo: false
#| output: false
os.remove("small_table_profile.json")
```

The `from_json()` classmethod works with JSON strings directly, and
[`from_dict()`](`DataScan.from_dict`) accepts the dictionary format produced by `to_dict()`. All
three approaches produce a fully restored `DataScan` that retains the column names, types,
statistics, and sample data from the original scan.

## Comparing Profiles for Drift

When your data changes over time, you can compare two `DataScan` profiles to detect drift. The
[`compare()`](`DataScan.compare`) method identifies schema changes (columns added, removed, or with
changed types) and statistical shifts in common columns.

```{python}
import polars as pl

# Simulate two versions of a dataset
orders_v1 = pl.DataFrame({
    "order_id": [1, 2, 3, 4, 5],
    "amount": [10.0, 25.0, 15.0, 30.0, 20.0],
    "status": ["paid", "paid", "refund", "paid", "paid"],
})

orders_v2 = pl.DataFrame({
    "order_id": [1, 2, 3, 4, 5, 6, 7, 8],
    "amount": [10.0, 25.0, 15.0, 30.0, 20.0, 150.0, 200.0, 175.0],
    "status": ["paid", "paid", "refund", "paid", "paid", "paid", "paid", "paid"],
    "region": ["US", "EU", "US", "EU", "US", "US", "EU", "EU"],
})

baseline = pb.DataScan(data=orders_v1, tbl_name="orders_v1")
current = pb.DataScan(data=orders_v2, tbl_name="orders_v2")

diff = current.compare(baseline)
```

The returned [`DataScanDiff`](`pointblank.DataScanDiff`) object provides programmatic access to the
changes:

```{python}
print("Has changes:", diff.has_changes)
print("Columns added:", diff.columns_added)
print("Row count (baseline vs current):", diff.row_count_diff)
```

You can also get the full comparison as a dictionary with [`to_dict()`](`DataScanDiff.to_dict`), or
view it as a styled report:

```{python}
diff.get_tabular_report()
```

The report shows each column's status (OK, Added, Removed, Stats Changed, or Type Changed) along
with any statistics that shifted between the baseline and current profiles. This makes it
straightforward to spot when your data's shape or distribution has changed in ways that might affect
downstream analyses or validation rules.

### Statistical Drift Measures

When both the baseline and current `DataScan` objects have their original data attached (i.e., they
haven't been loaded from disk), the comparison also computes formal drift statistics for each
column:

- **PSI (Population Stability Index)**: Measures how much a distribution has shifted. Computed for
both numeric and categorical columns. Values below 0.1 indicate no meaningful drift, 0.1 to 0.25
suggests moderate drift, and above 0.25 signals a significant population shift.
- **KS (Kolmogorov-Smirnov) test**: For numeric columns only, this measures the maximum distance
between two empirical CDFs and provides a p-value. A small p-value (e.g., below 0.05) indicates the
two distributions are statistically different.

These scores appear in the `drift_scores` field of the comparison dictionary and in the "Drift
Scores" column of the generated report. Let's look at a more dramatic drift example to see them in
action:

```{python}
# Baseline: normal revenue range
revenue_v1 = pl.DataFrame({
    "amount": [10.0, 25.0, 15.0, 30.0, 20.0, 12.0, 18.0, 22.0, 28.0, 35.0,
               14.0, 26.0, 19.0, 31.0, 23.0, 11.0, 17.0, 29.0, 33.0, 21.0],
    "category": (["electronics"] * 8 + ["clothing"] * 7 + ["food"] * 5),
})

# Current: large orders have appeared, category mix shifted
revenue_v2 = pl.DataFrame({
    "amount": [10.0, 25.0, 15.0, 30.0, 20.0, 150.0, 200.0, 175.0, 180.0, 160.0,
               14.0, 26.0, 190.0, 170.0, 165.0, 11.0, 17.0, 155.0, 185.0, 195.0],
    "category": (["electronics"] * 12 + ["clothing"] * 3 + ["food"] * 5),
})

baseline = pb.DataScan(data=revenue_v1, tbl_name="revenue_v1")
current = pb.DataScan(data=revenue_v2, tbl_name="revenue_v2")

diff = current.compare(baseline)
diff.get_tabular_report()
```

The drift scores are also available programmatically through [`to_dict()`](`DataScanDiff.to_dict`):

```{python}
drift = diff.to_dict().get("drift_scores", {})
for col_name, scores in drift.items():
    print(f"{col_name}: {scores}")
```

When a `DataScan` is loaded from a saved JSON file (with no original data attached), drift scores
cannot be computed. The comparison still reports schema changes and raw stat differences, but the
`drift_scores` field will be empty. If you need drift scores in a persist-and-compare workflow,
keep the original data available when calling `compare()`.

### A Typical Workflow

In practice, drift detection works best as a two-phase process. First, you establish a baseline
profile once your data quality checks pass and save it to disk. Then, on each subsequent pipeline
run, you load that baseline and compare it against a fresh scan of the incoming data.

```python
# On the first run: establish the baseline
baseline = pb.DataScan(data=production_table, tbl_name="orders")
baseline.save_to_json("orders_baseline.json")

# On subsequent runs: compare against the baseline
baseline = pb.DataScan.load_from_json("orders_baseline.json")
current = pb.DataScan(data=production_table, tbl_name="orders")
diff = current.compare(baseline)

if diff.has_changes:
    print("Data drift detected!")
    print(diff.to_dict())
```

The key thing to notice is that the baseline is loaded from a saved JSON file, so it won't have
original data attached. This means drift scores (PSI and KS) won't be computed for the loaded
baseline. If you need those scores, keep the baseline data accessible and create a fresh `DataScan`
from it rather than loading from JSON.

The `has_changes` property gives you a simple boolean for CI pipelines or alerting. So if anything
shifted (schema or statistics) you can easily flag it. For more granular decisions, inspect
`columns_added`, `columns_removed`, `columns_type_changed`, and the per-column `stat_diffs` and
`drift_scores` in the object provided by `to_dict()`.

## Conclusion

The `col_summary_tbl()` function and the `DataScan` class give you two levels of access to
column-level profiling. The function is a quick, visual summary you can drop into a notebook to
understand a dataset at a glance. The class provides the same statistics in a structured form that
you can save, load, and compare over time. When your data evolves, `compare()` catches schema drift
(columns added, removed, or retyped) and statistical drift (shifts in means, quantiles, null rates,
and frequency distributions), optionally backed by formal measures like PSI and the KS test.
Together, these tools make it straightforward to move from initial data discovery to ongoing data
quality monitoring.


### Missing Values Reporting

```{python}
#| echo: false
#| output: false
import pointblank as pb
```

Sometimes values just aren't there: they're missing. This can either be expected or another thing to
worry about. Either way, we can dig a little deeper if need be and use the `missing_vals_tbl()`
function to generate a summary table that can elucidate how many values are missing, and roughly
where.

## Using and Understanding `missing_vals_tbl()`{.qd-no-link}

The missing values table is arranged a lot like the column summary table (generated via the
`col_summary_tbl()` function) in that columns of the input table are arranged as rows in the
reporting table. Let's use `missing_vals_tbl()` on the `nycflights` dataset, which has a lot of
missing values:

```{python}
import pointblank as pb

nycflights = pb.load_dataset(dataset="nycflights", tbl_type="polars")

pb.missing_vals_tbl(nycflights)
```

There are 18 columns in `nycflights` and they're arranged down the missing values table as rows. To
the right we see column headers indicating 10 columns that are row sectors. Row sectors are groups
of rows and each sector contains a tenth of the total rows in the table. The leftmost sectors are
the rows at the top of the table whereas the sectors on the right are closer to the bottom. If you'd
like to know which rows make up each row sector, there are details on this in the table footer area
(click the `ROW SECTORS` text or the disclosure triangle).

Now that we know about row sectors, we need to understand the visuals here. A light blue cell
indicates there are no (`0`) missing values within a given row sector of a column. For `nycflights`
we can see that several columns have no missing values at all (i.e., the light blue color makes up
the entire row in the missing values table).

When there are missing values in a column's row sector, you'll be met with a grayscale color. The
proportion of missing values corresponds to the color ramp from light gray to solid black.
Interestingly, most of the columns that have missing values appear to be related to each other in
terms of the extent of missing values (i.e., the appearance in the reporting table looks roughly the
same, indicating a sort of systematic missingness). These columns are `dep_time`, `dep_delay`,
`arr_time`, `arr_delay`, and `air_time`.

The odd column out with regard to the distribution of missing values is `tailnum`. By scanning the
row and observing that the grayscale color values are all a little different we see that the degree
of missingness of more variable and not related to the other columns containing missing values.

## Missing Value Tables from the Other Datasets

The `small_table` dataset has only 13 rows to it. Let's use that as a Pandas DataFrame with
`missing_vals_tbl()`:

```{python}
small_table = pb.load_dataset(dataset="small_table", tbl_type="pandas")

pb.missing_vals_tbl(small_table)
```

It appears that only column `c` has missing values. And since the table is very small in terms of
row count, most of the row sectors contain only a single row.

The `game_revenue` dataset has *no* missing values. And this can be easily proven by using
`missing_vals_tbl()` with it:

```{python}
game_revenue = pb.load_dataset(dataset="game_revenue", tbl_type="duckdb")

pb.missing_vals_tbl(game_revenue)
```

We see nothing but light blue in this report! The header also indicates that there are no missing
values by displaying a large green check mark (the other report tables provided a count of total
missing values across all columns).

## Structured Missingness by Reason

So far we've treated missingness as binary: a value is either `Null` or it isn't. But real-world
data often encodes *why* a value is absent. Survey data distinguishes *refused* from *not asked*
from *don't know*; clinical and statistical-package data use sentinel codes like `-99`, `".A"`, or
`"NOT DONE"`. Pointblank captures this with the [`MissingSpec`](`pointblank.MissingSpec`) class,
which maps sentinel values to human-readable *reasons*.

When you pass a `missing=` mapping of column names to `MissingSpec` objects, `missing_vals_tbl()`
switches from the sector heatmap to a *structured breakdown*: one row per column with the count and
percentage of complete values and of each missing reason.

::: {.callout-note}
## Supplying `missing=` produces a different report

The structured breakdown is a *distinct visualization*, not an annotated version of the default
sector heatmap. Adding `missing=` changes the table's whole layout. The report title changes too
(from "Missing Values" to "Missing Values by Reason", or "Missing Pattern Heatmap" with
`as_heatmap=True`), and the shared title styling and monospaced column list keep the two views
recognizably part of the same family.
:::

```{python}
import polars as pl

survey = pl.DataFrame(
    {
        "age": [34, -98, 41, -99, 29, -98, 55, None],
        "income": [50000, -99, -1, None, 42000, -99, 38000, 61000],
    }
)

specs = {
    "age": pb.MissingSpec(reasons={-99: "not_asked", -98: "refused", -97: "dont_know"}),
    "income": pb.MissingSpec(reasons={-99: "not_asked", -1: "below_threshold"}),
}

pb.missing_vals_tbl(survey, missing=specs)
```

Each `MissingSpec` declares the sentinel values for a column and the reason each one represents.
Those declared (coded) reasons are grouped under the **Missing Reasons** spanner. By default, actual
`Null` values are also counted as missing; because those are raw `Null`/`None`/`NA` values and *not*
part of the spec, they're tallied in a fixed **Null** column at the far right (styled like
**Complete**), rather than as a reason. Set `null_is_missing=False` on the spec if raw nulls should
be treated as real values instead — then there's no **Null** column at all.

The reason columns are the *union* of reasons across all the specs you provide. When a reason isn't
defined for a particular column, that cell shows an em dash (`—`) rather than `0`. This signals
"not applicable to this column", as distinct from a reason that *is* defined but simply wasn't
observed (which shows `0 (0%)`).

### Viewing the pattern as a heatmap

For a more visual read of *where* missingness concentrates, pass `as_heatmap=True`. The reason
columns are then shaded from light to dark by the proportion missing:

```{python}
pb.missing_vals_tbl(survey, missing=specs, as_heatmap=True)
```

### Pre-built specs for common standards

You don't always have to define reasons by hand. `MissingSpec` provides factory methods for common
encodings, including CDISC/HL7 null flavors and SAS special missing values:

```{python}
cdisc = pb.MissingSpec.from_cdisc_null_flavors()
print("NASK ->", cdisc.reason_for("NASK"))   # not_asked
print("UNK  ->", cdisc.reason_for("UNK"))     # unknown
```

When metadata is imported from SPSS, Stata, or SAS files (see the *Metadata Import* section),
[`MetadataImport.missing_specs()`](`pointblank.MetadataImport`) auto-generates a `{column:
MissingSpec}` mapping from the variables' declared missing values, ready to pass straight to
`missing_vals_tbl()`.

::: {.callout-note}
The same `MissingSpec` objects power missingness-aware *validation*, not just reporting. You can
pass `missing=` to the `col_vals_*()` methods (to exclude sentinel values from a check) and use the
dedicated [`col_pct_missing()`](`pointblank.Validate.col_pct_missing`),
[`col_missing_coded()`](`pointblank.Validate.col_missing_coded`),
[`col_missing_only_coded()`](`pointblank.Validate.col_missing_only_coded`), and
[`col_missing_consistent()`](`pointblank.Validate.col_missing_consistent`) validation steps. See the
*Validation Methods* article for details.
:::


### Finding Differences Between Tables

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

A very common data question is *how do these two tables differ?* Maybe you refactored a pipeline
and want to know whether its output changed, migrated a table from one database to another, or
received a new extract of a data feed. In each case there's a *source* table (the reference, or the
data you expect) and a *target* table (the data you want to check), and you want to know:

- which records are missing from the target
- which records in the target are extra
- which records have values that differ, and in which columns
- whether the columns themselves (names, types, order) changed

Pointblank answers all of these with the [`compare()`](`pointblank.compare`) function, and the same
comparison is available as a validation step through `Validate.tbl_match()` and on the command line
through `pb compare`.

## A First Comparison

Let's make two small tables of orders. The target is a later extract of the same data: one order
had its amount corrected, one order is gone, and a new order has appeared.

```{python}
import polars as pl

source = pl.DataFrame(
    {
        "order_id": [1001, 1002, 1003, 1004, 1005],
        "customer": ["Ana", "Ben", "Cho", "Dev", "Eli"],
        "amount": [25.00, 40.50, 12.75, 99.99, 5.00],
        "status": ["shipped", "shipped", "pending", "shipped", "cancelled"],
    }
)

target = pl.DataFrame(
    {
        "order_id": [1001, 1002, 1004, 1005, 1006],
        "customer": ["Ana", "Ben", "Dev", "Eli", "Fay"],
        "amount": [25.00, 45.50, 99.99, 5.00, 18.25],
        "status": ["shipped", "shipped", "shipped", "cancelled", "pending"],
    }
)
```

Since every order has a unique `order_id`, we'll use it as the *key* to line up the rows of the two
tables:

```{python}
import pointblank as pb

comparison = pb.compare(source, target, keys="order_id")

comparison
```

The report's header gives the verdict and a breakdown of the rows. Below it, each section shows one
kind of difference: which columns have mismatched values (and how often), then samples of the
changed, missing, and extra rows. For changed rows, only the differing values are shown, as
*old → new* with the numeric difference alongside.

## Row Statuses

Every row that can be aligned between the two tables gets exactly one status:

| Status | Meaning |
|---|---|
| `match` | the row is in both tables and all compared values are equal |
| `changed` | the row is in both tables but at least one compared value differs |
| `missing` | the row is in the source but not in the target |
| `extra` | the row is in the target but not in the source |
| `dup_key` | the row's key appears more than once in either table, so it can't be aligned |

The counts are available directly on the `Comparison` object:

```{python}
comparison.status_counts
```

```{python}
comparison.n_changed, comparison.n_missing, comparison.n_extra
```

A `Comparison` is truthy only if the tables match, so it can be used directly in a test or an
`if` statement:

```{python}
bool(comparison)
```

## Getting the Differing Rows

Use the `rows()` method to get the rows with a given status as a table. Changed rows come with a
`<column>_source` and a `<column>_target` column for each compared column:

```{python}
comparison.rows("changed")
```

Missing rows come from the source and extra rows from the target:

```{python}
comparison.rows("missing")
```

```{python}
comparison.rows("extra")
```

Only up to `limit=` rows (100 by default) are collected, so this is safe even for very large
tables. For a per-column view of the mismatches, use `column_summary()`:

```{python}
comparison.column_summary()
```

## How Rows Are Aligned

The `keys=` argument decides how the rows of the two tables are lined up:

- **By key** (`keys="order_id"`, or a list for a composite key): rows are matched on the key
  values, so their order doesn't matter. This is the most informative mode, and the one to use
  whenever the data has an identifier.
- **By position** (`keys=None`, the default): the first row of the source is compared with the first
  row of the target, and so on. This suits outputs that are supposed to be identical row for row.
- **As multisets** (`keys="*"`): whole rows are compared regardless of their order, with
  duplicates counted. This is for tables that have no key at all.

Here's what positional alignment makes of the same two tables. Since a row was removed and another
added, most rows no longer line up:

```{python}
pb.compare(source, target).status_counts
```

## Tolerances, Normalization, and Nulls

Not every difference matters. Numeric and datetime columns can be compared with a tolerance, given
for all numeric columns or per column. An absolute tolerance of `5` absorbs the corrected amount:

```{python}
pb.compare(source, target, keys="order_id", tolerance={"amount": 5}).n_changed
```

Tolerances can be absolute (`atol`), relative to the source value (`rtol`), or both, as in
`tolerance={"amount": {"atol": 0.01, "rtol": 1e-6}}`. Two values are equal if
`|source - target| <= atol + rtol * |source|`. Datetime tolerances are durations like `"1s"`,
`"500ms"`, or `"2m"`.

String values can be normalized before comparing with `normalize=`. For example,
`normalize={"customer": ["strip", "lower"]}` ignores surrounding whitespace and letter case (the
available operations are `"strip"`, `"lower"`, `"upper"`, `"collapse_whitespace"`, and
`{"round": n}`). Reports always show the original values.

By default, two nulls count as equal, and NaN values in floating-point columns are treated as nulls.
Use `null_equal=False` to count two nulls as a difference.

## Schema Differences

Column differences are reported in the *Schema differences* section of the report and in the
`schema_diff` attribute. Suppose the target renamed a column and added another:

```{python}
target_v2 = target.rename({"customer": "customer_name"}).with_columns(
    region=pl.lit("EU")
)

comparison_v2 = pb.compare(source, target_v2, keys="order_id")

comparison_v2.schema_diff
```

With the default `schema="strict"`, any schema difference means the tables don't match and every
row counts as failing. Two arguments help here. `column_map=` pairs up renamed columns, and
`schema="common"` compares only the columns that both tables share (still reporting the schema
differences, but without failing any rows):

```{python}
pb.compare(
    source,
    target_v2,
    keys="order_id",
    column_map={"customer": "customer_name"},
    schema="common",
)
```

To compare just some of the columns, list them in `columns=`. The schema check is then limited to
the keys and those columns.

## Duplicate Keys

A key that appears more than once in either table can't be aligned unambiguously, so its rows get
the `dup_key` status (they always count as failing). By default the report lists the duplicated keys
with their number of occurrences in each table. With `dup_keys="compare"`, the report also compares
the duplicated rows pairwise to show which columns differ between them.

## Table Comparison as a Validation Step

The `tbl_match()` validation method uses the same comparison, which makes it a validation step like
any other: it can have thresholds and actions, and it shows up in the validation report. The table
being validated is the target, and `tbl_compare=` is the source. Every aligned row is a *test
unit*, so thresholds can be set as a proportion of rows:

```{python}
validation = (
    pb.Validate(data=target, tbl_name="orders", thresholds=(0.1, 0.25))
    .tbl_match(tbl_compare=source, keys="order_id")
    .interrogate()
)

validation
```

:::{.callout-note}
In Pointblank `v1.0.1` and earlier, `tbl_match()` had a single test unit for the whole table. Steps
still pass or fail in the same situations, but fractional thresholds now refer to the share of
differing rows. To react to *any* difference, as before, use an absolute threshold of `1`.
:::

Three of the six aligned rows differ, exceeding both the 'warning' and 'error' thresholds. The step
report holds the full comparison:

```{python}
validation.get_step_report(i=1)
```

The data extract for the step contains the failing rows from both tables, with a `_status_` column
and source/target value pairs:

```{python}
validation.get_data_extracts(i=1, frame=True)
```

All the comparison arguments (`keys=`, `tolerance=`, `schema=`, and so on) are available in
`tbl_match()`, and they can be written in YAML too:

```yaml
tbl: orders_target.csv
steps:
  - tbl_match:
      tbl_compare: orders_source.csv
      keys: order_id
      tolerance:
        amount: 0.01
```

## Reconciling Aggregates

Sometimes two tables shouldn't have the same rows but should add up to the same numbers: a
warehouse table loaded from a source system, a summary table built from detailed transactions, or
two systems that record the same activity at different levels of detail. For these, the
`tbl_reconcile()` validation method compares *aggregates* instead of rows. It groups both tables by
the `keys=` columns, aggregates the `measures=` columns (with `agg="sum"` by default, or `"count"`,
`"mean"`, `"min"`, or `"max"`), and compares the results group by group. Each group is a test unit.

Here, the orders in the target have been loaded at a finer grain than in the source, so we
reconcile the total amount per customer:

```{python}
order_lines = pl.DataFrame(
    {
        "order_id": [1001, 1002, 1002, 1004, 1005, 1006],
        "customer": ["Ana", "Ben", "Ben", "Dev", "Eli", "Fay"],
        "amount": [25.00, 40.50, 5.00, 99.99, 5.00, 18.25],
    }
)

validation = (
    pb.Validate(data=order_lines, tbl_name="order_lines")
    .tbl_reconcile(tbl_compare=source, keys="customer", measures="amount")
    .interrogate()
)

validation.get_step_report(i=1)
```

`Ben`'s two order lines add up to more than his order in the source, `Cho` is missing, and `Fay` is
new. Without `keys=`, `tbl_reconcile()` compares grand totals (a single test unit), and
`agg="count"` without `measures=` compares the number of rows in each group. Tolerances,
`column_map=`, and `partitions=` work as they do in `tbl_match()`. Rows with a null key form a group
of their own, which is matched with the other table's null-key group.

## Exporting the Results

The results can be serialized with [`to_dict()`](`Comparison.to_dict`) or
[`to_json()`](`Comparison.to_json`), which include the counts, the schema differences, and (by
default) samples of the differing rows. If the data could contain sensitive values, leave the
samples out:

```{python}
print(comparison.to_json(samples=False))
```

The tabular report can be saved as HTML, like any Great Tables table:

```python
comparison.get_tabular_report().write_raw_html("orders_diff.html")
```

## From the Command Line

The `pb compare` command runs a comparison on files, database tables, or built-in datasets, prints a
summary, and can save the HTML report and the JSON results:

```bash
pb compare orders_source.csv orders_target.csv --key order_id --output-html diff.html
```

Its exit code makes it easy to use in scripts and CI pipelines: `0` if the tables match, `1` if
they differ, and `2` if there was an error. Use `--format json` to print the results as JSON. All
the comparison settings have command-line options (e.g., `--tolerance amount=0.01`,
`--schema common`); see the [`pb compare` reference](../../reference/cli/compare.qmd) for the
full list.

For tables that are too large to fit in memory, see
[Scaling Up to Large Files](02-comparing-large-files.qmd).


### Scaling Up to Large Files

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_header=False, report_incl_footer_timings=False)
```

Here's a situation that comes up all the time: a pipeline run produces a target CSV file from a
source CSV file, and you need to check that nothing went wrong. Specifically, you want to find

1. records missing from the target,
2. extra records in the target,
3. records whose column values differ, and
4. schema or column-name mismatches,

and the files are big enough that loading both of them into memory is slow, or impossible.

This article looks at why the usual approaches struggle with this, and how to do it with
Pointblank's [`compare()`](../05-table-comparison/01-comparing-tables.qmd) function and the
`pb compare` command.

## Why the Usual Approaches Fall Short

**A pandas merge.** Reading both files and doing an outer merge on the key (with `indicator=True`)
does identify missing and extra rows, and the merged frame can be compared column by column to find
changed values. But it needs both files in memory *plus* the merged result, which is wider than
either input. For large files that's where memory runs out. Nulls take extra care too, since
`NaN != NaN`.

**Python sets.** Turning each file's rows into a set of tuples and taking set differences is quick to
write, but it can't tell a *changed* row from a missing one plus an extra one, can't say which
column changed, silently collapses duplicate rows, and still holds every row of both files in
memory (as Python objects, which is worse than a DataFrame).

**Reading in chunks.** Processing the files chunk by chunk keeps memory in check, but a row's
counterpart in the other file is usually in a *different* chunk, unless both files happen to be
sorted the same way. Making chunked comparison correct means partitioning both files by key, so
that each key's rows from both files land in the same chunk. That's a lot of code to get right.

## How Pointblank Approaches It

Three things make [`compare()`](`pointblank.compare`) suitable for large files:

1. **Files are scanned, not loaded.** CSV and Parquet paths are read lazily (with Polars, or
   DuckDB), so the comparison is a query plan rather than two DataFrames in memory.
2. **The work happens in the engine.** The join on the key, the per-column comparisons, and the
   counting are all done by the query engine. Python only receives the counts and small samples of
   the differing rows (the `limit=` of each sample).
3. **Chunking done correctly.** With `partitions=N`, the comparison runs in N passes, each covering
   the keys whose hash falls into one bucket, on *both* tables. A key's rows always land in the same
   pass, so the results are exactly those of a single-pass comparison, while each pass only joins
   about `1/N` of the rows.

Let's try it on a pair of CSV files. We'll generate a source file with 200,000 orders and a target
file in which some orders are missing, some amounts changed, a few orders were added, and the rows
were shuffled:

```{python}
import tempfile
from pathlib import Path

import numpy as np
import polars as pl

n = 200_000
rng = np.random.default_rng(23)

source = pl.DataFrame(
    {
        "order_id": np.arange(n),
        "amount": rng.normal(100, 20, n).round(2),
        "region": rng.choice(["north", "south", "east", "west"], n),
    }
)

target = (
    source.filter(pl.col("order_id") % 1000 != 7)  # 200 orders missing
    .with_columns(  # 400 amounts changed
        amount=pl.when(pl.col("order_id") % 500 == 3)
        .then(pl.col("amount") + 1)
        .otherwise("amount")
    )
    .vstack(source.tail(50).with_columns(pl.col("order_id") + n))  # 50 extra orders
    .sample(fraction=1.0, shuffle=True, seed=1)
)

data_dir = Path(tempfile.mkdtemp())
source.write_csv(data_dir / "source.csv")
target.write_csv(data_dir / "target.csv")
```

Now we compare the two files by giving `compare()` their paths:

```{python}
import pointblank as pb

comparison = pb.compare(
    data_dir / "source.csv",
    data_dir / "target.csv",
    keys="order_id",
)

comparison
```

All four questions are answered in one place: missing and extra rows, changed rows (with the
`amount` column identified as the one that differs), and the schema check in the header.

The same comparison in four partitions gives identical results:

```{python}
partitioned = pb.compare(
    data_dir / "source.csv",
    data_dir / "target.csv",
    keys="order_id",
    partitions=4,
)

partitioned.status_counts == comparison.status_counts
```

## Choosing an Engine and a Number of Partitions

When given file paths, `compare()` scans them with Polars, which is the fastest option. DuckDB uses
less memory, and combined with partitions it uses the least. To use DuckDB, read the files as DuckDB
relations and pass those to `compare()`:

```python
import duckdb

con = duckdb.connect()

comparison = pb.compare(
    con.read_csv("source.csv"),
    con.read_csv("target.csv"),
    keys="order_id",
    partitions=8,
)
```

Here's how the options compare on two CSV files with 10 million rows and 10 columns each (about
730 MB per file), with the rows of the target shuffled. Peak memory is the memory the process
actually owns (its physical footprint):

| Approach | Wall time | Peak memory |
|---|---|---|
| Polars: read both files, then a full join (for reference) | 2.1 s | 5.7 GB |
| `compare()` on file paths (Polars) | 5.7 s | 4.2 GB |
| `compare()` on file paths (Polars), `partitions=4` | 6.8 s | 1.6 GB |
| `compare()` on file paths (Polars), `partitions=8` | 10.2 s | 1.1 GB |
| `compare()` on DuckDB relations | 6.8 s | 1.6 GB |
| `compare()` on DuckDB relations, `partitions=8` | 19.0 s | 0.8 GB |

And with 100 million rows (about 7.4 GB per file), where comparing without partitions needs more
memory than most machines have:

| Approach | Wall time | Peak memory |
|---|---|---|
| `compare()` on file paths (Polars), `partitions=8` | 139 s | 6.6 GB |
| `compare()` on file paths (Polars), `partitions=16` | 217 s | 4.4 GB |
| `compare()` on DuckDB relations, `partitions=8` | 166 s | 2.0 GB |

Every approach found exactly the expected differences. The numbers will vary with your machine and
data, but the pattern is consistent: partitions trade time (each pass reads the files again) for
memory, Polars is the faster engine, and DuckDB the more memory-efficient one. Some practical
guidance:

- If the files fit comfortably in memory, use the defaults.
- If memory is tight, increase `partitions=` until the peak memory is acceptable; for the lowest
  memory use, read the files as DuckDB relations.
- Setting DuckDB's own `memory_limit` (so it can spill to disk) without partitions doesn't hold
  memory to the limit: with a 4 GB limit, the 100-million-row comparison peaked at 6.3 GB.
  Partitions give more predictable memory use.
- A good key matters most: with `keys=`, row order doesn't matter, and partitioning is possible.
  Without a key, rows are compared by position (or as multisets, with `keys="*"`), and partitions
  can't be used.

You can run the same benchmark on your own machine with the script in the Pointblank repository
(`benchmark/compare_large_files.py`).

## Tables in Databases

Tables don't have to be files. If both tables are in the same database (e.g., two Ibis tables from
one DuckDB, PostgreSQL, or Snowflake connection), the comparison runs entirely inside the database,
and only the counts and samples are transferred. Tables from *different* databases are first
brought into Polars, so for large tables it's better to have both in one place.

## In Scripts and CI Pipelines

The `pb compare` command does all of this from the command line, and its exit code tells a script or
CI job whether the files match (`0`), differ (`1`), or couldn't be compared (`2`):

```bash
pb compare source.csv target.csv --key order_id --engine duckdb --partitions 8 \
  --output-html diff.html --output-json diff.json
```

The HTML report can be kept as a build artifact for anyone to look at. If the data is sensitive, add
`--no-samples` to keep row values out of the JSON output. To make the comparison one check among
others in a validation, use `Validate.tbl_match()`, which accepts the same settings, including
`partitions=` (see [Finding Differences Between Tables](01-comparing-tables.qmd)).


### Generating Test Data

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Pointblank provides a built-in test data generation system that creates realistic, locale-aware
synthetic data based on schema definitions. This is useful for testing validation rules, creating
sample datasets, and generating fixture data for development.

This page covers the basics: defining a schema with typed fields, the available presets, and
the output formats. The rest of this section covers [profile fields](profile-fields.qmd) for
bundles of person data, [country-specific data](country-specific-data.qmd) for locale-aware
generation, and the [pytest fixture](pytest-fixture.qmd) for using generated data in tests.

::: {.callout-note}
Throughout this guide, we use `pb.preview()` to display generated datasets with nice HTML
formatting. This is optional: `pb.generate_dataset()` returns a standard DataFrame that you can
display or manipulate however you prefer.
:::

## Quick Start

Generate test data using a schema with field constraints:

```{python}
import pointblank as pb

# Define a schema with typed field specifications
schema = pb.Schema(
    user_id=pb.int_field(min_val=1, unique=True),
    name=pb.string_field(preset="name"),
    email=pb.string_field(preset="email"),
    age=pb.int_field(min_val=18, max_val=80),
    status=pb.string_field(allowed=["active", "pending", "inactive"]),
)

# Generate 100 rows of test data (seed ensures reproducibility)
pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

## Field Types

Pointblank provides helper functions for defining typed columns with constraints:

| Function | Description | Key Parameters |
|----------|-------------|----------------|
| `int_field()` | Integer columns | `min_val`, `max_val`, `allowed`, `unique` |
| `float_field()` | Float columns | `min_val`, `max_val`, `allowed` |
| `string_field()` | String columns | `preset`, `pattern`, `allowed`, `unique` |
| `bool_field()` | Boolean columns | `p_true` (probability of True) |
| `date_field()` | Date columns | `min_date`, `max_date` |
| `datetime_field()` | Datetime columns | `min_date`, `max_date` |
| `time_field()` | Time columns | `min_time`, `max_time` |
| `duration_field()` | Duration columns | `min_duration`, `max_duration` |
| `profile_fields()` | Bundled person-profile fields | `set`, `split_name`, `include`, `exclude`, `prefix` |

### Integer Fields

Integer fields support range constraints with `min_val` and `max_val`, discrete allowed values with
`allowed`, and uniqueness enforcement with `unique=True`:

```{python}
schema = pb.Schema(
    id=pb.int_field(min_val=1000, max_val=9999, unique=True),
    quantity=pb.int_field(min_val=1, max_val=100),
    rating=pb.int_field(allowed=[1, 2, 3, 4, 5]),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

The `unique=True` constraint ensures no duplicate values appear in that column, which is useful for
generating primary keys or identifiers.

### Float Fields

Float fields work similarly to integers, with `min_val` and `max_val` defining the range of
generated values:

```{python}
schema = pb.Schema(
    price=pb.float_field(min_val=0.0, max_val=1000.0),
    discount=pb.float_field(min_val=0.0, max_val=0.5),
    temperature=pb.float_field(min_val=-40.0, max_val=50.0),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

Values are uniformly distributed across the specified range, making this useful for simulating
measurements, prices, or any continuous numeric data.

### String Fields with Presets

Presets generate realistic data like names, emails, and addresses. When you include related
fields like `name` and `email` in the same schema, Pointblank ensures **coherence** (e.g., the email
address will be derived from the person's name), making the generated data more realistic:

```{python}
schema = pb.Schema(
    full_name=pb.string_field(preset="name"),
    email=pb.string_field(preset="email"),
    company=pb.string_field(preset="company"),
    city=pb.string_field(preset="city"),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

This coherence extends to other related fields like `user_name`, which will also reflect the
person's name when included alongside name and email fields.

### String Fields with Patterns

Use regex patterns to generate strings matching specific formats:

```{python}
schema = pb.Schema(
    product_code=pb.string_field(pattern=r"[A-Z]{3}-[0-9]{4}"),
    phone=pb.string_field(pattern=r"\([0-9]{3}\) [0-9]{3}-[0-9]{4}"),
    hex_color=pb.string_field(pattern=r"#[0-9A-F]{6}"),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

Patterns support standard regex character classes and quantifiers, giving you flexibility to
generate data matching virtually any format specification.

### Boolean Fields

Control the probability of `True` values:

```{python}
schema = pb.Schema(
    is_active=pb.bool_field(p_true=0.8),      # 80% True
    is_premium=pb.bool_field(p_true=0.2),     # 20% True
    is_verified=pb.bool_field(),              # 50% True (default)
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

This probabilistic control is helpful when you need to simulate real-world distributions where
certain states are more common than others.

### Date and Datetime Fields

Temporal fields accept Python `date` and `datetime` objects for their range boundaries, generating
values uniformly distributed within the specified period:

```{python}
from datetime import date, datetime

schema = pb.Schema(
    birth_date=pb.date_field(
        min_date=date(1960, 1, 1),
        max_date=date(2005, 12, 31)
    ),
    created_at=pb.datetime_field(
        min_date=datetime(2024, 1, 1),
        max_date=datetime(2024, 12, 31)
    ),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

The same pattern applies to `time_field()` and `duration_field()`, allowing you to generate
realistic temporal data for any use case.

## Available Presets

The `preset=` parameter in `string_field()` supports many data types:

**Personal Data:**

- `name`: full name (first + last)
- `name_full`: full name with optional prefix/suffix (e.g., "Dr. Ana Sousa", "Prof. Tanaka Yuki")
- `first_name`: first name only
- `last_name`: last name only
- `gender`: person's gender (`"male"` or `"female"`), coherent with name fields
- `email`: email address
- `phone_number`: phone number in country-specific format

**Location Data:**

- `address`: full street address
- `city`: city name
- `state`: state/province name
- `country`: country name
- `country_code_2`: ISO 3166-1 alpha-2 country code (e.g., `"US"`)
- `country_code_3`: ISO 3166-1 alpha-3 country code (e.g., `"USA"`)
- `postcode`: postal/ZIP code
- `latitude`: latitude coordinate
- `longitude`: longitude coordinate

**Business Data:**

- `company`: company name
- `job`: job title
- `catch_phrase`: business catch phrase

**Internet Data:**

- `url`: website URL
- `domain_name`: domain name
- `ipv4`: IPv4 address
- `ipv6`: IPv6 address
- `user_name`: username
- `password`: password

**Financial Data:**

- `credit_card_number`: credit card number
- `credit_card_provider`: card network name (Visa, Mastercard, American Express, or Discover); coherent with `credit_card_number`
- `iban`: International Bank Account Number
- `currency_code`: currency code (USD, EUR, etc.)

**Identifiers:**

- `uuid4`: UUID version 4
- `md5`: MD5 hash (32 hex characters)
- `sha1`: SHA-1 hash (40 hex characters)
- `sha256`: SHA-256 hash (64 hex characters)
- `ssn`: Social Security Number (country-specific format)
- `license_plate`: vehicle license plate (location-aware for CA, US, DE, AU, GB)

**Barcodes:**

- `ean8`: EAN-8 barcode with valid check digit
- `ean13`: EAN-13 barcode with valid check digit

**Date/Time:**

- `date_this_year`: a date within the current year
- `date_this_decade`: a date within the current decade
- `date_between`: a random date between 2000 and 2025
- `date_range`: two dates joined with an en-dash (e.g., `"2012-05-12 – 2015-11-22"`)
- `future_date`: a date up to 1 year in the future
- `past_date`: a date up to 10 years in the past
- `time`: a time value

**Text:**

- `word`: single word
- `sentence`: full sentence
- `paragraph`: paragraph of text
- `text`: multiple paragraphs

**Miscellaneous:**

- `color_name`: color name
- `file_name`: file name
- `file_extension`: file extension
- `mime_type`: MIME type
- `user_agent`: browser user agent string (country-weighted)
- `locale_code`: locale identifier (e.g., `"en_US"`, `"de_DE"`; multilingual countries return a random official locale)

## Output Formats

The `generate_dataset()` function supports multiple output formats via the `output=` parameter,
making it easy to integrate with your preferred data processing library.

```{python}
schema = pb.Schema(
    id=pb.int_field(min_val=1),
    name=pb.string_field(preset="name"),
)
```

The default output is a Polars DataFrame, which offers excellent performance and a modern API for
data manipulation:

```{python}
polars_df = pb.generate_dataset(schema, n=100, seed=23, output="polars")

pb.preview(polars_df)
```

If your workflow uses Pandas, simply specify `output="pandas"` to get a **Pandas DataFrame**:

```{python}
pandas_df = pb.generate_dataset(schema, n=100, seed=23, output="pandas")

pb.preview(pandas_df)
```

Both formats work seamlessly with Pointblank's validation functions, so you can choose whichever
fits best with your existing data pipeline.

## Using Generated Data for Validation Testing

A common use case is generating test data to validate your validation rules:

```{python}
# Define a schema with constraints
schema = pb.Schema(
    user_id=pb.int_field(min_val=1, unique=True),
    email=pb.string_field(preset="email"),
    age=pb.int_field(min_val=18, max_val=100),
    status=pb.string_field(allowed=["active", "pending", "inactive"]),
)

# Generate test data
test_data = pb.generate_dataset(schema, n=100, seed=23)

# Validate the generated data (it should pass all checks)
validation = (
    pb.Validate(test_data)
    .col_vals_gt("user_id", 0)
    .col_vals_regex("email", r".+@.+\..+")
    .col_vals_between("age", 18, 100)
    .col_vals_in_set("status", ["active", "pending", "inactive"])
    .interrogate()
)

validation
```

Since the generated data respects the constraints defined in the schema, it should pass all
validation checks. This workflow is particularly useful for testing validation logic before
applying it to production data, or for creating reproducible test fixtures in your CI/CD pipeline.

## Conclusion

Test data generation provides a convenient way to create realistic synthetic datasets directly from
schema definitions. While the concept is straightforward (defining field types and constraints, then
generating matching data), the feature can be invaluable in many development and testing workflows.
By incorporating test data generation into your process, you can:

- quickly prototype validation rules before working with production data
- create reproducible test fixtures for automated testing and CI/CD pipelines
- generate locale-specific data for internationalization testing across 100 countries
- ensure coherent relationships between related fields like names, emails, addresses, jobs, and
license plates
- produce datasets of any size with consistent, realistic values

Whether you're building validation logic, testing data pipelines, or simply need sample data for
development, the schema-based generation approach gives you precise control over data
characteristics while maintaining the realism needed to uncover edge cases and validate your
assumptions about data quality.


### Profile Fields

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

When generating person-profile data, you often need several related presets together: a name, an
email derived from that name, an address, a phone number, and so on. Rather than wiring up each
column individually, the `profile_fields()` helper returns a ready-made dictionary of
`StringField` objects that you can unpack directly into a `Schema()`.

## Basic Usage

With no arguments, `profile_fields()` returns the **standard** set of seven columns:
`first_name`, `last_name`, `email`, `city`, `state`, `postcode`, and `phone_number`. All
coherence rules apply automatically: emails are derived from names, and city/state/postcode/phone
are internally consistent.

```{python}
schema = pb.Schema(
    user_id=pb.int_field(unique=True, min_val=1),
    **pb.profile_fields(),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

The `**` operator unpacks the dictionary into keyword arguments, as if you had written each
`string_field(preset=...)` call by hand.

## Choosing a Set

Three built-in sets control how many columns are generated:

| Set | Columns |
|-----|------|
| `"minimal"` | `first_name`, `last_name`, `email`, `phone_number` |
| `"standard"` | `first_name`, `last_name`, `email`, `city`, `state`, `postcode`, `phone_number` |
| `"full"` | `first_name`, `last_name`, `email`, `address`, `city`, `state`, `postcode`, `phone_number`, `company`, `job` |

```{python}
# Minimal profile: just name, email, and phone
pb.preview(
    pb.generate_dataset(
        pb.Schema(**pb.profile_fields(set="minimal")),
        n=100, seed=23,
    )
)
```

```{python}
# Full profile: includes address, company, and job title
pb.preview(
    pb.generate_dataset(
        pb.Schema(**pb.profile_fields(set="full")),
        n=100, seed=23,
    )
)
```

## Combined vs. Split Names

By default, names are split into `first_name` and `last_name` columns. Set `split_name=False` to
get a single `name` column instead:

```{python}
pb.preview(
    pb.generate_dataset(
        pb.Schema(**pb.profile_fields(set="minimal", split_name=False)),
        n=100, seed=23,
    )
)
```

## Adding and Removing Columns

Use `include=` to add presets to the base set and `exclude=` to remove them. Both accept lists of
preset names. The available profile presets are: `first_name`, `last_name`, `name`, `email`,
`address`, `city`, `state`, `postcode`, `phone_number`, `company`, and `job`.

```{python}
# Standard set + company column
pb.preview(
    pb.generate_dataset(
        pb.Schema(**pb.profile_fields(include=["company"])),
        n=100, seed=23,
    )
)
```

```{python}
# Standard set without city and state
pb.preview(
    pb.generate_dataset(
        pb.Schema(**pb.profile_fields(exclude=["city", "state"])),
        n=100, seed=23,
    )
)
```

You can combine `include=` and `exclude=` in the same call, as long as the same preset does not
appear in both.

## Column Prefixes

The `prefix=` parameter prepends a string to every column name. This is especially useful when a
schema needs two independent profiles (e.g., sender and recipient):

```{python}
schema = pb.Schema(
    **pb.profile_fields(set="minimal", prefix="sender_"),
    **pb.profile_fields(set="minimal", prefix="recipient_"),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23))
```

Each prefixed profile is a separate person with its own coherence: the sender's email is derived
from the sender's name, and the recipient's email from the recipient's name. The two profiles also
get their own locations and employers, so a row's sender and recipient are independent of each
other. Other preset columns whose names start with a profile's prefix (e.g., a hand-written
`sender_city=pb.string_field(preset="city")` added to a `"minimal"` sender profile) belong to that
profile.

## Combining with Other Field Types

Since `profile_fields()` returns a plain dictionary, it composes naturally with any other field
types:

```{python}
schema = pb.Schema(
    id=pb.int_field(unique=True, min_val=1000),
    **pb.profile_fields(),
    active=pb.bool_field(p_true=0.8),
    signup_date=pb.date_field(
        min_date="2024-01-01",
        max_date="2025-12-31",
    ),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23, country="DE"))
```


### Country-Specific Data

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

One of the most powerful features is generating locale-aware data. Use the `country=` parameter
to generate data specific to a country. This affects names, cities, addresses, and other
locale-sensitive presets.

Let's create a schema that includes several location-related fields. When generating data for a
specific country, Pointblank ensures *consistency across related fields*. The city, address,
postcode, and coordinates will all correspond to the same location:

```{python}
# Schema with linked location fields
schema = pb.Schema(
    name=pb.string_field(preset="name"),
    city=pb.string_field(preset="city"),
    address=pb.string_field(preset="address"),
    postcode=pb.string_field(preset="postcode"),
    latitude=pb.string_field(preset="latitude"),
    longitude=pb.string_field(preset="longitude"),
)
```

Here's German data with authentic names and addresses from cities like Berlin, Munich, and Hamburg.
Notice how the latitude/longitude coordinates match real locations in Germany:

```{python}
pb.preview(pb.generate_dataset(schema, n=200, seed=23, country="DE"))
```

Japanese data includes names in romanized form and addresses from cities like Tokyo, Osaka, and
Kyoto. The coordinates fall within Japan's geographic boundaries:

```{python}
pb.preview(pb.generate_dataset(schema, n=200, seed=23, country="JP"))
```

Brazilian data features Portuguese names and addresses from cities like São Paulo, Rio de Janeiro,
and Brasília. The postal codes follow Brazil's CEP format:

```{python}
pb.preview(pb.generate_dataset(schema, n=200, seed=23, country="BR"))
```

This location coherence is valuable when testing geospatial applications, address validation
systems, or any scenario where realistic, internally-consistent location data matters.

## Data Coherence

Pointblank automatically links related columns to produce realistic rows. There are three
coherence systems that activate based on which presets appear together in a schema:

**Address coherence** activates when *any* address-related preset is present (`address`, `city`,
`state`, `postcode`, `latitude`, `longitude`, `phone_number`, `license_plate`). All of these
fields will refer to the same location within each row, and the postcode within an `address` is
the same as the one in the `postcode` column.

**Person coherence** activates when *any* person-related preset is present (`name`, `name_full`,
`first_name`, `last_name`, `email`, `user_name`). The email and username are derived from the
person's name.

**Business coherence** activates when *both* `job` and `company` are present. When active:

- the company and job title are drawn from the same industry (e.g., a nurse will work at a
hospital, not a law firm).
- `name_full` gains profession-matched titles: a doctor may appear as "Dr. Ana Sousa" and a
professor as "Prof. Tanaka Yuki". For German-speaking countries (DE, AT, CH), the honorific
stacks before the professional title (e.g., "Herr Dr. med. Klaus Weber").
- integer columns whose name contains `age` (e.g., `age`, `person_age`) are automatically
constrained to working-age range (22–65).

Here's an example showing all three coherence systems working together:

```{python}
schema = pb.Schema(
    name=pb.string_field(preset="name_full"),
    email=pb.string_field(preset="email"),
    company=pb.string_field(preset="company"),
    job=pb.string_field(preset="job"),
    city=pb.string_field(preset="city"),
    state=pb.string_field(preset="state"),
    license_plate=pb.string_field(preset="license_plate"),
    age=pb.int_field(),
)

pb.preview(pb.generate_dataset(schema, n=100, seed=23, country="DE"))
```

**License plate coherence** is part of address coherence. For `CA`, `US`, `DE`, `AU`, and `GB`,
license plates follow real subregion-specific formats when location fields are present. For example,
an Ontario row produces plates like `"CABC 123"` while a British Columbia row produces `"AB1 23C"`.
Letters I, O, Q, and U are excluded from plate generation, matching real-world restrictions.

## Supported Countries

Pointblank currently supports 100 countries with full locale data for realistic test data generation.
You can use either ISO 3166-1 alpha-2 codes (e.g., `"US"`) or alpha-3 codes (e.g., `"USA"`).

**Europe (38 countries):**

- Armenia (`AM`), Austria (`AT`), Azerbaijan (`AZ`), Belgium (`BE`), Bulgaria (`BG`), Croatia (`HR`), Cyprus (`CY`), Czech Republic (`CZ`), Denmark (`DK`), Estonia (`EE`), Finland (`FI`), France (`FR`), Georgia (`GE`), Germany (`DE`), Greece (`GR`), Hungary (`HU`), Iceland (`IS`), Ireland (`IE`), Italy (`IT`), Latvia (`LV`), Lithuania (`LT`), Luxembourg (`LU`), Malta (`MT`), Moldova (`MD`), Netherlands (`NL`), Norway (`NO`), Poland (`PL`), Portugal (`PT`), Romania (`RO`), Russia (`RU`), Serbia (`RS`), Slovakia (`SK`), Slovenia (`SI`), Spain (`ES`), Sweden (`SE`), Switzerland (`CH`), Ukraine (`UA`), United Kingdom (`GB`)

**Americas (19 countries):**

- Argentina (`AR`), Bolivia (`BO`), Brazil (`BR`), Canada (`CA`), Chile (`CL`), Colombia (`CO`), Costa Rica (`CR`), Dominican Republic (`DO`), Ecuador (`EC`), El Salvador (`SV`), Guatemala (`GT`), Honduras (`HN`), Jamaica (`JM`), Mexico (`MX`), Panama (`PA`), Paraguay (`PY`), Peru (`PE`), United States (`US`), Uruguay (`UY`)

**Asia-Pacific (22 countries):**

- Australia (`AU`), Bangladesh (`BD`), Cambodia (`KH`), China (`CN`), Hong Kong (`HK`), India (`IN`), Indonesia (`ID`), Japan (`JP`), Kazakhstan (`KZ`), Malaysia (`MY`), Myanmar (`MM`), Nepal (`NP`), New Zealand (`NZ`), Pakistan (`PK`), Philippines (`PH`), Singapore (`SG`), South Korea (`KR`), Sri Lanka (`LK`), Taiwan (`TW`), Thailand (`TH`), Uzbekistan (`UZ`), Vietnam (`VN`)

**Middle East & Africa (21 countries):**

- Algeria (`DZ`), Cameroon (`CM`), Egypt (`EG`), Ethiopia (`ET`), Ghana (`GH`), Israel (`IL`), Jordan (`JO`), Kenya (`KE`), Lebanon (`LB`), Morocco (`MA`), Mozambique (`MZ`), Nigeria (`NG`), Rwanda (`RW`), Saudi Arabia (`SA`), Senegal (`SN`), South Africa (`ZA`), Tanzania (`TZ`), Tunisia (`TN`), Turkey (`TR`), Uganda (`UG`), United Arab Emirates (`AE`)

Additional countries and expanded coverage are planned for future releases.

## Mixing Multiple Countries

When you need test data that spans multiple locales (e.g., simulating an international customer
base), you can pass a list or dict to the `country=` parameter instead of a single string.

Passing a list of country codes splits rows equally across those countries. Here, 200 rows are
divided evenly among the US, Germany, and Japan (~67 each):

```{python}
schema = pb.Schema(
    name=pb.string_field(preset="name"),
    city=pb.string_field(preset="city"),
    postcode=pb.string_field(preset="postcode"),
)

pb.preview(pb.generate_dataset(schema, n=200, seed=23, country=["US", "DE", "JP"]))
```

To control the proportion of rows per country, pass a dict mapping country codes to weights. The
following generates 200 rows with 70% from the US, 20% from Germany, and 10% from France:

```{python}
pb.preview(
    pb.generate_dataset(
        schema, n=200, seed=23,
        country={"US": 0.7, "DE": 0.2, "FR": 0.1},
    )
)
```

Weights are auto-normalized, so `{"US": 7, "DE": 2, "FR": 1}` is equivalent to the example above.
Row counts are allocated using largest-remainder apportionment, ensuring they always sum to exactly
`n`.

By default, rows from different countries are interleaved randomly (`shuffle=True`). Set
`shuffle=False` to keep rows grouped by country in the order the countries are listed:

```{python}
pb.preview(
    pb.generate_dataset(
        schema, n=120, seed=23,
        country=["US", "DE", "JP"], shuffle=False,
    )
)
```

All coherence systems (address, person, business) work correctly within each country's batch of
rows. A French row will have a French name with a matching French email; a Japanese row will have a
Japanese name with a matching Japanese email. Non-preset columns (integers, floats, booleans, dates)
are generated independently for each batch but still respect their field constraints.

## Frequency-Weighted Sampling

By default, names and cities are sampled uniformly at random from the locale data, giving every
entry the same probability of being selected. Real-world distributions are far from uniform though:
"James" and "Maria" appear orders of magnitude more often than "Thaddeus" or "Xiomara", and more
people live in New York City than in Flagstaff. The `weighted=True` parameter makes generated data
reflect this natural skew.

```{python}
schema = pb.Schema(
    name=pb.string_field(preset="name"),
    city=pb.string_field(preset="city"),
)

pb.preview(pb.generate_dataset(schema, n=200, seed=23, country="US", weighted=True))
```

With weighting enabled you will see popular names like James, John, Mary, and Patricia appear more
frequently, while unusual names surface only occasionally. Similarly, cities like New York, Los
Angeles, and Chicago dominate the output while smaller cities appear less often.

The feature works by organizing locale data into four frequency tiers. Each tier has a sampling
probability that determines how likely its members are to be selected:

| Tier | Probability | Contents |
|------|-------------|----------|
| very_common | 45% | The top ~10% of entries by real-world frequency |
| common | 30% | The next ~20% of entries |
| uncommon | 20% | The next ~30% of entries |
| rare | 5% | The remaining ~40% of entries |

When a value is needed, a tier is first chosen according to these probabilities and then a single
entry is picked uniformly at random within that tier. This two-step approach keeps sampling fast
while producing a realistic long-tail distribution. Setting `weighted=False` pools all entries
across every tier and samples them uniformly, which can be useful when you want an even spread
rather than a realistic distribution.

Weighted sampling combines seamlessly with multi-country mixing. Each country's batch uses its own
tiered data independently, so a mixed dataset will have weighted US names alongside weighted German
names:

```{python}
pb.preview(
    pb.generate_dataset(
        schema,
        n=200,
        seed=23,
        country={"US": 0.6, "DE": 0.4},
        weighted=True,
    )
)
```

All 100 supported country locales have tiered name and location data, so `weighted=True` produces
realistic frequency distributions for every country.


### The pytest Fixture

When Pointblank is installed, a `generate_dataset` **pytest fixture** is automatically available
in all your test files. There is no need to import anything or add configuration to `conftest.py`:
the fixture is registered via pytest's plugin system.

The fixture works identically to `pb.generate_dataset()`, but with one key difference: when you
don't supply a `seed=` parameter, a deterministic seed is automatically derived from the test's
fully-qualified name. This means:

- the **same test** always produces the **same data**: no manual seed management required.
- *different tests* get different seeds, so they exercise different datasets.
- you can still pass an explicit `seed=` to override the automatic seed when needed.

## Basic Usage

Use it by adding `generate_dataset` to your test function's parameter list:

```{.python filename="test_pipeline.py"}
import pointblank as pb

def test_etl_handles_nulls(generate_dataset):
    schema = pb.Schema(
        user_id=pb.int_field(unique=True),
        email=pb.string_field(preset="email", nullable=True, null_probability=0.3),
        age=pb.int_field(min_val=0, max_val=120),
    )

    df = generate_dataset(schema, n=500)
    result = my_etl_pipeline(df)
    assert result.filter(pl.col("email").is_null()).shape[0] == 0
```

All parameters from `generate_dataset()` are supported: `n=`, `seed=`, `output=`, and `country=`:

```python
def test_german_data(generate_dataset):
    schema = pb.Schema(
        name=pb.string_field(preset="name"),
        city=pb.string_field(preset="city"),
    )

    df = generate_dataset(schema, n=200, country="DE", output="pandas")
    assert len(df) == 200
```

## Multiple Datasets in One Test

Calling the fixture multiple times within the same test produces different (but still deterministic) data on each call:

```python
def test_merge_pipeline(generate_dataset):
    customers = generate_dataset(customer_schema, n=1000, country="US")
    orders = generate_dataset(order_schema, n=5000)

    # Each call gets a unique seed derived from the test name + call index,
    # so both DataFrames are deterministic and different from each other.
    result = merge_pipeline(customers, orders)
    assert result.shape[0] > 0
```

## Testing Across Locales

The fixture makes locale testing particularly concise when combined with `pytest.mark.parametrize`:

```python
import pytest
import pointblank as pb

@pytest.mark.parametrize("country", ["US", "DE", "JP", "BR"])
def test_name_normalizer(generate_dataset, country):
    schema = pb.Schema(name=pb.string_field(preset="name_full"))
    df = generate_dataset(schema, n=100, country=country)
    result = normalize_names(df)
    assert result["name"].str.len_chars().min() > 0
```

## Sharing Schemas Across Tests

Define schemas as fixtures in `conftest.py` and compose them with `generate_dataset`:

```{.python filename="conftest.py"}
import pytest
import pointblank as pb

@pytest.fixture
def customer_schema():
    return pb.Schema(
        id=pb.int_field(unique=True),
        name=pb.string_field(preset="name"),
        email=pb.string_field(preset="email"),
        city=pb.string_field(preset="city"),
    )
```

```{.python filename="test_validation.py"}
def test_customer_validation(generate_dataset, customer_schema):
    df = generate_dataset(customer_schema, n=200, country="DE")
    validation = pb.Validate(df).col_vals_not_null(columns="email").interrogate()
    assert validation.all_passed()
```

```{.python filename="test_export.py"}
def test_customer_export(generate_dataset, customer_schema):
    df = generate_dataset(customer_schema, n=50, country="JP")
    exported = export_to_parquet(df)
    assert exported.exists()
```

## Debugging with Seed Introspection

The fixture callable exposes two attributes that make debugging failed tests straightforward:

- `generate_dataset.default_seed`: the base seed derived from the test name (available before any call)
- `generate_dataset.last_seed`: the seed actually used for the most recent call (accounts for the call counter and explicit overrides)

Include `.last_seed` in assertion messages so failures are immediately reproducible:

```python
def test_age_range(generate_dataset):
    schema = pb.Schema(age=pb.int_field(min_val=18, max_val=100))
    df = generate_dataset(schema, n=500)
    min_age = df["age"].min()
    assert min_age >= 18, (
        f"Expected min age >= 18, got {min_age} (seed={generate_dataset.last_seed})"
    )
```

You can also use `.default_seed` to reproduce the exact dataset outside of pytest:

```python
# In a REPL or notebook, reproduce the data from a failed test:
import pointblank as pb
df = pb.generate_dataset(schema, n=500, seed=<default_seed_from_output>)
```

## Seed Stability

A given seed (whether explicit or auto-derived) is guaranteed to produce identical output **within
the same Pointblank version**. Across versions, changes to country data files or generator logic
may alter the output for a given seed.

For CI pipelines that require bit-exact data across library upgrades, we recommend saving
generated DataFrames as Parquet or CSV snapshot files rather than relying on cross-version seed
stability. This is the same approach used by snapshot-testing tools like `pytest-snapshot` and
`syrupy`.


### YAML Validation Workflows

Pointblank supports defining validation workflows using YAML configuration files, providing a
declarative, readable, and maintainable approach to data validation. YAML workflows are particularly
useful for teams, version control, automation pipelines, and scenarios where you want to separate
validation logic from application code.

YAML validation workflows offer several advantages: they're easy to read and write, can be version
controlled alongside your data processing code, enable non-programmers to contribute to data quality
definitions, and provide a clear separation between validation logic and execution code.

The YAML approach complements Pointblank's Python API, giving you flexibility to choose the right
tool for each situation. Simple, repetitive validations work well in YAML, while complex logic with
custom functions might be better suited for the Python API.

## Basic YAML Validation Structure

A YAML validation workflow consists of a few key components:

- `tbl`: specifies the data source (file path, dataset name, or Python expression)
- `steps`: defines the validation checks to perform
- Optional metadata: table name, label, thresholds, actions, and other configuration

Here's a simple example validating the built-in `small_table` dataset:

```yaml
tbl: small_table
df_library: polars                     # Optional: specify DataFrame library
tbl_name: "Small Table Validation"
label: "Basic data quality checks"
steps:
  - rows_distinct
  - col_exists:
      columns: [a, b, c, d]
  - col_vals_not_null:
      columns: [a, b]
```

You can save this configuration to a .yaml file and execute it using the `yaml_interrogate()`
function:

```{python}
import pointblank as pb
from pathlib import Path

# Save the YAML configuration to a file
yaml_content = """
tbl: small_table
df_library: polars
tbl_name: "Small Table Validation"
label: "Basic data quality checks"
steps:
  - rows_distinct
  - col_exists:
      columns: [a, b, c, d]
  - col_vals_not_null:
      columns: [a, b]
"""

yaml_file = Path("basic_validation.yaml")
yaml_file.write_text(yaml_content)

# Execute the validation from the file
result = pb.yaml_interrogate(yaml_file)
result
```

The validation table shows the results of each step, just as if you had written the equivalent
Python code. You can also pass YAML content directly as a string for quick testing, but working
with files is the recommended approach for production workflows.

## Data Sources in YAML

The `tbl` field supports various data source types, making it easy to work with different kinds of
data. You can also control the DataFrame library used for loading data with the `df_library`
parameter.

### DataFrame Library Selection

By default, Pointblank loads data as Polars DataFrames, but you can specify alternative libraries:

```yaml
# Load as Polars DataFrame (default)
tbl: small_table
df_library: polars

# Load as Pandas DataFrame
tbl: small_table
df_library: pandas

# Load as DuckDB table (via Ibis)
tbl: small_table
df_library: duckdb
```

This is particularly useful when using validation expressions that require specific DataFrame APIs:

```yaml
# Using Pandas-specific operations
tbl: small_table
df_library: pandas
steps:
  - specially:
      expr: "lambda df: df.assign(total=df['a'] + df['d'])"

# Using Polars-specific operations
tbl: small_table
df_library: polars
steps:
  - specially:
      expr: "lambda df: df.select(pl.col('a') + pl.col('d') > 0)"
```

### File-based Sources

```yaml
# CSV files (respects df_library setting)
tbl: "data/customers.csv"
df_library: pandas

# Parquet files
tbl: "warehouse/sales.parquet"
df_library: polars

# Multiple files with patterns
tbl: "logs/*.parquet"
```

### Built-in Datasets

```yaml
# Use Pointblank's built-in datasets
tbl: small_table
tbl: game_revenue
tbl: nycflights
```

### Python Expressions for Complex Sources

For more complex data loading, use the `python:` block syntax. This syntax can be used with several
parameters throughout your YAML configuration:

- `tbl`: For complex data source loading (as shown below)
- `expr`: For custom validation expressions in `col_vals_expr`
- `pre`: For data preprocessing before validation steps
- `actions`: For callable action functions (`warning`, `error`, `critical`, and `default`)

```yaml
# Load data with custom Polars operations
tbl:
  python: |
    pl.scan_csv("sales_data.csv")
    .filter(pl.col("date") >= "2024-01-01")
    .head(1000)

# Load from a database connection
tbl:
  python: |
    pl.read_database(
        query="SELECT * FROM customers WHERE active = true",
        connection="postgresql://user:pass@localhost/db"
    )
```

### Allowing Python Code to Run

Python code in a YAML configuration only runs if you allow it, with `allow_python=True` in
`yaml_interrogate()` (or the `--allow-python` flag of `pb run`):

```python
result = pb.yaml_interrogate("validation.yaml", allow_python=True)
```

Without it, a configuration that contains Python code raises an error saying so. Python code in YAML
isn't sandboxed: it runs with the same access as any Python script you run, so it can read and
write files, make network requests, or run commands. Treat a YAML configuration that contains Python
code like a Python script, and only allow it for configurations you wrote or otherwise trust.

A few `pb` functions don't need `allow_python=True` when they're called with plain values (strings,
numbers, lists, and so on): `pb.load_dataset()`, `pb.col()`, `pb.ref()`, `pb.seg_group()`,
`pb.has_columns()`, `pb.has_rows()`, and the column selectors (`pb.starts_with()`, `pb.ends_with()`,
`pb.contains()`, `pb.matches()`, `pb.everything()`, `pb.first_n()`, and `pb.last_n()`). So do
`Decimal()`, `ZoneInfo()`, and the `datetime.date()`, `datetime.time()`, `datetime.datetime()`, and
`datetime.timedelta()` constructors, for values like `Decimal("2.2")` that YAML can't write. These
only load a bundled dataset or build an object, so a configuration like this one works without it:

```yaml
tbl:
  python: pb.load_dataset("small_table", tbl_type="polars")
steps:
  - col_vals_gt:
      columns: [d]
      value: pb.col("c")
```

## Reusable Templates with `set_tbl=`

One of the most powerful features of YAML validation workflows is the ability to create reusable
templates that can be applied to different datasets. Using the `set_tbl=` parameter with
`yaml_interrogate()`, you can define validation logic once and apply it to multiple data sources.

### Creating Validation Templates

When creating templates for use with `set_tbl=`, the `tbl` field is still required but its value
will be overridden. The recommended approach is to use `tbl: null`:

```yaml
tbl: null
tbl_name: "Sales Data Validation Template"
label: "Standard validation checks for sales data"
steps:
  - col_exists:
      columns: [customer_id, revenue, region, date]
  - col_vals_not_null:
      columns: [customer_id, revenue]
  - col_vals_gt:
      columns: [revenue]
      value: 0
  - col_vals_in_set:
      columns: [region]
      set: [North, South, East, West]
```

### Applying Templates to Multiple Datasets

Here's a practical example showing how to apply the same validation template to multiple quarterly
datasets, demonstrating the power of reusable YAML configurations:

```{python}
import pointblank as pb
import polars as pl

# Define the template once
sales_template = """
tbl: null  # Will be overridden
tbl_name: "Sales Data Validation"
label: "Standard sales validation checks"
thresholds:
  warning: 0.05
  error: 0.1
steps:
  - col_exists:
      columns: [customer_id, revenue, region]
  - col_vals_not_null:
      columns: [customer_id, revenue]
  - col_vals_gt:
      columns: [revenue]
      value: 0
  - col_vals_in_set:
      columns: [region]
      set: [North, South, East, West]
"""

# Create different datasets
q1_data = pl.DataFrame({
    "customer_id": [1, 2, 3, 4],
    "revenue": [100, 200, 150, 300],
    "region": ["North", "South", "East", "West"]
})

q2_data = pl.DataFrame({
    "customer_id": [5, 6, 7, 8],
    "revenue": [250, 180, 220, 350],
    "region": ["South", "North", "West", "East"]
})

# Apply the same template to both datasets
q1_result = pb.yaml_interrogate(sales_template, set_tbl=q1_data)
q2_result = pb.yaml_interrogate(sales_template, set_tbl=q2_data)

print(f"Q1 validation: {all(v.all_passed for v in q1_result.validation_info)}")
print(f"Q2 validation: {all(v.all_passed for v in q2_result.validation_info)}")
```

### Template Best Practices

1. **Use `tbl: null`**: this clearly indicates the template expects a data source to be provided
2. **Include comprehensive metadata**: use `tbl_name`, `label`, and `brief` to make results
self-documenting
3. **Set appropriate thresholds**: define warning/error levels that make sense for your use case
4. **Version control templates**: store templates in your repository alongside your data processing
code
5. **Test with sample data**: validate your templates work with representative datasets

### Common Template Patterns

For API response validation, you can ensure that responses have the expected structure and valid
status codes:

```yaml
tbl: null
tbl_name: "API Response Validation"
brief: "Standard checks for API response data"
steps:
  - col_exists:
      columns: [user_id, status, timestamp]
  - col_vals_in_set:
      columns: [status]
      set: [success, error, pending]
  - col_vals_not_null:
      columns: [user_id, timestamp]
```

For file upload validation, you can check file sizes and formats to ensure they meet your
requirements:

```yaml
tbl: null
tbl_name: "File Upload Validation"
steps:
  - col_vals_gt:
      columns: [file_size]
      value: 0
  - col_vals_lt:
      columns: [file_size]
      value: 10485760  # 10MB limit
  - col_vals_in_set:
      columns: [file_type]
      set: [csv, json, xlsx, parquet]
```

This template approach is particularly valuable in data pipelines, ETL processes, and automated
testing scenarios where you need to apply consistent validation logic across multiple similar
datasets.

## Validation Steps

YAML supports all of Pointblank's validation methods. Here are some common patterns:

### Column-based Validations

```yaml
tbl: worldcities.csv
steps:
  # Check for missing values
  - col_vals_not_null:
      columns: [city_name, country]

  # Validate value ranges
  - col_vals_between:
      columns: latitude
      left: -90
      right: 90

  # Check set membership
  - col_vals_in_set:
      columns: country_code
      set: [US, CA, MX, UK, DE, FR]

  # Regular expression validation
  - col_vals_regex:
      columns: postal_code
      pattern: "^[0-9]{5}(-[0-9]{4})?$"
```

### Row-based Validations

```yaml
tbl: sales_data.csv
steps:
  # Check for duplicate rows
  - rows_distinct

  # Ensure complete rows (no missing values)
  - rows_complete

  # Check row count
  - row_count_match:
      count: 1000
```

### Schema Validations

Schema validation ensures your data has the expected structure and column types. The
`col_schema_match` validation method uses a `schema` key that contains a `columns` list, where each
item in the list can specify a column name alone or a column name with its expected data type.

Each `column` entry can be specified as:

- `column_name`: column name as a scalar string (structure validation, no type checking)
- `[column_name, "data_type"]`: column name with type validation (as a list with two elements)
- `[column_name]`: column name in a single-item list (equivalent to scalar, for consistency)

```yaml
tbl: customer_data.csv
steps:
  # Complete schema validation (structure and types)
  - col_schema_match:
      schema:
        columns:
          - [customer_id, "int64"]
          - [name, "object"]
          - [email, "object"]
          - [signup_date, "datetime64[ns]"]

  # Structure-only validation (column names without types)
  - col_schema_match:
      schema:
        columns:
          - customer_id
          - name
          - email
      complete: false
      brief: "Check that core columns exist"
```

#### Schema Validation Options

Schema validations support the full range of validation options:

```yaml
tbl: data_file.csv
steps:
  - col_schema_match:
      schema:
        columns:
          - [id, "int64"]
          - name
      complete: false                  # Allow extra columns
      in_order: false                  # Column order doesn't matter
      case_sensitive_colnames: false   # Case-insensitive column names
      case_sensitive_dtypes: false     # Case-insensitive type names
      full_match_dtypes: false         # Allow partial type matching
      brief: "Flexible schema validation"
```

#### Other Structure Validations

```yaml
tbl: customer_data.csv
steps:
  # Check column count
  - col_count_match:
      count: 4
```

### Trend Validations

Validate that values follow increasing or decreasing patterns across rows:

```yaml
tbl: time_series_data.csv
steps:
  # Ensure timestamp values increase
  - col_vals_increasing:
      columns: timestamp
      brief: "Timestamps must be in chronological order"

  # Validate countdown timer decreases
  - col_vals_decreasing:
      columns: countdown
      allow_stationary: true
      brief: "Countdown values should decrease (ties allowed)"

  # Check trend with tolerance
  - col_vals_increasing:
      columns: temperature
      decreasing_tol: 0.5
      brief: "Temperature trends upward (small drops < 0.5°C allowed)"
```

### Specification-based Validations

Validate values against common data specifications like email addresses, URLs, postal codes, and
more:

```yaml
tbl: user_contact_info.csv
steps:
  # Validate email addresses
  - col_vals_within_spec:
      columns: email
      spec: "email"

  # Validate US ZIP codes
  - col_vals_within_spec:
      columns: zip_code
      spec: "postal_code[US]"

  # Validate URLs
  - col_vals_within_spec:
      columns: website
      spec: "url"
      na_pass: true
```

Available specifications include: `"email"`, `"url"`, `"phone"`, `"ipv4"`, `"ipv6"`, `"mac"`,
`"isbn"`, `"vin"`, `"credit_card"`, `"swift"`, `"postal_code[<country>]"`, `"iban[<country>]"`.

### Table Comparison

Validate that an entire table matches a reference table:

```yaml
tbl: processed_output.csv
steps:
  # Compare against expected output
  - tbl_match:
      tbl_compare:
        python: |
          pb.load_dataset("expected_output", tbl_type="polars")
      brief: "Output matches expected results"
```

The `tbl_match()` validation performs comprehensive comparison including column count, row count,
schema, and data values. It supports cross-backend validation (e.g., comparing Polars vs. Pandas
DataFrames).

### AI-Powered Validation

Use Large Language Models to validate data based on natural language criteria:

```yaml
tbl: customer_feedback.csv
steps:
  # Validate sentiment
  - prompt:
      prompt: "Customer feedback should express positive sentiment"
      model: "anthropic:claude-sonnet-5-5"
      columns_subset: [feedback_text, rating]
      batch_size: 500
      thresholds:
        warning: 0.1

  # Validate semantic correctness
  - prompt:
      prompt: "Product descriptions should mention the product category and at least one benefit"
      model: "openai:gpt-5.6-sol"
      columns_subset: [product_name, description, category]
```

**Note**: AI validations require API keys to be set as environment variables (e.g.,
`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`) or in a `.env` file. These validations are best suited for
semantic, context-dependent, or subjective quality checks rather than simple numeric comparisons.

## Thresholds and Severity Levels

Thresholds determine when validation failures trigger different severity levels. You can set global
thresholds for the entire workflow:

```yaml
tbl: sales_data.csv
tbl_name: "Sales Data Quality Check"
thresholds:
  warning: 0.05    # 5% failure rate triggers warning
  error: 0.10      # 10% failure rate triggers error
  critical: 0.15   # 15% failure rate triggers critical
steps:
  - col_vals_not_null:
      columns: [customer_id, amount]
  - col_vals_gt:
      columns: amount
      value: 0
```

You can also set thresholds for individual validation steps:

```yaml
tbl: user_data.csv
steps:
  - col_vals_not_null:
      columns: email
      thresholds:
        warning: 1      # Any missing email is a warning
        error: 0.01     # 1% missing emails is an error

  - col_vals_regex:
      columns: email
      pattern: "^[\\w\\.-]+@[\\w\\.-]+\\.[a-zA-Z]{2,}$"
      thresholds:
        error: 1        # Any invalid email format is an error
```

## Actions: Responding to Validation Failures

Actions define what happens when validation thresholds are exceeded. You can use string templates
with placeholder variables or callable functions.

### String Template Actions

```yaml
tbl: orders.csv
thresholds:
  warning: 0.02
  error: 0.05
actions:
  warning: "Warning: Step {step} found {n_failed} failures in {col} column"
  error: "Error in {TYPE} validation: {n_failed}/{n} rows failed (Step {step})"
  critical: "Critical failure detected at {time}"
steps:
  - col_vals_not_null:
      columns: [order_id, customer_id]
```

Available template variables include:

- `{step}`: validation step number
- `{col}`: column name being validated
- `{val}`: specific failing value (when applicable)
- `{n_failed}`: number of failing rows
- `{n}`: total number of rows checked
- `{TYPE}`: validation method name (e.g., "COL_VALS_NOT_NULL")
- `{LEVEL}`: severity level ("WARNING", "ERROR", "CRITICAL")
- `{time}`: timestamp of the validation

### Callable Actions

For more complex responses, use Python callable functions:

```yaml
tbl: critical_data.csv
thresholds:
  error: 1
actions:
  error:
    python: |
      lambda: print("ALERT: Critical data validation failed!")
  critical:
    python: |
      lambda: print("CRITICAL: Validation failure - manual intervention required!")
steps:
  - col_vals_not_null:
      columns: [transaction_id, amount]
```

Callable actions are Python code, so they only run with `allow_python=True` (see [Allowing Python
Code to Run](#allowing-python-code-to-run)). The code isn't sandboxed and has the same access as any
Python script, so only allow it for configurations you trust. String template actions are plain
text and always work.

### Step-level Actions

You can also define actions for individual validation steps:

```yaml
tbl: financial_data.csv
steps:
  - col_vals_not_null:
      columns: account_balance
      thresholds:
        error: 1
      actions:
        error: "Missing account balance detected in step {step}."

  - col_vals_gt:
      columns: account_balance
      value: 0
      actions:
        warning:
          python: |
            lambda: print("Negative balance warning triggered.")
```

## Advanced Features

### Pre-processing with the `pre` Parameter

You can apply data transformations before validation using the `pre` parameter:

```yaml
tbl: transactions.csv
steps:
  # Validate only recent transactions
  - col_vals_gt:
      columns: amount
      value: 0
      pre:
        python: |
          lambda df: df.filter(
              pl.col("transaction_date") >= "2024-01-01"
          )

  # Check completeness for active customers only
  - col_vals_not_null:
      columns: [email, phone]
      pre: |
        lambda df: df.filter(pl.col("status") == "active")
```

Note that you can use either the explicit `python:` block syntax or the shortcut syntax (just
`pre: |`) for the lambda expressions. Either way, `pre` functions are Python code and need
`allow_python=True`, as do the custom expressions in the next section.

### Complex Expressions

For advanced validation logic, use a `col_vals_expr` step with custom expressions:

```yaml
tbl: sales_data.csv
steps:
  # Custom business logic validation
  - col_vals_expr:
      expr:
        python: |
          (
            pl.when(pl.col("product_type") == "premium")
            .then(pl.col("price") >= 100)
            .when(pl.col("product_type") == "standard")
            .then(pl.col("price").is_between(20, 99))
            .otherwise(pl.col("price") <= 19)
          )
```

### Brief Descriptions

Add human-readable descriptions to validation steps. The `brief` parameter supports string
templating and automatic generation:

```yaml
tbl: customer_data.csv
brief: "Customer data quality validation for {auto}"
steps:
  - col_vals_not_null:
      columns: customer_id
      brief: "Ensure all customers have valid IDs"

  - col_vals_regex:
      columns: email
      pattern: "^[\\w\\.-]+@[\\w\\.-]+\\.[a-zA-Z]{2,}$"
      brief: "Validate email format compliance"

  - col_vals_between:
      columns: age
      left: 13
      right: 120
      brief: "Check reasonable age ranges"

  # Use automatic brief generation
  - col_vals_not_null:
      columns: phone_number
      brief: true

  # Template variables in briefs
  - col_vals_in_set:
      columns: status
      set: [active, inactive, pending]
      brief: "Column '{col}' must be one of: {set}"
```

Brief Templating Options:

- custom strings: Write your own descriptive text
- `true`: Automatically generates a brief based on the validation method and parameters
- `{auto}`: Placeholder for auto-generated text within custom strings
- template variables: Use the same variables available in actions:
  - `{col}`: column name(s) being validated
  - `{step}`: the step number in the validation plan
  - `{value}`: the comparison value used in the validation (for single-value comparisons)
  - `{pattern}`: for regex validations, the pattern being matched

### Governance Metadata

YAML workflows support governance metadata that identifies ownership and usage of validation
workflows. These fields are embedded in the validation report:

```yaml
tbl: sales_data.csv
tbl_name: "Sales Pipeline"
owner: "Data Engineering"
consumers: [Analytics Team, Finance, Compliance]
version: "2.1.0"
steps:
  - col_vals_not_null:
      columns: [customer_id, revenue]
  - col_vals_gt:
      columns: [revenue]
      value: 0
```

The `owner`, `consumers`, and `version` fields are forwarded to the `Validate` constructor and
appear in the validation report header. These fields are optional and do not affect validation
behavior.

### Data Freshness and Null Percentage

Two additional validation methods support common data quality checks:

**`data_freshness`**: Validate that a date/datetime column has recent data:

```yaml
steps:
  - data_freshness:
      column: event_date
      max_age: "24h"
```

**`col_pct_null`**: Validate that the percentage of null values is within bounds:

```yaml
steps:
  - col_pct_null:
      columns: [email, phone]
      value: 0.05
```

### Aggregate Validations

Aggregate methods validate column-level statistics like sum, average, and standard deviation:

```yaml
steps:
  # Check that total revenue is positive
  - col_sum_gt:
      columns: [revenue]
      value: 0

  # Validate average rating is at most 5
  - col_avg_le:
      columns: [rating]
      value: 5

  # Ensure temperature variation is bounded
  - col_sd_lt:
      columns: [temperature]
      value: 10
```

Available methods follow the `col_{stat}_{comparator}` pattern where `{stat}` is `sum`, `avg`, or
`sd`, and `{comparator}` is `gt`, `lt`, `ge`, `le`, `eq`, `between`, or `outside`.

### Step Activation Control

The `active` parameter allows you to temporarily disable validation steps without removing them
from the configuration:

```yaml
steps:
  # This step is disabled
  - col_vals_gt:
      columns: [amount]
      value: 0
      active: false

  # This step runs normally (active: true is the default)
  - col_vals_not_null:
      columns: [customer_id]
```

This is useful for debugging, phased rollouts, or temporarily skipping steps that are known to fail.

### Reference Tables

The `reference` top-level key specifies a reference table for comparison-based validations:

```yaml
tbl: current_data.csv
reference:
  python: |
    pb.load_dataset("baseline_data", tbl_type="polars")
steps:
  - tbl_match:
      tbl_compare:
        python: |
          pb.load_dataset("baseline_data", tbl_type="polars")
```

## Working with YAML Files

### Loading from Files

You can save your YAML configuration to files and load them:

```{python}
# Create a YAML file
yaml_content = """
tbl: small_table
tbl_name: "File-based Validation"
steps:
  - col_vals_between:
      columns: c
      left: 1
      right: 10
  - col_vals_in_set:
      columns: f
      set: [low, mid, high]
"""

# Save to file
from pathlib import Path
yaml_file = Path("validation_config.yaml")
yaml_file.write_text(yaml_content)

# Load and execute
result = pb.yaml_interrogate(yaml_file)
result
```

### Converting YAML to Python

Use `yaml_to_python()` to generate equivalent Python code from your YAML configuration:

```{python}
yaml_config = """
tbl: small_table
tbl_name: "Example Validation"
thresholds:
  warning: 0.1
  error: 0.2
actions:
  warning: "Warning: {TYPE} validation failed"
steps:
  - col_vals_gt:
      columns: a
      value: 0
  - col_vals_in_set:
      columns: f
      set: [low, mid, high]
"""

# Generate Python code
python_code = pb.yaml_to_python(yaml_config)
print(python_code)
```

`yaml_to_python()` never runs Python code from the configuration: it copies the code into the
generated Python as written, so it doesn't need `allow_python=True`. Read the generated code before
you run it.

This is useful for:

- learning how YAML maps to Python API calls
- transitioning from YAML to code-based workflows
- generating documentation that shows both approaches
- debugging YAML configurations

## Practical Examples

### Data Pipeline Validation

Here's a comprehensive example for validating data in a processing pipeline:

```yaml
tbl:
  python: |
    (
      pl.scan_csv("raw_data/customer_events.csv")
      .filter(pl.col("event_date") >= "2024-01-01")
    )

tbl_name: "Customer Events Pipeline Validation"
label: "Daily data quality check for customer events"

thresholds:
  warning: 0.01   # 1% failure rate
  error: 0.05     # 5% failure rate

actions:
  warning: "Pipeline warning: {TYPE} validation found {n_failed} issues"
  error:
    python: |
      lambda: print("ERROR: Pipeline validation failed - manual review required")

steps:
  # Schema validation
  - col_schema_match:
      schema:
        columns:
          - [customer_id, "int64"]
          - [event_type, "object"]
          - [event_date, "object"]
          - [revenue, "float64"]
      brief: "Validate table structure matches expected schema"

  # Data completeness
  - col_vals_not_null:
      columns: [customer_id, event_type, event_date]
      brief: "Critical fields must be complete"

  # Business logic validation
  - col_vals_in_set:
      columns: event_type
      set: [signup, purchase, cancellation, upgrade]
      brief: "Event types must be from approved list"

  # Data quality checks
  - col_vals_gt:
      columns: revenue
      value: 0
      na_pass: true
      brief: "Revenue values must be positive when present"

  # Temporal validation
  - col_vals_expr:
      expr:
        python: |
          pl.col("event_date").str.strptime(pl.Date, "%Y-%m-%d").is_not_null()
      brief: "Event dates must be valid YYYY-MM-DD format"
```

This configuration contains Python code (the `tbl` expression, the `error` action, and the custom
expression), so it needs `allow_python=True` in `yaml_interrogate()` or `--allow-python` with
`pb run`.

### Quality Monitoring Dashboard

For ongoing data quality monitoring:

```yaml
tbl: warehouse/daily_metrics.parquet
tbl_name: "Daily Metrics Quality Check"

thresholds:
  warning: 5      # 5 failing rows
  error: 50       # 50 failing rows
  critical: 100   # 100 failing rows

actions:
  warning: "Quality check warning: {n_failed} rows failed {TYPE} validation"
  error: "Quality degradation detected: Step {step} failed for {n_failed}/{n} rows"
  critical:
    python: |
      lambda: print("CRITICAL: Data quality failure detected - immediate attention required")
  highest_only: false

steps:
  - row_count_match:
      count: 10000
      brief: "Verify expected daily record count"

  - col_vals_not_null:
      columns: [date, metric_value, source_system]
      brief: "Core fields must be complete"

  - col_vals_between:
      columns: metric_value
      left: 0
      right: 1000000
      brief: "Metric values within reasonable range"

  - rows_distinct:
      columns_subset: [date, metric_name, source_system]
      brief: "No duplicate metric records per day"
```

## Best Practices

### Organization and Structure

1. use descriptive names: give your validations clear `tbl_name` and `label` values
2. add brief descriptions: document what each validation step checks
3. group related validations: organize steps logically (schema, completeness, business rules)
4. version control: store YAML files in git alongside your data processing code

### Error Handling and Monitoring

1. set appropriate thresholds: start conservative and adjust based on your data patterns
2. use actions for alerting: set up notifications for critical failures
3. document expected failures: some data quality issues might be acceptable
4. monitor validation results: track validation performance over time

### Performance Considerations

1. use the `pre` parameter efficiently: apply filters early to reduce data volume
2. order validations strategically: put fast, likely-to-fail checks first
3. consider data source location: local files are faster than remote sources
4. use appropriate column selections: only validate the columns you need

## Wrapping Up

YAML validation workflows provide a powerful, declarative approach to data validation in Pointblank.
Such workflows are great at expressing common validation patterns in a readable format that can be
easily shared, version controlled, and maintained by teams.

Key advantages of YAML workflows:

- readable: non-programmers can understand and contribute to validation logic
- maintainable: easy to modify validation rules without changing application code
- portable: YAML files can be shared between projects and teams
- version controlled: track changes to validation logic over time
- flexible: support for simple checks and complex custom logic

Use YAML workflows when you want declarative, maintainable validation definitions, and fall back to
the Python API when you need complex programmatic logic or tight integration with application code.
The two approaches complement each other well and can be used together as your validation needs
evolve.


### YAML Reference

This reference provides a comprehensive guide to all YAML keys and parameters supported by
Pointblank's YAML validation workflows. Use this document as a quick lookup when building validation
configurations.

## Global Configuration Keys

### Top-level Structure

```yaml
tbl: data_source                       # REQUIRED: Data source specification
df_library: "polars"                   # OPTIONAL: DataFrame library ("polars", "pandas", "duckdb")
tbl_name: "Custom Table Name"          # OPTIONAL: Human-readable table name
label: "Validation Description"        # OPTIONAL: Description for the validation workflow
lang: "en"                             # OPTIONAL: Language code (default: "en")
locale: "en"                           # OPTIONAL: Locale setting (default: "en")
brief: "Global brief: {auto}"          # OPTIONAL: Global brief template
owner: "Data Engineering"              # OPTIONAL: Data owner (governance metadata)
consumers: [Analytics, Finance]        # OPTIONAL: Data consumers (governance metadata)
version: "1.0.0"                       # OPTIONAL: Validation version (governance metadata)
reference:                             # OPTIONAL: Reference table for comparison validations
  python: |
    pb.load_dataset("ref_table")
thresholds:                            # OPTIONAL: Global failure thresholds
  warning: 0.1
  error: 0.2
  critical: 0.3
actions:                               # OPTIONAL: Global failure actions
  warning: "Warning message template"
  error: "Error message template"
  critical: "Critical message template"
  highest_only: false
final_actions:                         # OPTIONAL: Actions triggered after all steps complete
  warning: "Post-validation warning"
  error: "Post-validation error"
missing_specs:                         # OPTIONAL: Named structured-missingness specs
  standard_survey:
    reasons:
      -99: not_asked
      -98: refused
steps:                                 # REQUIRED: List of validation steps
  - validation_method_name
  - validation_method_name:
      parameter: value
```

### Data Source (`tbl`)

The `tbl` key specifies the data source and supports multiple formats:

```yaml
# File paths
tbl: "data/file.csv"
tbl: "data/file.parquet"

# Built-in datasets
tbl: small_table
tbl: game_revenue
tbl: nycflights

# Python expressions for complex data loading
tbl:
  python: |
    pl.scan_csv("data.csv").filter(pl.col("date") >= "2024-01-01")
```

#### Using Templates with `set_tbl=`

For reusable validation templates that will always use a custom data source via the `set_tbl=`
parameter in `yaml_interrogate()`, the `tbl` field is still required but its value doesn't matter
since it will be overridden. Recommended approaches:

```yaml
# Option 1: Use a valid dataset name (gets overridden anyway)
tbl: small_table  # Will be ignored when `set_tbl=` is used

# Option 2: Use YAML null (clearest semantic intent)
tbl: null  # Indicates table will be provided via `set_tbl=`
```

When using `yaml_interrogate()` with `set_tbl=`, the validation template becomes fully reusable:

```python
# Define reusable template
template = """
tbl: null  # Will be overridden
tbl_name: "Sales Validation"
steps:
  - col_exists:
      columns: [customer_id, revenue, region]
  - col_vals_gt:
      columns: [revenue]
      value: 0
"""

# Apply to different datasets
q1_result = pb.yaml_interrogate(template, set_tbl=q1_data)
q2_result = pb.yaml_interrogate(template, set_tbl=q2_data)
```

### DataFrame Library (`df_library`)

The `df_library` key controls which DataFrame library is used to load data sources. This parameter
affects both built-in datasets and file loading:

```yaml
# Use Polars DataFrames (default)
df_library: polars

# Use Pandas DataFrames
df_library: pandas

# Use DuckDB tables (via Ibis)
df_library: duckdb
```

Examples with different libraries:

```yaml
# Load built-in dataset as Pandas DataFrame
tbl: small_table
df_library: pandas
steps:
  - specially:
      expr: "lambda df: df.assign(validation_result=df['a'] > 0)"

# Load CSV file as Polars DataFrame
tbl: "data/sales.csv"
df_library: polars
steps:
  - col_vals_gt:
      columns: amount
      value: 0

# Load dataset as DuckDB table
tbl: nycflights
df_library: duckdb
steps:
  - row_count_match:
      count: 336776
```

The `df_library` parameter is particularly useful when:

- using validation expressions that require specific DataFrame APIs (e.g., Pandas `.assign()`,
Polars `.select()`)
- integrating with existing pipelines that use a specific DataFrame library
- optimizing performance for different data sizes and operations
- ensuring compatibility with downstream processing steps

### Global Thresholds

Thresholds define when validation failures trigger different severity levels:

```yaml
thresholds:
  warning: 0.05    # 5% failure rate triggers warning
  error: 0.10      # 10% failure rate triggers error
  critical: 0.15   # 15% failure rate triggers critical
```

- values: numbers between `0` and `1` (percentages) or integers (row counts)
- levels: `warning`, `error`, `critical`

### Global Actions

Actions define responses when thresholds are exceeded. When supplying a string to a severity level
('warning', 'error', 'critical'), you can use template variables that will be automatically
substituted with contextual information:

```yaml
actions:
  warning: "Warning: {n_failed} failures in step {step}"
  error:
    python: |
      lambda: print("Error detected!")
  critical: "Critical failure at {time}"
  highest_only: false        # Execute all applicable actions vs. only highest severity
```

Template variables available for action strings:

- `{step}`: current validation step number
- `{col}`: column name(s) being validated
- `{val}`: validation value or threshold
- `{n_failed}`: number of failing records
- `{n}`: total number of records
- `{type}`: validation method type
- `{level}`: severity level ('warning'/'error'/'critical')
- `{time}`: timestamp of validation

### Structured Missingness (`missing_specs`)

The optional top-level `missing_specs` key defines named [`MissingSpec`](`pointblank.MissingSpec`)
objects that steps can reference. Each named spec maps sentinel values to reason labels, and may
declare `categories`, `null_is_missing`, and `null_reason`:

```yaml
missing_specs:
  standard_survey:
    reasons:
      -99: not_asked
      -98: refused
      -97: dont_know
    categories:
      nonresponse: [refused, dont_know]
    null_is_missing: true        # OPTIONAL (default true)
    null_reason: unknown         # OPTIONAL (default "unknown")
```

Steps reference a named spec by name through the `missing:` parameter. This works both on the
`col_vals_*` methods (to exclude sentinel values from a check) and on the dedicated missingness
methods (`col_pct_missing`, `col_missing_coded`, `col_missing_only_coded`, `col_missing_consistent`):

```yaml
missing_specs:
  standard_survey:
    reasons:
      -99: not_asked
      -98: refused

steps:
  - col_vals_between:
      columns: age
      left: 0
      right: 120
      missing: standard_survey       # excludes -99/-98 (and nulls) from the range check
  - col_pct_missing:
      columns: age
      missing: standard_survey
      reason: refused
      max_pct: 0.30
```

A step can also define a spec inline (an anonymous mapping) instead of referencing a named one:

```yaml
steps:
  - col_pct_missing:
      columns: age
      max_pct: 0.5
      missing:
        reasons:
          -99: not_asked
          -98: refused
```

## Validation Methods Reference

### Column Value Validations

#### Comparison Methods

`col_vals_gt`: are column data greater than a fixed value or data in another column?

```yaml
- col_vals_gt:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    value: 100                         # REQUIRED: Comparison value
    na_pass: true                      # OPTIONAL: Pass NULL values (default: false)
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Values must be > 100"      # OPTIONAL: Step description
```

To compare against the data in another column, give the column as the value. It can be written
directly or in a `python:` block:

```yaml
- col_vals_gt:
    columns: [column_name]
    value: pb.col("other_column")      # Same as `value: {python: pb.col("other_column")}`
```

`col_vals_lt`: are column data less than a fixed value or data in another column?

```yaml
- col_vals_lt:
    columns: [column_name]
    value: 100
    na_pass: true
    # ... (same parameters as col_vals_gt)
```

`col_vals_ge`: are column data greater than or equal to a fixed value or data in another column?

```yaml
- col_vals_ge:
    columns: [column_name]
    value: 100
    na_pass: true
    # ... (same parameters as col_vals_gt)
```

`col_vals_le`: are column data less than or equal to a fixed value or data in another column?

```yaml
- col_vals_le:
    columns: [column_name]
    value: 100
    na_pass: true
    # ... (same parameters as col_vals_gt)
```

`col_vals_eq`: are column data equal to a fixed value or data in another column?

```yaml
- col_vals_eq:
    columns: [column_name]
    value: "expected_value"
    na_pass: true
    # ... (same parameters as col_vals_gt)
```

`col_vals_ne`: are column data not equal to a fixed value or data in another column?

```yaml
- col_vals_ne:
    columns: [column_name]
    value: "forbidden_value"
    na_pass: true
    # ... (same parameters as col_vals_gt)
```

#### Range Methods

`col_vals_between`: are column data between two specified values (inclusive)?

```yaml
- col_vals_between:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    left: 0                            # REQUIRED: Lower bound
    right: 100                         # REQUIRED: Upper bound
    inclusive: [true, true]            # OPTIONAL: Include bounds [left, right]
    na_pass: false                     # OPTIONAL: Pass NULL values
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Values between 0 and 100"  # OPTIONAL: Step description
```

`col_vals_outside`: are column data outside of two specified values?

```yaml
- col_vals_outside:
    columns: [column_name]
    left: 0
    right: 100
    inclusive: [false, false]          # OPTIONAL: Exclude bounds [left, right]
    na_pass: false
    # ... (same parameters as col_vals_between)
```

#### Set Membership Methods

`col_vals_in_set`: are column data part of a specified set of values?

```yaml
- col_vals_in_set:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    set: [value1, value2, value3]      # REQUIRED: Allowed values
    na_pass: false                     # OPTIONAL: Pass NULL values
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Values in allowed set"     # OPTIONAL: Step description
```

`col_vals_not_in_set`: are column data not part of a specified set of values?

```yaml
- col_vals_not_in_set:
    columns: [column_name]
    set: [forbidden1, forbidden2]      # REQUIRED: Forbidden values
    na_pass: false
    # ... (same parameters as col_vals_in_set)
```

#### NULL Value Methods

`col_vals_null`: are column data null (missing)?

```yaml
- col_vals_null:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Values must be NULL"       # OPTIONAL: Step description
```

`col_vals_not_null`: are column data not null (not missing)?

```yaml
- col_vals_not_null:
    columns: [column_name]
    # ... (same parameters as col_vals_null)
```

#### Pattern Matching Methods

`col_vals_regex`: do string-based column data match a regular expression?
```yaml
- col_vals_regex:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    pattern: "^[A-Z]{2,3}$"            # REQUIRED: Regular expression pattern
    na_pass: false                     # OPTIONAL: Pass NULL values
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Values match pattern"      # OPTIONAL: Step description
```

`col_vals_within_spec`: do column data conform to a specification (email, URL, postal codes, etc.)?

```yaml
- col_vals_within_spec:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    spec: "email"                      # REQUIRED: Specification type
    na_pass: false                     # OPTIONAL: Pass NULL values
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Values match spec"         # OPTIONAL: Step description
```

Available specification types:

- `"email"` - Email addresses
- `"url"` - Internet URLs
- `"phone"` - Phone numbers
- `"ipv4"` - IPv4 addresses
- `"ipv6"` - IPv6 addresses
- `"mac"` - MAC addresses
- `"isbn"` - International Standard Book Numbers (10 or 13 digit)
- `"vin"` - Vehicle Identification Numbers
- `"credit_card"` - Credit card numbers (uses Luhn algorithm)
- `"swift"` - Business Identifier Codes (SWIFT-BIC)
- `"postal_code[<country_code>]"` - Postal codes for specific countries (e.g., `"postal_code[US]"`, `"postal_code[CA]"`)
- `"zip"` - Alias for US ZIP codes (`"postal_code[US]"`)
- `"iban[<country_code>]"` - International Bank Account Numbers (e.g., `"iban[DE]"`, `"iban[FR]"`)

Examples:

```yaml
# Email validation
- col_vals_within_spec:
    columns: user_email
    spec: "email"

# US postal codes
- col_vals_within_spec:
    columns: zip_code
    spec: "postal_code[US]"

# German IBAN
- col_vals_within_spec:
    columns: account_number
    spec: "iban[DE]"
```

`col_vals_str_len`: are string lengths within a range?

```yaml
- col_vals_str_len:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    min_val: 2                         # OPTIONAL: Minimum length (at least one bound needed)
    max_val: 10                        # OPTIONAL: Maximum length
    na_pass: false                     # OPTIONAL: Pass NULL values
```

#### Custom Expression Methods

`col_vals_expr`: do column data agree with a predicate expression?

```yaml
- col_vals_expr:
    expr:                              # REQUIRED: Custom validation expression
      python: |
        pl.when(pl.col("status") == "active")
        .then(pl.col("value") > 0)
        .otherwise(pl.lit(True))
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Custom validation rule"    # OPTIONAL: Step description
```

#### Trend Validation Methods

`col_vals_increasing`: are column data increasing row-by-row?

```yaml
- col_vals_increasing:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    allow_stationary: false            # OPTIONAL: Allow consecutive equal values (default: false)
    decreasing_tol: 0.5                # OPTIONAL: Tolerance for negative movement (default: null)
    na_pass: false                     # OPTIONAL: Pass NULL values
    order_by: date                     # OPTIONAL: Column(s) that put the rows in order
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Values must increase"      # OPTIONAL: Step description
```

This validation checks whether values in a column increase as you move down the rows. Useful for
validating time-series data, sequence numbers, or any monotonically increasing values.

Parameters:

- `allow_stationary`: If `true`, allows consecutive values to be equal (stationary phases). For
example, `[1, 2, 2, 3]` would pass when `true` but fail at the third value when `false`.
- `decreasing_tol`: Absolute tolerance for negative movement. Setting this to `0.5` means values can
decrease by up to 0.5 units and still pass. Setting any value also sets `allow_stationary` to `true`.
- `order_by`: Column (or list of columns) that puts the rows in order before comparing values.
Required for database tables, which have no inherent row order.

Examples:

```yaml
# Strict increasing validation
- col_vals_increasing:
    columns: timestamp_seconds
    brief: "Timestamps must strictly increase"

# Allow stationary values
- col_vals_increasing:
    columns: version_number
    allow_stationary: true
    brief: "Version numbers should increase (ties allowed)"

# With tolerance for small decreases
- col_vals_increasing:
    columns: temperature
    decreasing_tol: 0.1
    brief: "Temperature trend (small drops allowed)"
```

`col_vals_decreasing`: are column data decreasing row-by-row?

```yaml
- col_vals_decreasing:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    allow_stationary: false            # OPTIONAL: Allow consecutive equal values (default: false)
    increasing_tol: 0.5                # OPTIONAL: Tolerance for positive movement (default: null)
    na_pass: false                     # OPTIONAL: Pass NULL values
    order_by: date                     # OPTIONAL: Column(s) that put the rows in order
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Values must decrease"      # OPTIONAL: Step description
```

This validation checks whether values in a column decrease as you move down the rows. Useful for
countdown timers, inventory depletion, or any monotonically decreasing values.

Parameters:

- `allow_stationary`: If `true`, allows consecutive values to be equal (stationary phases). For
example, `[10, 8, 8, 5]` would pass when `true` but fail at the third value when `false`.
- `increasing_tol`: Absolute tolerance for positive movement. Setting this to `0.5` means values can
increase by up to 0.5 units and still pass. Setting any value also sets `allow_stationary` to `true`.

Examples:

```yaml
# Strict decreasing validation
- col_vals_decreasing:
    columns: countdown_timer
    brief: "Timer must strictly decrease"

# Allow stationary values
- col_vals_decreasing:
    columns: priority_score
    allow_stationary: true
    brief: "Priority scores should decrease (ties allowed)"

# With tolerance for small increases
- col_vals_decreasing:
    columns: stock_level
    increasing_tol: 5
    brief: "Stock levels decrease (small restocks allowed)"
```

### Row-based Validations

`rows_distinct`: are row data distinct?

```yaml
- rows_distinct                        # Simple form

- rows_distinct:                       # With parameters
    columns_subset: [col1, col2]       # OPTIONAL: Check subset of columns
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "No duplicate rows"         # OPTIONAL: Step description
```

`rows_complete`: are row data complete?

```yaml
- rows_complete                        # Simple form

- rows_complete:                       # With parameters
    columns_subset: [col1, col2]       # OPTIONAL: Check subset of columns
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Complete rows only"        # OPTIONAL: Step description
```

### Structure Validations

`col_exists`: does column exist in the table?

```yaml
- col_exists:
    columns: [col1, col2, col3]        # REQUIRED: Column(s) that must exist
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Required columns exist"    # OPTIONAL: Step description
```

`col_schema_match`: does the table have expected column names and data types?

```yaml
- col_schema_match:
    schema:                            # REQUIRED: Expected schema
      columns:
        - [column_name, "data_type"]   # Column with type validation
        - column_name                  # Column name only (no type check)
        - [column_name]                # Alternative syntax
    complete: true                     # OPTIONAL: Require exact column set
    in_order: true                     # OPTIONAL: Require exact column order
    case_sensitive_colnames: true      # OPTIONAL: Case-sensitive column names
    case_sensitive_dtypes: true        # OPTIONAL: Case-sensitive data types
    full_match_dtypes: true            # OPTIONAL: Exact type matching
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Schema validation"         # OPTIONAL: Step description
```

`row_count_match`: does the table have n rows?

```yaml
- row_count_match:
    count: 1000                        # REQUIRED: Expected row count
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Expected row count"        # OPTIONAL: Step description
```

`col_count_match`: does the table have n columns?

```yaml
- col_count_match:
    count: 10                          # REQUIRED: Expected column count
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Expected column count"     # OPTIONAL: Step description
```

`tbl_match`: does the table match a comparison table?

```yaml
- tbl_match:
    tbl_compare:                       # REQUIRED: Comparison table
      python: |
        pb.load_dataset("reference_table", tbl_type="polars")
    pre: |                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.0
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Table structure matches"   # OPTIONAL: Step description
```

This validation performs a comprehensive comparison between the target table and a comparison table,
using progressively stricter checks:

1. **Column count match**: both tables have the same number of columns
2. **Row count match**: both tables have the same number of rows
3. **Schema match (loose)**: column names and dtypes match (case-insensitive, any order)
4. **Schema match (order)**: columns in correct order (case-insensitive names)
5. **Schema match (exact)**: column names match exactly (case-sensitive, correct order)
6. **Data match**: values in corresponding cells are identical

The validation fails at the first check that doesn't pass, making it easy to diagnose mismatches.
This operates over a single test unit (pass/fail for complete table match).

**Cross-backend validation**: `tbl_match()` supports automatic backend coercion when comparing tables
from different backends (e.g., Polars vs. Pandas, DuckDB vs. SQLite). The comparison table is
automatically converted to match the target table's backend.

Examples:

```yaml
# Compare against reference dataset
- tbl_match:
    tbl_compare:
      python: |
        pb.load_dataset("expected_output", tbl_type="polars")
    brief: "Output matches expected results"

# Compare against CSV file
- tbl_match:
    tbl_compare:
      python: |
        pl.read_csv("reference_data.csv")
    brief: "Matches reference CSV"

# Compare with preprocessing on target table only
- tbl_match:
    tbl_compare:
      python: |
        pb.load_dataset("reference_table", tbl_type="polars")
    pre: |
      lambda df: df.select(["id", "name", "value"])
    brief: "Selected columns match reference"
```

### Special Validation Methods

`conjointly`: are multiple validations having a joint dependency?

```yaml
- conjointly:
    expressions:                       # REQUIRED: List of lambda expressions
      - "lambda df: df['d'] > df['a']"
      - "lambda df: df['a'] > 0"
      - "lambda df: df['a'] + df['d'] < 12000"
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "All conditions must pass"  # OPTIONAL: Step description
```

`specially`: do table data pass a custom validation function?

```yaml
- specially:
    expr:                              # REQUIRED: Custom validation function
      "lambda df: df.select(pl.col('a') + pl.col('d') > 0)"
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Custom validation"         # OPTIONAL: Step description
```

Alternative syntax with Python expressions:

```yaml
- specially:
    expr:
      python: |
        lambda df: df.select(pl.col('amount') > 0)
```

For Pandas DataFrames (when using `df_library: pandas`):

```yaml
- specially:
    expr: "lambda df: df.assign(is_valid=df['a'] + df['d'] > 0)"
```

### AI-Powered Validation

`prompt`: validate rows using AI/LLM-powered analysis

```yaml
- prompt:
    prompt: "Values should be positive and realistic"  # REQUIRED: Natural language criteria
    model: "anthropic:claude-sonnet-5-5"               # REQUIRED: Model identifier
    columns_subset: [column1, column2]                 # OPTIONAL: Columns to validate
    batch_size: 1000                                   # OPTIONAL: Rows per batch (default: 1000)
    max_concurrent: 3                                  # OPTIONAL: Concurrent API requests (default: 3)
    pre: |                                             # OPTIONAL: Data preprocessing
      lambda df: df.filter(condition)
    thresholds:                                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "AI validation"                             # OPTIONAL: Step description
```

This validation method uses Large Language Models (LLMs) to validate rows of data based on natural
language criteria. Each row becomes a test unit that either passes or fails the validation criteria,
producing binary True/False results that integrate with standard Pointblank reporting.

**Supported models:**

- **Anthropic**: `"anthropic:claude-sonnet-5-5"`, `"anthropic:claude-opus-5-5"`
- **OpenAI**: `"openai:gpt-5.6-sol"`, `"openai:gpt-5.6-terra"`
- **Ollama**: `"ollama:<model-name>"` (e.g., `"ollama:llama3"`)
- **Bedrock**: `"bedrock:<model-name>"`

**Authentication**: API keys are automatically loaded from environment variables or `.env` files:

- **OpenAI**: Set `OPENAI_API_KEY` environment variable or add to `.env` file
- **Anthropic**: Set `ANTHROPIC_API_KEY` environment variable or add to `.env` file
- **Ollama**: No API key required (runs locally)
- **Bedrock**: Configure AWS credentials through standard AWS methods

Example `.env` file:

```plaintext
ANTHROPIC_API_KEY="your_anthropic_api_key_here"
OPENAI_API_KEY="your_openai_api_key_here"
```

**Performance optimization**: The validation process uses row signature memoization to avoid
redundant LLM calls. When multiple rows have identical values in the selected columns, only one
representative row is validated, and the result is applied to all matching rows. This dramatically
reduces API costs and processing time for datasets with repetitive patterns.

Examples:

```yaml
# Basic AI validation
- prompt:
    prompt: "Email addresses should look realistic and professional"
    model: "anthropic:claude-sonnet-5-5"
    columns_subset: [email]

# Complex semantic validation
- prompt:
    prompt: "Product descriptions should mention the product category and include at least one benefit"
    model: "openai:gpt-5.6-sol"
    columns_subset: [product_name, description, category]
    batch_size: 500
    max_concurrent: 5

# Sentiment analysis
- prompt:
    prompt: "Customer feedback should express positive sentiment"
    model: "anthropic:claude-sonnet-5-5"
    columns_subset: [feedback_text, rating]

# Context-dependent validation
- prompt:
    prompt: "For high-value transactions (amount > 1000), a detailed justification should be provided"
    model: "openai:gpt-5.6-sol"
    columns_subset: [amount, justification, approver]
    thresholds:
      warning: 0.05
      error: 0.15

# Local model with Ollama
- prompt:
    prompt: "Transaction descriptions should be clear and professional"
    model: "ollama:llama3"
    columns_subset: [description]
```

**Best practices for AI validation:**

- Be specific and clear in your prompt criteria
- Include only necessary columns in `columns_subset` to reduce API costs
- Start with smaller `batch_size` for testing, increase for production
- Adjust `max_concurrent` based on API rate limits
- Use thresholds appropriate for probabilistic validation results
- Consider cost implications for large datasets
- Test prompts on sample data before full deployment

**When to use AI validation:**

- Semantic checks (e.g., "does the description match the category?")
- Context-dependent validation (e.g., "is the justification appropriate for the amount?")
- Subjective quality assessment (e.g., "is the text professional?")
- Pattern recognition that's hard to express programmatically
- Natural language understanding tasks

**When NOT to use AI validation:**

- Simple numeric comparisons (use `col_vals_gt`, `col_vals_lt`, etc.)
- Exact pattern matching (use `col_vals_regex`)
- Schema validation (use `col_schema_match`)
- Performance-critical validations with large datasets
- When deterministic results are required

### Data Quality Methods

`col_pct_null`: is the percentage of null values in a column within bounds?

```yaml
- col_pct_null:
    columns: [column_name]             # REQUIRED: Column(s) to validate
    value: 0.05                        # REQUIRED: Maximum allowed null fraction
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Null rate check"           # OPTIONAL: Step description
```

`data_freshness`: is the data in a date/datetime column recent?

```yaml
- data_freshness:
    column: date_column                # REQUIRED: Date/datetime column
    max_age: "24h"                     # REQUIRED: Maximum age of data
    timezone: "UTC"                    # OPTIONAL: Timezone for naive datetimes
    allow_future: "5m"                 # OPTIONAL: Allow data this far ahead (true for any amount)
    thresholds:                        # OPTIONAL: Step-level thresholds
      warning: 0.1
    actions:                           # OPTIONAL: Step-level actions
      warning: "Custom message"
    brief: "Data is recent"            # OPTIONAL: Step description
```

### Aggregate Validations

Aggregate methods validate column-level statistics (sum, average, standard deviation) against a
threshold. They follow the pattern `col_{stat}_{comparator}`:

```yaml
# Sum validations
- col_sum_gt:
    columns: [revenue]
    value: 0
    brief: "Total revenue is positive"

# Average validations
- col_avg_le:
    columns: [rating]
    value: 5
    brief: "Average rating at most 5"

# Standard deviation validations
- col_sd_lt:
    columns: [temperature]
    value: 10
    brief: "Temperature variation is bounded"
```

Available aggregate methods:

- **Sum**: `col_sum_gt`, `col_sum_lt`, `col_sum_ge`, `col_sum_le`, `col_sum_eq`
- **Average**: `col_avg_gt`, `col_avg_lt`, `col_avg_ge`, `col_avg_le`, `col_avg_eq`
- **Standard deviation**: `col_sd_gt`, `col_sd_lt`, `col_sd_ge`, `col_sd_le`, `col_sd_eq`

All aggregate methods accept these common parameters: `columns`, `value`, `thresholds`, `actions`,
`brief`, `active`, and `pre`.

## Column Selection Patterns

All validation methods that accept a `columns` parameter support these selection patterns:

```yaml
# Single column
columns: column_name

# Multiple columns as list
columns: [col1, col2, col3]

# Column selector functions (when used in Python expressions)
columns:
  python: |
    starts_with("prefix_")

# Examples of common patterns
columns: [customer_id, order_id]     # Specific columns
columns: user_email                  # Single column
```

## Parameter Details

### Common Parameters

These parameters are available for most validation methods:

- `columns`: column selection (string, list, or selector expression)
- `na_pass`: whether to pass NULL/missing values (boolean, default: false)
- `pre`: data preprocessing function (Python lambda expression)
- `segments`: split the step by segment; a column name (one step per distinct value), or a list
  of column names and two-item `[column, value]` pairs (e.g., `[region, [quarter, Q1]]`), where
  the value can also be a list of values
- `thresholds`: step-level failure thresholds (dict)
- `actions`: step-level failure actions (dict)
- `brief`: step description (string, boolean, or template)
- `active`: whether the step is active (boolean, default: true)

### Active Parameter

The `active` parameter controls whether a validation step runs. It defaults to `true`; set it to
`false` to skip a step without removing it from the configuration:

```yaml
steps:
  # This step will be skipped
  - col_vals_gt:
      columns: [amount]
      value: 0
      active: false

  # This step runs normally (default active: true)
  - col_vals_not_null:
      columns: [customer_id]
```

### Brief Parameter Options

The `brief` parameter supports several formats:

```yaml
brief: "Custom description"          # Custom text
brief: true                         # Auto-generated description
brief: false                        # No description
brief: "Step {step}: {auto}"        # Template with auto-generated text
brief: "Column '{col}' validation"  # Template with variables
```

template variables: `{step}`, `{col}`, `{value}`, `{set}`, `{pattern}`, `{auto}`

### Python Expressions

Several parameters support Python expressions using the `python:` block syntax:

```yaml
# Data source loading
tbl:
  python: |
    pl.scan_csv("data.csv").filter(pl.col("active") == True)

# Preprocessing
pre:
  python: |
    lambda df: df.filter(pl.col("date") >= "2024-01-01")

# Custom expressions
expr:
  python: |
    pl.col("value").is_between(0, 100)

# Callable actions
actions:
  error:
    python: |
      lambda: print("VALIDATION ERROR: Critical data quality issue detected!")
```

Python code only runs with `allow_python=True`, and it isn't sandboxed (see [Running Python
Code](#running-python-code)).

You can also use the shortcut syntax for lambda expressions:

```yaml
# Shortcut syntax (equivalent to python: block)
pre: |
  lambda df: df.filter(pl.col("status") == "active")
```

### Running Python Code

Python code in a YAML configuration (`python:` blocks and the `expr`, `pre`, and `active`
shortcuts) only runs if you allow it, with `allow_python=True`:

```python
pb.yaml_interrogate("validation.yaml", allow_python=True)
```

From the command line, use `pb run validation.yaml --allow-python`. Without it, a configuration that
contains Python code raises an error saying so.

The code isn't sandboxed. It runs with the same access as any Python script you run, so it can read
and write files, import modules, make network requests, or run commands. Treat a YAML configuration
that contains Python code like a Python script, and only allow it for configurations you wrote or
otherwise trust.

When the code runs, these names are available without importing them:

- `pb` (`pointblank`)
- `Path` from pathlib
- `datetime`, `Decimal` (from decimal), and `ZoneInfo` (from zoneinfo)
- `pl` (`polars`) if it's installed
- `pd` (`pandas`) if it's installed
- any modules given to `yaml_interrogate()` with `namespaces=`

A few `pb` functions don't count as Python code when they're called with plain values (strings,
numbers, lists, and so on), so they work without `allow_python=True`:

- `pb.load_dataset()`
- `pb.col()` and `pb.ref()`
- `pb.seg_group()`
- `pb.has_columns()` and `pb.has_rows()`
- the column selectors: `pb.starts_with()`, `pb.ends_with()`, `pb.contains()`, `pb.matches()`,
  `pb.everything()`, `pb.first_n()`, and `pb.last_n()` (combined with `&`, `|`, `-`, and `~` if you
  like)

The same goes for building the values that YAML has no way of writing: `Decimal()`,
`datetime.date()`, `datetime.time()`, `datetime.datetime()`, `datetime.timedelta()`, and
`ZoneInfo()`. For example, `value: {python: 'Decimal("2.2")'}` compares a column against an exact
decimal, and `value: {python: 'datetime.time(9, 30)'}` against a time of day. This is how
`Validate.to_yaml()` writes these values.

These only load a bundled dataset or build an object. Anything else, such as `pl.col("a") > 0`,
`pb.col("a").str.len()`, or `datetime.date.today()`, is Python code.

A date or datetime can also be written as an ISO 8601 string (e.g., `"2024-01-15"` or
`"2024-01-15T10:30:00"`), with a UTC offset for a timezone-aware datetime (e.g.,
`"2024-01-15T10:30:00+05:00"` or `"2024-01-15T10:30:00Z"`).

`yaml_to_python()` never runs Python code from the configuration. It copies the code into the
generated Python as written, so it doesn't need `allow_python=True`.

**Examples of callable actions:**

```yaml
# Simple output with built-in functions
actions:
  warning:
    python: |
      lambda: print(f"WARNING: {sum([1, 2, 3])} validation issues detected")

# Using available variables and string formatting
actions:
  error:
    python: |
      lambda: print("ERROR: Data validation failed at " + str(len("validation")))

# Multiple statements in lambda (using parentheses)
actions:
  critical:
    python: |
      lambda: (
          print("CRITICAL ALERT:"),
          print("Immediate attention required"),
          print("Contact data team")
      )[-1]  # Return the last value
```

String template actions are plain text, so they work without `allow_python=True`. For alerting
from configurations you don't fully trust, use string template actions and handle the external
communication in your application code after validation completes.

## Best Practices

### Organization

- use descriptive `tbl_name` and `label` values
- add `brief` descriptions for complex validations
- group related validations logically
- use consistent indentation and formatting

### Performance

- apply `pre` filters early to reduce data volume
- order validations from fast to slow
- use `columns_subset` for row-based validations when appropriate
- consider data source location (local vs. remote)
- choose `df_library` based on data size and operations:
  - `polars`: fastest for large datasets and analytical operations
  - `pandas`: best for complex transformations and data science workflows
  - `duckdb`: optimal for analytical queries on very large datasets

### Maintainability

- store YAML files in version control
- use template variables in actions and briefs
- document expected failures with comments
- test configurations with `validate_yaml()` before deployment
- specify `df_library` explicitly when using library-specific validation expressions
- keep DataFrame library choice consistent within related validation workflows

### Error Handling

- set appropriate thresholds based on data patterns
- use actions for monitoring and alerting
- start with conservative thresholds and adjust
- consider using `highest_only: false` for comprehensive reporting


### Data Contracts

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

A **data contract** is a declarative specification of what data should look like at a particular
point in your system. Rather than writing imperative validation code every time you need to check
data, contracts let you define *expectations as data* and then enforce them anywhere.

Pointblank's `Contract` class combines three things into a single, portable unit:

1. **Schema**: what columns exist and what types they are
2. **Validation steps**: semantic rules the data must satisfy
3. **Metadata**: who owns it, what version it is, and what to do on failure

Contracts can be serialized to YAML, version-controlled, and shared across teams. They can serve as
the single source of truth for data quality expectations at a given boundary.

## Creating Your First Contract

The simplest contract just names a set of validation steps:

```{python}
import pointblank as pb
import polars as pl

# Define a contract for customer data
customer_contract = pb.Contract(
    name="customer_records",
    steps=[
        pb.Step("col_vals_not_null", columns=["customer_id", "email"]),
        pb.Step("col_vals_regex", columns="email", pattern=r"^[^@]+@[^@]+\.[^@]+$"),
        pb.Step("rows_distinct", columns_subset=["customer_id"]),
    ],
)
```

This contract essentially says: "Any data calling itself `customer_records` must have non-null IDs
and emails, valid email formats, and unique customer IDs."

Now validate some data against it:

```{python}
customers = pl.DataFrame(
    {
        "customer_id": ["C001", "C002", "C003", "C004"],
        "email": ["alice@example.com", "bob@corp.io", "charlie@mail.org", "dave@startup.co"],
        "name": ["Alice", "Bob", "Charlie", "Dave"],
        "signup_date": ["2024-01-15", "2024-02-20", "2024-03-10", "2024-04-05"],
    }
)

customer_contract.validate(customers)
```

The `.validate()` method compiles the contract into a `Validate` object, runs all the checks, and
returns the interrogated result. You get the same rich validation report you're used to but the
contract itself is a reusable, declarative artifact.

## Adding a Schema

Contracts can include a `Schema` to enforce structural expectations (column names and data types):

```{python}
order_contract = pb.Contract(
    name="order_data",
    direction="source",
    schema=pb.Schema(
        order_id="String",
        customer_id="String",
        amount="Float64",
        quantity="Int64",
        status="String",
    ),
    steps=[
        pb.Step("col_vals_not_null", columns=["order_id", "customer_id", "amount"]),
        pb.Step("col_vals_gt", columns="amount", value=0),
        pb.Step("col_vals_ge", columns="quantity", value=1),
        pb.Step("col_vals_in_set", columns="status", set=["pending", "shipped", "delivered"]),
        pb.Step("rows_distinct", columns_subset=["order_id"]),
    ],
    version="1.0.0",
    owner="data-platform-team",
)
```

When a schema is defined, the contract automatically adds a `col_schema_match()` step before all
other validation steps. This means the schema check runs first, and if the table doesn't have the
right columns and types, you'll know immediately.

```{python}
orders = pl.DataFrame(
    {
        "order_id": ["ORD-001", "ORD-002", "ORD-003", "ORD-004", "ORD-005"],
        "customer_id": ["C001", "C002", "C001", "C003", "C002"],
        "amount": [29.99, 149.50, 9.99, 75.00, 220.00],
        "quantity": [1, 3, 1, 2, 5],
        "status": ["shipped", "pending", "delivered", "pending", "shipped"],
    }
)

order_contract.validate(orders)
```

## The `Step` Class

Each validation rule in a contract is represented by a `Step`, which is a declarative description of
a single validation method call. Steps store the method name and its arguments as plain data:

```{python}
# These are equivalent ways to express a validation rule:
step1 = pb.Step("col_vals_gt", columns="revenue", value=0)
step2 = pb.Step("col_vals_between", columns="age", left=0, right=150)
step3 = pb.Step("col_vals_in_set", columns="country", set=["US", "UK", "CA", "AU"])
step4 = pb.Step("col_vals_regex", columns="phone", pattern=r"^\+?[0-9\-\(\) ]+$")

# Steps are data so you can inspect them
print(step1)
print(step1.method)
print(step1.kwargs)
```

Because steps are data (not function calls), they serialize cleanly to YAML/JSON and can be
reconstructed anywhere. The `method` field corresponds directly to a validation method on the
`Validate` class, and the `kwargs` are passed through verbatim.

### Available Methods

A Step can use most of the core validation methods of the `Validate` class (the full list is in the
[`Contract`](`Contract`) reference). A Step that names any other method is rejected when the
contract is created. Here are the most common ones for contract definitions:

| Method | Purpose |
|--------|---------|
| `col_vals_gt()`, `col_vals_lt()`, `col_vals_ge()`, `col_vals_le()` | Numeric bounds |
| `col_vals_between()`, `col_vals_outside()` | Range checks |
| `col_vals_eq()`, `col_vals_ne()` | Equality checks |
| `col_vals_in_set()`, `col_vals_not_in_set()` | Categorical membership |
| `col_vals_not_null()`, `col_vals_null()` | Null checks |
| `col_vals_regex()` | Pattern matching |
| `col_exists()` | Column presence |
| `rows_distinct()` | Uniqueness (use `columns_subset=`) |
| `rows_complete()` | No nulls in any column |
| `row_count_match()` | Expected row count |
| `col_count_match()` | Expected column count |

## Contract Metadata

Contracts support rich metadata that makes them self-documenting and suitable for team workflows:

```{python}
production_contract = pb.Contract(
    name="clean_sales_report",
    direction="target",              # "source" or "target" boundary
    version="2.1.0",                 # Semantic versioning for evolution
    owner="data-platform-team",      # Who maintains this contract
    consumers=["analytics-team", "ml-team"],  # Who depends on it
    description="Validated, deduplicated sales data ready for downstream consumption.",
    on_violation="warn",             # "warn", "raise", or "log"
    schema=pb.Schema(
        sale_id="String",
        revenue="Float64",
        region="String",
    ),
    steps=[
        pb.Step("col_vals_not_null", columns=["sale_id", "revenue", "region"]),
        pb.Step("col_vals_gt", columns="revenue", value=0),
        pb.Step("rows_distinct", columns_subset=["sale_id"]),
    ],
    thresholds=pb.Thresholds(warning=0.01, error=0.05, critical=0.10),
)

print(production_contract)
```

### Direction

The `direction` parameter is metadata that signals where in a pipeline this contract applies:

- `"source"`: for inbound/raw data arriving from upstream
- `"target"`: for outbound data leaving your transform

Direction doesn't change validation behavior, but it's used in pipeline reports and helps teams
understand the contract's role in the system.

### Violation Handling

The `on_violation` parameter controls what happens when validation fails (used by the `Pipeline`
class, covered in the next guide page):

- `"warn"` (default): issue a Python `UserWarning`
- `"raise"`: raise a `RuntimeError` (halts execution)
- `"log"`: log via the `pointblank.contract` logger

## Using `to_validate()` for Custom Workflows

If you need more control, [`to_validate()`](`Contract.to_validate`) gives you back an
un-interrogated `Validate` object that you can extend with additional checks:

```{python}
# Start from the contract, then add ad-hoc checks
validation = (
    order_contract
    .to_validate(orders)
    .col_vals_lt(columns="amount", value=500)  # Additional check not in the contract
    .interrogate()
)

validation
```

This is useful when a contract captures your *baseline* expectations, but a specific workflow needs
extra checks on top.

## Contract Equality and Composition

Steps support equality comparison, making it easy to verify contracts:

```{python}
# Steps are equal if they have the same method and kwargs
s1 = pb.Step("col_vals_gt", columns="x", value=0)
s2 = pb.Step("col_vals_gt", columns="x", value=0)
s3 = pb.Step("col_vals_gt", columns="x", value=10)

print(f"s1 == s2: {s1 == s2}")
print(f"s1 == s3: {s1 == s3}")
```

You can compose contracts by combining steps from multiple sources:

```{python}
# Common checks shared across all tables
common_steps = [
    pb.Step("col_vals_not_null", columns=["id"]),
    pb.Step("rows_distinct", columns_subset=["id"]),
]

# Table-specific checks
sales_steps = [
    pb.Step("col_vals_gt", columns="revenue", value=0),
    pb.Step("col_vals_in_set", columns="region", set=["NA", "EU", "APAC"]),
]

# Compose them
sales_contract = pb.Contract(
    name="sales_data",
    steps=common_steps + sales_steps,
    version="1.0.0",
)

print(f"Total steps: {len(sales_contract.steps)}")
```

## Example: Validating with Failures

Let's see what happens when data doesn't meet the contract:

```{python}
# Data with quality issues
bad_orders = pl.DataFrame(
    {
        "order_id": ["ORD-001", "ORD-002", "ORD-001", "ORD-004", None],   # duplicates + null
        "customer_id": ["C001", None, "C001", "C003", "C002"],            # null
        "amount": [29.99, -5.00, 9.99, 0.0, 220.00],                      # negative + zero
        "quantity": [1, 3, 1, 0, 5],                                      # zero (should be >= 1)
        "status": ["shipped", "pending", "invalid", "pending", "shipped"],# invalid value
    }
)

order_contract.validate(bad_orders)
```

The validation report clearly shows which steps failed and how many test units were affected. This
makes it easy to diagnose data quality issues and communicate them to data producers.

## What's Next

Now that you understand contracts, the next guide page covers **Pipelines**. They combine a source
contract, a transform, and a target contract into a complete boundary enforcement workflow. This
lets you validate data at both the *input* and *output* of your data transformations.


### Pipeline Boundary Enforcement

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

A **pipeline** in Pointblank represents the enforcement of data quality at both boundaries of a data
transformation: the input ("source") and the output ("target"). The `Pipeline` class binds a source
contract and a target contract together, providing a unified way to validate data as it flows
through your system.

This approach embodies a key principle: **Pointblank enforces boundaries, it doesn't orchestrate the
transform.** You write the transformation logic; Pointblank guards the edges. This deliberate
separation of concerns means you can use any DataFrame library, any transformation approach, and any
orchestration tool you prefer. Pointblank's only job is to verify that data entering your transform
meets expectations and that data leaving your transform satisfies guarantees you've made to
downstream consumers.

Think of it like type-checking for data in motion: the source contract is your function's input type
signature, and the target contract is the return type. The pipeline ties them together.

## Why Boundary Enforcement?

Data pipelines rarely fail in obvious ways. More often, quality degrades silently: an upstream team
adds a column and reorders the schema, a third-party API starts returning nulls in a field that was
always populated, or a subtle join bug duplicates rows without raising any exception. These failures
propagate downstream and compound, making root-cause analysis painful.

Even with a perfectly correct transformation, data quality can degrade if:

- Upstream data changes shape, types, or semantics without notice
- A transform introduces bugs (null propagation, incorrect joins, off-by-one errors)
- Schema drift occurs silently over time
- External data sources degrade in quality between scheduled runs

Boundary enforcement addresses these problems by placing validation checkpoints at the two places
that matter most: the entry point and the exit point of your transformation. By validating at
**both** boundaries, you can:

1. **Catch upstream issues early**: before they enter your transform and corrupt derived tables
2. **Detect transform bugs**: by verifying output against a separate contract defined independently
3. **Short-circuit expensive processing**: fail fast when source data is invalid, saving compute
4. **Generate compliance evidence**: produce audit-ready reports showing data quality at each stage
5. **Assign accountability**: source contracts belong to producers, target contracts to consumers

This dual-boundary model makes it possible to distinguish between "the data I received was bad" and
"my code introduced a bug". This is a distinction that a single-point validation cannot make.

## Creating a Pipeline

A Pipeline needs at least one contract (source, target, or both). In practice, most pipelines define
both contracts because that gives you full boundary coverage. However, there are legitimate cases
for single-boundary pipelines (covered later in this guide).

The example below demonstrates a real-world pattern: a transaction processing pipeline that receives
raw financial data in one format (amounts in cents, all fields as raw strings) and produces clean,
normalized output (amounts in decimal dollars, validated types). Notice how the source and target
contracts encode fundamentally different expectations about the data's shape and values (this is
intentional). The source contract reflects what your upstream system actually produces, while the
target contract reflects what downstream consumers need.

```{python}
import pointblank as pb
import polars as pl

# Define the source contract (what raw data must look like)
source_contract = pb.Contract(
    name="raw_transactions",
    direction="source",
    schema=pb.Schema(
        txn_id="String",
        amount_cents="Int64",
        currency="String",
        timestamp="String",
    ),
    steps=[
        pb.Step("col_vals_not_null", columns=["txn_id", "amount_cents", "currency"]),
        pb.Step("col_vals_ge", columns="amount_cents", value=0),
        pb.Step("col_vals_in_set", columns="currency", set=["USD", "EUR", "GBP", "JPY"]),
        pb.Step("rows_distinct", columns_subset=["txn_id"]),
    ],
    version="1.0.0",
    owner="ingestion-team",
)

# Define the target contract (what clean data must look like)
target_contract = pb.Contract(
    name="clean_transactions",
    direction="target",
    schema=pb.Schema(
        txn_id="String",
        currency="String",
        timestamp="String",
        amount="Float64",
    ),
    steps=[
        pb.Step("col_vals_not_null", columns=["txn_id", "amount", "currency", "timestamp"]),
        pb.Step("col_vals_gt", columns="amount", value=0),
        pb.Step("col_vals_between", columns="amount", left=0.01, right=100_000.0),
        pb.Step("rows_distinct", columns_subset=["txn_id"]),
    ],
    version="1.0.0",
    consumers=["analytics-team", "fraud-team"],
)

# Create the pipeline
pipeline = pb.Pipeline(
    source=source_contract,
    target=target_contract,
    label="Transaction Processing Pipeline",
)

print(pipeline)
```

## Running the Pipeline

The `Pipeline.run()` method executes the full boundary enforcement workflow in a single call. This
is the primary way to use a pipeline: you hand it your input data and a transform function, and it
handles the validation orchestration for you. The method follows a strict three-step sequence:

1. Validate source data against the source contract
2. Execute your transform function (only if source validation passes)
3. Validate the output against the target contract

This ordering is deliberate. By validating the source *before* running the transform, you avoid
wasting compute on data that was already invalid. And by validating the target *after* the
transform, you confirm that your code produced output matching your guarantees.

```{python}
# Sample raw transaction data
raw_transactions = pl.DataFrame(
    {
        "txn_id": ["TXN-001", "TXN-002", "TXN-003", "TXN-004", "TXN-005"],
        "amount_cents": [4999, 12500, 750, 89900, 3200],
        "currency": ["USD", "EUR", "GBP", "USD", "JPY"],
        "timestamp": [
            "2024-06-01T10:30:00",
            "2024-06-01T11:45:00",
            "2024-06-01T12:00:00",
            "2024-06-01T14:20:00",
            "2024-06-01T15:55:00",
        ],
    }
)

# Your transformation logic
def process_transactions(df: pl.DataFrame) -> pl.DataFrame:
    """Convert cents to dollars and clean up the data."""
    return df.with_columns(
        (pl.col("amount_cents") / 100).alias("amount")
    ).drop("amount_cents")

# Run the full pipeline
result = pipeline.run(data=raw_transactions, transform=process_transactions)

print(f"Source passed: {result.source_passed}")
print(f"Target passed: {result.target_passed}")
print(f"Overall passed: {result.passed}")
```

The `result` is a `PipelineResult` object containing:

- `result.source_validation`: the interrogated Validate object for the source check
- `result.target_validation`: the interrogated Validate object for the target check
- `result.transform_output`: the data returned by your transform function
- `result.passed`: `True` only if **both** boundaries pass

The `PipelineResult` is designed to give you full introspection. You can access the raw validation
objects (which are standard Pointblank `Validate` instances), inspect the transformed data directly,
or simply check the boolean `passed` property for a quick go/no-go decision. Holding onto the result
object also gives you the ability to generate reports or log details after the fact.

## Inspecting Results

Because `PipelineResult` holds full `Validate` objects, you have access to Pointblank's rich
reporting capabilities. Each validation result can render its standard HTML report table, which
shows exactly which steps passed, which failed, and by how much. This is especially valuable during
development when you need to understand *why* a boundary check failed, not just *that* it failed.

You can access individual validation results to see the standard Pointblank report:

```{python}
# View the source boundary validation
result.source_validation
```

```{python}
# View the target boundary validation
result.target_validation
```

You can also get a text summary of the full pipeline run. The `get_report()` method produces a
human-readable summary that includes both the source and target validation results, making it
suitable for logging or printing in CI/CD output:

```{python}
print(result.get_report())
```

## Validating Boundaries Independently

You don't have to use `run()`. You can validate each boundary separately when you need more
control over the workflow. The `validate_source()` and `validate_target()` methods each return
a standard `Validate` object (not a `PipelineResult`), giving you direct access to the
interrogation results.

This pattern is particularly useful when:

- Your transform has side effects (writing to a database, calling an API) that you don't want
triggered during validation testing
- You need to inspect or modify intermediate data between the two validation steps
- You're building a custom orchestration layer that needs finer-grained control
- You want to run just one boundary check in a unit test

```{python}
# Validate just the source
source_result = pipeline.validate_source(raw_transactions)
print(f"Source all passed: {source_result.all_passed()}")
```

```{python}
# Run your transform manually
clean_data = process_transactions(raw_transactions)

# Then validate the target
target_result = pipeline.validate_target(clean_data)
print(f"Target all passed: {target_result.all_passed()}")
```

This is useful when your transform is complex, has side effects, or when you want to inspect
intermediate results before proceeding. Notice that these methods return `Validate` objects
directly, so you call `.all_passed()` on them rather than checking `.passed` as you would on a
`PipelineResult`.

## Short-Circuiting on Source Failure

By default, `Pipeline.run()` uses **short-circuit** behavior: if the source validation fails, the
transform is never executed and the target is never validated. This is a safety mechanism that
prevents invalid data from flowing into your transformation logic, where it could cause unexpected
exceptions, produce corrupt output, or trigger irreversible side effects like writing bad records to
a production database.

Short-circuiting also saves compute resources. If your transform is expensive (large joins, ML
inference, API calls), there's no reason to run it on data you already know is invalid. The pipeline
fails fast and reports exactly why.

In the example below, the source data contains a null `txn_id` and a negative `amount_cents`, both
of which violate the source contract. Because short-circuiting is enabled (the default), the
transform function is never called and the target validation is skipped entirely:

```{python}
# Data with problems (null txn_id, negative amount)
bad_raw_data = pl.DataFrame(
    {
        "txn_id": ["TXN-001", None, "TXN-003", "TXN-004", "TXN-005"],
        "amount_cents": [4999, -100, 750, 89900, 3200],
        "currency": ["USD", "EUR", "GBP", "USD", "JPY"],
        "timestamp": [
            "2024-06-01T10:30:00",
            "2024-06-01T11:45:00",
            "2024-06-01T12:00:00",
            "2024-06-01T14:20:00",
            "2024-06-01T15:55:00",
        ],
    }
)

# Source contract uses on_violation="warn" (default)
import warnings

with warnings.catch_warnings(record=True):
    warnings.simplefilter("always")
    result = pipeline.run(data=bad_raw_data, transform=process_transactions)

print(f"Source passed: {result.source_passed}")
print(f"Target validation: {result.target_validation}")  # None, was skipped
print(f"Transform output: {result.transform_output}")  # None, was skipped
```

To disable short-circuiting and always run both validations, set `short_circuit=False`. This is
useful when you want to collect validation reports from *both* boundaries regardless of whether the
source passed (for example, during development when you want to see the full picture of all data
quality issues at once):

```{python}
pipeline_no_shortcircuit = pb.Pipeline(
    source=source_contract,
    target=target_contract,
    short_circuit=False,
)

with warnings.catch_warnings(record=True):
    warnings.simplefilter("always")
    result = pipeline_no_shortcircuit.run(
        data=bad_raw_data, transform=process_transactions
    )

print(f"Source passed: {result.source_passed}")
print(f"Target validated: {result.target_validation is not None}")
```

## Violation Handling

When a contract's validation fails, the pipeline needs to *do something* about it. The
`on_violation` parameter on each `Contract` controls this behavior. There are three modes, each
suited to different stages of your development and deployment lifecycle:

- **`"warn"`** (default): Issues a Python `UserWarning`. The pipeline continues, and the failure
is recorded in the result. Ideal for development and monitoring.
- **`"raise"`**: Raises a `ContractViolationError` (a `RuntimeError`). The pipeline halts. Ideal
for production gates where invalid data must never proceed.
- **`"log"`**: Writes a warning-level message to the `pointblank.contract` logger. The pipeline
continues silently. Ideal for background monitoring and observability integrations.

Each contract in a pipeline can use a different violation mode. For example, you might use `"warn"`
on the source contract (so you can still inspect the data) and `"raise"` on the target contract (to
guarantee output quality).

### Warning Mode (Default)

The warning mode is non-blocking: your pipeline completes, but Python's warning system records that
a violation occurred. You can capture these warnings programmatically using the standard `warnings`
module:

```{python}
warn_contract = pb.Contract(
    name="soft_check",
    steps=[pb.Step("col_vals_not_null", columns=["txn_id"])],
    on_violation="warn",  # Issues a UserWarning
)

pipeline = pb.Pipeline(source=warn_contract)

with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    pipeline.validate_source(bad_raw_data)
    if w:
        print(f"Warning raised: {w[0].message}")
```

### Raise Mode

Use `on_violation="raise"` to halt execution immediately when data is invalid. This is the strictest
mode and the right choice for production pipelines where bad data must never reach downstream
systems. When a validation fails with this mode, a `ContractViolationError` (a subclass of
`RuntimeError`) is raised with a message identifying which contract failed. This applies to
`run()` as well as to `validate_source()` and `validate_target()`. The error keeps the results, so
the code that catches it can still look at what went wrong:

```{python}
strict_contract = pb.Contract(
    name="strict_check",
    steps=[pb.Step("col_vals_not_null", columns=["txn_id"])],
    on_violation="raise",  # Raises ContractViolationError
)

pipeline = pb.Pipeline(source=strict_contract)

try:
    pipeline.run(data=bad_raw_data, transform=process_transactions)
except pb.ContractViolationError as e:
    print(f"Stopped: {e}")
    print(f"Source passed: {e.result.source_passed}")
```

The error's `validation` attribute holds the failing validation, and when it comes from `run()`,
its `result` attribute holds the `PipelineResult` of the run. With `short_circuit=False`, a failing
source contract with `"raise"` doesn't stop the run right away: the transformation runs and the
target is validated first, and then the error is raised.

### Log Mode

Use `on_violation="log"` for background monitoring without interrupting the pipeline. This mode
integrates with Python's standard `logging` module, writing violation messages to the
`pointblank.contract` logger at the WARNING level. This is ideal for production systems with
structured logging and alerting infrastructure where you want visibility into data quality without
blocking data flow:

```python
import logging
logging.basicConfig(level=logging.WARNING)

log_contract = pb.Contract(
    name="monitored_check",
    steps=[pb.Step("col_vals_not_null", columns=["txn_id"])],
    on_violation="log",  # Logs via pointblank.contract logger
)
```

## Pipeline-Level Thresholds

You can set thresholds at the pipeline level to override contract-level thresholds. This is a
powerful mechanism for reusing the same contracts across different environments with different
tolerance levels. A contract defines *what* to check; the pipeline's thresholds define *how strict*
those checks should be in a given context.

For example, during development you might tolerate up to 10% of rows failing a given check (so you
can work with imperfect sample data), while in production you might require less than 1% failure.
Rather than maintaining separate contracts for each environment, you define the contracts once and
let the pipeline's thresholds control the sensitivity:

```{python}
# Relaxed thresholds for development
dev_pipeline = pb.Pipeline(
    source=source_contract,
    target=target_contract,
    thresholds=pb.Thresholds(warning=0.10, error=0.25, critical=0.50),
    label="Development Pipeline",
)

# Strict thresholds for production
prod_pipeline = pb.Pipeline(
    source=source_contract,
    target=target_contract,
    thresholds=pb.Thresholds(warning=0.01, error=0.05, critical=0.10),
    label="Production Pipeline",
)
```

## Source-Only and Target-Only Pipelines

You don't always need both boundaries. In many real-world systems, you'll encounter situations where
only one boundary is relevant. The `Pipeline` class supports this naturally: simply omit the
contract you don't need, and the pipeline will only validate the boundary you've defined.

Common patterns:

### Source-Only (Ingestion Guard)

Validate incoming data without a specific output contract. This pattern is common at the edges of
your system where you receive data from external sources (APIs, file uploads, partner feeds) and
want to reject or flag invalid input before it enters your data lake or warehouse:

```{python}
ingestion_pipeline = pb.Pipeline(source=source_contract)

result = ingestion_pipeline.run(
    data=raw_transactions,
    transform=lambda df: df,  # Pass-through
)
print(f"Ingestion valid: {result.passed}")
```

### Target-Only (Output Guard)

Validate data before it leaves your system. This pattern is useful for data exports, API responses,
or any point where you're publishing data to external consumers. You've already transformed the data
and simply want to confirm it meets the contract you've promised:

```{python}
export_pipeline = pb.Pipeline(target=target_contract)

clean_data = process_transactions(raw_transactions)
result = export_pipeline.run(
    data=clean_data,
    transform=lambda df: df,  # Already transformed
)
print(f"Export valid: {result.passed}")
```

## Detecting Transform Bugs

One of the most powerful applications of boundary enforcement is catching bugs in your
transformation logic. When you validate at both boundaries, you can distinguish between "the input
data was bad" (source failure) and "my code introduced a problem" (source passes, target fails).
This distinction is invaluable during debugging and code review.

Here's an example where a buggy transform introduces data quality issues. The source data is
perfectly valid, but the transformation logic has a conditional branch that inadvertently nulls out
small transaction amounts. Without boundary enforcement, this bug might go undetected until a
downstream consumer reports missing data (potentially days or weeks later):

```{python}
def buggy_transform(df: pl.DataFrame) -> pl.DataFrame:
    """A transform with a bug (introduces nulls for certain records)."""
    return df.with_columns(
        pl.when(pl.col("amount_cents") < 1000)
        .then(None)  # Bug! Small amounts become null
        .otherwise(pl.col("amount_cents") / 100)
        .alias("amount")
    ).drop("amount_cents")

# The source data is fine...
result_source = pipeline.validate_source(raw_transactions)
print(f"Source passes: {result_source.all_passed()}")

# ...but the buggy transform breaks the target contract
with warnings.catch_warnings(record=True):
    warnings.simplefilter("always")
    result = pipeline.run(data=raw_transactions, transform=buggy_transform)

print(f"\nAfter buggy transform:")
print(f"  Source passed: {result.source_passed}")
print(f"  Target passed: {result.target_passed}")
print(f"  Overall passed: {result.passed}")
```

```{python}
# The target validation shows exactly what went wrong
result.target_validation
```

The target validation report above pinpoints the failure: the `col_vals_not_null` step on the
`amount` column fails because the buggy transform introduced nulls. This makes the root cause
obvious and you can see exactly which contract step failed and trace it back to the transformation
logic. Without the target contract, this bug would silently produce a table with missing values.

## Working with Pandas

Pipelines work with any DataFrame library supported by Pointblank, including Pandas. The interface
is identical: you pass a Pandas DataFrame to `run()` or `validate_source()`, and Pointblank handles
the rest. This means your contracts are portable across DataFrame backends and you can define a
contract once and use it to validate both Polars and Pandas data without modification.

Here's the same transaction-processing workflow using Pandas DataFrames:

```{python}
import pandas as pd

# Pandas DataFrame
raw_pd = pd.DataFrame(
    {
        "txn_id": ["TXN-001", "TXN-002", "TXN-003"],
        "amount_cents": [4999, 12500, 750],
        "currency": ["USD", "EUR", "GBP"],
        "timestamp": ["2024-06-01T10:30:00", "2024-06-01T11:45:00", "2024-06-01T12:00:00"],
    }
)

# Pandas transform
def pandas_transform(df: pd.DataFrame) -> pd.DataFrame:
    df = df.copy()
    df["amount"] = df["amount_cents"] / 100
    df = df.drop(columns=["amount_cents"])
    return df

# Same pipeline works with Pandas
source_only = pb.Pipeline(
    source=pb.Contract(
        name="pandas_source",
        steps=[
            pb.Step("col_vals_not_null", columns=["txn_id", "amount_cents"]),
            pb.Step("col_vals_ge", columns="amount_cents", value=0),
        ],
    ),
    target=pb.Contract(
        name="pandas_target",
        steps=[
            pb.Step("col_vals_not_null", columns=["txn_id", "amount"]),
            pb.Step("col_vals_gt", columns="amount", value=0),
        ],
    ),
)

result = source_only.run(data=raw_pd, transform=pandas_transform)
print(f"Pandas pipeline passed: {result.passed}")
```

## Multiple Pipeline Runs

Pipelines are stateless and can be run multiple times with different data. Each call to `run()`
produces an independent `PipelineResult` with no side effects on the pipeline object itself. This
makes pipelines safe to reuse in loops, parallel processing, or long-running services where the same
validation logic applies to many batches of data.

The example below demonstrates running a single pipeline against multiple data batches (a common
pattern in batch-processing systems where data arrives in chunks):

```{python}
# Same pipeline, different data batches
batches = [
    pl.DataFrame({
        "txn_id": [f"BATCH1-{i}" for i in range(3)],
        "amount_cents": [1000, 2000, 3000],
        "currency": ["USD", "USD", "EUR"],
        "timestamp": ["2024-06-01T10:00:00"] * 3,
    }),
    pl.DataFrame({
        "txn_id": [f"BATCH2-{i}" for i in range(3)],
        "amount_cents": [500, 1500, 4500],
        "currency": ["GBP", "JPY", "USD"],
        "timestamp": ["2024-06-02T10:00:00"] * 3,
    }),
]

for i, batch in enumerate(batches):
    result = pipeline.run(data=batch, transform=process_transactions)
    print(f"Batch {i + 1}: {'PASSED' if result.passed else 'FAILED'}")
```

## What's Next

You now have a solid understanding of how `Pipeline` enforces data quality at both boundaries of
a transformation. You've seen how to run full pipelines, inspect results, validate boundaries
independently, handle violations, tune thresholds per environment, and detect transform bugs
through dual-boundary validation.

The next guide page covers **YAML Contracts**: how to serialize contracts and pipelines to YAML
files for version control, team collaboration, and configuration-driven validation workflows. YAML
serialization is especially powerful in production environments where contracts should be managed as
configuration rather than code, enabling non-developers to review and modify data quality rules.


### YAML Contracts & Pipelines

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Contracts and pipelines are designed to be **data, not code**. They serialize cleanly to YAML,
making them useful for version control, team collaboration, and configuration-driven workflows. This
guide covers how to write, load, and use YAML-based contracts and pipelines.

When you define a contract in Python, it lives inside your application code. That's fine for quick
validation tasks, but in production data platforms you often want the *specification* of what data
should look like to live separately from the *code* that processes it. YAML contracts give you
exactly that separation with a human-readable, machine-parseable definition of data expectations
that anyone on your team can read, review, and modify.

## Why YAML?

YAML contracts offer several advantages over defining contracts purely in Python:

- **Version-controllable**: track contract changes alongside your code in Git, with clear diffs
  when expectations evolve
- **Language-agnostic**: contracts can be understood (and even authored) by non-Python team members
  such as data analysts, product managers, or compliance officers
- **Separates concerns**: validation rules live apart from transform logic, meaning you can update
  expectations without touching application code
- **CI/CD friendly**: load and enforce contracts in automated pipelines and gate deployments on
  contract compliance
- **Reviewable**: pull requests that change data contracts are easy to review because the YAML
  format highlights exactly what changed in a readable way

The YAML approach doesn't replace the Python API but, rather, it complements it. Use YAML for
stable, well-defined contracts that change infrequently, and Python for exploratory validation or
complex logic that benefits from programmatic construction.

## Contract YAML Format

Every contract YAML file uses a `contract:` top-level key as its root element. Inside this key you
define all the metadata, schema, and validation steps that make up your contract. The structure is
designed to be self-documenting so someone reading the file should immediately understand what data
this contract governs, who owns it, and what rules are enforced.

Here's a complete example showing all the available fields:

```yaml
contract:
  name: customer_records
  direction: source
  version: "1.2.0"
  owner: data-platform-team
  consumers:
    - analytics-team
    - ml-team
  description: "Raw customer data from the CRM system"
  on_violation: warn

  schema:
    customer_id: String
    email: String
    name: String
    signup_date: String
    plan: String

  steps:
    - col_vals_not_null:
        columns:
          - customer_id
          - email
    - col_vals_regex:
        columns: email
        pattern: "^[^@]+@[^@]+\\.[^@]+$"
    - rows_distinct:
        columns_subset:
          - customer_id
    - col_vals_in_set:
        columns: plan
        set:
          - free
          - pro
          - enterprise

  thresholds:
    warning: 0.01
    error: 0.05
    critical: 0.10
```

### Structure Reference

The table below summarizes every key available inside the `contract:` block. Only `name` is
required and everything else has sensible defaults or can be omitted if not applicable to your use
case.

| Key | Required | Description |
|-----|----------|-------------|
| `name` | Yes | Human-readable identifier for this contract |
| `direction` | No | `"source"` (default) or `"target"`: indicates where in a pipeline this contract applies |
| `version` | No | Semantic version string (e.g., `"1.2.0"`) for tracking contract evolution over time |
| `owner` | No | Team or individual responsible for maintaining this contract |
| `consumers` | No | List of teams or systems that depend on data conforming to this contract |
| `description` | No | Longer prose describing the contract's purpose and context |
| `on_violation` | No | Behavior when validation fails: `"warn"` (default), `"raise"`, or `"log"` |
| `schema` | No | Column name -> data type mapping; triggers a `col_schema_match()` check |
| `steps` | No | Ordered list of validation steps to execute |
| `thresholds` | No | Threshold levels (`warning`, `error`, `critical`) as fractions or absolute counts |

### Step Syntax in YAML

Steps are the heart of a contract. They define the actual validation rules that will be checked
against your data. In YAML, each step is a list item whose key is the name of a Pointblank
validation method (like `col_vals_gt` or `rows_distinct`), and whose value is a dictionary of the
arguments to pass to that method.

This mapping is direct: the YAML step `col_vals_gt: {columns: amount, value: 0}` is exactly
equivalent to calling `.col_vals_gt(columns="amount", value=0)` in Python. Here are examples
covering the most common patterns:

```yaml
steps:
  # Step with keyword arguments
  - col_vals_gt:
      columns: amount
      value: 0

  # Step with a list of columns
  - col_vals_not_null:
      columns:
        - id
        - name
        - email

  # Step with multiple arguments
  - col_vals_between:
      columns: age
      left: 0
      right: 150

  # Step with a set/list of values
  - col_vals_in_set:
      columns: status
      set:
        - active
        - inactive
        - suspended

  # Step with no arguments
  - rows_complete: {}
```

## Writing and Loading Contracts

Pointblank supports a complete round-trip workflow: define contracts in Python, save them to YAML
for version control and sharing, then load them back in any environment. This section walks through
each step of that workflow.

### Saving to YAML

The `.to_yaml()` method serializes a `Contract` object to a YAML string. You can optionally pass a
file path to write directly to disk. The resulting YAML is clean, readable, and contains all the
information needed to reconstruct the contract exactly.

```{python}
import pointblank as pb
from pathlib import Path

# Create a contract in Python
contract = pb.Contract(
    name="product_catalog",
    direction="source",
    version="1.0.0",
    owner="catalog-team",
    schema=pb.Schema(
        product_id="String",
        name="String",
        price="Float64",
        category="String",
        in_stock="Boolean",
    ),
    steps=[
        pb.Step("col_vals_not_null", columns=["product_id", "name", "price"]),
        pb.Step("col_vals_gt", columns="price", value=0),
        pb.Step("col_vals_in_set", columns="category", set=["electronics", "clothing", "food", "home"]),
        pb.Step("rows_distinct", columns_subset=["product_id"]),
    ],
    thresholds=pb.Thresholds(warning=0.01, error=0.05),
)

# Save to YAML
yaml_output = contract.to_yaml("product_contract.yaml")
print(yaml_output)
```

### Loading from YAML

The `Contract.from_yaml()` class method reads a YAML file and reconstructs the full `Contract`
object, including schema, steps, thresholds, and all metadata. The loaded contract is identical in
behavior to one created directly in Python (there's no functional difference).

```{python}
# Load the contract back
loaded_contract = pb.Contract.from_yaml("product_contract.yaml")
print(loaded_contract)
print(f"Steps: {len(loaded_contract.steps)}")
print(f"Version: {loaded_contract.version}")
```

### Validating with a Loaded Contract

Once loaded, a contract works exactly like one defined in Python. Call `.validate(data)` to compile
it into a `Validate` object, execute all checks, and get the standard Pointblank validation report.
This is the core value proposition of YAML contracts: define once, validate anywhere.

```{python}
import polars as pl

products = pl.DataFrame(
    {
        "product_id": ["P001", "P002", "P003", "P004", "P005"],
        "name": ["Laptop", "T-Shirt", "Coffee Beans", "Desk Lamp", "Headphones"],
        "price": [999.99, 29.99, 12.50, 45.00, 79.99],
        "category": ["electronics", "clothing", "food", "home", "electronics"],
        "in_stock": [True, True, False, True, True],
    }
)

loaded_contract.validate(products)
```

## Pipeline YAML Format

While individual contracts validate data at a single point, **pipelines** enforce quality at both
the input and output boundaries of a transformation. A pipeline YAML file packages the source
contract, target contract, and pipeline-level settings (like global thresholds and short-circuit
behavior) into a single, self-contained document.

This is especially powerful for team workflows: a data engineer defines the pipeline specification
in YAML, commits it to version control, and any downstream system can load and execute it without
knowing anything about the implementation details.

Let's build a pipeline in Python and export it to see what the YAML looks like:

```{python}
# Create a pipeline and export to YAML
source = pb.Contract(
    name="raw_events",
    direction="source",
    version="1.0.0",
    schema=pb.Schema(
        event_id="String",
        user_id="String",
        event_type="String",
        value_cents="Int64",
    ),
    steps=[
        pb.Step("col_vals_not_null", columns=["event_id", "user_id", "event_type"]),
        pb.Step("col_vals_ge", columns="value_cents", value=0),
        pb.Step("col_vals_in_set", columns="event_type", set=["click", "purchase", "view", "signup"]),
        pb.Step("rows_distinct", columns_subset=["event_id"]),
    ],
)

target = pb.Contract(
    name="enriched_events",
    direction="target",
    version="1.0.0",
    schema=pb.Schema(
        event_id="String",
        user_id="String",
        event_type="String",
        value="Float64",
    ),
    steps=[
        pb.Step("col_vals_not_null", columns=["event_id", "user_id", "event_type", "value"]),
        pb.Step("col_vals_gt", columns="value", value=0),
        pb.Step("rows_distinct", columns_subset=["event_id"]),
    ],
)

pipeline = pb.Pipeline(
    source=source,
    target=target,
    label="Event Processing Pipeline",
    thresholds=pb.Thresholds(warning=0.02, error=0.10),
)

pipeline_yaml = pipeline.to_yaml("event_pipeline.yaml")
print(pipeline_yaml)
```

### Pipeline YAML Structure

The generated YAML has three top-level sections that correspond to the three components of boundary
enforcement: the `pipeline:` section for global settings, the `source:` section for the inbound
contract, and the `target:` section for the outbound contract. Each section can be understood
independently, which makes reviewing changes straightforward.

Here's the annotated structure:

```yaml
# Pipeline metadata and global settings
pipeline:
  label: "Event Processing Pipeline"
  thresholds:
    warning: 0.02
    error: 0.10
  # short_circuit: true   (default, omitted when true)

# Source contract (inbound boundary)
source:
  name: raw_events
  version: "1.0.0"
  schema:
    event_id: String
    user_id: String
    event_type: String
    value_cents: Int64
  steps:
    - col_vals_not_null:
        columns: [event_id, user_id, event_type]
    - col_vals_ge:
        columns: value_cents
        value: 0

# Target contract (outbound boundary)
target:
  name: enriched_events
  version: "1.0.0"
  schema:
    event_id: String
    user_id: String
    event_type: String
    value: Float64
  steps:
    - col_vals_not_null:
        columns: [event_id, user_id, event_type, value]
    - col_vals_gt:
        columns: value
        value: 0
```

### Loading and Running a Pipeline from YAML

Loading a pipeline from YAML works the same way as loading a contract: call the
[`from_yaml()`](`Pipeline.from_yaml`) class method with the file path. The returned `Pipeline`
object is fully functional and ready to validate data or run a complete boundary enforcement
workflow.

```{python}
# Load the pipeline from YAML
loaded_pipeline = pb.Pipeline.from_yaml("event_pipeline.yaml")
print(loaded_pipeline)
```

With the pipeline loaded, you provide data and a transform function just as you would with a
pipeline defined in Python. The YAML only stores the *contract specifications* whereas the transform
logic always lives in your Python code (since transforms are arbitrary functions that can't be
meaningfully serialized to YAML).

```{python}
# Create test data and run the pipeline
events = pl.DataFrame(
    {
        "event_id": ["EVT-001", "EVT-002", "EVT-003", "EVT-004"],
        "user_id": ["U100", "U200", "U100", "U300"],
        "event_type": ["purchase", "click", "view", "signup"],
        "value_cents": [2500, 0, 0, 0],
    }
)

def enrich_events(df: pl.DataFrame) -> pl.DataFrame:
    """Convert cents to dollars, defaulting zero to a minimum value."""
    return df.with_columns(
        pl.when(pl.col("value_cents") > 0)
        .then(pl.col("value_cents") / 100)
        .otherwise(0.01)  # Minimum trackable value
        .alias("value")
    ).drop("value_cents")

result = loaded_pipeline.run(data=events, transform=enrich_events)
print(f"Pipeline passed: {result.passed}")
```

```{python}
result.target_validation
```

## Practical Patterns

The following patterns illustrate common ways teams use YAML contracts and pipelines in real-world
data platforms. These aren't exhaustive: they're starting points that you can adapt to your own
workflow.

### Pattern 1: Environment-Specific Pipelines

A common need is to apply different tolerance levels in different environments. In development, you
might accept 20% failure rates while iterating on a transform. In production, anything above 1% is
a critical issue. YAML contracts make this easy: store the same validation logic in a shared
contract, but load different pipeline configurations that set environment-appropriate thresholds.

```{python}
# Same contracts, different thresholds
dev_pipeline = pb.Pipeline(
    source=source,
    target=target,
    thresholds=pb.Thresholds(warning=0.20, error=0.50),  # Lenient for dev
    label="DEV: Event Pipeline",
)

prod_pipeline = pb.Pipeline(
    source=source,
    target=target,
    thresholds=pb.Thresholds(warning=0.01, error=0.05, critical=0.10),  # Strict
    label="PROD: Event Pipeline",
)

# In practice, you'd select based on an env var:
# import os
# pipeline = pb.Pipeline.from_yaml(f"contracts/{os.environ['ENV']}_pipeline.yaml")
```

### Pattern 2: Contract Library

As your data platform grows, you'll accumulate many contracts. A recommended practice is to organize
them into a directory structure that mirrors your data architecture. This creates a discoverable,
browsable "library" of data quality expectations that serves as living documentation of your
system's data contracts.

Here's a typical layout:

```
contracts/
├── sources/
│   ├── raw_events.yaml
│   ├── raw_transactions.yaml
│   └── raw_customers.yaml
├── targets/
│   ├── clean_events.yaml
│   ├── clean_transactions.yaml
│   └── clean_customers.yaml
└── pipelines/
    ├── event_pipeline.yaml
    └── transaction_pipeline.yaml
```

```python
# Load any contract from the library
events_source = pb.Contract.from_yaml("contracts/sources/raw_events.yaml")
events_target = pb.Contract.from_yaml("contracts/targets/clean_events.yaml")

# Build pipeline dynamically
pipeline = pb.Pipeline(source=events_source, target=events_target)
```

### Pattern 3: CI/CD Integration

YAML contracts integrate naturally into CI/CD pipelines. Because contracts are plain files, you can
load them in test suites, gate deployments on contract compliance, and fail builds when data quality
regresses. This turns data quality from a manual spot-check into an automated, enforceable
standard.

Here's what a pytest-based contract test might look like:

```python
# In your test suite or CI script:
import pointblank as pb

def test_pipeline_contract():
    pipeline = pb.Pipeline.from_yaml("contracts/pipelines/event_pipeline.yaml")
    test_data = load_test_fixture("events_sample.parquet")

    result = pipeline.run(data=test_data, transform=my_transform)
    assert result.passed, result.get_report()
```

### Pattern 4: Composing Steps from Multiple Sources

Sometimes you want to define a set of "base" validation rules that apply to all tables (like "every
table must have a non-null `id` column that's unique"), and then layer on table-specific rules.
YAML makes this easy because you can parse step definitions from YAML fragments and combine them
programmatically.

This pattern is especially useful when you have organization-wide data quality standards that every
contract should inherit.

```{python}
# Base steps that apply to all tables
base_steps_yaml = """
- col_vals_not_null:
    columns: [id]
- rows_distinct:
    columns_subset: [id]
"""

import yaml
base_steps = [pb.Step.from_dict(s) for s in yaml.safe_load(base_steps_yaml)]

# Compose with table-specific steps
full_contract = pb.Contract(
    name="composed_contract",
    steps=base_steps + [
        pb.Step("col_vals_gt", columns="score", value=0),
        pb.Step("col_vals_le", columns="score", value=100),
    ],
)

print(f"Total steps: {len(full_contract.steps)}")
for step in full_contract.steps:
    print(f"  - {step}")
```

## Round-Trip Fidelity

A critical property of Pointblank's YAML serialization is **round-trip fidelity**: when you save a
contract to YAML and load it back, you get an identical object. No information is lost, no defaults
are silently changed, and no steps are reordered. This guarantee means you can confidently use YAML
as your source of truth without worrying about subtle differences between the YAML representation
and the in-memory object.

Let's verify this with a fully-specified contract:

```{python}
# Original contract
original = pb.Contract(
    name="fidelity_test",
    direction="target",
    version="3.0.0",
    owner="qa-team",
    consumers=["reporting", "dashboards"],
    on_violation="raise",
    schema=pb.Schema(x="Int64", y="Float64", z="String"),
    steps=[
        pb.Step("col_vals_not_null", columns=["x", "y", "z"]),
        pb.Step("col_vals_between", columns="y", left=-1.0, right=1.0),
    ],
    thresholds=pb.Thresholds(warning=0.01, error=0.05, critical=0.10),
)

# Round-trip through YAML
original.to_yaml("fidelity_test.yaml")
restored = pb.Contract.from_yaml("fidelity_test.yaml")

# Verify fidelity
print(f"Name matches: {original.name == restored.name}")
print(f"Direction matches: {original.direction == restored.direction}")
print(f"Version matches: {original.version == restored.version}")
print(f"Owner matches: {original.owner == restored.owner}")
print(f"Steps match: {original.steps == restored.steps}")
print(f"Thresholds match: {original.thresholds.warning == restored.thresholds.warning}")
```

## Cleanup

The examples in this guide created several YAML files on disk to demonstrate the serialization
workflow. The following hidden cell removes them so they don't clutter your working directory. In
your own projects, these files would live in version control and persist across sessions. You'd
never need to clean them up like this.

```{python}
#| echo: false
import os
for f in ["product_contract.yaml", "event_pipeline.yaml", "fidelity_test.yaml",
           "basic_validation.yaml"]:
    if os.path.exists(f):
        os.remove(f)
```

## Conclusion

The table below summarizes the key operations and how they map between the Python API and YAML:

| Operation | Python API | YAML |
|-----------|-----------|------|
| Define contract | `pb.Contract(...)` | `contract:` block |
| Define pipeline | `pb.Pipeline(...)` | `pipeline:` + `source:` + `target:` |
| Save | `.to_yaml("path.yaml")` | --- |
| Load contract | `pb.Contract.from_yaml("path.yaml")` | --- |
| Load pipeline | `pb.Pipeline.from_yaml("path.yaml")` | --- |
| Validate | `.validate(data)` | Load -> validate in Python |
| Run pipeline | `.run(data, transform)` | Load -> run in Python |

YAML contracts and pipelines give you the best of both worlds: declarative, reviewable
specifications that integrate seamlessly with Pointblank's Python execution engine. The separation
of *what* (YAML contracts) from *how* (Python transforms) creates a clean architecture where data
quality expectations can evolve independently from processing logic: reviewed by different people,
on different timelines, with clear accountability at each boundary.


### Importing External Schemas

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Many teams already have data schemas defined in other tools: JSON Schema files for API validation,
Frictionless Table Schemas for open data, dbt `schema.yml` files for analytics pipelines, or ODCS
data contracts for cross-team governance. Rather than manually rewriting these specifications as
Pointblank validation steps, you can **import** them directly.

The [`import_contract()`](`pointblank.import_contract`) function reads an external schema definition
and produces a `ContractImport` object containing everything Pointblank needs to validate data:
column types, constraints, and mapped validation steps. From there you can create a `Validate`
workflow, a `Contract` object, generate equivalent Python code, or produce a YAML definition, all
with a single function call.

## Quick Start

The fastest path from an external schema to running validation:

```{python}
import pointblank as pb
import polars as pl

# Define a JSON Schema (could also be loaded from a file)
user_schema = {
    "type": "object",
    "properties": {
        "user_id": {"type": "integer"},
        "email": {"type": "string", "format": "email"},
        "age": {"type": "integer", "minimum": 0, "maximum": 150},
        "status": {"type": "string", "enum": ["active", "inactive", "pending"]},
    },
    "required": ["user_id", "email"],
}

# Import the schema
result = pb.import_contract(user_schema, format="json_schema")

# Create sample data and validate
users = pl.DataFrame(
    {
        "user_id": [1, 2, 3, 4, 5],
        "email": [
            "alice@example.com",
            "bob@corp.io",
            "charlie@mail.org",
            "dave@startup.co",
            "eve@company.net",
        ],
        "age": [28, 34, 45, 22, 31],
        "status": ["active", "active", "inactive", "pending", "active"],
    }
)

result.to_validate(data=users).interrogate()
```

That's it. The JSON Schema `minimum`, `maximum`, `enum`, `format`, and `required` keywords were
automatically translated into the appropriate Pointblank validation steps. Each keyword becomes a
dedicated validation check: `minimum` becomes `col_vals_ge()`, `maximum` becomes `col_vals_le()`,
`enum` becomes `col_vals_in_set()`, and so on. The schema's `required` array generates
`col_vals_not_null()` steps for each listed field, ensuring that null values are caught at
validation time.

## How It Works

The import process has three stages:

1. **Parse**: the external schema is read (from a file path, a dict, or a Python object)
2. **Map**: each constraint in the source format is translated to a Pointblank validation method
3. **Package**: the results are stored in a `ContractImport` object with multiple output options

```{mermaid}
flowchart LR
    A[External Schema] --> B[import_contract]
    B --> C[ContractImport]
    C --> D[.to_validate‹data›]
    C --> E[.to_contract‹›]
    C --> F[.to_python‹›]
    C --> G[.to_yaml‹›]
```

The `ContractImport` object is your bridge between the external world and Pointblank. It doesn't
execute anything on its own. Rather, it holds the *translated specification* and lets you choose
how to use it. This separation is intentional: you can inspect the translation results, check
for any warnings, and decide how to proceed before committing to a particular output format.

## The `ContractImport` Object

After calling `import_contract()`, you get back a `ContractImport` with these key attributes and
methods:

| Attribute / Method | Description |
|---|---|
| `.columns` | List of `(column_name, dtype)` tuples detected from the source |
| `.constraints` | List of `MappedConstraint` objects (method + kwargs) |
| `.warnings` | Messages about constraints that couldn't be translated |
| `.coverage` | Fraction of source constraints successfully mapped (0.0–1.0) |
| `.metadata` | Extra metadata (title, description) from the source |
| `.to_validate(data)` | Build a `Validate` object ready for `.interrogate()` |
| `.to_contract(name)` | Build a `Contract` object for pipeline use |
| `.to_python()` | Generate equivalent Python code as a string |
| `.to_yaml()` | Generate Pointblank YAML configuration |
| `.summary()` | Return a human-readable summary |

### Inspecting What Was Imported

Before running validation, it's often useful to inspect what the import produced:

```{python}
result = pb.import_contract(user_schema, format="json_schema")
print(result.summary())
```

If any constraints couldn't be mapped, they appear in `.warnings`:

```{python}
# A schema with an unmappable format
schema_with_date = {
    "type": "object",
    "properties": {
        "created_at": {"type": "string", "format": "date-time"},
    },
}
result = pb.import_contract(schema_with_date, format="json_schema")

if result.warnings:
    for w in result.warnings:
        print(f"⚠ {w}")

print(f"\nCoverage: {result.coverage:.0%}")
```

The `coverage` metric tells you what fraction of the source constraints were successfully
translated. A coverage of 100% means everything mapped cleanly; lower values mean some constraints
were skipped (with details in `warnings`). This transparency is important because no translation
between formats is perfect. By checking `coverage` and `warnings` before running validation, you
can be confident about exactly which parts of your original schema are being enforced and which
parts might need manual attention.

## Supported Formats

Pointblank ships with adapters for four widely-used schema and contract formats.

```{python}
pb.list_adapters()
```

### JSON Schema

[JSON Schema](https://json-schema.org/) is a widely used format for describing the structure of JSON
data. Because tabular data (DataFrames) can be modeled as arrays of JSON objects, JSON Schema is a
natural fit for defining column-level constraints.

**Constraint mapping:**

| JSON Schema Keyword | Pointblank Method |
|---|---|
| `type: "integer"` / `"number"` / `"string"` / `"boolean"` | Schema dtype check |
| `minimum` | `col_vals_ge()` |
| `maximum` | `col_vals_le()` |
| `exclusiveMinimum` | `col_vals_gt()` |
| `exclusiveMaximum` | `col_vals_lt()` |
| `enum` | `col_vals_in_set()` |
| `pattern` | `col_vals_regex()` |
| `format: "email"` | `col_vals_within_spec(spec="email")` |
| `format: "uri"` | `col_vals_within_spec(spec="url")` |
| `format: "ipv4"` / `"ipv6"` | `col_vals_within_spec(spec="ipv4"/"ipv6")` |
| `const` | `col_vals_eq()` |
| `required` (array of field names) | `col_vals_not_null()` |

**Importing from a file:**

```python
# File-based import (auto-detects .schema.json extension)
result = pb.import_contract("models/user_profile.schema.json")
```

**Importing from a dict:**

```{python}
product_schema = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "Product Catalog",
    "description": "Expected structure for product data",
    "type": "object",
    "properties": {
        "sku": {"type": "string", "pattern": "^[A-Z]{3}-[0-9]{4}$"},
        "price": {"type": "number", "exclusiveMinimum": 0},
        "category": {"type": "string", "enum": ["electronics", "clothing", "food", "other"]},
        "in_stock": {"type": "boolean"},
    },
    "required": ["sku", "price", "category"],
}

result = pb.import_contract(product_schema, format="json_schema")

# Inspect what was mapped
for c in result.constraints:
    print(f"  {c.method}({c.kwargs})")
```

Each constraint from the JSON Schema has been translated into the corresponding Pointblank method
call. The `pattern` keyword on the `sku` field became a `col_vals_regex()` step, `exclusiveMinimum`
became `col_vals_gt()` (note the strict inequality), and the three `required` fields each generated
a `col_vals_not_null()` step. You can iterate over `result.constraints` like this to verify the
translation before running any validation.

### Frictionless Data Table Schema

[Frictionless Data](https://frictionlessdata.io/) is a set of standards for describing and
packaging data. The **Table Schema** format is particularly well-suited for tabular data validation,
with explicit support for column types, constraints, and primary/foreign keys.

**Constraint mapping:**

| Frictionless Feature | Pointblank Method |
|---|---|
| `type` | Schema dtype check |
| `constraints.required: true` | `col_vals_not_null()` |
| `constraints.unique: true` | `rows_distinct()` |
| `constraints.minimum` / `maximum` | `col_vals_ge()` / `col_vals_le()` |
| `constraints.enum` | `col_vals_in_set()` |
| `constraints.pattern` | `col_vals_regex()` |
| `primaryKey` | `col_vals_not_null()` + `rows_distinct()` |
| `foreignKeys` | ⚠ Warning (cross-table not yet supported) |

**Importing a standalone Table Schema:**

```{python}
inventory_schema = {
    "fields": [
        {"name": "item_id", "type": "integer", "constraints": {"required": True, "unique": True}},
        {"name": "name", "type": "string", "constraints": {"required": True}},
        {"name": "quantity", "type": "integer", "constraints": {"minimum": 0}},
        {"name": "warehouse", "type": "string", "constraints": {"enum": ["NYC", "LAX", "ORD"]}},
    ],
    "primaryKey": "item_id",
}

result = pb.import_contract(inventory_schema, format="frictionless")
print(result.summary())
```

Notice how the `primaryKey` field generates both a not-null check and a uniqueness check for
`item_id`. This is the correct semantic interpretation: a primary key must always be present and
must uniquely identify each row. The field-level `constraints.required` and `constraints.unique`
also contribute their own checks, so the adapter deduplicates where appropriate.

**Importing from a Data Package:**

Data Packages bundle multiple resources (tables) together. You can select which resource to import
by name or index:

```{python}
data_package = {
    "name": "ecommerce-data",
    "resources": [
        {
            "name": "customers",
            "path": "customers.csv",
            "schema": {
                "fields": [
                    {"name": "id", "type": "integer", "constraints": {"required": True}},
                    {"name": "email", "type": "string"},
                ],
            },
        },
        {
            "name": "orders",
            "path": "orders.csv",
            "schema": {
                "fields": [
                    {"name": "order_id", "type": "integer", "constraints": {"required": True}},
                    {"name": "amount", "type": "number", "constraints": {"minimum": 0}},
                ],
            },
        },
    ],
}

# Import a specific resource by name
orders_import = pb.import_contract(data_package, format="frictionless", resource="orders")
print(f"Columns: {[name for name, _ in orders_import.columns]}")
print(f"Constraints: {len(orders_import.constraints)}")
```

The `resource=` parameter accepts either a string (the resource name) or an integer (the resource
index). When omitted, the first resource in the package is used. This makes it straightforward to
work with multi-table data packages where each table has its own schema definition.

### dbt `schema.yml`

[dbt](https://www.getdbt.com/) (data build tool) is the standard for transformation workflows in
modern analytics stacks. dbt models declare column-level tests in `schema.yml` files, and Pointblank
can import these definitions directly.

**Constraint mapping:**

| dbt Feature | Pointblank Method |
|---|---|
| `data_type` | Schema dtype check |
| `not_null` test | `col_vals_not_null()` |
| `unique` test | `rows_distinct()` |
| `accepted_values` test | `col_vals_in_set()` |
| `relationships` test | Warning (cross-table not yet supported) |

Both the newer `data_tests` key (dbt v1.8+) and the legacy `tests` key are supported.

**Importing from a dict:**

```{python}
dbt_schema = {
    "version": 2,
    "models": [
        {
            "name": "orders",
            "description": "Cleaned order data",
            "columns": [
                {
                    "name": "order_id",
                    "data_type": "integer",
                    "data_tests": ["not_null", "unique"],
                },
                {
                    "name": "status",
                    "data_type": "string",
                    "data_tests": [
                        {"accepted_values": {"values": ["pending", "shipped", "delivered"]}}
                    ],
                },
                {
                    "name": "amount",
                    "data_type": "float",
                },
            ],
        }
    ],
}

result = pb.import_contract(dbt_schema, format="dbt")
print(result.summary())
```

**Importing from a file:**

```python
# File-based import (auto-detects .yml / .yaml with dbt structure)
result = pb.import_contract("models/schema.yml", format="dbt")
```

**Selecting a specific model:**

When a `schema.yml` file contains multiple models, use the `model=` parameter to pick one:

```{python}
multi_model = {
    "version": 2,
    "models": [
        {"name": "users", "columns": [{"name": "id", "data_tests": ["not_null"]}]},
        {"name": "orders", "columns": [{"name": "order_id", "data_tests": ["not_null", "unique"]}]},
    ],
}

result = pb.import_contract(multi_model, format="dbt", model="orders")
print(f"Imported model: {result.metadata.get('title')}")
print(f"Constraints: {len(result.constraints)}")
```

dbt sources are also supported. When a file contains `sources` with `tables`, each table is treated
as a model candidate:

```python
result = pb.import_contract("sources.yml", format="dbt", model="raw_events")
```

### ODCS (Open Data Contract Standard)

The [Open Data Contract Standard](https://github.com/bitol-io/open-data-contract-standard) (ODCS)
is a vendor-neutral format for defining data contracts between producers and consumers. Pointblank
supports both ODCS v2.x and v3.x documents.

**Constraint mapping:**

| ODCS Feature | Pointblank Method |
|---|---|
| `logicalType` | Schema dtype check |
| `isNullable: false` | `col_vals_not_null()` |
| `isUnique: true` | `rows_distinct()` |
| `isPrimaryKey: true` | `col_vals_not_null()` + `rows_distinct()` |
| `enum` / `values` | `col_vals_in_set()` |
| `pattern` | `col_vals_regex()` |
| `minimum` / `maximum` | `col_vals_ge()` / `col_vals_le()` |
| `minLength` / `maxLength` | Warning (no direct equivalent) |
| Custom checks / SodaCL | Warning (no automatic mapping) |

**Importing an ODCS v3 contract:**

```{python}
odcs_contract = {
    "kind": "DataContract",
    "apiVersion": "v3.0.0",
    "info": {
        "title": "Customer Data",
        "description": "Contract for customer records",
    },
    "dataset": [
        {
            "table": "customers",
            "columns": [
                {
                    "column": "customer_id",
                    "logicalType": "integer",
                    "isNullable": False,
                    "isUnique": True,
                },
                {
                    "column": "email",
                    "logicalType": "string",
                    "isNullable": False,
                    "pattern": r"^[^@]+@[^@]+\.[^@]+$",
                },
                {
                    "column": "tier",
                    "logicalType": "string",
                    "enum": ["free", "pro", "enterprise"],
                },
                {
                    "column": "lifetime_value",
                    "logicalType": "float",
                    "minimum": 0,
                },
            ],
        }
    ],
}

result = pb.import_contract(odcs_contract, format="odcs")
print(result.summary())
```

**Selecting a specific table:**

For contracts with multiple dataset tables, use `table=`:

```python
result = pb.import_contract("contract.odcs.yml", format="odcs", table="orders")
```

**Importing from a file:**

```python
# YAML files
result = pb.import_contract("contracts/customer_data.odcs.yml", format="odcs")

# JSON files
result = pb.import_contract("contracts/customer_data.odcs.json", format="odcs")
```

## Output Options

Once you have a `ContractImport`, you can use it in several ways depending on your workflow.

### Direct Validation with `.to_validate()`

The most common path is to get a `Validate` object, pass your data, and run it:

```{python}
schema = {
    "type": "object",
    "properties": {
        "temperature": {"type": "number", "minimum": -50, "maximum": 60},
        "humidity": {"type": "number", "minimum": 0, "maximum": 100},
        "station_id": {"type": "string"},
    },
    "required": ["temperature", "humidity", "station_id"],
}

weather_data = pl.DataFrame(
    {
        "temperature": [22.5, 18.3, -5.1, 35.0, 28.7],
        "humidity": [45.0, 78.2, 30.0, 92.5, 55.0],
        "station_id": ["WX-001", "WX-002", "WX-003", "WX-001", "WX-004"],
    }
)

imported = pb.import_contract(schema, format="json_schema")
imported.to_validate(data=weather_data).interrogate()
```

The `.to_validate()` method returns a fully configured `Validate` object with all imported
constraints already applied as validation steps. You get the familiar validation report showing
pass/fail counts for each check. Because the `Validate` object is not yet interrogated when
created, you also have the option of adding additional validation steps before calling
`.interrogate()`.

You can also pass additional arguments to the `Validate` constructor:

```python
imported.to_validate(
    data=weather_data,
    tbl_name="weather_readings",
    label="Daily sensor check",
    thresholds=pb.Thresholds(warning=0.05, error=0.10),
)
```

Any keyword argument accepted by the `Validate` class can be passed through here, including
`tbl_name`, `label`, `thresholds`, `owner`, and `consumers`. This gives you full control over
how the validation is configured without needing to modify the import result.

### Creating a Reusable Contract with `.to_contract()`

If you want to store the imported schema as a Pointblank `Contract` for use in pipelines or
repeated validation:

```{python}
imported = pb.import_contract(inventory_schema, format="frictionless")
contract = imported.to_contract(name="inventory_check", version="1.0.0", owner="warehouse-team")
print(contract)
```

The resulting `Contract` can be serialized to YAML, used in `Pipeline`, or shared with other teams.
This is particularly valuable when you want to maintain a stable contract definition that outlives
the original external schema file. The `Contract` object carries all the metadata (version, owner,
description) that makes it suitable for team workflows and CI/CD pipelines.

### Generating Python Code with `.to_python()`

When you want to see (or save) the equivalent Pointblank Python code that would be generated from
the import:

```{python}
imported = pb.import_contract(user_schema, format="json_schema")
print(imported.to_python())
```

The generated code is syntactically valid Python that you can copy directly into a script or
notebook. It uses the standard Pointblank method-chaining style, making it easy to read and modify.
This is especially useful for:

- understanding exactly what validation steps an import produces
- generating starter code that you can then customize
- documentation and onboarding (show teams what their schema "means" in validation terms)

Once you have the generated code, you can paste it into your project and modify it freely. Add
extra validation steps, remove checks that don't apply, or adjust parameter values. The generated
code has no dependency on the original schema file, so it serves as a clean handoff point between
the schema world and your Python codebase.

### Generating YAML with `.to_yaml()`

For workflows that use Pointblank's YAML-based validation:

```{python}
imported = pb.import_contract(user_schema, format="json_schema")
print(imported.to_yaml())
```

The YAML output follows Pointblank's validation YAML format, with each constraint appearing as a
separate step entry. You can save this output to a file and use it with `pb.yaml_interrogate()` or
`pb.validate_yaml()` for configuration-driven workflows where validation rules are managed as YAML
files rather than Python code.

## Exporting Contracts

The reverse operation (taking a Pointblank `Contract` or `Validate` object and writing it out in
an external format) is handled by [`export_contract()`](`pointblank.export_contract`):

```{python}
# Create a contract
contract = pb.Contract(
    name="sensor_data",
    schema=pb.Schema(temperature="Float64", humidity="Float64", station_id="String"),
    steps=[
        pb.Step("col_vals_ge", columns="temperature", value=-50),
        pb.Step("col_vals_le", columns="temperature", value=60),
        pb.Step("col_vals_ge", columns="humidity", value=0),
        pb.Step("col_vals_le", columns="humidity", value=100),
        pb.Step("col_vals_not_null", columns=["temperature", "humidity", "station_id"]),
    ],
)

# Export to JSON Schema
json_schema = pb.export_contract(contract, format="json_schema")
json_schema
```

```{python}
# Export to Frictionless Table Schema
table_schema = pb.export_contract(contract, format="frictionless")
table_schema
```

```{python}
# Export to dbt schema.yml
dbt_doc = pb.export_contract(contract, format="dbt")
dbt_doc
```

```{python}
# Export to ODCS
odcs_doc = pb.export_contract(contract, format="odcs")
odcs_doc
```

Each format produces the output structure that is native to that standard. JSON Schema export
creates a valid `$schema`-annotated document with `properties`, `type`, and `required` fields.
Frictionless export creates a Table Schema with `fields` and `constraints` entries. dbt export
creates a `version: 2` document with models and column-level `data_tests`. ODCS export creates a
`kind: DataContract` document with typed columns and constraint properties. All formats can be fed
directly into tools that consume those standards.

You can also write directly to a file:

```python
pb.export_contract(contract, "output/sensor_data.schema.json", format="json_schema")
pb.export_contract(contract, "output/sensor_data.resource.json", format="frictionless")
pb.export_contract(contract, "output/schema.yml", format="dbt")
pb.export_contract(contract, "output/sensor_data.odcs.yml", format="odcs")
```

When a `destination` path is provided, the output is written to that file (creating parent
directories as needed) and also returned from the function. This makes it convenient to both
persist the output and inspect it in the same call.

::: {.callout-tip}
## Shortcut for JSON Schema

If you're working with a `Validate` object and want JSON Schema output specifically, you can use the
[`to_json_schema()`](`Validate.to_json_schema`) convenience method directly on the `Validate`
object. It delegates to `export_contract()` and additionally enriches the schema with column type
information inferred from the data. See the
[Serializing Validation Plans](../02-advanced-validation/05-plan-serialization.qmd#exporting-a-plan-as-json-schema)
guide for details.
:::

## Round-Trip Fidelity

Importing a schema and then exporting it back should produce an equivalent result. This is important
for workflows where you maintain schemas in an external format but want to validate with Pointblank:

```{python}
# Start with a JSON Schema
original = {
    "type": "object",
    "properties": {
        "score": {"type": "integer", "minimum": 0, "maximum": 100},
        "grade": {"type": "string", "enum": ["A", "B", "C", "D", "F"]},
    },
    "required": ["score"],
}

# Import → Contract → Export
imported = pb.import_contract(original, format="json_schema")
contract = imported.to_contract(name="grades")
exported = pb.export_contract(contract, format="json_schema")

# The exported schema preserves the constraints
print(f"Original constraints on 'score': minimum={original['properties']['score']['minimum']}, "
      f"maximum={original['properties']['score']['maximum']}")
print(f"Exported constraints on 'score': minimum={exported['properties']['score'].get('minimum')}, "
      f"maximum={exported['properties']['score'].get('maximum')}")
```

Round-trip fidelity is tested as part of Pointblank's test suite. The general guarantee is that
any constraint that can be expressed in both Pointblank and the target format will survive the
round trip. Constraints that are unique to one format (like JSON Schema's `$ref` or Pointblank's
`pre=` argument) may not survive, but the core numeric bounds, enum checks, null checks, and
pattern constraints will always round-trip cleanly.

## Auto-Detection

When the format is obvious from the source content, you can omit the `format=` parameter:

```{python}
# JSON Schema: detected by presence of "$schema" or "type" + "properties"
result = pb.import_contract({"type": "object", "properties": {"x": {"type": "integer"}}})
print(f"Detected: {result.source_format}")

# Frictionless: detected by presence of "fields" list
result = pb.import_contract({"fields": [{"name": "x", "type": "integer"}]})
print(f"Detected: {result.source_format}")

# dbt: detected by "models" or "sources" key
result = pb.import_contract({"version": 2, "models": [{"name": "t", "columns": []}]})
print(f"Detected: {result.source_format}")

# ODCS: detected by "kind": "DataContract"
result = pb.import_contract({
    "kind": "DataContract", "apiVersion": "v3.0.0", "info": {"title": "x"},
    "dataset": [{"table": "t", "columns": []}],
})
print(f"Detected: {result.source_format}")
```

For file-based imports, the extension is also used for detection (`.schema.json` maps to JSON
Schema, `.resource.json` or `.datapackage.json` maps to Frictionless, `.odcs.yml` or `.odcs.json`
maps to ODCS). Note that `.yml` and `.yaml` extensions are shared between dbt and ODCS, so for YAML
files without a distinguishing extension, auto-detection falls through to content-based inspection.
When working with ambiguous files or dict inputs that could match multiple formats, it is best to
specify `format=` explicitly to avoid any possibility of misdetection.

## Combining Imports with Extra Checks

An imported schema gives you a baseline, but you can always add more Pointblank checks on top:

```{python}
imported = pb.import_contract(user_schema, format="json_schema")

# Start from the import but add custom checks
validation = (
    imported
    .to_validate(data=users, tbl_name="enriched_check")
    .col_vals_regex(columns="email", pattern=r".*\.(com|io|org|net|co)$")
    .rows_distinct(columns_subset="user_id")
    .interrogate()
)

validation
```

This pattern works well when the external schema covers structural and type constraints, but your
team has additional business rules that only make sense in the Pointblank context. The imported
constraints form the foundation, and your additional `.col_vals_*()` or `.rows_*()` calls layer on
top. Because `.to_validate()` returns a standard `Validate` object, you have full access to the
entire Pointblank API for adding checks, setting thresholds, or attaching actions.

## Migration from Other Tools

A key use case for `import_contract()` is **migration**: bringing existing validation definitions
from other tools into Pointblank without manual rewriting.

### Coming from JSON Schema

If your team uses JSON Schema for API validation and you want the same rules applied to DataFrames:

```python
# Your existing JSON Schema (maybe generated by your API framework)
result = pb.import_contract("api/schemas/user.schema.json")

# Now use it for DataFrame validation in your data pipeline
validation = result.to_validate(data=raw_users_df).interrogate()
```

This approach is particularly powerful when your API team already maintains JSON Schema definitions
for request/response validation. Those same schemas can now serve double duty: validating API
payloads at the service boundary and validating the resulting DataFrames in your analytics pipeline.
You get consistent enforcement across both layers without writing the rules twice.

### Coming from Frictionless

If you have data packages from open data sources or research datasets:

```python
# Import from an existing data package descriptor
result = pb.import_contract("data/datapackage.json", resource="observations")

# Validate the actual CSV data against the declared schema
validation = result.to_validate(data=observations_df).interrogate()
```

Frictionless Data Packages are common in open data portals, government datasets, and academic
research repositories. By importing their Table Schemas directly, you can validate downloaded data
against its declared structure without needing to manually inspect the descriptor file and rewrite
each constraint. This is especially valuable when working with unfamiliar datasets where the schema
descriptor is your primary documentation of what the data should contain.

### Coming from dbt

If your analytics team maintains dbt models with schema tests:

```python
# Import the tests from your dbt schema.yml
result = pb.import_contract("models/schema.yml", format="dbt", model="orders")

# Now validate source data before it reaches dbt
validation = result.to_validate(data=raw_orders_df).interrogate()
```

This is really valuable for "shift-left" validation: catching data quality issues in raw source data
before it enters the dbt transformation pipeline. Your dbt schema tests define what the transformed
data should look like. Importing those same rules into Pointblank lets you apply them at the
ingestion boundary.

### Coming from ODCS

If your organization uses the Open Data Contract Standard for cross-team data governance:

```python
# Import a data contract
result = pb.import_contract("contracts/customer_data.odcs.yml", format="odcs")

# Validate data against the contract
validation = result.to_validate(data=customer_df).interrogate()
```

ODCS contracts are increasingly used in data mesh architectures where domain teams publish contracts
for their data products. Importing these contracts into Pointblank lets consumers validate incoming
data against the producer's declared contract, without needing to manually rewrite the constraints.

### Generating a Starting Point

Even if you don't plan to keep using the external format, importing is a great way to bootstrap
a Pointblank contract:

```python
# Import from your existing schema
imported = pb.import_contract("legacy_schema.json", format="json_schema")

# Save as a YAML contract you'll maintain going forward
contract = imported.to_contract(name="my_table", version="1.0.0")
contract.to_yaml("contracts/my_table.yaml")
```

Now you have a Pointblank-native contract that you can extend and evolve independently of the
original source. You can add new validation steps, adjust thresholds, or incorporate business rules
that go beyond what the original schema format could express.

## Conclusion

The contract import/export system lets you bridge the gap between external schema definitions and
Pointblank's validation engine. Rather than maintaining duplicate specifications across tools, you
can keep your source of truth in whichever format suits your team and import it into Pointblank
whenever you need runtime validation. The key points to remember:

- Use `pb.import_contract()` to read external schemas and translate them into Pointblank checks
- The `ContractImport` object gives you multiple output options: direct validation, reusable
  contracts, generated Python code, or YAML
- Check `.coverage` and `.warnings` to understand how completely the translation covered your
  original schema
- Use `pb.export_contract()` to write Pointblank contracts back to external formats for sharing
  with other tools
- Combine imports with additional Pointblank-specific checks for the most thorough validation
  coverage

Whether you are migrating from another validation tool, bootstrapping contracts from existing
schemas, or maintaining interoperability with external systems, the adapter framework gives you a
clean path between external specifications and Pointblank's validation engine. The four built-in
adapters (JSON Schema, Frictionless, dbt, and ODCS) cover the most common schema formats in the data
ecosystem. And if you need to support a proprietary format, the custom adapter system (covered in
the next section) makes it straightforward to extend the framework.


### Custom Adapters

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Pointblank's contract import/export system is designed to be extensible. If your organization uses
a proprietary schema format, an internal data catalog, or any other schema definition tool that
isn't covered by the built-in adapters, you can write a **custom adapter** and register it with
the framework.

Once registered, your custom adapter works seamlessly with
[`import_contract()`](`pointblank.import_contract`) and
[`export_contract()`](`pointblank.export_contract`), the same API surface your team already uses for
JSON Schema, Frictionless, dbt, and ODCS.

## The Adapter Architecture

Every adapter follows the same pattern:

1. **Subclass** `ContractAdapter` and set a few class attributes
2. **Implement** `detect()` (for auto-detection),
   [`import_contract()`](`ContractAdapter.import_contract`), and optionally
   [`export_contract()`](`ContractAdapter.export_contract`)
3. **Register** the adapter with the `@register_adapter` decorator

Here's a minimal example to illustrate the structure:

```{python}
from pointblank.adapters import ContractAdapter, ContractImport, MappedConstraint, register_adapter


@register_adapter("my_format")
class MyFormatAdapter(ContractAdapter):
    """Adapter for My Company's internal schema format."""

    format_name = "my_format"
    file_extensions = [".myschema"]
    supports_import = True
    supports_export = False  # export not implemented yet

    @staticmethod
    def detect(source) -> bool:
        """Return True if this adapter can handle the source."""
        if isinstance(source, dict):
            return "my_format_version" in source
        return False

    def import_contract(self, source, **kwargs) -> ContractImport:
        """Parse the source and return a ContractImport."""
        # Your parsing logic here
        columns = [("id", "Int64"), ("value", "Float64")]
        constraints = [
            MappedConstraint(
                method="col_vals_not_null",
                kwargs={"columns": "id"},
                source_description="id is required",
            ),
        ]
        return ContractImport(
            source_format="my_format",
            columns=columns,
            constraints=constraints,
        )
```

After registration, it's immediately usable:

```{python}
# Now this works
result = pb.import_contract({"my_format_version": "1.0", "fields": []}, format="my_format")
print(result)
```

The adapter is now part of the Pointblank ecosystem. Any call to `import_contract()` with
`format="my_format"` will route through this adapter, and the auto-detection system will call
`detect()` when no format is specified.

```{python}
# And it shows up in the adapter list
pb.list_adapters()
```

The `list_adapters()` output confirms your adapter is registered alongside the built-in ones,
showing its supported file extensions and whether it handles import, export, or both.

## The `ContractAdapter` Base Class

Here are the class attributes and methods you can define:

| Attribute | Type | Purpose |
|---|---|---|
| `format_name` | `str` | Short identifier (e.g., `"json_schema"`, `"my_format"`) |
| `file_extensions` | `list[str]` | File extensions for auto-detection (e.g., `[".schema.json"]`) |
| `supports_import` | `bool` | Whether `import_contract()` is implemented |
| `supports_export` | `bool` | Whether `export_contract()` is implemented |

| Method | Required? | Purpose |
|---|---|---|
| `detect(source)` | Recommended | Returns `True` if this adapter handles the given source |
| `import_contract(source, **kwargs)` | If `supports_import` | Parses source, returns `ContractImport` |
| `export_contract(obj, destination, **kwargs)` | If `supports_export` | Exports to the format |

## Building an Import Adapter

Let's build a more realistic adapter, one that reads a simple YAML-based schema format used
internally at a hypothetical company:

```yaml
# company_schema.yaml
version: "2.0"
table: user_events
columns:
  - name: event_id
    type: string
    required: true
    unique: true
  - name: user_id
    type: integer
    required: true
  - name: event_type
    type: string
    values: [click, view, purchase, signup]
  - name: amount
    type: float
    min: 0
```

Here's the adapter that handles this format:

```{python}
import yaml
from pointblank.adapters import ContractAdapter, ContractImport, MappedConstraint, register_adapter


@register_adapter("company_schema")
class CompanySchemaAdapter(ContractAdapter):
    """Adapter for our company's internal YAML schema format."""

    format_name = "company_schema"
    file_extensions = [".company.yaml", ".company.yml"]
    supports_import = True
    supports_export = False

    # Type mapping from our format to Pointblank dtypes
    TYPE_MAP = {
        "string": "String",
        "integer": "Int64",
        "float": "Float64",
        "boolean": "Boolean",
        "date": "Date",
        "datetime": "Datetime",
    }

    @staticmethod
    def detect(source) -> bool:
        """Detect our format by looking for the 'version' + 'columns' keys."""
        if isinstance(source, dict):
            return "version" in source and "columns" in source and "table" in source
        return False

    def import_contract(self, source, **kwargs) -> ContractImport:
        """Import from our company schema format."""
        # Load from file or use dict directly
        if isinstance(source, str):
            from pathlib import Path

            with open(Path(source)) as f:
                doc = yaml.safe_load(f)
        elif isinstance(source, dict):
            doc = source
        else:
            raise TypeError(f"Expected str or dict, got {type(source).__name__}")

        columns = []
        constraints = []
        warnings = []
        total = 0

        for col_def in doc.get("columns", []):
            col_name = col_def["name"]
            col_type = col_def.get("type", "string")
            dtype = self.TYPE_MAP.get(col_type)
            columns.append((col_name, dtype))

            if col_def.get("required", False):
                total += 1
                constraints.append(
                    MappedConstraint(
                        method="col_vals_not_null",
                        kwargs={"columns": col_name},
                        source_description=f"{col_name} is required",
                    )
                )

            if col_def.get("unique", False):
                total += 1
                constraints.append(
                    MappedConstraint(
                        method="rows_distinct",
                        kwargs={"columns_subset": col_name},
                        source_description=f"{col_name} must be unique",
                    )
                )

            if "values" in col_def:
                total += 1
                constraints.append(
                    MappedConstraint(
                        method="col_vals_in_set",
                        kwargs={"columns": col_name, "set": col_def["values"]},
                        source_description=f"{col_name} allowed values: {col_def['values']}",
                    )
                )

            if "min" in col_def:
                total += 1
                constraints.append(
                    MappedConstraint(
                        method="col_vals_ge",
                        kwargs={"columns": col_name, "value": col_def["min"]},
                        source_description=f"{col_name} >= {col_def['min']}",
                    )
                )

            if "max" in col_def:
                total += 1
                constraints.append(
                    MappedConstraint(
                        method="col_vals_le",
                        kwargs={"columns": col_name, "value": col_def["max"]},
                        source_description=f"{col_name} <= {col_def['max']}",
                    )
                )

        coverage = 1.0 if total == 0 else (total - len(warnings)) / total

        return ContractImport(
            source_format="company_schema",
            source_path=source if isinstance(source, str) else None,
            source_version=doc.get("version"),
            columns=columns,
            constraints=constraints,
            metadata={"table": doc.get("table")},
            warnings=warnings,
            coverage=coverage,
        )
```

Now let's use it:

```{python}
import polars as pl

# Simulate a company schema document
company_schema = {
    "version": "2.0",
    "table": "user_events",
    "columns": [
        {"name": "event_id", "type": "string", "required": True, "unique": True},
        {"name": "user_id", "type": "integer", "required": True},
        {"name": "event_type", "type": "string", "values": ["click", "view", "purchase", "signup"]},
        {"name": "amount", "type": "float", "min": 0},
    ],
}

# Import using our custom adapter
result = pb.import_contract(company_schema, format="company_schema")
print(result.summary())
```

The summary shows four columns detected and five constraints mapped (two `required` fields, one
`unique` field, one `values` check, and one `min` bound). All constraints were successfully
translated, giving 100% coverage.

```{python}
# Validate some data
events = pl.DataFrame(
    {
        "event_id": ["E001", "E002", "E003", "E004", "E005"],
        "user_id": [101, 102, 101, 103, 104],
        "event_type": ["click", "view", "purchase", "signup", "click"],
        "amount": [0.0, 0.0, 49.99, 0.0, 0.0],
    }
)

result.to_validate(data=events).interrogate()
```

The validation report shows each imported constraint as a separate step, just as if you had written
the validation by hand. From the user's perspective, there is no difference between validation steps
that came from a custom adapter and those written directly in Python.

## The `MappedConstraint` Class

Each constraint from the external format gets mapped to a `MappedConstraint`, which is a simple
data container holding:

- `method`: the Pointblank `Validate` method name (e.g., `"col_vals_gt"`)
- `kwargs`: the keyword arguments to pass to that method
- `source_description`: optional human-readable note about what this was in the source format

```{python}
# Creating constraints manually
c1 = MappedConstraint(
    method="col_vals_between",
    kwargs={"columns": "temperature", "left": -40, "right": 60},
    source_description="Temperature must be in physical range",
)
print(c1)
```

The `source_description` is stored for debugging and documentation but doesn't affect validation.
When users call `.summary()` or inspect the `ContractImport` object, these descriptions help them
understand the provenance of each validation step. This is especially useful when debugging why a
particular check was generated or when comparing the import output against the original schema.

## Handling Unmappable Constraints

Not every constraint in every format has a clean Pointblank equivalent. When you encounter something
that can't be translated, add it to the warnings list rather than silently dropping it:

```python
# In your import_contract() method:
if "custom_check" in col_def:
    total += 1
    warnings.append(
        f"Column '{col_name}': 'custom_check' has no Pointblank equivalent, skipped."
    )
```

This follows Pointblank's design principle of **best-effort translation**: generate everything you
can, be transparent about what was skipped, and never silently lose information. Users can then
review the warnings list and decide whether to add manual validation steps for the missing
constraints or whether the gap is acceptable for their use case.

## Auto-Detection Tips

The `detect()` method enables format auto-detection. Good detection should be:

- **Fast**: don't load the entire file just to check if it's your format
- **Specific**: avoid false positives that could conflict with other adapters
- **Graceful**: return `False` (never raise) if the source isn't your format

The detection system iterates through all registered adapters and calls `detect()` on each one.
Because of this, your detection logic should be as lightweight as possible. Checking for the
presence of a few distinctive keys in a dict is ideal. Avoid expensive operations like parsing
large files or making network requests inside `detect()`.

```python
@staticmethod
def detect(source) -> bool:
    if isinstance(source, dict):
        # Check for a distinctive key combination
        return "my_format_version" in source and "tables" in source

    if isinstance(source, str):
        # Check file extension first (cheapest check)
        return source.lower().endswith(".myformat.yaml")

    return False
```

## Best Practices

1. **Map as much as possible**: users expect high coverage. If a constraint is *close* to
   something Pointblank supports, map it (possibly with reduced precision) rather than skipping it.

2. **Use descriptive source_description**: this helps users understand what each generated
   validation step corresponds to in their original schema.

3. **Set coverage accurately**: track the total number of source constraints and how many were
   successfully mapped. This gives users confidence in the import quality.

4. **Handle both file paths and dicts**: users should be able to pass either a path string or
   pre-loaded data. Most adapters check `isinstance(source, str)` for file paths and
   `isinstance(source, dict)` for pre-parsed content.

5. **Fail clearly on bad input**: raise `TypeError` for wrong source types, `FileNotFoundError`
   for missing files, and `ValueError` for malformed content. Don't return partial results
   silently.

6. **Keep dependencies optional**: if your adapter needs a third-party library, check for it at
   import time and give a clear installation hint if it's missing.

## Conclusion

Custom adapters let you extend Pointblank's import/export system to handle any schema format your
organization uses. The plugin architecture is intentionally simple: subclass `ContractAdapter`,
implement one or two methods, and register it with a decorator. From that point forward, your
format participates in the same `import_contract()` and `export_contract()` workflow that the
built-in adapters use.

This extensibility means that Pointblank can serve as a universal validation layer regardless of
where your data contracts originate. Whether your schemas live in a proprietary YAML format, an
internal data catalog API, or a custom metadata store, a short adapter class is all you need to
bring them into the Pointblank ecosystem and benefit from its validation reporting, threshold
system, and pipeline integration.


### Data Inspection

Pointblank’s CLI (`pb`) makes it easy to view your data before running validations. It has several
commands that are exceedingly useful for understanding your data’s structure, checking for obvious
issues, and confirming that your data source is being read correctly. We also make it easy to
explore data in various formats and locations. Let's go through each of the commands for inspecting
and exploring data.

## `pb info`: Inspecting the Data Structure

Use `pb info` to display basic information about your data source. Here's how this works with a
local CSV file:

```bash
pb info worldcities.csv
```

![](/assets/pb-info-worldcities-csv.png){width=100%}

This command shows the (1) table type (e.g., `pandas`, `polars`, etc.), (2) the number of rows and
columns, and (3) the data source path or identifier.

That example used a local CSV file. The same file is also present in Pointblank's GitHub repository
(in the `data-raw` directory) and the CLI is able to load the data from there as well:

```bash
pb info https://github.com/posit-dev/pointblank/blob/main/data_raw/worldcities.csv
```

![](/assets/pb-info-worldcities-github-csv.png){width=100%}

The `pb info` command is useful before running validations to confirm your data source's dimensions,
and, whether it can even be loaded.

::: {.callout-info}

You can inspect a wide variety of data sources using the CLI! Here are some examples with `pb info`:

```bash
pb info small_table         # built in dataset
pb info worldcities.csv     # single CSV file
pb info meteo.parquet       # single Parquet file
pb info "*.parquet"         # several Parquet files
pb info "data/*.parquet"    # partitioned Parquet files
pb info "duckdb:///warehouse/analytics.ddb::customer_metrics" # DB table via connection string
pb info https://github.com/posit-dev/pointblank/blob/main/data_raw/global_sales.csv # GitHub URL
```

And these input schemes work with all other commands that accept a `DATA_SOURCE`.
:::

## `pb preview`: Previewing Data

Use `pb preview` to view the first and last rows of your data. Let's try it out with the
`worldcities.csv` file:

```bash
pb preview worldcities.csv
```

![](/assets/pb-preview-worldcities-csv.png){width=100%}

As can be seen, `pb preview` gives you a preview of the dataset as a table in the console. The
dataset has 41K rows but we're electing to show only five rows from the head and from the tail.

Let's go over some features of the table preview. First off, the table header provides information
on the data source and the DataFrame library that handled the reading of the CSV. Below the column
names are simplified representations of the data types (e.g., `<obj>` for `object`, `<f64>` for
`Float64`). We provide row numbers (in gray) in the table stub to indicate which of the rows are
from the head or the tail (and a divider helps to distinguish these row groups). If you'd prefer
to eliminate the row numbers, use the `--no-row-numbers` option:

```bash
pb preview worldcities.csv --no-row-numbers
```

![](/assets/pb-preview-worldcities-csv-no-row-numbers.png){width=100%}

While `pb preview` purposefully displays only a few rows, the number of columns shown can be more
than you might need. Furthermore, if a table has *a lot* of columns, you'll only see some of the
first and some of the last columns. This is where column selection becomes useful and there are a
few methods available for subsetting the preview table's columns. A good one (provided you know the
column names) is to use the `--columns` option along with a comma-delimted set of column names.
Let's look at a preview of the included `game_revenue` dataset before subsetting the columns:

```bash
pb preview game_revenue
```

![](/assets/pb-preview-game_revenue-all-columns.png){width=100%}

That's 11 columns in total and while the all columns *are* shown (i.e., none in the middle are
truncated from view), we start to see some necessary instances of abbreviating via `…` within the
column names and in the displayed values.

Let's now use the `--columns` with a set of column names:

```bash
pb preview game_revenue -columns "player_id, item_type, item_name, start_day"
```

![](/assets/pb-preview-game_revenue-column-names.png){width=100%}

With that, the few columns that are displayed no longer have to abbreviate their data values. This
is an important consideration since a selective display of column becomes more necessary if column
content is large or if the width of the terminal (in terms of characters) cannot be increased.

You may want to view ranges of columns by their indices. This is convenient when you want to get a
closer look at a few side-by-side columns and you don't want to bother with getting the set of
column names exactly right (i.e., for quick inspection). For this, we need to use the `--col-range`
option with the desired left/right column bounds separated by a colon:

```bash
pb preview game_revenue —-col-range "3:6"
```

![](/assets/pb-preview-game_revenue-column-range.png){width=100%}

In the case that you want to save a table preview as an HTML table in a standalone file, you can add
in the `--output-html` option (just add a path/filename with an .html extension).

And there are many more options that allow for quick iteration while previewing a table. Use
`pb preview --help` to get a helpful listing.

## `pb scan`: Getting Column Summaries

We can use `pb scan` for fairly comprehensive summaries of column data, including:

- data types
- missing value counts
- unique value counts
- summary statistics (mean, standard deviation, min, max, quartiles, and the interquartile range)

Let's use this on the `worldcities.csv` dataset:

```bash
pb scan worldcities.csv
```

![](/assets/pb-scan-worldcities-csv.png){width=100%}

Each row in the summary table represents a column in the input dataset. Just as in `pb preview` we
get simplified dtypes (in the `Type` column). The `NA` and `UQ` indicate how many missing and unique
values are in the column. The remaining columns are statistical measures and there's an important
thing to note here: the values provided for any string-based columns (here, `city_name` and
`country`) are derived from string lengths.

When using `pb scan`, it's helpful to know that large numbers in the summary table are automatically
abbreviated for readability, so you'll see values like `39.8k` or `38.0M` instead of long numbers
that would require many more characters. For the best experience, try to use a terminal window
that's at least 150 characters wide. This will help ensure that all column values are fully visible
and not adversely abbreviated by the underlying table mechanism.

If your table has many columns, that's not much of a problem for the reporting! Each column is
represented as a row in the report, so you'll simply see more lines in the output (and you could
always limit the number of columns reported).

There are two options for `pb scan`:

- `--columns "col1,col2"`: scan only specified columns
- `--output-html "file.html"`: save scan as an HTML file

Both of these options are also in the `pb preview` command and they behave the same way here.

## `pb missing`: Reporting on Missing Values

Use `pb missing` to generate a missing values report, visualizing missingness across columns and 10
*row sectors*. Here's an example using `worldcities.csv`:

```bash
pb missing worldcities.csv
```

![](/assets/pb-missing-worldcities-csv.png){width=100%}

This report is arranged similarly to that of `pb scan`, where each column in the input table gets a
row in this report table. Each of the 10 row sectors represents 1/10 of the rows in the dataset,
where sector `1` encompasses the head of the table, and `10` the tail.

More often than not, we expect few missing values so a filled green circle signifies that the
collection of rows in a sector (for a column) has no missing values. We don't see any red circles in
the `worldcities.csv`-based example but, if we did, that would mean that sectors for a given column
are entirely filled with missing values.

What's in between the no-missing and completely-missing cases are percentages of missing values. For
instance, we can see that row sector `3` of the `population` column has 18% missing values (which is
very odd for a table with the sole purpose of providing population values).

We also have cases where we see <1% of values in a row sector missing. The reporting of `pb missing`
is very careful not to 'round down' in cases where there could be very few missing values (or even
just one) in a large table.

Seeing this type of missing value report can be really important! You might not expect *any* missing
values but finding them will inform decisions on whether to institute checks for them. Another case
is that missing values will pop up in specific sectors, indicating a change in how data is processed
and appended to the table.

By way of options, there's only one for `pb missing` and it is `--output-html`. With that (as in the
previous two commands discussed), we can write the missing values report to a standalone HTML file.

## Wrapping Up

Pointblank’s CLI provides a set of commands that make it easy to inspect, understand, and diagnose
your data before you move on to validation or analysis. Using these tools can help you catch issues
early and gain confidence in your data sources.

- use `pb info` and before running validations to confirm your data source can be loaded
- use `pb preview` to quickly understand what the data looks like
- use `pb scan` for a quick data profile and to spot outliers or data quality issues
- use `pb missing` to visualize and diagnose missing data patterns

By incorporating these commands into your workflow, you’ll be better equipped to work efficiently
with your data (and avoid surprises down the line).


### Data Validation

Validating data directly in the terminal with the Pointblank CLI offers a fast, scriptable, and
repeatable way to check your data. This approach is especially useful for quick checks, CI/CD
pipelines, and automation workflows, where you want immediate feedback and clear pass/fail results.

The CLI commands are designed for efficiency: you can run validations with a single line, integrate
them easily into shell scripts or data pipelines, and benefit from clear, color-coded output that’s
easy to interpret at a glance.

The `pb validate` command lets you perform common validation checks directly on your data source
with a simple command-line interface. This works well both for quick, one-off checks and for use in
automated pipelines.

For more complex validation logic, the `pb run` command serves as a runner for validation scripts
written with the Pointblank Python API, allowing you to execute custom validation workflows from the
command line.

## `pb validate`: Quick, One-Line Data Checks

The `pb validate` command is your go-to for running common validation checks directly on your data
source. It’s great for quick, one-off checks or for use in automated pipelines. You specify
exactly which check you want to run using the `--check` option, making your intent clear and your
validation explicit.

Here’s how you construct a validation command:

```bash
pb validate worldcities.csv --check <check-name> [other options]
```

You always provide the data source first, then specify one or more checks with `--check`. Each check
can have its own options, such as `--column` or `--value`, depending on what you want to validate.

### Checking for Duplicate and Complete Rows

To check for duplicate rows, use the `rows-distinct` check:

```bash
pb validate worldcities.csv --check rows-distinct
```

![](/assets/pb-validate-rows-distinct-worldcities-csv.png){width=100%}

The output shows you whether your data contains any duplicate rows, how many rows were checked, and
if any duplicates were found. The color-coding of the results helps you quickly interpret the
results, using green for pass and red for fail. Here, no duplicate rows were detected out of the 41K
rows checked.

To check that every row is complete (i.e., no missing values in any column), use the `rows-complete`
check:

```bash
pb validate worldcities.csv --check rows-complete
```

![](/assets/pb-validate-rows-complete-worldcities-csv.png){width=100%}

With this check we see that the `worldcities.csv` dataset has 739 rows containing at least one
Null/missing value. And with any dataset, it's easy to quickly spot if there are any rows with
missing data using this command.

### Checking for Nulls and Value Ranges

You can easily check for missing values in a column, or ensure that values fall within a certain
range. Here’s how to check that all values in the `population` column are not null:

```bash
pb validate worldcities.csv --check col-vals-not-null --column city_name
```

![](/assets/pb-validate-worldcities-not-null-city_name.png){width=100%}

Perhaps surprisingly, we find that one row has a missing city name.

Let's now check whether all values in the `population` column are greater than zero:

```bash
pb validate worldcities.csv --check col-vals-gt --column population --value 0
```

![](/assets/pb-validate-worldcities-gt-0-population.png){width=100%}

With that we find that there are 741 rows where the `population` value is not greater than 0 (note
that this check also fails when cells are null or missing).

### Multiple Checks in One Command

You can chain several checks together in a single command. This is handy for comprehensive data
quality checks:

```bash
pb validate worldcities.csv --check rows-distinct --check col-vals-not-null --column city_name --check col-vals-gt --column population --value 0
```

![](/assets/pb-validate-multi-check.png){width=100%}

Each check is shown one after the other in the terminal output, so you can review the result of each
validation step individually as the command proceeds.

### Seeing and Saving Failing Rows

If a check fails, you might want to see which rows caused the failure. Use the `--show-extract`
option to display failing rows right in the terminal:

```bash
pb validate worldcities.csv --check rows-complete --show-extract
```

![](/assets/pb-validate-show-extract.png){width=100%}

Or, save the failing rows to a CSV file for further investigation:

```bash
pb validate worldcities.csv --check rows-complete --show-extract --write-extract incomplete_failing_rows
```

![](/assets/pb-validate-write-extract.png){width=100%}

Note here in the output the additional lines stating that failing rows were saved to a folder
(`incomplete_failing_rows`) and, within that folder the `step_01_rows_complete.csv` file was
written. Using a folder for extracts is necessary in practice since there may be multiple
validations defined in a `pb validate` command.

### Advanced Options and CI/CD Integration

- use `--exit-code` to make the command exit with a non-zero code if any check fails; useful for
CI/CD pipelines
- use `--limit` to control how many failing rows are shown or saved
- use `--list-checks` to see all available validation checks and their options

```bash
pb validate worldcities.csv --check col-vals-not-null --column city_name --exit-code
```

![](/assets/pb-validate-exit-code.png){width=100%}

## `pb run`: Custom Validation Workflows with Python

For more complex validation logic, use the `pb run` command. This lets you execute a Python script
containing Pointblank validation steps, combining the flexibility of the Python API with the
convenience of the CLI.

You can always scaffold a template script using the `pb make-template` command:

```bash
pb make-template my_validation.py
```

![](/assets/pb-make-template.png){width=100%}

But for our example, we'll elect to make our own `worldcities_validation.py` file from scratch. It
will:

- use the `worldcities.csv` file
- apply two thresholds (one for 'warning', another for 'error')
- have six validation steps

Here's what it looks like:

```python
import pointblank as pb

validation = (
    pb.Validate(
        data="worldcities.csv",
        thresholds=pb.Thresholds(
            warning=1,  # 1 failure
            error=0.05,  # 5% of rows failing
        ),
    )
    .col_schema_match(
        schema=pb.Schema(
            columns=[
                ("city_name", "object"),
                ("latitude", "float64"),
                ("longitude", "float64"),
                ("country", "object"),
                ("population", "float64"),
            ]
        ),
    )
    .col_vals_not_null(columns="city_name")
    .col_vals_not_null(columns="population")
    .col_vals_gt(columns="population", value=0, na_pass=True)
    .col_vals_between(columns="latitude", left=-90, right=90)
    .col_vals_between(columns="longitude", left=-180, right=180)
    .interrogate()
)
```

Now, we'll run the .py script from the terminal:

```bash
pb run worldcities_validation.py
```

![](/assets/pb-run-worldcities_validation.png){width=100%}

You’ll see a summary table that lists all of the steps and their results and you can include as many
steps and as much logic as you need.

### Output Options

You could save the validation report as HTML or JSON (or both) for the purposes of sharing or
for automation:

```bash
pb run worldcities_validation.py --output-html report.html --output-json report.json
```

![](/assets/pb-run-worldcities_validation-output.png){width=100%}

There are also the options to produce extracts (subset of failing rows) with `--show-extract` or
`--write-extract` (just like with `pb validate`). Let's do both in the following example:

```bash
pb run worldcities_validation.py --show-extract --write-extract worldcities_failures
```

![](/assets/pb-run-worldcities_validation-extracts.png){width=100%}

This shows a preview of each extract for those validation steps where extracts were produced (steps
2, 3, and 4). Individual CSV files with extracted rows for those steps were written to the
`worldcities_failures` directory.

### Controlling Failure Behavior

It's possible to use the `--fail-on` option to control when the command should exit with an error,
based on the severity of validation failures. This is especially useful for automated workflows and
CI/CD pipelines.

Let's try that with our `worldcities_validation.py` validation, which we've seen exeeds the
'warning' in steps 2, 3, and 4:

```bash
pb run worldcities_validation.py --fail-on warning
```

![](/assets/pb-run-worldcities_validation-fail-on-warning.png){width=100%}

Notice the final line states `Exiting with error due to warning, error, or critical validation
failures`. Because we applied `--fail-on warning`, any presence of `warning' (or higher levels such
as 'error' or 'critical') will yield a non-zero exit code that should stop a pipeline process. We
can prove this by running the following lines in the terminal

```bash
pb run worldcities_validation.py --fail-on warning > /dev/null 2>&1
echo $?
```

which returns `1`.

Using `--fail-on any` is stricter still: it exits with an error when any step has failing test
units, or when a step couldn't be evaluated at all (say, because its column isn't in the table).
Such steps aren't counted as passing in the report header, which gives them their own `Not
evaluated` count (inactive steps get an `Inactive` count) and lists the reason for each one below
the table.

## Wrapping Up

Pointblank’s CLI gives you powerful tools for validating your data, whether you need a quick check
or a custom workflow. Use `pb validate` for fast, one-liner checks and `pb run` for more advanced,
scriptable validation logic. With clear output and flexible options, you can catch data issues early
and keep your workflows running smoothly.


### MCP Server

Transform your data validation workflow with conversational AI in VS Code or Positron IDE. Here are three simple steps to start validating data through conversation (and no complex configuration required).

### 1. Install

```bash
pip install pointblank[mcp]
```

### 2. Configure Your IDE

**For VS Code**:

**Option 1: Workspace Configuration (Recommended for teams)**

1. Create a `.vscode/mcp.json` file in your project folder
2. Add this configuration:

```json
{
  "servers": {
    "pointblank": {
      "command": "python",
      "args": ["-m", "pointblank.mcp"]
    }
  }
}
```

**Option 2: User Configuration (Personal use)**

1. Run command: `MCP: Open User Configuration` (Cmd/Ctrl + Shift + P)
2. Add the same JSON configuration above

> ⚠️ **Security Note**: Only add MCP servers from trusted sources. VS Code will ask you to confirm trust when starting the server for the first time.

**For Positron**:

1. Open Positron Settings
2. Navigate to MCP Server configuration
3. Add the configuration (format may vary)

> **Note**: If you don't see MCP settings, you may need to install an MCP extension first. Search for "MCP" in the Extensions marketplace.

### 3. Start Chatting

```
"Load my sales data and check its quality"
```

That's basically how you get started.

## Essential Commands

Master these five command patterns and you'll be able to handle most data validation scenarios. Think of these as your fundamental vocabulary for talking to Pointblank.

### Load Data

```
"Load the file /path/to/data.csv"
"Load my Netflix dataset from the working directory"
"Load the CSV file with sales metrics"
"Load customer_data.csv as my main dataset"
```

### Explore Data

```
"Analyze the data for netflix_data"
"Show me a preview of the loaded data"
"Create a column summary table"
"Generate a missing values analysis"
```

**What you'll get**: Comprehensive data profiling with statistics including missing values, data types, distributions, and summary statistics for each column. The preview and summary tables are automatically generated as beautiful HTML files that open in your browser. This gives you a complete picture of your dataset's structure and characteristics before you define quality rules.

### Check Quality

```
"Create a validator for netflix_data"
"Add validation that ratings are between 0 and 10"
"Check that all release years are reasonable"
"Apply the basic_quality template"
```

**What you'll get**: Actual data quality validation that checks your data against business rules and domain knowledge. This tells you if your data meets your specific quality requirements and identifies rows that fail validation criteria.

### Create Data Validations

```
"Add validation that show_id values are unique"
"Check that cast field is not empty for movies"
"Ensure vote_count is greater than 0"
"Validate that country field follows ISO format"
```

**What you'll get**: Individual validation rules added to your validator. Each rule tests a specific business requirement and can be customized with thresholds and actions.

### Run and Export

```
"Run the validation and show results"
"Export validation failures to CSV"
"Get failed rows for step 2"
"Save the validation report"
```

---

## Top Three Use Cases

Once you have Pointblank set up, these scenarios show the most common and powerful ways to use natural language data validation. Each example demonstrates the conversational flow and immediate value you'll experience.

### 1. 📊 Quick Data Check

**When**: You get a new dataset

```
👤 "I just got netflix_data.csv. Is it clean?"
🤖 "Loading dataset... Found 16,000 movies/shows with 18 columns.
    Quality analysis complete:
    - ✅ No duplicate show IDs
    - ⚠️ 466 missing countries (2.9%)
    - ⚠️ 132 missing directors (0.8%)
    Opening data preview in your browser..."
```

### 2. 🔄 Daily Validation

**When**: Regular data quality checks

```
👤 "Apply the basic_quality template to my data"
🤖 "Creating validator with basic quality checks...
    ✅ Data types validated
    ✅ Missing value thresholds checked
    ✅ Duplicate detection passed
    Validation complete! All checks passed."
```

### 3. 🚨 Issue Investigation

**When**: Something looks wrong

```
👤 "Show me movies with ratings above 9.5"
🤖 "Creating validator for high ratings...
    Found 47 entries with ratings > 9.5
    Exporting suspicious rows to CSV
    Most are documentaries - could be valid!"
```

These examples show how natural conversation can quickly identify and resolve data quality issues that might take hours to diagnose manually.

## Core Capabilities

Pointblank's MCP server provides powerful tools for comprehensive data validation with beautiful, interactive HTML reports:

### Data Exploration

- **Interactive HTML previews** with automatic browser opening showing head/tail rows
- **Column summary tables** with detailed statistics and color-coded data types
- **Missing values analysis** with visual patterns and percentages
- **Data quality analysis** with comprehensive profiling insights

### Validation Workflows

- **Validator creation** with flexible thresholds and configuration
- **Many validation types** for comprehensive data quality checking
- **Step-by-step validation** building with natural language commands
- **Template-based validation** for common data quality patterns

### HTML Reports & Analysis

- **Interactive validation reports** automatically opened in your browser
- **Timestamped HTML files** for easy sharing and documentation
- **Python code generation** for reproducible validation scripts

All interactions use natural language, making advanced data validation accessible to users at any technical level while producing publication-ready HTML reports.

## Common Validation Rules

Understanding what validation rules to ask for will help you quickly build comprehensive data quality checks. These examples cover the most frequent validation scenarios using Pointblank's built-in validation functions.

### Data Integrity

- "Check for duplicate show IDs"
- "Ensure no missing required fields like title"
- "Validate that release years are between 1900 and 2025"

### Business Logic

- "Ratings must be between 0 and 10"
- "Budget must be positive numbers"
- "Duration should be greater than 0"

### Cross-Field Validation

- "Release year should match date_added year"
- "Vote count should correlate with popularity"
- "Movies should have directors specified"

### Available Templates

Pointblank includes pre-built validation templates:

- `basic_quality` - Essential data quality checks
- `financial_data` - Money and numeric validations
- `customer_data` - Personal information validations
- `sensor_data` - Time series and measurement checks
- `survey_data` - Response and rating validations

These rule patterns can be combined and customized for your specific data and business requirements. The natural language interface makes it easy to express complex validation logic without learning technical syntax.

## Some Tips and Tricks

These recommendations will help you get more value from your Pointblank MCP server and avoid some common pitfalls.

### Talk Naturally

✅ **Good:** "Check if customer emails look valid"

❌ **Avoid:** "Execute col_vals_regex on email column"

### Provide Context

✅ **Good:** "This is for the board presentation"

❌ **Avoid:** Just asking for validation without explanation

### Build Incrementally

1. Start with data profiling
2. Add basic validation rules
3. Create templates for reuse
4. Set up automated checks

### Save Templates

```
"Save these rules as 'customer_validation'"
"Apply the financial_data template"
"Use our standard survey validation"
```

### Interactive Visual Tables

Pointblank automatically generates beautiful, interactive HTML tables for data exploration:

```
"Show me a preview of the data"
"Generate a column summary table"
"Create a missing values analysis"
```

These commands create professional HTML tables with:

- **Color-coded data types** (numeric in purple, text in yellow)
- **Gradient styling** tailored to each table type
- **Automatic browser opening** for immediate viewing
- **Timestamped files** for easy reference and sharing

The tables open automatically in your default browser, making it easy to share data insights with colleagues or include in presentations.

These practices help you build data quality workflows that scale with your needs while remaining accessible to those with varying technical backgrounds.

## File Support

Pointblank works with many major data file formats, making it easy to validate data regardless of how it's stored. This support means you can maintain consistent validation practices across your entire data ecosystem.

| Type | Extensions | Example | Backend Support |
|------|------------|---------|-----------------|
| **CSV** | `.csv` | `sales_data.csv` | pandas, polars |
| **Parquet** | `.parquet` | `big_data.parquet` | pandas, polars |
| **JSON** | `.json` | `api_response.json` | pandas, polars |
| **JSON Lines** | `.jsonl` | `streaming_data.jsonl` | pandas, polars |

The consistent natural language interface works the same regardless of file format, so you can focus on validation logic rather than technical details. Polars provides faster processing for large datasets, while Pandas offers broader format support.

## Quick Troubleshooting

When you encounter issues, these quick fixes resolve the most common problems. Furthermore, the natural language interface means you can always ask for help and explanations.

| Problem | Quick Fix |
|---------|-----------|
| "File not found" | Use absolute path: `/Users/name/Downloads/data.csv` |
| "DataFrame not found" | Check loaded datasets with "List my loaded dataframes" |
| "Validator not found" | Use "List active validators" to see available validators |
| "Validation too slow" | Try "Use pandas backend" or sample your data first |
| "HTML tables won't open" | Check your default browser settings |
| "Need validation ideas" | Ask "Show me validation templates" or "Suggest validations for my data" |

**Browser Issues**: The HTML tables automatically open in your default browser. If they don't appear, check that your browser isn't blocking pop-ups and that you have a default browser set in your system preferences.

Remember, you can always ask the AI to explain what's happening or suggest solutions when you run into problems.

## Now You're Ready!

You now have everything needed to start validating data through conversation. The beauty of Pointblank's MCP server is that it grows with your expertise: start simple and gradually build more sophisticated validation workflows as you become comfortable with the interface.

Start with simple commands and build up to more complex validation workflows. The AI will guide you through the process and help you create robust data quality checks!


### OpenTelemetry Integration

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

OpenTelemetry (OTel) is the industry-standard framework for vendor-neutral observability. Pointblank
can export validation results (pass/fail counts, durations, threshold breaches) as OTel signals
so they appear alongside your application metrics in Grafana, Datadog, New Relic, or any
OTel-compatible backend.

Instead of building separate integrations for each monitoring tool, one OTel export covers all of
them. And when Pointblank runs inside an Airflow, Prefect, or Dagster task that already uses OTel
instrumentation, validation spans appear inside the pipeline's distributed trace automatically.

## Installation

The OTel integration is an optional dependency. Install it with:

```bash
pip install pointblank[otel]
```

This adds `opentelemetry-api` and `opentelemetry-sdk`. For sending data to a collector you'll also
need an exporter package, most commonly:

```bash
pip install opentelemetry-exporter-otlp-proto-grpc
# or
pip install opentelemetry-exporter-otlp-proto-http
```

With these packages installed, Pointblank can push signals to any OTel-compatible backend. The
following section describes exactly what gets sent.

## What Gets Exported

Pointblank maps its validation outputs onto three OTel signal types:

| Signal | What's Exported | Use Case |
|--------|----------------|----------|
| **Metrics** | Pass/fail counters, pass rate gauge, step duration histogram, threshold breach counts | Dashboards, alerting, SLA tracking |
| **Traces** | A root span per `interrogate()` call with child spans per validation step | Pipeline traceability, performance analysis |
| **Logs** | Structured log records for threshold breaches | Incident investigation, audit trails |

Each signal type captures a different dimension of the validation run. Metrics provide aggregate
counts and rates suitable for dashboards, traces capture timing and parent-child relationships for
debugging, and logs produce structured records for incident review.

### Metric Instruments

The metrics signal is the most detailed. Twelve instruments cover step-level and aggregate
statistics, all sharing a configurable name prefix (default: `pb.validation`).

| Metric Name | Kind | Description |
|-------------|------|-------------|
| `{prefix}.steps.total` | Counter | Total validation steps executed |
| `{prefix}.steps.passed` | Counter | Steps where all test units passed |
| `{prefix}.steps.failed` | Counter | Steps with at least one failure |
| `{prefix}.test_units.total` | Counter | Total test units evaluated |
| `{prefix}.test_units.passed` | Counter | Test units that passed |
| `{prefix}.test_units.failed` | Counter | Test units that failed |
| `{prefix}.pass_rate` | Gauge | Overall pass fraction (0--1) |
| `{prefix}.step.duration` | Histogram | Per-step processing duration (seconds) |
| `{prefix}.duration` | Gauge | Total interrogation wall-clock time (seconds) |
| `{prefix}.threshold.warning` | Counter | Steps exceeding warning threshold |
| `{prefix}.threshold.error` | Counter | Steps exceeding error threshold |
| `{prefix}.threshold.critical` | Counter | Steps exceeding critical threshold |

Every metric carries attributes derived from the `Validate` object (`pb.tbl_name`, `pb.label`,
`pb.owner`, `pb.version`) plus any `extra_attributes` you provide. Per-step metrics also include
`pb.step.i`, `pb.step.assertion_type`, and `pb.step.column`.

Together these instruments give you enough information to build dashboards, set alerts, and track
data quality SLAs across every table Pointblank validates.

## Two Usage Patterns

Pointblank offers two ways to export OTel signals, matching how different teams prefer to work.

### Pattern A: Explicit Export

Import `OTelExporter`, run your validation, then call `export()`. This gives you full control over
when and how signals are emitted.

```{python}
#| eval: false
from pointblank.integrations.otel import OTelExporter

validation = (
    pb.Validate(data=df, tbl_name="orders")
    .col_vals_not_null(columns="order_id")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)

otel = OTelExporter(enable_metrics=True, enable_tracing=True)
otel.export(validation)
```

Because the export happens after `interrogate()`, you can inspect the validation results first and
decide whether to export at all. This pattern works well for ad-hoc analysis or when you want to
conditionally emit signals based on the outcome.

### Pattern B: FinalActions (Automatic)

Use `emit_otel()` inside `FinalActions` so the signals are emitted automatically at the end of
every `interrogate()` call (no extra code needed after the validation).

```{python}
#| eval: false
import pointblank as pb

validation = (
    pb.Validate(
        data=df,
        tbl_name="orders",
        final_actions=pb.FinalActions(
            pb.emit_otel(enable_metrics=True),
        ),
    )
    .col_vals_not_null(columns="order_id")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()  # metrics emitted here automatically
)
```

You can combine `emit_otel()` with other final actions like `send_slack_notification()`:

```{python}
#| eval: false
import os

validation = (
    pb.Validate(
        data=df,
        tbl_name="orders",
        final_actions=pb.FinalActions(
            pb.emit_otel(enable_metrics=True),
            pb.send_slack_notification(webhook_url=os.environ["SLACK_WEBHOOK"]),
        ),
    )
    .col_vals_not_null(columns="order_id")
    .interrogate()
)
```

Both patterns send the same signals, with the same names and attributes: `emit_otel()` takes the
same `enable_tracing=`, `enable_logging=` and provider arguments as `OTelExporter`. Since
`emit_otel()` runs inside `interrogate()`, its `pb.validate` span becomes a child of whatever span
is current at that point (such as the span of an Airflow or Dagster task). Choose explicit export
when you want to look at the results before deciding whether to send them, and `FinalActions` when
you want every validation run to report automatically.

## Working Example with Console Output

Let's walk through a complete example using in-memory providers so you can see exactly what gets
emitted (no external collector needed).

### Setup

Before exporting any signals we need OTel providers to receive them. The SDK's in-memory providers
are very useful for exploration: they capture every metric, span, and log record in Python lists so
we can print them without running a collector.

```{python}
#| eval: false
import polars as pl
import pointblank as pb
from pointblank.integrations.otel import OTelExporter

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import InMemoryLogRecordExporter, SimpleLogRecordProcessor

# In-memory providers capture signals without a collector
metric_reader = InMemoryMetricReader()
meter_provider = MeterProvider(metric_readers=[metric_reader])

span_exporter = InMemorySpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))

log_exporter = InMemoryLogRecordExporter()
logger_provider = LoggerProvider()
logger_provider.add_log_record_processor(SimpleLogRecordProcessor(log_exporter))
```

Each provider is paired with an in-memory reader or exporter that we'll query later to see what was
captured.

### Run Validation and Export

With the providers ready, we'll create a small DataFrame with some intentional quality issues (a
negative amount, a zero amount, and a missing customer name), validate it, and then export the
results.

```{python}
#| eval: false
df = pl.DataFrame(
    {
        "order_id": [101, 102, 103, 104, 105],
        "amount": [29.99, 150.00, -5.00, 75.50, 0.00],
        "customer": ["Alice", "Bob", None, "Dana", "Eve"],
    }
)

validation = (
    pb.Validate(
        data=df,
        tbl_name="orders",
        label="Daily orders check",
        owner="data-platform-team",
        thresholds=(0.05, 0.20, 0.40),
    )
    .col_vals_not_null(columns="customer")
    .col_vals_gt(columns="amount", value=0)
    .interrogate()
)

validation
```

The report shows two steps with failures. Now we create an `OTelExporter` with all three signal types
enabled and export the validation results.

```{python}
#| eval: false
otel = OTelExporter(
    meter_provider=meter_provider,
    tracer_provider=tracer_provider,
    logger_provider=logger_provider,
    enable_metrics=True,
    enable_tracing=True,
    enable_logging=True,
    extra_attributes={"env": "production", "pipeline": "daily-orders"},
)
otel.export(validation)
```

The `export()` call is instantaneous as the in-memory providers simply store the data. In production,
this is where the SDK would batch and send signals to your collector.

### Inspect the Emitted Metrics

The metric reader holds all counters, gauges, and histograms that were recorded. We can iterate
through them to see every instrument and its current value.

```{python}
#| eval: false
metrics_data = metric_reader.get_metrics_data()

for resource_metrics in metrics_data.resource_metrics:
    for scope_metrics in resource_metrics.scope_metrics:
        for metric in sorted(scope_metrics.metrics, key=lambda m: m.name):
            for dp in metric.data.data_points:
                val = getattr(dp, "value", None) or getattr(dp, "count", "N/A")
                print(f"{metric.name}: {val}")
```

You'll see counters like `pb.validation.steps.total` and `pb.validation.test_units.failed`, the
`pb.validation.pass_rate` gauge, and a histogram of step durations. These are the same instruments
that would appear in Prometheus, Datadog, or any OTel-compatible metrics backend.

### Inspect the Trace Spans

Each `interrogate()` call produces a root span with one child per validation step:

```{python}
#| eval: false
spans = span_exporter.get_finished_spans()

for span in spans:
    parent = "(root)" if span.parent is None else "(child)"
    print(f"Span: {span.name} {parent}")
    for k, v in sorted(span.attributes.items()):
        print(f"  {k} = {v}")
    for event in span.events:
        print(f"  EVENT: {event.name} -> {dict(event.attributes)}")
    print()
```

The root span (`pb.validate`) carries table-level attributes and timing, while each child span
(`pb.validate.step`) carries per-step details. Steps that breach a threshold also include an event
recording the severity level.

### Inspect the Log Records

Log records are emitted for each validation step that exceeds a threshold. The severity of the log
record corresponds to the highest threshold level breached by that step.

```{python}
#| eval: false
logs = log_exporter.get_finished_logs()

for log in logs:
    rec = log.log_record
    print(f"[{rec.severity_text}] {rec.body}")
    print(f"  attributes: {dict(rec.attributes)}")
    print()
```

Each log record includes the step index, assertion type, column name, and the threshold level that
was breached. In a production setup these records would flow to your log aggregator (e.g., Loki,
Elasticsearch, or CloudWatch) where you can search and alert on them.

```{python}
#| echo: false
#| eval: false
meter_provider.shutdown()
tracer_provider.shutdown()
logger_provider.shutdown()
```

That completes the end-to-end walkthrough. The same three signal types (metrics, traces, logs) work
identically whether you use in-memory providers for development or OTLP exporters in production.

## Configuration Reference

This section documents every `OTelExporter` option and how the log-level filter works.

### `OTelExporter` Parameters

The table below lists all constructor parameters for `OTelExporter`.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `meter_name` | `"pointblank"` | Name for the OTel Meter |
| `meter_version` | Package version | Version string for the Meter |
| `enable_metrics` | `True` | Emit metric counters, gauges, and histograms |
| `enable_tracing` | `False` | Emit trace spans |
| `enable_logging` | `False` | Emit log records for threshold breaches |
| `meter_provider` | Global default | Custom `MeterProvider` instance |
| `tracer_provider` | Global default | Custom `TracerProvider` instance |
| `logger_provider` | `None` | Custom `LoggerProvider` (required for logging) |
| `metric_prefix` | `"pb.validation"` | Prefix for all metric instrument names |
| `log_level` | `"warning"` | Minimum threshold severity for log emission |
| `extra_attributes` | `None` | Additional key-value pairs on all signals |

The `emit_otel()` factory function accepts the same parameters and returns a callable suitable for
use in `FinalActions`. Both raise a `ValueError` when they're created if `enable_logging=True` is
given without a `logger_provider=`, or if `log_level=` isn't `"warning"`, `"error"` or `"critical"`
(in any case), so a misconfigured exporter fails before any validation runs.

### Log Level Filtering

Not every threshold breach warrants a log record. The `log_level=` parameter controls which breaches
produce log records, letting you filter out lower-severity noise.

| `log_level` value | warning breaches | error breaches | critical breaches |
|-------------------|:---:|:---:|:---:|
| `"warning"` | ✓ | ✓ | ✓ |
| `"error"` | — | ✓ | ✓ |
| `"critical"` | — | — | ✓ |

For most production deployments, `"warning"` (the default) is a good starting point. Switch to
`"error"` or `"critical"` if your log volume is too high or you only want to be notified about
serious issues.

## Bringing Your Own Provider

The in-memory providers used in the walkthrough above are great during development, but in production
you'll point the SDK at an actual collector. Here's how to configure a `MeterProvider` with the OTLP
gRPC exporter.

```{python}
#| eval: false
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="http://otel-collector:4317")
)
provider = MeterProvider(metric_readers=[reader])

otel = OTelExporter(meter_provider=provider)
otel.export(validation)
```

If you set the standard `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable and use the global
provider (the default when `meter_provider=None`), no provider configuration is needed at all.

The same approach works for `TracerProvider` and `LoggerProvider`. Pass your custom providers to
`OTelExporter` and they'll be used instead of the global defaults.

## Pipeline Integration

Pointblank's OTel integration becomes especially powerful inside orchestrated data pipelines. When
Pointblank runs inside an Airflow, Prefect, or Dagster task that has OTel auto-instrumentation,
trace context propagation happens automatically. The `pb.validate` span attaches as a child of the
current task span, giving you a unified view of pipeline and validation performance.

```
Span: airflow.task (root)
└── Span: pb.validate
    ├── Span: pb.validate.step  (`col_vals_not_null` on 'order_id')
    ├── Span: pb.validate.step  (`col_vals_gt` on 'amount')
    └── Span: pb.validate.step  (`col_vals_in_set` on 'status')
```

This means validation timing and failures appear in your existing pipeline traces without any
additional configuration. If a validation step is slow or failing, you can trace it back to the
exact pipeline run and task that triggered it.

## Example PromQL Queries

Once metrics are flowing into a Prometheus-compatible backend, you can query them with PromQL. Below
are a few starter queries that cover common data-quality monitoring scenarios.

```promql
# Pass rate for a specific table
pb_validation_pass_rate{pb_tbl_name="orders"}

# Alert on any critical threshold breach in the last hour
increase(pb_validation_threshold_critical_total[1h]) > 0

# Failed test units by table (24h window)
sum by (pb_tbl_name) (increase(pb_validation_test_units_failed_total[24h]))

# p95 validation step duration
histogram_quantile(0.95, rate(pb_validation_step_duration_bucket[5m]))
```

These queries use Prometheus naming conventions (dots replaced with underscores, `_total` suffix on
counters). Adjust the metric names if you've changed the `metric_prefix=` parameter.

With metrics, traces, and logs all flowing through OpenTelemetry, you have a complete observability
picture for your data validation workflows. Whether you're running ad-hoc checks in a notebook or
validating tables in a production pipeline, the same instrumentation gives you dashboards, alerts,
and audit trails with no additional tooling.


### Importing Metadata from External Standards

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Many data files carry rich metadata that goes beyond raw values: variable labels describing what
each column means, value labels mapping codes to human-readable categories, controlled
terminologies defining permitted values, and constraints specifying valid ranges and formats. This
metadata lives in files like SPSS `.sav` archives, CDISC Define-XML documents, and Frictionless
Data Packages, and it represents a significant investment in documenting data expectations.

Pointblank's metadata import system reads these external descriptions and converts them into
validation workflows automatically. Rather than manually translating a Define-XML specification
or an SPSS codebook into validation code, you can call `import_metadata()` and let Pointblank
generate the appropriate checks for you. The result is a `MetadataImport` object that bridges
between domain-specific formats and Pointblank's validation engine.

## Quick Start

The fastest path from a metadata file to a running validation uses three steps: import the metadata,
provide your data, and call [`to_validate()`](`MetadataImport.to_validate`). Here is the basic
pattern:

```python
import pointblank as pb

# Import metadata from any supported format
meta = pb.import_metadata("define.xml", format="cdisc_define")

# Convert to a validation workflow and run it
validation = meta.to_validate(data=my_dataframe).interrogate()
```

The `import_metadata()` function is the single entry point for all supported formats. It returns
a `MetadataImport` object containing parsed variable definitions, codelists, missing value codes,
and dataset-level metadata. From there, you can generate a `Schema`, a full `Validate` workflow,
or inspect individual variables and their constraints.

## Supported Formats

Pointblank can import metadata from a range of domain-specific standards. Each format is handled
by a dedicated reader that understands the structure and semantics of that standard.

| Format | Description | File Types |
|--------|-------------|------------|
| `spss` / `sav` | SPSS variable labels, value labels, missing codes | `.sav`, `.zsav` |
| `xpt` / `sas` | SAS Transport variable labels and formats | `.xpt` |
| `stata` / `dta` | Stata variable labels and value labels | `.dta` |
| `frictionless` / `datapackage` | Frictionless Data Package schemas and constraints | `.json` |
| `table_schema` | Standalone Frictionless Table Schema | `.json` |
| `csvw` | W3C CSV on the Web metadata | `.json`, `.jsonld` |
| `cdisc_define` / `define_xml` | CDISC Define-XML variable definitions and codelists | `.xml` |
| `cdisc_ct` | CDISC Controlled Terminology packages | `.xml` |
| `cdisc_sdtm` | SDTM domain validation templates | (built-in) |
| `cdisc_adam` | ADaM dataset validation templates | (built-in) |

The format can be specified explicitly or auto-detected from the file extension in many cases.
For ambiguous formats (like XML files that could be Define-XML or Controlled Terminology),
Pointblank inspects the file content to determine the correct reader.

## The `MetadataImport` Object

Every call to `import_metadata()` returns a `MetadataImport` instance. This object holds all
the information extracted from the source file, organized into a structure that Pointblank can
work with. Understanding its components helps you get the most out of imported metadata.

The key attributes are:

| Attribute | Type | Description |
|-----------|------|-------------|
| `source_format` | `str` | Which format was parsed (e.g., `"spss"`, `"cdisc_define"`) |
| `source_path` | `str` or `None` | File path that was read |
| `dataset_name` | `str` or `None` | Name of the dataset from the metadata |
| `dataset_label` | `str` or `None` | Human-readable dataset label |
| `variables` | `list[VariableMetadata]` | Per-variable metadata (labels, constraints, types) |
| `codelists` | `dict[str, Codelist]` | Named controlled terminologies / value sets |
| `missing_value_codes` | `dict[str, list]` | Sentinel values that indicate missingness |

Each `VariableMetadata` object describes a single column, including its name, data type, label,
and any constraints that were defined in the source file. Constraints are automatically mapped
to Pointblank validation methods when you call `to_validate()`.

## Converting to a Schema

The `to_schema()` method produces a Pointblank `Schema` reflecting the column names and data types
from the metadata. This is useful for structural validation (ensuring a DataFrame has the expected
shape) without running value-level checks.

```python
meta = pb.import_metadata("clinical_data.xpt", format="xpt")

# Get just the schema (column names + types)
schema = meta.to_schema()

# Use it in a validation
validation = (
    pb.Validate(data=my_data)
    .col_schema_match(schema=schema)
    .interrogate()
)
```

The schema captures what the metadata says the table *should* look like, not what the data
actually contains. This makes it valuable for catching structural drift: columns that were
renamed, retyped, or dropped since the metadata was last updated.

## Converting to a Validation Workflow

The [`to_validate()`](`MetadataImport.to_validate`) method is where all the power lives. It reads
all constraints from the metadata and generates a complete `Validate` object with the appropriate
validation steps. Each constraint type maps to a specific Pointblank method:

| Metadata Constraint | Generated Validation Step |
|---------------------|--------------------------|
| `required=True` | `col_vals_not_null()` |
| `unique=True` | `rows_distinct()` |
| `min_val` / `max_val` | `col_vals_between()` |
| `max_length` | `col_vals_expr()` (string length check) |
| `pattern` | `col_vals_regex()` |
| `allowed_values` or codelist | `col_vals_in_set()` |
| Schema (column names + types) | `col_schema_match()` |

The generated workflow covers everything the metadata specifies. You can run it as-is for
comprehensive validation, or add your own steps on top for business rules that go beyond
what the metadata captures:

```python
meta = pb.import_metadata("survey_data.sav", format="spss")

# Generate validation from metadata, then add custom checks
validation = (
    meta.to_validate(data=df)
    .col_vals_between(columns="response_time_ms", left=100, right=30000)
    .interrogate()
)
```

## Format Auto-Detection

When the format is unambiguous from the file extension, you can omit the `format=` parameter
and let Pointblank detect it automatically:

```python
# These are equivalent:
meta = pb.import_metadata("data.sav", format="spss")
meta = pb.import_metadata("data.sav")  # auto-detected from .sav extension

# Same for other unambiguous extensions:
meta = pb.import_metadata("delivery.xpt")   # detected as SAS Transport
meta = pb.import_metadata("panel.dta")       # detected as Stata
```

For JSON files, you need to specify the format explicitly because a `.json` file could be a
Frictionless Data Package, a Table Schema, or a CSVW document. For XML files, Pointblank
inspects the content (namespace declarations and root element) to distinguish between
Define-XML and Controlled Terminology formats.

## Working with Variables

The `variables` list on a `MetadataImport` contains one `VariableMetadata` object per column.
Each carries all the information that was available in the source file for that column.

```python
meta = pb.import_metadata("demographics.sav", format="spss")

# Inspect individual variables
for var in meta.variables:
    print(f"{var.name}: {var.dtype}, label={var.label!r}")
    if var.allowed_values:
        print(f"  Allowed: {var.allowed_values}")
    if var.required:
        print(f"  Required (non-null)")
```

Different source formats populate different fields. An SPSS file provides value labels and
missing value codes but not controlled terminology references. A CDISC Define-XML provides
computational methods and codelist references but not display formats. The `VariableMetadata`
dataclass is a superset that accommodates all source formats.

## Working with Codelists

Codelists represent controlled terminologies: named sets of permitted values with labels and
optional descriptions. They appear in CDISC files, SPSS value labels, and Frictionless enum
constraints.

```python
meta = pb.import_metadata("define.xml", format="cdisc_define")

# List available codelists
for name, codelist in meta.codelists.items():
    print(f"{name}: {len(codelist)} entries, extensible={codelist.extensible}")

# Get the valid values from a codelist
sex_values = meta.codelists["C66731"].to_set()
# e.g., ["M", "F", "U", "UNDIFFERENTIATED"]

# Get value-to-label mapping
sex_labels = meta.codelists["C66731"].to_dict()
# e.g., {"M": "Male", "F": "Female", "U": "Unknown", ...}
```

When `to_validate()` encounters a variable with a `codelist_ref`, it generates a
`col_vals_in_set()` step using the codelist's values. For non-extensible codelists, any value
outside the set is a failure. For extensible codelists, additional values are permitted (the
check still runs but serves as documentation of expected values).

## Handling Missing Value Codes

Statistical packages like SPSS and SAS use sentinel values to represent different kinds of
missingness. A value of `-99` might mean "not asked", while `-98` means "refused". These are
not null values in the data; they are valid numeric entries that carry semantic meaning.

The `missing_value_codes` dictionary maps variable names to their defined missing value codes:

```python
meta = pb.import_metadata("survey.sav", format="spss")

# Check what missing codes are defined
for var_name, codes in meta.missing_value_codes.items():
    for code in codes:
        print(f"{var_name}: {code.value} = {code.label}")
```

When generating validation, these codes inform Pointblank about which values should be treated
as missing rather than as data errors. This prevents false positives where a valid missing
code like `-99` would otherwise fail a `col_vals_ge(value=0)` check.

## Conclusion

The metadata import system transforms domain-specific data descriptions into actionable
validation workflows. Whether you are working with statistical package files from survey
research, CDISC documents from clinical trials, or Frictionless schemas from open data
platforms, the pattern is the same: call `import_metadata()`, inspect what was extracted, and
then convert it to a `Schema` or `Validate` object. The following pages in this section cover
each format family in detail, starting with statistical packages and then moving to CDISC
clinical data standards.


### Statistical Package Metadata

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Statistical software packages like SPSS, SAS, and Stata store rich metadata alongside data values.
Variable labels describe what each column represents, value labels map numeric codes to meaningful
categories, and missing value definitions distinguish between different reasons for absent data.
This metadata represents a carefully curated description of data expectations, often built up over
years of survey design and data management work.

Pointblank can read this embedded metadata and translate it directly into validation rules. When
you import a `.sav`, `.xpt`, or `.dta` file, Pointblank extracts the full metadata catalog and
maps each element to the appropriate validation method. Value labels become `col_vals_in_set()`
checks, data types become schema constraints, and missing value codes inform how validation
handles sentinel values.

## Prerequisites

Reading metadata from statistical package files requires the `pyreadstat` library. This is an
optional dependency that you can install separately:

```bash
pip install pyreadstat
```

Or install Pointblank with the stats extra to get everything you need:

```bash
pip install pointblank[stats]
```

The `pyreadstat` library reads SPSS, SAS, and Stata file metadata without loading the full dataset
into memory. This makes the import fast even for large files, since only the metadata header is
parsed.

## SPSS (.sav) Files

SPSS `.sav` files are the most metadata-rich of the statistical package formats. They carry
variable labels, value labels for categorical variables, defined missing value codes, display
formats, and variable measurement levels. Pointblank extracts all of these and maps them to
validation concepts.

### What Gets Extracted

When you import an SPSS file, Pointblank reads the following metadata:

| Metadata Element | Pointblank Mapping |
|------------------|-------------------|
| Variable names | Column names in Schema |
| Variable labels | Stored as `label` on `VariableMetadata` |
| Variable types (numeric/string) | Mapped to `dtype` (Float64, Int64, String, Date, etc.) |
| Value labels | `allowed_values` list and `Codelist` objects |
| Missing value codes | `MissingValueCode` entries with labels |
| Display formats (F8.2, A20, etc.) | Stored as `display_format` |
| Date/time formats | Mapped to Date, Time, or Datetime dtypes |

### Basic Usage

The simplest usage reads the file and converts to a validation workflow:

```python
import pointblank as pb

# Import metadata from an SPSS file
meta = pb.import_metadata("survey_responses.sav", format="spss")

# See what was extracted
print(f"Dataset: {meta.dataset_name}")
print(f"Variables: {len(meta.variables)}")
print(f"Codelists: {len(meta.codelists)}")

# Generate validation from the metadata
validation = meta.to_validate(data=my_data).interrogate()
```

The `format="spss"` parameter is optional here because Pointblank auto-detects `.sav` files from
their extension.

### Value Labels and Codelists

SPSS value labels define the permitted values for categorical variables. For example, a variable
`GENDER` might have labels `{1: "Male", 2: "Female", 3: "Non-binary"}`. Pointblank converts
these into `Codelist` objects and generates `col_vals_in_set()` checks:

```python
meta = pb.import_metadata("demographics.sav")

# Inspect a specific variable's allowed values
for var in meta.variables:
    if var.allowed_values:
        print(f"{var.name}: {var.allowed_values}")

# The codelists are also available directly
for cl_name, codelist in meta.codelists.items():
    values = codelist.to_set()
    labels = codelist.to_dict()
    print(f"{cl_name}: {labels}")
```

When validation is generated, each variable with value labels gets a `col_vals_in_set()` step
that ensures all data values appear in the labeled set. Values outside the set are flagged as
failures in the validation report.

### Missing Value Codes

SPSS supports up to three discrete missing value codes per variable, plus an optional range of
missing values. These codes carry semantic meaning: a value of `-99` might indicate "question not
asked", while `-98` means "respondent refused to answer".

```python
meta = pb.import_metadata("survey.sav")

# Examine defined missing value codes
for var_name, codes in meta.missing_value_codes.items():
    for code in codes:
        print(f"  {var_name}: value={code.value}, meaning={code.label}")
```

Missing value codes are preserved in the `MetadataImport` object so downstream tools can handle
them appropriately. When validation is generated, these codes are documented in the metadata
rather than generating explicit exclusion rules, since the correct handling depends on your
analysis context.

#### Turning missing codes into `MissingSpec` objects

To put these codes to work in validation and reporting, convert them into
[`MissingSpec`](`pointblank.MissingSpec`) objects. The
[`MetadataImport.missing_specs()`](`pointblank.MetadataImport`) method does this for every variable
that declares missing values, returning a `{column: MissingSpec}` mapping (the reason labels are
derived from the variables' value labels):

```python
meta = pb.import_metadata("survey.sav")

# Auto-generate a {column: MissingSpec} mapping from the declared missing values
specs = meta.missing_specs()

# Use the specs in a structured missingness report...
pb.missing_vals_tbl(data, missing=specs)

# ...or in missingness-aware validation
validation = (
    pb.Validate(data=data)
    .col_vals_between(columns="age", left=0, right=120, missing=specs["age"])
    .interrogate()
)
```

You can also build a spec for a single variable with
[`VariableMetadata.to_missing_spec()`](`pointblank.VariableMetadata`), or construct one directly
from SPSS-style values via `pb.MissingSpec.from_spss(missing_values=[...], labels={...})`. See the
*Missing Values Reporting* and *Validation Methods* articles for what you can do with these specs.

### Type Detection from Formats

SPSS stores numeric variables with format strings that indicate how they should be displayed. These
formats also carry type information. A format like `DATE11` indicates a date variable, `DATETIME20`
indicates a datetime, and `F8.0` (eight characters, zero decimal places) suggests an integer.
Pointblank uses these format strings to infer the most appropriate Pointblank dtype:

| SPSS Format | Inferred Dtype |
|-------------|---------------|
| `F8.2`, `F5.1` | Float64 |
| `F8.0`, `F3.0` | Int64 |
| `A20`, `A8` | String |
| `DATE11`, `ADATE10` | Date |
| `TIME8` | Time |
| `DATETIME20` | Datetime |

This inference makes the generated schema more precise than simply marking everything as
numeric or string.

## SAS Transport (.xpt) Files

SAS Transport (`.xpt`) files are the standard delivery format for regulatory submissions,
particularly in pharmaceutical clinical trials. They carry variable names, labels, types, and
length constraints. While less metadata-rich than SPSS files (no value labels), they provide the
structural foundation for CDISC-compliant data packages.

### What Gets Extracted

| Metadata Element | Pointblank Mapping |
|------------------|-------------------|
| Variable names | Column names in Schema |
| Variable labels | Stored as `label` on `VariableMetadata` |
| Variable types (numeric/character) | Mapped to `dtype` |
| Variable lengths | `max_length` constraint (for character variables) |
| SAS formats (DATE9., etc.) | `display_format` + dtype inference |
| Dataset name | `dataset_name` on `MetadataImport` |
| Dataset label | `dataset_label` on `MetadataImport` |

### Basic Usage

```python
import pointblank as pb

# Import metadata from a SAS Transport file
meta = pb.import_metadata("demographics.xpt", format="xpt")

# Examine the extracted metadata
print(f"Dataset: {meta.dataset_name}")
print(f"Label: {meta.dataset_label}")

for var in meta.variables:
    constraint_info = []
    if var.max_length:
        constraint_info.append(f"max_length={var.max_length}")
    if var.required:
        constraint_info.append("required")
    print(f"  {var.name} ({var.dtype}): {var.label}")
    if constraint_info:
        print(f"    Constraints: {', '.join(constraint_info)}")
```

### Length Constraints

Character variables in SAS Transport files have defined maximum lengths. Pointblank captures these
as `max_length` constraints on the `VariableMetadata` object. When you call
[`to_validate()`](`MetadataImport.to_validate`), variables with length constraints get a
`col_vals_expr()` step that checks string length does not exceed the specified maximum.

This is particularly important for CDISC submissions where variable lengths are strictly defined
in the submission specification. A variable defined as `$200.` (200 characters) must not contain
values longer than 200 characters, and Pointblank will flag any violations.

### Format-Based Type Detection

Like SPSS, SAS formats encode type information. A variable with format `DATE9.` is a date, one
with `DATETIME20.` is a datetime, and `$CHAR200.` is a 200-character string:

| SAS Format | Inferred Dtype |
|------------|---------------|
| `DATE9.`, `MMDDYY10.` | Date |
| `TIME8.` | Time |
| `DATETIME20.` | Datetime |
| `$CHAR200.`, `$50.` | String |
| Numeric (no date format) | Float64 |

## Stata (.dta) Files

Stata `.dta` files provide variable labels, value labels (similar to SPSS), and typed storage
with distinct integer and floating-point types. The format is commonly used in economics, public
health, and social science research.

### What Gets Extracted

| Metadata Element | Pointblank Mapping |
|------------------|-------------------|
| Variable names | Column names in Schema |
| Variable labels | Stored as `label` on `VariableMetadata` |
| Storage types (byte, int, long, float, double, strN) | Mapped to `dtype` |
| Value labels | `allowed_values` list and `Codelist` objects |
| Dataset label | `dataset_label` |

### Basic Usage

```python
import pointblank as pb

# Import metadata from a Stata file
meta = pb.import_metadata("panel_data.dta", format="stata")

# Inspect what was found
print(f"Variables: {len(meta.variables)}")
for var in meta.variables:
    type_info = f"({var.dtype})"
    label_info = f" - {var.label}" if var.label else ""
    print(f"  {var.name} {type_info}{label_info}")
```

### Type Mapping

Stata has more granular numeric types than SPSS, which Pointblank maps to appropriate dtypes:

| Stata Type | Inferred Dtype |
|------------|---------------|
| `byte`, `int`, `long` | Int64 |
| `float`, `double` | Float64 |
| `str1` through `str2045` | String |

The distinction between integer and floating-point types is preserved, which produces more
accurate schema validation. A variable stored as `int` in Stata should contain only integer
values, and the generated schema reflects that expectation.

## Generating Validation from Statistical Metadata

Once you have imported metadata from any statistical package file, the workflow for generating
validation is the same. The [`to_validate()`](`MetadataImport.to_validate`) method examines every
variable's constraints and creates the appropriate validation steps.

For a typical SPSS file with value labels and types defined, the generated validation includes:

1. A `col_schema_match()` step verifying column names and data types
2. `col_vals_in_set()` steps for every variable with value labels
3. `col_vals_not_null()` steps for variables marked as required
4. `col_vals_expr()` steps for variables with length constraints (from SAS Transport)

```python
import pointblank as pb

# Import and validate in one chain
meta = pb.import_metadata("study_data.sav")
validation = meta.to_validate(data=df).interrogate()

# Or generate just the schema for a lightweight structural check
schema = meta.to_schema()
lightweight = (
    pb.Validate(data=df)
    .col_schema_match(schema=schema)
    .interrogate()
)
```

You can also combine metadata-generated validation with your own custom steps. The `to_validate()`
method returns an un-interrogated `Validate` object, so you can chain additional methods before
calling `.interrogate()`:

```python
meta = pb.import_metadata("survey.sav")

# Start from metadata, add custom business rules
validation = (
    meta.to_validate(data=df)
    .col_vals_between(columns="completion_time_min", left=5, right=120)
    .rows_distinct(columns_subset=["respondent_id"])
    .interrogate()
)
```

## Conclusion

Statistical package metadata provides a ready-made specification of data expectations that you can
leverage directly in Pointblank. Rather than manually inspecting a codebook and writing validation
rules by hand, importing the metadata gives you instant, comprehensive coverage of the constraints
that the data's creators intended. The next page covers CDISC standards for clinical trial data,
which build on these same concepts with additional domain-specific validation rules for regulatory
compliance.


### CDISC Clinical Data Standards

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Clinical trial data follows strict organizational standards defined by CDISC (Clinical Data
Interchange Standards Consortium). These standards specify exactly which variables must appear in
each dataset, what values are permitted, how dates should be formatted, and how analysis datasets
trace back to their source observations. Regulatory agencies like the FDA and PMDA require
CDISC-compliant data for drug submissions, making adherence to these standards mandatory for
pharmaceutical organizations.

Pointblank provides native support for the three major CDISC data models: **SDTM** (Study Data
Tabulation Model) for raw collected data, **ADaM** (Analysis Data Model) for analysis-ready
datasets, and **Define-XML** for the metadata documents that describe both. Whether you are
preparing a regulatory submission, running quality checks on incoming CRO data, or building
automated validation pipelines for clinical data warehouses, Pointblank can generate the
appropriate checks directly from the standard specifications.

## Prerequisites

CDISC XML parsing (Define-XML and Controlled Terminology files) requires the `lxml` library:

```bash
pip install lxml
```

Or install Pointblank with the CDISC extra:

```bash
pip install pointblank[cdisc]
```

Reading metadata from SAS Transport (`.xpt`) files (used in the *Exporting Metadata* example below)
additionally requires the `pyreadstat` library:

```bash
pip install pyreadstat
```

The SDTM and ADaM domain templates are built into Pointblank and require no additional
dependencies. They encode the structural requirements from the SDTM Implementation Guide 3.4 and
the ADaM Implementation Guide 1.1 directly in Python, so you can validate clinical datasets
without needing the original XML specification documents.

## Define-XML Import

Define-XML is the CDISC standard for documenting dataset structure. It describes every variable in
a submission package: its name, label, data type, length, origin, and associated controlled
terminology. Pointblank can parse Define-XML 2.0 and 2.1 documents and extract this metadata into
a form suitable for validation.

### Importing a Define-XML File

The `import_metadata()` function with `format="cdisc_define"` reads a Define-XML file and returns
a `MetadataPackage` containing metadata for all datasets defined in the document. So that you can
run these examples without supplying your own files, Pointblank bundles a small set of sample
metadata documents that you can locate with `load_metadata_example()`:

```{python}
import pointblank as pb

# Locate the bundled Define-XML example and import all of its datasets
define_path = pb.load_metadata_example("define.xml")
package = pb.import_metadata(define_path, format="cdisc_define")

# List the datasets defined in the document
for name in package.keys():
    meta = package[name]
    print(f"{name}: {meta.dataset_label} ({len(meta.variables)} variables)")
```

Each dataset in the package is a `MetadataImport` object with full variable-level metadata. You
can access individual datasets by name and generate validation from them. Here we validate a small
Demographics table against the metadata extracted from the Define-XML:

```{python}
import polars as pl

# Get metadata for the Demographics domain
dm_meta = package["DM"]

# A small Demographics dataset to validate
dm_data = pl.DataFrame({
    "STUDYID": ["XYZ789"] * 4,
    "DOMAIN": ["DM"] * 4,
    "USUBJID": ["XYZ789-001", "XYZ789-002", "XYZ789-003", "XYZ789-004"],
    "SUBJID": ["001", "002", "003", "004"],
    "RFSTDTC": ["2024-01-15", "2024-01-20", "2024-02-01", "2024-02-10"],
    "RFENDTC": ["2024-06-15", "2024-06-20", "2024-07-01", "2024-07-10"],
    "SITEID": ["SITE01", "SITE01", "SITE02", "SITE02"],
    "AGE": [45, 62, 38, 55],
    "AGEU": ["YEARS"] * 4,
    "SEX": ["M", "F", "M", "F"],
    "RACE": ["WHITE", "BLACK OR AFRICAN AMERICAN", "ASIAN", "WHITE"],
    "ARMCD": ["DRUG", "PLACEBO", "DRUG", "PLACEBO"],
    "ARM": ["Active Drug 10mg", "Placebo", "Active Drug 10mg", "Placebo"],
})

# Generate validation for the Demographics data from the Define-XML metadata
validation = dm_meta.to_validate(data=dm_data).interrogate()
validation
```

### What Gets Extracted

Define-XML documents contain rich structural metadata. Pointblank extracts the following elements:

| Define-XML Element | Pointblank Mapping |
|--------------------|-------------------|
| ItemGroupDef (dataset) | `MetadataImport` per dataset |
| ItemDef (variable) | `VariableMetadata` with name, label, dtype |
| DataType attribute | Mapped to Pointblank dtype (String, Int64, Float64, etc.) |
| Length attribute | `max_length` constraint on `VariableMetadata` |
| SignificantDigits | `significant_digits` on `VariableMetadata` |
| Origin (CRF, Derived, etc.) | `origin` field |
| CodeListRef | `codelist_ref` linking to the associated codelist |
| ComputationalMethod | `computational_method` for derived variables |
| Role/RoleCodeListOID | `cdisc_role` (Identifier, Topic, etc.) |
| CodeList | `Codelist` object with all permitted values |
| Mandatory="Yes" | `required=True` on `VariableMetadata` |

### Controlled Terminology from Define-XML

Define-XML documents embed the codelists that constrain variable values. When Pointblank parses a
Define-XML, all codelists are extracted and linked to their respective variables. The
[`to_validate()`](`MetadataImport.to_validate`) method then generates `col_vals_in_set()` checks for
each variable that references a codelist:

```{python}
# Inspect codelists referenced by the Demographics domain
for cl_name, codelist in dm_meta.codelists.items():
    print(f"{cl_name}: {codelist.to_set()[:5]}...")  # first 5 values
    print(f"  Extensible: {codelist.extensible}")
```

The `extensible` flag records an important distinction. Non-extensible codelists require strict
adherence: any value not in the codelist is a conformance issue. Extensible codelists permit
sponsor-defined additions, so a value outside the published set is not necessarily an error.

Pointblank currently generates the same `col_vals_in_set()` check for both kinds of codelist. For
extensible codelists you will typically want to relax that step to reflect that additions are
allowed, for example by attaching a warning-level threshold to the step rather than treating every
out-of-set value as a hard failure.

## CDISC Controlled Terminology Import

Beyond the codelists embedded in Define-XML, CDISC publishes standalone Controlled Terminology
packages as XML files. These contain the canonical value sets for concepts like SEX, RACE,
ROUTE OF ADMINISTRATION, and hundreds of others. Pointblank can parse these directly.

Importing a CT package returns a `MetadataPackage` whose entries are keyed by codelist name. Each
entry is a `MetadataImport` that holds the codelist itself:

```{python}
# Import the bundled CDISC CT example
ct_path = pb.load_metadata_example("sdtm_ct.xml")
ct = pb.import_metadata(ct_path, format="cdisc_ct")

# List the codelists in the package
print(list(ct.keys()))

# Access a codelist by name
sex_codelist = ct["Sex"].codelists["Sex"]
print(f"SEX values: {sex_codelist.to_set()}")
print(f"Extensible: {sex_codelist.extensible}")
```

Once you have a codelist, its permitted values feed directly into a `col_vals_in_set()` check:

```{python}
# Use the CT-derived value set in a validation
validation = (
    pb.Validate(data=dm_data)
    .col_vals_in_set(columns="SEX", set=sex_codelist.to_set())
    .interrogate()
)
validation
```

Controlled Terminology packages version quarterly (e.g., 2024-03-29, 2024-06-28). Referencing a
specific version ensures reproducible validation results. In production pipelines, you would pin
the CT version to match what was specified in your study's Define-XML.

## SDTM Domain Templates

The Study Data Tabulation Model organizes clinical trial data into domains: Demographics (DM),
Adverse Events (AE), Laboratory Results (LB), Vital Signs (VS), and many others. Each domain has a
defined set of required and expected variables, with specific roles, types, and length constraints.

Pointblank includes built-in templates for eight commonly used SDTM domains. These templates encode
the structural requirements from the SDTM Implementation Guide 3.4 directly, so you can validate
data against the standard without needing a Define-XML file.

### Available Domains

```{python}
import pointblank as pb
from pointblank.metadata import list_sdtm_domains, get_sdtm_domain

# List all available SDTM domain templates
domains = list_sdtm_domains()
for d in domains:
    template = get_sdtm_domain(d)
    req_count = sum(1 for v in template.variables if v.required)
    print(f"  {d}: {template.label} ({req_count} required, {len(template.variables)} total vars)")
```

Each template provides the full variable specification for its domain, including which variables
are required (core="Req"), expected (core="Exp"), or permissible (core="Perm") per the
Implementation Guide.

### Inspecting a Domain Template

You can examine the variable specifications for any domain to understand what Pointblank will
check:

```{python}
# Get the Demographics domain template
dm = get_sdtm_domain("DM")

print(f"Domain: {dm.domain} - {dm.label}")
print(f"Class: {dm.domain_class}")
print(f"Repeating: {dm.repeating}")
print()

# Show required variables
print("Required variables (core='Req'):")
for var in dm.variables:
    if var.required:
        ct_info = f" [CT: {var.controlled_term}]" if var.controlled_term else ""
        print(f"  {var.name:12s} {var.dtype:4s} {var.role:12s} {var.label}{ct_info}")
```

### Structural Validation

The `validate_sdtm_structure()` function performs a quick check that a dataset contains all
required variables for its domain. This is useful as a fast pre-check before running the full
validation workflow:

```{python}
import polars as pl
from pointblank.metadata import validate_sdtm_structure

# A minimal Demographics dataset
dm_data = pl.DataFrame({
    "STUDYID": ["STUDY01"] * 4,
    "DOMAIN": ["DM"] * 4,
    "USUBJID": ["STUDY01-001", "STUDY01-002", "STUDY01-003", "STUDY01-004"],
    "SUBJID": ["001", "002", "003", "004"],
    "RFSTDTC": ["2024-01-15", "2024-01-20", "2024-02-01", "2024-02-10"],
    "RFENDTC": ["2024-06-15", "2024-06-20", "2024-07-01", "2024-07-10"],
    "SITEID": ["SITE01", "SITE01", "SITE02", "SITE02"],
    "AGE": [45, 62, 38, 55],
    "AGEU": ["YEARS"] * 4,
    "SEX": ["M", "F", "M", "F"],
    "RACE": ["WHITE", "BLACK OR AFRICAN AMERICAN", "ASIAN", "WHITE"],
    "ARMCD": ["DRUG", "PLACEBO", "DRUG", "PLACEBO"],
    "ARM": ["Active Drug 10mg", "Placebo", "Active Drug 10mg", "Placebo"],
    "COUNTRY": ["USA", "USA", "GBR", "GBR"],
})

result = validate_sdtm_structure(dm_data, domain="DM")
print(f"Valid: {result['valid']}")
if result["missing_required"]:
    print(f"Missing required: {result['missing_required']}")
if result["unknown_variables"]:
    print(f"Unknown variables: {result['unknown_variables'][:5]}")
```

### Full SDTM Validation

The `validate_sdtm()` function generates a comprehensive validation workflow that checks far more
than just structure. It produces a `Validate` object with checks for required variable non-nullness,
DOMAIN value correctness, sequence number positivity, string length constraints, and ISO 8601 date
formatting:

```{python}
from pointblank.metadata import validate_sdtm

# Generate and run the full SDTM DM validation
validation = validate_sdtm(data=dm_data, domain="DM").interrogate()
validation
```

The validation checks the following rules automatically:

| Check | Description |
|-------|-------------|
| Required variables non-null | Every variable with core="Req" must have no nulls |
| DOMAIN value | The DOMAIN column must contain only the expected domain code |
| Sequence numbers | `--SEQ` variables must be positive integers |
| String lengths | Character variables must not exceed their defined max length |
| ISO 8601 dates | All `--DTC` timing variables must match the CDISC date pattern |

### ISO 8601 Date Validation

CDISC uses a specific subset of ISO 8601 that allows partial dates. A date might be fully
specified as `2024-03-15T10:30:00` or partially specified as just `2024-03` (year and month known,
day unknown). The validation checks that all timing variables (`--DTC` columns like `RFSTDTC`,
`AESTDTC`, `LBDTC`) conform to this pattern:

```
Valid:   2024-03-15T10:30:00  (full datetime)
Valid:   2024-03-15            (date only)
Valid:   2024-03               (year-month only)
Valid:   2024                   (year only)
Invalid: 03/15/2024            (wrong format)
Invalid: 15-Mar-2024           (wrong format)
```

This catches a common data quality issue where dates are entered in locale-specific formats
rather than the required ISO 8601 pattern.

### Catching Conformance Problems

The examples so far have used clean data, so every check passes. The point of validation, of course,
is to surface problems. Here is the same DM domain with three deliberate defects: a missing
`USUBJID` (a required variable), a row where `DOMAIN` is `"AE"` instead of `"DM"`, and a `RFSTDTC`
value entered in the wrong date format:

```{python}
# A Demographics dataset with three injected conformance issues
dm_data_with_issues = pl.DataFrame({
    "STUDYID": ["STUDY01"] * 4,
    "DOMAIN": ["DM", "DM", "AE", "DM"],                       # row 3: wrong domain code
    "USUBJID": ["STUDY01-001", "STUDY01-002", None, "STUDY01-004"],  # row 3: required but null
    "SUBJID": ["001", "002", "003", "004"],
    "RFSTDTC": ["2024-01-15", "03/15/2024", "2024-02-01", "2024-02-10"],  # row 2: not ISO 8601
    "SITEID": ["SITE01", "SITE01", "SITE02", "SITE02"],
    "AGE": [45, 62, 38, 55],
    "AGEU": ["YEARS"] * 4,
    "SEX": ["M", "F", "M", "F"],
    "RACE": ["WHITE", "ASIAN", "WHITE", "ASIAN"],
    "ARMCD": ["DRUG", "PLACEBO", "DRUG", "PLACEBO"],
    "ARM": ["Active Drug 10mg", "Placebo", "Active Drug 10mg", "Placebo"],
})

validation = validate_sdtm(data=dm_data_with_issues, domain="DM").interrogate()
validation
```

Three steps now report a failing unit: the `col_vals_not_null()` check on `USUBJID`, the
`col_vals_in_set()` check on `DOMAIN`, and the ISO 8601 `col_vals_regex()` check on `RFSTDTC`. Each
failing step links to the specific rows so you can trace a conformance issue back to the offending
records.

### Converting SDTM Templates to MetadataImport

If you prefer to work with the standard `MetadataImport` interface (for example, to use
`to_schema()` or combine SDTM metadata with other sources), you can convert a domain template:

```{python}
from pointblank.metadata import sdtm_to_metadata

# Convert the DM template to a MetadataImport
dm_meta = sdtm_to_metadata(domain="DM", study_id="STUDY01")

print(f"Format: {dm_meta.source_format}")
print(f"Dataset: {dm_meta.dataset_name}")
print(f"Variables: {len(dm_meta.variables)}")

# Generate a schema from it
schema = dm_meta.to_schema()
print(f"Schema columns: {len(schema.columns)}")
```

## ADaM Dataset Templates

The Analysis Data Model builds on top of SDTM by adding derived variables, population flags,
and analysis-specific structures. ADaM datasets are the basis for statistical analyses in
clinical trials, and their structure is tightly specified to ensure reproducibility and
traceability back to the source data.

Pointblank includes templates for four ADaM dataset structures: **ADSL** (subject-level analysis),
**BDS** (Basic Data Structure for repeated measures), **ADAE** (adverse events analysis), and
**ADTTE** (time-to-event analysis).

### Available ADaM Datasets

```{python}
from pointblank.metadata import list_adam_datasets, get_adam_dataset

# List all available ADaM dataset templates
datasets = list_adam_datasets()
for d in datasets:
    template = get_adam_dataset(d)
    req_count = sum(1 for v in template.variables if v.required)
    flag_count = sum(1 for v in template.variables if v.is_population_flag)
    print(f"  {d}: {template.label}")
    print(f"      {req_count} required vars, {flag_count} population flags")
```

### ADSL: Subject-Level Analysis

ADSL is the foundational ADaM dataset. It contains one row per subject with all the key
demographic and treatment information needed for analysis. Every other ADaM dataset merges back
to ADSL for population definitions.

```{python}
adsl_template = get_adam_dataset("ADSL")
print(f"Dataset class: {adsl_template.dataset_class}")
print(f"\nPopulation flags:")
for var in adsl_template.variables:
    if var.is_population_flag:
        print(f"  {var.name}: {var.label}")
```

Population flags (SAFFL, ITTFL, EFFFL, etc.) define which subjects belong to each analysis
population. They must contain only the values "Y" or "N", with no nulls. Pointblank's ADaM
validation checks this automatically.

### Full ADaM Validation

The `validate_adam()` function generates comprehensive checks tailored to each dataset type:

```{python}
import polars as pl
from pointblank.metadata import validate_adam

# Create a minimal ADSL dataset
adsl_data = pl.DataFrame({
    "STUDYID": ["STUDY01"] * 5,
    "USUBJID": [f"STUDY01-{i:03d}" for i in range(1, 6)],
    "SUBJID": [f"{i:03d}" for i in range(1, 6)],
    "SITEID": ["SITE01", "SITE01", "SITE02", "SITE02", "SITE01"],
    "TRT01P": ["Drug A", "Placebo", "Drug A", "Placebo", "Drug A"],
    "TRT01A": ["Drug A", "Placebo", "Drug A", "Placebo", "Drug A"],
    "AGE": [45, 62, 38, 55, 48],
    "AGEU": ["YEARS"] * 5,
    "SEX": ["M", "F", "M", "F", "M"],
    "RACE": ["WHITE", "BLACK OR AFRICAN AMERICAN", "ASIAN", "WHITE", "WHITE"],
    "SAFFL": ["Y", "Y", "Y", "Y", "N"],
    "ITTFL": ["Y", "Y", "Y", "Y", "Y"],
    "EFFFL": ["Y", "Y", "N", "Y", "N"],
    "TRTSDT": ["2024-01-15", "2024-01-20", "2024-02-01", "2024-02-10", "2024-02-15"],
    "TRTEDT": ["2024-06-15", "2024-06-20", "2024-07-01", "2024-07-10", "2024-07-15"],
})

# Run ADaM ADSL validation
validation = validate_adam(data=adsl_data, dataset="ADSL").interrogate()
validation
```

### ADaM Validation Checks by Dataset Type

The checks generated by `validate_adam()` vary depending on the dataset type. Each type has its
own domain-specific rules in addition to the common required-variable and population-flag checks:

| Dataset | Specific Checks |
|---------|----------------|
| **ADSL** | TRT01P non-null, all population flags are Y/N |
| **BDS** | PARAMCD length at most 8 characters |
| **ADAE** | TRTEMFL is Y/N, AESEQ is positive |
| **ADTTE** | CNSR is 0 or 1, AVAL (time) is non-negative |

Here is an example validating a BDS (Basic Data Structure) dataset:

```{python}
# Create a minimal BDS dataset (e.g., ADLB - laboratory analysis)
bds_data = pl.DataFrame({
    "STUDYID": ["STUDY01"] * 6,
    "USUBJID": ["STUDY01-001"] * 3 + ["STUDY01-002"] * 3,
    "PARAMCD": ["ALT", "AST", "BILI"] * 2,
    "PARAM": [
        "Alanine Aminotransferase (U/L)",
        "Aspartate Aminotransferase (U/L)",
        "Bilirubin (umol/L)",
    ] * 2,
    "AVAL": [25.0, 30.0, 12.0, 45.0, 38.0, 15.0],
    "ABLFL": ["Y", "Y", "Y", "N", "N", "N"],
    "ANL01FL": ["Y"] * 6,
    "TRTA": ["Drug A"] * 3 + ["Placebo"] * 3,
})

validation = validate_adam(data=bds_data, dataset="BDS").interrogate()
validation
```

### Structural Validation

Like SDTM, ADaM provides a quick structural check via `validate_adam_structure()`:

```{python}
from pointblank.metadata import validate_adam_structure

result = validate_adam_structure(adsl_data, dataset="ADSL")
print(f"Valid: {result['valid']}")
print(f"Missing required: {result['missing_required']}")
print(f"Population flags present: {result['population_flags_found']}")
```

### Converting ADaM Templates to MetadataImport

The `adam_to_metadata()` function converts an ADaM template into the standard `MetadataImport`
format, giving you access to `to_schema()` and [`to_validate()`](`MetadataImport.to_validate`):

```{python}
from pointblank.metadata import adam_to_metadata

# Convert ADSL template to MetadataImport
adsl_meta = adam_to_metadata(dataset="ADSL", study_id="STUDY01")

print(f"Format: {adsl_meta.source_format}")
print(f"Version: {adsl_meta.source_version}")
print(f"Variables: {len(adsl_meta.variables)}")

# You can also use it through the import_metadata dispatcher
meta = pb.import_metadata("ADSL", format="cdisc_adam", dataset="ADSL")
print(f"Same result: {meta.dataset_name}")
```

## Frictionless Data Packages

While not a clinical standard, Frictionless Data Packages are widely used in open data and
research contexts. They describe tabular data with JSON schemas that specify column types,
constraints (minimum, maximum, enum, pattern), and primary keys. Pointblank imports these
seamlessly.

### Importing a Frictionless Schema

The bundled `datapackage.json` example describes a transactions table. Importing it yields a
`MetadataImport` with one `VariableMetadata` per column:

```{python}
# Import the bundled Frictionless Data Package example
datapackage_path = pb.load_metadata_example("datapackage.json")
meta = pb.import_metadata(datapackage_path, format="frictionless")

print(f"Dataset: {meta.dataset_name}")
for v in meta.variables:
    print(f"  {v.name:16s} {v.dtype}")
```

Frictionless constraints map directly onto Pointblank validation steps:

| Frictionless constraint | Pointblank step |
|-------------------------|-----------------|
| `"required": true` | `col_vals_not_null()` |
| `"unique": true` | `rows_distinct()` |
| `"minimum": 0` | `col_vals_ge(value=0)` |
| `"maximum": 100` | `col_vals_le(value=100)` |
| `"pattern": "..."` | `col_vals_regex(pattern="...")` |
| `"enum": [...]` | `col_vals_in_set(set=[...])` |

The constraint mapping is direct and complete. Every constraint expressible in a Frictionless
Table Schema has a corresponding Pointblank validation step, making the translation lossless.

A standalone Table Schema (without the surrounding data package wrapper) imports the same way with
`format="table_schema"`.

### CSVW (CSV on the Web)

The W3C's CSVW standard provides similar capabilities to Frictionless but uses JSON-LD and aligns
with linked data principles. Pointblank imports CSVW metadata with the same interface:

```{python}
# Import the bundled CSVW example
csvw_path = pb.load_metadata_example("weather_csvw.json")
meta = pb.import_metadata(csvw_path, format="csvw")

# CSVW column descriptors become VariableMetadata
print(f"Dataset: {meta.dataset_name} ({len(meta.variables)} variables)")
print("Columns:", ", ".join(v.name for v in meta.variables))
```

## Exporting Metadata

Pointblank can also export validation metadata in Frictionless format. This is useful when you want
to share data quality expectations with tools that understand the Frictionless ecosystem:

```{python}
import json
import tempfile
from pathlib import Path

# Read metadata from the bundled SAS Transport (XPT) example
xpt_path = pb.load_metadata_example("dm.xpt")
meta = pb.import_metadata(xpt_path, format="xpt")

# Export it as a Frictionless Table Schema
out_path = Path(tempfile.gettempdir()) / "dm_table_schema.json"
pb.export_metadata(meta, out_path, format="frictionless")

# Inspect the exported schema: how many fields, and the first two definitions
schema = json.loads(out_path.read_text())
print(f"Exported {len(schema['fields'])} field definitions. First two:")
print(json.dumps(schema["fields"][:2], indent=2))
```

The exported document contains the column definitions and constraints from the original metadata,
formatted as a valid Frictionless Table Schema that other tools can consume.

## Combining Multiple Metadata Sources

In practice, clinical data validation often combines metadata from multiple sources. The
Define-XML provides the authoritative variable definitions, but you might also want to check
against SDTM domain rules and controlled terminology packages. Pointblank supports this by
letting you compose validation workflows from different metadata sources:

```{python}
from pointblank.metadata import validate_sdtm

# Load the Define-XML for variable-level constraints
package = pb.import_metadata(pb.load_metadata_example("define.xml"), format="cdisc_define")
dm_meta = package["DM"]

# Generate validation from Define-XML metadata
validation = dm_meta.to_validate(data=dm_data)

# The SDTM template adds domain-specific rules not in the Define-XML
# (ISO 8601 checks, sequence number rules, etc.)
sdtm_validation = validate_sdtm(data=dm_data, domain="DM")

# Run both and compare how many checks each contributes
define_results = validation.interrogate()
sdtm_results = sdtm_validation.interrogate()

print(f"Define-XML checks: {len(define_results.validation_info)}")
print(f"SDTM template checks: {len(sdtm_results.validation_info)}")
```

This layered approach gives you the flexibility to apply different levels of validation depending
on your needs. The Define-XML checks enforce what was specifically documented for your study,
while the SDTM template checks enforce the broader standard requirements that apply universally.

## Null Flavors and Structured Missingness

Clinical data uses standardized HL7/CDISC *null flavors* to record *why* a value is absent (e.g.,
`"NASK"` = not asked, `"UNK"` = unknown, `"NA"` = not applicable). Pointblank ships a pre-built
[`MissingSpec`](`pointblank.MissingSpec`) for these codes via
`MissingSpec.from_cdisc_null_flavors()`:

```{python}
cdisc = pb.MissingSpec.from_cdisc_null_flavors()

print("NASK ->", cdisc.reason_for("NASK"))   # not_asked
print("UNK  ->", cdisc.reason_for("UNK"))     # unknown
print("boundary codes:", cdisc.values_for_category("boundary"))
```

This spec can be passed to `missing_vals_tbl()` for a reason-by-reason breakdown, or to the
`col_vals_*()` and dedicated missingness validation methods (`col_pct_missing()`,
`col_missing_coded()`, `col_missing_only_coded()`, `col_missing_consistent()`) to validate data
while accounting for the null flavor codes. See the *Missing Values Reporting* and *Validation
Methods* articles for the full set of capabilities.

## Conclusion

CDISC data validation with Pointblank covers the full spectrum of clinical trial data management:
from parsing Define-XML documents and controlled terminology packages to validating individual
datasets against SDTM and ADaM structural rules. The built-in domain templates encode years of
regulatory guidance into ready-to-use validation workflows, letting you check data compliance
with a single function call. For teams preparing regulatory submissions, this means catching
structural issues, date format errors, and terminology violations early in the data pipeline,
well before the formal submission review process begins.


### CDISC Submission Conformance

```{python}
#| echo: false
#| output: false
import pointblank as pb
pb.config(report_incl_footer_timings=False)
```

Regulatory submissions require that every dataset in a study package passes CDISC conformance
rules. Pointblank provides a built-in conformance engine that checks your SDTM datasets against the
full SDTMIG rule catalog with no external dependencies (no subprocess, no CLI, no network). For
those preparing a final submission gate, Pointblank can also delegate to the external
[CDISC CORE engine](https://github.com/cdisc-org/cdisc-rules-engine).

Two engines are available:

- **Built-in** (primary): Pointblank's own engine runs 362 SDTMIG 3.4 rules against your datasets
in-process. Results are immediately renderable in a notebook. No installation beyond Pointblank is
required.
- **CORE** (advanced): delegates to the open-source CDISC CORE CLI, which runs the authoritative
CDISC-certified rule set. Requires the CORE executable to be installed separately.

## SDTMIG Conformance

The built-in engine covers the complete SDTMIG 3.4 rule catalog, spanning per-record value checks,
variable metadata, dataset-level constraints, domain presence, and Define-XML cross-references. The
sections below walk through the three output surfaces (the tabular conformance report, the
record-level findings table, and the programmatic findings DataFrame) and describe the rule catalog,
controlled terminology handling, and optional inputs such as custom CT packages and Define-XML.

### Quick Start

[`validate_sdtmig()`](`pointblank.validate_sdtmig`) is the entry point for SDTMIG conformance.
Pass a dictionary mapping domain names to DataFrames and it returns a
[`ConformanceReport`](`pointblank.ConformanceReport`) that renders as a color-coded summary table
in Jupyter and Quarto notebooks:

```{python}
import polars as pl
import pointblank as pb

dm = pl.DataFrame({
    "STUDYID": ["STUDY01"] * 4,
    "DOMAIN":  ["DM"] * 4,
    "USUBJID": ["STUDY01-01-001", "STUDY01-01-002", "STUDY01-01-003", "STUDY01-01-004"],
    "SUBJID":  ["001", "002", "003", "004"],
    "RFSTDTC": ["2024-01-15", "2024-01-16", "2024-01-17", "2024-01-18"],
    "SEX":     ["M", "F", "M", "F"],
    "RACE":    ["WHITE", "ASIAN", "WHITE", "BLACK OR AFRICAN AMERICAN"],
    "ETHNIC":  ["NOT HISPANIC OR LATINO"] * 4,
    "ARMCD":   ["TRT", "PBO", "TRT", "PBO"],
    "ARM":     ["Treatment", "Placebo", "Treatment", "Placebo"],
    "COUNTRY": ["USA"] * 4,
    "AGE":     [45, 62, 38, 55],
    "AGEU":    ["YEARS"] * 4,
    "DMDTC":   ["2024-01-10"] * 4,
    "DMDY":    [1] * 4,
    "SITEID":  ["01"] * 4,
})

report = pb.validate_sdtmig({"DM": dm})
report
```

Domain keys are matched case-insensitively (`"dm"` and `"DM"` are equivalent). Polars, pandas, and
any other narwhals-compatible DataFrame are accepted. The report object itself renders the tabular
summary in a notebook; read on to learn how to drill into failures and access findings
programmatically.

### The Conformance Report

The tabular report shown above is produced by
[`get_tabular_report()`](`pointblank.ConformanceReport.get_tabular_report`). Each row is one rule
from the SDTMIG 3.4 catalog; the colored bar on the left edge encodes its status:

| Color | Status | Meaning |
|---|---|---|
| Green | `pass` | The rule was evaluated and no violations were found |
| Red | `fail` | One or more records or datasets violated the rule |
| Amber | `error` | The rule could not be evaluated (unexpected data state) |
| Grey | `not_applicable` | The rule requires an input (such as a Define-XML) that was not supplied |
| Grey | `not_supported` | The rule type is not yet implemented in the built-in engine |

Rules are sorted by severity: failures first, then errors, then passes, then not-applicable.

The header shows a `PASS` or `FAIL` badge alongside the standard, version, and a full status
breakdown (e.g., `SDTMIG 3-4 · 346 passed · 4 failed · 12 n/a`).

Call `get_tabular_report()` to get the `GT` object directly if you need to embed it in a report
pipeline:

```{python}
#| eval: false
gt = report.get_tabular_report()
gt
```

The `GT` object can be passed to any Great Tables export method (for example, `gt.as_raw_html()` to
embed the table in a custom HTML report, or `gt.gtsave("report.png")` to render it as an image).
When failures are present, use the findings surfaces below to investigate the specific records that
triggered each rule.

### Findings Drill-Down

When rules fail, the tabular report shows how many records violated each rule but not which ones.
[`get_findings_table()`](`pointblank.ConformanceReport.get_findings_table`) provides that
drill-down: one row per failing record, showing the subject, the specific column that violated the
rule, the offending value, and the 1-based row number in the source dataset.

To see it in action, introduce a few deliberate violations:

```{python}
dm_with_issues = pl.DataFrame({
    "STUDYID": ["STUDY01"] * 4,
    "DOMAIN":  ["DM"] * 4,
    "USUBJID": ["STUDY01-01-001", "STUDY01-01-002", "STUDY01-01-003", "STUDY01-01-004"],
    "SUBJID":  ["001", "002", "003", "004"],
    "RFSTDTC": ["2024-01-15", "2024-01-16", "2024-01-17", "2024-01-18"],
    "SEX":     ["M", "F", "UNKNOWN", "F"],      # "UNKNOWN" is not in the SEX codelist
    "RACE":    ["WHITE", "ASIAN", "WHITE", "BLACK OR AFRICAN AMERICAN"],
    "ETHNIC":  ["NOT HISPANIC OR LATINO"] * 4,
    "ARMCD":   ["TRT", "PBO", "TRT", "PBO"],
    "ARM":     ["Treatment", "Placebo", "Treatment", "Placebo"],
    "COUNTRY": ["USA"] * 4,
    "AGE":     [45, -5, 38, 55],               # -5 violates AGE >= 0
    "AGEU":    ["YEARS"] * 4,
    "DMDTC":   ["2024-01-10", "not-a-date", "2024-01-10", "2024-01-10"],  # invalid ISO date
    "DMDY":    [1] * 4,
    "SITEID":  ["01"] * 4,
})

report_with_issues = pb.validate_sdtmig({"DM": dm_with_issues})
report_with_issues.get_findings_table()
```

The findings table groups its columns into two spanners:

- **SDTM Rule Definition**: `Rule`, `Domain` and `Description` identify which rule fired and where.
- **Finding**: `USUBJID` identifies the subject; `Column` and `Value` show exactly what was wrong;
`Row` is the 1-based row number in the source dataset for quick lookup.

At most 100 findings per rule are shown in the table. The true total for each rule is always visible
in `get_tabular_report()`. To export findings to a spreadsheet or join them back to your source
data, use `findings_df()` instead.

### Programmatic Access with findings_df()

[`findings_df()`](`pointblank.ConformanceReport.findings_df`) returns the same findings as a Polars
DataFrame, which is better suited for filtering, grouping, exporting to CSV, or joining back to the
source data:

```{python}
df = report_with_issues.findings_df()
df
```

The DataFrame schema is:

| Column | Description |
|---|---|
| `rule_id` | CDISC CORE rule identifier (e.g., `"SDTM-007"`) |
| `dataset` | SDTM domain the failing record belongs to |
| `row_index` | 0-based row position in the source dataset |
| `usubjid` | Unique Subject Identifier (`USUBJID`) for the failing record |
| `checked_column` | The variable that violated the rule (e.g., `"SEX"`) |
| `checked_value` | The actual value found in `checked_column` |
| `description` | Human-readable rule description |

```{python}
import polars as pl

# Filter to a specific rule
df.filter(pl.col("rule_id") == "SDTM-007")
```

```{python}
# Group by rule to count violations
df.group_by("rule_id").agg(pl.len().alias("n_violations")).sort("n_violations", descending=True)
```

The DataFrame is empty (with the same schema) when all rules pass, so it is safe to call
unconditionally in a script. Use it to triage failures before running the more time-intensive CDISC
CORE engine, or to generate a custom summary report tailored to your team's workflow.

### What the Rule Catalog Covers

The bundled SDTMIG 3.4 catalog contains 362 rules across seven types:

| Type | Description | Examples |
|---|---|---|
| `RECORD_CHECK` | Per-row value checks | Codelist membership, ISO 8601 dates, numeric ranges |
| `VARIABLE_METADATA_CHECK` | Variable presence, types and ordering | DM must contain SEX, RACE and COUNTRY |
| `DATASET_METADATA_CHECK` | Identifier variables present in every dataset | STUDYID must be present in every domain |
| `DATASET_CONTENTS_CHECK` | Dataset-level value constraints | All rows in a domain must share the same STUDYID |
| `DOMAIN_PRESENCE_CHECK` | Required domain presence | DM, TS and TA must be present |
| `DEFINE_ITEM_METADATA_CHECK` | Variable declarations against Define-XML | Activated when `define_xml` is supplied |
| `DEFINE_CODELIST_CHECK` | Codelist values against Define-XML | Activated when `define_xml` is supplied |

Only `RECORD_CHECK`, `DATASET_CONTENTS_CHECK` and `DEFINE_CODELIST_CHECK` rules produce row-level
findings accessible via `findings_df()` and `get_findings_table()`. The other types report a violation count in
`get_tabular_report()` but do not have individual record detail.

Rules that target a domain not present in your datasets have nothing to check, so they are reported
as `pass`. For example, a rule that checks `AESTDTC` in the AE domain passes when no AE dataset is
supplied. A `pass` therefore doesn't mean that a domain was checked: adding more domains to the
dictionary passed to `validate_sdtmig()` gives the rules for those domains something to evaluate,
giving a more complete conformance picture. Only rules that need an input that wasn't supplied
(such as a Define-XML) are marked `not_applicable`.

### Controlled Terminology

Codelist checks use the bundled CT package `sdtm-ct-2024-09-27`. Two important behaviors:

- **Case-insensitive matching**: a value of `"beats/min"` matches the codelist term `"BEATS/MIN"`.
This avoids false positives for studies that applied mixed-case CT values.
- **SAS/XPT empty strings treated as null**: SAS Transport files encode character missing values as
`""` (empty string) rather than `None`. The engine recognizes this and skips codelist and format
checks for such cells, preventing the large volumes of false positives that occur with a naive
string comparison.

Both behaviors apply automatically with no configuration. If you need to pin a different CT version
or supply additional packages, see the next section.

### Supply a Custom CT Package

By default the most recent bundled CT package is used. Pass `ct_packages` to pin a specific version
or supply additional packages:

```{python}
#| eval: false
report = pb.validate_sdtmig(
    {"DM": dm},
    ct_packages=["sdtm-ct-2024-09-27"],
)
```

Pinning the CT version is useful when your study was locked against a specific CDISC CT release and
you want the conformance check to reflect that snapshot rather than the latest terms.

### Activating Define-XML Rules

Pass a path to `define.xml` to activate the Define-XML-aware rule types:

```{python}
#| eval: false
report = pb.validate_sdtmig(
    {"DM": dm, "AE": ae},
    define_xml="path/to/define.xml",
)
```

Without `define_xml`, the 12 rules that need it (the `DEFINE_ITEM_METADATA_CHECK` and
`DEFINE_CODELIST_CHECK` rules, and two `VARIABLE_METADATA_CHECK` rules) are marked
`not_applicable`. Supplying the file allows the engine to verify that every variable present
in your datasets is declared in Define-XML with the correct type and, where applicable, a valid
codelist.

Define-XML also tells the CT checks about sponsor extensions. Many CDISC codelists are extensible
(EPOCH, FREQ, ROUTE, and UNIT among them): a sponsor may add terms, provided Define-XML declares
them with `def:ExtendedValue="Yes"` on the codelist item. When `define_xml` is supplied, such
declared terms pass the codelist checks for extensible codelists. For example, a crossover study can
use the epochs `TREATMENT 1` and `TREATMENT 2`. Without Define-XML, or for a codelist that isn't
extensible (such as SEX), terms outside the codelist are still reported.

### Loading XPT Files

For datasets stored as SAS Transport (XPT) files, use `pyreadstat` to load them before passing to
`validate_sdtmig()`. Install it with `pip install pyreadstat`:

```{python}
#| eval: false
import pyreadstat
import polars as pl
import pointblank as pb

def load_xpt(path: str) -> pl.DataFrame:
    df, _ = pyreadstat.read_xport(path)
    return pl.from_pandas(df)

report = pb.validate_sdtmig(
    {
        "DM": load_xpt("sdtm/dm.xpt"),
        "AE": load_xpt("sdtm/ae.xpt"),
        "LB": load_xpt("sdtm/lb.xpt"),
        "VS": load_xpt("sdtm/vs.xpt"),
    },
    study_id="STUDY01",
)
```

Pointblank handles the SAS empty-string convention automatically, so XPT data does not require any
preprocessing before being passed to `validate_sdtmig()`. The same built-in handling applies whether
data arrives from XPT files, in-memory DataFrames, or any other narwhals-compatible source.

## Prerequisites

`validate_sdtmig()` and the `ConformanceReport` methods require no additional dependencies beyond
Pointblank itself. The optional extras are:

- **Reading XPT files** with `pyreadstat`: `pip install pyreadstat`
- **Writing XPT files** for CORE (automatic, handled internally): `pip install pyreadstat`
- **Excel export** with `to_excel()`: `pip install openpyxl`

```bash
pip install pointblank[cdisc]    # adds pyreadstat
pip install pointblank[excel]    # adds openpyxl
```

The CDISC CORE engine is not a Python dependency of Pointblank and must be installed separately when
needed. See [Installing the CDISC CORE Engine](#installing-the-cdisc-core-engine) below.

## CDISC CORE Engine (Advanced)

The CDISC CORE engine runs the full authoritative conformance rule set and produces reports accepted
by FDA and PMDA review tools. Use it as the final pre-submission gate after the built-in engine
gives a clean result.

### Installing the CDISC CORE Engine

CORE can be obtained in three ways depending on your environment. The standalone executable is
easiest for local use, Docker is preferred in containerized CI pipelines, and the repo checkout is
useful when you need a specific unreleased version or want to inspect the rule definitions directly.

**Option 1: Standalone executable.** Download the pre-built binary from the
[CDISC CORE releases page](https://github.com/cdisc-org/cdisc-rules-engine/releases). Place it
somewhere on your `PATH` under the name `core`:

```bash
chmod +x core
sudo mv core /usr/local/bin/
core --version
```

Once the binary is on `PATH`, Pointblank will discover it automatically with no additional
configuration required.

**Option 2: Docker.** CDISC publishes an official Docker image that includes the engine and its
full rules cache:

```bash
docker pull cdisc/cdisc-rules-engine:latest

docker run --rm \
    -v /path/to/study/data:/data \
    cdisc/cdisc-rules-engine:latest \
    validate -s sdtmig -v 3-4 -d /data -of JSON -o /data/report
```

When running via Docker you will typically invoke CORE directly rather than through Pointblank's
subprocess wrapper. The resulting JSON output can be loaded with `parse_core_report()` and wrapped
in a `ConformanceReport` for further analysis.

**Option 3: Repo checkout.**

```bash
git clone https://github.com/cdisc-org/cdisc-rules-engine.git
cd cdisc-rules-engine
pip install -r requirements.txt
python core.py --version
```

When using a repo checkout, pass the repo root as `core_cwd` so Pointblank sets the subprocess
working directory correctly.

### Telling Pointblank Where to Find CORE

Pointblank discovers CORE through three mechanisms, tried in order:

1. An explicit `core=` argument to `validate_cdisc_submission()` or
`validate_conformance(engine="core")`.
2. The `POINTBLANK_CDISC_CORE` environment variable (e.g.,
`export POINTBLANK_CDISC_CORE="python /path/to/core.py"`).
3. A `core` or `cdisc-rules-engine` executable on `PATH`.

The environment variable approach is convenient for CI systems where the CORE path differs between
machines, while the explicit argument is useful in notebooks where you want the path to be
self-documenting.

### Running CORE

`validate_cdisc_submission()` is the simplest way to run a CORE validation. It accepts an in-memory
dictionary of DataFrames, a folder path, or an existing `SubmissionPackage`:

```{python}
#| eval: false
report = pb.validate_cdisc_submission(
    {"DM": dm, "AE": ae},
    standard="sdtmig",
    version="3.4",
    agency="FDA",
)
```

For a folder of XPT files pass the path directly. Pointblank skips the materialization step and
passes the folder straight to CORE:

```{python}
#| eval: false
report = pb.validate_cdisc_submission(
    "path/to/sdtm/",
    standard="sdtmig",
    version="3.4",
    agency="FDA",
)
```

For repo-checkout CORE, pass the command prefix and working directory:

```{python}
#| eval: false
report = pb.validate_cdisc_submission(
    {"DM": dm},
    standard="sdtmig",
    version="3.4",
    core=["python", "/path/to/cdisc-rules-engine/core.py"],
    core_cwd="/path/to/cdisc-rules-engine",
    cache="/path/to/cdisc-rules-engine/resources/cache",
)
```

In all cases the return value is a `ConformanceReport` with `is_core=True`. The sections below
describe the accessor methods available on a CORE report.

### Working with a CORE ConformanceReport

The examples below use a captured real CORE report so that the code runs without requiring CORE to
be installed in the docs environment. The structure is identical to what a live run produces.

```{python}
import json
from pathlib import Path
from pointblank.metadata import parse_core_report, ConformanceReport

_fixtures = Path(pb.__file__).parent.parent / "tests" / "metadata_fixtures" / "cdisc_core"
raw = json.loads((_fixtures / "core_report_full.json").read_text())

report = ConformanceReport.from_core_report(raw, agency="FDA")
print(report)
```

**Overall result.** [`all_passed()`](`ConformanceReport.all_passed`) returns `True` when no rules
reported issues; `is_core` confirms the report originated from the CORE engine rather than the
built-in engine:

```{python}
print("All passed:", report.all_passed())
print("Is CORE report:", report.is_core)
```

**Summary dictionary.** [`summary()`](`ConformanceReport.summary`) returns run provenance and rule
counts as a plain dictionary, useful for logging, assertions in CI scripts, or building a custom
status page:

```{python}
s = report.summary()

print(f"Standard:        {s['standard']} {s['version']}")
print(f"Engine:          {s['engine_version']}")
print(f"Rules evaluated: {s['n_rules']}")
print(f"Total issues:    {s['n_issues']}")
print()

print("Rule status breakdown:")
for status, count in sorted(s["status_counts"].items()):
    print(f"  {status:20s}: {count}")
```

The `status_counts` dictionary enumerates every status bucket CORE reported. Use it to build
pass/fail thresholds or trend charts across multiple submission runs.

**Issues.** `issues()` returns one record per (dataset, rule) pair that reported at least one
issue. Pass a `status` filter to separate conformance violations from execution errors:

```{python}
issues = report.issues()
print(f"Issue entries: {len(issues)}")

for issue in issues[:3]:
    print()
    print(f"  Dataset:  {issue['dataset']}")
    print(f"  Rule:     {issue['rule_id']}")
    print(f"  Message:  {issue['message']}")
    print(f"  Issues:   {issue['issues']}")
```

```{python}
from pointblank.metadata._cdisc_core import STATUS_ISSUE, STATUS_ERROR

reported_issues = report.issues(status=STATUS_ISSUE)
print(f"Rules with issues: {len(reported_issues)}")

exec_errors = report.issues(status=STATUS_ERROR)
print(f"Execution errors:  {len(exec_errors)}")
```

Execution errors (`STATUS_ERROR`) indicate that CORE could not evaluate a rule, typically because
a required variable or dataset was absent or had an unexpected format. They are distinct from
conformance issues (`STATUS_ISSUE`) where the rule ran successfully but found a violation.

**Row-level findings.** `findings()` returns the row-level detail from CORE's `Issue_Details`
section, giving the exact record that triggered each rule:

```{python}
findings = report.findings()
print(f"Row-level findings: {len(findings)}")

f = findings[0]
print()
print(f"Rule:          {f.rule_id}")
print(f"Dataset:       {f.dataset}")
print(f"Row:           {f.row}")
print(f"USUBJID:       {f.usubjid}")
print(f"Variables:     {f.variables}")
print(f"Values:        {f.values}")
```

Each finding carries the `USUBJID`, the affected variables and their values, and the 1-based row
number in the source dataset, matching the information shown in the built-in engine's findings table.

**Per-rule run results.** `rules()` returns one `CoreRuleResult` per rule with its run status:

```{python}
from pointblank.metadata._cdisc_core import STATUS_SUCCESS, STATUS_SKIPPED

all_rules = report.rules()
print(f"Total rules:  {len(all_rules)}")
print(f"Successful:   {len(report.rules(status=STATUS_SUCCESS))}")
print(f"Skipped:      {len(report.rules(status=STATUS_SKIPPED))}")

failing = [r for r in all_rules if r.is_failing]
print(f"Failing:      {len(failing)}")
for r in failing:
    print(f"  {r.rule_id:15s} [{r.status}]  {(r.message or '')[:60]}")
```

Rules marked skipped are not failures. CORE skips rules when the dataset or variable they check is
absent from the submission. A skipped rule is equivalent to `not_applicable` in the built-in engine
and should not be treated as a conformance problem.

## Exporting Reports

Both built-in and CORE reports can be exported for archiving, sharing with a biometrics team, or
loading into a review tool. Two formats are supported: JSON for round-trippable machine-readable
output, and Excel for spreadsheet-based review workflows.

### JSON

[`to_json()`](`ConformanceReport.to_json`) saves the report as a JSON file. For CORE reports the
structure mirrors CORE's native output and is parseable by `parse_core_report()`:

```{python}
import tempfile

with tempfile.TemporaryDirectory() as tmp:
    dest = report.to_json(Path(tmp) / "conformance_report.json")
    print(f"Written to: {dest.name}")
    print(f"File size:  {dest.stat().st_size:,} bytes")

    reloaded = json.loads(dest.read_text())
    reparsed = parse_core_report(reloaded)
    print(f"Rules after round-trip: {len(reparsed.rules)}")
```

The round-trip fidelity means a JSON file written by Pointblank can be re-loaded later without
re-running CORE, which is useful for archiving the state of a submission at a specific point in time.

### Excel

`to_excel()` writes the report as a workbook. Requires `openpyxl`:

```{python}
#| eval: false
dest = report.to_excel("conformance_report.xlsx")
```

For CORE reports the workbook has four sheets: `Issue_Summary`, `Issue_Details`, `Rules_Report`,
and `Conformance_Details`.

## Using Both Engines Together

The built-in engine provides fast feedback during development; CORE is the final gate. The typical
workflow is to iterate with the built-in engine until it reports a clean result, then run CORE once
as a pre-submission check:

```{python}
#| eval: false
# Step 1: fast iteration with the built-in engine
report = pb.validate_sdtmig({"DM": dm, "AE": ae}, study_id="STUDY01")

if not report.all_passed():
    print("Fix rule violations before running CORE:")
    df = report.findings_df()
    print(df.group_by("rule_id").agg(pl.len().alias("n")).sort("n", descending=True))
    raise SystemExit(1)

# Step 2: final gate with CDISC CORE
core_report = pb.validate_cdisc_submission(
    {"DM": dm, "AE": ae},
    standard="sdtmig",
    version="3.4",
    agency="FDA",
)

if not core_report.all_passed():
    failing = [r for r in core_report.rules() if r.is_failing]
    print(f"CORE found {len(failing)} failing rules.")
    core_report.to_json("core_report.json")
    raise SystemExit(1)

print("Submission passed all conformance checks.")
```

This pattern catches the majority of conformance problems early (when fixing them is cheap) and
reserves the slower CORE invocation for the final verification step. Both reports can be exported
and archived alongside the submission package.

## Running the Integration Tests

Pointblank ships integration tests for the full CORE pipeline. They are skipped automatically when
CORE is not discoverable:

```bash
# Run when CORE is on PATH:
pytest -m cdisc_core

# With a repo-checkout CORE:
export POINTBLANK_CDISC_CORE="python /path/to/core.py"
export POINTBLANK_CDISC_CORE_CWD="/path/to/cdisc-rules-engine"
pytest -m cdisc_core
```

The integration tests validate the full subprocess pipeline end-to-end, including XPT materialization,
CORE invocation, JSON parsing, and report construction. Running them against your local CORE
installation is a good sanity check after upgrading either Pointblank or the CORE engine.


### Glossary

This page explains the terms used throughout the Pointblank documentation, in alphabetical order.
Where a term matches something in Pointblank's API, the name of the class, function, or method is
shown in `code`. Terms in **bold** have their own entry on this page.

Action
: Something Pointblank does when a step's failures reach a **threshold** level, such as printing a
  message or running a function you supply. You set actions with the `Actions` class, either for
  the whole validation (`Validate(actions=...)`) or for a single step. Messages can include
  placeholders like `{step}`, `{col}`, and `{level}`, which are filled in when the action runs. By
  default, only the action for the most serious level reached is run. Inside a function used as an
  action, `get_action_metadata()` gives details such as the step number, the column, and the level
  reached. See also **final action**. [Learn more](../validation-plan/actions.qmd).

Active / inactive step
: Whether a **validation step** runs. A step with `active=False` is skipped, but it still appears in
  the **validation report** so that step numbers don't shift. You can also give `active=` a
  function that decides at run time. The helpers `has_columns()` and `has_rows()` make this easy:
  for example, `active=pb.has_columns("email")` runs the step only if the table has an `email`
  column.

ADaM (Analysis Data Model)
: A **CDISC** standard for the analysis-ready datasets used in clinical trials, which are built from
  **SDTM** data. Common examples are ADSL (one row per trial subject) and ADAE (adverse events).
  Pointblank can check that an ADaM dataset has the columns the standard requires, using
  `validate_adam_structure()` or `validate_adam()`. [Learn
  more](../metadata-import/cdisc-validation.qmd).

Aggregate check
: A check on a summary of a whole column, such as its sum, average, or standard deviation, instead
  of on each value. Examples include `col_sum_gt()`, `col_avg_eq()`, and `col_sd_lt()`. Because
  the whole column is summarized into one number, an aggregate check has a single **test unit**.
  Use `tol=` to allow some leeway (see **tolerance**), and set **reference data** to compare with
  the same summary of another table. [Learn more](../validation-plan/validation-methods.qmd).

AI check
: A **validation step** where a large language model (LLM) reads each row and judges whether it
  follows a rule you describe in plain words. This helps with rules that are hard to write as code,
  like "the description should mention the product's color". Added with `prompt()`. [Learn
  more](../validation-plan/validation-methods.qmd).

AI Validation Editor
: A tool that changes an existing **validation plan** based on a plain-English instruction, such as
  "also check that the email column has no missing values". It uses an LLM to write the revised
  plan. With the `EditValidation` class you can see what changed (`diff()` or `review()`) and then
  get the new plan with `accept()`. It's also available from the **CLI** as `pb edit`. See also
  **DraftValidation**. [Learn more](../advanced-validation/ai-validation-editor.qmd).

Assertion
: A way to stop your code when data doesn't pass validation. After building a plan, call
  `assert_passing()` to raise an error if any **test unit** failed, or `assert_below_threshold()`
  to raise an error if a **threshold** level was reached. Both run **interrogation** first if it
  hasn't happened yet. This is handy in data pipelines and tests. There's also
  `assert_dimension_scores()`, which raises an error if the score for any **quality dimension** is
  below a minimum you set. If you'd rather get `True` or `False` than an error, use a **status
  check**. [Learn more](../advanced-validation/assertions.qmd).

Assistant
: A chat window where you can ask an LLM questions about your data and about Pointblank. You start
  it with `assistant()`, and it opens in your web browser or in the terminal. If you give it a
  table, a summary of that table is shared with the model so its answers fit your data.

Backend
: The library or database that holds your table. Pointblank works with Polars, Pandas, and PySpark
  tables, and (through the Ibis library) with databases such as DuckDB, PostgreSQL, MySQL, SQLite,
  SQL Server, Snowflake, Databricks, and BigQuery. It can also read CSV and Parquet files directly,
  and database tables given as a **connection string**. You don't need to tell Pointblank which
  backend you're using, since it works this out from the data you pass in. See also **Narwhals**.
  [Learn more](../getting-started/installation.qmd).

Boundary enforcement
: See **pipeline**.

Brief
: A short description of what a **validation step** checks, shown in the STEP column of the
  **validation report**. You can write your own (`brief="..."`) or let Pointblank write one for you
  (`brief=True`). Briefs can include placeholders such as `{col}` (the column name), `{step}` (the
  step number), and `{auto}` (Pointblank's generated description). Generated briefs use the
  **report language**. [Learn more](../validation-plan/briefs.qmd).

CDISC (Clinical Data Interchange Standards Consortium)
: The organization that sets data standards for clinical trials, including **SDTM**, **ADaM**, and
  **Define-XML**. Pointblank can read CDISC metadata, check that datasets have the expected
  structure, and check a set of trial datasets against CDISC's rules. It can do that last part with
  its own built-in rules (`validate_sdtmig()`) or by running the **CORE** engine. [Learn
  more](../metadata-import/cdisc-validation.qmd).

CLI (command-line interface)
: The `pb` command, which you run in a terminal to use Pointblank without writing Python. `pb
  preview`, `pb scan`, `pb missing`, and `pb info` let you look at a table. `pb validate` runs quick
  one-off checks, and `pb run` runs a full validation written in Python or YAML. `pb compare` runs
  a **table comparison**, `pb make-template` creates a starter file, and `pb edit` opens the **AI
  Validation Editor**. See also **exit code**. [Learn
  more](../../reference/cli/index.qmd).

Coded missing value
: A stand-in value, such as `-99` or `"UNK"`, that means a value is missing for a known reason (for
  example, the person refused to answer or wasn't asked). The `MissingSpec` class lists which codes
  a column uses and what each one means. Validation methods like `col_missing_coded()` and the
  **missing values table** can then treat these codes as missing data. See also **null flavor**.
  [Learn more](../data-inspection/missing-vals-tbl.qmd).

Codelist
: A named list of allowed values, often with a label for each (for example, `"M"` means Male and
  `"F"` means Female). Codelists usually come from a **metadata import**, such as CDISC **controlled
  terminology** or SPSS value labels. Pointblank represents them with the `Codelist` class. [Learn
  more](../metadata-import/metadata-import.qmd).

Column comparison
: Checking each value in a column against the value of another column in the same row, rather than
  against a fixed value. Wrap the other column's name in `col()`: for example,
  `col_vals_lt(columns="a", value=pb.col("c"))` checks that each value of `a` is less than the value
  of `c` in the same row. [Learn more](../validation-plan/validation-methods.qmd).

Column selection
: How you tell a **validation step** which columns to check. You can give a single column name, a
  list of names, or a helper that picks columns for you: `starts_with()`, `ends_with()`,
  `contains()`, [`matches()`](`pointblank.matches`), `everything()`, `first_n()`, or `last_n()`.
  Helpers can be combined inside `col()` using `&` (and), `|` (or), and `-` (but not). Each column
  that's picked gets its own step. [Learn more](../validation-plan/column-selection-patterns.qmd).

Column summary table
: A table that describes each column of your data: its type, how many values are missing, how many
  are distinct, and statistics like the mean and quartiles. You make one with `col_summary_tbl()`.
  See also **DataScan**, **preview**, and **missing values table**. [Learn
  more](../data-inspection/col-summary-tbl.qmd).

Comparison
: See **table comparison**.

Composable steps
: A reusable set of **validation steps** that isn't tied to any particular table. You build the set
  with the `Steps` class, using the same method names as `Validate`, and add it to any validation
  with `Validate.add_steps()`. This lets you write common checks once and reuse them for many
  tables. [Learn more](../advanced-validation/composable-steps.qmd).

Conformance report
: The results of checking a set of clinical trial datasets against **CDISC** rules, given by
  `validate_sdtmig()` and `validate_cdisc_submission()` as a `ConformanceReport`. Its table shows
  one row per rule, with a status of pass, fail, error, not applicable, or not supported. The
  *findings* (one for each record that broke a rule, with the column and value at fault) are
  available as a table with `get_findings_table()` or as a DataFrame with `findings_df()`. [Learn
  more](../metadata-import/cdisc-submission-conformance.qmd).

Conjointly
: A single **validation step** that checks several conditions on each row at once. A row passes
  only if every condition is true for that row. This is different from adding one step per
  condition, where each condition is judged on its own. Added with `conjointly()`. [Learn
  more](../validation-plan/validation-methods.qmd).

Connection string
: A piece of text that points to a database table, such as
  `"duckdb:///path/to/db.ddb::table_name"`. The part before `::` says how to reach the database, and
  the part after it names the table. You can pass a connection string as the data for `Validate`
  or for **CLI** commands, instead of a table you've already loaded. [Learn
  more](../the-pointblank-cli/cli-data-inspection.qmd).

Contract
: A written-down agreement about what a table should look like. It combines a **schema** (the
  expected columns and their types), a list of rules, and details like the owner and version. You
  create one with the `Contract` class and check a table against it with `.validate(data)`.
  Contracts can be saved as YAML, imported from other formats (such as JSON Schema, Frictionless,
  dbt, and ODCS) with [`import_contract()`](`pointblank.import_contract`), and exported with
  [`export_contract()`](`pointblank.export_contract`). See also **pipeline**, **contract import**,
  and **violation handling**. [Learn more](../contracts-and-pipelines/contracts.qmd).

Contract adapter
: The part of Pointblank that translates between a **contract** and one outside format. There are
  built-in adapters for JSON Schema, Frictionless, dbt, and ODCS, and `list_adapters()` shows which
  ones are available. For another format, you can write your own adapter by making a subclass of
  `ContractAdapter` and registering it with `@register_adapter`.
  [`import_contract()`](`pointblank.import_contract`) and
  [`export_contract()`](`pointblank.export_contract`) can then use it like any other. [Learn
  more](../contracts-and-pipelines/custom-adapters.qmd).

Contract import
: What you get back from [`import_contract()`](`pointblank.import_contract`): a `ContractImport`
  object holding the checks that were translated from an outside format. Nothing is checked yet. Its
  `coverage` is the share of the outside rules that could be translated, and `warnings` lists any
  that couldn't. From there, `to_validate(data)` gives a **validation plan**, `to_contract()` gives
  a **contract**, and `to_python()` or [`to_yaml()`](`ContractImport.to_yaml`) give code. [Learn
  more](../contracts-and-pipelines/importing-contracts.qmd).

Controlled terminology (CT)
: CDISC's published lists of the allowed values for clinical trial data, such as the codes for sex,
  race, or units of measure. They're released as dated packages a few times a year, so it's good
  practice to use the same version your study was set up with. `import_metadata(...,
  format="cdisc_ct")` reads a package as a set of **codelists**, and `validate_sdtmig()` checks
  values against a package that comes with Pointblank (or one you pick with `ct_packages=`). Some
  codelists are *extensible*, meaning a study may add its own values. [Learn
  more](../metadata-import/cdisc-validation.qmd).

CORE (CDISC Conformance Rules Engine)
: CDISC's official, open-source program for checking clinical trial data against CDISC's rules.
  You install it separately, and `validate_cdisc_submission()` runs it for you and turns its output
  into a `ConformanceReport`. Pointblank also has built-in rules that don't need CORE (see
  **CDISC**). [Learn more](../metadata-import/cdisc-submission-conformance.qmd).

Critical
: See **warning, error, and critical**.

Custom check
: A **validation step** that uses a function you write, for rules that none of the built-in
  validation methods cover. Your function receives the table and returns `True` or `False` values
  showing what passed. Added with `specially()`. [Learn
  more](../validation-plan/validation-methods.qmd).

Data coherence
: In **test data generation**, the way related columns are made to fit together within each row.
  For example, an email address is made from the person's name, and the address, city, postcode,
  phone number, and coordinates all belong to the same place. This happens on its own when related
  **presets** are used together, and it still holds when rows from several countries are mixed.
  [Learn more](../test-data-generation/country-specific-data.qmd#data-coherence).

Data extract
: The rows that failed a **validation step**, saved during **interrogation** so you can look at
  them afterwards. You get them with `Validate.get_data_extracts()`. Only steps that check values
  row by row produce extracts. Options on `interrogate()`, such as `extract_limit=`, control how
  many rows are kept. See also **sundering**. [Learn more](../post-interrogation/extracts.qmd).

Data freshness
: Whether a table has recent data. `data_freshness()` checks that the latest date or time in a
  column is no older than `max_age=`, given as text like `"6h"`, `"2d"`, or `"1d 12h"`. The
  **validation report** shows how old the data was, which helps tell a small delay from a pipeline
  that has stopped. [Learn more](../validation-plan/validation-methods.qmd).

DataScan
: An object that holds a column-by-column profile of a table, with statistics like counts, missing
  values, and averages. It's what the **column summary table** is built from. You can save a
  `DataScan` to a JSON file, load it again later, and compare two scans to spot **drift**. [Learn
  more](../data-inspection/col-summary-tbl.qmd).

Define-XML
: A **CDISC** file format (written in XML) that describes a clinical trial dataset: its column
  names, labels, types, and allowed values. Pointblank can read Define-XML files with
  `import_metadata()` and can pass them to **CORE**. [Learn
  more](../metadata-import/cdisc-validation.qmd).

Direction
: See **source and target**.

Domain
: See **SDTM**.

DraftValidation
: A tool that uses an LLM to write a first draft of a **validation plan** for your table. It sends
  the model a summary of your table (column types, counts, and basic statistics) rather than the
  full data, and gives back Python code for you to review and adjust. Available through the
  `DraftValidation` class. See also **AI Validation Editor**. [Learn
  more](../advanced-validation/draft-validation.qmd).

Drift
: A change in a table between two points in time, like a column being added or removed, or the
  typical values in a column shifting. You can look for drift by comparing two **DataScan** objects
  with `DataScan.compare()`. It returns a `DataScanDiff`, which lists the columns that were added,
  removed, or changed, along with **drift scores**. [Learn
  more](../data-inspection/col-summary-tbl.qmd).

Drift score
: A number that measures how far a column's values have shifted between two **DataScan** objects.
  `DataScan.compare()` gives two kinds. The PSI (Population Stability Index) works for numeric and
  categorical columns: under 0.1 means little change, and over 0.25 means a large shift. The KS
  (Kolmogorov-Smirnov) test works for numeric columns, and a small p-value (say, under 0.05) means
  the two sets of values differ. Drift scores need the original data, so they can't be worked out
  for a scan loaded from a JSON file. [Learn more](../data-inspection/col-summary-tbl.qmd).

Error
: See **warning, error, and critical**.

Exit code
: The number a command gives back when it finishes, which scripts and CI systems use to tell
  whether it succeeded (`0` means success). `pb validate --exit-code` gives a non-zero code if any
  check fails, and `pb run --fail-on` does so when a chosen **threshold** level is reached. `pb
  compare` gives `0` if the tables match, `1` if they differ, and `2` if there was an error. [Learn
  more](../the-pointblank-cli/cli-data-validation.qmd).

Expression check
: A **validation step** that checks each row against a condition written in your table library's
  own syntax. For example, with Polars you might check `pl.col("end") > pl.col("start")`. Added
  with `col_vals_expr()`. [Learn more](../advanced-validation/expressions.qmd).

Field
: A description of one column for **test data generation**, made with a helper such as
  `int_field()`, `float_field()`, `string_field()`, `bool_field()`, or `date_field()`. The options
  say what values are allowed, for example `min_val=`, `max_val=`, `allowed=`, `pattern=`,
  `preset=`, or `unique=True`, depending on the type. You put fields together in a `Schema` and
  pass it to `generate_dataset()`. [Learn
  more](../test-data-generation/generating-test-data.qmd#field-types).

Final action
: An **action** that runs once, after all the **validation steps** are finished, whatever the
  results were. Final actions are useful for sending a summary, such as a message to a chat
  channel (`send_slack_notification()` makes one for Slack). You set them with the `FinalActions`
  class. Inside a final action, `get_validation_summary()` gives you the results. [Learn
  more](../validation-plan/actions.qmd).

Global settings
: Default settings that apply to every validation in your Python session, changed with
  `pb.config()`. They control what the **validation report** includes (for example, the header,
  footer, and timings) and how the **health score** is calculated. Each call to `pb.config()` sets
  every option, so any option you leave out goes back to its default.

Governance metadata
: Details about who looks after a validation and who relies on it, given with `owner=`,
  `consumers=`, and `version=` in `Validate` (the same fields can be set in a **YAML workflow** or
  a **contract**). They're shown in the footer of the **validation report** and don't change how
  the checks run. [Learn more](../post-interrogation/validation-reports.qmd).

Great Tables
: The Python library Pointblank uses to make its tables, such as the **validation report**, the
  **step report**, and the **column summary table**. These come back as Great Tables `GT` objects,
  so you can show them, turn them into HTML with `as_raw_html()`, or save them as an image with
  `gtsave()`.

Health score
: A single number from 0 to 100 that sums up data quality: the share of all **test units** that
  passed. Pointblank also works out a score for each **quality dimension**. Get the scores with
  `get_health_score()` and `get_dimension_scores()`, or show them with `get_scorecard()`. They
  appear in the **validation report** when you use `incl_dimensions=True`. [Learn
  more](../post-interrogation/quality-dimensions-and-scoring.qmd).

Interrogation
: Running the checks in a **validation plan**, which happens when you call `.interrogate()`.
  Nothing is checked until then. Interrogation runs every step, counts what passed and failed,
  compares the counts to the **thresholds**, runs any **actions**, and saves **data extracts**.
  [Learn more](../validation-plan/validation-overview.qmd).

JSON report
: The validation results as JSON text, which other programs can read. You get it with
  `Validate.get_json_report()`, and you can choose which details to include with `use_fields=` or
  leave some out with `exclude_fields=`. See also **validation report**.

MCP server
: A way for AI assistants (such as the AI chat in VS Code or Positron) to use Pointblank for
  you. MCP (Model Context Protocol) is a standard way for AI tools to connect to
  other programs. Through it, an assistant can load data, build and run validations, and show
  previews and summaries. [Learn more](../integrations/mcp-quick-start.qmd).

Metadata import
: Reading a description of a dataset from a file made by another tool, and turning it into
  Pointblank checks. `import_metadata()` reads files from SPSS, SAS, and Stata, Frictionless Data
  Packages, CSV on the Web (CSVW), and **CDISC** standards like **Define-XML**. You can turn the
  result into a validation plan with `to_validate(data)` or into a **schema** with `to_schema()`.
  [Learn more](../metadata-import/metadata-import.qmd).

Missing values table
: A table showing where the missing values are in your data, made with `missing_vals_tbl()`. By
  default it splits the rows into chunks and shades each chunk by how much of each column is
  missing. If you describe your **coded missing values** with `missing=`, it can also break them
  down by reason. [Learn more](../data-inspection/missing-vals-tbl.qmd).

`na_pass`
: An option on many validation methods that decides whether missing values (nulls) count as
  passing. The default, `na_pass=False`, counts them as failures. Set `na_pass=True` when missing
  values are acceptable. [Learn more](../validation-plan/validation-methods.qmd).

Narwhals
: The library Pointblank uses to work with tables from different **backends** in the same way. It's
  why a validation you write once works whether your table is in Polars, Pandas, or a database.
  Pointblank installs it for you, so you won't usually need to use it directly.

Null flavor
: A standard code from the HL7 and **CDISC** standards that says why a value is missing. For
  example, `"NI"` means no information, `"UNK"` means unknown, and `"ASKU"` means asked but
  unknown. `MissingSpec.from_cdisc_null_flavors()` sets up all 14 codes for you. See also **coded
  missing value**. [Learn more](../metadata-import/cdisc-validation.qmd).

OpenTelemetry integration
: OpenTelemetry is a widely used standard for sending measurements from programs to monitoring
  tools such as Grafana or Datadog. Adding `emit_otel()` as a **final action** sends numbers like
  pass rates and **threshold** breaches after each validation. This lets you track data quality
  on the same dashboards as the rest of your systems. [Learn
  more](../integrations/otel-integration.qmd).

Partitions
: A way to compare tables that are too large to fit in memory. With `partitions=N`,
  [`compare()`](`pointblank.compare`) (and `tbl_match()` and `tbl_reconcile()`) works through the
  tables in N passes, each covering a share of the keys from both tables. The results are the same
  as in a single pass, but less memory is needed at once. More partitions use less memory but take
  longer. See also **table comparison**. [Learn
  more](../table-comparison/comparing-large-files.qmd).

Pipeline
: A data-processing step with **contracts** on both sides. A `Pipeline` checks the incoming data
  (the source), runs your transformation function, then checks the result (the target), all in
  one call to `run(data, transform)`. By default, if the incoming data fails, the transformation is
  skipped. The `PipelineResult` that comes back holds both validations and the transformed data,
  and its `passed` is `True` only if both checks passed. Checking data at these two points is
  sometimes called *boundary enforcement*. See also **source and target** and **violation
  handling**. [Learn more](../contracts-and-pipelines/pipelines.qmd).

Plan serialization
: Turning a **validation plan** into text so it can be saved, shared, reviewed, or rebuilt later.
  `Validate.to_code()` gives Python code, `Validate.to_yaml()` gives a **YAML workflow**, and
  `Validate.to_json_schema()` gives a JSON Schema document. See also **round trip**. [Learn
  more](../advanced-validation/plan-serialization.qmd).

Preprocessing
: Changing the table for just one step before it's checked, using the `pre=` option. You give
  `pre=` a function that takes the table and returns a changed version, for example with some rows
  filtered out or a new column added. The original data and other steps are not affected. The
  number of **test units** is counted after the change. [Learn
  more](../validation-plan/preprocessing.qmd).

Preset
: A ready-made kind of realistic text for **test data generation**, chosen with
  `string_field(preset=...)`. There are presets for names, email addresses, street addresses,
  cities, phone numbers, company names, URLs, credit card numbers, and more. `profile_fields()`
  gives a whole set of person-related presets at once. Presets follow the `country=` given to
  `generate_dataset()`, and related presets fit together within a row (see **data coherence**).
  [Learn more](../test-data-generation/generating-test-data.qmd#available-presets).

Preview
: A quick look at a table showing the first and last few rows along with each column's type, made
  with `preview()`. See also **column summary table**. [Learn
  more](../data-inspection/preview.qmd).

`python:` block
: A way to write Python code inside a **YAML workflow**, for settings that need it, such as `tbl`,
  `pre`, `expr`, and `actions`. The code isn't sandboxed, so it only runs if you allow it with
  `allow_python=True` (or `pb run --allow-python`). A few simple calls, such as `pb.col()` and
  `pb.load_dataset()`, work without it. [Learn more](../yaml/yaml-reference.qmd).

Quality dimension
: A category describing what kind of data quality a step measures. The categories are
  *completeness*, *validity*, *uniqueness*, *consistency*, *timeliness*, and *volume*. Each
  validation method has a default category (for example, `col_vals_not_null()` measures
  completeness), and you can change it with `dimension=`. Each dimension gets its own score, and
  together they make up the **health score**. [Learn
  more](../post-interrogation/quality-dimensions-and-scoring.qmd).

Reconciliation
: Checking that two tables add up to the same numbers even though their rows differ, for example
  that a summary table agrees with the detailed records it was built from. `tbl_reconcile()`
  groups both tables by the `keys=` columns, totals the `measures=` columns (with `agg="sum"` by
  default, or `"count"`, `"mean"`, `"min"`, or `"max"`), and compares the results group by group.
  Each group is one **test unit**. See also **table comparison**. [Learn
  more](../table-comparison/comparing-tables.qmd).

Reference data
: A second table, such as last month's data, that **aggregate checks** can compare with. You set it
  with `Validate(reference=...)`. When an aggregate check is given no `value=`, it compares with
  the same column in the reference data, so `col_sum_eq(columns="revenue", tol=50)` checks that the
  total revenue is within 50 of the reference table's total. To compare with a different column of
  the reference data, use `value=pb.ref("column_name")`. [Learn
  more](../validation-plan/validation-methods.qmd).

Referential check
: A check that every value in a column also appears in a column of another table, for example that
  every `customer_id` in an orders table exists in the customers table. Added with
  `col_vals_in_table()`. [Learn more](../validation-plan/validation-methods.qmd).

Report language
: The language used for the text in reports and in generated **briefs**, set with
  `Validate(lang=...)`. Forty languages are available. A separate option, `locale=`, controls how
  numbers are formatted (for example, whether a comma or a period is used as the decimal mark).
  [Learn more](../validation-plan/briefs.qmd).

Round trip
: Turning something into another form and back again, ending up with the same thing. For example, a
  **validation plan** saved with [`to_yaml()`](`Validate.to_yaml`) and loaded again with
  `yaml_interrogate()`, or a **contract** imported from JSON Schema and exported back. Anything that
  both forms can express survives the round trip. Things that only one form can express, such as a
  `pre=` function, are replaced with a placeholder or left out. [Learn
  more](../advanced-validation/plan-serialization.qmd).

Row-based check
: A check on whole rows rather than on single values. `rows_distinct()` checks that no row appears
  more than once, and `rows_complete()` checks that no row has missing values. Both can look at
  just some of the columns with `columns_subset=`. [Learn
  more](../validation-plan/validation-methods.qmd).

Row status
: The result for one row in a **table comparison**: `match` (in both tables, with equal values),
  `changed` (in both tables, but at least one value differs), `missing` (only in the source table),
  `extra` (only in the target table), or `dup_key` (its key appears more than once, so it can't be
  matched up). The counts are in the `status_counts` of a `Comparison`. See also **source and
  target**. [Learn more](../table-comparison/comparing-tables.qmd).

Schema
: A description of the columns a table should have: their names and, optionally, their types. You
  write one with the `Schema` class and check a table against it with `col_schema_match()`, with
  options for how strict to be about column order, capital letters, and types. `schema_from_tbl()`
  builds a schema from an existing table. [Learn
  more](../advanced-validation/schema-validation.qmd).

Scorecard
: A compact table showing the **health score** and the score for each **quality dimension**, made
  with `Validate.get_scorecard()`. [Learn
  more](../post-interrogation/quality-dimensions-and-scoring.qmd).

SDTM (Study Data Tabulation Model)
: A **CDISC** standard for organizing the data collected in a clinical trial into tables called
  *domains*, such as DM (demographics), AE (adverse events), and LB (lab results). Pointblank can
  check that a domain has the columns the standard requires (`validate_sdtm_structure()`), and it
  can check datasets against the SDTM rules (`validate_sdtmig()`). [Learn
  more](../metadata-import/cdisc-validation.qmd).

SDTMIG (SDTM Implementation Guide)
: CDISC's detailed rules for how data should be laid out in **SDTM** domains: which columns each
  domain needs, what values they can hold, and how the domains relate to each other.
  `validate_sdtmig()` checks a set of domains against the SDTMIG 3.4 rules, without needing
  **CORE**, and gives back a **conformance report**. [Learn
  more](../metadata-import/cdisc-submission-conformance.qmd).

Seed
: A number that makes randomly generated data repeatable: `generate_dataset(schema, seed=23)` gives
  the same rows each time with the same version of Pointblank. The `generate_dataset` pytest
  fixture picks a seed from each test's name, so every test gets its own data, which stays the
  same from one run to the next. [Learn more](../test-data-generation/pytest-fixture.qmd).

Segment
: A group of rows that gets checked and reported on its own. Using `segments=` splits one step
  into several, one per group. For example, `segments="region"` makes one step for each region in
  the table, and `segments=("region", ["East", "West"])` makes steps for just those two regions. To
  treat several values as one group, use `seg_group()`. [Learn
  more](../validation-plan/segmentation.qmd).

Severity level
: See **warning, error, and critical**.

Source and target
: The two sides of a **table comparison** or a **pipeline**. In a table comparison, the source is
  the table you start from and the target is the one checked against it. With `tbl_match()`, the
  table being validated is the target, and `tbl_compare=` is the source. In a pipeline, the source
  is the data going into your transformation and the target is what comes out, and each can have
  its own **contract**. A contract's `direction=` (`"source"` or `"target"`) records which side
  it's meant for. [Learn more](../contracts-and-pipelines/pipelines.qmd).

Specification check
: A check that text values follow a well-known format, such as email addresses, URLs, postal
  codes, credit card numbers, ISBNs, VINs, or IBANs. Added with `col_vals_within_spec()`, for
  example with `spec="email"` or `spec="postal_code[USA]"`. [Learn
  more](../validation-plan/validation-methods.qmd).

Status check
: A method that tells you how a validation went, as `True` or `False`, without stopping your code.
  [`all_passed()`](`Validate.all_passed`) tells you whether every **test unit** passed, and
  `above_threshold()`, `warning()`, `error()`, and `critical()` tell you whether **threshold**
  levels were reached. See also **assertion**. [Learn more](../advanced-validation/assertions.qmd).

Step
: Usually short for **validation step**. In a **contract**, the `Step` class describes one
  validation method call by its name and arguments (for example, `Step("col_vals_not_null",
  columns="id")`), so that it can be saved as YAML or JSON. [Learn
  more](../contracts-and-pipelines/contracts.qmd).

Step report
: A detailed view of a single **validation step**, showing what it checked and, where it makes
  sense, the rows that failed. You get one with `Validate.get_step_report()`. See also **validation
  report**. [Learn more](../post-interrogation/step-reports.qmd).

Structured missingness
: See **coded missing value**.

Sundering
: Splitting a table in two after validation: the rows that passed every row-by-row check, and the
  rows that failed at least one. You get either part with `Validate.get_sundered_data()`, using
  `type="pass"` or `type="fail"`. See also **data extract**. [Learn
  more](../post-interrogation/sundering.qmd).

Table comparison
: Finding the differences between two tables. [`compare()`](`pointblank.compare`) matches up the
  rows of a source table and a target table, gives each row a **row status** (such as `changed` or
  `missing`), and reports any differences in their columns. It returns a `Comparison` object, which
  counts as `True` only if the tables match. Rows can be matched by key columns (`keys="id"`), by
  position (the default), or regardless of order (`keys="*"`), and small differences can be allowed
  with `tolerance=` or ignored with `normalize=` (for example, letter case). As a **validation
  step**, `tbl_match()` runs the same comparison with each row as a **test unit**, and
  `tbl_reconcile()` compares totals instead (see **reconciliation**). From the **CLI**, use
  `pb compare`. To check only the number of rows or columns, use `row_count_match()` or
  `col_count_match()`. See also **source and target** and **partitions**. [Learn
  more](../table-comparison/comparing-tables.qmd).

Test data generation
: Making realistic fake tables for testing and demos. You describe each column in a `Schema` with
  **fields**, such as `int_field(min_val=1)` or `string_field(preset="email")`, and then create
  rows with `generate_dataset()`. Use `country=` for data that fits one or more countries, and
  `seed=` to get the same rows every time. When Pointblank is installed, a `generate_dataset`
  fixture is also available in pytest. Not to be confused with **test datasets**. See also
  **preset**, **data coherence**, and **seed**. [Learn
  more](../test-data-generation/generating-test-data.qmd).

Test dataset
: One of the sample tables that comes with Pointblank, loaded with `load_dataset()`. The choices
  are `small_table`, `game_revenue`, `nycflights`, and `global_sales`, and `tbl_type=` sets which
  **backend** you get (for example, `"polars"`, `"pandas"`, or `"duckdb"`). They're used in
  examples throughout the documentation. [Learn more](../getting-started/quickstart.qmd).

Test unit
: A single thing that passes or fails in a **validation step**. When a step checks the values in a
  column, each value is one test unit, so checking a column in a 100-row table gives 100 test units.
  When a step checks the table or column as a whole (like `col_exists()` or `col_sum_gt()`), there
  is just one test unit. The pass and fail counts in the **validation report** are counts of test
  units. [Learn more](../getting-started/quickstart.qmd).

Threshold
: A limit on how many failing **test units** a step can have before it's flagged. A threshold can
  be a fraction (`0.1` means 10% of test units) or a count (`5` means five test units); any value
  of 1 or more is treated as a count. There are three levels: **warning, error, and critical**. A
  level is reached when the failures meet or go past its threshold. You set thresholds for the
  whole validation (`Validate(thresholds=...)`) or for a single step, either with the
  `Thresholds` class or with a shortcut like `(0.1, 0.2, 0.3)`. [Learn
  more](../validation-plan/thresholds.qmd).

Tolerance
: How far a result can be from the expected one and still pass. **Aggregate checks** and
  `col_pct_null()` take `tol=`: for example, `col_avg_eq(columns="x", value=10, tol=0.5)` passes
  for any average from 9.5 to 10.5. In a **table comparison**, `tolerance=` sets how much two
  values may differ, as an absolute amount (`atol`), as a share of the source value (`rtol`), or,
  for dates and times, as a length of time like `"1s"`. [Learn
  more](../validation-plan/validation-methods.qmd).

Trend check
: A check that a column's values go up (`col_vals_increasing()`) or down (`col_vals_decreasing()`)
  from one row to the next, as in a time series. Repeated values fail unless
  `allow_stationary=True`. Rows are compared in the table's own order, or in the order given by
  `order_by=`, which database tables need since their rows have no fixed order. [Learn
  more](../validation-plan/validation-methods.qmd).

Validate
: The main class in Pointblank. A `Validate` object holds your table, settings like
  **thresholds** and **actions**, and the **validation plan**. You add checks by calling
  validation methods on it, one after another, and run them with `.interrogate()`. Afterwards, the
  same object gives you reports, **data extracts**, and results. [Learn
  more](../validation-plan/validation-overview.qmd).

Validation method
: A method of `Validate` that adds a check to the **validation plan**, such as `col_vals_gt()`
  (values are greater than some number) or `rows_distinct()` (no duplicate rows). Each call adds
  one or more **validation steps**. [Learn more](../validation-plan/validation-methods.qmd).

Validation plan
: The list of checks you've added to a `Validate` object. Building the plan doesn't check anything
  yet: it only records what should be checked. The checks happen during **interrogation**. Plans
  can also be written in YAML (see **YAML workflow**). [Learn
  more](../validation-plan/validation-overview.qmd).

Validation report
: The table that summarizes the results of every **validation step** after **interrogation**. It
  shows each step's description, which columns it checked, how many **test units** passed and
  failed, and which **threshold** levels were reached. You get it with
  `Validate.get_tabular_report()`. See also **step report**, **JSON report**, and **report
  language**. [Learn more](../post-interrogation/validation-reports.qmd).

Validation step
: One check in a **validation plan**. Each call to a **validation method** creates at least one
  step: one for each column it checks and one for each **segment**. Steps are numbered from 1.
  After **interrogation**, each step records how many **test units** passed and failed and which
  **threshold** levels were reached. [Learn more](../validation-plan/validation-methods.qmd).

Validation template
: A **YAML workflow** written to be used with many tables. Leave the data out with `tbl: null`,
  then give a table each time it runs with `yaml_interrogate(..., set_tbl=...)`. This way, a set of
  checks written once can be run on, say, each quarter's data. [Learn
  more](../yaml/yaml-validation-workflows.qmd).

Violation handling
: What happens when data fails a **contract** inside a **pipeline**, set with the contract's
  `on_violation=`. With `"warn"` (the default), Python shows a warning; with `"raise"`, an error
  stops the run; and with `"log"`, a message is written to the `pointblank.contract` logger. [Learn
  more](../contracts-and-pipelines/contracts.qmd).

Warning, error, and critical
: The three **threshold** levels, from least to most serious. The names are only labels, so what
  each one means is up to you. For example, you might send a note to your team at the warning
  level and stop a data pipeline at the critical level. The **validation report** shows which
  levels each step reached in its W, E, and C columns. [Learn
  more](../validation-plan/thresholds.qmd).

YAML workflow
: A validation written in a YAML file instead of Python code. The file lists the data to use, the
  **validation steps**, the **thresholds**, and other settings. This makes validations easy to
  store in version control and share with people who don't write Python. You run one with
  `yaml_interrogate()` or with `pb run` on the command line, and you can convert one to Python
  with `yaml_to_python()`. See also **validation template** and **`python:` block**. [Learn
  more](../yaml/yaml-validation-workflows.qmd).