Execute a YAML-based validation workflow.
Usage
This is the main entry point for YAML-based validation workflows. It takes YAML configuration (as a string or file path) and returns a validated Validate object with interrogation results.
The YAML configuration defines the data source, validation steps, and optional settings like thresholds and labels. This function automatically loads the data, builds the validation plan, executes all validation steps, and returns the interrogated results.
yaml: Union[str, Path]YAML configuration as string or file path. Can be: (1) a YAML string containing the validation configuration, or (2) a Path object or string path to a YAML file.
set_tbl: Any = NoneAn optional table to override the table specified in the YAML configuration. This allows you to apply a YAML-defined validation workflow to a different table than what’s specified in the configuration. If provided, this table is used in place of the table defined in the YAML’s tbl field (which is then not loaded, so it can be a placeholder or a source that isn’t available). This can be any supported table type including DataFrame objects, Ibis table objects, CSV file paths, Parquet file paths, GitHub URLs, or database connection strings.
namespaces: Optional[Union[Iterable[str], Mapping[str, str]]] = NoneOptional module namespaces to make available for Python code execution in YAML configurations. Can be a dictionary mapping aliases to module names or a list of module names. This only has an effect with allow_python=True. See the “Using Namespaces” section below for detailed examples.
allow_python: bool = Falsepython: blocks, and the expr:, pre:, and active: shortcuts). This is False by default, which makes a configuration with Python code raise a YAMLValidationError. Python code in YAML runs with the full access of your Python session (it can read and write files or run commands), so only set this to True for configurations you trust. A few pb functions called with literal arguments (e.g., pb.col("b") or pb.has_columns("a")) don’t need it. See the “Python Code in YAML” section below.
Validate YAMLValidationErrorallow_python=True.
A YAML configuration can include Python code: a python: block for the data source, a parameter value, or an action, and the expr:, pre:, and active: shortcuts for code written as a plain string. This code isn’t run unless you set allow_python=True.
Python code in YAML is not sandboxed. It runs with the same access as the rest of your Python session, so a configuration with Python code should be treated like a Python script: only allow it for configurations you wrote or otherwise trust, never for ones supplied by other users or systems.
A small set of pb functions can always be used, when called with literal arguments (strings, numbers, lists, and so on): load_dataset(), col(), ref(), seg_group(), has_columns(), has_rows(), and the column selectors (starts_with(), ends_with(), contains(), matches(), everything(), first_n(), and last_n()). So can the constructors for values that YAML has no literal for: Decimal(), ZoneInfo(), datetime.date(), datetime.time(), datetime.datetime(), and datetime.timedelta(). These only load a bundled dataset or build an object, so value: pb.col("b"), value: {python: 'Decimal("2.2")'}, or active: pb.has_columns("a") work without allow_python=True. In code that is allowed to run, datetime, Decimal, and ZoneInfo are available along with pb, pl, pd, and Path.
The namespaces= parameter enables custom Python modules and functions in YAML configurations. This is particularly useful for custom action functions and advanced Python expressions. Since these are Python code, both options below need allow_python=True.
Namespace formats:
{"alias": "module.name"} maps aliases to module names["module.name", "another.module"] imports modules directlyOption 1: Inline expressions (no namespaces needed)
import pointblank as pb
# Simple inline custom action
yaml_config = '''
tbl: small_table
thresholds:
warning: 0.01
actions:
warning:
python: "lambda: print('Custom warning triggered')"
steps:
- col_vals_gt:
columns: [a]
value: 1000
'''
result = pb.yaml_interrogate(yaml_config, allow_python=True)
resultCustom warning triggeredOption 2: External functions with namespaces
# Define a custom action function
def my_custom_action():
print("Data validation failed: please check your data.")
# Add to current module for demo
import sys
sys.modules[__name__].my_custom_action = my_custom_action
# YAML that references the external function
yaml_config = '''
tbl: small_table
thresholds:
warning: 0.01
actions:
warning:
python: actions.my_custom_action
steps:
- col_vals_gt:
columns: [a]
value: 1000 # This will fail
'''
# Use namespaces to make the function available
result = pb.yaml_interrogate(
yaml_config, namespaces={'actions': '__main__'}, allow_python=True
)
resultData validation failed: please check your data.This approach enables modular, reusable validation workflows with custom business logic.
For the examples here, we’ll use YAML configurations to define validation workflows. Let’s start with a basic YAML workflow that validates the built-in small_table dataset.
The validation table shows the results of our YAML-defined workflow. We can see that the rows_distinct() validation failed (because there are duplicate rows in the table), while the column existence checks passed.
Now let’s create a more comprehensive validation workflow with thresholds and metadata:
# Advanced YAML configuration with thresholds and metadata
yaml_config = '''
tbl: small_table
tbl_name: small_table_demo
label: Comprehensive data validation
thresholds:
warning: 0.1
error: 0.25
critical: 0.35
steps:
- col_vals_gt:
columns: [d]
value: 100
- col_vals_regex:
columns: [b]
pattern: '[0-9]-[a-z]{3}-[0-9]{3}'
- col_vals_not_null:
columns: [date, a]
'''
# Execute the validation workflow
result = pb.yaml_interrogate(yaml_config)
print(f"Table name: {result.tbl_name}")
print(f"Label: {result.label}")
print(f"Total validation steps: {len(result.validation_info)}")Table name: small_table_demo
Label: Comprehensive data validation
Total validation steps: 4The validation results now include our custom table name and label. The thresholds we defined will determine when validation steps are marked as warnings, errors, or critical failures.
You can also load YAML configurations from files. Here’s how you would work with a YAML file:
from pathlib import Path
import tempfile
# Create a temporary YAML file for demonstration
yaml_content = '''
tbl: small_table
tbl_name: File-based Validation
steps:
- col_vals_between:
columns: [c]
left: 1
right: 10
- col_vals_in_set:
columns: [f]
set: [low, mid, high]
'''
with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
f.write(yaml_content)
yaml_file_path = Path(f.name)
# Load and execute validation from file
result = pb.yaml_interrogate(yaml_file_path)
resultThis approach is particularly useful for storing validation configurations as part of your data pipeline or version control system, allowing you to maintain validation rules alongside your code.
YAML workflows support governance metadata via owner, consumers, and version top-level keys. These are forwarded to the Validate constructor and embedded in the validation report:
yaml_config = '''
tbl: small_table
tbl_name: sales_pipeline
owner: Data Engineering
consumers: [Analytics, Finance, Compliance]
version: "2.1.0"
steps:
- col_vals_not_null:
columns: [a, b]
'''
result = pb.yaml_interrogate(yaml_config)
print(f"Owner: {result.owner}")
print(f"Consumers: {result.consumers}")
print(f"Version: {result.version}")Owner: Data Engineering
Consumers: ['Analytics', 'Finance', 'Compliance']
Version: 2.1.0YAML supports aggregate validation methods for checking column-level statistics. These methods validate that a column’s sum, average, or standard deviation meets a threshold:
yaml_config = '''
tbl: small_table
steps:
- col_sum_gt:
columns: [d]
value: 0
- col_avg_le:
columns: [a]
value: 10
'''
result = pb.yaml_interrogate(yaml_config)
result| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_sum_gt()
|
d | 0 | ✓ | 1 | 1 1.00 |
0 0.00 |
— | — | — | — | |
| #4CA64C | 2 |
col_avg_le()
|
a | 10 | ✓ | 1 | 1 1.00 |
0 0.00 |
— | — | — | — |
The 15 available aggregate methods follow the pattern col_{stat}_{comparator} where {stat} is sum, avg, or sd and {comparator} is gt, lt, ge, le, or eq.
Check that a date/datetime column has recent data using data_freshness:
The active= parameter controls whether a validation step runs. It supports boolean values and Python expression shortcuts:
Use col_pct_null to validate that the percentage of null values in a column is within bounds:
set_tbl= to Override the TableThe set_tbl= parameter allows you to override the table specified in the YAML configuration. This is useful when you have a template validation workflow but want to apply it to different tables:
import polars as pl
# Create a test table with similar structure to small_table
test_table = pl.DataFrame({
"date": ["2023-01-01", "2023-01-02", "2023-01-03"],
"a": [1, 2, 3],
"b": ["1-abc-123", "2-def-456", "3-ghi-789"],
"d": [150, 200, 250]
})
# Use the same YAML config but apply it to our test table
yaml_config = '''
tbl: small_table # This will be overridden
tbl_name: Test Table # This name will be used
steps:
- col_exists:
columns: [date, a, b, d]
- col_vals_gt:
columns: [d]
value: 100
'''
# Execute with table override
result = pb.yaml_interrogate(yaml_config, set_tbl=test_table)
print(f"Validation applied to: {result.tbl_name}")
resultValidation applied to: Test Table| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
col_exists()
|
date | — | ✓ | 1 | 1 1.00 |
0 0.00 |
— | — | — | — | |
| #4CA64C | 2 |
col_exists()
|
a | — | ✓ | 1 | 1 1.00 |
0 0.00 |
— | — | — | — | |
| #4CA64C | 3 |
col_exists()
|
b | — | ✓ | 1 | 1 1.00 |
0 0.00 |
— | — | — | — | |
| #4CA64C | 4 |
col_exists()
|
d | — | ✓ | 1 | 1 1.00 |
0 0.00 |
— | — | — | — | |
| #4CA64C | 5 |
col_vals_gt()
|
d | 100 | ✓ | 3 | 3 1.00 |
0 0.00 |
— | — | — | — |
This feature makes YAML configurations more reusable and flexible, allowing you to define validation logic once and apply it to multiple similar tables.