WDP Import Pipeline

The Waste Data Platform (WDP) import pipeline handles flat-file (CSV) imports of billing and weight data into SQL Server. It validates each file with Pandera schemas before loading.

What It Does

Flow	Input	Output	Purpose
wdp-details-import	CSV billing/charge file	portal.rpt_details_new	Import daily billing detail records
wdp-weight-import	CSV weight/tonnage file	portal.rpt_weight_new	Import daily weight records

Both flows read from Azure Blob Storage and write to SQL Server (wasteology.database.windows.net).

Architecture

Azure Blob Storage
       │
       ▼
 Prefect Flow (Pandera validation)
       │
       ▼
SQL Server (wasteology)
  portal.rpt_details_new
  portal.rpt_weight_new

Validation

Each import applies a 10-field Pandera schema before any rows reach SQL Server. Validation failures halt the import and surface in Prefect Cloud run logs — no partial loads.

Deployment

Detail	Value
Source repo	~/projects/orchestration/wdp-import-pipeline
Work pool	cietrade-aci-pool (Dev subscription)
Image	cietradeacr.azurecr.io/wdp-imports:latest
Trigger	On-demand (via Prefect API or n8n workflow)
ADO repo	dev.azure.com/wasteology/Wasteology/_git/wdp-import-pipeline

Triggering an Import

From Prefect Cloud

1. Go to app.prefect.cloud → Deployments
2. Find wdp-details-import or wdp-weight-import
3. Click Run → provide the blob path as the file_path parameter
4. Monitor in Flow Runs

From the CLI

cd ~/projects/orchestration/wdp-import-pipeline
source .env

# Run details import
uv run python -c "
from flows.wdp_imports import details_import_flow
result = details_import_flow('data/details_test_import_v1.csv')
print(result)
"

Deploying

cd ~/projects/orchestration/wdp-import-pipeline
source .env && uv run python scripts/deploy-wdp-imports.py

Data Validation Schema

Each CSV must match the expected schema before loading. If you receive a validation error, check:

1. Column names match exactly (case-sensitive)
2. Data types are correct (dates as YYYY-MM-DD, amounts as numeric)
3. Required fields are not null

Pandera validation errors appear in the Prefect run logs with the specific failing row and column.

note

WDP imports run on the cietrade-aci-pool (Dev subscription, cietradeacr.azurecr.io). New flows for this pipeline go in the same pool — do not mix with enevo-aci-pool.