Skip to main content

Prefect — Overview

Prefect is the workflow orchestrator behind every Wasteology ETL pipeline. This page introduces how we use Prefect Cloud together with Azure Container Instances (ACI) to run flows on demand, without standing up any long-lived worker infrastructure.

If you're new to the stack, read this page end-to-end before touching a deployment. The Flow Schedules section lists the specific deployments you'll most likely interact with day-to-day.

Prefect + Azure Container Instances Architecture

What is Prefect?​

Prefect Cloud is a managed orchestration service. It schedules flow runs, stores run history, surfaces logs and failures in a UI, and exposes an API for programmatic control. At Wasteology, Prefect Cloud is the source of truth for "what pipelines exist, when they run, and whether they succeeded."

AspectDetail
Hosted byPrefect (SaaS — app.prefect.cloud)
PlanFree
WorkspaceWasteology
Self-hosted?No — we do not run Prefect Server

The Prefect Python library lives inside each flow's container image. It registers deployments, emits run state back to Prefect Cloud, and handles retries/logging. But no Prefect process runs continuously in our own infrastructure — execution is delegated to Azure Container Instances (see below).

Execution Model: Azure Container Instances (ACI)​

Every flow runs inside an ephemeral Azure Container Instance. Prefect Cloud tells Azure when to start one; Azure tears it down when the flow finishes.

You trigger a flow run (scheduled or manual)
  │
  ▼
Prefect Cloud schedules the run
  │
  ▼
Work pool (push type) calls Azure API
  │  Uses service principal credentials
  │
  ▼
Azure spins up an ephemeral Container Instance
  │  Pulls image from Azure Container Registry
  │  Runs your flow code inside the container
  │  Container exits when flow completes
  │
  ▼
Prefect Cloud records result (Completed / Failed / Crashed)
ACI container is automatically deleted

The key design choice: no persistent worker process. There is no Prefect agent or worker VM sitting idle between runs. ACI containers spin up per run and are destroyed after.

Consequences of this model:

  • No infra to maintain — no workers to patch, no queues to drain, no VMs to size.
  • Cold start (~30s) per run — flows pay a brief startup cost while Azure provisions the container and pulls the image. Acceptable for batch ETL, not suitable for low-latency use cases.
  • Pay only for container runtime — billing is per-second of actual flow execution, not for idle capacity.
  • Container image is the deployment artifact — to change flow code, rebuild and push the image; Prefect will use the new image on the next run.

The Two Work Pools​

A Prefect work pool is the bridge between Prefect Cloud and a specific Azure subscription. Each pool holds the service principal credentials Prefect uses to launch ACI containers in that subscription.

We run two push-type ACI work pools, split by Azure subscription and container registry:

PoolUsed ByAzure SubscriptionACR
cietrade-aci-poolAll CieTrade ETL flows (29 deployments)Wasteology Development (669a4356)cietradeacr.azurecr.io
enevo-aci-poolEnevo, Goodwill, QuickBooks flowsWasteology Production (a5a6818e)wasteologypipelinesacr.azurecr.io
Free plan pool limit

We're on Prefect Cloud's Free plan, which caps us at 2 work pools. Do not create new work pools without coordinating with kgray — adding a third requires either a plan upgrade or retiring an existing pool.

When you create a new deployment, you pick which pool it runs in. That choice determines which Azure subscription provisions the container and which ACR the image must live in.

Credential Strategy: KV Injection​

All non-CieTrade flows (enevo, goodwill, qbo, ach-remittance) use Azure Key Vault injection for app secrets. Secrets are fetched from wg-orchestration-kv (Prod subscription, wdp-palantiri-rg) at deploy time and stored as literal values in the Prefect deployment record. The running container sees them as plain env vars — no network call to Key Vault or Prefect at runtime.

Two narrow exceptions remain as Prefect block refs:

  • ACI bootstrap credentials (aci_credentials, image_registry) — the work pool infrastructure resolves these before the container starts; KV injection cannot substitute here.
  • QBO OAuth tokens (qbo-access-token, qbo-refresh-token) — the QBO flow refreshes and writes new token values back to these blocks on every run. Injecting them as literals would bake in a stale token.

Rotating a KV-injected secret requires re-deploying (no container rebuild needed):

  1. az keyvault secret set --vault-name wg-orchestration-kv --name <secret-name> --value "NEW-VALUE"
  2. Push to main to trigger the CI RegisterDeployment stage, or run scripts/deploy.py manually.

See Deploying a Flow for the full deploy script template with KV injection.

Flow Schedules (enevo-aci-pool flows)​

These are the non-CieTrade deployments — the ones you'll most often touch on the enevo-aci-pool side:

DeploymentScheduleWhat it does
enevo-service-events-normalized-wasteologyDaily 6:30 AM UTCPulls Enevo service event data to PostgreSQL
goodwill-monthly-etlMonthly, 12thPulls Goodwill API data to PostgreSQL
qbo-daily-extractDaily 6:00 AM UTCFull QuickBooks extract to PostgreSQL (11 entities)
qbo-full-refreshManual onlyOn-demand QuickBooks full refresh
ach-remittance-dailyWeekdays 9:00 AM ETReads previous-day ACH payments from QuickBooks, generates Excel remittance per vendor, emails from remittance@wasteologygroup.com

All five write to our PostgreSQL warehouse (or send email in the case of ach-remittance-daily). Note that ach-remittance-daily lives in the wg-orchestration repo, not a dedicated ETL repo. The QuickBooks deployments are the heaviest — qbo-daily-extract pulls 11 entities every morning; qbo-full-refresh is the nuclear option when you need to rebuild from scratch.

CieTrade deployments live on the other pool and are documented separately. If your work touches CieTrade flows, loop in that team before making schedule changes.

Accessing Prefect Cloud​

The Prefect Cloud UI is the primary interface for day-to-day operations.

URL: https://app.prefect.cloud

  1. Sign in with your Wasteology account.
  2. Navigate to the Wasteology workspace.
  3. Use the Deployments tab to see all flows and trigger runs manually.
  4. Use the Flow Runs tab to monitor active and recent runs, inspect logs, and view failure details.

From the Deployments tab you can:

  • Trigger an ad-hoc run (useful for qbo-full-refresh and for re-running a failed scheduled job).
  • Pause/resume a schedule.
  • View the deployment's work pool, image tag, and parameters.

From the Flow Runs tab you can:

  • Filter by state (Running, Completed, Failed, Crashed) or by deployment.
  • Open a run to see its full log stream (captured from the ACI container's stdout/stderr).
  • See timing — start, end, duration — and any exceptions.

Self-Healing: Automatic Failure Response​

Flows in our environment are wired up with an on_failure hook that creates an Azure DevOps Work Item when the flow crashes or fails. That work item is tagged to trigger the ADW (agentic developer workflow) pipeline, which spins up an agent to diagnose the failure and attempt a fix.

The short version of the loop:

Flow run fails
  │
  ▼
on_failure hook → creates ADO Work Item (tagged for ADW)
  │
  ▼
ADW pipeline picks up the work item
  │  Reads logs, plans a fix, implements + tests, opens a PR
  │
  ▼
Auto-fix succeeds → PR ready for review
Auto-fix fails   → engineers notified to investigate manually

You generally don't need to do anything to opt in — the hook is applied at the flow level and inherited by all deployments. If you write a new flow, wire up the same on_failure hook so your failures participate in the self-healing loop.