Prefect — Deploying a Flow

This page walks through deploying a brand-new ETL flow to the enevo-aci-pool work pool on the Production Azure subscription. If you are onboarding and have Prefect Cloud access but have never pushed a flow, start here.

Before deploying anything, read Prefect — Overview to understand how Prefect Cloud, the ACR images, and Azure Container Instances fit together. This page is purely the mechanics.

Prerequisites

Confirm each of the following before you start. Missing any one of these will cause the deploy to fail partway through, usually with a confusing error.

Requirement	Check
Docker installed and running	`docker --version`
AcrPush on `wasteologypipelinesacr.azurecr.io`	`az acr login --name wasteologypipelinesacr` succeeds
Prefect Cloud Editor on the Wasteology workspace	You can see Deployments in app.prefect.cloud
Python env with Prefect + Azure plugin	`pip install prefect prefect-azure`
Azure CLI logged in	`az login`

Step 1 — Write your flow

A Prefect flow is just a Python function decorated with @flow. Any functions called inside it can be decorated with @task for granular run tracking in the Prefect UI.

from prefect import flow, task
from flows.hooks.self_healing_hook import self_healing_hook  # add to any production flow

@task
def extract_data():
    ...

@task
def load_data(records):
    ...

@flow(name="my-etl-flow", on_failure=[self_healing_hook])
def my_etl():
    records = extract_data()
    load_data(records)

Self-healing hook is required

self_healing_hook is required for any production flow — it creates an ADO work item on failure so the ADW agent can attempt auto-repair. Do not omit it on production deployments.

Step 2 — Build and push the Docker image

All flows run inside Docker containers on Azure Container Instances. Build your image and push it to wasteologypipelinesacr.azurecr.io (the Prod ACR):

# Log in to the Prod ACR
az acr login --name wasteologypipelinesacr

# Build the image (from your project root where Dockerfile is)
docker build -t wasteologypipelinesacr.azurecr.io/<your-flow-name>:latest .

# Push
docker push wasteologypipelinesacr.azurecr.io/<your-flow-name>:latest

The Dockerfile should use prefecthq/prefect:3-latest as the base image and install your project's dependencies on top:

FROM prefecthq/prefect:3-latest
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

Step 3 — Write the deploy script

The deploy script registers your deployment in Prefect Cloud. The key section is job_variables — this tells Prefect how to launch the ACI container — and the credential strategy, which governs which secrets use KV injection vs Prefect block refs.

Credential strategy

Credential type	How to pass	Why
App secrets (DB passwords, API keys)	KV-injected literals	Simpler, no runtime block lookup
ACI bootstrap (`aci_credentials`, `image_registry`)	Prefect block refs	Work pool resolves these before container starts
QBO OAuth tokens	Prefect block refs	Flow writes refreshed tokens back to these blocks

Full deploy script template (with KV injection)

Copy this and adapt the image name, schedule, and env_vars:

import os
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

KV_URL = "https://wg-orchestration-kv.vault.azure.net/"
PROD_SUB = "a5a6818e-a0c0-4c6a-8b5c-21e91d75caa8"
PROD_RG = "prefect-resources"
PROD_ACR = "wasteologypipelinesacr.azurecr.io"
IMAGE = os.getenv("CONTAINER_IMAGE", f"{PROD_ACR}/<your-flow-name>:latest")

_kv = SecretClient(vault_url=KV_URL, credential=DefaultAzureCredential())

def kv(name: str) -> str:
    return _kv.get_secret(name).value

job_variables = {
    "image": IMAGE,
    "cpu": 1.0,
    "memory": 2.0,
    "subscription_id": PROD_SUB,
    "resource_group_name": PROD_RG,
    "auto_remove": True,
    # ACI bootstrap — must stay as Prefect block refs
    "aci_credentials": {
        "subscription_id": PROD_SUB,
        "resource_group_name": PROD_RG,
        "tenant_id":     "{{ prefect.blocks.secret.azure-sp-tenant-id }}",
        "client_id":     "{{ prefect.blocks.secret.azure-sp-client-id }}",
        "client_secret": "{{ prefect.blocks.secret.azure-sp-client-secret }}",
    },
    "image_registry": {
        "registry_url": PROD_ACR,
        "username": "{{ prefect.blocks.secret.azure-sp-client-id }}",
        "password": "{{ prefect.blocks.secret.azure-sp-client-secret }}",
    },
    # App secrets — KV-injected as literals at deploy time
    "env": {
        "MY_SECRET":   kv("my-kv-secret-name"),
        "MY_PASSWORD": kv("my-db-password"),
        # Add other secrets here
    },
}

deployment = my_etl.to_deployment(
    name="my-etl-daily",
    work_pool_name="enevo-aci-pool",
    job_variables=job_variables,
    cron="0 8 * * *",
    description="My ETL flow — does X",
    tags=["my-project", "etl"],
    on_failure=[self_healing_hook],
)

deployment.apply()
print("Deployed!")

IMAGE env var

The IMAGE env var is set by the CI pipeline to the commit SHA tag (e.g. wasteologypipelinesacr.azurecr.io/my-flow:<sha>). The :latest fallback is for local manual runs only.

Then run it:

source .env && uv run python scripts/deploy.py

Step 4 — Verify the deployment

After the deploy script exits cleanly, check Prefect Cloud:

Go to app.prefect.cloud → Deployments
Find your deployment — it should show as Ready
Click Run → Quick Run to trigger a manual test

You can also trigger from the CLI:

prefect deployment run "my-etl-flow/my-etl-daily"

Then watch the Flow Runs tab. A successful run moves through: Scheduled → Pending → Running → Completed. If it fails in Pending for more than a minute or two, the issue is almost always image pull or ACI credentials — check the job variables against the rules below.

Critical Rules for `enevo-aci-pool`

Read before editing any job_variables

These rules prevent the most common class of deployment failure — flows accidentally pointing at the wrong subscription or registry.

Rule	Why
Always set `subscription_id` and `resource_group_name` explicitly in `job_variables`	These must point at the Prod sub. Never use `{{ prefect.blocks.secret.azure-subscription-id }}` — that block points at Dev and is shared by CieTrade flows
Always set `image_registry.registry_url` to `wasteologypipelinesacr.azurecr.io`	CieTrade flows use `cietradeacr.azurecr.io` — wrong registry = image pull failure
Always include `self_healing_hook` in `on_failure`	Ensures failures auto-create ADO work items
Do not modify `cietrade-aci-pool` or its deployments	That pool runs mission-critical daily revenue data
Do not create a third work pool	The Free plan allows only 2. Coordinate with kgray first

Reference: Existing Working Examples

When in doubt, copy from a working deploy script. These are all live on enevo-aci-pool and known to work end-to-end:

Repo	Deploy Script	Notes
`~/projects/quickbooks-etl`	`scripts/deploy.py`	Cleanest example; Prod ACR, 2 deployments (scheduled + manual)
`~/projects/api/enevo/enevo-service-events-etl`	`deployment/prefect.yaml`	YAML-based deployment
`~/projects/api/goodwill/goodwill-api-etl`	`prefect.yaml`	Monthly schedule example

tip

~/projects/quickbooks-etl/scripts/deploy.py is the canonical example of KV injection — it injects QBO client credentials, DB passwords, and the Prefect API key, while keeping OAuth tokens as block refs. Read it before writing your first deploy script.

Prerequisites​

Step 1 — Write your flow​

Step 2 — Build and push the Docker image​

Step 3 — Write the deploy script​

Credential strategy​

Full deploy script template (with KV injection)​

Step 4 — Verify the deployment​

Critical Rules for enevo-aci-pool​

Reference: Existing Working Examples​