Migration Tool

The Migration Tool page allows you to move data between Wasteology databases using either Parquet chunking or Sling streaming modes.

Highlight:allauthmigrationprefectdbt

Page Overview

Migration Tool

Configure and run database migrations between PostgreSQL and Azure SQL

How to Use

1Select your source database from the dropdown

2Select your target database

3Choose migration mode: Single Table, All Tables, or Schedule

4Configure advanced options if needed

5Click Run Migration to start

Migration Modes

Single Table Mode

Migrate a specific table from source to target database.

1. Select the source schema
2. Select the table to migrate
3. Optionally specify a datetime column for chunking
4. Click Run Migration

All Tables Mode

Migrate all tables from a schema:

1. Select the source schema
2. Tables are discovered automatically
3. Migration runs sequentially with progress tracking

Schedule Mode

Schedule a migration via Prefect Cloud:

Prefect Required

This mode requires Prefect Cloud configuration. Set PREFECT_API_URL and PREFECT_API_KEY in your environment.

1. Configure source and target databases
2. Select tables or schema
3. Choose a schedule (hourly, daily, weekly)
4. Click Schedule Migration

Advanced Options

Migration Configuration

Option	Type	Description	Default
method	string	Migration method: parquet for chunked processing or sling for streaming	parquet
chunk_size	int	Number of rows per Parquet chunk (parquet mode only)	100000
connection_type	string	Connection method: native (psycopg2/pymssql) or sling (Sling CLI)	native
datetime_column	string	Column used for time-based chunking	-

Migration Methods

Parquet Mode

✓Recommended for large tables

Uses Apache Parquet as an intermediate format:

Source DB → Parquet Files → Target DB

Advantages:

• Chunked processing prevents memory issues
• Checkpointing for resumable migrations
• Works well with datetime-partitioned data

Sling Mode

ℹStreaming transfer

Direct database-to-database streaming via Sling CLI:

Source DB → Sling CLI → Target DB

Advantages:

• No intermediate files
• Faster for smaller tables
• Lower disk space requirements

Progress Monitoring

During migration, the page displays:

• Progress bar - Overall completion percentage
• Current chunk - Which chunk is being processed
• Records processed - Running count of migrated rows
• Log output - Real-time migration logs

tip

Large migrations can be monitored from the Prefect Flows page for better visibility and history tracking.

Troubleshooting

Connection Errors

Error	Solution
"Unable to connect to source"	Check source database credentials in .env
"Unable to connect to target"	Check target database credentials in .env
"Azure AD authentication failed"	Verify Service Principal credentials

Migration Failures

Error	Solution
"Out of memory"	Reduce chunk_size or switch to Sling mode
"Timeout"	Increase connection timeout or use smaller batches
"Schema mismatch"	Ensure target table exists or enable auto-create