Optimizing migration performance

You can leverage the EDB Data Migration Service (DMS), a service included with the Hybrid Manager (HM), to migrate your data from existing databases (Oracle, self-managed Postgres, etc.) to a HM-managed database cluster. EDB DMS supports two migration methods: snapshot and snapshot + streaming, allowing you to migrate your data with minimal to no downtime.

Understanding migration workloads in HM

The EDB Data Migration Service runs within the same Kubernetes environment where your EDB Hybrid Manager is deployed (on-premises, AWS, Red Hat OpenShift, and Google Cloud). This means that most of the workloads required for migration operations are managed and optimized by your Kubernetes cluster.

Generally, you don't need to extensively fine-tune resources, as your Kubernetes environment meeting the general requirements for running the Hybrid Manager should ensure a smooth migration. However, for large datasets, optimizing specific components can significantly accelerate the process.

The key components to consider for resource allocation are:

• EDB DMS Reader: This component can run on the same machine as the source database, or on an auxiliary machine that can connect to the source database and to the Hybrid Manager.

• Destination database cluster: This can be the database managed by HM or a different external self-managed database, however here we'll focus on an HM-managed database. This is where your data will reside.

• EDB DMS Writer: This component is only required if your migration destination is an external (self-managed) database cluster, and it runs on that database's host machine.

Whether you perform a snapshot or snapshot + streaming migration also affects performance differently:

• Snapshot migrations: They primarily focus on efficiently transferring a large volume of existing data from a source database to the destination. Performance for these is heavily influenced by total data size and sustained disk I/O.

• Snapshot + streaming migrations: They first transfer a snapshot of existing data and then continuously stream ongoing changes (transactions) from the source to the destination. Performance here is driven by the sustained transaction rate (TPS) and the ability to keep up with the source's write activity.

Optimizing migration performance for large datasets

The HM and DMS are designed to handle resource usage automatically. For datasets smaller than 100 GB, they are configured to perform fast migrations without further tuning. However, for datasets larger than 100 GB, a few targeted alterations can lead to significantly better performance and quicker migrations.

Tuning destination database performance

Optimizing the destination database cluster (whether an HM-managed database or a self-managed database) provides the most significant performance benefits from resource tuning.

Storage recommendations

During a migration, storage IOPS can peak at 2000, with an average throughput of roughly 300 MB/s and peaks up to 350 MB/s. To accommodate these demands:

• Separate WAL and data volumes: Aim for storage volumes that can handle a minimum of 2500-3000 IOPS and at least 200-250 MB/s of throughput for both your data and Write-Ahead Log (WAL) volumes. Using separate volumes for WAL and data is highly recommended for optimal performance.

• Unified storage and WAL volume: A unified storage and WAL volume can adversely affect performance. For example, using a single volume with 3000 IOPS and 125 MB/s throughput can increase migration time by over 40%. If you must use a unified storage and WAL volume, we recommend a throughput of 300-400 MB/s. Keep in mind that if the WAL becomes full, the migration will fail.

Memory recommendations

While 5 GB of memory is sufficient, a larger amount will facilitate optimal performance during a migration. Use at least 10 GB of memory for the destination database.

CPU recommendations

The destination database cluster should have at least 4 CPU cores. A lower core count will negatively impact migration performance.

Extensive testing has shown that the biggest factor affecting migration performance is volume speed (IOPS and throughput). While decreasing CPU below recommendations can reduce performance, it's not nearly to the degree that low IOPS and throughput for storage will. Halving the storage throughput can increase migration time by over 40%, whereas halving the CPU count to 2 cores might increase it by 10-15%. These recommendations are designed to yield the best migration performance; using lower values may still result in successful migrations but at a slower speed.

Tuning EBD DMS Reader and Writer performance

Optimal performance for the EDB DMS Reader and Writer is achieved when the host machines where these components are installed follow these specifications.

EDB DMS Reader

CPU: 3 CPU cores

Memory: 4 GB

EDB DMS Writer

CPU: 2 CPU cores

Memory: 2.5 GB

Shared host considerations

For optimal migration performance, carefully consider resource allocation when EDB DMS components and source database run on the same machine/host.

• DMS Reader/Writer share a host: If both DMS Reader and Writer are on the same machine, provision combined resources (at least 5 CPU cores and 6.5 GB memory).

• DMS Component and source database share a host: If a DMS Reader or Writer is on the same host as the source database, ensure the host's resources are significantly increased to accommodate both the database's operational workload and the migration's demands.