Skip to main content
Data Storage Modules 0.42.0Last updated in version 0.40.3

DMS Module

View Source Release Notes

This module creates AWS Database Migration Service (DMS) resources for database migration and replication.

What This Module Creates

  • DMS replication instance
  • Source and target endpoints
  • Replication tasks
  • Required IAM roles and policies
  • CloudWatch log groups
  • Security groups for network access

Supported Engines

Currently supports:

  • MySQL
  • MariaDB
  • Aurora MySQL

(As source and target)

Migration Types

The module supports three migration types via migration_type variable:

  • full-load - One-time migration
  • cdc - Ongoing replication only
  • full-load-and-cdc - Full migration plus ongoing replication

Usage

module "dms" {
source = "../modules/dms"

name = "my-database-migration"

# Replication instance
replication_instance_class = "dms.t3.medium"
allocated_storage = 100

# Source endpoint
source_endpoint_config = {
endpoint_id = "source-mysql"
endpoint_type = "source"
engine_name = "mysql"
server_name = "source.example.com"
port = 3306
username = var.source_username
password = var.source_password
}

# Target endpoint
target_endpoint_config = {
endpoint_id = "target-aurora"
endpoint_type = "target"
engine_name = "aurora"
server_name = "target.cluster.amazonaws.com"
port = 3306
username = var.target_username
password = var.target_password
}

migration_type = "full-load-and-cdc"
}

Configuration

  • See the root README for instructions on using Terraform modules.
  • See the variables.tf for all the variables you can set on this module.
  • See the dms-mysql examples folder for instruction on how to setup the modules to migrate data from an AWS RDS MySQL Instance to another AWS RDS MySQL Instance.
  • See the dms-aurora examples folder for instructions on how to setup the modules to migrate data from an AWS RDS MySQL Instance to another AWS RDS MySQL Instance.

Key Variables

  • name - Name prefix for all DMS resources
  • replication_instance_class - Instance size (e.g., dms.t3.medium)
  • allocated_storage - Storage in GB
  • vpc_id - VPC for deployment
  • subnet_ids - Subnets for replication instance
  • migration_type - full-load, cdc, or full-load-and-cdc

Common Issues

  • Network: Ensure replication instance can reach both databases
  • Storage: Allocate enough for your data volume
  • Primary keys: Required for CDC performance
  • LOB columns: May need special handling

Outputs

  • replication_instance_arn - ARN of replication instance
  • source_endpoint_arn - Source endpoint ARN
  • target_endpoint_arn - Target endpoint ARN
  • replication_task_arn - Task ARN

Sample Usage

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S DMS MODULE
# ------------------------------------------------------------------------------------------------------

module "dms" {

source = "git::git@github.com:gruntwork-io/terraform-aws-data-storage.git//modules/dms?ref=v0.42.0"

# ----------------------------------------------------------------------------------------------------
# REQUIRED VARIABLES
# ----------------------------------------------------------------------------------------------------

# The compute and memory capacity of the replication instance as specified by
# the replication instance class
instance_type = <string>

# The name used to namespace all resources created by these templates,
# including the DB instance (e.g. drupaldb). Must be unique for this region.
# May contain only lowercase alphanumeric characters, hyphens, underscores,
# periods, and spaces.
name = <string>

# A list of subnet ids where the Replication Instance should be deployed. In
# the standard Gruntwork VPC setup, these should be the private persistence
# subnet ids. This is ignored if create_subnet_group=false.
subnet_ids = <list(string)>

# The migration type. Can be one of `full-load` | `cdc` | `full-load-and-cdc`.
task_migration_type = <string>

# An escaped JSON string that contains the table mappings. For information on
# table mapping see
# http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html
task_table_mappings = <string>

# The id of the VPC in which this Replication Instance should be deployed.
vpc_id = <string>

# ----------------------------------------------------------------------------------------------------
# OPTIONAL VARIABLES
# ----------------------------------------------------------------------------------------------------

# If false, the DMS instance will bind to `instance_subnet_group_id` variable.
create_subnet_group = true

# A map of custom tags to apply to the DMS Instance and the Security Group
# created for it. The key is the tag name and the value is the tag value.
custom_tags = {}

# The amount of storage (in gigabytes) to be initially allocated for the
# replication instance. Min: 5, Max: 6144, Default: 50
instance_allocated_storage = null

# Indicates that major version upgrades are allowed
instance_allow_major_version_upgrade = true

# Indicates whether the changes should be applied immediately or during the
# next maintenance window
instance_apply_immediately = null

# Indicates that minor engine upgrades will be applied automatically to the
# replication instance during the maintenance window
instance_auto_minor_version_upgrade = true

# The EC2 Availability Zone that the replication instance will be created in
instance_availability_zone = null

# The [engine
# version](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReleaseNotes.html)
# number of the replication instance
instance_engine_version = null

# The replication instance identifier. This parameter is stored as a lowercase
# string
instance_id = null

# The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt
# the connection parameters
instance_kms_key_arn = null

# Specifies if the replication instance is a multi-az deployment. You cannot
# set the `availability_zone` parameter if the `multi_az` parameter is set to
# `true`
instance_multi_az = null

# The weekly time range during which system maintenance can occur, in
# Universal Coordinated Time (UTC)
instance_preferred_maintenance_window = null

# Specifies the accessibility options for the replication instance
instance_publicly_accessible = null

# An existing subnet group to associate with the replication instance
instance_subnet_group_id = null

# A map of additional tags to apply to the replication instance
instance_tags = {}

# A map of timeouts for replication instance create/update/delete operations
instance_timeouts = {}

# A list of VPC security group IDs to be used with the replication instance
instance_vpc_security_group_ids = null

# Name of the endpoint database
source_endpoint_database_name = null

# Type of engine for the endpoint.
source_endpoint_engine_name = "mysql"

# Additional attributes associated with the connection. For available
# attributes for a source Endpoint, see
# https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For
# available attributes for a target Endpoint, see
# https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.
source_endpoint_extra_connection_attributes = null

# ARN for the KMS key that will be used to encrypt the connection parameters.
# If you do not specify a value for kms_key_arn, then AWS DMS will use your
# default encryption key. AWS KMS creates the default encryption key for your
# AWS account. Your AWS account has a different default encryption key for
# each AWS region
source_endpoint_kms_key_arn = null

# Password to be used to login to the endpoint database.
source_endpoint_password = null

# Port used by the endpoint database.
source_endpoint_port = null

# (Required if using non-s3 source) Host name of the server.
source_endpoint_server_name = null

# SSL mode to use for the connection. Valid values are none, require,
# verify-ca, verify-full
source_endpoint_ssl_mode = "none"

# User name to be used to login to the endpoint database.
source_endpoint_username = null

# (Required if using s3) S3 bucket name.
source_s3_bucket_name = null

# (Required if using s3 for CDC; otherwise, Optional) Folder path of CDC
# files. If cdc_path is set, AWS DMS reads CDC files from this path and
# replicates the data changes to the target endpoint. Supported in AWS DMS
# versions 3.4.2 and later.
source_s3_cdc_path = null

# (Required if using s3) JSON document that describes how AWS DMS should
# interpret the data.
source_s3_external_table_definition = null

# ARN of the IAM role with permissions to the S3 Bucket. Default one will be
# created if not specified
source_s3_service_access_role_arn = null

# The description of the aws_dms_replication_subnet_group that is created.
# Defaults to 'Subnet group for the var.name DB' if not specified.
subnet_group_description = null

# The name of the aws_dms_replication_subnet_group that is created, or an
# existing one to use if create_subnet_group is false. Defaults to var.name if
# not specified.
subnet_group_name = null

# Name of the endpoint database
target_endpoint_database_name = null

# Type of engine for the endpoint.
target_endpoint_engine_name = "mysql"

# Additional attributes associated with the connection. For available
# attributes for a source Endpoint, see
# https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For
# available attributes for a target Endpoint, see
# https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.
target_endpoint_extra_connection_attributes = null

# ARN for the KMS key that will be used to encrypt the connection parameters.
# If you do not specify a value for kms_key_arn, then AWS DMS will use your
# default encryption key. AWS KMS creates the default encryption key for your
# AWS account. Your AWS account has a different default encryption key for
# each AWS region
target_endpoint_kms_key_arn = null

# Password to be used to login to the endpoint database.
target_endpoint_password = null

# Port used by the endpoint database.
target_endpoint_port = null

# (Required if using non-s3 target) Host name of the server.
target_endpoint_server_name = null

# SSL mode to use for the connection. Valid values are none, require,
# verify-ca, verify-full
target_endpoint_ssl_mode = "none"

# User name to be used to login to the endpoint database.
target_endpoint_username = null

# (Required if using s3) S3 bucket name.
target_s3_bucket_name = null

# Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from
# this path and replicates the data changes to the target endpoint. Supported
# in AWS DMS versions 3.4.2 and later.
target_s3_cdc_path = null

# JSON document that describes how AWS DMS should interpret the data.
target_s3_external_table_definition = null

# ARN of the IAM role with permissions to the S3 Bucket. Default one will be
# created if not specified
target_s3_service_access_role_arn = null

# (Conflicts with task_cdc_start_time) Indicates when you want a change data
# capture (CDC) operation to start. The value can be in date, checkpoint, or
# LSN/SCN format depending on the source engine. For more information, see
# https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html#CHAP_Task.CDC.StartPoint.Native.
task_cdc_start_position = null

# (Conflicts with task_cdc_start_position) The Unix timestamp integer for the
# start of the Change Data Capture (CDC) operation.
task_cdc_start_time = null

# An escaped JSON string that contains the task settings. For a complete list
# of task settings, see
# http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html.
task_settings = null

# Whether to run or stop the replication task.
task_start_replication_task = false

}


Reference

Required

instance_typestringrequired

The compute and memory capacity of the replication instance as specified by the replication instance class

namestringrequired

The name used to namespace all resources created by these templates, including the DB instance (e.g. drupaldb). Must be unique for this region. May contain only lowercase alphanumeric characters, hyphens, underscores, periods, and spaces.

subnet_idslist(string)required

A list of subnet ids where the Replication Instance should be deployed. In the standard Gruntwork VPC setup, these should be the private persistence subnet ids. This is ignored if create_subnet_group=false.

task_migration_typestringrequired

The migration type. Can be one of full-load | cdc | full-load-and-cdc.

task_table_mappingsstringrequired

An escaped JSON string that contains the table mappings. For information on table mapping see http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html

vpc_idstringrequired

The id of the VPC in which this Replication Instance should be deployed.

Optional

If false, the DMS instance will bind to instance_subnet_group_id variable.

true
custom_tagsmap(string)optional

A map of custom tags to apply to the DMS Instance and the Security Group created for it. The key is the tag name and the value is the tag value.

{}

The amount of storage (in gigabytes) to be initially allocated for the replication instance. Min: 5, Max: 6144, Default: 50

null

Indicates that major version upgrades are allowed

true

Indicates whether the changes should be applied immediately or during the next maintenance window

null

Indicates that minor engine upgrades will be applied automatically to the replication instance during the maintenance window

true

The EC2 Availability Zone that the replication instance will be created in

null

The engine version number of the replication instance

null
instance_idstringoptional

The replication instance identifier. This parameter is stored as a lowercase string

null
instance_kms_key_arnstringoptional

The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt the connection parameters

null
instance_multi_azbooloptional

Specifies if the replication instance is a multi-az deployment. You cannot set the availability_zone parameter if the multi_az parameter is set to true

null

The weekly time range during which system maintenance can occur, in Universal Coordinated Time (UTC)

null

Specifies the accessibility options for the replication instance

null

An existing subnet group to associate with the replication instance

null
instance_tagsmap(string)optional

A map of additional tags to apply to the replication instance

{}
instance_timeoutsmap(string)optional

A map of timeouts for replication instance create/update/delete operations

{}

A list of VPC security group IDs to be used with the replication instance

null

Name of the endpoint database

null

Type of engine for the endpoint.

"mysql"

Additional attributes associated with the connection. For available attributes for a source Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For available attributes for a target Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.

null

ARN for the KMS key that will be used to encrypt the connection parameters. If you do not specify a value for kms_key_arn, then AWS DMS will use your default encryption key. AWS KMS creates the default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS region

null

Password to be used to login to the endpoint database.

null
source_endpoint_portnumberoptional

Port used by the endpoint database.

null

(Required if using non-s3 source) Host name of the server.

null

SSL mode to use for the connection. Valid values are none, require, verify-ca, verify-full

"none"

User name to be used to login to the endpoint database.

null
source_s3_bucket_namestringoptional

(Required if using s3) S3 bucket name.

null
source_s3_cdc_pathstringoptional

(Required if using s3 for CDC; otherwise, Optional) Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from this path and replicates the data changes to the target endpoint. Supported in AWS DMS versions 3.4.2 and later.

null

(Required if using s3) JSON document that describes how AWS DMS should interpret the data.

null

ARN of the IAM role with permissions to the S3 Bucket. Default one will be created if not specified

null

The description of the aws_dms_replication_subnet_group that is created. Defaults to 'Subnet group for the name DB' if not specified.

null
subnet_group_namestringoptional

The name of the aws_dms_replication_subnet_group that is created, or an existing one to use if create_subnet_group is false. Defaults to name if not specified.

null

Name of the endpoint database

null

Type of engine for the endpoint.

"mysql"

Additional attributes associated with the connection. For available attributes for a source Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For available attributes for a target Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.

null

ARN for the KMS key that will be used to encrypt the connection parameters. If you do not specify a value for kms_key_arn, then AWS DMS will use your default encryption key. AWS KMS creates the default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS region

null

Password to be used to login to the endpoint database.

null
target_endpoint_portnumberoptional

Port used by the endpoint database.

null

(Required if using non-s3 target) Host name of the server.

null

SSL mode to use for the connection. Valid values are none, require, verify-ca, verify-full

"none"

User name to be used to login to the endpoint database.

null
target_s3_bucket_namestringoptional

(Required if using s3) S3 bucket name.

null
target_s3_cdc_pathstringoptional

Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from this path and replicates the data changes to the target endpoint. Supported in AWS DMS versions 3.4.2 and later.

null

JSON document that describes how AWS DMS should interpret the data.

null

ARN of the IAM role with permissions to the S3 Bucket. Default one will be created if not specified

null

(Conflicts with task_cdc_start_time) Indicates when you want a change data capture (CDC) operation to start. The value can be in date, checkpoint, or LSN/SCN format depending on the source engine. For more information, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html#CHAP_Task.CDC.StartPoint.Native.

null
task_cdc_start_timestringoptional

(Conflicts with task_cdc_start_position) The Unix timestamp integer for the start of the Change Data Capture (CDC) operation.

null
task_settingsstringoptional

An escaped JSON string that contains the task settings. For a complete list of task settings, see http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html.

null

Whether to run or stop the replication task.

false