Data Storage Modules 0.41.1Last updated in version 0.40.3

DMS Module

This module creates an Amazon Database Migration Service (DMS) that makes it possible to migrate data from one source database to another target database. You can use this module to migrate your data into the AWS Cloud or between combinations of cloud and on-premise setups. Currently, this module only support MySQL, MariaDB, and Aurora MySQL engines as source and target databases. Additional support for more database engines will be added in the future.

Sample Usage

Terraform
Terragrunt

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S DMS MODULE
# ------------------------------------------------------------------------------------------------------

module "dms" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-data-storage.git//modules/dms?ref=v0.41.1"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The compute and memory capacity of the replication instance as specified by
  # the replication instance class
  instance_type = <string>

  # The name used to namespace all resources created by these templates,
  # including the DB instance (e.g. drupaldb). Must be unique for this region.
  # May contain only lowercase alphanumeric characters, hyphens, underscores,
  # periods, and spaces.
  name = <string>

  # A list of subnet ids where the Replication Instance should be deployed. In
  # the standard Gruntwork VPC setup, these should be the private persistence
  # subnet ids. This is ignored if create_subnet_group=false.
  subnet_ids = <list(string)>

  # The migration type. Can be one of `full-load` | `cdc` | `full-load-and-cdc`.
  task_migration_type = <string>

  # An escaped JSON string that contains the table mappings. For information on
  # table mapping see
  # http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html
  task_table_mappings = <string>

  # The id of the VPC in which this Replication Instance should be deployed.
  vpc_id = <string>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # If false, the DMS instance will bind to `instance_subnet_group_id` variable.
  create_subnet_group = true

  # A map of custom tags to apply to the DMS Instance and the Security Group
  # created for it. The key is the tag name and the value is the tag value.
  custom_tags = {}

  # The amount of storage (in gigabytes) to be initially allocated for the
  # replication instance. Min: 5, Max: 6144, Default: 50
  instance_allocated_storage = null

  # Indicates that major version upgrades are allowed
  instance_allow_major_version_upgrade = true

  # Indicates whether the changes should be applied immediately or during the
  # next maintenance window
  instance_apply_immediately = null

  # Indicates that minor engine upgrades will be applied automatically to the
  # replication instance during the maintenance window
  instance_auto_minor_version_upgrade = true

  # The EC2 Availability Zone that the replication instance will be created in
  instance_availability_zone = null

  # The [engine
  # version](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReleaseNotes.html)
  # number of the replication instance
  instance_engine_version = null

  # The replication instance identifier. This parameter is stored as a lowercase
  # string
  instance_id = null

  # The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt
  # the connection parameters
  instance_kms_key_arn = null

  # Specifies if the replication instance is a multi-az deployment. You cannot
  # set the `availability_zone` parameter if the `multi_az` parameter is set to
  # `true`
  instance_multi_az = null

  # The weekly time range during which system maintenance can occur, in
  # Universal Coordinated Time (UTC)
  instance_preferred_maintenance_window = null

  # Specifies the accessibility options for the replication instance
  instance_publicly_accessible = null

  # An existing subnet group to associate with the replication instance
  instance_subnet_group_id = null

  # A map of additional tags to apply to the replication instance
  instance_tags = {}

  # A map of timeouts for replication instance create/update/delete operations
  instance_timeouts = {}

  # A list of VPC security group IDs to be used with the replication instance
  instance_vpc_security_group_ids = null

  # Name of the endpoint database
  source_endpoint_database_name = null

  # Type of engine for the endpoint.
  source_endpoint_engine_name = "mysql"

  # Additional attributes associated with the connection. For available
  # attributes for a source Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For
  # available attributes for a target Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.
  source_endpoint_extra_connection_attributes = null

  # ARN for the KMS key that will be used to encrypt the connection parameters.
  # If you do not specify a value for kms_key_arn, then AWS DMS will use your
  # default encryption key. AWS KMS creates the default encryption key for your
  # AWS account. Your AWS account has a different default encryption key for
  # each AWS region
  source_endpoint_kms_key_arn = null

  # Password to be used to login to the endpoint database.
  source_endpoint_password = null

  #  Port used by the endpoint database.
  source_endpoint_port = null

  # (Required if using non-s3 source) Host name of the server.
  source_endpoint_server_name = null

  # SSL mode to use for the connection. Valid values are none, require,
  # verify-ca, verify-full
  source_endpoint_ssl_mode = "none"

  # User name to be used to login to the endpoint database.
  source_endpoint_username = null

  # (Required if using s3) S3 bucket name.
  source_s3_bucket_name = null

  # (Required if using s3 for CDC; otherwise, Optional) Folder path of CDC
  # files. If cdc_path is set, AWS DMS reads CDC files from this path and
  # replicates the data changes to the target endpoint. Supported in AWS DMS
  # versions 3.4.2 and later.
  source_s3_cdc_path = null

  # (Required if using s3) JSON document that describes how AWS DMS should
  # interpret the data.
  source_s3_external_table_definition = null

  # ARN of the IAM role with permissions to the S3 Bucket. Default one will be
  # created if not specified
  source_s3_service_access_role_arn = null

  # The description of the aws_dms_replication_subnet_group that is created.
  # Defaults to 'Subnet group for the var.name DB' if not specified.
  subnet_group_description = null

  # The name of the aws_dms_replication_subnet_group that is created, or an
  # existing one to use if create_subnet_group is false. Defaults to var.name if
  # not specified.
  subnet_group_name = null

  # Name of the endpoint database
  target_endpoint_database_name = null

  # Type of engine for the endpoint. 
  target_endpoint_engine_name = "mysql"

  # Additional attributes associated with the connection. For available
  # attributes for a source Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For
  # available attributes for a target Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.
  target_endpoint_extra_connection_attributes = null

  # ARN for the KMS key that will be used to encrypt the connection parameters.
  # If you do not specify a value for kms_key_arn, then AWS DMS will use your
  # default encryption key. AWS KMS creates the default encryption key for your
  # AWS account. Your AWS account has a different default encryption key for
  # each AWS region
  target_endpoint_kms_key_arn = null

  # Password to be used to login to the endpoint database.
  target_endpoint_password = null

  #  Port used by the endpoint database.
  target_endpoint_port = null

  # (Required if using non-s3 target) Host name of the server.
  target_endpoint_server_name = null

  # SSL mode to use for the connection. Valid values are none, require,
  # verify-ca, verify-full
  target_endpoint_ssl_mode = "none"

  # User name to be used to login to the endpoint database.
  target_endpoint_username = null

  # (Required if using s3) S3 bucket name.
  target_s3_bucket_name = null

  # Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from
  # this path and replicates the data changes to the target endpoint. Supported
  # in AWS DMS versions 3.4.2 and later.
  target_s3_cdc_path = null

  # JSON document that describes how AWS DMS should interpret the data.
  target_s3_external_table_definition = null

  # ARN of the IAM role with permissions to the S3 Bucket. Default one will be
  # created if not specified
  target_s3_service_access_role_arn = null

  # (Conflicts with task_cdc_start_time) Indicates when you want a change data
  # capture (CDC) operation to start. The value can be in date, checkpoint, or
  # LSN/SCN format depending on the source engine. For more information, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html#CHAP_Task.CDC.StartPoint.Native.
  task_cdc_start_position = null

  # (Conflicts with task_cdc_start_position) The Unix timestamp integer for the
  # start of the Change Data Capture (CDC) operation.
  task_cdc_start_time = null

  # An escaped JSON string that contains the task settings. For a complete list
  # of task settings, see
  # http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html.
  task_settings = null

  # Whether to run or stop the replication task.
  task_start_replication_task = false

}

terragrunt.hcl

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S DMS MODULE
# ------------------------------------------------------------------------------------------------------

terraform {
  source = "git::git@github.com:gruntwork-io/terraform-aws-data-storage.git//modules/dms?ref=v0.41.1"
}

inputs = {

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The compute and memory capacity of the replication instance as specified by
  # the replication instance class
  instance_type = <string>

  # The name used to namespace all resources created by these templates,
  # including the DB instance (e.g. drupaldb). Must be unique for this region.
  # May contain only lowercase alphanumeric characters, hyphens, underscores,
  # periods, and spaces.
  name = <string>

  # A list of subnet ids where the Replication Instance should be deployed. In
  # the standard Gruntwork VPC setup, these should be the private persistence
  # subnet ids. This is ignored if create_subnet_group=false.
  subnet_ids = <list(string)>

  # The migration type. Can be one of `full-load` | `cdc` | `full-load-and-cdc`.
  task_migration_type = <string>

  # An escaped JSON string that contains the table mappings. For information on
  # table mapping see
  # http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html
  task_table_mappings = <string>

  # The id of the VPC in which this Replication Instance should be deployed.
  vpc_id = <string>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # If false, the DMS instance will bind to `instance_subnet_group_id` variable.
  create_subnet_group = true

  # A map of custom tags to apply to the DMS Instance and the Security Group
  # created for it. The key is the tag name and the value is the tag value.
  custom_tags = {}

  # The amount of storage (in gigabytes) to be initially allocated for the
  # replication instance. Min: 5, Max: 6144, Default: 50
  instance_allocated_storage = null

  # Indicates that major version upgrades are allowed
  instance_allow_major_version_upgrade = true

  # Indicates whether the changes should be applied immediately or during the
  # next maintenance window
  instance_apply_immediately = null

  # Indicates that minor engine upgrades will be applied automatically to the
  # replication instance during the maintenance window
  instance_auto_minor_version_upgrade = true

  # The EC2 Availability Zone that the replication instance will be created in
  instance_availability_zone = null

  # The [engine
  # version](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReleaseNotes.html)
  # number of the replication instance
  instance_engine_version = null

  # The replication instance identifier. This parameter is stored as a lowercase
  # string
  instance_id = null

  # The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt
  # the connection parameters
  instance_kms_key_arn = null

  # Specifies if the replication instance is a multi-az deployment. You cannot
  # set the `availability_zone` parameter if the `multi_az` parameter is set to
  # `true`
  instance_multi_az = null

  # The weekly time range during which system maintenance can occur, in
  # Universal Coordinated Time (UTC)
  instance_preferred_maintenance_window = null

  # Specifies the accessibility options for the replication instance
  instance_publicly_accessible = null

  # An existing subnet group to associate with the replication instance
  instance_subnet_group_id = null

  # A map of additional tags to apply to the replication instance
  instance_tags = {}

  # A map of timeouts for replication instance create/update/delete operations
  instance_timeouts = {}

  # A list of VPC security group IDs to be used with the replication instance
  instance_vpc_security_group_ids = null

  # Name of the endpoint database
  source_endpoint_database_name = null

  # Type of engine for the endpoint.
  source_endpoint_engine_name = "mysql"

  # Additional attributes associated with the connection. For available
  # attributes for a source Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For
  # available attributes for a target Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.
  source_endpoint_extra_connection_attributes = null

  # ARN for the KMS key that will be used to encrypt the connection parameters.
  # If you do not specify a value for kms_key_arn, then AWS DMS will use your
  # default encryption key. AWS KMS creates the default encryption key for your
  # AWS account. Your AWS account has a different default encryption key for
  # each AWS region
  source_endpoint_kms_key_arn = null

  # Password to be used to login to the endpoint database.
  source_endpoint_password = null

  #  Port used by the endpoint database.
  source_endpoint_port = null

  # (Required if using non-s3 source) Host name of the server.
  source_endpoint_server_name = null

  # SSL mode to use for the connection. Valid values are none, require,
  # verify-ca, verify-full
  source_endpoint_ssl_mode = "none"

  # User name to be used to login to the endpoint database.
  source_endpoint_username = null

  # (Required if using s3) S3 bucket name.
  source_s3_bucket_name = null

  # (Required if using s3 for CDC; otherwise, Optional) Folder path of CDC
  # files. If cdc_path is set, AWS DMS reads CDC files from this path and
  # replicates the data changes to the target endpoint. Supported in AWS DMS
  # versions 3.4.2 and later.
  source_s3_cdc_path = null

  # (Required if using s3) JSON document that describes how AWS DMS should
  # interpret the data.
  source_s3_external_table_definition = null

  # ARN of the IAM role with permissions to the S3 Bucket. Default one will be
  # created if not specified
  source_s3_service_access_role_arn = null

  # The description of the aws_dms_replication_subnet_group that is created.
  # Defaults to 'Subnet group for the var.name DB' if not specified.
  subnet_group_description = null

  # The name of the aws_dms_replication_subnet_group that is created, or an
  # existing one to use if create_subnet_group is false. Defaults to var.name if
  # not specified.
  subnet_group_name = null

  # Name of the endpoint database
  target_endpoint_database_name = null

  # Type of engine for the endpoint. 
  target_endpoint_engine_name = "mysql"

  # Additional attributes associated with the connection. For available
  # attributes for a source Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For
  # available attributes for a target Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.
  target_endpoint_extra_connection_attributes = null

  # ARN for the KMS key that will be used to encrypt the connection parameters.
  # If you do not specify a value for kms_key_arn, then AWS DMS will use your
  # default encryption key. AWS KMS creates the default encryption key for your
  # AWS account. Your AWS account has a different default encryption key for
  # each AWS region
  target_endpoint_kms_key_arn = null

  # Password to be used to login to the endpoint database.
  target_endpoint_password = null

  #  Port used by the endpoint database.
  target_endpoint_port = null

  # (Required if using non-s3 target) Host name of the server.
  target_endpoint_server_name = null

  # SSL mode to use for the connection. Valid values are none, require,
  # verify-ca, verify-full
  target_endpoint_ssl_mode = "none"

  # User name to be used to login to the endpoint database.
  target_endpoint_username = null

  # (Required if using s3) S3 bucket name.
  target_s3_bucket_name = null

  # Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from
  # this path and replicates the data changes to the target endpoint. Supported
  # in AWS DMS versions 3.4.2 and later.
  target_s3_cdc_path = null

  # JSON document that describes how AWS DMS should interpret the data.
  target_s3_external_table_definition = null

  # ARN of the IAM role with permissions to the S3 Bucket. Default one will be
  # created if not specified
  target_s3_service_access_role_arn = null

  # (Conflicts with task_cdc_start_time) Indicates when you want a change data
  # capture (CDC) operation to start. The value can be in date, checkpoint, or
  # LSN/SCN format depending on the source engine. For more information, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html#CHAP_Task.CDC.StartPoint.Native.
  task_cdc_start_position = null

  # (Conflicts with task_cdc_start_position) The Unix timestamp integer for the
  # start of the Change Data Capture (CDC) operation.
  task_cdc_start_time = null

  # An escaped JSON string that contains the task settings. For a complete list
  # of task settings, see
  # http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html.
  task_settings = null

  # Whether to run or stop the replication task.
  task_start_replication_task = false

}

Reference

Inputs
Outputs

Required

instance_typestringrequired

The compute and memory capacity of the replication instance as specified by the replication instance class

namestringrequired

The name used to namespace all resources created by these templates, including the DB instance (e.g. drupaldb). Must be unique for this region. May contain only lowercase alphanumeric characters, hyphens, underscores, periods, and spaces.

subnet_idslist(string)required

A list of subnet ids where the Replication Instance should be deployed. In the standard Gruntwork VPC setup, these should be the private persistence subnet ids. This is ignored if create_subnet_group=false.

task_migration_typestringrequired

The migration type. Can be one of full-load | cdc | full-load-and-cdc.

task_table_mappingsstringrequired

An escaped JSON string that contains the table mappings. For information on table mapping see http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html

vpc_idstringrequired

The id of the VPC in which this Replication Instance should be deployed.

Optional

create_subnet_groupbooloptional

If false, the DMS instance will bind to instance_subnet_group_id variable.

Default:true

custom_tagsmap(string)optional

A map of custom tags to apply to the DMS Instance and the Security Group created for it. The key is the tag name and the value is the tag value.

Default:{}

instance_allocated_storagenumberoptional

The amount of storage (in gigabytes) to be initially allocated for the replication instance. Min: 5, Max: 6144, Default: 50

Default:null

instance_allow_major_version_upgradebooloptional

Indicates that major version upgrades are allowed

Default:true

instance_apply_immediatelybooloptional

Indicates whether the changes should be applied immediately or during the next maintenance window

Default:null

instance_auto_minor_version_upgradebooloptional

Indicates that minor engine upgrades will be applied automatically to the replication instance during the maintenance window

Default:true

instance_availability_zonestringoptional

The EC2 Availability Zone that the replication instance will be created in

Default:null

instance_engine_versionstringoptional

The engine version number of the replication instance

Default:null

instance_idstringoptional

The replication instance identifier. This parameter is stored as a lowercase string

Default:null

instance_kms_key_arnstringoptional

The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt the connection parameters

Default:null

instance_multi_azbooloptional

Specifies if the replication instance is a multi-az deployment. You cannot set the availability_zone parameter if the multi_az parameter is set to true

Default:null

instance_preferred_maintenance_windowstringoptional

The weekly time range during which system maintenance can occur, in Universal Coordinated Time (UTC)

Default:null

instance_publicly_accessiblebooloptional

Specifies the accessibility options for the replication instance

Default:null

instance_subnet_group_idstringoptional

An existing subnet group to associate with the replication instance

Default:null

instance_tagsmap(string)optional

A map of additional tags to apply to the replication instance

Default:{}

instance_timeoutsmap(string)optional

A map of timeouts for replication instance create/update/delete operations

Default:{}

instance_vpc_security_group_idslist(string)optional

A list of VPC security group IDs to be used with the replication instance

Default:null

source_endpoint_database_namestringoptional

Name of the endpoint database

Default:null

source_endpoint_engine_namestringoptional

Type of engine for the endpoint.

Default:"mysql"

source_endpoint_extra_connection_attributesstringoptional

Additional attributes associated with the connection. For available attributes for a source Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For available attributes for a target Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.

Default:null

source_endpoint_kms_key_arnstringoptional

ARN for the KMS key that will be used to encrypt the connection parameters. If you do not specify a value for kms_key_arn, then AWS DMS will use your default encryption key. AWS KMS creates the default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS region

Default:null

source_endpoint_passwordstringoptional

Password to be used to login to the endpoint database.

Default:null

source_endpoint_portnumberoptional

Port used by the endpoint database.

Default:null

source_endpoint_server_namestringoptional

(Required if using non-s3 source) Host name of the server.

Default:null

source_endpoint_ssl_modestringoptional

SSL mode to use for the connection. Valid values are none, require, verify-ca, verify-full

Default:"none"

source_endpoint_usernamestringoptional

User name to be used to login to the endpoint database.

Default:null

source_s3_bucket_namestringoptional

(Required if using s3) S3 bucket name.

Default:null

source_s3_cdc_pathstringoptional

(Required if using s3 for CDC; otherwise, Optional) Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from this path and replicates the data changes to the target endpoint. Supported in AWS DMS versions 3.4.2 and later.

Default:null

source_s3_external_table_definitionstringoptional

(Required if using s3) JSON document that describes how AWS DMS should interpret the data.

Default:null

source_s3_service_access_role_arnstringoptional

ARN of the IAM role with permissions to the S3 Bucket. Default one will be created if not specified

Default:null

subnet_group_descriptionstringoptional

The description of the aws_dms_replication_subnet_group that is created. Defaults to 'Subnet group for the name DB' if not specified.

Default:null

subnet_group_namestringoptional

The name of the aws_dms_replication_subnet_group that is created, or an existing one to use if create_subnet_group is false. Defaults to name if not specified.

Default:null

target_endpoint_database_namestringoptional

Name of the endpoint database

Default:null

target_endpoint_engine_namestringoptional

Type of engine for the endpoint.

Default:"mysql"

target_endpoint_extra_connection_attributesstringoptional

Default:null

target_endpoint_kms_key_arnstringoptional

Default:null

target_endpoint_passwordstringoptional

Password to be used to login to the endpoint database.

Default:null

target_endpoint_portnumberoptional

Port used by the endpoint database.

Default:null

target_endpoint_server_namestringoptional

(Required if using non-s3 target) Host name of the server.

Default:null

target_endpoint_ssl_modestringoptional

SSL mode to use for the connection. Valid values are none, require, verify-ca, verify-full

Default:"none"

target_endpoint_usernamestringoptional

User name to be used to login to the endpoint database.

Default:null

target_s3_bucket_namestringoptional

(Required if using s3) S3 bucket name.

Default:null

target_s3_cdc_pathstringoptional

Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from this path and replicates the data changes to the target endpoint. Supported in AWS DMS versions 3.4.2 and later.

Default:null

target_s3_external_table_definitionstringoptional

JSON document that describes how AWS DMS should interpret the data.

Default:null

target_s3_service_access_role_arnstringoptional

ARN of the IAM role with permissions to the S3 Bucket. Default one will be created if not specified

Default:null

task_cdc_start_positionstringoptional

(Conflicts with task_cdc_start_time) Indicates when you want a change data capture (CDC) operation to start. The value can be in date, checkpoint, or LSN/SCN format depending on the source engine. For more information, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html#CHAP_Task.CDC.StartPoint.Native.

Default:null

task_cdc_start_timestringoptional

(Conflicts with task_cdc_start_position) The Unix timestamp integer for the start of the Change Data Capture (CDC) operation.

Default:null

task_settingsstringoptional

An escaped JSON string that contains the task settings. For a complete list of task settings, see http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html.

Default:null

task_start_replication_taskbooloptional

Whether to run or stop the replication task.

Default:false

endpoints

A map of maps containing the endpoints created and their full output of attributes and values