In this article, I will demonstrate “How to migrate AWS EC2 Instance to Google Cloud’s Compute Engine VM instance using Google’s native Migrate for Compute Engine Migration tool”.
In this rapidly growing technological world, Most companies are refactoring their infra to leverage the benefits of Machine learning, AI & Bigdata solutions, and Google Cloud Platform is the Leading Cloud Provider for modern technological solutions. Migrating workload virtual Machines is the first step to getting started with GCP & this may be a little bit of a blurring process for newcomers. Migrate for Compute Engine is one the most popular & Google’s native migration tools to achieve this & in this article, we’ll deep dig out into this tool while migrating demo EC2 instance to GCP.
- VPN Setup between AWS & GCP Environment
- Configuring required firewalls in GCP
- Creating Migration Manager in Migrate for Compute Engine
- Setting up AWS Environment
- Migration Agent Installation in AWS VMs
Let’s start the learning……..
Step 1: Creating a VPN between the VPC’s of AWS and GCP
For setting up your VPN, follow the below doc, in which you’ll find step-by-step instructions.
Step 2: Setup Firewall Configurations needed on the GCP side
- Below-specified firewall rules with specific tags are required in the migration process. For this demonstration: AWS VPC CIDR Range is 10.20.0.0/16 & GCP VPC CIDR Range is 172.16.0.0/16
Please follow the link for all firewall rules with detailed information.
Step 3: Creation & configuring migrate for compute engine Migration Manager
Navigation menu → compute engine → migrate on compute engine → try the new version
- Configure Network Connectivity
- Configure migration manager
Fill the all required fields according to your specifications.
Note:- Here we’re going to create two service accounts, so be sure that you’ve required roles with you so that you can create these service accounts.
- Migration Service account
- Migration Cloud Extension Service account
Don’t worry about what permissions are required by these service accounts. GCP console will automatically add the permissions to the service account if you’re creating new ones, so just click create a new survive account.
After adding a service account, your screen will look like this. Also, you can see what the permissions are going to use in the migration process.
At last, you’ve to create one user named apiuser you create that one also. Keep this password with you, you’ll need ID & Password for logging into the migration console & in the process of migration agent installation in AWS VM’s which are going to migrate to GCP (Especially Windows VM)
You can create a service account alone as well from IAM Console or CLI, The scripts mentioned in the given link provides the command to create the service account which will be required for the VM migration process.
Note:- For the workloads (to be migrated) to use in a later step, create a separate service account other than mentioned in the link.
Now just review & create the migration manager
You can see the same after the manager created, note the internal IP of the Migration Manager, we will use this IP address when installing the migration agent in the AWS Windows VMs.
Step 4: Setup AWS environment for Migrate for Compute Engine (Velostrata):
Required Policies: IAM – CreateGroup, PutGroupPolicy, CreateUser, AddUserToGroup
- Create an IAM group named VelosMgrGroup for use by a Migrate for Compute Engine service account. This group enforces an access policy with the minimum privileges required by Migrate for Compute Engine and allows provisioning and monitoring of cloud-side components and worker VMs.
- To set the AWS credentials, we have to create a new stack in the Cloud Formation, providing the JSON file available on the link below. After the stack is created, a velostrata IAM group is generated with all the required permissions.
The following links→ download page → CloudFormation Stack
Note:- Here, We’re using velostrata version 4.11.9 so if you’re using a different version then download the proper stack accordingly from the download page.
After downloading, extracting the file you’ll get the JSON file like this VxCF-GA-V3-IAMONLY.rev1.json
In the AWS Management Console
Search → cloud formation → create stack → with existing resources(import resources)
Click on the upload file & upload your JSON file you extracted in the above step.
Just specify some more details like VPC in further steps & create your stack.
3. Now create an IAM user with programmatic access and add it to the newly created Velostrata Group (VelosMgrGroup).
AWS Console -> Click on Account Name -> Security Credentials -> left menu -> users -> Create New User
Now, add the new user to the Velostrata Group (VelosMgrGroup)
Note: Be sure that you’ve given the Programmatic access to the new user, Also save the Access key & Secret key somewhere, we need both keys in the further process.
Step 5: Installation of the agent in AWS VMs (Which will be migrate)
On the VM that we want to migrate from AWS to GCP, we need to install the prep package according to our OS. In the below example, we have shown different operating systems.
Note:- before starting migration agent/package installation, check the connectivity using private IP
Installation in windows EC2 Instance:
- Connect to your Windows instance using RDP
- Open the browser & download the file from the below link & install it
- Run your Windows PowerShell as an Administrator & commands which are given below
Important: This will prompt you for login so keep your migration manager VM’s internal IP & remember we have created one user called apiuser & their password, so we’re going to use that details here.
This will download some packages, so be patient.
Note:- This is for velostrata version 4.11.9 only, so according to your version, download the file.
Migrate for compute engine downloads section → click PowerShell Module
Installation in Ubuntu EC2 Instance :
Download the migration agent package & install it on the machine.
wget https://storage.googleapis.com/velostrata-release/4.11.9/migrate-for-gce-prep-4.11.9-0.debsudo apt-get updatesudo apt --fix-broken installsudo apt install multipath-tools*sudo dpkg -i migrate-for-gce-prep-4.11.9–0.debsudo apt-get update && sudo apt-get install -f -y
Now, we’re ready for the migration process
Important: To access velostrata console, you’ve to add your PC’s IP to the firewall rule velos-webui.
Search for your IP in the browser → note your IPv4 address & add to velos-webui firewall rule as source IP ranges <Your IP>/32
Step 6: Migration
GCP Console → Navigation menu → migrate for compute engine → click on your migration instance → Instance details → you’ll redirect to compute engine → open your migration manager VM's external IP address in a new tab.
Note:- If you get some error then in the page click on advanced options & proceed to <vm external ip> (unsafe).
Click on Advanced & then Click on Proceed to unsafe
After getting the UI, we get the page to enable logs for the StackDriver, we can enable them.
6.1: Set up the ‘Source Cloud’ section with (AWS) credentials
Here, we need to provide the source credentials from the AWS IAM user credentials generated for velostrata in Step 4 (Setup AWS IAM for Migrate on compute Engine (Velostrata)).
For adding the source cloud credentials,
Here you’ll need to give User ID & Password so user apiuser as username & their password, we’ve created at the time of migration manager VM creation.
Source Cloud → Cloud Credentials → Add the details like region, access and secret key.
After the cloud credentials are created, go to the source cloud then cloud details
Note:- In this section, select the proper subnet that virtual Machines are running on the AWS side
6.2: Set up the ‘Target Cloud’ section:
Click home → Target Cloud
In the Target Cloud, we create the Cloud Extensions.
- A Cloud Extension is a conduit for VM storage between two hosting environments.
- A Cloud Extension uses a dual-node active/passive configuration for high availability; each node serves its own workloads while providing backup for the other.
- A Cloud Extension is a pair of Cloud Edge nodes, known as Node A and Node B.
- When we create a cloud extension, 2 VMs requiring 500 GB SSD volume each are created. So make sure you have increased the project quota to at least 1 TB.
Now, creating the Cloud Extensions requires filling 4 sections as given below:
6.2.1 Network Setting:
a. Select your GCP Project, Region, and VPC from the dropdown.
b. Specify the network tags ‘fw-velostrata’ for the edge nodes and ‘fw-workload’ for workloads to be migrated.
c. Select the GCP project from the dropdown
d. In the ‘Service Account for Workload’ select the service account specially meant for a specific VM service or workload.
f. leave other fields as default.
6.2.2 Cloud Extension:
- Name of the cloud extension of any choice
- Service account created for the edge nodes at the time of migration manager VM creation.
- Select size:
- Small for migrating 15 or fewer VMs simultaneously.
- Large for migrating more than 20 VMs
- Two cloud extension edge nodes will be created in the process of migration. In order to have HA in place, select two different zones of the region.
- Select the same subnet for both zones to reduce complexity.
- The workload subnet is the subnet where the VM will be migrated to. Select accordingly as planned for the same.
Labels are tags that will be attached to all the VMs created after running the migration wave (workload + edge nodes).
Velostrata edge node created in the GCP environment
Once cloud extension is done, enable HTTP and HTTPS firewall for edge nodes
Common Issue with Cloud Extension:
There could be various reasons, but very common is “could not connect to null instance on port 443”
Solution: Check the firewall rules and verify if necessary permission is given to the service account used.
6.3: Setting up Migration waves
Home → migration waves
Migration Waves is a four-step process:
- Generate Runbook csv (containing workload info)
- Modify the CSV file as per need
- Create Wave for Migration using the CSV
- Validate and Start Migration.
- Generate Runbook CSV
- Once the Target and Source cloud is Set, we can go ahead and generate the Runbook CSV, which contains all the details of the workload to be migrated from source to GCP.
- This CSV file will be used to create a ‘wave’ which is used to perform various types of migration jobs.
In the above fields:
- Source: Source from where VM is to be migrated. (E.g. AWS, Azure)
- Source Cloud Details: The credentials which were generated in the source cloud and from that cloud details which have been created.
- Source Tag: The tags given to the source AWS/Azure VMs, multiple tags can be added.
- Target Cloud Extension: The Cloud Extension created which is to be used is mentioned here.
- Target Network: Tick the option (Populate with Cloud Extension Defaults)
After clicking on the “Create” button, a CSV file will be downloaded which contains all details of the AWS VM to be migrated.
The next step will be editing the CSV file as mentioned earlier and creating a ‘wave’
B. Modify CSV file as per need
Few edits are to be performed on the generated CSV and save in your local system.
- The values in the first column in the CSV are negative, we have to suppress the negative sign.
- We have to manually enter the machine type (GCP) in the ‘TargetInstanceType’ column. E.g. n1-standard-1
- Give a name to the instance
Below is the sample runbook file
C. Create Wave for Migration using the CSV
- Now we create ‘wave’ by clicking ‘New Wave’, give it a name and upload the CSV file which was saved after modifications.
Provide the name and upload the generated runbook CSV file here.
D. Validate and Start Migration.
Once the wave is created using the runbook CSV, it has to be validated and only after validation we can perform the jobs of migration.
Once validated successfully, You’ll get passed status. Now, various types of migration can be performed by clicking on the ‘New Job’
In the operation, Choose a Migration method as I’ve chosen full migration for this demo but you can choose according to your need & requirements.
VM Migration Life Cycle:
1. Run-in-cloud –This operation moves the source VMs from AWS to GCP. This method does not completely move VM storage to the GCP. The run-in-cloud operation performs the following actions:
- Shuts down the source VMs in AWS.
- Attaches the source VM’s volumes to temporary exporter VMs created in AWS.
- Starts the VM on GCP, streaming the storage data to GCP disk.
Key Points to Note:
- Takes around 40 minutes to migrate a Windows VM with 40 GB volume.
- Source VM is stopped throughout the migration process.
- Doesn’t give the option to detach the GCP VMs from Velostrata Manager.
2. Full Migration – The full migration operation moves VMs in one step from source to target. In doing so, it
- Performs the run-in-cloud process described above.
- Waits for the VMs to be in the Cache on Demand state when storage is streamed to the GCP disks.
- Migrates the VM data to GCP VMs. After storage is fully copied to GCP VM, prepares the VM to detach from Velostarata manager.
Key Points to Note:
- When this process has been completed, the VM state changes to ‘Ready to Detach’.
- Took more than 1 hour 40 minutes to get the migration in the “Ready to Detach” state.
- Source VM is stopped throughout the migration process.
Once the full migration is done and shows the ready to detach status then go to the action then detach once detach is done go to action then clean up
Importer server in AWS:
AWS moving to GCP cloud
2. Aws instances are migrating to GCP:
AWS migrated Ubuntu Server login in GCP:
[Failed to prepare VM to detach] (Exporter is terminated)
6. Cleanup – After the VMs are detached, and you have completed any required validation, you can start to detach cleanup. Each VM is then marked as unmanaged by Migrate for Compute Engine.
Other Migration Jobs available with Migrate for Compute Engine
7. Test Clone
Note:- Test clones are available only for migrations from on-premises data centres to GCP. Test clones are not available for cloud-to-cloud migrations.
In this article, we had successfully migrated our EC2 instance ( Windows / Linux Server) to Google Cloud Platform compute engine Instance using Google Cloud’s native migration tool Migrate for compute Engine formally Velostrata. I hope this will give a better understanding of the Migrate for Compute Engine migration tool & the process we follow during the migration.