Google Vault to Google Cloud Storage Migration

Permanently deleted user

Updated September 01, 2023 16:10

This quick start guide provides you with the basic information to configure CloudM Migrate for a migration from Google Vault to Google Cloud Storage. It is highly recommended that you read the documentation in full for your platforms in order to understand all of the options available to you during a migration.

Any original folder structure from Vaulted files will be lost when migrating. Google exports all files into a zip file with their original folder structure stripped. As a result, you can migrate what is provided by Google and simply drop exported files into the destination root by default, or a top level folder, if specified in the general config setting 'Top Level Folder'.

Vault file migrations can only be done to 'User' import object types. Shared drive or SharePoint site imports are not currently allowed.

System Requirements

Environment

64 bit Operating system: Windows Server 2016+ (Clean build recommended)
Microsoft .NET Framework 4.8
Recommended system specification - Primary Server:

3GHz 8 Core Processor or better
200+GB Disk space
16+GB Memory

Recommended system specification - Secondary Server(s):

Server names longer than 15 characters will cause communication issues between servers in a multi-server environment.
3GHz 4 Core Processor or better
100GB Disk space
8+GB Memory
Server name shorter than 16 characters.

If you are looking to complete a 'large migration' (a migration of more than 25,000 users or 10 million objects), refer to the following articles for additional considerations:

If you are storing Drive document mappings and running a large migration, contact the support team.

Required ports

Both primary and secondary CloudM Migrate servers communicate with source and destination platforms, and the CloudM Migrate licensing platform using HTTPS.

Typically, it is ports 80 and 443, but that can depend on the source platform and local network configuration.

There is also the following to consider:

You may need to whitelist our license server: portal.thecloudmigrator.com
SQL server runs on TCP 1433 and UDP 1434
Redis runs on port 6379
Microsoft Message Queue on port 1801
All of Google's API endpoints are listed here: https://support.google.com/a/answer/60764

Basic installation for single-server migrations

If you are completing a multi-server migration, skip to the Multi-Server Installation section of this guide.

Installation

Basic installation installs all components to a single machine and is suitable for performing migrations from a single server or as the main server in a multi-server migration.

If you have specialised requirements, have a pre-existing SQL Server instance you would like to use, or will be performing multi-server migrations using a server farm, you should also review the advanced installation documentation.

The following components makeup and are used by CloudM Migrate:

Web Application and Primary Service
Secondary Service
SQL Server Express 2017
Redis

To start the installation:

Open a web browser, download the latest x64 CloudM Migrate executable and launch the installer as an administrator: CloudM Migrate Changelog and Download Link
Review and accept the license terms by checking the checkbox.
Click 'Next'.
All relevant fields will be selected / populated by default (with the exception of your web access credentials). To select an alternative installation directory:

Click the ellipsis button (...) to the right of the 'Installation Path' field and browse to the required install destination.

Set up your web access credentials with an email username and a password of your choice (which you will need to use to log in to the web application).

Click Install to proceed. This installation process can take some time.

Start CloudM Migrate by loading the web interface. Alternatively, the interfaces are available in the Start menu, under CloudM.

If you have any issues regarding the installation, refer to the next section of this article.
If the installation was successful, skip to Launching CloudM Migrate

Installation problems

If you have any issues when installing CloudM Migrate:

Ensure that you have no pending Windows updates waiting to install. If you have Windows updates waiting to install, please install them and reboot.

Check the following in the System Requirements:

You must be running a 64bit version of Windows.
You must be running Windows 8, 10, Windows Server 2016 or Windows Server 2016 R2.

Try clearing your computer's temporary directory. This can be done using the disk cleanup application installed with Windows or by navigating in a Windows explorer window to %TEMP% and deleting the files from this directory.

Check if any security or antivirus software may be preventing installation. CloudM Migrate needs to install custom firewall rules and make other system changes that some security packages may prevent. It is recommended to uninstall any non-Windows security software.
Verify the checksum on your CloudM Migrate installation package to be sure the download is not corrupt. The latest checksums can be found on the CloudM Migrate Changelog page and files can be verified using CertUtil. MD5 and SHA1 checksums are provided.

If the installer fails on the 'Prerequisites' package, and for any reason you have uninstalled SQL Server Express before an upgrade or reinstallation, make sure to delete the database files. These are in the following location by default:

C:\Program Files\Microsoft SQL Server\MSSQL12.POWEREDBY\MSSQL\DATA\PoweredBy.mdf
C:\Program Files\Microsoft SQL Server\MSSQL12.POWEREDBY\MSSQL\DATA\PoweredBy_log.ldf

Log Files

If you encounter any problems during the installation of CloudM Migrate, you will be presented with a message box outlining the error encountered, or you will be shown the failed installation dialog.

Clicking on the Log link in the above screen will open a zip file. The number of files in this zip file can vary depending on the installation mode (e.g Install, Update or Uninstall) components being installed and the error encountered.

If you have a problem installing CloudM Migrate and the tips above do not help, start by reviewing the logs. If you are still unable to diagnose or solve the issue, please report it to our support team with a copy of the installer logs.

Clicking Close will close the installer.

Installation zip files can be also found in the following directory and prefixed with 'CloudM Migrate' or "CloudMigrator" followed by the date and time:

%USERPROFILE%\AppData\Local\Temp

Multi-Server Installation

Virtual Machines are not mandatory and installations can be performed locally.

If you are not setting up Virtual Machines, and:

Using Windows Server, skip to Step 1 in the Installing CloudM Migrate section.
Not using Windows Server, skip to Step 2 in the Installing CloudM Migrate section.

Installing and configuring Virtual Machines

The following instructions explain how to set up servers using Virtual Machines set up in Google Cloud Platform (GCP). For more information, see here.

There are several other methods that you can use to set up your servers, including using Microsoft Azure (Windows Azure).

CloudM recommends setting up Virtual Machines using Google Cloud Platform when migrating to Google (and Microsoft Azure when migrating to Microsoft) to limit network traffic costs.

All CloudM Migrate secondary servers must be on the same VM network as the primary server. The default network configuration for new VM instances in GCP does not need to be modified to meet this requirement.

You will also need a GCP billing account and associated GCP project for creating VM instances and to be mindful of the associated costs.

The instructions differ at various points throughout this guide depending on the type of server being created and configured (primary or secondary).

Creating a VM instance

Access the Google Cloud Platform console
Open the Google Cloud Platform project to be used for the CloudM Migrate server/s, using the dropdown icon at the top of the screen.
Navigate to Compute Engine > VM instances

Select CREATE INSTANCE

Enter a name for the VM instance.

Keep the name to less than 15 characters to avoid connection issues in multi-server environments. See Resource naming conventions.
Standardize the naming convention for all instances, using numbering (1,2,3) or lettering (a,b,c) to help you differentiate between instances.

Select the ‘Machine Type’ dropdown menu and then, select ‘Custom’
- In larger migrations, where numerous secondary servers will be utilized, you should change the Virtual Memory of the Primary VM from the default Custom, which has a low threshold, to 'System Managed'. This is to improve performance of the Primary Server as it communicates with the REDIS database.
- Alternatively, select e2-standard-4 (4 vCPUs, 16 GB memory) as the Machine Type for a Primary Server and e2-standard-2 (2 vCPUs, 8 GB memory) for any Secondary Servers. These settings should be able to accommodate most migrations without spiralling cost.
Change the number of cores and memory to meet the CloudM Migrate System Requirements (note the different requirements for primary and secondary servers.
Select ‘Change’ in the ‘Boot disk’ section:

On the Public Images tab, change the:

‘Operating system’ to ‘Windows Server’.
‘Version’ to ‘Windows Server 2019 Datacenter’ (or another supported version of your choice).
‘Size (GB)’ to ‘300’ (the combined minimum requirements for both CloudM Migrate and Windows Server).

Click Select at the bottom of the page

If you are planning on configuring a public facing URL, in the ‘Firewall’ section, check the box to ‘Allow HTTPS traffic’. This allows easy access to download the installation files required, but is optional.

Select Create to create the VM instance.

Connecting to a VM instance

Go to the VM instances screen. Under the Connect column, select the small down arrow and then Set Windows password:

Leave the Username field as the default, and then select SET:

Copy the password and make a note of it for later:

Under the Connect column, select the small down arrow and then Download the RDP file:

Using the RDP (Remote Desktop Protocol) file, establish a connection and login to the VM instance using the password noted down earlier:

If it fails to connect, disable your local environment Windows firewall and any other network firewalls.

Installing CloudM Migrate

If using Windows Server, open Server Manager and:

Select Local Server (1).
Select On next to IE Enhanced Security Configuration (2).
Set the radio button to Off for Administrators (3).
Select OK (4).

Open a web browser, download the latest x64 CloudM Migrate executable and launch the installer as an administrator: CloudM Migrate Changelog and Download Link
Agree to the terms and conditions, select Next:

For primary server installations: Leave the default installation options and enter the email address and password that will be used to log in to the CloudM Migrate web interface, and then select Install:

Normally, you should use the default SQL Server Express Instance that is included with CloudM Migrate when installing a primary service. Using an existing SQL Server Instance should only be considered in large migrations, where you may expect to exceed the default SQL Server Express database size limit of 10GB.

Once the installation completes, select Export Details to take note of the database details of the primary server (which will be needed when creating a secondary CloudM Migrate server), and then select Close:

For secondary server installations: Install only the ‘Secondary Migration Service’ and enter any email address and password that will be used to log into the CloudM Migrate instance. Select “Use Existing SQL Server” and enter the Data Configuration details from the Primary Server, then select ‘Install’:

Launching CloudM Migrate

CloudM Migrate can be configured using the local web interface on the primary server. This can be found in the Windows start menu > CloudM > CloudM Migrate Web, or by navigating to https://cloudm.local/ in your web browser.
Sign-in using the web interface credentials entered during installation:

Platform Prerequisites

General and Source Platform prerequisites

Google Vault is connected to your Google Workspace account and so please perform the same steps as described for Google Workspace.
You will need at least 30GB free disk space on each migration server.
Additional Google APIs must be enabled using API Manager and Scopes added using the Google Admin Console

Google Vault API, Scopes (https://www.googleapis.com/auth/ediscovery,https://www.googleapis.com/auth/ediscovery.readonly)
Google Storage API, Scope (https://www.googleapis.com/auth/devstorage.read_write)

Billing must be enabled for the Google project being used for the migration. This is necessary to avoid very low Google Vault export quota limits. Enabling it allows for higher export quotas, however, with a possible resulting cost. This cost is something your Google account manager should be able to discuss with you.
Exporting from Google Vault is subject to quota limitations. If you find user migrations are failing with quota errors you should contact Google to have the limits increased.

Setting up the Service Account and enable the APIs within Google Workspace for CloudM Migrate

Please refer to the Setting up the Service Account and enable the APIs within Google Workspace for CloudM Migrate article.

Destination Platform prerequisites

You will need to:

Create a bucket - Open the Google Cloud Platform, navigate to Storage > Browser and create a bucket for the items to be migrated to. Make a note of the bucket name you'd like to migrate data into.
Back in the storage browser, click on the three dots on the right of the bucket you made and select Edit bucket permissions,
Click Add member and in the New members field, paste the service account email address that you made earlier,
Click Select a role and add Storage Object admin permissions.
Set encryption and decryption permissions to the KMS key - Permissions need to be given to the Cloud Storage Service account. To do this:

Open the Google Cloud Platform,
Navigate to Storage
Select Settings
Copy Service Account under Cloud Storage Service Account
Navigate to Security > Cryptographic Keys
Check the key ring you are going to use
Click add member under the permissions tab on the right side
Paste the service account email in the new members text box.
Select Cloud KMS -> Cloud KMS CryptoKey Encrypter/Decrypter role and save.

For more information on KMS keys, please refer to the Using customer-managed encryption keys article on the Google Cloud help site.

Start a new configuration project

Login to CloudM Migrate,
Select Projects in the Functions column on the left side of the screen,
Select the Add button (+ icon) to prompt the section to expand,

Select the Create a new project option,
In the pop-up box, select Google Cloud Storage as the Destination platform (the platform that you want to migrate all of your users and data to) and enter the Domain Name (the destination domain you are migrating to), a Configuration Name of your choice and License Key.

Select Check License to ensure that the license entered is valid for the destination platform that you want to migrate to.
Select Continue.
The project will now be displayed in the domain drop down menu at the top of the screen. Make sure that the project is selected.

Create a new configuration

To create a new configuration:

Select the Create a new configuration to get started option.
In the pop-up box, select Google Cloud Storage as the Destination platform (the platform that you want to migrate all of your users and data to) and enter a Configuration Name.

Create new project config basic.PNG

Select Continue.
The configuration will now be displayed in the Configuration Overview.

Once the configuration is created, it will be displayed in the Configuration Overview table. Select Configure, enter a CloudM license key and select Apply to start the configuration process.

Configure Source Platform settings

Google Vault

Choose Google Vault as the migration source and enter your Google Workspace settings into CloudM Migrate and then click Next.

Domain Name - The domain name you will be migrating from. This should be the Internet domain name, not the local domain name.
Admin Username - An administrator account for the domain specified, this will usually be an email address for a Super Admin.
Authentication Method - Set whether to use a P12 key or a JSON key as the authentication method.
Service Account Email Address - Before attempting to configure CloudM Migrate, you should have created a Google Cloud platform project and created a service account for it. If you have selected to use a P12 key, you will need to input the service account's email address in this field.
Private Key - The file path to the P12 or JSON key that was generated and downloaded when creating the OAuth service account.

If you cannot find the private key, go back to Google Cloud Platform service accounts, select your project (if not already selected), use the option button on the right of the service account and click "Create Key". Select P12 or JSON and download the key file.

Select Next to test the connection to the platform.

Configure Destination Platform settings

Google Cloud Storage

Account Details

Domain Name – The name of the Google domain to migrate to. This may be either a primary or secondary domain. Note: you can only migrate users to one domain at a time. If you have both primary and secondary domain users, they must be processed in separate migrations.
Authentication Method - Set whether to use a P12 key or a JSON key as the authentication method.
Service Account Email Address - Before attempting to configure CloudM Migrate, you should have created a Google Cloud platform project and created a service account for it. If you have selected to use a P12 key, you will need to input the service account's email address in this field.
Private Key - The file path to the P12 or JSON key that was generated and downloaded when creating the OAuth service account.

Google Cloud Storage Details

Bucket Name - The name of the bucket that has been created in the Google Cloud Platform, under Storage > Browser.

Google Cloud Storage (Advanced Settings)

Max File Size - The maximum size of the files to be uploaded (in bytes).
Compress Objects - Compress the objects before they are uploaded. Setting to True will use less cloud storage space at the expense of slowing down the import.

Encryption (Advanced Settings)

Please note that only one encryption method is required - either KMS or Encryption Key File.

KMS Key Full Name - The full name of the Google Cloud key management service key. The service account must have the Encrypter / Decrypter role on the key. Please note that the service account referred to here is Cloud Storage service account and not the service account for migration.
Encryption Key File Path - The location of your encryption key file. The key file is a text file containing a 32 bytes base64 string.

For more information on KMS keys, please refer to the Using customer-managed encryption keys article on the Google Cloud help site.

Select which users to migrate

It's now time to add which users you'd like to migrate.

When migrating, you may be able to Get Users from the actions menu. If this option is unavailable, you can manually import users via a CSV file or simply add them individually via the plus icon.

Selecting a star next to any specific user or users will prioritize their migration. When a migration starts, threads will be assigned to any starred user so that their migration can start immediately.

At this point you can choose what to migrate for each user, you can migrate mail and drive.

Enter your user's full email address within the Export Name field. If you have already created your users, then you will just need to enter their username.

Select how much mail to migrate

CloudM Migrate lets you decide how much mail you'd like to migrate to your shiny new system.

If you are changing your email address as part of the migration you can verify that the domain names are correct here. You can also specify specific address replacements in the respective section of the advanced settings.

For more information on domain and address replacements, see this page.

Start your migration

We know that you may want to start your migration in the middle of the night, or over the weekend, but we don't expect you to stay up in order to do so. With CloudM Migrate, you can decide to schedule exactly when you'd like the migration to occur.

If you want to start the migration straight away, select Start.

Migration Readiness Test

The Migration Readiness Test provides a last check before starting a migration (regardless of the source or destination platform - with the exception of PST Archives, Google Cloud Storage and Microsoft Azure Storage). The Drive Readiness Test is now included as part of the Migration Readiness Test.

The test will be optional but is highly recommended, especially in cases where the user has made numerous changes to their configuration. You can only run one Migration Readiness Test at a time.

The test includes:

Checking where changes have been made to general and advanced migration settings (differing from the default values) and suggest any potential issues that these settings could cause,
Scanning the Source and Destination platform to ensure that the connections work properly,
Flagging up any additional limitations, allowing users to resolve any issues before their migration begins.

To run the Migration Readiness Test:

After conducting an Environment Scan, select Next,
On the Summary screen, select Start (or click on the Migration Readiness Tests field),

A Migration Readiness Test pop-up screen will be displayed, as shown below,

You can choose to skip the Migration Readiness Test (by placing a tick in the checkbox), but we highly recommend running the test,
Select Start Readiness Test,
The results will start to propagate as settings are checked,

If the setting is properly configured, the check results will be highlighted in Green,
If the setting is different to the default value, it will be highlighted in Blue with a description of the setting. These results reflect changes that you have made, but should still be checked here to make sure that they are still correct for your migration,
If the setting has failed (usually when attempting to connect to a source or destination platform), the check result will be highlighted in Red. These settings need to be rectified or your migration will fail.

Select Close to close the pop-up screen.
Once all errors are corrected, you can start your migration.

To export the report (for example, if prompted by our Support Team):

Once the test has been completed, select the Export option,
The report will be exported in CSV format and will save in your computer's Downloads folder.
Attach the CSV file to your Support ticket.

Review your migration results

During the migration process, CloudM Migrate will report back in real time exactly who is being migrated and the items being processed. All you now need to do is sit back, relax and wait for your migration to complete.

Check the progress of your migration.

Once complete you can download a full report for your migration.