When running large migrations using CloudM Migrate Self Hosted, some extra considerations should be made with regard to server infrastructure in addition to the standard system requirements. For these purposes, a 'large migration' is considered to be a migration of over 25,000 users or 10 million objects.
CloudM Migrate uses a SQL Express database to store data by default, and while this is more than suitable for most migrations, if running a migration over this size, it is recommended a seperate SQL Server instance is used to prevent the SQL Express max database size of 10GB being reached, causing migrations to fail.
Large migrations should utilise multiple secondary servers to ensure optimal performance. The more secondary servers used, the higher the overall migration throughput. If you are running more than 30 secondary servers, you should also follow these recommendations:
- Use an independent Redis instance
- Do not run the secondary migration service on the primary server. Processing migrations on the primary server will take up resource required to run the UI and services smoothly
- Ensure the system requirements are met or exceeded for all servers
- Have at least 500GB available disk space on the primary server
If these recommendations are not followed, you may experience performance issues or server errors caused by high memory usage on the migration instances.
The statistics processor can require lots of memory when processing high numbers of users or items. Increasing the primary instance’s memory and using an independent Redis instance helps to reduce this. Increasing the primary instance disk can also allow you to use more virtual memory in addition to any physical memory assigned to the instance, and may be required to store migration history which is synced from Redis to SQLite.