Skip to main content
  1. CloudM
  2. Migrate
  3. Errors & Troubleshooting
  4. Troubleshooting
  5. Performance

Throttling, Rate Limiting, and API Quotas: General Troubleshooting

Updated

Migrating data between platforms is a complex process. Throttling, rate limiting, and API quotas are common and expected challenges that can impact migration speed. This article provides general troubleshooting techniques that apply to all migrations, regardless of the source or destination platform.

Understanding Throttling and Rate Limiting

Although these terms are often used interchangeably, they refer to distinct platform mechanisms, and CloudM is built to handle all of them.

  • Throttling: A platform's reactive measure to slow down a user or application that is making too many requests. It's like a warning that tells you to slow down before you get cut off.
  • Rate Limiting: A proactive, hard limit on the number of requests you can make within a specific, short timeframe (e.g., 100 requests per second). When you hit this limit, the platform will block your requests.
  • API Quotas: A hard, cumulative limit on the total number of requests you can make over a longer period, such as a day. Once this limit is reached, all further requests are blocked until the quota resets.

Immediate Steps (During a Migration)

If your migration speed drops or a migration stops unexpectedly due to throttling, these are the first steps you should take:

Important Note: Before making any configuration changes, you must first pause your migration. Changes will not take effect until the migration is resumed.

  1. Look for Throttling Indicators: Check your CloudM Migrate dashboard for specific visual cues or status messages that indicate throttling is occurring. The platform provides real-time visibility into when a source or destination is slowing down the migration.
    Throttling Indicators
  2. Check the Migration Logs: Your CloudM Migrate dashboard provides real-time status updates. Look for specific error messages or status codes that point to throttling or rate limiting issues.
    • Examples:
      • 429 (Too Many Requests) or 503 (Service Unavailable)
        • A 503 is a more general server error that is not limited to throttling, but in the context of a high-volume migration, it often indicates the server is too busy to process the request and is a form of throttling.
      • "User Rate Limit Exceeded", or "rateLimitExceeded"
      • "Quota exceeded for quota metric..."

  3. Lower the number of concurrent migration entities running at the same time via the CloudM Migrate Service Manager > Maximum User Migrations setting. This is the single most effective way to reduce the load on the source and destination APIs.

    Note: This setting is configurable for CloudM Migrate Self-Hosted. For CloudM Migrate Hosted migrations, you must contact CloudM Support to request a change to your concurrency settings.

    For a more detailed overview of these settings, see the CloudM Migrate Service Manager Overview & Configuration.

  4. Implement an Item Export Delay: Introducing a small delay between item exports can also help manage the rate of API requests. Although this setting is named "Export Delay," it can be used to manage throttling at both the source (export) and destination (import) platforms.
    1. In your CloudM Migrate project, go to Configuration > Advanced Settings > System > Item Export Delay.
    2. Enter a small value (in milliseconds) to create a pause between each item being processed. Start with a low value like 100 or 200.
  5. Reduce Drive Thread Count (for Google Drive errors): If the errors are specific to file migrations, reducing the number of threads used for Drive data can help.
    1. In your CloudM Migrate project, go to Configuration > Advanced Settings > System > Drive Thread Count.
    2. Reduce the value. Lowering it from the default of 3 to 2 can make a substantial difference.

  6. Pause and Resume: Pausing the migration for 5-10 minutes can give the platform's throttling policy time to reset. When you resume, CloudM Migrate's exponential backoff will handle the retries.

    For more severe throttling or daily API quotas, you may need to pause for a longer period. The quota for Google and most cloud platforms typically resets at midnight Pacific Time. Depending on your time zone, this may require waiting until the next day to resume.

  7. Check Platform-Specific Documentation: Some platforms have unique throttling rules or settings. Consult the specific articles for your migration:
  8. Contact CloudM Support: If you have tried these steps and are still experiencing issues, please contact our support team with your migration logs.

Proactive and Advanced Strategies

These strategies can be used to prevent throttling and optimize performance for very large or complex migrations.

  • Hosting Environment & Performance: For any large-scale migration, performance is heavily dependent on the infrastructure used. For migrations to Google Workspace or Microsoft 365, the most effective strategy is to leverage the respective cloud infrastructure. 
    CloudM Migrate Hosted: Services automatically follow this practice by using GCP or Azure virtual machines (VMs) for all migrations.

  • Advanced Strategy: Multiple projects and service accounts: For very large migrations, a single project using a single service account can become a bottleneck due to API limits. A single project can typically handle up to 500 concurrent threads for Google or 200-300 for Microsoft before performance degrades. To overcome this and achieve significantly higher throughput, a multi-project strategy is recommended. This involves distributing the workload across multiple GCP Projects, service accounts and Migrate projects.


    How It Works:

    • Set up multiple migration projects in Migrate, each with its own separate destination connection.
    • For a Google Workspace migration, each destination connection should use a service account on it's own dedicated GCP project.
    • For a Microsoft 365 migration, each destination connection should use its own dedicated Azure application.

    This approach helps ensure that your overall migration performance is not limited by the API quotas of a single project or application. However, this does not entirely negate the risk of rate limiting.

  • Consult CloudM Support: If you're planning a very large or complex migration, contact our support team in advance. We can help you plan the best strategy to minimize the impact of throttling.

Related Information

Was this article helpful?
0 out of 0 found this helpful