This article explains what determines how long a link remediation pass takes, how to estimate it for your own environment, and what you can do to speed it up. For what the feature does, see Link remediation. For running it, see Set up and run link remediation.
Link remediation is a separate pass that runs after your documents have migrated. It does not slow down the document migration itself, and it can run while other users are still migrating documents.
What decides the run time
Four things, in roughly this order of importance.
How many documents each user owns, not how many links they contain
Every Google Doc, Sheet and Slides file a user owns has to be opened and scanned to find out whether it contains links that need changing. There is no way to tell from the file list. A user with 2,000 documents and ten links to fix takes about as long as a user with 2,000 documents and a thousand links to fix. When you estimate, count documents.
For a Microsoft 365 source, the same applies to the Word, Excel and PowerPoint files that were migrated. Other file types are not opened.
The mix of file types
Different file types cost different amounts of work.
| File type | What happens | Relative cost |
|---|---|---|
| Google Doc | Read once. If links are found, they are rewritten in one or two batched updates. | Low |
| Google Slides | Read once. Same rewrite pattern as Docs, including layouts and speaker notes. | Low |
| Google Sheet | Read once for the list of tabs, then once more for every tab. A ten-tab workbook costs eleven reads before any rewriting. | Medium to high, rising with tab count |
| Word, Excel or PowerPoint file kept in its original format | Downloaded, rewritten and uploaded again, then checked against Google's hash of the uploaded file. | Proportional to file size |
Sheets are the type most likely to set the pace of a large run, both because of the per-tab reads and because Google's Sheets API has the tightest limits (see below).
How much runs in parallel
CloudM Migrate processes several documents at once for each user, and several users at once on each server.
- Documents per user. Controlled by the Documents Thread Count setting. The default is 3. On a shared drive, each organiser adds another set of threads, which spreads the load across their API allowances.
- Users per server. Controlled by Maximum Users Per Secondary Service. The default is 20. Adding secondary servers adds capacity in the usual way.
Raising the document thread count helps up to the point where Google's per-user limits take over, which for Sheets is quickly.
Google's API limits
Google caps how many requests each API accepts per minute. Two sets of limits apply at the same time: one for each user, and one for the whole Google Cloud project your service account lives in. The per-user limits are separate for every user being migrated, so more users in parallel means more throughput. The project limits are shared across the entire migration, no matter how many servers you run.
| API | Reads per minute, per user | Reads per minute, whole project | Writes per minute, per user | Writes per minute, whole project |
|---|---|---|---|---|
| Google Docs | 300 | 3,000 | 60 | 600 |
| Google Slides | 600 | 3,000 | 60 | 600 |
| Google Sheets | 60 | 300 | 60 | 300 |
These are Google's default quotas at the time of writing. When a limit is reached, CloudM Migrate waits and retries automatically, so the run slows down rather than failing. If your estate is large or Sheets-heavy, ask Google to raise the Sheets and Docs API quotas on your project before you start. This is done from the Quotas page of the Google Cloud console and is normally approved quickly.
Estimating your run
There is no single figure that fits every environment, so treat the following as a way to get to your own estimate rather than a promise.
- Count documents. Use the migration report from your document migration, or an environment scan, to find the number of Docs, Sheets and Slides files (or Office files) per user, and the total.
- Check the Sheets share. If a large proportion of the estate is Sheets, the whole-project Sheets read limit is the number to plan around: at the default quota, roughly 300 tab reads a minute across the entire migration.
- Run a pilot. Run link remediation on a batch of a few representative users first. The migration report shows how long each user took. Scale from that, allowing for the number of users you will run in parallel.
Worked examples
The three examples below show how the factors interact. They are estimates worked through from the number of API calls the product makes per document, assuming half a second per API call, 5 MB per second for file transfers, default settings and default Google quotas. They are here to show you what limits a run and what to change, not as a promise of run time. Your pilot batch is the real measure.
Example 1: a small Google Workspace estate
| Input | Value |
|---|---|
| Users | 50 |
| Documents per user | 400 Docs, 100 Sheets with 3 tabs each, 50 Slides |
| Share of documents with links to fix | 1 in 5 |
| Servers and settings | 1 server, 3 document threads, 20 users at a time |
Each user's 550 documents take around 8 minutes on their own. Running 20 users at a time, the 50 users would finish in about 25 minutes if nothing else got in the way. Something does: 50 users' worth of Sheets is 20,000 Sheets API reads, and the whole project is allowed 300 a minute, so the run stretches to about an hour. Even here, on a small estate, the Sheets project quota is what sets the finish time. Adding a second server would not change it. Raising the Sheets read quota to 1,500 a minute would bring it back to roughly 25 minutes.
Example 2: a large, Sheets-heavy estate
| Input | Value |
|---|---|
| Users | 2,000 |
| Documents per user | 300 Docs, 300 Sheets with 6 tabs each, 50 Slides |
| Share of documents with links to fix | About 1 in 7 |
| Servers and settings | 3 servers, 3 document threads, 20 users at a time on each |
Each user takes around 35 minutes, and it is the per-user Sheets read limit of 60 a minute that sets that, not the servers. Across 2,000 users the estate holds 1.3 million documents and needs about 4.2 million Sheets API reads. At the project default of 300 a minute that is close to ten days of continuous running, and it is the same ten days with one server or three, because the project quota is shared. With the Sheets read quota raised five times, to 1,500 a minute, the same run comes down to about two days. For an estate like this, request the quota increase before you start and plan the pass over a week rather than a weekend.
Example 3: Microsoft 365 to Google, Office files kept in their original format
| Input | Value |
|---|---|
| Users | 500 |
| Documents per user | 600 Word, Excel and PowerPoint files averaging 2 MB, kept as Office files |
| Share of documents with links to fix | 1 in 10 |
| Servers and settings | 1 server, then 2 servers, 3 document threads, 20 users at a time on each |
Every file is downloaded, checked and, where links were found, rewritten and uploaded again, so time follows file size. Each user takes around 9 minutes. With one server the 500 users run in 25 batches of 20, about 4 hours. Adding a second server halves that to about 2 hours, because no Google quota is in play on this path. This is the one case where adding servers scales the run in a straight line. If the same files had been converted to Google format during the document migration, they would instead be rewritten in place through the Docs, Sheets and Slides APIs and the Sheets quota would apply.
What the examples show
- The Sheets API project quota is the limit on most Google-destination runs of any size. Ask for the increase before a large run.
- Adding servers helps when the limit is your own processing capacity, which is the case for Office files kept in their original format, and does not help once a project quota is the limit.
- Per-user times are short. Long run times come from the number of users and the shared quota, not from any single user being slow.
Re-running link remediation
A second pass over the same users costs about the same as the first. Every document is opened and scanned again, because the pass has to check for links that were not resolvable the first time, for example because the linked file had not yet migrated. Plan re-runs with the same allowance as the original run.
Speeding it up
- Run users in parallel. Per-user API limits are separate for each user, so twenty users at once complete far sooner than twenty users one after another. Use the default of 20 users per server unless you have a reason not to.
- Add secondary servers for very large estates, in the same way as for a document migration. This helps until you reach the whole-project Google quotas.
- Request higher Google quotas for the Sheets and Docs APIs on your project before a large run. This is the only lever that moves the whole-project ceiling.
- Convert Office files to Google format during the document migration if that suits your project. Native Docs, Sheets and Slides are rewritten through the API in place; Office files kept as Office files have to be downloaded and uploaded.
- Do not raise Documents Thread Count far above the default for Sheets-heavy users. The per-user Sheets limit of 60 reads a minute is reached quickly, and extra threads then spend their time waiting to retry.