An easier way to import lots of email
One of the main challenges in moving from an on-premise Exchange installation to the Microsoft hosted Office 365 service is data migration. The slowest method to import data from existing Exchange mailboxes to Office 365 is to use Outlook to upload the data directly to Office 365.
Microsoft caps the import speed at roughly 500 MB an hour per mailbox from each instance of Outlook which can easily cause problems due to time and logistical constraints – even over a long weekend. It is more efficient to use their new PST Import Tool which involves uploading all the data to a temporary Azure location and then starting a mass import. The procedure for bulk Office 365 imports is described in this article.
Preparation for the Data Import
Before you can begin the data imports, you need to complete several tasks:
- Set up Office 365 with user accounts and passwords
- Copy the PST files from every user that contains all their current Outlook/Exchange data to a centralized file share that is accessible from the import machine
- Assign rights to one of your Office 365 users to allow the import of the PST data
- Assemble a CSV file that maps the user accounts to the PST files that you have exported
Office 365 Setup
This step should be fairly self-explanatory. Just create an Office 365 account, create your users and choose passwords for them.
Assemble the PST Files for Import
This step is a little more complex and depends on your current setup. The first step is to create a share on a machine that has enough space to hold all your existing email. It doesn’t have to be a true server – it can be a workstation. The only requirement is that the PC that will upload the PSTs to the temporary Azure location has access to it. In the example below, we’ll be using \SERVERPSTFiles.
If your current setup is POP or IMAP it’s really easy – you probably already have PST files on each machine that contain all the user data so just copy those to the share.
If you’re currently using Exchange you’ll need to do an export through Outlook. This is easy too but will take a little longer. On each machine you need to export the entire user data to a PST file. This is slightly different in each version of Outlook.
Screenshots for the Outlook 2013 export process are below. The procedure on earlier versions of Outlook will be fairly similar.
Select File, then Open & Export, then Export to a File, then Outlook Data File (.pst):
You then select the very top of the folder hierarchy, make sure Include Subfolders is checked and click Next. Name the PST and pick a place for it to be saved (it can be directly to the PST share created earlier) and click Finish.
Assigning Import Permissions to a User
The next step is to assign mailbox import rights to one of your users. We recommend just using an administrator account.
Open the Office 365 admin center and select Admin and then Exchange on the left side:
You’ll then be in the Exchange admin center. Chose Permissions and make sure Admin Roles is selected. Click the plus symbol:
The New Role Group dialog box will appear. You just need to fill in three values:
- Name – we used “Office 365 PST Import”
- Under Roles, click the plus symbol and add “Mailbox Import Export”
- Under Members, click the plus symbol and add the user that will gain import rights. In the example below the user name is “symmetrixtest.”
The completed form looks like the image below:
Preparing and Uploading the CSV File
The next step is to prepare the user to mailbox mapping file which needs to be in CSV format. You can do this in either Excel or Notepad.
There are a lot of variables for particularly customized setups but the options below will be fine for most installations. The important fields are:
Workload – Just set this to ‘Exchange’
FilePath – set this to the file share location of your PSTs. Note that the syntax is different than a standard Windows share so “\SERVERPSTSource” is rewritten as “SERVER/PSTSource” with a forward slash.
Name – The file name of the PST.
Mailbox – the primary email address of the destination mailbox in Office 365
IsArchive – This option allows you to upload into the user’s archive mailbox. You will probably use “FALSE” in most cases.
TargetRootFolder – The location where the PST data will be uploaded. If you just use a forward slash, the import will merge everything into the root of the mailbox. You could optionally put everything into a subfolder using something like “/Import” and let the user move the data according to their preference. It’s also useful if you have multiple PSTs per user – you can add an additional line in the CSV for a user and upload the other PST into a subfolder.
SPFileContainer – leave blank
SPManifestContainer – leave blank
SPSiteUrl – leave blank
If you create the file in Excel it should look similar to this:
And here is the same file in Notepad:
Azure Upload Preparation
Now that you’ve prepared all the data and the import account, you’re ready to get the data and mapping file uploaded to Azure.
Log in to the administrative console in Office 365 and click on the Admin icon. Then select Import, click on the Plus symbol and select Upload Files over Network.
A window will then pop up. There are several things to do here:
- Download the Azure AzCopy tool. Click on the tool’s download link and save it to your hard drive. As of this writing, the link is http://aka.ms/downloadazcopy. The filename is “MicrosoftAzureStorageTools.msi”. Once it is downloaded, open it and install the software.
- Retrieve the two pieces of information that are specific to your account – the secure storage account key and the secure network upload URL. Click on both Show Key and Show URL and make a copy of them in a temporary Notepad file. Make very sure you capture the entire length of the keys – depending on your browser some of the trailing characters can be hidden but you can copy the whole thing by using your mouse. You can see the difference between what’s displayed on the page and what the entire keys contain below.
To get the key and URL, click on Import and Upload files over the Network and then copy the keys:
The browser screenshot is below:
The Notepad screenshot is below (note the difference in the string lengths):
Uploading the PSTs to Azure
Now that you have the PSTs ready in a share on the server, it’s time to upload the data to the temporary Azure location.
Open the Microsoft Azure Storage Command Line Tool by selecting it from your Start Menu:
You should get a PowerShell box like this:
The next command is where it gets EXTREMELY particular about your syntax. Note that this is all in one line in the previous box and it’s illustrated further below in another screenshot.
AzCopy.exe /Source:\SERVERPSTSource /Dest:https://f6b5a2xxxxxxxxxxxxxxx72.blob.core.windows.net/ingestiondata/SERVER/PSTSource/ /Destkey:uVv0dj9saqAv5mTNzZaO7M7CqcuMu89e1LO49DALxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxpW1tJJMGBw== /S /V:C:Uploadlog.log
Pay attention to the forward and backslashes in the paths – essentially Microsoft is using both Windows and UNIX path conventions.
You won’t have to authenticate with an account because you’ve already done that by having the long URL and key in the Azure copy command. Azcopy.exe will upload any files you have in the source directory so be careful about sensitive files.
Performing the Import into Office 365
Now that the PSTs uploaded to Azure and the CSV file is prepared we can actually begin populating the mailboxes with data.
Select the Import option in the Office 365 admin center, click on the plus symbol and select Upload Files over the Network:
Select both check boxes in the pop-up window and click Next.
On the next screen, enter a job name following the specified conventions and click Next. The actual name doesn’t matter.
On the next screen, click the Plus symbol, select your CSV mapping file that you created earlier, click the check box and click Finish:
Click Close on the next screen:
On the Import screen you can see the job you just created:
Click on the blue View Details link on the right to see the progress:
The status does not update often so be patient. You can click on the refresh symbol next to the magnifying glass to get the current progress. We have seen the percentages increase, decrease and then increase again so it’s not entirely accurate. The status seems to update about every five to ten minutes.
While the imports are running, start converting your Outlook configurations on each machine to point to Office 365. Because Outlook keeps a local cache of all the mail data it will have to sync all of the information from Office 365 to a local OST file. This may take a while depending on your bandwidth but if a user clicks on a particular folder in Outlook it will prioritize that folder and the user will see the data pretty quickly.
As the imports complete the data will populate into Outlook. If you have a massive data set and bandwidth restrictions you can show the user how to get into the webmail interface (OWA) and they’ll be able to see everything as soon as it’s imported.
We have done repeated import speed tests over the course of a week and have come up with the following results which are accurate as of this writing.
50 Mbit up/down Internet connection
10 test mail boxes
10 identical PST files with 85,796 items and 2.22 GB of data in each PST and a total of 857,960 items and 22.2 GB of data
A mix of mail, calendar items, contacts and tasks
We were able to upload PST files to Azure at an average rate of 7 MB/sec on a 50 Mbit up/down Internet connection. You could probably increase this to some extent by splitting up the PSTs to multiple computers and having them each upload using the same Azure URL and key from separate Internet connections although we have not tested that method.
22.2 GB in 10 mailboxes took 53 minutes to upload.
All ten mailboxes were imported simultaneously but exhibited wildly varying import rates. The completion percentages on all of them were increasing over the import duration but some finished much more quickly than others. The individual rates seem to be independent of the others – the rate of the ones that completed later did not increase as the earlier ones finished.
One mailbox finished in two hours
Two additional mailboxes finished in three hours
Three additional mailboxes finished in four hours
The remaining four mailboxes finished in five hours
Microsoft’s bulk import tool makes data migration to Office 365 much simpler if you have a lot of mailboxes and PSTs. You are still subject to your bandwidth limitations at your business but by staging the data to Azure first and then doing a multithreaded import you can greatly speed up and automate the process.