Data Migration from CRM 4.0 to MS Dynamics CRM 2011 online

This is my second post for data migration from CRM 4.0 to CRM 2011. I think i mentioned in the last webcast that was shared as part of my post that we use our integration framework at the backend of our migration utility which allows us to take data from disparate solutions and feed it to MS Dynamics CRM 2011 on-premises and online. Although this webcast and the last one focused on migration from CRM 4.0 to CRM 2011 but the migration utility can be used for Goldmine and SalesForce as well. We are targeting CRM 4.0 to CRM 2011 just based on the demand and I will be preparing the next webcast for Goldmine. Anyways in this webcast we take data from CRM 4.0 to CRM 2011 online.

I think this webcast addresses the core issue for many who were just stuck with CRM 4.0 because they wanted to migrate to CRM 2011 online and the migration was not obvious. I would like to mention here that no matter what you have on your CRM 4.0 on-premises version you should be able to transition to Cloud provided you are working with intelligent consultants. There is always a work-around for some restrictions with the online solutions and Solution Experts should be able to share the solution with all the pros and cons.

The migration utility works perfectly for basic sales entities. We are still looking into reports to some how find a way to automate report migration from CRM 4.0 to CRM 2011 online but as it is reports require lots of manual work as one needs to change each report.

Enjoy this video and contact me if you have any questions. Hang in there for more videos on data migration from different solutions to CRM 2011 online.

MS CRM Dynamics Online and claim based authentication

MS CRM 4.0 implementation often involves customization. This customization might be custom web application that open up inside CRM or some content from CRM that opens up in the web applications. If the customizations reside on the same machine as CRM then authentication is not an issue. However at times its is imperative that these custom applications reside on separate servers. This requirement would increase with custom applications pulling data from CRM moving to the cloud. There would be an increased requirement for  single sign on or claim based authentication using windows identity foundation. Moreover incase of custom application the security considerations of end users can only be satisfied with a robust security provider in place.

The purpose of this article and ones to follow in this series is to configure STS between the IP and RP. In this post I will configure a web application on windows azure and then will use Microsoft Federation Gateway as the IP by establishing STS between the application and STS service. This would enable the authentication of our azure application to be handled by third party identity provider and would also enable using windows live Id to log into our application. As the token received would be based on the live Id it can be used for claim based authentication. Unfortunately CRM Dynamics does not support claim based authentication so we will be using the SDK class ‘Wlidticket’ for the generation of ticket to be used for accessing CRM service. The only problem is that for the generation of this ticket we will have to use the LiveId and password. Ideally we should have used the claims for subsequent calls to CRM service and this is what we would do with CRM 2011 in the next article. This is huge because it would mean that external application would have a windows Live login and they will be able to pull up CRM data without having to store user credentials. Many of the applications I have seen today are rejected by consumers because they either require multiple logins or they store user credentials. However that is for the next article. In this blog we will have to live with what is available in CRM Dynamics online. Before we start with the actual scenario I just want to highlight that we are using the federation gateway so that we could use windows live authentication service and this requires WIF. If you are not hosting application on windows Azure then you can also use RPS (Relying Party Suite). Anyways so lets get to setting up our windows azure application and configuring STS. Usually service account is used for accessing the CRM service so setup the live id for the service account in CRM online. In-order to enable impersonation on CRM online assign the proxy role to the service account.

Here is the summary of steps we need to do:

1. Register the azure service DNS on Microsoft federation service, msm.live.com.

2. Configure azure application to accept the windows live security token. This is done using WIF.

3. Configure CRM online for the service account and enable proxy account.

4. Configure azure to use ‘Wlidticket’ class from CRM 4.0 SDK and generate ticket to be used for accessing CRM online service.

 

1. Register the azure service DNS on Microsoft federation service:

a. Use the service account linked with live id to access http://msm.live.com

b. Keep in mind that here you would start with test platform(INT) however in-order to work with CRM online you will have to use production platform.

c. Register a new site on msm.live.com.

 

image

d. DNS would be the unique identifier for your website. I am using tayyab.cloudapp.net as DNS in this example. DNS name has to be a unique URI. Use the address for your cloud application if it is already available. This can be modified later..

e. On submission you will se the confirmation page. Confirm and go to ‘Modify editable site properties’.

f. Modify the site properties. I have changed the DNS to a URN. Later we will use this in our cloud application to establish STS. Note the return Url is using https.This is because we will be using the MBI_FED_SSL authentication policy.

image

g. Domain name is ‘Cloudapp.net’ as we plan to deploy our application on Azure Cloud.

h. Set Override Authentication Policy field to MBI_FED_SSL.

image

i. Click Submit and confirm. You have successfully established STS between MSM live and your cloud application. Now we will setup the cloud application and configure it to receive token from STS service.

 

2. Configure azure application to accept the windows live security token:

a. Make sure you have WIF, VS 2010 and azure account.

b. Start with your cloud application. This blog is not going to cover the details of creating Azure applications. Once the cloud application is in place we need to first of all configure it to use HTTPS. Generate a certificate and add it to your cloud application on windows.azure.com. (you can use selfssl for this)

c. Configuring your cloud application to use this certificate requires modifying the role properties in cloud application using VS 2010.

d. Go to Web Role properties under the cloud service project and perform following:

         i ) Under certifications add the certificate that you just added to the windows.azure.com.

         ii) Under Endpoints check https and uncheck http. Choose the certificate we just added.

        iii) Under configuration setup trust level to Full Trust. Check HTTPS endpoint.

e. Add WIF classes to enable the cloud applicationto receive and process the claims.

Add following classes:

  • CustomIssuerTokenResolver.cs
  • WIFSampleRequestValidator.cs
  • Wlidticketforazure.cs

Use the link for the labs on WIF.

http://channel9.msdn.com/Learn/Courses/IdentityTrainingCourse/WebSitesAndIdentity/WebSitesAndIdentityLab

f. Add required assemblies for accessing CRM online webservice and using Identity Model. Keep in mind you will be deploying on cloud platform so you have to make sure Copy Local is “True” for all Dlls that you will need on Azure platform.

g. Add web reference for the crm web service.

h. Add STS reference to the windows Live STS. Right click on the Web Role and add STS. The federation utility will start.

image

In the application URI use the DNS name configured for the URN or azure application on msm.live.

i. In the ‘Security Token Service’ screen specify “Use an Existing STS’ and copy paste this link.

https://nexus.passport.com/FederationMetaData/2007-06/FederationMetaData.xml

j. Use no encryption and no chaining and click finish. A dialog would open up that federation utility finished successfully.

k. Update the default page to get the ticket that will be used for authentication.

 

string ticket = LiveIdTicketManager.RetrieveTicket("devicePassword(add anything here>", "crm.dynamics.com", "serviceAccount@live.com (this is received in the claim)", "<passWord>", "MBI_SSL", LiveIdTicketManager.LiveIdEnvironment.PROD (use INT for INT environment, true, "hostedServiceName" (url for azure application));

 

3. Configure CRM online for the service account and enable proxy account:

a. Make sure you have the account setup on CRM online and it has the proxy role.

4. Deploy application on Azure

a. Deploy the cloud app on Azure.

5. Test Application

a. Access the azure application. If all is configured properly you will be redirected to the windows live login page. Once you have given the login credentials you will receive the claim in your default application.

 

I have tried to skip the details and the post is long as is so please let me know if you have any questions. Make sure that you look into WIF in some detail so that you have clear understanding as to what is going on with the ticket and claim.

Ideally speaking once we have the claim we should use it for authentication in other applications. However CRM Dynamics online does not support claims so we used the ticket thanks to the Microsoft for providing with this work around. The actual purpose of this blog is to set ground for using claims with CRM 2011 online as it supports claim based authentication. I will be writing about that soon. Happy coding !!!

Data Migration from Legacy apps to CRM Dynamics 4.0

CRM Data Migration

Data migration is one of the most complicated aspects of a Microsoft Dynamics CRM implementation; however it is also the most critical.  If the data quality in a CRM system is poor, users will not rely on the system as their one stop for accurate customer data.

The data migration and import features let you upload data from varios customer relationship management systems and data sources into Microsoft Dynamics CRM. You can import data into standard and customized attributes of most entity types, including custom entities. During migration, you can upload related data of different types, such as notes and attachments. Data import lets you import new data or update existing data in Microsoft Dynamics CRM.

Microsoft Dynamics CRM includes the following tools for importing and migrating data:

  • Import Data Wizard, which is a Web application that is used for importing data records from multiple comma-separated values (CSV) files or from multiple XML Spreadsheet files.
  • Data Migration Manager, which is a stand-alone tool that is used for migrating data from multiple comma-separated values (CSV) files.

 Data migration can occur in two ways: Initial (one-time) or Ongoing (selective & scheduled).

 Initial Data Migration

During an initial data migration, the focus is on extracting data from your legacy systems, cleansing the data and importing it into your new Microsoft Dynamics CRM system.  This process can be time consuming as often the system values or fields have changed in the new system. Once the data has been cleansed and imported into Microsoft Dynamics CRM the task is complete.

Repetitive or Ongoing Data Migration

Many systems have needs for ongoing data migration. This can be caused by using some support applications to gather business related data and import that data into CRM to help sales team or can be caused by having specific departments continue to use legacy systems. 

Ways to import external data into Microsoft Dynamics CRM are the following: 

  1. Data Migration Manager (DMM)
  2. CRM Web Client Import / Import Wizard
  3. Custom utility using SDK for Microsoft Dynamics CRM
Data Migration manager (DMM)
 

 

Though there are many details to track when migrating data, the basic process is straightforward.

Data Migration Manager is a tool you can use to convert information from another customer relationship management (CRM) system or from a database to Microsoft Dynamics CRM. It can only be used by someone with the Microsoft Dynamics CRM Online System Administrator security role.

The Data Migration Manager (DMM) is a new version of the Data Migration Framework (DMF) in Microsoft CRM. The tool and the data migration process have been improved and simplified. For example:

  1. It is now possible to remove the migrated data.
  2. You can create custom entities without any previous mapping

There are four distinct stages:

  • Export source data – Export your data to comma-separated values (CSV) files, with data for each type of record in a separate file.
  • Install Data Migration Manager – Most of the features that are available in the Data Migration Manager are now available in the Import Data Wizard in Microsoft Dynamics CRM. However, it can be downloaded from the Microsoft site via the following link: http://www.microsoft.com/downloads/details.aspx?FamilyID=6766880a-da8f-4336-a278-9a5367eb79ca&displaylang=en
  • Create and migrate test data – Prepare a small number of test records, and use the Data Migration Manager to map and migrate the data.

During this step you’ll identify any changes you need to make to your source data and to Microsoft Dynamics CRM so that the data migrates correctly. You’ll need to use both the built-in wizard and the guidelines in the Help to identify potential issues you need to address.

After the test migration is successful, you can delete all data migrated by your test migration.

  • Migrate all data – Once your test data migrates successfully, and you’ve made any corrections you identified to your data map and to your data, you’re ready to migrate your entire data set.

The following diagram illustrates the process of migrating data.

CRM Data Migration

CRM Web Client Import

The native Microsoft Dynamics CRM Web Client Import receives data from a CSV or text file, allows field and pick-list value mapping.  The Microsoft Dynamics CRM import only works for Leads, Accounts, Contacts and Campaign Responses. It does not include the option for importing relationship mappings, notes, activities or email attachments in version 4.0.

Most of the features that are available in the Data Migration Manager are now available in the Import Data Wizard in Microsoft Dynamics CRM.

To ensure a successful import, you need to prepare your source data so that it can be mapped to Microsoft Dynamics CRM. For each default entity that can have data imported to it, the following attributes are listed:

  • Required: Source files should have one column for each required field in the Microsoft Dynamics CRM record type. All records must have data in each of these columns.
  • Relationship: If you want to connect one record to a record in another record type, you must have a column in your source file for the data that creates the relationship. If there is no data in a relationship column, the record will not be connected.
  • Calculated: Data in calculated fields is not imported. Instead, when the data used for the calculation is imported, Microsoft Dynamics CRM does the calculation and populates the field. You should identify any columns in your source data that map to calculated fields, so that you can select Ignore when mapping the column.
Custom Migration Utility

A custom migration utility provide the best results as it enables imports from multiple sources and not limited to Excel files, SQL Databases or CSVs.  Additionally they allow you to import relationship mappings, notes, activities and email attachments.  Import can be completed once or scheduled for regular imports.