I’ve performed many data migrations over the years. Many have been successful, some have not. The failures have taught me as much about successful migration as the victories, if not more. The following are a few lessons I’ve learned along the way.
Plan More, Rework Less
The majority of migration failures occur due to a lack of planning. Data complications, scope creep, resource unavailability, and miscommunication all lead to rework. Rework consumes time and development resources, which eventually leads to a broken project budget and missed deadline. To prevent this, nail down all of these particulars up front.
Work with the end-users (or their representative) to generate the complete list of entities in the legacy system that need to move. Compare this list to entities in CRM to find out early what needs to move where, and what new entities need to be created to accomplish your goals, then use this information to create an entity-relationship diagram. Take your time here and make sure all entity relationships are named and labeled up front. This is the snowball at the top of a hill; if something goes wrong here there will be an avalanche of problems at the end.
Once the entities and relationships are nailed down, create a single map shared with all team members that shows where fields in each entity need to go. Guesswork is a good start, but verification with the end-client is the only way to know for sure. With that in mind…
Think Agile, Not Waterfall
Not everything can be planned for. In most cases the core entities and visible form-level fields in a legacy system are known by the end users, but often the underlying data relationships are hidden from a business user’s sight. Your project plan needs to be flexible enough to adapt when new data requirements are discovered.
Unfortunately, the nature of a migration project lends itself to a waterfall methodological approach due to the vast planning effort that needs to be done up front. A good way to counter this is by breaking the migration effort into entity-sized modules and planning your project’s agile sprints accordingly. If the central pillar of your project is the account entity, devote sprint one (sprint zero being your planning phase) to that, and only tackle the next related layer of entities in the next cycle.
Assuming the migration is using a full-featured data tool (Scribe Insight, SSIS with Dynamics Adapter, etc.) day one of your account entity sprint should begin by creating a quick draft of the account migration package using only those fields that exist out of the box and running it over a sample data set (more on this below). This will serve as a proof of concept as well as provide the team with an example of what the entity looks like in CRM as configured with data in it. From there, iterate to expand the scope of the entity, adding fields and mappings with daily (or more) test runs to fill out the entity. When everyone on the team agrees that the entity looks good with all of the mapped data, move on to adjusting the migration package/process to appropriately address the entire entity record set.
Some migrations are relatively small with record sets numbering in the low thousands, however more often a business’s legacy data will stretch back decades and include tens to hundreds of millions of rows. Regardless of the tool used to migrate such data the full process will be slow, so full record set migrations should only happen at the end of a project before the go-live date. Until then, the best approach to accommodate an agile development cycle is to select a sample set of data that represents 90% of business use cases. Work with end users if possible to get a good idea of the most common test cases with the most complete record relationships available.
The sample set should not be more than a few hundred records per entity, with the aim of a migration package runtime less than 5 minutes per entity. The goal of this is to be able to perform “complete” migration test runs of all entities in less than an hour. This will allow for faster iteration based on user feedback towards the end of a project.
Preserve Source Identifiers wherever possible
With the exception of flat files and very old and poorly-formatted databases, most legacy data sources will have a primary key unique identifier on every entity. This should be preserved within CRM as a custom field or alternate key, or outside of CRM using a cross-reference table. Not only will familiar record IDs help to get legacy users acclimated to the new system, keeping a link to the old system intact will allow you to easily update records programmatically during iteration. This will help to prevent duplicates and will cut down on the need to delete and recreate records during testing.
Code for the rule, not the exceptions
Almost every business has some set of rules or processes that define how data is created and maintained within their legacy system. Also, most rules or processes have exceptions. While it’s vital to take every known exception into consideration when performing your final migration, getting bogged down with writing a logical maze of exceptions early on will slow development.
Instead, begin each entity sprint by writing code for only the hard rules. As the sprint progresses add and test exceptions incrementally, making sure to keep the speed of migration development up to pace with form design so that daily tests can be performed on schedule. It’s easier to write exceptions for the missing 5% later in the sprint than it is to make the vast bulk wade through a maze of conditionals.
Latest posts by Avtex (see all)
- Avtex Joins Twin-Cities Based Norwest Equity Partners (NEP) Portfolio of Business Services Companies - February 12, 2018
- CRM Special Report: Fundamental Strategy for a Pivotal Objective - January 29, 2018
- Creating Meaningful Engagements in the Financial Services Industry - January 17, 2018