Legacy Data Migration

Legacy. Data. Migration.

Sounds ominous, right? Well, what we’ll do in this post is explain what legacy data is, why you (may) need migration to a new system, and the steps involved in a migration or conversion project. It’s not really as bad as it seems, and with the right people involved it can be a smooth process. Of course, there are always obstacles and traps in any project so we’ll point out a few of those so that you can avoid them.

What Is Legacy Data?

According to BusinessDictionary, legacy data is “information stored in an old or obsolete format or computer system that is, therefore, difficult to access or process.”

That’s a clear and concise definition, and we’ll break it down even more so we can understand the parts that make the whole.

Information

Part 1 of the Legacy Data trio is “Information.” This can be various kinds of data, such as personnel records, finance records, CJIS files (criminal reports and criminal data), student records, proprietary company data, and so on. It’s the data and/or images that you, at one point, decided worth saving and kept it somewhere like an electronic document management system or on some sort of backup media, like a hard drive or on network servers.

Stored in an old or obsolete format or computer system

Part 2 is that the information mentioned above is stored in an old or obsolete format or on legacy systems that are past their prime. Or both! You may have files that are stored in an obsolete format and also on an obsolete system.

Obsolete here doesn’t definitely mean that the format or system doesn’t work; obsolete just means that it’s out of date and may not be supported. The problem with using obsolete media formats or systems is that if they break for whatever reason, you may lose all your information because no one can fix the issue or recover the data. That’s bad.

Difficult to access or process

And Part 3 of the Legacy Data whole is that your information is difficult to access in its current environment. Like we mentioned above, just because you’re storing your information on obsolete legacy systems or in a format that is old and out of date doesn’t mean you can’t access it, but only that it’s harder to access that it could or should be. Your data quality can be just fine, but if you can’t get to it, what good does that do you?

We’re going to focus on already-digital data sources in this article, but the ideas and suggestions can also refer to physical media. Imagine you have microfilm (what’s microfilm, you ask? Look here) and use a Canon reader-printer to access the data. At some point the reader machine is going to break, and it’s old and no one can repair it because Canon doesn’t make the parts anymore. Both your physical media and your hardware “system” are obsolete, and the information is difficult to access.

Digitally speaking, you may have a document management system that was created by a small company years ago, using the leading-edge technology of the time. Let’s call it “DVDImage.” That company isn’t around anymore and no one else can maintain the environment, so you’ve been running on fumes for the past few years hoping your Windows Vista computer won’t break and take all your information with it. Even if it continues to run, the application is so old and out of date that most of your staff don’t know how to use it that well, and it makes finding documents harder than it needs to be. This is an obsolete system.

What Does Data Migration Mean?

In her article “Data Migration: The Strategy to Succeed,” Christine Taylor describes data migration as “ the process of transferring data from a source system to a target system.” She also makes a note to differentiate data migration from data conversion and data integration:

Note that data migration services are not the same thing as data conversion or data integration. To clarify:

Data Migration. Moving data between storage devices, locations, or systems. Includes subsets like quality assurance, cleansing, validation, and profiling.
Data Conversion. Transforms data from a legacy application to an updated or new application. The process is ETL: extract, transform, load.
Data Integration. Combines stored data residing in different systems to create a unified view and global analytics.

We like this clarification because in the real world the terms “migration” and “conversion” are often used interchangeably. Technically it’s incorrect to do this, but if you’re not someone in the weeds of the tech world, why would you know otherwise?

Throughout our article we’ll refer to both – for your specific project you may be running a pure legacy data migration or you could be both converting and migrating your data. Don’t get too concerned about specifics, just know that whichever you need, we can help you understand the process and help creat a successful project.

Why Do You Need Data Migration Services?

If your archived digital information and data is currently being stored in an electronic records management system that’s failing to meet your needs or is, as described earlier, obsolete, it’s probably time for you to consider data migration.

One reason to consider data migration is that you don’t want to lose all the records and information that’s currently stored on your legacy system. In a worst-case scenario, your system breaks or dies and all the data stored there is now inaccessible and lost forever. If the records on your system are critical to your organization then you can’t let this happen.

Another reason to migrate your legacy data is to preserve the original investment you made when you first created your digital database. We’re assuming that many of your legacy digital files came from media types such as microfilm and paper and were scanned to digital as part of a backfile conversion project. Even if they’re not mission-critical files, the original project was likely a significant investment that you don’t want to just throw away. additionally, you probably don’t even have the original hard copy records anymore! If you lose your digital information, there’s nothing to fall back on.

A final consideration for migration to a new system is that your end-users (staff, customers, or otherwise) will very likely enjoy an enhanced user experience utilizing a contemporary application. If your old system is obsolete it’s probably because it wasn’t continually updated or developed by the creator of the system, keeping it stuck in time and now making progress through the years. With a contemporary system, this shouldn’t be an issue.

What Are The Steps In A Data Migration Project?

A legacy data migration project typically includes the following steps:

Project Discovery
- Kickoff to figure out the high-level scope of your project.
Initial Assessment
- Legacy system assessment including document file formats, document classification, index and database schemas, etc.
Project Documentation
- Scope of work creation and review/approval.
Data Acquisition
- Legacy data securely transferred to our facility.
- Methods include encrypted USB drives and secure FTP electronic transfer.
Milestone 1 Proof of Concept
- Project process flow creation and testing using small batches of your legacy data.
Data Migration
- Once the Milestone 1 is approved, the remaining data will be converted and/or migrated for import into your new system.
In-Line Data Migration
- Projects can take some time, or may be done in batches, so some data may still be added to your legacy system while you’re migrating to a new system.
- These documents and files will be identified and processed so they are also migrated to your new records management system.
Exception Identification and Adjudication
- Any exceptions to the rules and process flow implemented for the conversion/migration will be identified.
- If contingency plans have already been decided, the exceptions will be processed using those contingency rules.
- If contingency plans have not been determined, exceptions will require adjudication so that they can be processed and included in the migration.
Data Delivery
- Delivery of your data to your new system, via secure USB or SFTP.
- Data will be formatted for automatic population into your new system.

How To Perform Data Mapping

Data mapping is the process of ensuring that the proper file identifiers get translated from one system to the next; basically ensuring data quality and data integrity. This means that fields that identify a file in your legacy data system are captured and populated in your new document management system and that they relate to the relevant file.

If the data mapping portion of your data migration isn’t executed properly, you could be in for a painful experience. Imagine looking for a file and either not finding it or actually having the index information point to the wrong document! This is what can happen if data mapping isn’t done correctly.

To mitigate the chances of incorrect data mapping, ensure your migration partner understands the fields and data that identify your documents, and that they’re clear about how that information will translate to your new system. Test small batches of documents to verify that the migration was done correctly and the mapping properly aligns in the new system.

Traps To Look Out For

As with most exciting endeavors, it’s important to know what to avoid as much as what you’re supposed to do correctly. Below are four traps to be aware of so that you don’t stumble as you move forward with your data migration project.

Not treating legacy data migration as its own project

Combining legacy data migration with another project, such as a backfile hard copy scanning project, can definitely work out if you do everything right. Even so, you’re probably better off treating the migration as its own project so that it gets the attention and resources it deserves; better to spend some more time and effort up front and have a successful ending instead of trying to cram everything in at once and then having to clean up a mess!

An example of combining projects is doing a large microfilm scanning conversion into Digital ReeL and including your migration from FileNet to Digital ReeL at the same time, assuming that it’s all the same. This assumption is normal because at the end of the whole job you’ll want to access all your files from Digital ReeL, so why not do it at once?

A cleaner way to move forward with the above scenario is to separate the two projects and create individual workflows: scan your microfilm first and then convert and migrate your existing FileNet docs after. This works because once you scan your film you may realize that you don’t need the records from the microfilm indexed (“mapped”) in the same way your other records are currently stored in FileNet. It may even make sense to keep the document types separate so that you know you’re looking for certain records on microfilm instead of what’s already been digitized. If instead you combined the projects, you may think you have to index the microfilm records the same as your FileNet docs, thereby creating more stress for yourself while you figure out how to identify the microfilm files, and also creating more work to achieve the end result – and more work usually means more expensive.

If you’re looking to execute multiple projects at once, spend some time and really think if it makes sense to do everything at once (“throwing in” data migration) or if it’s cleaner and simpler to separate the projects in a phased approach.

Expecting your data migration partner to handle everything

One of our worst fears! This isn’t an issue because we can’t do the technical work, but because without someone from your side providing necessary info, feedback, and guidance on what you actually want, it’s pretty tough to end up with a successful project.

Like anything else, you need to be involved. Imagine hiring a contractor to build you a house and you don’t tell them what you want. How will they know what to make?! Identify a primary point of contact on your side that knows your legacy system as well as the system that you’re moving to. Or if you need one person to represent each system, that’s fine, too.

We love legacy data migration and conversion projects, but if your team isn’t involved then you can expect delayed timelines, confusion, and overall sluggishness in getting the project completed.

Not involving your business people

Sure, migration of data is a technical project, but without the knowledge of how the information will be used once it’s migrated, or which data is important to keep or get rid of, all the tech time in the world won’t matter. Analogies are helpful: the most talented baker in the world could craft a perfect 7-layer chocolate cake with raspberry frosting (an award-winning cake!), but if you wanted a vanilla angel food cake, then who cares about the awards?

The same goes with legacy data migration: smart technical people creating complex rules and mapping processes won’t be much good if they don’t know where the information needs to go and how it’ll be used. You need the proper business people involved!

“Business people” is really meant as “stakeholders,” the ones who actually use the information or know its purpose. They can include end-users, C-level executives, analysts, and even the technical support people making the project work. The key is that you don’t push ahead with the project in a technical vacuum and just assume that because you have IT involved, it’ll all work out. Get the business people onboard and you’ll be happy you did.

Underestimating the effort of data migration

Legacy data conversion and migration s a lot of science and a lot of art, and you need both to be successful. Similar in theory to the section above about involving business people, having one piece of the puzzle by itself won’t work.

Just having the science of migration locked down won’t get you to the end of your project – at best, you’ll have a technically complete application that doesn’t function with the effectiveness that you want; at worst, it won’t work at all! All art doesn’t fly either: having a cool-looking electronic management system with all the bells and whistles doesn’t add up to much when you can’t find any documents you need, or the system crashes every time you load a new record. Include the scientists (tech folks) and the artists (business people) when you’re developing your migration plan and the collaborative effort will pay off tenfold in saved pain later on!

Also important to realize is that timelines expand and evolve. Humans are terrible at estimating how long a task will take, even for simple tasks, so you can imagine how estimating something as complex as a data migration is almost like a dart throw. It’s important to have an end date in mind so that there’s a target you can steer towards, but be ready to flex your project schedule if unanticipated events occur or other items become the priority and put your migration on the backburner.

Note: some of the ideas for this section of our article come from “Migrate Legacy Data Without Screwing It Up” by Marian Paul. His article is focused on Salesforce migration, but the information he provides is applicable to most any data migration.

Electronic Records Management Systems We Work With

There are hundreds, maybe thousands, of digital records systems available nowadays, and even if you have something obscure you can still move forward with a legacy data conversion or migration. Most systems are built in a similar way, so once you’ve done a few migrations it’s fairly simple to do another. Notice that we say “simple” and not “easy,” because although the general concept of a migration might be simple, the nuances of your specific requirements of the oddities of your particular content system can throw curveballs at us as we implement the project.

The takeaway from this section is that whatever records management system you currently use, or that you plan to migrate your data to, we can almost certainly provide the guidance and services you’ll need to make your project a success.

Here are some example systems that we’ve successfully worked with:

Laserfiche
OnBase
FileNet
Tiburon
Tyler
OpenText ApplicationXtender
DocuLynx
Canon Canofile
PaperVision

If you don’t see your records management system on this list, not to worry! Let us know what you have and we can work with you to figure out a solution.

Can You Provide A Content Management System?

Yes! If you’re looking for something that’s simple in design, easy to use, and cost-effective, we provide our Digital ReeL hosted application.

Digital ReeL was originally developed for microfilm, but over the years we’ve integrated microfiche and aperture cards to round out the microfilm media type and also pushed ahead into paper file and digital-to-digital import.

You may be thinking “I don’t need a microfilm and microfiche application for my boxes of files and digital records!” We agree. Digital ReeL was developed for microfilm, but because of its intuitiveness and simplicity, it’s organically grown into something that can ingest almost any digital material. Many of its functionalities are based on customer input, so only the most used and most effective capabilities are available. Just think, how many functions do you use when you’re in Microsoft Excel? Maybe 10 most of the time? And there are probably thousands available to you! Instead of loading up Digital ReeL with every possible function, we keep it slim so that your user experience is clean and clear.

Not sure if Digital ReeL is right for your legacy data migration or digital conversion project? We’ve put together a piece that describes the differences between a “traditional” microfilm conversion and a Digital ReeL microfilm conversion. Although it’s primarily about microfilm and microfiche scanning, the general ideas still hold true overall.

Next Steps

Reach out to us today! Click the “Get Your Quote” button below, fill out the form, and we’ll quickly reply to you to discuss your project.

GET YOUR QUOTE!