Image: knssr/Adobe Stock Migration of data from on-premises to cloud systems or between several cloud systems is a common and complex occasion across companies of all sizes and industries. The kinds of information being migrated can range from e-mail messages to Office documents and PDF files to databases, site information and code repositories.
SEE: Data migration testing list: Through pre- and post-migration (TechRepublic Premium)
No matter the intricacy of the information migration you’re completing, it’s important to complete testing in the pre-migration, migration and post-migration phases.
This can be a laborious process: It’s extremely simple to miss a key action and hurt the total security, performance and/or precision of your migration. However, if you make the effort to automate your information migration testing processes, you can save yourself time in the long run while establishing a clear and regulated testing plan.
Dive to:
Kinds of information migration screening
It’s challenging to specify the “types” of information migration screening that exist, since data migration testing can be classified in a variety of methods. For starters, testing methodologies might look different depending upon the kind of systems you’re migrating to and from.
SEE: Best practices to follow for information migration (TechRepublic)
For each of the list below types of information migration testing, it is necessary to consider how much information is kept in the system, how the data is formatted and how it might require to be transformed moving forward. Think about likewise any security or compliance functions that are constructed into the system and how important that information is to day-to-day company operations.
With that framing in mind, these are the various types of information migration testing, based on source system format:
- Database migration screening
- Running system migration testing
- Server migration screening
- Application migration testing
- Data center migration screening
- Cloud migration screening
The type of information migration testing you select to do may depend upon a range of other aspects as well, such as your timeline, your spending plan, and the in-house resources and teams you have on-hand to support the procedure.
Aspects to consider when screening migrated data
The following ten data migration elements should be tested and validated functional to guarantee the success of the migration cutover. While much of these aspects should be tested pre-migration, several others need to be evaluated throughout the migration procedure– even post-migration.
- Ease of access: The data can be accessed on the target source(s).
- Precision: The data is intact and functional.
- Reliability of transfer: Whether all of the information is moved over to accomplish a 100% transfer rate. Testing this will likely involve comparing dataset sizes on the source versus the target.
- Dependability of automation: Whether the automated transfers can be counted on to kick off and finish their tasks as anticipated.
- Speed: The rate at which data is transferred so as to develop a predictable standard.
- Repeatability: Whether the test can be run various times with the very same results.
- Mistake monitoring: Whether any mistakes happen in reading, moving or composing the information in other places, and how these mistakes can be fixed.
- Security: Making sure only the proper people and groups have access to the information on the target source(s).
- Enrichment: Whether the information and gain access to can be optimized on the target source(s).
- Defense: The information is supported and can be restored on the target source(s).
While there are a lot of consumer-focused tools that can move relatively little sets of data from a single system to another, the focus of this short article is on business-level migration tools, planned for larger datasets:
- Peak Data Loader: An open source Salesforce information migrator.
- AWS Data Pipeline: An option that migrates data between AWS data stores.
- Azure Cosmos DB: An open source command line tool that deals with numerous data sources.
- Azure DocumentDB: An open source data migration tool by Microsoft.
- Configero Data Loader: A web-based data loader application for Salesforce.
- Dell EMC Rainfinity: An information migration tool that works across heterogeneous environments.
- IBM Informix: An SQL-based data migration tool that works across multiple os.
- Informatica Cloud Data Wizard: A Salesforce data loader application that works with typical and custom-made things.
- SnapLogic: A combination platform as a service tool.
- Stitch Data: A cloud-based ETL platform.
Even the plain old rsync command is a quality information migration tool I myself consider a go-to option. When vetting out a possible information migration vendor, concentrate on compatibility with your environment, reliability, speed, security and scalability.
Methods for automating information migration tests
Evaluating with lots of time before the main cutover due date is generally the bulk of the hard work associated with data migration. The testing might be brief or extended, however it ought to be completely performed and verified before the procedure is moved forward into the “live” stage.
An automated data migration method is a crucial element here. You desire this process to work seamlessly while likewise operating in the background with very little human intervention. This is why I prefer constant or frequent replication to keep things in sync.
SEE: A guide to reliable data migration testing (TechRepublic)
One typical strategy is to run automated information synchronizations in the background through a scheduler or cron job, which only syncs brand-new information. Each time the procedure runs, the quantity of details transferred will end up being less and less.
This is known as drip data migration, and it works well due to the fact that many business use and upgrade a small set of their information every day. An initial migration of 10TB of information on the first day of screening might lead to a migration of simply 30GB of just recently altered or upgraded data during the minutes before the real cutover.
Steps for automating data migration testing
Back up your information
Always ensure to back up your data prior to continuing, even if your migration involves simply copying data from source to target. System and human mistakes can be a fearsome combination; I’ve seen instances of rsync operations gone terribly awry where target data was wrongly rsynced versus source data such that information was inadvertently removed.
Determine datasets, source systems and target systems for migration
Must-read big data protection
Determine the information to be moved and where it is to be moved. There may be multiple sources and multiple targets included and various top priority levels for different datasets. Guarantee you’re just going to migrate information you really require– think about running an information deduplication solution to improve your dataset at this moment– but be cognizant of any requirements involving data retention policies so you comply with them.
You ought to have a full understanding of what is located where. Many most importantly, you ought to know the total quantity of data to be migrated. You need to guarantee you have sufficient resources on the target end, particularly for information storage.
Use a drip information migration technique to test and move existing information
Whenever possible, strategy to execute a trickle data migration copy strategy, where your source is synced to target regularly and just the brand-new files need to be transferred in subsequent runs. Obviously, this implies your first migration operation will be the longest and most complicated. Employ supplier assistance as required.
Identify your automation technique and spot-check its accuracy
Recognize the automatic techniques and principles that will make sure the information migration works on its own. These should be applied throughout the board, no matter the information sources and/or urgency, for consistency and simpleness’s sake.
Tracking and notifies that inform your team of data migration development are key elements to consider now. Handbook information confirmation on the target end can be conducted through a “spot check” procedure, but you simply can’t inspect hundreds or thousands of files on a one-by-one basis.
Apply required security procedures
Ensure security is correctly applied in the source target environments, not only for data security however to ensure migration tools can function properly. Particularly for certain markets and running regions, it’s also important to consider what data governance and regulative protocols need to be included or preserved.
Enter into live testing with test information
Implement the solution and conduct a live test of irrelevant information. This typically involves using dummy files, however you must prevent using empty files; empty files will not work, as you wish to validate the contents appear the same on the target and the source system.
Set up automation and screen results
Configure and run the automated information migration procedure and monitor the results. Ensure every component in the Types Of Data Migration Screening in this article is sufficiently fulfilled.
This task, along with the rest of these actions, can be handled by an internal data migration group, however it may likewise be necessary to bring in vendor support to implement this level of automation and testing.
Check out next: Top cloud and application migration tools (TechRepublic)
