Migrating your information warehouse workloads is without doubt one of the most difficult but important duties for any group. Whether or not the motivation is the expansion of your small business and scalability necessities or decreasing the excessive license and {hardware} value of your present legacy techniques, migrating just isn’t so simple as transferring information. At Databricks, our Skilled Companies crew (PS), has labored with tons of of shoppers and companions on migration tasks and have a wealthy report of profitable migrations. This weblog put up will discover greatest practices and classes discovered that any information skilled ought to think about when scoping, designing, constructing, and executing a migration.
5 phases for a profitable migration
At Databricks, we have now developed a five-phase course of for our migration tasks primarily based on our expertise and experience.
Earlier than beginning any migration mission, we start with the discovery part. Throughout this part, we intention to know the explanations behind the migration and the challenges of the prevailing legacy system. We additionally spotlight the advantages of migrating workloads to the Databricks Information Intelligence Platform. The invention part includes collaborative Q&A periods and architectural discussions with key stakeholders from the shopper, Databricks. Moreover, we use an automatic discovery profiler to achieve insights into the legacy workloads and estimate the consumption prices of the Databricks Platform to calculate TCO discount.
After finishing the invention part, we transfer on to a extra in-depth evaluation. Throughout this stage, we make the most of automated analyzers to judge the complexity of the prevailing code and acquire a high-level estimate of the trouble and price required. This course of offers helpful insights into the structure of the present information platform and the functions it helps. It additionally helps us refine the scope of the migration, eradicate outdated tables, pipelines, and jobs, and start contemplating the goal structure.
Within the migration technique and design part, we are going to finalize the main points of the goal structure and the detailed design for information migration, ETL, saved process code translation, and Report and BI modernization. At this stage, we can even map out the expertise between the supply and goal belongings. As soon as we have now finalized the migration technique, together with the goal structure, migration patterns, toolings, and chosen supply companions, Databricks PS, together with the chosen SI companion, will put together a migration Assertion of Work (SOW) for the Pilot (Part I) or a number of phases for the mission. Databricks has a number of licensed Migration Brickbuilder SI companions who present automated tooling to make sure profitable migrations. Moreover, Databricks Skilled Companies can present Migration Assurance companies together with an SI companion.
After the assertion of labor (SOW) is signed, Databricks Skilled Companies (PS) or the chosen Supply Accomplice carries out a manufacturing pilot part. On this part, a clearly outlined end-to-end use case is migrated to Databricks from the legacy platform. The info, code, and experiences are modernized to Databricks utilizing automated instruments and code converter accelerators. Finest practices are documented, and a Dash retrospective captures all the teachings discovered to establish areas for enchancment. A Databricks onboarding information is created to function the blueprint for the remaining phases, that are sometimes executed in parallel sprints utilizing agile Scrum groups.
Lastly, we progress to the full-fledged Migration execution part. We repeat our pilot execution strategy, integrating all the teachings discovered. This helps in establishing a Databricks Middle of Excellence (CoE) throughout the group and scaling the groups by collaborating with buyer groups, licensed SI companions, and our Skilled Companies crew to make sure migration experience and success.
Classes discovered
Suppose Large, Begin Small
It is essential in the course of the technique part to completely perceive your small business’s information panorama. Equally essential is to check a number of particular end-to-end use circumstances in the course of the manufacturing pilot part. Irrespective of how nicely you propose, some points could solely come up throughout implementation. It is higher to face them early to seek out options. An effective way to decide on a pilot use case is to start out with the top objective – for instance, choose a reporting dashboard that is essential for your small business, determine the information and processes wanted to create it, after which attempt creating the identical dashboard in your goal platform as a take a look at. This provides you with a good suggestion of what the migration course of will contain.
Automate the invention part
We start through the use of questionnaires and interviewing the database directors to know the scope of the migration. Moreover, our automated platform profilers scan by the information dictionaries of databases and hadoop system metadata to supply us with precise data-driven numbers on CPU utilizations, % ETL vs % BI utilization, utilization patterns by varied customers, and repair principals. This data may be very helpful in estimating the Databricks prices and the ensuing TCO Financial savings. Code complexity analyzers are additionally helpful as they supply us with the variety of DDLs, DMLs, Saved procedures, and different ETL jobs to be migrated, together with their complexity classification. This helps us decide the migration prices and timelines.
Leverage Automated Code Converters
Using automated code conversion instruments is crucial to expedite migration and reduce bills. These instruments assist in changing legacy code, comparable to saved procedures or ETL, to Databricks SQL. This ensures that no enterprise guidelines or features carried out within the legacy code are missed as a result of lack of documentation. Moreover, the conversion course of sometimes saves builders over 80% of improvement time, enabling them to promptly evaluation the transformed code, make vital changes, and deal with unit testing. It’s essential to make sure that the automated tooling can convert not solely the database code but in addition the ETL code from legacy GUI-based platforms.
Past Code Conversion—Information Issues Too
Migrations typically create a deceptive impression of a clearly outlined mission. After we take into consideration migration, we often deal with changing code from the supply engine to the goal. Nevertheless, it is essential to not overlook different particulars which are essential to make the brand new platform usable.
For instance, it’s essential to finalize the strategy for information migration, much like code migration and conversion. Information migration may be successfully achieved through the use of Databricks LakeFlow Join the place relevant or by selecting one among our CDC Ingestion companion instruments. Initially, in the course of the improvement part, it might be vital to hold out historic and catch-up hundreds from the legacy EDW, whereas concurrently constructing the information ingestion from the precise sources to Databricks. Moreover, it is very important have a well-defined orchestration technique utilizing Databricks Workflows, Delta Dwell Tables, or comparable instruments. Moreover, your migrated information platform ought to align together with your software program improvement and CI/CD practices earlier than the migration is taken into account full.
Do not ignore governance and safety
Governance and safety are different elements which are typically missed when designing and scoping a migration. No matter your present governance practices, we advocate utilizing the Unity Catalog at Databricks as your single supply of fact for centralized entry management, auditing, lineage, and information discovery capabilities. Migrating and enabling the Unity Catalog will increase the trouble required for the entire migration. Additionally, discover the distinctive capabilities that a few of our Governance companions present.
Information Validation and Consumer Testing is crucial for profitable migration
It’s essential for the success of the mission to have correct information validation and lively participation from enterprise Topic Matter Consultants (SMEs) throughout Consumer Acceptance Testing part. The Databricks migration crew and our licensed System Integrators (SIs) use parallel testing and information reconciliation instruments to make sure that the information meets all the information high quality requirements with none discrepancies. Sturdy alignment with executives ensures well timed and targeted participation of enterprise SMEs throughout user-acceptance testing, facilitating a fast transition to manufacturing and settlement on decommissioning older techniques and experiences as soon as the brand new system is in place.
Make It Actual – operationalize and observe your migration
Implement good operational greatest practices, comparable to information high quality frameworks, exception dealing with, reprocessing, and information pipeline observability controls, to seize and report course of metrics. This can assist establish and report any deviations or delays, permitting for fast corrective actions. Databricks options like Lakehouse Monitoring and our system billing tables assist in observability and FinOps monitoring.
Belief the specialists
Migrations may be difficult. There’ll all the time be tradeoffs to stability and surprising points and delays to handle. You want confirmed companions and options for the folks, course of, and expertise features of the migration. We advocate trusting the specialists at Databricks Skilled Companies and our licensed migration companions, who’ve in depth expertise in delivering high-quality migration options in a well timed method. Attain out to get your migration evaluation began.