Finest practices for upgrading from Amazon Redshift DC2 to RA3 and Amazon Redshift Serverless

Amazon Redshift is a quick, petabyte-scale cloud knowledge warehouse that makes it easy and cost-effective to research your knowledge utilizing customary SQL and your present enterprise intelligence (BI) instruments. Tens of hundreds of shoppers depend on Amazon Redshift to research exabytes of information and run complicated analytical queries, delivering the very best price-performance.

With a totally managed, AI-powered, massively parallel processing (MPP) structure, Amazon Redshift drives enterprise decision-making shortly and cost-effectively. Beforehand, Amazon Redshift provided DC2 (Dense Compute) node sorts optimized for compute-intensive workloads. Nonetheless, they lacked the flexibleness to scale compute and storage independently and didn’t assist most of the fashionable options now out there. As analytical calls for develop, many shoppers are upgrading from DC2 to RA3 or Amazon Redshift Serverless, which supply impartial compute and storage scaling, together with superior capabilities similar to knowledge sharing, zero-ETL integration, and built-in synthetic intelligence and machine studying (AI/ML) assist with Amazon Redshift ML.

This publish supplies a sensible information to plan your goal structure and migration technique, overlaying improve choices, key concerns, and finest practices to facilitate a profitable and seamless transition.

Improve course of from DC2 nodes to RA3 and Redshift Serverless

Step one in the direction of improve is to know how the brand new structure ought to be sized; for this, AWS supplies a suggestion desk for provisioned clusters. When figuring out the configuration for Redshift Serverless endpoints, you possibly can assess compute capability particulars by analyzing the connection between RPUs and reminiscence. Every RPU allocates 16 GiB of RAM. To estimate the bottom RPU requirement, divide your DC2 nodes cluster’s complete RAM by 16. These suggestions present steering in sizing the preliminary goal structure however depend upon the computing necessities of your workload. To raised estimate your necessities, take into account conducting a proof of idea that makes use of Redshift Take a look at Drive to run potential configurations. To study extra, see Discover the very best Amazon Redshift configuration in your workload utilizing Redshift Take a look at Drive and Efficiently conduct a proof of idea in Amazon Redshift. After you resolve on the goal configuration and structure, you possibly can construct the technique for upgrading.

Structure patterns

Step one is to outline the goal structure in your resolution. You possibly can select the principle structure sample that finest aligns together with your use case from the choices offered in Structure patterns to optimize Amazon Redshift efficiency at scale. There are two principal eventualities, as illustrated within the following diagram.

On the time of writing, Redshift Serverless doesn’t have guide workload administration; the whole lot runs with computerized workload administration. Think about isolating your workload into a number of endpoints based mostly on use case to allow impartial scaling and higher efficiency. For extra data, discuss with Structure patterns to optimize Amazon Redshift efficiency at scale.

Improve methods

You possibly can select from two attainable improve choices when upgrading from DC2 nodes to RA3 nodes or Redshift Serverless:

• Full re-architecture – Step one is to guage and assess the workloads to find out whether or not you may gain advantage from a contemporary knowledge structure, then re-architect the prevailing platform in the course of the improve course of from DC2 nodes.
• Phased strategy– It is a two-stage technique. The primary stage includes an easy migration to the goal RA3 or Serverless configuration. Within the second stage, you possibly can modernize the goal structure by profiting from cutting-edge Redshift options.

We normally suggest a phased strategy, which permits for a smoother transition whereas enabling future optimization. The primary stage of a phased strategy consists of the next steps:

• Consider an equal RA3 nodes or Redshift Serverless configuration in your present DC2 cluster, utilizing the sizing tips for provisioned clusters or the compute capability choices for serverless endpoints.
• Completely validate the chosen goal configuration in a non-production setting utilizing Redshift Take a look at Drive. This automated instrument simplifies the method of simulating your manufacturing workloads on numerous potential goal configurations, enabling a complete what-if evaluation. This step is strongly beneficial.
• Proceed to the improve course of when you’re happy with the price-performance ratio of a specific goal configuration, utilizing one of many strategies detailed within the following part.

Redshift RA3 cases and Redshift Serverless present entry to highly effective new capabilities, together with zero-ETL, Amazon Redshift Streaming Ingestion, knowledge sharing writes, and impartial compute and storage scaling. To maximise these advantages, we suggest conducting a complete assessment of your present structure (the second stage of a phased strategy) to establish alternatives for modernization utilizing Amazon Redshift’s newest options. For instance:

Improve choices

You possibly can select from 3 ways to resize or improve a Redshift cluster from DC2 to RA3 or Redshift Serverless: snapshot restore, traditional resize, and elastic resize.

Snapshot restore

The snapshot restore technique follows a sequential course of that begins with capturing a snapshot of your present (supply) cluster. This snapshot is then used to create a brand new goal cluster together with your desired specs. After creation, it’s important to confirm knowledge integrity by confirming that knowledge has been accurately transferred to the goal cluster. An necessary consideration is that any knowledge written to the supply cluster after the preliminary snapshot have to be manually transferred to keep up synchronization.

This technique provides the next benefits:

• Permits for the validation of the brand new RA3 or Serverless setup with out affecting the prevailing DC2 cluster
• Supplies the flexibleness to revive to totally different AWS Areas or Availability Zones
• Minimizes cluster downtime for write operations in the course of the transition

Remember the next concerns:

• Setup and knowledge restore may take longer than elastic resize.
• You may encounter knowledge synchronization challenges. Any new knowledge written to the supply cluster after snapshot creation requires guide copying to the goal. This course of may want a number of iterations to attain full synchronization and require downtime earlier than cutoff.
• A brand new Redshift endpoint is generated, necessitating connection updates. Think about renaming each clusters so as to preserve the unique endpoint (ensure the brand new goal cluster adopts the unique supply cluster’s identify)

Traditional resize

Amazon Redshift creates a goal cluster and migrates your knowledge and metadata to it from the supply cluster utilizing a backup and restore operation. All of your knowledge, together with database schemas and person configurations, is precisely transferred to the brand new cluster. The supply cluster restarts initially and is unavailable for a couple of minutes, inflicting minimal downtime. It shortly resumes, permitting each learn and write operations because the resize continues within the background.

Traditional resize is a two-stage course of:

• Stage 1 (essential path) – Throughout this stage, metadata migration happens between the supply and goal configurations, quickly putting the supply cluster in read-only mode. This preliminary section is often transient. When this section is full, the cluster is made out there for learn and write queries. Though tables initially configured with KEY distribution type are quickly saved utilizing EVEN distribution, they are going to be redistributed to their unique KEY distribution throughout Stage 2 of the method.
• Stage 2 (background operations) – This stage focuses on restoring knowledge to its unique distribution patterns. This operation runs within the background with low precedence with out interfering with the first migration course of. The period of this stage varies based mostly on a number of components, together with the quantity of information being redistributed, ongoing cluster workload, and the goal configuration getting used.

The general resize period is primarily decided by the information quantity being processed. You possibly can monitor progress on the Amazon Redshift console or by utilizing the SYS_RESTORE_STATE system view, which shows the share accomplished for the desk being transformed (accessing this view requires superuser privileges).

The traditional resize strategy provides the next benefits:

• All attainable goal node configurations are supported
• A complete reconfiguration of the supply cluster rebalances the information slices to default per node, resulting in even knowledge distribution throughout the nodes

Nonetheless, remember the next:

• Stage 2 redistributes the information for optimum efficiency. Nonetheless, Stage 2 runs at a decrease precedence, and in busy clusters, it might probably take a very long time to finish. To hurry up the method, you possibly can manually run the ALTER TABLE DISTSTYLE command in your tables having KEY DISTSTYLE. By executing this command, you possibly can prioritize the information redistribution to occur quicker, mitigating any potential efficiency degradation because of the ongoing Stage 2 course of.
• As a result of Stage 2 background redistribution course of, queries can take longer to finish in the course of the resize operation. Think about enabling concurrency scaling as a mitigation technique.
• Drop pointless and unused tables earlier than initiating a resize to hurry up knowledge distribution.
• The snapshot used for the resize operation turns into devoted to this operation solely. Subsequently, it might probably’t be used for a desk restore or different function.
• The cluster should function inside a digital non-public cloud (VPC).
• This strategy requires a brand new or a current guide snapshot taken earlier than initiating a traditional resize.
• We suggest scheduling the operation throughout off-peak hours or upkeep home windows for minimal enterprise influence.

Elastic resize

When utilizing elastic resize to vary the node kind, Amazon Redshift follows a sequential course of. It begins by making a snapshot of your present cluster, then provisions a brand new goal cluster utilizing the latest knowledge from that snapshot. Whereas knowledge transfers to the brand new cluster within the background, the system stays in read-only mode. Because the resize operation approaches completion, Amazon Redshift robotically redirects the endpoint to the brand new cluster and stops all connections to the unique one. If any points come up throughout this course of, the system usually performs an computerized rollback with out requiring guide intervention, although such failures are uncommon.

Elastic resize provides a number of benefits:

• It’s a fast course of that takes 10–quarter-hour on common
• Customers preserve learn entry to their knowledge in the course of the course of, experiencing solely minimal interruption
• The cluster endpoint stays unchanged all through and after the operation

When contemplating this strategy, remember the next:

• Elastic resize operations can solely be carried out on clusters utilizing the EC2-VPC platform. Subsequently, it’s not out there for Redshift Serverless.
• The goal node configuration should present adequate storage capability for present knowledge.
• Not all goal cluster configurations assist elastic resize. In such instances, think about using traditional resize or snapshot restore.
• After the method is began, elastic resize can’t be stopped.
• Knowledge slices stay unchanged; this will probably trigger some knowledge or CPU skew.

Improve suggestions

The next flowchart visually guides the decision-making course of for selecting the suitable Amazon Redshift improve technique.

When upgrading Amazon Redshift, the strategy is dependent upon the goal configuration and operational constraints. For Redshift Serverless, at all times use the snapshot restore technique. If upgrading to an RA3 provisioned cluster, you possibly can select from two choices: use snapshot restore if a full upkeep window with downtime is suitable, or select traditional resize for minimal downtime, as a result of it rebalances the information slices to default per node, resulting in even knowledge distribution throughout the nodes. Though you should use elastic resize for sure node kind adjustments (for instance, DC2 to RA3) inside particular ranges, it’s not beneficial as a result of elastic resize doesn’t change the variety of slices, probably resulting in knowledge or CPU skew, which might later influence the efficiency of the Redshift cluster. Nonetheless, elastic resize stays the first suggestion when you might want to add or cut back nodes in an present cluster.

Finest practices for migration

When planning your migration, take into account the next finest practices:

• Conduct a pre-migration evaluation utilizing Amazon Redshift Advisor or Amazon CloudWatch.
• Select the fitting goal structure based mostly in your use instances and workloads. You need to use Redshift Take a look at Drive to find out the fitting goal structure.
• Backup utilizing guide snapshots, and allow automated rollback.
• Talk timelines, downtime, and adjustments to stakeholders.
• Replace runbooks with new structure particulars and endpoints.
• Validate workloads utilizing benchmarks and knowledge checksum.
• Use upkeep home windows for last syncs and cutovers.

By following these practices, you possibly can obtain a managed, low-risk migration that balances efficiency, price, and operational continuity.

Conclusion

Migrating from Redshift DC2 nodes to RA3 nodes or Redshift Serverless requires a structured strategy to assist efficiency, cost-efficiency, and minimal disruption. By choosing the fitting structure in your workload, and validating knowledge and workloads post-migration, organizations can seamlessly modernize their knowledge platforms. This improve facilitates long-term success, serving to groups totally harness RA3’s scalable storage or Redshift Serverless auto scaling capabilities whereas optimizing prices and efficiency.

In regards to the authors