Improve information safety and governance for Amazon Redshift Spectrum with VPC endpoints

Many shoppers are extending their information warehouse capabilities to their information lake with Amazon Redshift. They want to additional improve their safety posture the place they will implement entry insurance policies on their information lakes primarily based on Amazon Easy Storage Service (Amazon S3). Moreover, they’re adopting safety fashions that require entry to the information lake by their non-public networks.

Amazon Redshift Spectrum lets you run Amazon Redshift SQL queries on information saved in Amazon S3. Redshift Spectrum makes use of the AWS Glue Knowledge Catalog as a Hive metastore. With a provisioned Redshift information warehouse, Redshift Spectrum compute capability runs from separate devoted Redshift servers owned by Amazon Redshift which are impartial of your Redshift cluster. When enhanced VPC routing is enabled on your Redshift cluster, Redshift Spectrum connects from the Redshift VPC to an elastic community interface (ENI) in your VPC. As a result of it makes use of separate Redshift devoted clusters, to drive all visitors between Redshift and Amazon S3 by your VPC, that you must activate enhanced VPC routing and create a particular community path between your Redshift information warehouse VPC and S3 information sources.

When utilizing an Amazon Redshift Serverless occasion, Redshift Spectrum makes use of the identical compute capability as your serverless workgroup compute capability. To entry your S3 information sources from Redshift Serverless with out visitors leaving your VPC, you should use the improved VPC routing choice with out the necessity for any extra community configuration.

AWS Lake Formation gives an easy and centralized method to entry administration for S3 information sources. Lake Formation permits organizations to handle entry management for Amazon S3-based information lakes utilizing acquainted database ideas akin to tables and columns, together with extra superior choices akin to row-level and cell-level safety. Lake Formation makes use of the AWS Glue Knowledge Catalog to supply entry management for Amazon S3.

On this submit, we exhibit find out how to configure your community for Redshift Spectrum to make use of a Redshift provisioned cluster’s enhanced VPC routing to entry Amazon S3 information by Lake Formation entry management. You possibly can arrange this integration in a non-public community with no connectivity to the web.

Resolution overview

With this resolution, community visitors is routed by your VPC by enabling Amazon Redshift enhanced VPC routing. This routing choice prioritizes the VPC endpoint as the primary route precedence over an web gateway, NAT occasion, or NAT gateway. To forestall your Redshift cluster from speaking with sources exterior of your VPC, it’s essential to take away all different routing choices. This ensures that every one communication is routed by the VPC endpoints.

The next diagram illustrates the answer structure.

The answer consists of the next steps:

1. Create a Redshift cluster in a non-public subnet community configuration:
   1. Allow enhanced VPC routing on your Redshift cluster.
   2. Modify the route desk to make sure no connectivity to the general public community.
2. Create the next VPC endpoints for Redshift Spectrum connectivity:
   1. AWS Glue interface endpoint.
   2. Lake Formation interface endpoint.
   3. Amazon S3 gateway endpoint.
3. Analyze Amazon Redshift connectivity and community routing:
   1. Confirm community routes for Amazon Redshift in a non-public community.
   2. Confirm community connectivity from the Redshift cluster to varied VPC endpoints.
   3. Take a look at connectivity utilizing the Amazon Redshift question editor v2.

This integration makes use of VPC endpoints to ascertain a non-public connection out of your Redshift information warehouse to Lake Formation, Amazon S3, and AWS Glue.

Stipulations

To arrange this resolution, You want primary familiarity with the AWS Administration Console, an AWS account, and entry to the next AWS providers:

Moreover, you have to have built-in Lake Formation with Amazon Redshift to entry your S3 information lake in non-private community. For directions, confer with Centralize governance on your information lake utilizing AWS Lake Formation whereas enabling a contemporary information structure with Amazon Redshift Spectrum.

Create a Redshift cluster in a non-public subnet community configuration.

Step one is to configure your Redshift cluster to solely permit community visitors by your VPC and forestall any public routes. To perform this, you have to allow enhanced VPC routing on your Redshift cluster. Full the next steps:

1. On the Amazon Redshift console, navigate to your cluster.
2. Edit your community and safety settings.
3. For Enhanced VPC routing, choose Activate.
4. Disable the Publicly accessible choice.
5. Select Save modifications and modify the cluster to use the updates. You now have a Redshift cluster that may solely talk by the VPC. Now you’ll be able to modify the route desk to make sure no connectivity to the general public community.
   
6. On the Amazon Redshift console, make an observation of the subnet group and determine the subnet related to this subnet group.
   
7. On the Amazon VPC console, determine the route desk related to this subnet and edit to take away the default path to the NAT gateway.

If you happen to cluster is in a public subnet, you might have to take away the web gateway route. If subnet is shared amongst different sources, it might influence their connectivity.

Your cluster is now in a non-public community and might’t talk with any sources exterior of your VPC.

Create VPC endpoints for Redshift Spectrum connectivity

After you configure your Redshift cluster to function inside a non-public community with out exterior connectivity, that you must set up connectivity to the next providers by VPC endpoints:

• AWS Glue
• Lake Formation
• Amazon S3

Create an AWS Glue endpoint

To start with, Redshift Spectrum connects to AWS Glue endpoints to retrieve data from the AWS Knowledge Glue Catalog. To create a VPC endpoint for AWS Glue, full the next steps:

1. On the Amazon VPC console, select Endpoints within the navigation pane.
2. Select Create endpoint.
3. For Title tag, enter an optionally available title.
4. For Service class, choose AWS providers.
5. Within the Providers part, seek for and choose your AWS Glue interface endpoint.
   
6. Select the suitable VPC and subnets on your endpoint.
   
7. Configure the safety group settings and overview your endpoint settings.
8. Select Create endpoint to finish the method.

After you create the AWS Glue VPC endpoint, Redshift Spectrum will have the ability to retrieve data from the AWS Glue Knowledge Catalog inside your VPC.

Create a Lake Formation endpoint

Repeat the identical course of to create a Lake Formation endpoint:

1. On the Amazon VPC console, select Endpoints within the navigation pane.
2. Select Create endpoint.
3. For Title tag, enter an optionally available title.
4. For Service class, choose AWS providers.
5. Within the Providers part, seek for and choose your Lake Formation interface endpoint.
   
6. Select the suitable VPC and subnets on your endpoint.
7. Configure the safety group settings and overview your endpoint settings.
8. Select Create endpoint.

You now have connectivity for Amazon Redshift to Lake Formation and AWS Glue, which lets you retrieve the catalog and validate permissions on the information lake.

Create an Amazon S3 endpoint

The following step is to create a VPC endpoint for Amazon S3 to allow Redshift Spectrum to entry information saved in Amazon S3 through VPC endpoints:

1. On the Amazon VPC console, select Endpoints within the navigation pane.
2. Select Create endpoint.
3. For Title tag, enter an optionally available title.
4. For Service class, choose AWS providers.
5. Within the Providers part, seek for and choose your Amazon S3 gateway endpoint.
   
6. Select the suitable VPC and subnets on your endpoint.
7. Configure the safety group settings and overview your endpoint settings.
8. Select Create endpoint.

With the creation of the VPC endpoint for Amazon S3, you have got accomplished all obligatory steps to make sure that your Redshift cluster can privately talk with the required providers through VPC endpoints inside your VPC.

It’s essential to make sure that the safety teams hooked up to the VPC endpoints are correctly configured, as a result of an incorrect inbound rule could cause your connection to timeout. Confirm that the safety group inbound guidelines are accurately set as much as permit obligatory visitors to move by the VPC endpoint.

Analyze visitors and community topology

You should utilize the next strategies to confirm the community paths from Amazon Redshift to different endpoints.

Confirm community routes for Amazon Redshift in a non-public community

You should utilize an Amazon VPC useful resource map to visualise Amazon Redshift connectivity. The useful resource map reveals the interconnections between sources inside a VPC and the move of visitors between subnets, NAT gateways, web gateways, and gateway endpoints. As proven within the following screenshot, the highlighted subnet the place the Redshift cluster is working doesn’t have connectivity to a NAT gateway or web gateway. The route desk related to the subnet can attain out to Amazon S3 through VPC endpoint solely.

Be aware that AWS Glue and Lake Formation endpoints are interface endpoints and never seen on a useful resource map.

Confirm community connectivity from the Redshift cluster to varied VPC endpoints

You possibly can confirm connectivity out of your Redshift cluster subnet to all VPC endpoints utilizing the Reachability Analyzer. The Reachability Analyzer is a configuration evaluation software that lets you carry out connectivity testing between a supply useful resource and a vacation spot useful resource in your VPCs. Full the next steps:

1. On the Amazon Redshift console, navigate to the Redshift cluster configuration web page and notice the interior IP deal with.
2. On the Amazon EC2 console, seek for your ENI by filtering by the IP deal with.
3. Select the ENI related along with your Redshift cluster and select Run Reachability Analyzer.
   
4. For Supply kind, select Community interfaces.
5. For Supply, select the Redshift ENI.
6. For Vacation spot kind, select VPC endpoints.
7. For Vacation spot, select your VPC endpoint.
8. Select Create and analyze path.
   
9. When evaluation is full, view the evaluation to see reachability.

As proven within the following screenshot, the Redshift cluster has connectivity to the Lake Formation endpoint.

You possibly can repeat these steps to confirm community reachability for all different VPC endpoints.

Take a look at connectivity by working a SQL question from the Amazon Redshift question editor v2

You possibly can confirm connectivity by working a SQL question along with your Redshift Spectrum desk utilizing the Amazon Redshift question editor, as proven within the following screenshot.

Congratulations! You’ll be able to efficiently question from Redshift Spectrum tables from a provisioned cluster whereas enhanced VPC routing is enabled for visitors to remain inside your AWS community.

Clear up

You need to clear up the sources you created as a part of this train to keep away from pointless price to your AWS account. Full the next steps:

1. On the Amazon VPC console, select Endpoints within the navigation pane.
2. Choose the endpoints you created and on the Actions menu, select Delete VPC endpoints.
   
3. On the Amazon Redshift console, navigate to your Redshift cluster.
4. Edit the cluster community and safety settings and choose Flip off for Enhanced VPC routing.
   
5. It’s also possible to delete your Amazon S3 information and Redshift cluster in case you are not planning to make use of them additional.

Conclusion

By shifting your Redshift information warehouse to a non-public community setting and enabling enhanced VPC routing, you’ll be able to improve the safety posture of your Redshift cluster by limiting entry to solely approved networks.

We need to acknowledge our fellow AWS colleagues Harshida Patel, Fabricio Pinto, and Soumyajeet Patra for offering their insights with this weblog submit.

You probably have any questions or strategies, depart your suggestions within the feedback part. If you happen to want additional help with securing your S3 information lakes and Redshift information warehouses, contact your AWS account group.

Further sources

In regards to the Authors

Kanwar Bajwa is an Enterprise Help Lead at AWS who works with clients to optimize their use of AWS providers and obtain their enterprise aims.

Swapna Bandla is a Senior Options Architect within the AWS Analytics Specialist SA Group. Swapna has a ardour in the direction of understanding clients information and analytics wants and empowering them to develop cloud-based well-architected options. Exterior of labor, she enjoys spending time together with her household.