Most Amazon Managed Streaming for Apache Kafka (Amazon MSK) prospects are simplifying and standardizing entry management to Kafka sources utilizing AWS Id and Entry Administration (IAM) authentication. This adoption can also be accelerated as Amazon MSK now helps IAM authentication in in style languages together with Java, Python, Go, JavaScript, and .NET.
Within the first a part of Configure a customized area title in your Amazon MSK cluster, we mentioned about why customized domains are necessary and offered particulars on how you can configure a customized area title in Amazon MSK when utilizing SASL_SCRAM authentication. On this put up, we focus on how you can configure a customized area title in Amazon MSK when utilizing IAM authentication. We suggest you learn the primary a part of this weblog because it captures answer particulars implementation steps.
Resolution overview
IAM authentication for Amazon MSK makes use of TLS to encrypt the Kafka protocol site visitors between the consumer and Kafka dealer. To make use of a customized area title, the Kafka dealer must current a server certificates that matches the customized area title. To realize this, this answer makes use of an Community Load Balancers (NLBs) with Amazon Certificates Supervisor to supply a customized certificates on behalf of the MSK brokers, and a Route 53 Non-public Hosted Zone to supply DNS for the customized area title.
The next diagram reveals all parts utilized by the answer.
Certificates administration
For shoppers to carry out TLS communication with the MSK cluster the cluster wants to supply a certificates with hostnames matching the customized area title. This answer makes use of a certificates in AWS Certificates Supervisor (ACM) signed with a Non-public Certificates Authority (PCA) for TLS with the customized area title. This answer makes use of a certificates with bootstrap.instance.com because the Frequent Identify (CN) in order that the certificates is legitimate for the bootstrap tackle, and Topic Various Names (SANs) are set for all dealer DNS names (comparable to b-1.instance.com). Since this answer makes use of a non-public certificates authority, the CA chain have to be imported into the consumer belief shops.
This answer works with any server certificates, whether or not certificates are signed by a public or personal Certificates Authority (CA). You’ll be able to import present certificates into ACM for use with this answer. Certificates should present a typical title and/or topic different names that match the bootstrap DNS tackle in addition to the person dealer DNS addresses. If the certificates is issued by a non-public CA, shoppers have to import the basis and intermediate CA certificates to the consumer belief retailer. If the certificates is issued by a public CA, the basis and intermediate CA certificates will likely be within the default belief retailer.
Community Load Balancer
The NLB supplies the flexibility to make use of a TLS listener. The ACM certificates is related to the listeners and permits TLS negotiation between the consumer and the NLB. The NLB performs a separate TLS negotiation between itself and the MSK brokers. Along with the above structure, this answer additionally permits utilizing AWS Non-public Hyperlink to attach the cluster to exterior VPCs. This enables safe entry to MSK between VPCs whereas utilizing a customized area title.
The next diagram illustrates the NLB port and goal configuration. A TLS listener with port 9000 is used for bootstrap connections with all MSK brokers set as targets. IAM authentication is configured to run on port 9098 of the MSK brokers utilizing a TLS goal sort. A TLS listener port is used to symbolize every dealer within the MSK cluster. On this put up, there are three brokers within the MSK cluster beginning with port 9001, representing dealer 1 and as much as port 9003, representing dealer 3.
Area Identify System (DNS)
For the consumer to resolve DNS queries for the customized area, we use an Amazon Route 53 personal hosted zone to host the DNS information, and affiliate it with the consumer’s VPC to allow DNS decision from the Route 53 VPC resolver. This answer makes use of a non-public MSK cluster and personal DNS. For publicly accessible MSK clusters a public NLB and DNS supplier comparable to a Route53 public hosted zone can be utilized.
Amazon MSK
Lastly, every dealer must have its marketed listeners configuration (marketed.listeners) up to date to match the customized area title and NLB ports. Marketed listeners is a configuration possibility utilized by Kafka shoppers to hook up with the brokers. By default, an marketed listener shouldn’t be set. As soon as set, Kafka shoppers use the marketed listener as a substitute of listeners to acquire the connection data for brokers. MSK brokers use the listener configuration to inform shoppers the DNS names and ports to make use of to hook up with the person brokers for every authentication sort enabled. Marketed listeners are distinctive to every dealer; and the cluster received’t begin if a number of brokers have the identical marketed listener tackle. For that reason, this answer makes use of a novel customized DNS title for every dealer (comparable to, b-1.instance.com).
Resolution Deployment
To deploy the answer, use the CloudFormation template from the GitHub repository.
This template deploys a VPC, NLB, PCA, ACM certificates, MSK cluster, and an Amazon EC2 occasion for cluster connectivity. The EC2 occasion features a script to deal with updating the dealer marketed.listeners settings to match the customized area title. For extra data on deploying a CloudFormation template, consult with Create a stack from the CloudFormation console.
After deploying the CloudFormation template, run the script to replace marketed listeners as follows:
- Retrieve the MSKClusterARN and CertificateAuthorityARN from the CloudFormation outputs in your stack as they are going to be utilized in subsequent steps.
- Navigate to the EC2 console and determine the KafkaClientInstance. Select Join to hook up with the occasion utilizing AWS Methods Supervisor Session Supervisor.
- Session Supervisor begins a session in shell. Begin a bash session with the command:
- The Kafka consumer SDKs have already been put in within the EC2 occasion. You’ll be able to replace the
marketed.listenersconfiguration as follows, changing CLUSTER_ARN with the ARN of your MSK cluster retrieved from CloudFormation in step 1:Observe that when this script completes, the brokers may have new marketed listeners configurations. Connections utilizing the usual IAM tackle for the MSK service is not going to work till we full the subsequent steps, because the brokers will redirect connections over this tackle again to the customized area title and TLS will fail.
- Subsequent, we have to create a truststore with the certificates for our AWS Non-public Certificates Authority (PCA) to permit TLS with the NLB. Within the following command, exchange PCA_ARN with the ARN of the PCA retrieved from CloudFormation in step 1:
We’re utilizing the default Java truststore which makes use of the password changeit.When requested “Belief this certificates?” enter “sure”. - Create a brand new properties file to permit IAM authentication with our customized truststore:
- Confirm you’ll be able to hook up with the cluster utilizing IAM authentication utilizing our new customized area title, changing bootstrap.instance.com with your personal customized area title in the event you used a special one in CloudFormation:
We’re utilizing the default Java truststore which makes use of the password 
Cleanup
To cease incurring prices navigate to CloudFormation and delete the CloudFormation stack to take away all sources provisioned by CloudFormation.
Continuously Requested Query about Customized Area Identify
Prospects have requested a couple of questions on implementing customized domains with MSK. You’ll find solutions to among the hottest questions right here.
Are there any limitations for this answer on MSK?
The marketed.listeners setting was eliminated as a dynamic dealer in KRaft-based Kafka clusters. Due to this fact, this answer is just supported in Zookeeper-based MSK clusters. Moreover, this answer is just relevant to SASL/SCRAM and IAM-authentication based mostly MSK clusters.
How the customized area title answer scales after we add new brokers?
When utilizing the NLB for dealer connectivity (possibility 2 within the configure a customized area title in your Amazon MSK cluster weblog put up), you have to so as to add an extra listener for every extra dealer created.
For TLS, if utilizing Topic Various Identify (SAN) to record particular person dealer DNS hostnames, you have to to create a brand new certificates that features the names of the extra brokers. One possibility is to create a certificates with SANs for extra brokers than wanted to permit for development.If a wildcard certificates is used, you do not want to change certificates when including brokers.
What adjustments are required after we take away brokers?
Amazon MSK helps scale-in by eradicating brokers from the cluster. Brokers are faraway from every availability zones (AZ). So a 6 dealer Amazon MSK cluster deployed in 3 AZ might be diminished to three dealer cluster deployed in 3 AZ. When brokers are eliminated, you’ll be able to take away the NLB listeners for the eliminated dealer together with the Route53 DNS endpoints. Nevertheless, you can too depart them as is, or simply take away the goal IP from the dealer numbers goal group. The NLB will mark the targets as unhealthy and cease directing site visitors to them. In the event you ever plan to scale-out the variety of brokers, you’ll be able to re-use the prevailing NLB listeners and Route 53 DNS entries and would solely have to replace the goal IPs used within the dealer numbers goal group.
Is there any change in configuration required if there may be any dealer failure?
No. When a dealer fails, Amazon MSK replaces the failed dealer with a brand new dealer occasion maintaining the configuration of the dealer precisely the identical. So, there can be no change within the marketed listener of the dealer. As soon as the dealer is wholesome, the dealer can settle for new connections and skim/write site visitors.
Can you utilize Amazon MSK Replicator between MSK clusters in a number of AWS Areas when utilizing the customized area title answer?
The Amazon MSK Replicator can be utilized when utilizing the customized area title answer, both in an active-passive or active-active setup. The identical course of might be adopted to set the customized area title.
You then comply with construct multi-Area resilient Apache Kafka purposes with an identical matter names utilizing Amazon MSK and Amazon MSK Replicator put up to configure MSK Replicator.
The next diagram reveals an active-active AWS multi-Area MSK setup utilizing the customized area title answer:
Can I take advantage of a world bootstrap DNS title to hook up with Amazon MSK clusters deployed throughout a number of AWS areas when IAM authentication is enabled?
No, it’s not doable to make use of a world bootstrap reference to symbolize MSK clusters deployed in a number of AWS Areas, until the consumer is conscious of the cluster’s area when connecting. To make use of IAM authentication, the right AWS Area have to be included within the IAM authentication request for a given cluster. It’s because the AWS Area is part of the Sigv4 authentication protocol utilized by IAM. This scope prevents the IAM authorization getting used to speak to a useful resource in one other AWS Area. You’ll be able to present the AWS Area in certainly one of two methods– with region-specific bootstrap URLs or by explicitly configuring the area.
For instance, if the bootstrap string is bootstrap.us-east-1.instance.com, then msk-iam-auth library will to extract the AWS Area from the dealer connection string and use us-east-1 in its IAM requests. If the bootstrap string is just bootstrap.instance.com, then the consumer should explicitly configure AWS_REGION=us-east-1 to hook up with the cluster whether it is in us-east-1, or us-west-2 whether it is in us-west-2.
Observe that this can be a limitation for IAM authentication, however not for SASL/SCRAM authentication. With SASL/SCRAM authentication, if the consumer’s credentials are utilized to each clusters the worldwide endpoint can level to both cluster and the consumer will be capable to join. The AWS Area shouldn’t be utilized in SASL/SCRAM authentication, so it doesn’t limit the authentication scope.
The best way to permit public entry to a non-public MSK cluster utilizing the customized area title answer?
To supply public entry to a MSK cluster utilizing the customized area answer, you have to to do the next:
- Create an Web-facing NLB, and affiliate public subnets (subnets which have a path to the Web Gateway hooked up to the VPC).
- Create ingress guidelines in each the NLB and MSK safety teams allowing the required public addresses. Observe: the port will likely be 9098 for the MSK safety group, and the ports you’re utilizing on the NLB listeners.
- Present public DNS decision for the Kafka shoppers, by utilizing a Route 53 public zone, or another public DNS resolver.
- The consumer wants have IAM credentials, with permission, to speak to the MSK brokers, utilizing an IAM position, IAM entry keys, IAM Roles Anyplace, or one other mechanism that makes use of the AWS Safety Token Service (AWS STS) to create and supply trusted customers with non permanent safety credentials.
Within the first a part of the weblog, two patterns have been highlighted. The best way to determine which sample to make use of and why?
Choice 1: Solely bootstrap connection by means of NLB
If the Kafka shoppers have direct entry to the dealer, then you need to use customized area title for the bootstrap connection whereas the shoppers can nonetheless hook up with the MSK Brokers with dealer DNS. That is the only possibility, because it doesn’t require customized TLS certificates or TLS listeners.Observe that this feature shouldn’t be mandatory when utilizing MSK Categorical brokers, as MSK Categorical brokers already manages bootstrapping by way of a broker-agnostic connection string. For MSK Categorical, this feature doesn’t add worth aside from configuring a customized area title for appearances / simplicity of consumer configuration. For MSK Customary brokers, this could enhance consumer connectivity by making connection strings dealer agnostic.
Choice 2: All connections by means of NLB
When Kafka shoppers don’t have direct entry to Amazon MSK Brokers, routing all connections by means of the NLB might be most popular. This may happen when a consumer is deployed in a special VPC than Amazon MSK VPC or the consumer is exterior, and when Amazon MSK Multi VPC Connectivity shouldn’t be an possibility. Usually, Amazon MSK Multi VPC Connectivity is most popular as this can be a easier sample for many organizations to handle MSK Connectivity throughout accounts and VPCs.When Multi VPC Connectivity shouldn’t be an possibility, NLB can be utilized to supply connectivity with Transit Gateway or PrivateLink, and the answer talked about within the weblog must be used.
Right here is an instance structure how Kafka consumer and Amazon MSK cluster deployed in two separate VPCs however linked by way of AWS Non-public Hyperlink.
Is Amazon Route 53 required to make use of a customized area title with Amazon MSK?
You should use another DNS resolver service, and don’t require Amazon Route 53 to make use of a customized area title with Amazon MSK. The one requirement is that your shoppers can resolve in opposition to your DNS resolver service. The one change required, is to make use of a CNAME for the DNS information, referencing the NLBs DNS report, rather than the Alias information, as that is report sort is just out there in Amazon Route 53.
We don’t use Amazon Certificates Supervisor (ACM), can NLB combine with different third occasion certificates managers?
NLB solely helps ACM to bind a certificates to a TLS listener. You’ll be able to import a certificates created utilizing your third occasion certificates supervisor into ACM, and don’t have to create a certificates utilizing ACM.
Getting connection to node terminated throughout authentication after setting marketed.listeners , what may very well be the problem?
As the problem began to happen after altering the marketed.listeners configuration, the problem is unlikely to be associated to permissions. The next could cause this problem:
- The NLB and/or consumer’s Safety Group doesn’t allow entry to the listener ports on the NLB from the consumer.
- A firewall equipment between the NLB and consumer doesn’t allow the consumer to speak to the NLB utilizing the listener ports.
- The
marketed.listenersconfiguration has an error inflicting the consumer to obtain invalid particulars, comparable to a typo within the title. If that is so, use a consumer in the identical VPC because the MSK dealer that has IAM permissions to speak to the MSK dealer, and Safety Group guidelines allowing connectivity, you then use the next command to delete themarketed.listenersconfiguration.
BROKERS_AMAZON_DNS_NAME comparable to b-1.clustername.xxxxxx.yy.kafka.area.amazonaws.com:9098.
Getting “surprising dealer id, anticipated 2 or empty string, however obtained 1”, what’s inflicting this error?
This error is usually offered when the marketed.listeners configuration for one of many brokers has the port utilized by one other dealer set. For instance dealer 2 has port 9001 set for IAM, however this port is used to hook up with dealer 1, so dealer 1 is responding with an error to say you offered dealer id 2, however I’m dealer 1.
To right this, you have to to replace the dealer with the inaccurate marketed.listeners configuration to make use of the right port. To realize entry to the dealer to make the change, you have to to make use of the next command to delete the inaccurate configuration:
BROKERS_AMAZON_DNS_NAME comparable to b-2.clustername.xxxxxx.yy.kafka.area.amazonaws.com:9098.
You then want to make use of the next command to set the marketed.listeners configuration for that dealer:
Observe: The marketed.listeners configuration within the under assumes solely IAM is used for authentication. In case you are utilizing extra authentication choices, you have to to incorporate them.
Abstract
On this put up, we defined how you need to use an NLB, Route 53, and the marketed listener configuration possibility in Amazon MSK to help customized domains with MSK clusters when utilizing IAM authentication. You should use this answer to maintain your present Kafka bootstrap DNS title and scale back or take away the necessity to change consumer purposes due to a migration, restoration course of, or to make use of a DNS title according to your group’s naming conference (for instance, msk.prod.instance.com).
Attempt the answer out for your self, and depart your questions and suggestions within the feedback part.