Stream mainframe knowledge to AWS in close to actual time with Exactly and Amazon MSK

It is a visitor publish by Supreet Padhi, Expertise Architect, and Manasa Ramesh, Expertise Architect at Exactly in partnership with AWS.

Enterprises depend on mainframes to run mission-critical purposes and retailer important knowledge, enabling real-time operations that assist obtain enterprise aims. These organizations face a standard problem: learn how to unlock the worth of their mainframe knowledge in at present’s cloud-first world whereas sustaining system stability and knowledge high quality. Modernizing these programs is crucial for competitiveness and innovation.

The digital transformation crucial has made mainframe knowledge integration with cloud companies a strategic precedence for enterprises worldwide. Organizations that may seamlessly bridge their mainframe environments with fashionable cloud platforms acquire important aggressive benefits by improved agility, diminished operational prices, and enhanced analytics capabilities. Nonetheless, implementing such integrations presents distinctive technical challenges that require specialised options. Among the challenges embrace changing EBCDIC knowledge to ASCII, the place the dealing with of information varieties is exclusive to the mainframe, akin to binary knowledge and COMP knowledge. Knowledge saved in Digital Storage Entry Methodology (VSAM) information could be fairly advanced as a consequence of practices to retailer a number of totally different file varieties in a single file. To deal with these challenges, Exactly—a worldwide chief in knowledge integrity, serving over 12,000 prospects—has partnered with Amazon Net Providers (AWS) to allow real-time synchronization between mainframe programs and Amazon Relational Database Service (Amazon RDS). For extra on this collaboration, try our earlier weblog publish: Unlock Mainframe Knowledge with Exactly Join and Amazon Aurora.

On this publish, we introduce another structure to synchronize mainframe knowledge to the cloud utilizing Amazon Managed Streaming for Apache Kafka (Amazon MSK) for better flexibility and scalability. This event-driven method offers further potentialities for mainframe knowledge integration and modernization methods.

A key enhancement on this resolution is the usage of the AWS Mainframe Modernization – Knowledge Replication for IBM z/OS Amazon Machine Picture (AMI) out there in AWS Market, which simplifies deployment and reduces implementation time.

Actual-time processing and event-driven structure advantages

Actual-time processing makes knowledge actionable inside seconds somewhat than ready for batch processing cycles. For instance, monetary establishments akin to World Funds have leveraged this resolution to modernize mission-critical banking operations, together with funds processing. By migrating these operations to the AWS Cloud, they enhanced consumer expertise, improved scalability and maintainability, whereas enabling superior fraud detection – all with out impacting the efficiency of current mainframe programs. Change knowledge seize (CDC) permits this by figuring out database adjustments and delivering them in actual time to cloud environments.

CDC gives two key benefits for mainframe modernization:

• Incremental knowledge motion – Eliminates disruptive bulk extracts by streaming solely modified knowledge to cloud targets, minimizing system impression and guaranteeing knowledge foreign money
• Actual-time synchronization – Retains cloud purposes in sync with mainframe programs, enabling fast insights and responsive operations

Resolution overview

On this publish, we offer an in depth implementation information for streaming mainframe knowledge adjustments from DB2z by AWS Mainframe Modernization – Knowledge Replication for IBM z/OS AMI to Amazon MSK after which making use of these adjustments to Amazon Relational Database Service (Amazon RDS) for PostgreSQL utilizing MSK Join with the Confluent JDBC Sink Connector.

By introducing Amazon MSK into structure and streamlining deployment by the AWS Market AMI, we create new potentialities for knowledge distribution, transformation, and consumption that increase upon our beforehand demonstrated direct replication method. This streaming-based structure gives a number of further advantages:

• Simplified deployment – Speed up implementation utilizing the preconfigured AWS Market AMI
• Decoupled programs – Separate the priority of information extraction from knowledge consumption, permitting each side to scale independently
• Multi-consumer assist – Allow a number of downstream purposes and companies to devour the identical knowledge stream based on their very own necessities
• Extensibility – Create a basis that may be prolonged to assist further mainframe knowledge sources akin to IMS and VSAM, in addition to further AWS targets utilizing MSK Join sink connectors

The next diagram illustrates the answer structure.

1. Seize/Writer – Join CDC Seize/Writer captures Db2 adjustments from Db2 logs utilizing IFI 306 Learn and communicates captured knowledge adjustments to a goal engine by TCP/IP.
2. Controller Daemon – The Controller Daemon authenticates all connection requests, managing safe communication between the supply and goal environments.
3. Apply Engine – The Apply Engine is a multifaceted and multifunctional element within the goal atmosphere. It receives the adjustments from the Writer agent and applies the modified knowledge to the goal Amazon MSK.
4. Join CDC Single Message Remodel (SMT) – Performs all mandatory knowledge filtering, transformation, and augmentation required by the sink connector.
5. JDBC Sink Connector – As knowledge arrives, an occasion of the JDBC Sink Connector together with Apache Kafka writes the information to focus on tables in Amazon RDS.

This structure offers a clear separation between the information seize course of and the information consumption course of, permitting every to scale independently. The usage of MSK as an middleman permits a number of programs to devour the identical knowledge stream, opening potentialities for advanced occasion processing, real-time analytics, and integration with different AWS companies.

Conditions

To finish the answer, you want the next stipulations:

1. Set up AWS Mainframe Modernization – Knowledge Replication for IBM z/OS
2. Have entry to Db2z on mainframe from AWS utilizing your authorized connectivity between AWS and your mainframe

Resolution walkthrough

The next code content material shouldn’t be deployed to manufacturing environments with out further safety testing.

Configure the AWS Mainframe Modernization Knowledge Replication with Exactly AMI on Amazon EC2

Comply with the steps outlined at Exactly AWS Mainframe Modernization Knowledge Replication. Upon the preliminary launch of the AMI, use the next command to hook up with the Amazon Elastic Compute Cloud (Amazon EC2) occasion:

ssh -i ami-ec2-user.pem ec2-user@$AWS_AMI_HOST

Configure the serverless cluster

To create an Amazon Aurora PostgreSQL-Suitable Version Serverless v2 cluster, full the next steps:

1. Create a DB cluster through the use of the next AWS Command Line Interface (AWS CLI) command. Substitute the placeholder strings with values that correspond to your cluster’s subnet and subnet group IDs.

   aws rds create-db-cluster 
      --db-cluster-identifier cdc-serverless-pg-cluster 
      --engine aurora-postgresql 
      --serverless-v2-scaling-configuration MinCapacity=1,MaxCapacity=2 
      --master-username connectcdcuser 
      --manage-master-user-password 
      --db-subnet-group-name "<subnet-security-group-id>" 
      --vpc-security-group-ids "<cluster-security-group-id>"

2. Confirm the standing of the cluster through the use of the next command:

   aws rds describe-db-clusters --db-cluster-identifier cdc-serverless-pg-cluster

3. Add a author DB occasion to the Aurora cluster:

   aws rds create-db-instance 
      --db-cluster-identifier cdc-serverless-pg-cluster 
      --db-instance-identifier cdc-serverless-pg-instance 
      --db-instance-class db.serverless 
      --engine aurora-postgresql

4. Confirm the standing of the author occasion:

   aws rds describe-db-instances --db-instance-identifier cdc-serverless-pg-instance

Create a database within the PostgreSQL cluster

After your Aurora Serverless v2 cluster is working, it’s worthwhile to create a database to your replicated mainframe knowledge. Comply with these steps:

1. Set up the psql shopper:

   sudo yum set up postgresql16

2. Retrieve the password from secret supervisor:

   aws secretsmanager get-secret-value --secret-id '<cdc-serverless-pg-cluster-secret ARN>' --query 'SecretString' --output textual content

3. Create a brand new database in PostgreSQL:

   PGPASSWORD="password" psql --host=<DATABASE-HOST> --username=connectcdcuser --dbname=postgres -c "CREATE DATABASE dbcdc"

Configure the serverless MSK cluster

To create a serverless MSK cluster, full the next steps:

1. Copy the next JSON and paste it into a brand new file create-msk-serverless-cluster.json. Substitute the placeholder strings with values that correspond to your cluster’s subnet and safety group IDs.

      {
        "VpcConfigs": [
          {
            "subnets": [
              "<cluster-subnet-1>",
              "<cluster-subnet-2>",
              "<cluster-subnet-3>"
            ],
            "securityGroups": ["<cluster-security-group-id>"]
          }
        ],
        "ClientAuthentication": {
          "Sasl": {
            "Iam": {
              "Enabled": true
            }
          }
        }
      }

2. Invoke the next AWS CLI command within the folder the place you saved the JSON file within the earlier step:

   aws kafka create-cluster-v2 --cluster-name pgsqlmsk --serverless file://create-msk-serverless-cluster.json

3. Confirm cluster standing by invoking the next AWS CLI command:

   aws kafka list-clusters-v2 --cluster-type-filter SERVERLESS

4. Get the bootstrap dealer deal with by invoking the next AWS CLI command:

   aws kafka get-bootstrap-brokers --cluster-arn "<msk-serverless-cluster-arn>"

5. Outline the atmosphere variable to retailer the bootstrap servers of the MSK cluster and regionally set up Kafka within the path atmosphere variable:

   export BOOTSTRAP_SERVERS=<kafka_bootstrap_servers_with_ports>

Create a subject on the MSK cluster

To create a Kafka matter, it’s worthwhile to set up the Kafka CLI first. Comply with these steps:

1. Obtain the binary distribution of Apache Kafka and extract the archive in folder kafka:

   wget https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz
      tar -xzf kafka_2.13-3.9.0.tgz
      ln -sfn kafka_2.13-3.9.0 kafka

2. To make use of IAM to authenticate with the MSK cluster, obtain the Amazon MSK Library for IAM and replica to the native Kafka library listing as proven within the following code. For full directions, check with Configure purchasers for IAM entry management.

   wget https://github.com/aws/aws-msk-iam-auth/releases/obtain/v2.3.1/aws-msk-iam-auth-2.3.1-all.jar
   cp aws-msk-iam-auth-2.3.1-all.jar kafka/libs

3. Within the listing, create a file to configure a Kafka shopper to make use of IAM authentication for the Kafka console producer and customers:

   safety.protocol=SASL_SSL
      sasl.mechanism=AWS_MSK_IAM
      sasl.jaas.config=software program.amazon.msk.auth.iam.IAMLoginModule required; sasl.shopper.callback.handler.class=software program.amazon.msk.auth.iam.IAMClientCallbackHandler

4. Create the Kafka matter, which you outlined within the connector config:

   kafka/bin/kafka-topics.sh --create --bootstrap-server $BOOTSTRAP_SERVERS --command-config kafka/config/client-config.properties --partitions 1 --topic pgsql-sink-topic

Configure the MSK Join plugin

Subsequent, create a {custom} plugin out there within the AMI at /decide/exactly/di/packages/sqdata-msk_connect_1.0.1.zip which accommodates the next:

• JDBC Sink Connector from Confluent
• MSK Config supplier
• AWS Mainframe Modernization – Knowledge Repication for IBM z/OS Customized SMT

Comply with these steps:

1. Invoke the next to add the .zip file to an S3 bucket to which you could have entry:

   aws s3 cp /decide/exactly/di/packages/sqdata-msk_connect_1.0.1.zip s3://<bucket>/

2. Copy the next JSON and paste it into a brand new file create-custom-plugin.json. Substitute the placeholder strings with values that correspond to your bucket.

   {
        "contentType": "ZIP",
        "description": "jdbc sink connector",
        "location": {
          "s3Location": {
            "bucketArn": "arn:aws:s3:::<bucket>",
            "fileKey": "sqdata-msk_connect_1.0.1.zip"
          }
        },
        "identify": "jdbc-sink-connector"
      }

3. Invoke the next AWS CLI command within the folder the place you saved the JSON file within the earlier step:

   aws kafkaconnect create-custom-plugin --cli-input-json file://create-custom-plugin.json

4. Confirm plugin standing by invoking the next AWS CLI command:

   aws kafkaconnect list-custom-plugins

Configure the JDBC Sink Connector

To configure the JDBC Sink Connector, observe these steps:

1. Copy the next JSON and paste it into a brand new file create-connector.json. Substitute the placeholder strings with acceptable values:

   {
        "connectorConfiguration": {
          "connector.class": "io.confluent.join.jdbc.JdbcSinkConnector",
          "connection.url": "jdbc:postgresql://<postgresql-endpoint>
   /dbcdc?currentSchema=public",
          "config.suppliers": "secretsmanager",
          "config.suppliers.secretsmanager.class": "com.amazonaws.kafka.config.suppliers.SecretsManagerConfigProvider",
          "connection.consumer": "${secretsmanager:MySecret-1234:username}",
          "connection.password": "${secretsmanager:MySecret-1234:password}",
          "config.suppliers.secretsmanager.param.area": "<area>",
          "duties.max": "1",
          "subjects": "pgsql-sink-topic",
          "insert.mode": "upsert",
          "delete.enabled": "true",
          "pk.mode": "record_key",
          "auto.evolve": "true",
          "auto.create": "true",
          "worth.converter": "org.apache.kafka.join.storage.StringConverter",
          "key.converter": "org.apache.kafka.join.storage.StringConverter",
          "transforms": "ConnectCDCConverter",
          "transforms.ConnectCDCConverter.kind": "com.exactly.kafkaconnect.ConnectCDCConverter",
          "transforms.ConnectCDCConverter.cdc.a number of.tables.enabled": "true",
          "transforms.ConnectCDCConverter.cdc.supply.desk.identify.ignore.schema": "true"
        },
        "connectorName": "pssql-sink-connector",
        "kafkaCluster": {
          "apacheKafkaCluster": {
            "bootstrapServers": "<msk-bootstrap-servers-string>",
            "vpc": {
              "subnets": [
                "<cluster-subnet-1>",
                "<cluster-subnet-2>",
                "<cluster-subnet-3>"
              ],
              "securityGroups": ["<cluster-security-group-id>"]
            }
          }
        },
        "capability": {
          "provisionedCapacity": {
            "mcuCount": 1,
            "workerCount": 1
          }
        },
        "kafkaConnectVersion": "3.7.x",
        "serviceExecutionRoleArn": "<arn-of-a-role-that-msk-connect-can-assume>",
        "plugins": [
          {
            "customPlugin": {
              "customPluginArn": "<arn-of-custom-plugin-that-contains-connector-code>",
              "revision": 1
            }
          }
        ],
        "kafkaClusterEncryptionInTransit": {"encryptionType": "TLS"},
        "kafkaClusterClientAuthentication": {"authenticationType": "IAM"},
        "logDelivery": {
          "workerLogDelivery": {
            "cloudWatchLogs": {
              "enabled": true,
              "logGroup": "<loggroup>"
            }
          }
        }
      }

2. Invoke the next AWS CLI command within the folder the place you saved the JSON file within the earlier step:

   aws kafkaconnect create-connector --cli-input-json file://create-connector.json

3. Confirm connector standing by invoking the next AWS CLI command:

   aws kafkaconnect list-connectors

Arrange Db2 Seize/Writer on Mainframe

To ascertain the Db2 Seize/Writer on the mainframe for capturing adjustments to the DEPT desk, observe these structured steps that construct upon our earlier weblog publish, Unlock Mainframe Knowledge with Exactly Join and Amazon Aurora:

1. Put together the supply desk. Earlier than configuring the Seize/Writer, make sure the DEPT supply desk exists in your mainframe Db2 system. The desk definition ought to match the construction outlined at $SQDATA_VAR_DIR/templates/dept.ddl. If it’s worthwhile to create this desk in your mainframe, use the DDL from this file as a reference to make sure compatibility with the replication course of.
2. Entry the Interactive System Productiveness Facility (ISPF) interface. Register to your mainframe system and entry the AWS Mainframe Modernization – Knowledge Repication for IBM z/OS ISPF panels by the equipped ISPF software menu. Choose choice 3 (CDC) to entry the CDC configuration panels, as demonstrated in our earlier weblog publish.
3. Add supply tables for seize:
   1. From the CDC Main Choice Menu, select choice 2 (Outline Subscriptions).
   2. Select choice 1 (Outline Db2 Tables) so as to add supply tables.
   3. On the (Add DB2 Supply Desk to CAB File panel), enter a wildcard worth (%) or the particular desk identify DEPT within the (Desk Identify) area.
   4. Press Enter to show the checklist of obtainable tables.
   5. Kind S subsequent to the DEPT desk to pick out it for replication, then press Enter to substantiate.

This course of is just like the desk choice course of proven in determine 3 and determine 4 of our earlier publish however now focuses particularly on the DEPT desk construction.

With the completion of each the Db2 Seize/Writer setup on the mainframe and the AWS atmosphere configuration (Amazon MSK, Apply Engine, and MSK Join JDBC Sink Connector), you now have a totally useful pipeline able to seize knowledge adjustments from the mainframe and stream them to the MSK matter. Inserts, updates, or deletions to the DEPT desk on the mainframe shall be robotically captured and pushed to the MSK matter in close to actual time. From there, the MSK Join JDBC Sink Connector and the {custom} SMT will course of these messages and apply the adjustments to the PostgreSQL database on Amazon RDS, finishing the end-to-end replication move.

Configure Apply Engine for Amazon MSK integration

Configure the AWS facet elements to obtain knowledge from the mainframe and ahead it to Amazon MSK. Comply with these steps to outline and handle a brand new CDC pipeline from DB2 z/OS to Amazon MSK:

1. Use the next command to modify to the join consumer:
2. Create the apply engine directories:

   mkdir -p $SQDATA_VAR_DIR/apply/DB2ZTOMSK/ddl
        join> mkdir -p $SQDATA_VAR_DIR/apply/DB2ZTOMSK/scripts

3. Copy the pattern script from dept.ddl:

   cp $SQDATA_VAR_DIR/templates/dept.ddl $SQDATA_VAR_DIR/apply/DB2ZTOMSK/ddl/

4. Copy the next content material and paste it in a brand new file $SQDATA_VAR_DIR/apply/DB2ZTOMSK/scripts/DB2ZTOMSK.sqd. Substitute the placeholder strings with values that correspond to the DB2z endpoint:

   -----------------------------------------------------------------------
      Identify: DB2TOKAF: Z/OS DB2 To Kafka
      -----------------------------------------------------------------------
      SUBSTITUTION PARMS USED IN THIS SCRIPT:
      ---------------------------------------------------------------------
      JOBNAME DB2TOKAFKA;
      -----------------------------
      TABLE DESCRIPTIONS
      ---------------------------
      BEGIN GROUP SOURCE_TABLES;
      DESCRIPTION Db2SQL /var/exactly/di/sqdata/apply/DB2ZTOMSK/ddl/dept.ddl AS DEPT KEY IS DEPTNO;
      END GROUP;
      -------------------------------------------------------------
      DATASTORE SECTION
      -------------------------------------------------------------
      SOURCE DATASTORE
      DATASTORE cdc://<DB2z endpoint with port>/dbcg/DBCG_TBTSS388T6 OF UTSCDC AS CDCIN DESCRIBED BY GROUP SOURCE_TABLES;
      -- TARGET DATASTORE
      DATASTORE kafka:///pgsql-sink-topic/table_key OF JSON AS TARGET KEY IS DEPTNO DESCRIBED BY GROUP SOURCE_TABLES;
      ---------------------------------
      PROCESS INTO TARGET
      SELECT { REPLICATE(TARGET) } FROM CDCIN;

5. Create the working listing:

   mkdir -p /var/exactly/di/sqdata_logs/apply/DB2ZTOMSK

6. Add the next to $SQDATA_DAEMON_DIR/cfg/sqdagents.cfg:

   [DB2ZTOMSK]
      kind=engine
      program=sqdata
      args=/var/exactly/di/sqdata/apply/DB2ZTOMSK/scripts/DB2ZTOMSK.prc --log-level=8
      working_directory=/var/exactly/di/sqdata_logs/apply/DB2ZTOMSK
      stdout_file=stdout.txt
      stderr_file=stderr.txt
      auto_start=0
      remark=Apply Engine for MSK from Db2z

7. After the previous code is added to the sqdagents.cfg part, reload for the adjustments to take impact:
8. Validate the apply engine job script through the use of the SQData parse command to create the compiled file anticipated by the SQData engine:

   sqdparse $SQDATA_VAR_DIR/apply/DB2ZTOMSK/scripts/DB2ZTOMSK.sqd $SQDATA_VAR_DIR/apply/DB2ZTOMSK/scripts/DB2ZTOMSK.prc

   The next is an instance of the output that you simply get while you invoke the command efficiently:

   SQDC042I mounting/working sqdparse with arguments:
   SQDC041I args[0]:sqdparse
   SQDC041I args[1]:/var/exactly/di/sqdata/apply/DB2ZTOMSK/scripts/DB2ZTOMSK.sqd
   SQDC041I args[2]:/var/exactly/di/sqdata/apply/DB2ZTOMSK/scripts/DB2ZTOMSK.prc
   SQDC000I *******************************************************
   SQDC021I sqdparse Model 5.0.1-rel (Linux-x86_64)
   SQDC022I Construct-id 4f2d7c16728aa2e40c610db7d5a6e373476a9889
   SQDC023I (c) 2001, 2025 Syncsort Integrated. All rights reserved.
   SQDC000I *******************************************************
   SQDC000I
   SQD0000I 2025-03-31 00:59:10
   >>> Begin Preprocessed /var/exactly/di/sqdata/apply/DB2ZTOMSK/scripts/DB2ZTOMSK.sqd
   000001 ----------------------------------------------------------------------
   000002 -- Identify: DB2TOKAF:  Z/OS DB2 To Kafka
   000003 ----------------------------------------------------------------------
   000004 --  SUBSTITUTION PARMS USED IN THIS SCRIPT:
   000005 ----------------------------------------------------------------------
   000006
   000007 JOBNAME DB2TOKAFKA;
   000008
   000009 ----------------------------
   000010 -- TABLE DESCRIPTIONS
   000011 ----------------------------
   000012 BEGIN GROUP SOURCE_TABLES;
   000013 DESCRIPTION Db2SQL /var/exactly/di/sqdata/apply/DB2ZTOMSK/ddl/dept.ddl  AS DEPT
   000014 KEY IS DEPTNO;
   000015 END GROUP;
   000016
   000017 ------------------------------------------------------------
   000018 --       DATASTORE SECTION
   000019 ------------------------------------------------------------
   000020
   000021 -- SOURCE DATASTORE
   000022 DATASTORE /var/exactly/di/sqdata/apply/DB2ZTOMSK/scripts/DB0A.ENGINE3.DEPT.COPY
   000023           OF UTSCDC
   000024           AS CDCIN
   000025           DESCRIBED BY GROUP SOURCE_TABLES;
   000026
   000027 -- TARGET DATASTORE
   000028 DATASTORE 
   000029           OF JSON
   000030           AS TARGET
   000031           KEY IS DEPTNO
   000032           DESCRIBED BY GROUP SOURCE_TABLES;
   000033
   000034 ----------------------------------
   000035
   000036 PROCESS INTO TARGET
   000037 SELECT
   000038 {
   000039     REPLICATE(TARGET)
   000040 }
   000041 FROM CDCIN;
   <<< Finish Preprocessed /var/exactly/di/sqdata/apply/DB2ZTOMSK/scripts/DB2ZTOMSK.sqd
   >>> Begin Preprocessed /var/exactly/di/sqdata/apply/DB2ZTOMSK/ddl/dept.ddl
   000001 CREATE TABLE DEPARTMENT
   000002 (
   000003    DEPTNO char(3) NOT NULL,
   000004    DEPTNAME varchar(36) NOT NULL,
   000005    MGRNO char(6),
   000006    ADMRDEPT char(3) NOT NULL,
   000007    LOCATION char(16),
   000008    CONSTRAINT PK_DEPTNO PRIMARY KEY (DEPTNO)
   000009 ) ;
   <<< Finish Preprocessed /var/exactly/di/sqdata/apply/DB2ZTOMSK/ddl/dept.ddl
   Variety of Knowledge Shops...................: 2
   Knowledge Retailer..............................: /var/exactly/di/sqdata/apply/DB2ZTOMSK/scripts/DB0A.ENGINE3.DEPT.COPY
     Alias.................................: CDCIN
     Kind..................................: UTS Change Knowledge Seize
     Variety of Information.....................: 1
       Report Identify.........................: DEPARTMENT
       Report Description Alias............: DEPT
       Report Description Size...........: 72
       Variety of Fields....................: 5
         ................................... TYPE            OFF   LEN   XLEN  EXT
         ................................... ---------- ----- ----- ----- -----
         DEPTNO............................: CHAR(3)             0     3     3
         DEPTNAME..........................: VARCHAR(36)         3    38    38
         MGRNO.............................: CHAR(6)             7     6     6
         ADMRDEPT..........................: CHAR(3)            14     3     3
         LOCATION..........................: CHAR(16)           17    16    16
   Knowledge Retailer..............................: 
     Alias.................................: TARGET
     Kind..................................: JSON
     Variety of Information.....................: 1
       Report Identify.........................: DEPARTMENT
       Report Description Alias............: DEPT
       Report Description Size...........: 70
       Variety of Fields....................: 5
         ................................... TYPE            OFF   LEN   XLEN  EXT
         ................................... ---------- ----- ----- ----- -----
         DEPTNO............................: CHAR(3)             0     3     3
         DEPTNAME..........................: VARCHAR(36)         3    38    38
         MGRNO.............................: CHAR(6)            41     6     6
         ADMRDEPT..........................: CHAR(3)            47     3     3
         LOCATION..........................: CHAR(16)           50    16    16
   Part.................................: SQDSTP000
     Variety of steps.......................: 1
   SQDC017I sqdparse(pid=4023) terminated efficiently

9. Copy the next content material and paste it in a brand new file /var/exactly/di/sqdata_logs/apply/DB2ZTOMSK/sqdata_kafka_producer.conf. Substitute the placeholder strings with values that correspond to your bootstrap server and AWS Area.

   metadata.dealer.checklist=<kafka_bootstrap_servers_with_ports>
        safety.protocol=SASL_SSL
        sasl.mechanism=OAUTHBEARER
        sasl.oauthbearer.config="extension_AWSMSKCB=python3,/usr/lib64/python3.9/site-packages/aws_msk_iam_sasl_signer/cli.py,--region,<area>"
        sasl.oauthbearer.technique="default"

10. Begin the apply engine utilizing the controller daemon through the use of the next command:

    sqdmon begin ///DB2ZTOMSK

11. Monitor the apply engine by the controller daemon through the use of the next command:

    sqdmon show ///DB2ZTOMSK --format=particulars

    The next is an instance of the output that you simply get while you invoke the command efficiently:

    Engine..................................: DB2ZTOMSK
    model.................................: 5.0.1-rel (Linux-x86_64)
    git.....................................: f021c29a84c1a99f59144288aeeb2cb8fa494485
    jobname.................................: DB2TOKAFKA
    parsed..................................: 20250320172610278108
    began.................................: 2025-03-20.17.47.23.444474
    began (UTC)...........................: 2025-03-20.17.47.23.444474 (1742492843444)
    up to date (UTC)...........................: 2025-03-20.17.47.25.901018 (1742492845901)
    Enter Datastore.........................: /var/exactly/di/sqdata/apply/DB2ZTOMSK/scripts/DB0A.ENGINE3.DEPT.COPY
    Alias...................................: CDCIN
    Kind....................................: UTS Change Knowledge Seize
      Information Learn..........................: 14
      Information Chosen......................: 14
      Bytes Learn............................: 2892
    Output Datastore........................: kafka:///pgsql-sink-topic/table_key
    Alias...................................: TARGET
    Kind....................................: JSON
      Information Inserted......................: 14
      Information Up to date.......................: 0
      Information Deleted.......................: 0
      Formatted bytes.......................: 3458
      Unformatted bytes.....................: 448
    Complete Output Formatted bytes............: 3458
    Complete Output Unformatted bytes..........: 448
    SQDC017I sqdmon(pid=123540) terminated efficiently

    Logs can be discovered at /var/exactly/di/sqdata_logs/apply/DB2ZTOMSK.

Confirm knowledge within the MSK matter

Invoke the Kafka CLI command to confirm the JSON knowledge within the MSK matter:

kafka/bin/kafka-console-consumer.sh --bootstrap-server $BOOTSTRAP_SERVERS --consumer.config kafka/config/client-config.properties --topic pgsql-sink-topic --from-beginning --property print.key=true

Confirm knowledge within the PostgreSQL database

Invoke the next command to confirm the information within the PostgreSQL database:

PGPASSWORD="password" psql --host=<DATABASE-HOST> --username=<consumer> --dbname=<database> -c "choose * from "DEPT""

With these steps accomplished, you’ve efficiently arrange end-to-end knowledge replication from DB2z to RDS for PostgreSQL, utilizing AWS Mainframe Modernization – Knowledge Replication for IBM z/OS AMI, Amazon MSK, MSK Join, and the Confluent JDBC Sink Connector.

Cleanup

Once you’re completed testing this resolution, you may clear up the sources to keep away from incurring further expenses. Comply with these steps in sequence to make sure correct cleanup.

Step 1: Delete the MSK Join elements

Comply with these steps:

1. Checklist current connectors:

   aws kafkaconnect list-connectors

2. Delete the sink connector:

   aws kafkaconnect delete-connector --connector-arn "<arn-of-connector>"

3. Checklist {custom} plugins:

   aws kafkaconnect list-custom-plugins

4. Delete the {custom} plugin:

   aws kafkaconnect delete-custom-plugin --custom-plugin-arn "<arn-of-custom-plugin>"

Step 2: Delete the MSK cluster

Comply with these steps:

1. Checklist MSK clusters:

   aws kafka list-clusters-v2 --cluster-type-filter SERVERLESS

2. Delete the MSK serverless cluster:

   aws kafka delete-cluster --cluster-arn "<arn-of-msk-serverless-cluster>"

Step 3: Delete the Aurora sources

Comply with these steps:

1. Delete the Aurora DB occasion:

   aws rds delete-db-instance --db-instance-identifier cdc-serverless-pg-instance --skip-final-snapshot

2. Delete the Aurora DB cluster:

   aws rds delete-db-cluster --db-cluster-identifier cdc-serverless-pg-cluster --skip-final-snapshot.

Conclusion

By capturing modified knowledge from DB2z and streaming it to AWS targets, organizations can modernize their legacy mainframe knowledge shops, enabling operational insights and AI initiatives. Companies can use this resolution to reap the benefits of cloud-based purposes with mainframe knowledge to offer scalability, cost-efficiency, and enhanced efficiency.

The combination of AWS Mainframe Modernization – Knowledge Replication for IBM z/OS AMI with Amazon MSK and RDS for PostgreSQL offers an enhanced framework for real-time knowledge synchronization that maintains knowledge integrity. This structure could be prolonged to assist further mainframe knowledge sources akin to VSAM and IMS, in addition to different AWS targets. Organizations can then tailor their knowledge integration technique to particular enterprise wants. Knowledge consistency and latency challenges could be successfully managed by AWS and Exactly’s monitoring capabilities. By adopting this structure, organizations hold their mainframe knowledge regularly out there for analytics, machine studying (ML), and different superior purposes.Streaming mainframe knowledge to AWS in close to actual time represents a strategic step towards modernizing legacy programs whereas unlocking new alternatives for innovation, with knowledge transfers occurring in subseconds. With Exactly and AWS, organizations can successfully navigate their modernization journey and keep their aggressive benefit.

Study extra about AWS Mainframe Modernization – Knowledge Replication for IBM z/OS AMI within the Exactly documentation. AWS Mainframe Modernization Knowledge Replication is offered for buy in AWS Market. For extra details about the answer or to see an illustration, contact Exactly.

In regards to the authors