Migrate to Apache Flink 2.2 on Amazon Managed Service for Apache Flink

Migrating to Apache Flink 2.2 on Amazon Managed Service for Apache Flink offers you entry to Java 17 runtime, quicker checkpoints and restoration by means of RocksDB 8.10.0, and SQL-native synthetic intelligence and machine studying (AI/ML) inference. If you happen to run Flink 1.x as we speak, you could be coping with an getting old Java 11 runtime that can now not obtain customary assist by the top of this yr, slower state backend efficiency, and a fragmented API floor cut up throughout DataSet, DataStream, and legacy connector interfaces. Flink 2.2 addresses these gaps in a single main model improve.

Apache Flink is an open supply distributed processing engine for stream and batch information, with first-class assist for stateful processing and event-time semantics. Amazon Managed Service for Apache Flink removes the operational overhead of operating Flink. You present your software code, and the service provisions, scales, checkpoints, and patches the infrastructure for you.

On this publish, we clarify what’s new in Amazon Managed Service for Apache Flink 2.2, present a guided migration utilizing CLI instructions, console directions, and code examples, and present you find out how to monitor the improve and roll again if wanted.

Earlier than you improve: Flink 2.2 removes the DataSet API, drops Java 11 assist, and replaces legacy connector interfaces. We suggest reviewing the Upgrading to Flink 2.2: Full Information and the State Compatibility Information for Flink 2.2 Improves earlier than upgrading manufacturing functions.

What’s new in Amazon Managed Service for Apache Flink 2.2

This launch spans runtime upgrades, SQL, and Desk API capabilities. The next sections break down every space.

Runtime and efficiency

These modifications enhance software efficiency and produce your runtime as much as present requirements.

• Java 17 runtime – Flink 2.2 requires Java 17. Construct your software code with JDK 17 for higher rubbish assortment, a safer runtime, and trendy language options like sealed lessons and information. Java 11 is now not supported.
• Python 3.12 – Flink 2.2 requires Python 3.9+, with Python 3.12 because the default. Python 3.8 is now not supported.
• RocksDB 8.10.0 – Your stateful functions profit from improved I/O efficiency with the upgraded state backend, leading to quicker checkpoints and restoration.
• Devoted assortment serializers – Improved serializers for Map, Listing, and Set varieties cut back serialization overhead, which lowers checkpoint sizes for functions that use these information constructions steadily.
• Kryo 5.6 – Kryo upgrades from model 2.24–5.6. This has state compatibility implications lined within the migration part.

SQL and Desk API highlights

With Flink 2.2, you may:

For particulars on these options, see the Apache Flink 2.2 launch documentation.

Migrating from Flink 1.x to 2.2

In-place model upgrades

You’ll be able to improve a operating Flink 1.x software to 2.2 utilizing the UpdateApplication API, the AWS Administration Console, AWS CloudFormation, the AWS SDK, and Terraform Modules. The improve preserves your software configuration, logs, metrics, tags, and, in case your state and binaries are suitable.

Auto-rollback

With auto-rollback turned on, binary incompatibilities detected throughout job startup set off an computerized revert to the earlier Flink model inside minutes, with no handbook intervention required. For state incompatibilities that floor as restart loops after a profitable improve, invoke the Rollback API to return to your earlier model and state.

Unsupported open supply options

The next Flink 2.2 options aren’t presently supported in Amazon Managed Service for Apache Flink as a result of they’re nonetheless thought of experimental: Materialized Tables, ForSt State Backend (disaggregated state storage), Java 21, and customized metric reporters/telemetry configurations. We proceed to guage these options as they mature within the Apache Flink venture and can share updates on availability. You’ll be able to have a better look to which options are supported in Apache Flink 2.2 options supported

Now that you already know what’s modified, the subsequent part walks by means of the migration course of.

Stipulations

Earlier than beginning the migration, affirm that you’ve got the next in place:

• An current Apache Flink 1.x software operating on Amazon Managed Service for Apache Flink.
• JDK 17 put in in your native construct setting.
• The AWS Command Line Interface (AWS CLI) put in and configured with permissions to name the kinesisanalyticsv2 APIs (UpdateApplication, CreateApplicationSnapshot, DescribeApplication, RollbackApplication).
• An Amazon Easy Storage Service (Amazon S3) bucket to add your up to date software JAR.

We suggest testing every part on a non-production duplicate of your software earlier than making use of the identical steps to manufacturing.

Step 1: Replace your software code

Begin by updating your Flink dependencies to model 2.2.0 and changing deprecated APIs. The next sections present the commonest modifications.

Replace your pom.xml:

<properties>
    <flink.model>2.2.0</flink.model>
    <java.model>17</java.model>
</properties>

Substitute legacy Kinesis connectors:

Flink 2.2 removes the FlinkKinesisConsumer and FlinkKinesisProducer lessons. The next instance exhibits find out how to migrate to the FLIP-27 based mostly KinesisStreamsSource.Earlier than (Flink 1.x):

FlinkKinesisConsumer<String> client = new FlinkKinesisConsumer<>(
    "my-stream",
    new SimpleStringSchema(),
    consumerConfig);
env.addSource(client);

After (Flink 2.2):

KinesisStreamsSource<String> supply = KinesisStreamsSource.<String>builder()
    .setStreamArn("arn:aws:kinesis:us-east-1:123456789012:stream/my-stream")
    .setDeserializationSchema(new SimpleStringSchema())
    .construct();
env.fromSource(supply, WatermarkStrategy.noWatermarks(), "Kinesis Supply");

Replace connector dependencies:

The next AWS connectors have Flink 2.x-compatible releases:

Throughout writing, the JDBC, OpenSearch, and Prometheus connectors don’t but have Flink 2.x-compatible releases. For the newest variations, see the Amazon Managed Service for Apache Flink connector documentation.

Past connector updates, make the next code modifications:

• Substitute DataSet API utilization with the DataStream API or Desk API/SQL.
• Substitute Scala API utilization with the Java API.
• Confirm that your construct targets JDK 17.

Construct your up to date software JAR and add it to Amazon S3 with a special file identify than your present JAR (for instance, my-app-flink-2.2.jar).

Step 2: Test state compatibility

Earlier than upgrading, assess whether or not your software state is suitable with Flink 2.2. The Kryo improve from model 2.24 to five.6 modifications the binary format of serialized state. Purposes utilizing POJOs with Java collections (HashMap, ArrayList, HashSet) are the commonest supply of incompatibility.

Fast compatibility verify:

Test your logs for Kryo fallback:

Search your software logs for this sample, which signifies a kind is falling again to Kryo serialization:Class class <className> can't be used as a POJO sort

Step 3: Activate auto-rollback and computerized snapshots

Activate auto-rollback so the service routinely reverts to the earlier model if the improve fails. Additionally, confirm that computerized snapshots are turned on. The service takes a snapshot earlier than the improve that serves as your rollback level.

Test present settings:

aws kinesisanalyticsv2 describe-application 
    --application-name MyApplication 
    --query 'ApplicationDetail.ApplicationConfigurationDescription.{
        AutoRollback: ApplicationSystemRollbackConfigurationDescription.RollbackEnabled,
        AutoSnapshots: ApplicationSnapshotConfigurationDescription.SnapshotsEnabled
    }'

Activate each in the event that they’re not already energetic:

aws kinesisanalyticsv2 update-application 
    --application-name MyApplication 
    --current-application-version-id <version-id> 
    --application-configuration-update '{
        "ApplicationSystemRollbackConfigurationUpdate": {
            "RollbackEnabledUpdate": true
        },
        "ApplicationSnapshotConfigurationUpdate": {
            "SnapshotsEnabledUpdate": true
        }
    }'

Step 4: Take a handbook snapshot (really useful)

Though the improve course of takes an computerized snapshot, taking a handbook snapshot offers you a named restore level you could rapidly determine.

aws kinesisanalyticsv2 create-application-snapshot 
    --application-name MyApplication 
    --snapshot-name pre-flink-2.2-upgrade

Confirm that the snapshot is prepared earlier than continuing:

aws kinesisanalyticsv2 describe-application-snapshot 
    --application-name MyApplication 
    --snapshot-name pre-flink-2.2-upgrade

Wait till SnapshotStatus is READY.

Step 5: Run the improve

Run the improve whereas the appliance is in RUNNING or READY (stopped) state. The next instance upgrades a operating software and factors to the brand new JAR.

AWS CLI:

aws kinesisanalyticsv2 update-application 
    --application-name MyApplication 
    --current-application-version-id <version-id> 
    --runtime-environment-update FLINK-2_2 
    --application-configuration-update '{
        "ApplicationCodeConfigurationUpdate": {
            "CodeContentUpdate": {
                "S3ContentLocationUpdate": {
                    "FileKeyUpdate": "my-app-flink-2.2.jar"
                }
            }
        }
    }'

AWS Administration Console:

To improve from the console, comply with these steps:

1. Navigate to your software within the Amazon Managed Service for Apache Flink console.
2. Select Configure.
3. Choose the Flink 2.2 runtime.
4. Level to your new software JAR on Amazon S3.
5. Choose the snapshot to revive from (use Newest to start out from the latest snapshot).
6. Select Replace.

AWS CloudFormation:

Replace the RuntimeEnvironment discipline in your template. AWS CloudFormation now performs an in-place replace as a substitute of deleting and recreating the appliance.

Terraform:

If you happen to handle your Flink software with Terraform, you may carry out the identical in-place improve by updating the runtime_environment and code reference in your aws_kinesisanalyticsv2_application useful resource. Be aware: Terraform assist for FLINK-2_2 requires AWS supplier model 6.40.0 or later (launched April 8, 2026). Earlier supplier variations don’t acknowledge this runtime worth. First, replace your supplier model constraint:

terraform {
  required_providers {
    aws = {
      supply  = "hashicorp/aws"
      model = ">= 6.40.0"
    }
  }
}

Then run terraform init -upgrade to tug the brand new supplier.Subsequent, replace your software useful resource. Change runtime_environment from “FLINK-1_20” to “FLINK-2_2” and level to your new JAR:

useful resource "aws_kinesisanalyticsv2_application" "my_app" {
  identify                   = "MyApplication"
  runtime_environment    = "FLINK-2_2"
  service_execution_role = aws_iam_role.flink.arn
  application_configuration {
    application_code_configuration {
      code_content_type = "ZIPFILE"
      code_content {
        s3_content_location {
          bucket_arn = aws_s3_bucket.app_code.arn
          file_key   = "my-app-flink-2.2.jar"
        }
      }
    }
    application_snapshot_configuration {
      snapshots_enabled = true
    }
    flink_application_configuration {
      checkpoint_configuration {
        configuration_type = "DEFAULT"
      }
      monitoring_configuration {
        configuration_type = "CUSTOM"
        log_level          = "INFO"
        metrics_level      = "APPLICATION"
      }
      parallelism_configuration {
        auto_scaling_enabled = true
        configuration_type   = "CUSTOM"
        parallelism          = 4
        parallelism_per_kpu  = 1
      }
    }
  }
}

Run the improve:

terraform plan    # Assessment the in-place replace
terraform apply   # Apply the runtime change

Terraform will carry out an in-place replace of the appliance, altering the runtime model and code location. The applying will restart with the brand new Flink 2.2 runtime. To roll again with Terraform, revert runtime_environment to “FLINK-1_20”, level file_key again to your authentic JAR, and run terraform apply once more. Be aware that you just can not restore a Flink 2.2 snapshot on Flink 1.x, so the rollback will begin from the final Flink 1.x snapshot.

Essential Terraform concerns:

• Auto-rollback and the RollbackApplication API aren’t straight uncovered as Terraform useful resource attributes. If you happen to want auto-rollback throughout the improve, allow it utilizing the AWS CLI (Step 3) earlier than operating terraform apply, or use a provisioner/null_resource to name the CLI.
• At all times take a handbook snapshot (Step 4) earlier than operating terraform apply for the improve. Terraform doesn’t routinely snapshot earlier than updating the runtime.

Step 6: Monitor the improve

After initiating the improve, monitor the appliance to confirm that it completes efficiently.

Test software standing:

The applying ought to transition by means of RUNNING → UPDATING → RUNNING. Affirm the runtime model modified to 2.2:

aws kinesisanalyticsv2 describe-application 
    --application-name MyApplication 
    --query 'ApplicationDetail.RuntimeEnvironment'

What to look at for:

Key CloudWatch metrics to watch:

1. numRestarts: must be zero after improve
2. lastCheckpointDuration: must be much like pre-upgrade values
3. numberOfFailedCheckpoints: ought to stay at zero
4. uptime: must be steadily growing

Step 7: Validate software conduct

After the appliance is operating on Flink 2.2:

• Affirm that information is being learn from sources and written to sinks.
• Examine the output along with your pre-upgrade baseline.
• Monitor latency, throughput, checkpoint period, and useful resource utilization.
• Run for at the least 24 hours to substantiate secure conduct: no reminiscence leaks, no surprising restarts, constant checkpoint sizes.

Step 8: Rollback (if wanted)

If the appliance is operating however is unhealthy after the improve, invoke the Rollback API:

AWS CLI:

aws kinesisanalyticsv2 rollback-application 
    --application-name MyApplication 
    --current-application-version-id <version-id>

AWS Administration Console:

• Navigate to your software.
• Select Actions, Roll again.
• Affirm the rollback.

Throughout rollback, the appliance stops, reverts to the earlier Flink model and software code, and restarts from the snapshot taken earlier than the improve.

Essential: You’ll be able to’t restore a Flink 2.2 snapshot on Flink 1.x. Rollback makes use of the snapshot taken earlier than the improve. For this reason Steps 3 and 4 are important.

Subsequent steps

Your path is determined by the place you might be as we speak:

1. If you happen to’re new to Apache Flink: Begin with the information to selecting the best API and language, the Amazon Managed Service for Apache Flink getting began information, and the Amazon Managed Service for Apache Flink workshop.
2. If you happen to’re operating Flink 1.x in manufacturing: Observe the migration steps on this publish on a non-production duplicate first, then apply to manufacturing. For the whole reference, see the Upgrading to Flink 2.2: Full Information and the State Compatibility Information for Flink 2.2 Upgrades.
3. If you happen to’re evaluating Flink 2.2 options: Launch a brand new software on the Flink 2.2 runtime to discover SQL/ML capabilities, the VARIANT information sort, and the brand new be a part of operators. See the Amazon Managed Service for Apache Flink pattern functions on GitHub for reference architectures.
4. If you happen to need assistance along with your migration: Use the Kiro Energy and Agent Ability for Amazon Managed Service for Apache Flink to determine compatibility points in your current codebase and obtain steerage on refactoring steps. You can even open a case by means of AWS Assist, publish a query on AWS re:Submit for Amazon Managed Service for Apache Flink, or attain out by means of the Apache Flink group.

For the Apache Flink 2.2 documentation, see nightlies.apache.org/flink/flink-docs-release-2.2. For Amazon Managed Service for Apache Flink documentation, see the Developer Information. For pricing, see the pricing web page.

Conclusion

With Apache Flink 2.2 on Amazon Managed Service for Apache Flink, you get a contemporary Java 17 runtime, SQL-native AI/ML inference, improved state administration efficiency, and a streamlined API floor. In-place upgrades with state preservation and auto-rollback make the migration easy. Check on a duplicate, comply with the steps on this publish, and begin constructing on Flink 2.2.

In regards to the authors