Amazon OpenSearch Ingestion 101: Set CloudWatch alarms for key metrics

Amazon OpenSearch Ingestion is a totally managed, serverless knowledge pipeline that simplifies the method of ingesting knowledge into Amazon OpenSearch Service and OpenSearch Serverless collections. Some key ideas embody:

• Supply – Enter element that specifies how the pipeline ingests the information. Every pipeline has a single supply which will be both push-based and pull-based.
• Processors – Intermediate processing models that may filter, remodel, and enrich data earlier than supply.
• Sink – Output element that specifies the vacation spot(s) to which the pipeline publishes knowledge. It might publish data to a number of locations.
• Buffer – It’s the layer between the supply and the sink. It serves as non permanent storage for occasions, decoupling the supply from the downstream processors and sinks. Amazon OpenSearch Ingestion additionally affords a persistent buffer possibility for push-based sources
• Useless-letter queues (DLQs) – Configures Amazon Easy Storage Service (Amazon S3) to seize data that fail to put in writing to the sink, enabling error dealing with and troubleshooting.

This end-to-end knowledge ingestion service can assist you gather, course of, and ship knowledge to your OpenSearch environments with out the necessity to handle underlying infrastructure.

This put up supplies an in-depth have a look at establishing Amazon CloudWatch alarms for OpenSearch Ingestion pipelines. It goes past our really useful alarms to assist determine bottlenecks within the pipeline, whether or not that’s within the sink, the OpenSearch clusters knowledge is being despatched to, the processors, or the pipeline not pulling or accepting sufficient from the supply. This put up will make it easier to proactively monitor and troubleshoot your OpenSearch Ingestion pipelines.

Overview

Monitoring your OpenSearch Ingestion pipelines is essential for catching and addressing points early. By understanding the important thing metrics and establishing the proper alarms, you’ll be able to proactively handle the well being and efficiency of your knowledge ingestion workflows. Within the following sections, we offer particulars about alarm metrics for various sources, displays, and sinks. The precise values for the edge, interval, and datapoints to alarm used for alarms can fluctuate primarily based on the person use case and necessities.

Conditions

To create an OpenSearch Ingestion pipeline, consult with Creating Amazon OpenSearch Ingestion pipelines. For creating CloudWatch alarms, consult with Create a CloudWatch alarm primarily based on a static threshold.

You possibly can allow logging for OpenSearch Ingestion Pipeline, which captures varied log messages throughout pipeline operations and ingestion exercise, together with errors, warnings, and informational messages. For particulars on enabling and monitoring pipeline logs, consult with Monitoring pipeline logs

Sources

The entry level of your pipeline is commonly the place monitoring ought to start. By setting acceptable alarms for supply parts, you’ll be able to shortly determine ingestion bottlenecks or connection points. The next desk summarizes key alarm metrics for various sources.

Processors

The next desk supplies particulars about alarm metrics for various processors.

Sinks and DLQs

The next desk incorporates particulars about alarm metrics for various sinks and DLQs.

Buffer

The next desk incorporates particulars about alarm metrics for buffers.

Reference Information

The next present steerage for resolving widespread pipeline points together with common reference.

REF-001: WARN-level Log Evaluate

Evaluate WARN-level logs within the pipeline logs to determine the exception particulars.

REF-002: ERROR-level Log Evaluate

Evaluate ERROR-level logs within the pipeline logs to determine the exception particulars.

REF-003: S3 Objects Failed

When troubleshooting growing s3ObjectsFailed.rely values, monitor these particular metrics to slim down the basis trigger:

• s3ObjectsAccessDenied.rely – This metric increments when the pipeline encounters Entry Denied or Forbidden errors whereas studying S3 objects. Widespread causes embody:
• Inadequate permissions within the pipeline function.
• Restrictive S3 bucket coverage not permitting the pipeline function entry.
• For cross-account S3 buckets, incorrectly configured bucket_owners mapping.
• s3ObjectsNotFound.rely – This metric increments when the pipeline receives Not Discovered errors whereas making an attempt to learn S3 objects.

For additional help with the really useful actions, contact AWS assist.

REF-004: Configuring Alarm for distinction in totalOpenShards.max and activeShardsInProcessing.worth for Amazon DynamoDB supply.

1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
2. Within the navigation pane, select Alarms, All alarms.
3. Select Create alarm.
4. Select Choose Metric.
5. Choose Supply.
6. In supply, following JSON can be utilized after updating the <sub-pipeline-name>, <pipeline-name> and <area>.

   {
       "metrics": [
           [ { "expression": "m1-e1", "label": "Expression2", "id": "e2", "period": 900 } ],
           [ { "expression": "FLOOR((m2/15)+0.5)", "label": "Expression1", "id": "activeShardsInProcessing", "visible": false, "period": 900 } ],
           [ "AWS/OSIS", "<sub-pipeline-name>.dynamodb.totalOpenShards.max", "PipelineName", "<pipeline-name>", { "stat": "Maximum", "id": "m1", "visible": false } ],
           [ ".", "<sub-pipeline name>.dynamodb.activeShardsInProcessing.value", ".", ".", { "stat": "Average", "id": "m2", "visible": false } ]
       ],
       "view": "timeSeries",
       "stacked": false,
       "interval": 900,
       "area": "<area>"
   }

Let’s overview couple of situations primarily based on the above metrics.

State of affairs 1 – Perceive and Decrease Pipeline Latency

Latency inside a pipeline is constructed up of three essential parts:

• The time it takes to ship paperwork by way of bulk requests to OpenSearch,
• the time it takes for knowledge to undergo the pipeline processors, and
• the time that knowledge sits within the pipeline buffer

Bulk requests and processors (final two objects within the earlier listing) are the basis causes for why the buffer builds up and results in latency.

To observe how a lot knowledge is being saved within the buffer, monitor the bufferUsage.worth metric. The one technique to decrease latency throughout the buffer is to optimize the pipeline processors and sink bulk request latency, relying on which of these is the bottleneck.

The bulkRequestLatency metric measures the time taken to execute bulk requests, together with retries, and can be utilized to observe write efficiency to the OpenSearch sink. If this metric experiences an unusually excessive worth, it signifies that the OpenSearch sink could also be overloaded, inflicting elevated processing time. To troubleshoot additional, overview the bulkRequestNumberOfRetries.rely metric to substantiate whether or not the excessive latency is because of rejections from OpenSearch which are resulting in retries, reminiscent of throttling (429 errors) or different causes. If doc errors are current, look at the configured DLQ to determine the failed doc particulars. Moreover, the max_retries parameter will be configured within the pipeline configuration to restrict the variety of retries. Nonetheless, if the documentErrors metric experiences zero, the bulkRequestNumberOfRetries.rely can be zero, and the bulkRequestLatency stays excessive, it’s probably an indicator that the OpenSearch sink is overloaded. On this case, overview the vacation spot metrics for extra particulars.

If the bulkRequestLatency metric is low (for instance, lower than 1.5 seconds) and the bulkRequestNumberOfRetries metric is reported as 0, then the bottleneck is probably going throughout the pipeline processors. To observe the efficiency of the processors, overview the <processorName>.timeElapsed.avg metric. This metric experiences the time taken for the processor to finish processing of a batch of data. For instance, if a grok processor is reporting a a lot greater worth than different processors for timeElapsed, it could be because of a gradual grok sample that may be optimized and even changed with a extra performant processor, relying on the use case.

State of affairs 2 – Understanding and Resolving Doc Errors to OpenSearch

The documentErrors.rely metric tracks the variety of paperwork that did not be despatched by bulk requests. The failure can occur because of varied causes reminiscent of mapping conflicts, invalid knowledge codecs, or schema mismatches. When this metric experiences a non-zero worth, it signifies that some paperwork are being rejected by OpenSearch. To determine the basis trigger, look at the configured Useless Letter Queue (DLQ), which captures the failed paperwork together with error particulars. The DLQ supplies details about why particular paperwork failed, enabling you to determine patterns reminiscent of incorrect area varieties, lacking required fields, or knowledge that exceeds dimension limits. For instance, discover the pattern DLQ objects for widespread points under:

Mapper parsing exception:

{"dlqObjects": [{
        "pluginId": "opensearch",
        "pluginName": "opensearch",
        "pipelineName": "<PipelineName>",
        "failedData": {
            "index": "<IndexName>",
            "indexId": null,
            "status": 400,
            "message": "failed to parse field [<fieldname>] of kind [integer] in doc with id '<DocumentId>'. Preview of area's worth: 'N/A' attributable to For enter string: "N/A"",
            "doc": {<OriginalDocument>}
        },
        "timestamp": "…"
    }]}

Right here, OpenSearch can not retailer the textual content string “N/A” in a area that’s just for numbers, so it rejects the doc and shops it within the DLQ.

Restrict of whole fields exceeded:

{"dlqObjects": [{
        "pluginId": "opensearch",
        "pluginName": "opensearch",
        "pipelineName": "<PipelineName>",
        "failedData": {
            "index": "<IndexName>",
            "indexId": null,
            "status": 400,
            "message": "Limit of total fields [<field limit>] has been exceeded",
            "doc": {<OriginalDocument>}
        },
        "timestamp": "…"
    }]}

The index.mapping.total_fields.restrict setting is the parameter that controls the utmost variety of fields allowed in an index mapping, and exceeding this restrict will trigger indexing operations to fail. You possibly can examine if all these fields are required or leverage varied processors offered by OpenSearch Ingestion to rework the information.

As soon as these points are recognized, you’ll be able to both right the supply knowledge, modify the pipeline configuration to rework the information appropriately, or modify the OpenSearch index mapping to accommodate the incoming knowledge format.

Clear up

When establishing alarms for monitoring your OpenSearch Ingestion pipelines, it’s vital to be aware of the potential prices concerned. Every alarm you configure will incur expenses primarily based on the CloudWatch pricing mannequin.

To keep away from pointless bills, we advocate rigorously evaluating your alarm necessities and configuring them accordingly. Solely arrange the alarms which are important in your use case, and recurrently overview your alarm configurations to determine and take away unused or redundant alarms.

Conclusion

On this put up, we explored the great monitoring capabilities for OpenSearch Ingestion pipelines via CloudWatch alarms, protecting key metrics throughout varied sources, processors, and sinks. Though this put up highlights probably the most important metrics, there’s extra to find. For a deeper dive, consult with the next sources:

Efficient monitoring via CloudWatch alarms is essential for sustaining wholesome ingestion pipelines and sustaining optimum knowledge move.

In regards to the authors