As information volumes proceed to develop exponentially, there may be rising stress to optimize search infrastructure prices whereas sustaining the excessive efficiency and reliability that mission-critical workloads demand. Many firms discover themselves managing advanced, costly search techniques that require important operational overhead and restrict their capacity to scale effectively. The problem turns into much more acute when organizations have to migrate between search techniques, a course of that historically includes substantial downtime, advanced information synchronization, and important affect on enterprise operations. Enterprise purposes can’t afford service interruptions that would affect buyer experiences, enterprise intelligence, or operational continuity. Migration methods have to ship price optimization and operational enhancements whereas sustaining zero downtime and facilitating full information integrity all through the transition course of.
Based in 2013, Octus, previously Reorg, is the important credit score intelligence and information supplier for the world’s main purchase facet corporations, funding banks, regulation corporations and advisory corporations. By surrounding unparalleled human experience with confirmed expertise, information and AI instruments, Octus unlocks highly effective truths that gas decisive motion throughout monetary industries.
This put up highlights how Octus migrated its Elasticsearch workloads working on Elastic Cloud to Amazon OpenSearch Service. The journey traces Octus’s shift from managing a number of techniques to adopting a cost-efficient answer powered by OpenSearch Service. Alongside the best way, we share the structure selections and implementation methods that made the migration profitable. The result’s uninterrupted service availability all through migration, with improved efficiency and larger price effectivity.
Strategic necessities
We recognized a number of necessities that made Amazon OpenSearch Service the fitting selection for his or her migration:
- Value effectivity: The OpenSearch Service pricing mannequin enabled us to optimize cloud spend with out compromising efficiency.
- Responsive help: AWS supplied reliable, high-quality help to speed up challenge decision and instill confidence.
- Constant reliability: OpenSearch Service offers an SLA as much as 99.99% providing the reliability required for Octus’s mission-critical workloads.
- Seamless migration with no question downtime: Migration Assistant for Amazon OpenSearch Service supplied Octus with a migration path whereas sustaining uninterrupted question availability in the course of the migration, facilitating enterprise continuity.
- Operational simplification: Consolidating onto AWS decreased infrastructure complexity whereas sustaining excessive safety requirements.
Answer overview
The Migration Assistant for Amazon OpenSearch Service offers a collection of instruments to help in Elasticsearch to OpenSearch Service migrations. Octus use the next capabilities for his or her migration:
- Metadata migration: The instrument enabled Octus emigrate dozens of indices with various mappings and settings. When a backward incompatibility was recognized with timestamp metadata, a customized JavaScript transformation, built-in instantly into the Migration Assistant tooling, was utilized to routinely modify the mappings throughout the indices and facilitate compatibility.
- Historic information migration: Octus used Reindex-from-Snapshot emigrate the historic paperwork from a point-in-time snapshot of the supply cluster, scaling this course of with out impacting the supply cluster because the snapshot was saved in Amazon Easy Storage Service (Amazon S3). Reindex-from-Snapshot additionally enabled Octus to regulate the sharding scheme throughout migration, serving to to optimize cluster efficiency on the goal.
- Stay Site visitors Replay: As soon as backfill was full, Octus used Migration Assistant’s Site visitors Replayer to ship the captured stay site visitors (from the Site visitors Seize Proxy) to the goal cluster with required request transformations for OpenSearch Service compatibility, ensuing within the goal cluster containing the paperwork from the supply cluster with updates being carried out in actual time.
The next diagram illustrates the implementation structure diagram for this migration.
Determine 1 – Migration Assistant structure with migration steps
For extra details about the Migration Assistant for Amazon OpenSearch Service, go to the AWS Options residence web page.
Every node within the diagram correlates to the next steps within the migration course of:
- Shopper site visitors is directed to the present cluster.
- An Utility Load Balancer with seize proxies relays site visitors to a supply whereas replicating information to Amazon Managed Streaming for Apache Kafka (Amazon MSK).
- Utilizing the migration console, a point-in-time snapshot is taken. As soon as the snapshot completes, the Metadata Migration Software is used to determine indexes, templates, part templates, and aliases on the goal cluster. With steady site visitors seize in place, Reindex-from-Snapshot, migrates information from the supply.
- As soon as Reindex-from-Snapshot is full, captured site visitors is replayed from Amazon Managed Streaming for Apache Kafka (Amazon MSK) to the goal cluster by Site visitors Replayer.
- Efficiency and conduct of site visitors despatched to the supply and goal clusters are in contrast by reviewing logs and metrics.
- After confirming that the goal cluster’s performance meets expectations, shoppers are redirected to the brand new goal.
Full migration and optimization journey
Octus’s migration from Elastic Cloud to Amazon OpenSearch Service encompassed each the core migration effort and subsequent optimization phases. The purpose was to efficiently migrate the search infrastructure, purposes, and information from Elastic Cloud to a brand new OpenSearch Service area with minimal disruption, whereas constantly optimizing efficiency and prices based mostly on real-world utilization information.
Octus used their in-house customized infrastructure frameworks (their inside tooling for infrastructure automation) to construct, deploy and monitor the goal OpenSearch Service 1.3 area, establishing a stable basis for the migration. This method used acquainted inside processes whereas shifting to the totally managed AWS service. Discuss with AWS documentation to implement safety finest practices when utilizing OpenSearch Service.
Pre-migration optimization
Previous to initiating the migration, Octus carried out optimization actions on the supply Elasticsearch cluster to streamline the migration course of. This included eradicating unused indexes that had amassed over time and eradicating massive paperwork that might unnecessarily lengthen migration length and enhance storage switch prices. These preparatory steps considerably decreased the info quantity requiring migration and minimized the general migration complexity, enabling extra environment friendly use of the Migration Assistant instruments.
Technical constraints and model concerns
The migration concerned particular model compatibility challenges that influenced the technical method. The supply Elasticsearch cluster was working model 7.17, and the Python shopper purposes have been additionally constrained to Elasticsearch 7.17 compatibility. To help the transition, the crew used Reindex-from-Snapshot, which allows cross-system migrations by reindexing information from current snapshots into a brand new OpenSearch Service cluster. RFS additionally rewrites indices created on older variations of Lucene, simplifying future upgrades to the newest model of OpenSearch Service. Whereas evaluating a transfer to OpenSearch 1 or 2, Octus chosen OpenSearch 1.3 because the goal to reduce client-side modifications and scale back migration complexity, whereas positioning themselves for less complicated upgrades later.
The model choice significantly impacted the R utility surroundings, as R language (an open-source programming language for statistical computing and information evaluation) lacked native OpenSearch 1.3 shopper help. This constraint required Octus to develop a customized shopper answer utilizing the ropensci/elastic library to combine with the brand new OpenSearch Service area. The Python surroundings introduced comparable challenges, the place the Elasticsearch 7.17 shopper constraints necessitated cautious consideration of the migration method. These shopper compatibility issues have been among the many elements that influenced the selection of Migration Assistant instruments over conventional snapshot-based strategies, because the Migration Assistant supplied higher help for managing version-specific shopper interactions in the course of the transition.
Wanting ahead, Octus plans to improve to newer OpenSearch variations as their utility stack evolves and shopper library help matures, in order that they will leverage the newest options and efficiency enhancements whereas sustaining the steadiness achieved by means of this migration.
Utility modernization throughout a number of languages
The applying modifications represented a major technical enterprise throughout a number of programming environments:
- Legacy PHP techniques (5.6 and Laravel 4.2): Octus dealt with mapping sort deprecation on OpenSearch requests as specifying these mapping varieties usually are not supported, whereas persevering with to make use of the elasticsearch connector library with username/password authentication.
- Fashionable PHP purposes (8.1 and Laravel 9): These underwent extra complete modifications, changing the elasticsearch/elasticsearch library with the opensearch-project/opensearch-php shopper and leveraging IAM authentication to hook up with the clusters.
- Python surroundings: Functions spanning variations 3.8, 3.10, 3.11, and three.13 with Django frameworks 2.1, 3.2, and 5.2 required changing the elasticsearch library with opensearch-py and transitioning to IAM authentication.
- R purposes: For R 4.5.1 purposes, Octus utilized a customized library ropensci/elastic to facilitate compatibility.
Site visitors routing and enhanced monitoring
To facilitate the migration, Octus redirected their current shoppers to route requests to the supply cluster by means of Migration Assistant’s Site visitors Seize Proxy, migrating the info from stay site visitors to their goal cluster.
The monitoring infrastructure underwent important enhancement throughout this course of. Octus’s observability infrastructure screens the general well being of OpenSearch Service clusters which incorporates cluster supervisor and information nodes, community, information storage, safety and IAM entry. It additionally screens the indexing and search efficiency of their purposes. This alleviated the necessity for a separate monitoring cluster as logs and metrics have been shipped on to Datadog, considerably bettering observability. The Datadog screens have been outlined utilizing Infrastructure-as-Code and built-in seamlessly into their infrastructure frameworks.
Cutover and preliminary outcomes
The Website Reliability Engineering crew meticulously deliberate the discharge, attaining a profitable migration from Elasticsearch to OpenSearch Service and cutover of the Elasticsearch shopper to the OpenSearch Service shoppers with no downtime for the system utility and nil information loss. The preliminary migration section resulted in a 52% price discount whereas attaining operational advantages together with zero downtime for the system app, no information loss, full Infrastructure-as-Code implementation for infrastructure and monitoring, and enhanced observability.
Submit-migration optimization
Following the migration, Octus carried out complete optimization based mostly on operational information from manufacturing and different environments within the new OpenSearch Service setup. This real-world utilization information supplied invaluable insights into precise useful resource consumption, enabling knowledgeable selections concerning additional cluster resizing.
By utilization metric evaluation and strategic resizing, Octus aligned cluster measurement extra exactly with operational wants, facilitating continued efficiency whereas minimizing expenditure. This optimization section delivered a further 33% price discount in comparison with the unique Elastic Cloud prices, bringing the whole discount to 85% whereas sustaining constant and optimum efficiency.
Operational monitoring
Octus makes use of Datadog to watch each search and indexing latency offering real-time visibility into Amazon OpenSearch Service cluster efficiency. The next screenshot showcases how customized Datadog dashboards present a stay view of the OpenSearch Service clusters. This visualization gives each a high-level overview and detailed insights into the ingestion course of, serving to us perceive the storage and doc depend. The underside half of the dashboard presents a time-series view of particular person node well being and efficiency metrics like learn and write latency, throughput and IOPS.
Determine 2 – DataDog dashboards
Migration observability
Migration Assistant for Amazon OpenSearch Service offers a number of dashboards to watch and validate the progress of a migration. Through the use of these observability options clients can monitor each backfill and stay seize and replay progress, facilitating confidence earlier than switching manufacturing workloads to the goal cluster.The next graphs are an instance from Octus’s migration, the place roughly 4TB of knowledge was migrated in about 9 hours (from 08:00 to 17:00).
Determine 3 – Backfill progress by disk utilization
Determine 4 – Backfill progress by searchable paperwork
As soon as the backfill is full, the captured site visitors is replayed to synchronize ongoing exercise between the supply and goal clusters.
On the time the backfill completed (round 17:00), the goal cluster was roughly 467 minutes behind the supply. The replay course of quickly decreased this lag by processing captured site visitors at a sooner charge than it was initially ingested on the supply.
Determine 5 – Replay lag after backfill completion
When the lag time reached 0, the goal cluster was totally in sync and manufacturing site visitors may safely be rerouted. Octus selected to watch replayed site visitors on the goal for a number of days earlier than making the ultimate switchover.
Attaining excellence
Octus’s migration to Amazon OpenSearch Service has yielded outstanding outcomes:
- Scalability – Octus has virtually doubled the variety of paperwork obtainable for Q&A throughout three environments in days as a substitute of weeks. Their use of Amazon Elastic Container Service (Amazon ECS) with AWS Fargate with auto scaling guidelines and controls provides them elastic scalability for his or her providers throughout peak utilization hours.
- Value discount – By shifting away from Elastic Cloud to OpenSearch Service, Octus’s month-to-month infrastructure prices are actually 85% decrease.
- Enhanced search efficiency – Octus maintained constant response occasions all through the migration with no unfavorable affect on latency, whereas attaining a 20% enchancment in question throughput and general search efficiency.
- Zero downtime – Octus skilled zero downtime throughout migration and 100% uptime general for the entire utility.
- Diminished operational overhead – Submit-migration, Octus’s DevOps and SRE groups see 30% much less upkeep burden and overheads. Supporting SOC2 compliance can be simple now that they’re utilizing one system.
- Accelerated timeline supply – The complete migration was accomplished forward of schedule, shifting from planning to full completion in below one quarter.
“Shifting from Elastic Cloud to Amazon OpenSearch Service was a key part of our broader technique to reduce third-party dependencies and strengthen the reliability of Octus’ system infrastructure. Migration Assistant for Amazon OpenSearch Service enabled us to execute a seamless transition with zero information loss and just about no downtime for our customers.” – Vishal Saxena, CTO, Octus
Conclusion
On this put up, we confirmed you the way Octus efficiently migrated their Elasticsearch workloads from Elastic Cloud to Amazon OpenSearch Service utilizing the Migration Assistant for OpenSearch Service, attaining zero downtime and important operational enhancements.
The Migration Assistant for OpenSearch Service supported this advanced migration by means of its complete suite of instruments. The Metadata Migration functionality migrated dozens of indices with various mappings and settings, with customized JavaScript transformations dealing with backward incompatibilities. Reindex-from-Snapshot migrated the historic paperwork from point-in-time snapshots with out impacting the supply cluster, whereas additionally optimizing the sharding scheme for improved efficiency. Stay Site visitors Replay made positive the goal cluster remained synchronized with real-time updates all through the migration course of.
The migration delivered substantial outcomes throughout the size. Octus achieved an 85% discount in month-to-month infrastructure prices whereas almost doubling the variety of paperwork obtainable for search throughout three environments. Search efficiency improved by 20% in question throughput with constant response occasions and no unfavorable affect on latency. The migration maintained zero downtime and 100% uptime for all the utility, with DevOps and SRE groups experiencing 30% much less upkeep burden and operational overhead. The complete migration was accomplished forward of schedule in below one quarter.
To study extra concerning the Migration Assistant for OpenSearch Service and the way it may help you obtain comparable outcomes, go to the AWS Options residence web page.
Go to Octus to find out how we ship rigorously verified intelligence at velocity and create an entire image for professionals throughout all the credit score lifecycle. Comply with Octus on LinkedIn and X.