Optimize multimodal search utilizing the TwelveLabs Embed API and Amazon OpenSearch Service

This weblog is co-authored by James Le, Head of Developer Expertise – TwelveLabs

The exponential development of video content material has created each alternatives and challenges. Content material creators, entrepreneurs, and researchers at the moment are confronted with the daunting process of effectively looking, analyzing, and extracting useful insights from huge video libraries. Conventional search strategies reminiscent of keyword-based textual content search usually fall quick when coping with video content material to investigate the visible content material, spoken phrases, or contextual components throughout the video itself, leaving organizations struggling to successfully search via and unlock the total potential of their multimedia property.

With the combination of TwelveLabs’ Embed API and Amazon OpenSearch Service, we are able to work together with and derive worth from video content material. Through the use of TwelveLabs‘ superior AI-powered video understanding know-how and OpenSearch Service’s search and analytics capabilities, we are able to now carry out superior video discovery and achieve deeper insights.

On this weblog put up, we present you the method of integrating TwelveLabs Embed API with OpenSearch Service to create a multimodal search answer. You’ll discover ways to generate wealthy, contextual embeddings from video content material and use OpenSearch Service’s vector database capabilities to allow search functionalities. By the top of this put up, you’ll be geared up with the data to implement a system that may rework the way in which your group handles and extracts worth from video content material.

TwelveLabs’ multimodal embeddings course of visible, audio, and textual content alerts collectively to create unified representations, capturing the direct relationships between these modalities. This unified strategy delivers exact, context-aware video search that matches human understanding of video content material. Whether or not you’re a developer trying to improve your purposes with superior video search capabilities, or a enterprise chief searching for to optimize your content material administration methods, this put up will offer you the instruments and steps to implement multimodal seek for your organizational knowledge.

About TwelveLabs

TwelveLabs is an Superior AWS Associate and AWS Market Vendor that provides video understanding options. Embed API is designed to revolutionize the way you work together with and extract worth from video content material.

At its core, the Embed API transforms uncooked video content material into significant, searchable knowledge through the use of state-of-the-art machine studying fashions. These fashions extract and characterize advanced video data within the type of dense vector embeddings, every a regular 1024-dimensional vector that captures the essence of the video content material throughout a number of modalities (picture, textual content, and audio).

Key options of TwelveLabs Embed API

Beneath are the important thing options of TwelveLabs Embed API:

• Multimodal understanding: The API generates embeddings that encapsulate varied facets of the video, together with visible expressions, physique language, spoken phrases, and total context.
• Temporal coherence: Not like static image-based fashions, TwelveLabs’ embeddings seize the interrelations between completely different modalities over time, offering a extra correct illustration of video content material.
• Flexibility: The API helps native processing of all modalities current in movies, eliminating the necessity for separate text-only or image-only fashions.
• Excessive efficiency: Through the use of a video-native strategy, the Embed API gives extra correct and temporally coherent interpretation of video content material in comparison with conventional CLIP-like fashions.

Advantages and use instances

The Embed API gives quite a few benefits for builders and companies working with video content material:

• Enhanced Search Capabilities: Allow highly effective multimodal search throughout video libraries, permitting customers to search out related content material primarily based on visible, audio, or textual queries.
• Content material Advice: Enhance content material suggestion techniques by understanding the deep contextual similarities between movies.
• Scene Detection and Segmentation: Robotically detect and phase completely different scenes inside movies for simpler navigation and evaluation.
• Content material Moderation: Effectively establish and flag inappropriate content material throughout giant video datasets.

Use instances embrace:

• Anomaly detection
• Variety sorting
• Sentiment evaluation
• Suggestions

Structure overview

The structure for utilizing TwelveLabs Embed API and OpenSearch Service for superior video search consists of the next parts:

• TwelveLabs Embed API: This API generates 1024-dimensional vector embeddings from video content material, capturing visible, audio, and textual components.
• OpenSearch Vector Database: Shops and indexes the video embeddings generated by TwelveLabs.
• Secrets and techniques Supervisor to retailer secrets and techniques reminiscent of API entry keys, and the Amazon OpenSearch Service username and password.
• Integration of TwelveLabs SDK and the OpenSearch Service consumer to course of movies, generate embeddings, and index them in OpenSearch Service.

The next diagram illustrates:

1. A video file is saved in Amazon Easy Storage Service (Amazon S3). Embeddings of the video file are created utilizing TwelveLabs Embed API.
2. Embeddings generated from the TwelveLabs Embed API at the moment are ingested to Amazon OpenSearch Service.
3. Customers can search the video embeddings utilizing textual content, audio, or picture. The person makes use of TwelveLabs Embed API to create the corresponding embeddings.
4. The person searches video embeddings in Amazon OpenSearch Service and retrieves the corresponding vector.

The use case

For the demo, you’ll work on these movies: Robin fowl forest Video by Federico Maderno from Pixabay and Island Video by Bellergy RC from Pixabay.

Nonetheless, the use case might be expanded to numerous different segments. For instance, the information group struggles with:

1. Needle-in-haystack searches via 1000’s of hours of archival footage
2. Guide metadata tagging that misses nuanced visible and audio context
3. Cross-modal queries reminiscent of querying a video assortment utilizing textual content or audio descriptions
4. Speedy content material retrieval for breaking information tie-ins

By integrating TwelveLabs Embed API with OpenSearch Service, you may:

• Generate 1024-dimensional embeddings capturing every video’s visible ideas. The embeddings are additionally able to extracting spoken narration, on-screen textual content, and audio cues.
• Allow multimodal search capabilities permitting customers to:
  • Discover particular demonstrations utilizing text-based queries.
  • Find actions via image-based queries.
  • Determine segments utilizing audio sample matching.
• Cut back search time from hours to seconds for advanced queries.

Answer walkthrough

GitHub repository accommodates a pocket book with detailed walkthrough directions for implementing superior video search capabilities by combining TwelveLabs’ Embed API with Amazon OpenSearch Service.

Conditions

Earlier than you proceed additional, confirm that the next stipulations are met:

• Affirm that you’ve got an AWS account. Check in to the AWS account.
• Create a TwelveLabs account as a result of will probably be required to get the API Key. TwelveLabs provide free tier pricing however you may improve if needed to satisfy your requirement.
• Have an Amazon OpenSearch Service area. For those who don’t have an current area, you may create one utilizing the steps outlined in our public documentation for Creating and Managing Amazon OpenSearch Service Area. Make it possible for the OpenSearch Service area is accessible out of your Python surroundings. You too can use Amazon OpenSearch Serverless for this use case and replace the interactions to OpenSearch Serverless utilizing AWS SDKs.

Step 1: Arrange the TwelveLabs SDK

Begin by organising the TwelveLabs SDK in your Python surroundings:

1. Get hold of your API key from TwelveLabs Dashboard.
2. Comply with steps right here to create a secret in AWS Secrets and techniques Supervisor. For instance, title the key as TL_API_Key.Notice down the ARN or title of the key (TL_API_Key) to retrieve. To retrieve a secret from one other account, you could use an ARN.For an ARN, we advocate that you simply specify a whole ARN reasonably than a partial ARN. See Discovering a secret from a partial ARN.Use this worth for the SecretId within the code block beneath.

import boto3
import json
secrets_manager_client=boto3.consumer("secretsmanager")
API_secret=secrets_manager_client.get_secret_value(
SecretId="TL_API_KEY"
)
TL_API_KEY=json.hundreds(API_secret["SecretString"])["TL_API_Key"]

Step 2: Generate video embeddings

Use the Embed API to create multimodal embeddings which might be contextual vector representations in your movies and texts. TwelveLabs video embeddings seize all of the refined cues and interactions between completely different modalities, together with the visible expressions, physique language, spoken phrases, and the general context of the video, encapsulating the essence of all these modalities and their interrelations over time.

To create video embeddings, you could first add your movies, and the platform should end processing them. Importing and processing movies require a while. Consequently, creating embeddings is an asynchronous course of comprised of three steps:

1. Add and course of a video: Once you begin importing a video, the platform creates a video embedding process and returns its distinctive process identifier.
2. Monitor the standing of your video embedding process: Use the distinctive identifier of your process to verify its standing periodically till it’s accomplished.
3. Retrieve the embeddings: After the video embedding process is accomplished, retrieve the video embeddings by offering the duty identifier. Study extra within the docs.

Video processing implementation

This demo relies upon upon some video knowledge. To make use of this, you’ll obtain two mp4 information and add it to an Amazon S3 bucket.

1. Click on on the hyperlinks containing the Robin fowl forest Video by Federico Maderno from Pixabay and Island Video by Bellergy RC from Pixabay movies.
2. Obtain the 21723-320725678_small.mp4 and 2946-164933125_small.mp4 information.
3. Create an S3 bucket should you don’t have one already. Comply with the steps within the Making a bucket doc. Notice down the title of the bucket and exchange it the code block beneath (Eg., MYS3BUCKET).
4. Add the 21723-320725678_small.mp4 and 2946-164933125_small.mp4 video information to the S3 bucket created within the step above by following the steps within the Importing objects doc. Notice down the title of the objects and exchange it the code block beneath (Eg., 21723-320725678_small.mp4 and 2946-164933125_small.mp4)

s3_client=boto3.consumer("s3")
bird_video_data=s3_client.download_file(Bucket="MYS3BUCKET",  Key='21723-320725678_small.mp4', Filename="robin-bird.mp4")
island_video_data=s3_client.download_file(Bucket="MYS3BUCKET",  Key='2946-164933125_small.mp4', Filename="island.mp4")

def print_segments(segments: Record[SegmentEmbedding], max_elements: int = 1024):
    for phase in segments:
        print(
            f"  embedding_scope={phase.embedding_scope} start_offset_sec={phase.start_offset_sec} end_offset_sec={phase.end_offset_sec}"
        )
        print(f"  embeddings: {phase.embeddings_float[:max_elements]}")

# Initialize consumer with API key
twelvelabs_client = TwelveLabs(api_key=TL_API_KEY)

video_files=["robin-bird.mp4", "island.mp4"]
duties=[]

Embedding technology course of

With the SDK configured, generate embeddings in your video and monitor process completion with real-time updates. Right here you utilize the Marengo 2.7 mannequin to generate the embeddings:

for video in video_files:
    # Create embedding process
    process = twelvelabs_client.embed.process.create(
        model_name="Marengo-retrieval-2.7",
        video_file=video
    )
    print(
        f"Created process: id={process.id} engine_name={process.model_name} standing={process.standing}"
    )
    
    def on_task_update(process: EmbeddingsTask):
        print(f"  Standing={process.standing}")
    
    standing = process.wait_for_done(
        sleep_interval=2,
        callback=on_task_update
    )
    print(f"Embedding finished: {standing}")
    
    # Retrieve and examine outcomes
    process = process.retrieve()
    if process.video_embedding just isn't None and process.video_embedding.segments just isn't None:
        print_segments(process.video_embedding.segments)
    duties.append(process)

Key options demonstrated embrace:

• Multimodal seize: 1024-dimensional vectors encoding visible, audio, and textual options
• Mannequin specificity: Utilizing Marengo-retrieval-2.7 optimized for scientific content material
• Progress monitoring: Actual-time standing updates throughout embedding technology

Anticipated output

Created process: id=67ca93a989d8a564e80dc3ba engine_name=Marengo-retrieval-2.7 standing=processing
  Standing=processing
  Standing=processing
  Standing=processing
  Standing=processing
  Standing=processing
  Standing=processing
  Standing=processing
  Standing=processing
  Standing=processing
  Standing=prepared
Embedding finished: prepared
  embedding_scope=clip start_offset_sec=0.0 end_offset_sec=6.0
  embeddings: [0.022429451, 0.00040668788, -0.01825908, -0.005862708, -0.03371106, 
-6.357456e-05, -0.015320076, -0.042556215, -0.02782445, -0.00019097517, 0.03258314, 
-0.0061399476, -0.00049206393, 0.035632476, 0.028209884, 0.02875258, -0.035486065, 
-0.11288028, -0.040782217, -0.0359422, 0.015908664, -0.021092793, 0.016303983, 
0.06351931,…………………

Step 3: Set up OpenSearch

To enable vector search capabilities, you first need to set up an OpenSearch client and test the connection. Follow these steps:

Install the required libraries

Install the necessary Python packages for working with OpenSearch:

!pip install opensearch-py
!pip install botocore
!pip install requests-aws4auth

Configure the OpenSearch client

Set up the OpenSearch client with your host details and authentication credentials:

from opensearchpy import OpenSearch, RequestsHttpConnection, helpers
from requests_aws4auth import AWS4Auth
from requests.auth import HTTPBasicAuth

# OpenSearch connection configuration
# host="your-host.aos.us-east-1.on.aws"
host="search-new-domain-mbgs7wth6r5w6hwmjofntiqcge.aos.us-east-1.on.aws"
port = 443  # Default HTTPS port

# Get OpenSearch username secret from Secrets Manager
opensearch_username=secrets_manager_client.get_secret_value(
    SecretId="AOS_username"
)
opensearch_username_string=json.loads(opensearch_username["SecretString"])["AOS_username"]

# Get OpenSearch password secret from Secrets and techniques Supervisor
opensearch_password = secrets_manager_client.get_secret_value(
    SecretId="AOS_password"
)
opensearch_password_string=json.hundreds(opensearch_password["SecretString"])["AOS_password"]

auth=(opensearch_username_string, opensearch_password_string)

# Create the consumer configuration
client_aos = OpenSearch(
    hosts=[{'host': host, 'port': port}],
    http_auth=auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

# Take a look at the connection
strive:
    # Get cluster data
    cluster_info = client_aos.information()
    print("Efficiently related to OpenSearch")
    print(f"Cluster information: {cluster_info}")
besides Exception as e:
    print(f"Connection failed: {str(e)}")

Anticipated output

If the connection is profitable, you must see a message like the next:

Efficiently related to OpenSearch
Cluster information: {'title': 'bb36e8d98ee7bd517891ecd714bfb9d7', ...}

This confirms that your OpenSearch consumer is correctly configured and prepared to be used.

Step 4: Create an index in OpenSearch Service

Subsequent, you create an index optimized for vector search to retailer the embeddings generated by the TwelveLabs Embed API.

Outline the index configuration

The index is configured to help k-nearest neighbor (kNN) search with a 1024-dimensional vector discipline. You’ll these values for this demo however observe these finest practices to search out acceptable values in your utility. Right here’s the code:

# Outline the improved index configuration
index_name="twelvelabs_index"
new_vector_index_definition = {
    "settings": {
        "index": {
            "knn": "true",
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    },
    "mappings": {
        "properties": {
            "embedding_field": {
                "sort": "knn_vector",
                "dimension": 1024
            },
            "video_title": {
                "sort": "textual content",
                "fields": {
                    "key phrase": {
                        "sort": "key phrase"
                    }
                }
            },
            "segment_start": {
                "sort": "date"
            },
            "segment_end": {
                "sort": "date"
            },
            "segment_id": {
                "sort": "textual content"
            }
        }
    }
}

Create the Index

Use the next code to create the index in OpenSearch Service:

# Create the index in OpenSearch
response = client_aos.indices.create(index=index_name, physique=new_vector_index_definition, ignore=400)

# Retrieve and show index particulars to verify creation
index_info = client_aos.indices.get(index=index_name)
print(index_info)

Anticipated output

After operating this code, you must see particulars of the newly created index. For instance:

{'twelvelabs_index': {'aliases': {}, 'mappings': {'properties': {'embedding_field': {'sort': 'knn_vector', 'dimension': 1024}}}, 'settings': {...}}}

The next screenshot confirms that an index named twelvelabs_index has been efficiently created with a knn_vector discipline of dimension 1024 and different specified settings. With these steps accomplished, you now have an operational OpenSearch Service area configured for vector search. This index will function the repository for storing embeddings generated from video content material, enabling superior multimodal search capabilities.

Step 5: Ingest embeddings to the created index in OpenSearch Service

With the TwelveLabs Embed API efficiently producing video embeddings and the OpenSearch Service index configured, the following step is to ingest these embeddings into the index. This course of helps be certain that the embeddings are saved in OpenSearch Service and made searchable for multimodal queries.

Embedding ingestion course of

The next code demonstrates how you can course of and index the embeddings into OpenSearch Service:

from opensearchpy.helpers import bulk

def generate_actions(duties, video_files):
    depend = 0
    for process in duties:
        # Test if video embeddings can be found
        if process.video_embedding just isn't None and process.video_embedding.segments just isn't None:
            embeddings_doc = process.video_embedding.segments
            
            # Generate actions for bulk indexing
            for doc_id, elt in enumerate(embeddings_doc):
                yield {
                    '_index': index_name,
                    '_id': doc_id,
                    '_source': {
                        'embedding_field': elt.embeddings_float,
                        'video_title': video_files[count],
                        'segment_start': elt.start_offset_sec,
                        'segment_end': elt.end_offset_sec,
                        'segment_id': doc_id
                    }
                }
        print(f"Ready bulk indexing knowledge for process {depend}")
        depend += 1

# Carry out bulk indexing
strive:
    success, failed = bulk(client_aos, generate_actions(duties, video_files))
    print(f"Efficiently listed {success} paperwork")
    if failed:
        print(f"Didn't index {len(failed)} paperwork")
besides Exception as e:
    print(f"Error throughout bulk indexing: {e}")

Rationalization of the code

1. Embedding extraction: The video_embedding.segments object accommodates a listing of phase embeddings generated by the TwelveLabs Embed API. Every phase represents a particular portion of the video.
2. Doc creation: For every phase, a doc is created with a key (embedding_field) that shops its 1024-dimensional vector, video_title with the title of the video, segment_start and segment_end indicating the timestamp of the video phase, and a segment_id.
3. Indexing in OpenSearch: The index() technique uploads every doc to the twelvelabs_index created earlier. Every doc is assigned a singular ID (doc_id) primarily based on its place within the checklist.

Anticipated output

After the script runs efficiently, you will note:

• A printed checklist of embeddings being listed.
• A affirmation message:

Ready bulk indexing knowledge for process 0
Ready bulk indexing knowledge for process 1
Efficiently listed 6 paperwork

End result

At this stage, all video phase embeddings at the moment are saved in OpenSearch and prepared for superior multimodal search operations, reminiscent of text-to-video or image-to-video queries. This units up the inspiration for performing environment friendly and scalable searches throughout your video content material.

Step 6: Carry out vector search in OpenSearch Service

After embeddings are generated, you utilize it as a question vector to carry out a kNN search within the OpenSearch Service index. Beneath are the features to carry out vector search and format the search outcomes:

# Operate to carry out vector search
def search_similar_segments(query_vector, okay=5):
    question = {
        "dimension": okay,
        "_source": ["video_title", "segment_start", "segment_end", "segment_id"],
        "question": {
            "knn": {
                "embedding_field": {
                    "vector": query_vector,
                    "okay": okay
                }
            }
        }
    }
    
    response = client_aos.search(
        index=index_name,
        physique=question
    )

    outcomes = []
    for hit in response['hits']['hits']:
        end result = {
            'rating': hit['_score'],
            'title': hit['_source']['video_title'],
            'start_time': hit['_source']['segment_start'],
            'end_time': hit['_source']['segment_end'],
            'segment_id': hit['_source']['segment_id']
        }
        outcomes.append(end result)

    return (outcomes)

# Operate to format search outcomes
def print_search_results(outcomes):
    print("nSearch Outcomes:")
    print("-" * 50)
    for i, lead to enumerate(outcomes, 1):
        print(f"nResult {i}:")
        print(f"Video: {end result['title']}")
        print(f"Time Vary: {end result['start_time']} - {end result['end_time']}")
        print(f"Similarity Rating: {end result['score']:.4f}")

Key factors:

• The _source discipline accommodates the video title, phase begin, phase finish, and phase id equivalent to the video embeddings.
• The embedding_field within the question corresponds to the sector the place video embeddings are saved.
• The okay parameter specifies what number of high outcomes to retrieve primarily based on similarity.

Step 7:Performing text-to-video search

You need to use text-to-video search to retrieve video segments which might be most related to a given textual question. On this answer, you’ll do that through the use of TwelveLabs’ textual content embedding capabilities and OpenSearch’s vector search performance. Right here’s how one can implement this step:

Generate textual content embeddings

To carry out a search, you first have to convert the textual content question right into a vector illustration utilizing the TwelveLabs Embed API:

from typing import Record
from twelvelabs.fashions.embed import SegmentEmbedding

def print_segments(segments: Record[SegmentEmbedding], max_elements: int = 1024):
    for phase in segments:
        print(
            f"  embedding_scope={phase.embedding_scope} start_offset_sec={phase.start_offset_sec} end_offset_sec={phase.end_offset_sec}"
        )
        print(f"  embeddings: {phase.embeddings_float[:max_elements]}")

# Create textual content embeddings for the question
text_res = twelvelabs_client.embed.create(
    model_name="Marengo-retrieval-2.7",
    textual content="Hen consuming meals",  # Substitute together with your desired question
)

print("Created a textual content embedding")
print(f" Mannequin: {text_res.model_name}")

# Extract and examine the generated textual content embeddings
if text_res.text_embedding just isn't None and text_res.text_embedding.segments just isn't None:
    print_segments(text_res.text_embedding.segments)

vector_search = text_res.text_embedding.segments[0].embeddings_float
print("Generated Textual content Embedding Vector:", vector_search)

Key factors:

• The Marengo-retrieval-2.7 mannequin is used to generate a dense vector embedding for the question.
• The embedding captures the semantic which means of the enter textual content, enabling efficient matching with video embeddings.

Carry out vector search in OpenSearch Service

After the textual content embedding is generated, you utilize it as a question vector to carry out a kNN search within the OpenSearch index:

# Outline the vector search question
query_vector = vector_search
text_to_video_search = search_similar_segments(query_vector)
# print(text_video_search)
print_search_results(text_to_video_search)

Anticipated output

The next illustrates related outcomes retrieved from OpenSearch.

Search Outcomes:
--------------------------------------------------

End result 1:
Video: robin-bird.mp4
Time Vary: 18.0 - 21.087732
Similarity Rating: 0.4409

End result 2:
Video: robin-bird.mp4
Time Vary: 12.0 - 18.0
Similarity Rating: 0.4300

End result 3:
Video: island.mp4
Time Vary: 0.0 - 6.0
Similarity Rating: 0.3624

Insights from outcomes

• Every end result features a similarity rating indicating how intently it matches the question, a time vary indicating the beginning and finish offset in seconds, and the video title.
• Observe that the highest 2 outcomes correspond to the robin fowl video segments matching the Hen consuming meals question.

This course of demonstrates how textual queries reminiscent of Hen consuming meals can successfully retrieve related video segments from an listed library utilizing TwelveLabs’ multimodal embeddings and OpenSearch’s highly effective vector search capabilities.

Step 8: Carry out audio-to-video search

You need to use audio-to-video search to retrieve video segments which might be most related to a given audio enter. Through the use of TwelveLabs’ audio embedding capabilities and OpenSearch’s vector search performance, you may match audio options with video embeddings within the index. Right here’s how you can implement this step:

Generate audio embeddings

To carry out the search, you first convert the audio enter right into a vector illustration utilizing the TwelveLabs Embed API:

# Create audio embeddings for the enter audio file
audio_res = twelvelabs_client.embed.create(
    model_name="Marengo-retrieval-2.7",
    audio_file="audio-data.mp3",  # Substitute together with your desired audio file
)

# Print particulars of the generated embedding
print(f"Created audio embedding: model_name={audio_res.model_name}")
print(f" Mannequin: {audio_res.model_name}")

# Extract and examine the generated audio embeddings
if audio_res.audio_embedding just isn't None and audio_res.audio_embedding.segments just isn't None:
    print_segments(audio_res.audio_embedding.segments)

# Retailer the embedding vector for search
vector_search = audio_res.audio_embedding.segments[0].embeddings_float
print("Generated Audio Embedding Vector:", vector_search)

Key factors:

• The Marengo-retrieval-2.7 mannequin is used to generate a dense vector embedding for the enter audio.
• The embedding captures the semantic options of the audio, reminiscent of rhythm, tone, and patterns, enabling efficient matching with video embeddings

Carry out vector search in OpenSearch Service

After the audio embedding is generated, you utilize it as a question vector to carry out a k-nearest neighbor (kNN) search in OpenSearch:

# Carry out vector search
query_vector = vector_search
audio_to_video_search = search_similar_segments(query_vector)
# print(text_video_search)
    
print_search_results(audio_to_video_search)

Anticipated output

The next exhibits video segments retrieved from OpenSearch Service primarily based on their similarity to the enter audio.

Search Outcomes:
--------------------------------------------------

End result 1:
Video: island.mp4
Time Vary: 6.0 - 12.0
Similarity Rating: 0.2855

End result 2:
Video: robin-bird.mp4
Time Vary: 18.0 - 21.087732
Similarity Rating: 0.2841

End result 3:
Video: robin-bird.mp4
Time Vary: 12.0 - 18.0
Similarity Rating: 0.2837

End result 4:
Video: island.mp4
Time Vary: 0.0 - 6.0
Similarity Rating: 0.2835

Right here discover that segments from each movies are returned with a low similarity rating.

Step 9: Performing images-to-video search

You need to use image-to-video search to retrieve video segments which might be visually much like a given picture. Through the use of TwelveLabs’ picture embedding capabilities and OpenSearch Service’s vector search performance, you may match visible options from a picture with video embeddings within the index. Right here’s how you can implement this step:

Generate Picture Embeddings

To carry out the search, you first convert the enter picture right into a vector illustration utilizing the TwelveLabs Embed API:

# Create picture embeddings for the enter picture file
image_res = twelvelabs_client.embed.create(
    model_name="Marengo-retrieval-2.7",
    image_file="image-data.jpg",  # Substitute together with your desired picture file
)

# Print particulars of the generated embedding
print(f"Created picture embedding: model_name={image_res.model_name}")
print(f" Mannequin: {image_res.model_name}")

# Extract and examine the generated picture embeddings
if image_res.image_embedding just isn't None and image_res.image_embedding.segments just isn't None:
    print_segments(image_res.image_embedding.segments)

# Retailer the embedding vector for search
vector_search = image_res.image_embedding.segments[0].embeddings_float
print("Generated Picture Embedding Vector:", vector_search)

Key factors:

• The Marengo-retrieval-2.7 mannequin is used to generate a dense vector embedding for the enter picture.
• The embedding captures visible options reminiscent of shapes, colours, and patterns, enabling efficient matching with video embeddings

Carry out vector search in OpenSearch

After the picture embedding is generated, you utilize it as a question vector to carry out a k-nearest neighbor (kNN) search in OpenSearch:

# Carry out vector search
query_vector = vector_search
image_to_video_search = search_similar_segments(query_vector)
# print(text_video_search)
    
print_search_results(image_to_video_search)

Anticipated output

The next exhibits video segments retrieved from OpenSearch primarily based on their similarity to the enter picture.

Search Outcomes:
--------------------------------------------------

End result 1:
Video: island.mp4
Time Vary: 6.0 - 12.0
Similarity Rating: 0.5616

End result 2:
Video: island.mp4
Time Vary: 0.0 - 6.0
Similarity Rating: 0.5576

End result 3:
Video: robin-bird.mp4
Time Vary: 12.0 - 18.0
Similarity Rating: 0.4592

End result 4:
Video: robin-bird.mp4
Time Vary: 18.0 - 21.087732
Similarity Rating: 0.4540

Observe that picture of an ocean was used to go looking the movies. Video clips from the island video are retrieved with the next similarity rating within the first 2 outcomes.

Clear up

To keep away from prices, delete assets created whereas following this put up. For Amazon OpenSearch Service domains, navigate to the AWS Administration Console for Amazon OpenSearch Service dashboard and delete the area.

Conclusion

The mixing of TwelveLabs Embed API with OpenSearch Service gives a cutting-edge answer for superior video search and evaluation, unlocking new prospects for content material discovery and insights. Through the use of TwelveLabs’ multimodal embeddings, which seize the intricate interaction of visible, audio, and textual components in movies, and mixing them with OpenSearch Service’s sturdy vector search capabilities, this answer allows extremely nuanced and contextually related video search.

As industries more and more depend on video content material for communication, schooling, advertising, and analysis, this superior search answer turns into indispensable. It empowers companies to extract hidden insights from their video content material, improve person experiences in video-centric purposes and make data-driven selections primarily based on complete video evaluation

This integration not solely addresses present challenges in managing video content material but in addition lays the inspiration for future improvements in how we work together with and derive worth from video knowledge.

Get began

Able to discover the ability of TwelveLabs Embed API? Begin your free trial at the moment by visiting TwelveLabs Playground to enroll and obtain your API key.

For builders trying to implement this answer, observe our detailed step-by-step information on GitHub to combine TwelveLabs Embed API with OpenSearch Service and construct your personal superior video search utility.

Unlock the total potential of your video content material at the moment!

Concerning the Authors

James Le runs the Developer Expertise operate at TwelveLabs. He works with companions, builders, and researchers to convey state-of-the-art video basis fashions to numerous multimodal video understanding use instances.

Gitika is an Senior WW Information & AI Associate Options Architect at Amazon Internet Providers (AWS). She works with companions on technical tasks, offering architectural steering and enablement to construct their analytics observe.

Kruthi is a Senior Associate Options Architect specializing in AI and ML. She gives technical steering to AWS Companions in following finest practices to construct safe, resilient, and extremely accessible options within the AWS Cloud.