Construct a Multimodal Agent for Product Ingredient Evaluation

Have you ever ever discovered your self looking at a product’s components checklist, googling unfamiliar chemical names to determine what they imply? It’s a standard wrestle – deciphering complicated product info on the spot may be overwhelming and time-consuming. Conventional strategies, like trying to find every ingredient individually, usually result in fragmented and complicated outcomes. However what if there was a wiser, sooner technique to analyze product components and get clear, actionable insights immediately? On this article, we’ll stroll you thru constructing a Product Components Analyzer utilizing Gemini 2.0, Phidata, and Tavily Net Search. Let’s dive in and make sense of these ingredient lists as soon as and for all!

Studying Goals

• Design a Multimodal AI Agent structure utilizing Phidata and Gemini 2.0 for vision-language duties.
• Combine Tavily Net Search into agent workflows for higher context and data retrieval.
• Construct a Product Ingredient Analyzer Agent that mixes picture processing and internet seek for detailed product insights.
• Learn the way system prompts and directions information agent conduct in multimodal duties.
• Develop a Streamlit UI for real-time picture evaluation, diet particulars, and health-based recommendations.

This text was printed as part of the Information Science Blogathon.

What are Multimodal Methods?

Multimodal programs course of and perceive a number of varieties of enter information—like textual content, photographs, audio, and video—concurrently. Imaginative and prescient-language fashions, corresponding to Gemini 2.0 Flash, GPT-4o, Claude Sonnet 3.5, and Pixtral-12B, excel at understanding relationships between these modalities, extracting significant insights from complicated inputs.

On this context, we deal with vision-language fashions that analyze photographs and generate textual insights. These programs mix laptop imaginative and prescient and pure language processing to interpret visible info primarily based on person prompts.

Multimodal Actual-world Use Instances

Multimodal programs are reworking industries:

• Finance: Customers can take screenshots of unfamiliar phrases in on-line types and get immediate explanations.
• E-commerce: Customers can {photograph} product labels to obtain detailed ingredient evaluation and well being insights.
• Schooling: College students can seize textbook diagrams and obtain simplified explanations.
• Healthcare: Sufferers can scan medical studies or prescription labels for simplified explanations of phrases and dosage directions.

Why Multimodal Agent?

The shift from single-mode AI to multimodal brokers marks a significant leap in how we work together with AI programs. Right here’s what makes multimodal brokers so efficient:

• They course of each visible and textual info concurrently, delivering extra correct and context-aware responses.
• They simplify complicated info, making it accessible to customers who might wrestle with technical phrases or detailed content material.
• As an alternative of manually trying to find particular person elements, customers can add a picture and obtain complete evaluation in a single step.
• By combining instruments like internet search and picture evaluation, they supply extra full and dependable insights.

Constructing Product Ingredient Analyzer Agent

Let’s break down the implementation of a Product Ingredient Evaluation Agent:

Step 1: Setup Dependencies 

• Gemini 2.0 Flash: Handles multimodal processing with enhanced imaginative and prescient capabilities
• Tavily Search: Offers internet search integration for extra context
• Phidata: Orchestrates the Agent system and manages workflows
• Streamlit: To develop the prototype into Net-based functions.

!pip set up phidata google-generativeai tavily-python streamlit pillow

Step 2: API Setup and Configuration 

On this step, we’ll arrange the surroundings variables and collect the required API credentials to run this use case. 

from phi.agent import Agent
from phi.mannequin.google import Gemini # wants a api key
from phi.instruments.tavily import TavilyTools # additionally wants a api key

import os
TAVILY_API_KEY = "<replace-your-api-key>"
GOOGLE_API_KEY = "<replace-your-api-key>"
os.environ['TAVILY_API_KEY'] = TAVILY_API_KEY
os.environ['GOOGLE_API_KEY'] = GOOGLE_API_KEY

Step 3: System immediate and Directions

To get higher responses from language fashions, it’s essential write higher prompts. This entails clearly defining the position and offering detailed directions within the system immediate for the LLM.

Let’s outline the position and obligations of an Agent with experience in ingredient evaluation and diet. The directions ought to information the Agent to systematically analyze meals merchandise, assess components, contemplate dietary restrictions, and consider well being implications.

SYSTEM_PROMPT = """
You're an professional Meals Product Analyst specialised in ingredient evaluation and diet science. 
Your position is to investigate product components, present well being insights, and establish potential issues by combining ingredient evaluation with scientific analysis. 
You make the most of your dietary data and analysis works to supply evidence-based insights, making complicated ingredient info accessible and actionable for customers.
Return your response in Markdown format. 
"""

INSTRUCTIONS = """
* Learn ingredient checklist from product picture 
* Keep in mind the person might not be educated in regards to the product, break it down in easy phrases like explaining to 10 12 months child
* Determine synthetic components and preservatives
* Examine towards main dietary restrictions (vegan, halal, kosher). Embrace this in response. 
* Fee dietary worth on scale of 1-5
* Spotlight key well being implications or issues
* Counsel more healthy alternate options if wanted
* Present transient evidence-based suggestions
* Use Search instrument for getting context
"""

Step 4: Outline the Agent Object

The Agent, constructed utilizing Phidata, is configured to course of markdown formatting and function primarily based on the system immediate and directions outlined earlier. The reasoning mannequin used on this instance is Gemini 2.0 Flash, identified for its superior means to grasp photographs and movies in comparison with different fashions.

For instrument integration, we’ll use Tavily Search, a sophisticated internet search engine that gives related context straight in response to person queries, avoiding pointless descriptions, URLs, and irrelevant parameters.

agent = Agent(
    mannequin = Gemini(id="gemini-2.0-flash-exp"),
    instruments = [TavilyTools()],
    markdown=True,
    system_prompt = SYSTEM_PROMPT,
    directions = INSTRUCTIONS
)

Step 5: Multimodal – Understanding the Picture

With the Agent elements now in place, the following step is to supply person enter. This may be achieved in two methods: both by passing the picture path or the URL, together with a person immediate specifying what info must be extracted from the supplied picture.

Strategy: 1 Utilizing Picture Path

agent.print_response(
    "Analyze the product picture",
    photographs = ["images/bournvita.jpg"],
    stream=True
)

Output:

Strategy: 2 Utilizing URL

agent.print_response(
    "Analyze the product picture",
    photographs = ["https://beardo.in/cdn/shop/products/9_2ba7ece4-0372-4a34-8040-5dc40c89f103.jpg?v=1703589764&width=1946"],
    stream=True
)

Output:

Step 6: Develop the Net App utilizing Streamlit

Now that we all know the right way to execute the Multimodal Agent, let’s construct the UI half utilizing Streamlit. 

import streamlit as st
from PIL import Picture
from io import BytesIO
from tempfile import NamedTemporaryFile

st.title("🔍 Product Ingredient Analyzer")

To optimize efficiency, outline the Agent inference beneath a cached operate. The cache decorator helps enhance effectivity by reusing the Agent occasion.

Since we’re utilizing Streamlit, which refreshes your complete web page after every occasion loop or widget set off, including st.cache_resource ensures the operate is just not refreshed and saves it within the cache.

@st.cache_resource
def get_agent():
    return Agent(
        mannequin=Gemini(id="gemini-2.0-flash-exp"),
        system_prompt=SYSTEM_PROMPT,
        directions=INSTRUCTIONS,
        instruments=[TavilyTools(api_key=os.getenv("TAVILY_API_KEY"))],
        markdown=True,
    )

When a brand new picture path is supplied by the person, the analyze_image operate runs and executes the Agent object outlined in get_agent. For real-time seize and the choice to add photographs, the uploaded file must be saved quickly for processing.

The picture is saved in a short lived file, and as soon as the execution is accomplished, the momentary file is deleted to release sources. This may be achieved utilizing the NamedTemporaryFile operate from the tempfile library. 

def analyze_image(image_path):
    agent = get_agent()
    with st.spinner('Analyzing picture...'):
        response = agent.run(
            "Analyze the given picture",
            photographs=[image_path],
        )
        st.markdown(response.content material)

def save_uploaded_file(uploaded_file):
    with NamedTemporaryFile(dir=".", suffix='.jpg', delete=False) as f:
        f.write(uploaded_file.getbuffer())
        return f.title

For a greater person interface, when a person selects a picture, it’s prone to have various resolutions and sizes. To take care of a constant format and correctly show the picture, we are able to resize the uploaded or captured picture to make sure it matches clearly on the display screen.

The LANCZOS resampling algorithm gives high-quality resizing, notably useful for product photographs the place textual content readability is essential for ingredient evaluation.

MAX_IMAGE_WIDTH = 300

def resize_image_for_display(image_file):
    img = Picture.open(image_file)
    
    aspect_ratio = img.top / img.width
    new_height = int(MAX_IMAGE_WIDTH * aspect_ratio)
    img = img.resize((MAX_IMAGE_WIDTH, new_height), Picture.Resampling.LANCZOS)
    
    buf = BytesIO()
    img.save(buf, format="PNG")
    return buf.getvalue()

Step 7: UI Options for Streamlit 

The interface is split into three navigation tabs the place the person can decide his selection of pursuits:

• Tab-1: Instance Merchandise that customers can choose to check the app
• Tab-2: Add an Picture of your selection if it’s already saved.
• Tab-3: Seize or Take a dwell picture and analyze the product.

We repeat the identical logical movement for all the three tabs:

• First, select the picture of your selection and resize it to show on the Streamlit UI utilizing st.picture.
• Second, save that picture in a short lived listing to course of it to the Agent object.
• Third, analyze the picture the place the Agent execution will happen utilizing Gemini 2.0 LLM and Tavily Search instrument.

State administration is dealt with via Streamlit’s session state, monitoring chosen examples and evaluation standing. 

def predominant():
    if 'selected_example' not in st.session_state:
        st.session_state.selected_example = None
    if 'analyze_clicked' not in st.session_state:
        st.session_state.analyze_clicked = False
    
    tab_examples, tab_upload, tab_camera = st.tabs([
        "📚 Example Products", 
        "📤 Upload Image", 
        "📸 Take Photo"
    ])
    
    with tab_examples:
        example_images = {
            "🥤 Vitality Drink": "photographs/bournvita.jpg",
            "🥔 Potato Chips": "photographs/lays.jpg",
            "🧴 Shampoo": "photographs/shampoo.jpg"
        }
        
        cols = st.columns(3)
        for idx, (title, path) in enumerate(example_images.objects()):
            with cols[idx]:
                if st.button(title, use_container_width=True):
                    st.session_state.selected_example = path
                    st.session_state.analyze_clicked = False
    
    with tab_upload:
        uploaded_file = st.file_uploader(
            "Add product picture", 
            sort=["jpg", "jpeg", "png"],
            assist="Add a transparent picture of the product's ingredient checklist"
        )
        if uploaded_file:
            resized_image = resize_image_for_display(uploaded_file)
            st.picture(resized_image, caption="Uploaded Picture", use_container_width=False, width=MAX_IMAGE_WIDTH)
            if st.button("🔍 Analyze Uploaded Picture", key="analyze_upload"):
                temp_path = save_uploaded_file(uploaded_file)
                analyze_image(temp_path)
                os.unlink(temp_path) 
    
    with tab_camera:
        camera_photo = st.camera_input("Take an image of the product")
        if camera_photo:
            resized_image = resize_image_for_display(camera_photo)
            st.picture(resized_image, caption="Captured Picture", use_container_width=False, width=MAX_IMAGE_WIDTH)
            if st.button("🔍 Analyze Captured Picture", key="analyze_camera"):
                temp_path = save_uploaded_file(camera_photo)
                analyze_image(temp_path)
                os.unlink(temp_path) 
    
    if st.session_state.selected_example:
        st.divider()
        st.subheader("Chosen Product")
        resized_image = resize_image_for_display(st.session_state.selected_example)
        st.picture(resized_image, caption="Chosen Instance", use_container_width=False, width=MAX_IMAGE_WIDTH)
        
        if st.button("🔍 Analyze Instance", key="analyze_example") and never st.session_state.analyze_clicked:
            st.session_state.analyze_clicked = True
            analyze_image(st.session_state.selected_example)

Necessary Hyperlinks

• Yow will discover the complete code right here.
• Change the “<replace-with-api-key>” placeholder along with your keys.
• For tab_examples, it’s essential have a folder picture. And save the pictures over there. Right here is the GitHub URL with photographs listing right here.
• If you’re excited about utilizing the use case, right here is the deployed App right here.

Conclusion 

Multimodal AI brokers symbolize a higher leap ahead in how we are able to work together with and perceive complicated info in our every day lives. By combining imaginative and prescient processing, pure language understanding, and internet search capabilities, these programs, just like the Product Ingredient Analyzer, can present immediate, complete evaluation of merchandise and their components, making knowledgeable decision-making extra accessible to everybody.

Key Takeaways

• Multimodal AI brokers enhance how we perceive product info. They mix textual content and picture evaluation.
• With Phidata, an open-source framework, we are able to construct and handle agent programs. These programs use fashions like GPT-4o and Gemini 2.0.
• Brokers use instruments like imaginative and prescient processing and internet search. This makes their evaluation extra full and correct. LLMs have restricted data, so brokers use instruments to deal with complicated duties higher.
• Streamlit makes it simple to construct internet apps for LLM-based instruments. Examples embody RAG and multimodal brokers.
• Good system prompts and directions information the agent. This ensures helpful and correct responses.

Steadily Requested Questions

The media proven on this article is just not owned by Analytics Vidhya and is used on the Creator’s discretion.

Information Scientist at AI Planet || YouTube- AIWithTarun || Google Developer Skilled in ML || Gained 5 AI hackathons || Co-organizer of TensorFlow Person Group Bangalore || Pie & AI Ambassador at DeepLearningAI