🌐 Multimodal RAG with LangChain, Pinecone & Groq

A production-ready Multimodal Retrieval-Augmented Generation (RAG) application designed to ingest, index, and query complex PDF documents containing a mix of narrative text, structured tabular data, and visual content (charts, diagrams, and operational photos).

Built with Streamlit, LangChain, Pinecone Vector Database, and ultra-fast inference powered by Groq Vision & Text LLMs.

---

📌 Table of Contents

• Overview
• System Architecture
• Key Features
• Project Structure
• Getting Started
  • Prerequisites
  • 1. Clone and Navigate
  • 2. Environment Setup (uv or venv)
  • 3. Configure API Credentials
  • 4. Launch the Web App
• How It Works
  • 1. Tri-Modal Extraction
  • 2. Visual AI Summarization
  • 3. Vector Indexing
  • 4. Hybrid Retrieval & Dynamic Multimodal Generation
• Sample Demo Queries
• Configuration Reference
• Troubleshooting

---

💡 Overview

Traditional RAG pipelines fail when faced with real-world enterprise documents because they strip out or ignore charts, process diagrams, financial graphs, and structured tables.

This application solves that limitation by implementing a tri-modal ingestion and retrieval pipeline:

1. Unstructured Text: Extracted cleanly per page with metadata preservation.
2. Structured Tables: Detected and rendered into Markdown format so tabular relationships and headers remain preserved.
3. Visuals & Charts: High-resolution raster images are extracted, analyzed by a Multimodal Vision LLM to generate descriptive textual summaries, and indexed in Pinecone. At query time, both the textual context and the actual visual images are fed to the Vision LLM for grounded answers.

---

🏗 System Architecture

flowchart TD
    subgraph Ingestion ["1. Document Ingestion Pipeline"]
        PDF[PDF Document] --> Extractor[PyMuPDF / Fitz Extractor]
        Extractor -->|Text Streams| TextChunks[Text Documents]
        Extractor -->|Table Grids| TableMarkdown[Markdown Tables]
        Extractor -->|Extracted Images| VisionDescriber[Groq Vision LLM\nPrompt: Factual Description]
        VisionDescriber --> VisualDocs[Visual Description Docs]
        
        TextChunks --> Embedder[HuggingFace Embeddings\nall-MiniLM-L6-v2]
        TableMarkdown --> Embedder
        VisualDocs --> Embedder
        Embedder --> VectorDB[(Pinecone Serverless Index\nCustom Namespace)]
    end

    subgraph Retrieval ["2. Retrieval & Multimodal Generation"]
        UserQuery[User Question] --> QueryEmbedder[Embed Query]
        QueryEmbedder --> PineconeSearch[Pinecone Similarity Search\nTop-K Context]
        PineconeSearch --> Router{Visual Docs\nin Top-K?}
        
        Router -->|Yes| VisionLLM[Groq Multimodal LLM\nContext + Image Data URIs]
        Router -->|No| TextLLM[Groq Text LLM\nopenai/gpt-oss-20b or llama-3.3]
        
        VisionLLM --> Answer[Grounded Multimodal Answer + Visual Evidence]
        TextLLM --> Answer
    end

Loading

---

✨ Key Features

• Multi-Modal Document Processing:
  • Automatically identifies and handles paragraphs, markdown tables, and charts/diagrams.
• AI-Driven Visual Descriptions:
  • Extracted visuals are summarized using multimodal models (qwen/qwen3.8-27b, llama-3.2-11b-vision-preview) with automatic retry logic.
• Pinecone Serverless Vector Store:
  • Automatically provisions and configures dimension-matched Pinecone indices with custom namespace isolation per document.
• Dynamic Dual-Path Generation:
  • If visual evidence is retrieved, the query routes to a multimodal Vision LLM with image attachments.
  • If text/tables only are retrieved, the query routes to an optimized text model for maximum throughput.
• Transparent Evidence & Citations:
  • Expandable visual inspection cards with extracted image previews, confidence scores, and source page citations.
• Interactive Streamlit UI:
  • Sidebar configuration for API keys, model selections, chunk parameters, and real-time ingestion progress.

---

📂 Project Structure

MultiModal-RAG/
├── .env.example                                    # Environment variable template
├── .gitignore                                      # Comprehensive Git ignore rules
├── README.md                                       # Project documentation
├── requirements.txt                                # Production dependencies
├── pyrightconfig.json                              # Pyright / IDE type-checking settings
│
├── app.py                                          # Streamlit interactive Multimodal Web App
├── rag_pipeline.py                                 # Core RAG pipeline logic & helper functions
│
├── data/                                           # Sample documents & PDF datasets
│   ├── NovaCore_Multimodal_Company_Report_2026.pdf # Sample corporate report
│   └── Multimodal_RAG_LangChain_Pinecone_With_Examples.pptx.pdf # Additional sample slides
│
├── notebooks/                                      # Research & exploration notebooks
│   └── Build_Multimodal_RAG_NovaCore.ipynb         # Step-by-step development notebook
│
├── novacore_extracted_images/                      # Pre-extracted visual elements for default demo
│   ├── page_1_img_1_4.png
│   ├── page_3_img_1_7.png
│   ├── page_4_img_1_10.png
│   └── ...
│
├── uploaded_documents/                             # Local storage for user-uploaded custom PDFs
└── extracted_images/                               # Runtime directory for extracted visual elements

---

🚀 Getting Started

Prerequisites

• Python: Version 3.10, 3.11, or 3.12.
• Package Manager: uv (recommended for 10-100x faster installs) or pip.
• API Keys:
  • Groq API Key
  • Pinecone API Key

1. Clone and Navigate

git clone <your-repository-url>
cd "MultiModal RAG"

2. Environment Setup (uv or venv)

Option A: Using uv (Fastest)

# Create virtual environment
uv venv .venv

# Activate environment
source .venv/bin/activate    # On macOS/Linux
# .venv\Scripts\activate     # On Windows

# Install all dependencies
uv pip install -r requirements.txt

Option B: Using Standard Python venv

python3 -m venv .venv
source .venv/bin/activate    # On macOS/Linux
pip install -r requirements.txt

3. Configure API Credentials

Create your local .env file by copying .env.example:

cp .env.example .env

Open .env and fill in your keys:

GROQ_API_KEY="gsk_your_groq_api_key"
PINECONE_API_KEY="pcsk_your_pinecone_api_key"
PINECONE_INDEX_NAME="novacore-multimodal-rag"
PINECONE_NAMESPACE="fy2026-demo"

Note: You can also enter API keys directly inside the Streamlit sidebar at runtime.

4. Launch the Web App

streamlit run app.py

The Streamlit interface will open in your browser at http://localhost:8501.

---

🔍 How It Works

1. Tri-Modal Extraction

When a PDF is ingested, extract_multimodal_documents() in rag_pipeline.py parses each page:

• Extracts narrative body text using PyMuPDF.
• Detects table structures, converts them into Pandas DataFrames, and converts them to Markdown tables.
• Extracts raster images, saves them to disk, and tracks page numbers and bounding boxes.

2. Visual AI Summarization

For each extracted image, describe_visual() calls a Groq multimodal model (qwen/qwen3.8-27b) to generate a concise, factual description capturing:

• Chart types, axes, labels, and trends.
• Specific values (peaks, lows, averages).
• Flowchart components and critical control points.

3. Vector Indexing

Each chunk (text, table, or visual summary) is transformed into a Document with metadata tags (page, modality, source, image_path) and embedded using sentence-transformers/all-MiniLM-L6-v2 into Pinecone Serverless.

4. Hybrid Retrieval & Dynamic Multimodal Generation

1. At query time, Pinecone retrieves the Top-$K$ most relevant chunks.
2. If any retrieved chunk is a visual, the pipeline extracts the corresponding image files, encodes them as base64 Data URIs, and invokes Groq Vision.
3. If no visual is present, the pipeline invokes Groq Text LLM (openai/gpt-oss-20b or llama-3.3-70b-versatile) for instant response generation.

---

💬 Sample Demo Queries

You can test the system against the included NovaCore Systems FY2026 Report:

Question Category	Sample Query	Modality Tested
Product Profitability	"Which product has the highest gross margin, and what does the product visual say about it?"	Visual + Table (AtlasEdge X1 @ 46%)
Regional Performance	"Which region grew fastest, and which region had the highest customer satisfaction score?"	Table + Bar Chart
Operational Quality	"According to the supply chain diagram, where is the critical quality-control point?"	Visual Diagram (Singapore Quality Lab)
Sustainability Mix	"What percentage of the Penang facility electricity came from solar and renewables?"	Pie Chart (Solar 34%, Renewables 15%)
Support Speed	"What was the support resolution time in August compared to January?"	Line Graph (14.2h down to 8.1h)

---

⚙️ Configuration Reference

Parameter	Default Value	Description
Embedding Model	sentence-transformers/all-MiniLM-L6-v2	Dense vector embedding model (384 dimensions)
Text LLM	openai/gpt-oss-20b	High-speed LLM for text and tabular answering
Vision LLM	qwen/qwen3.8-27b	Multimodal model for visual analysis & visual Q&A
Pinecone Index	novacore-multimodal-rag	Serverless vector index name (auto-sanitized)
Top-K Retrieval	5	Number of context chunks retrieved per query
Max Visual Attachments	3	Maximum number of images passed into Vision LLM per query

---

🛠 Troubleshooting

Pinecone Index Name Error

Pinecone index names must contain only lowercase alphanumeric characters and hyphens (max 45 chars). The system automatically sanitizes index names using sanitize_index_name(), but ensure custom names do not consist solely of special characters.

Groq Rate Limits / Quotas

Vision LLM inference may hit per-minute token limits on free tiers. The pipeline includes automatic exponential backoff retries (describe_visual(..., max_retries=2)). If you encounter rate limits during ingestion, reduce page ranges or process in smaller batches.

Type Checking / Linter Warnings

The project includes pre-configured pyrightconfig.json. To run type check across all pipeline files:

uv run --with pyright pyright rag_pipeline.py app.py

---

📄 License

This project is licensed under the MIT License. See LICENSE for details.