Skip to content
Day-RavalPublic

About

Takes a fridge photo and returns ranked recipes with explanations

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

ย 

History

48 Commits

Folders and files

NameName
Last commit message
Last commit date
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

๐ŸงŠ๐Ÿณ DishSense

Fridge photo in โ†’ ranked, explainable recipes out.

DishSense is a FRIDGE-RAG multimodal system that combines:

  • Computer Vision (YOLOv8 + DETR + CLIP) to detect ingredients from a fridge photo.
  • RAG retrieval (Sentence-Transformers + ChromaDB) to fetch relevant recipes.
  • LLM reranking (OpenAI GPT-4o-mini / Local-LLM Reranker) to produce explainable final recommendations.
  • FastAPI + Streamlit for API-first serving plus a demo UI.

Python FastAPI Streamlit Vector DB

๐Ÿ”„ Latest updates snapshot

To keep this README aligned with today's added/modified files and commits:

  • Added src/evaluation.py, a retrieval/ranking evaluation harness for P@K, Recall@K, and nDCG@K.
  • Updated src/config.py and .env.example with clearer compute/reranker configuration, including documented offline/local reranker usage.
  • Enhanced src/rag/reranker.py and src/rag/ingest.py behavior for stronger ranking/ingestion robustness.
  • Updated dashboard/app.py integration details to align with the latest API request flow.
  • Removed docs/images/workflow-overview.svg as part of deleting the docs/images directory.

๐Ÿ“ธ Project image context

Sample fridge image used for end-to-end testing:

Sample fridge image used by FRIDGE-RAG

Why this project is useful

Most demos stop at โ€œobject detection.โ€ This project is built like a small production prototype:

  • Ensemble detection for stronger ingredient coverage.
  • Semantic retrieval with optional calorie/time filters.
  • Human-readable ranking reasons + missing ingredients.
  • Clear module boundaries (API, pipeline, vision, retrieval, UI, tests).

๐Ÿงญ End-to-end architecture

๐ŸŸข Online inference flow (production-style)

flowchart LR
    subgraph Clients["๐Ÿ–ฅ๏ธ Client Layer"]
        UI["๐Ÿงช Streamlit UI"]
        APIClient["๐Ÿ“ฑ API Client"]
    end

    subgraph Serving["๐Ÿšฆ Serving Layer"]
        FastAPI["โšก FastAPI /recommend"]
        Pipeline["๐Ÿง  src.pipeline.recommend_from_photo"]
    end

    subgraph Vision["๐Ÿ‘๏ธ Vision Layer"]
        YOLO["๐Ÿ“ฆ YOLOv8"]
        DETR["๐Ÿงฒ DETR"]
        CLIP["๐Ÿงท CLIP"]
        Fusion["๐Ÿงฎ Ensemble Fusion"]
    end

    subgraph Retrieval["๐Ÿ”Ž Retrieval + Ranking Layer"]
        Embed["๐Ÿงพ Embeddings (MiniLM)"]
        Chroma["๐Ÿ—‚๏ธ ChromaDB Retriever"]
        Rerank["๐Ÿค– LLM Reranker (GPT-4o-mini)"]
    end

    subgraph Ops["๐Ÿ›ก๏ธ Production Controls"]
        Config["โš™๏ธ Env + thresholds"]
        Guardrails["๐Ÿšจ Validation + fallback"]
        Logs["๐Ÿ“Š Logging + health checks"]
    end

    UI --> FastAPI
    APIClient --> FastAPI
    FastAPI --> Pipeline

    Pipeline --> YOLO
    Pipeline --> DETR
    Pipeline --> CLIP
    YOLO --> Fusion
    DETR --> Fusion
    CLIP --> Fusion

    Fusion --> Embed
    Embed --> Chroma
    Chroma --> Rerank

    Config -.-> Pipeline
    Guardrails -.-> Rerank
    Logs -.-> FastAPI

    Rerank --> Output["โœ… Ranked recipes + explanations"]
Loading

๐ŸŸฃ Offline indexing flow (data pipeline)

flowchart LR
    Dataset["๐Ÿงบ Kaggle Food.com CSV"] --> BuildScript["๐Ÿ› ๏ธ scripts/build_vectordb.py"]
    BuildScript --> Ingest["๐Ÿ“ฅ src.rag.ingest.ingest_recipes"]
    Ingest --> Chunk["๐Ÿงฉ Clean + chunk + metadata"]
    Chunk --> EmbedOffline["๐Ÿงพ Embedding generation"]
    EmbedOffline --> Persist["๐Ÿ’พ ChromaDB @ data/recipe_db"]
Loading

Repository map (all tracked files)

FRIDGE-RAG/
โ”œโ”€โ”€ README.md
โ”œโ”€โ”€ requirements.txt
โ”œโ”€โ”€ .env.example
โ”œโ”€โ”€ Sample-image.jpg
โ”œโ”€โ”€ api/
โ”‚   โ”œโ”€โ”€ __init__.py
โ”‚   โ”œโ”€โ”€ main.py                 # FastAPI app (/health, /recommend)
โ”‚   โ””โ”€โ”€ schemas.py              # Pydantic response models
โ”œโ”€โ”€ dashboard/
โ”‚   โ”œโ”€โ”€ __init__.py
โ”‚   โ””โ”€โ”€ app.py                  # Streamlit UI client for API
โ”œโ”€โ”€ scripts/
โ”‚   โ””โ”€โ”€ build_vectordb.py       # CLI wrapper to ingest recipe CSV into Chroma
โ”œโ”€โ”€ src/
โ”‚   โ”œโ”€โ”€ __init__.py
โ”‚   โ”œโ”€โ”€ config.py               # global constants (models, thresholds, paths)
โ”‚   โ”œโ”€โ”€ pipeline.py             # online orchestration entrypoint
โ”‚   โ”œโ”€โ”€ evaluation.py           # retrieval/ranking metrics harness (P@K, Recall@K, nDCG)
โ”‚   โ”œโ”€โ”€ rag/
โ”‚   โ”‚   โ”œโ”€โ”€ __init__.py
โ”‚   โ”‚   โ”œโ”€โ”€ ingest.py           # CSV parsing + embedding + Chroma upsert
โ”‚   โ”‚   โ”œโ”€โ”€ retriever.py        # semantic query + metadata filters
โ”‚   โ”‚   โ””โ”€โ”€ reranker.py         # OpenAI LLM reranking + enrichment
โ”‚   โ””โ”€โ”€ vision/
โ”‚       โ”œโ”€โ”€ __init__.py
โ”‚       โ”œโ”€โ”€ yolo_detector.py    # YOLO detector wrapper
โ”‚       โ”œโ”€โ”€ detr_detector.py    # DETR detector wrapper
โ”‚       โ”œโ”€โ”€ clip_detector.py    # CLIP zero-shot detector wrapper
โ”‚       โ””โ”€โ”€ ensemble.py         # confidence fusion + dedupe
โ””โ”€โ”€ tests/
    โ”œโ”€โ”€ __init__.py
    โ”œโ”€โ”€ test_pipeline.py        # pipeline behavior with mocks
    โ”œโ”€โ”€ test_vision.py          # ensemble logic tests
    โ””โ”€โ”€ test_rag.py             # retriever behavior tests

๐Ÿงฑ Core components

1) ๐Ÿ‘๏ธ Vision detection (src/vision/*)

  • YOLOv8: fast baseline detector.
  • DETR: transformer detector, useful for different detection bias.
  • CLIP: zero-shot ingredient matching over candidate vocabulary.
  • Ensemble fusion:
    • normalize labels,
    • average confidence,
    • apply multi-model boost,
    • threshold and rank.

2) ๐Ÿ”Ž Retrieval (src/rag/retriever.py)

  • Embeds ingredient query with all-MiniLM-L6-v2.
  • Runs vector query in ChromaDB.
  • Supports metadata filters:
    • max_calories
    • max_minutes

3) ๐Ÿค– LLM reranking (src/rag/reranker.py)

  • Sends candidate summaries + user preferences to OpenAI.
  • Returns JSON-ranked recipes with:
    • coverage_pct
    • missing_ingredients
    • nutrition_score
    • reason
  • Fallback: similarity-order if JSON parse fails.

4) โšก API (api/main.py)

  • GET /health: model/db health summary.
  • POST /recommend: image + optional constraints to recipe recommendations.

5) ๐Ÿ–ฅ๏ธ Dashboard (dashboard/app.py)

  • Upload image + sliders for filters.
  • Displays ingredient list and expandable recommendation cards.

๐Ÿ› ๏ธ Setup

1) ๐Ÿ“‹ Prerequisites

  • Python 3.10+
  • Kaggle API credentials (for dataset download)
  • OpenAI API key

2) ๐Ÿ“ฆ Install dependencies

git clone <your-repo-url>
cd FRIDGE-RAG
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

3) โš™๏ธ Configure environment

cp .env.example .env

Set key in .env:

OPENAI_API_KEY=your_key_here

4) ๐Ÿ” Configure Kaggle credentials

mkdir -p ~/.kaggle
echo '{"username":"YOUR_KAGGLE_USERNAME","key":"YOUR_KAGGLE_KEY"}' > ~/.kaggle/kaggle.json
chmod 600 ~/.kaggle/kaggle.json

5) ๐Ÿงฌ Build the vector DB

python scripts/build_vectordb.py

Useful options:

python scripts/build_vectordb.py --help
python scripts/build_vectordb.py --limit 10000
python scripts/build_vectordb.py --batch-size 64

โ–ถ๏ธ Run locally

โšก Start API

uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload

๐Ÿ–ฅ๏ธ Start dashboard

streamlit run dashboard/app.py

โค๏ธ Quick health check

curl http://localhost:8000/health

API reference

GET /health

Response fields:

  • status
  • models_loaded
  • db_ready
  • recipe_count

POST /recommend

Multipart form fields:

  • photo (required): jpeg/png/webp
  • preferences (optional text)
  • max_calories (optional int; 0 = no limit)
  • max_minutes (optional int; 0 = no limit)
  • top_n (1-10)

Example curl:

curl -X POST http://localhost:8000/recommend \
  -F "photo=@Sample-image.jpg" \
  -F "preferences=high protein, low carb" \
  -F "max_calories=600" \
  -F "max_minutes=30" \
  -F "top_n=5"

Testing

Run all tests:

pytest tests/ -v

Targeted runs:

pytest tests/test_vision.py -v
pytest tests/test_rag.py -v
pytest tests/test_pipeline.py -v

Common enhancements to add next

  1. Caching layer (CAG-style memory/cache) for recurring user sessions.
  2. Expand the evaluation harness with dataset slices, baselines, and automated reports.
  3. Observability: tracing spans for detection/retrieval/rerank latency.
  4. Containerization: Docker + Compose for one-command local startup.
  5. Allergen & diet taxonomy filters beyond free-text preference string.
  6. CI pipeline with lint/test checks and model-mocking for speed.
  7. MLflow integration with experiment tracking/test checks and model-versioning for maintainability.

Notes on local data/.

data/ is generated locally during ingestion and is usually gitignored. That keeps the repository small and avoids committing generated vector DB artifacts.


License and contribution

If you open PRs, include:

  • motivation,
  • expected quality/runtime impact,
  • testing evidence.

Contributions are welcome.


โœ… README compatibility note

This revamp intentionally preserves all existing README content and adds an update layer for recent commits, plus contextual images where applicable.

About

Takes a fridge photo and returns ranked recipes with explanations

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages