Fridge photo in โ ranked, explainable recipes out.
DishSense is a FRIDGE-RAG multimodal system that combines:
- Computer Vision (YOLOv8 + DETR + CLIP) to detect ingredients from a fridge photo.
- RAG retrieval (Sentence-Transformers + ChromaDB) to fetch relevant recipes.
- LLM reranking (OpenAI GPT-4o-mini / Local-LLM Reranker) to produce explainable final recommendations.
- FastAPI + Streamlit for API-first serving plus a demo UI.
To keep this README aligned with today's added/modified files and commits:
- Added
src/evaluation.py, a retrieval/ranking evaluation harness for P@K, Recall@K, and nDCG@K. - Updated
src/config.pyand.env.examplewith clearer compute/reranker configuration, including documented offline/local reranker usage. - Enhanced
src/rag/reranker.pyandsrc/rag/ingest.pybehavior for stronger ranking/ingestion robustness. - Updated
dashboard/app.pyintegration details to align with the latest API request flow. - Removed
docs/images/workflow-overview.svgas part of deleting thedocs/imagesdirectory.
Sample fridge image used for end-to-end testing:
Most demos stop at โobject detection.โ This project is built like a small production prototype:
- Ensemble detection for stronger ingredient coverage.
- Semantic retrieval with optional calorie/time filters.
- Human-readable ranking reasons + missing ingredients.
- Clear module boundaries (API, pipeline, vision, retrieval, UI, tests).
flowchart LR
subgraph Clients["๐ฅ๏ธ Client Layer"]
UI["๐งช Streamlit UI"]
APIClient["๐ฑ API Client"]
end
subgraph Serving["๐ฆ Serving Layer"]
FastAPI["โก FastAPI /recommend"]
Pipeline["๐ง src.pipeline.recommend_from_photo"]
end
subgraph Vision["๐๏ธ Vision Layer"]
YOLO["๐ฆ YOLOv8"]
DETR["๐งฒ DETR"]
CLIP["๐งท CLIP"]
Fusion["๐งฎ Ensemble Fusion"]
end
subgraph Retrieval["๐ Retrieval + Ranking Layer"]
Embed["๐งพ Embeddings (MiniLM)"]
Chroma["๐๏ธ ChromaDB Retriever"]
Rerank["๐ค LLM Reranker (GPT-4o-mini)"]
end
subgraph Ops["๐ก๏ธ Production Controls"]
Config["โ๏ธ Env + thresholds"]
Guardrails["๐จ Validation + fallback"]
Logs["๐ Logging + health checks"]
end
UI --> FastAPI
APIClient --> FastAPI
FastAPI --> Pipeline
Pipeline --> YOLO
Pipeline --> DETR
Pipeline --> CLIP
YOLO --> Fusion
DETR --> Fusion
CLIP --> Fusion
Fusion --> Embed
Embed --> Chroma
Chroma --> Rerank
Config -.-> Pipeline
Guardrails -.-> Rerank
Logs -.-> FastAPI
Rerank --> Output["โ
Ranked recipes + explanations"]
flowchart LR
Dataset["๐งบ Kaggle Food.com CSV"] --> BuildScript["๐ ๏ธ scripts/build_vectordb.py"]
BuildScript --> Ingest["๐ฅ src.rag.ingest.ingest_recipes"]
Ingest --> Chunk["๐งฉ Clean + chunk + metadata"]
Chunk --> EmbedOffline["๐งพ Embedding generation"]
EmbedOffline --> Persist["๐พ ChromaDB @ data/recipe_db"]
FRIDGE-RAG/
โโโ README.md
โโโ requirements.txt
โโโ .env.example
โโโ Sample-image.jpg
โโโ api/
โ โโโ __init__.py
โ โโโ main.py # FastAPI app (/health, /recommend)
โ โโโ schemas.py # Pydantic response models
โโโ dashboard/
โ โโโ __init__.py
โ โโโ app.py # Streamlit UI client for API
โโโ scripts/
โ โโโ build_vectordb.py # CLI wrapper to ingest recipe CSV into Chroma
โโโ src/
โ โโโ __init__.py
โ โโโ config.py # global constants (models, thresholds, paths)
โ โโโ pipeline.py # online orchestration entrypoint
โ โโโ evaluation.py # retrieval/ranking metrics harness (P@K, Recall@K, nDCG)
โ โโโ rag/
โ โ โโโ __init__.py
โ โ โโโ ingest.py # CSV parsing + embedding + Chroma upsert
โ โ โโโ retriever.py # semantic query + metadata filters
โ โ โโโ reranker.py # OpenAI LLM reranking + enrichment
โ โโโ vision/
โ โโโ __init__.py
โ โโโ yolo_detector.py # YOLO detector wrapper
โ โโโ detr_detector.py # DETR detector wrapper
โ โโโ clip_detector.py # CLIP zero-shot detector wrapper
โ โโโ ensemble.py # confidence fusion + dedupe
โโโ tests/
โโโ __init__.py
โโโ test_pipeline.py # pipeline behavior with mocks
โโโ test_vision.py # ensemble logic tests
โโโ test_rag.py # retriever behavior tests
- YOLOv8: fast baseline detector.
- DETR: transformer detector, useful for different detection bias.
- CLIP: zero-shot ingredient matching over candidate vocabulary.
- Ensemble fusion:
- normalize labels,
- average confidence,
- apply multi-model boost,
- threshold and rank.
- Embeds ingredient query with
all-MiniLM-L6-v2. - Runs vector query in ChromaDB.
- Supports metadata filters:
max_caloriesmax_minutes
- Sends candidate summaries + user preferences to OpenAI.
- Returns JSON-ranked recipes with:
coverage_pctmissing_ingredientsnutrition_scorereason
- Fallback: similarity-order if JSON parse fails.
GET /health: model/db health summary.POST /recommend: image + optional constraints to recipe recommendations.
- Upload image + sliders for filters.
- Displays ingredient list and expandable recommendation cards.
- Python 3.10+
- Kaggle API credentials (for dataset download)
- OpenAI API key
git clone <your-repo-url>
cd FRIDGE-RAG
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtcp .env.example .envSet key in .env:
OPENAI_API_KEY=your_key_heremkdir -p ~/.kaggle
echo '{"username":"YOUR_KAGGLE_USERNAME","key":"YOUR_KAGGLE_KEY"}' > ~/.kaggle/kaggle.json
chmod 600 ~/.kaggle/kaggle.jsonpython scripts/build_vectordb.pyUseful options:
python scripts/build_vectordb.py --help
python scripts/build_vectordb.py --limit 10000
python scripts/build_vectordb.py --batch-size 64uvicorn api.main:app --host 0.0.0.0 --port 8000 --reloadstreamlit run dashboard/app.pycurl http://localhost:8000/healthResponse fields:
statusmodels_loadeddb_readyrecipe_count
Multipart form fields:
photo(required): jpeg/png/webppreferences(optional text)max_calories(optional int; 0 = no limit)max_minutes(optional int; 0 = no limit)top_n(1-10)
Example curl:
curl -X POST http://localhost:8000/recommend \
-F "photo=@Sample-image.jpg" \
-F "preferences=high protein, low carb" \
-F "max_calories=600" \
-F "max_minutes=30" \
-F "top_n=5"Run all tests:
pytest tests/ -vTargeted runs:
pytest tests/test_vision.py -v
pytest tests/test_rag.py -v
pytest tests/test_pipeline.py -v- Caching layer (CAG-style memory/cache) for recurring user sessions.
- Expand the evaluation harness with dataset slices, baselines, and automated reports.
- Observability: tracing spans for detection/retrieval/rerank latency.
- Containerization: Docker + Compose for one-command local startup.
- Allergen & diet taxonomy filters beyond free-text preference string.
- CI pipeline with lint/test checks and model-mocking for speed.
- MLflow integration with experiment tracking/test checks and model-versioning for maintainability.
data/ is generated locally during ingestion and is usually gitignored.
That keeps the repository small and avoids committing generated vector DB artifacts.
If you open PRs, include:
- motivation,
- expected quality/runtime impact,
- testing evidence.
Contributions are welcome.
This revamp intentionally preserves all existing README content and adds an update layer for recent commits, plus contextual images where applicable.
