GitHub Codebase Analyzer

A Retrieval-Augmented Generation (RAG) application that lets users ask natural-language questions about GitHub repositories. It retrieves relevant source code and documentation chunks and uses Google Gemini to generate context-grounded answers.

Features

• GitHub repository ingestion to retrieve project files for analysis.
• Code-aware text chunking with metadata to preserve file and location context.
• Semantic embeddings using Sentence Transformers (all-MiniLM-L6-v2).
• Persistent vector storage with ChromaDB.
• RAG-based question answering powered by LangChain and Google Gemini.
• Interactive Streamlit interface for repository analysis.
• Retrieval experiments comparing similarity search and Maximum Marginal Relevance (MMR).
• Evaluation framework with a 22-question reference dataset and Ragas evaluation scripts.

Architecture

GitHub Repository → Repository Loader → Code-Aware Chunking → Sentence Transformer Embeddings → ChromaDB → Retriever → Retrieved Context + User Question → Google Gemini → Answer

Technology Stack

Python · Streamlit · LangChain · ChromaDB · Sentence Transformers · Google Gemini · Ragas

Project Structure

github-codebase-analyzer/
├── app.py
├── github_loader.py
├── vector_store.py
├── rag_chain.py
├── requirements.txt
├── .env.example
├── .gitignore
└── evaluation/
    ├── dataset.json
    ├── evaluate_rag.py
    ├── score_baseline.py
    ├── compare_retrieval.py
    ├── check_retrieval_coverage.py
    ├── test_similarity_retrieval.py
    ├── test_similarity_answer.py
    └── score_similarity_test.py

Generated databases and evaluation outputs are excluded from version control.

Setup

Prerequisites

• Python installed
• A Google Gemini API key
• Internet access for GitHub requests, model downloads, and Gemini calls

1. Clone the repository

git clone https://github.com/Vedant-Salve/github-codebase-analyzer.git
cd github-codebase-analyzer

2. Create and activate a virtual environment

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

3. Install dependencies

python -m pip install -r requirements.txt

4. Configure environment variables

Create a local configuration file:

Copy-Item .env.example .env

Set your own GOOGLE_API_KEY in .env. You may optionally configure GITHUB_TOKEN and GEMINI_MODEL.

Never commit your .env file or expose API keys.

5. Run the application

python -m streamlit run app.py

Open the local URL printed by Streamlit in your browser.

Evaluation

The evaluation/ directory contains a reference dataset of 22 questions, baseline answer generation, Ragas scoring, and retrieval comparison scripts.

Metrics explored include:

• Faithfulness
• Answer relevancy
• Context precision
• Context recall

Retrieval experiments compare MMR and similarity search. Preliminary keyword-coverage checks reached 60/63 checks for both methods, but keyword coverage is only a diagnostic and does not establish answer correctness.

Small Ragas smoke tests and individual-question experiments are preliminary results, not a final benchmark. Evaluation requires a configured Gemini API key and may incur API usage costs.

Limitations

• Answer quality depends on the relevance and completeness of retrieved context.
• First-time indexing downloads the embedding model.
• Large repositories may take longer to index.
• Gemini requests depend on API availability and quotas.
• The local vector database must be created when setting up a fresh environment.

Future Improvements

• Add automated unit and integration tests.
• Improve source citations and line-level references in generated answers.
• Evaluate retrieval with validated semantic relevance checks.
• Add performance benchmarks and indexing optimizations.
• Expand language support and large-repository handling.

Author

Vedant Salve