FixPilot is an AI-powered IT support assistant that answers employee questions using company documents. If the documents do not have a helpful answer, it can search the web. It shows how it found each answer and lets authorized staff add new documents. Built with LangGraph, FastAPI, Pinecone, Groq, Tavily, HTML, CSS, and JavaScript.
NexoraRetail, a fictional 3,000-employee retail company.
The internal IT team maintains many documents: VPN instructions, password rules, MFA policy, software installation rules, laptop troubleshooting guides, and service-desk runbooks.
Employees still create repetitive support tickets because:
- They do not know where the correct document is.
- Traditional keyword search returns too many results.
- A normal chatbot may hallucinate an answer.
- Internal documents can be incomplete or outdated.
- Some questions require current vendor information from the public web.
An employee asks:
“How do I connect to the company VPN from home?”
The answer exists in the private company KB, so the system should not search the public internet.
Another employee asks:
“What is the latest Microsoft Teams outage guidance?”
The internal KB may not contain current outage information. The system should recognize weak private evidence, use an external search, grade that evidence, and answer with an external-source warning.
Build a secure IT Support Copilot that:
- Searches trusted private knowledge first.
- Checks whether retrieved evidence is good enough.
- Uses web search only when private knowledge is insufficient.
- Rewrites weak queries and retries.
- Generates grounded answers.
- Shows the decision path for transparency and debugging.
- Lets authorized staff add new company documents.
A Forward Deployed Engineer does more than create an LLM notebook. The FDE must turn the customer's problem into a usable product.
Customer Problem
↓
Discovery & Requirements
↓
Solution Architecture
↓
Data / Knowledge Integration
↓
Agentic RAG Development
↓
API Development
↓
User Interface
↓
Security + Audit + Testing
↓
Deployment
↓
Observe + Improve
This repository demonstrates each layer.
┌─────────────────────┐
│ Employee / User │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ HTML/CSS/JavaScript │
│ Web UI │
└──────────┬──────────┘
│ POST /api/chat
▼
┌─────────────────────┐
│ FastAPI │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ LangGraph │
│ Agentic RAG Control │
└──────────┬──────────┘
│
┌──────────────┴──────────────┐
│ │
▼ ▼
┌───────────────┐ ┌──────────────┐
│ Private KB │ │ Tavily Web │
│ Pinecone │ │ Search │
└───────┬───────┘ └──────┬───────┘
│ │
└──────────────┬──────────────┘
▼
┌────────────────┐
│ Groq LLM │
│ Grounded Answer│
└────────────────┘
This follows the same core implementation pattern as the reference notebook.
Question
↓
[1] Route Question
├── Greeting / simple chat ─────────────→ Direct Answer
│
└── IT support question
↓
[2] Retrieve from Private Pinecone KB
↓
[3] Grade Private Evidence
┌────────┴────────┐
│ │
GOOD WEAK
│ │
▼ ▼
Generate from KB [4] Tavily Web Search
↓
[5] Grade Web Evidence
┌────┴─────┐
│ │
GOOD WEAK
│ │
▼ ▼
Generate Web [6] Rewrite Query
↓
Retry Private KB
↓
Max retry reached?
↓
Insufficient Evidence
Normal RAG does approximately this:
Question → Retrieve → Generate
This project makes decisions:
Question
→ Route
→ Retrieve
→ Evaluate evidence
→ Choose KB or Web
→ Rewrite if needed
→ Retry
→ Generate grounded answer
The system therefore controls what to do next based on its current state.
| Layer | Technology | Purpose |
|---|---|---|
| Agent workflow | LangGraph | Stateful routing and conditional decisions |
| LLM | Groq | Routing, grading, rewriting, answer generation |
| Embeddings | HuggingFace all-MiniLM-L6-v2 |
Local 384-dimensional embeddings |
| Vector DB | Pinecone | Private enterprise knowledge base |
| External search | Tavily | Fallback when company KB is insufficient |
| API | FastAPI | Backend and REST endpoints |
| Frontend | HTML/CSS/JavaScript | Employee-facing interface |
| Audit | SQLite | Basic decision-path logging |
| Packaging | Docker | Reproducible deployment |
FixPilot/
│
├── app/
│ ├── api/
│ │ └── routes.py # Chat, health and ingestion APIs
│ │
│ ├── core/
│ │ ├── config.py # Environment configuration
│ │ └── logging.py # Logging configuration
│ │
│ ├── rag/
│ │ ├── state.py # LangGraph state + structured decisions
│ │ ├── vectorstore.py # Pinecone + embeddings
│ │ └── workflow.py # Complete Agentic RAG graph
│ │
│ ├── services/
│ │ ├── audit.py # SQLite query audit
│ │ └── ingestion.py # PDF/TXT/MD/DOCX loading + chunking
│ │
│ └── main.py # FastAPI application
│
├── data/
│ └── sample_kb/
│ ├── company_it_handbook.md
│ └── service_desk_runbook.md
│
├── static/
│ ├── css/style.css
│ └── js/app.js
│
├── templates/
│ └── index.html
│
├── tests/
│ └── test_ingestion.py
│
├── uploads/
├── .env.example
├── Dockerfile
├── ingest_sample_kb.py
├── requirements.txt
├── run.py
└── README.md
python -m venv venvWindows:
venv\Scripts\activatemacOS/Linux:
source venv/bin/activatepip install -r requirements.txtCreate a .env file in the project root directory with the following variables:
# LLM API Keys
GROQ_API_KEY=your_groq_api_key_here
OPENAI_API_KEY=your_openai_api_key_here
# External Search
TAVILY_API_KEY=your_tavily_api_key_here
# Vector Database (Pinecone)
PINECONE_API_KEY=your_pinecone_api_key_here
PINECONE_INDEX_NAME=fde-it-support-rag
PINECONE_NAMESPACE=company-it-kb
# LLM Models
GROQ_MODEL=openai/gpt-oss-20b
OPENAI_MODEL=gpt-4o-mini
EMBEDDING_MODEL=text-embedding-3-small
# Security
ADMIN_API_KEY=change-me-in-production
# Application Settings
APP_ENV=development| Variable | Description | Example | Required |
|---|---|---|---|
GROQ_API_KEY |
API key for Groq LLM (routing, grading, generation) | gsk_... |
✅ Yes |
OPENAI_API_KEY |
API key for OpenAI (embeddings and fallback LLM) | sk-... |
✅ Yes |
TAVILY_API_KEY |
API key for Tavily web search | tvly-... |
✅ Yes |
PINECONE_API_KEY |
API key for Pinecone vector database | pckey-... |
✅ Yes |
PINECONE_INDEX_NAME |
Pinecone index name | fde-it-support-rag |
fde-it-support-rag) |
PINECONE_NAMESPACE |
Pinecone namespace for document isolation | company-it-kb |
company-it-kb) |
GROQ_MODEL |
Groq model identifier | openai/gpt-oss-20b |
openai/gpt-oss-20b) |
OPENAI_MODEL |
OpenAI model identifier | gpt-4o-mini |
gpt-4o-mini) |
EMBEDDING_MODEL |
Embedding model for vectorization | text-embedding-3-small |
text-embedding-3-small) |
ADMIN_API_KEY |
Secret key for admin endpoints (document ingestion) | your-secure-key |
change-me) |
APP_ENV |
Application environment | development or production |
development) |
- Never commit
.envto version control. Add it to.gitignore. - Change
ADMIN_API_KEYto a secure random value in production. - Use environment variables instead of hardcoding secrets.
- Restrict API key access to only the services that need them.
python ingest_sample_kb.pyThis demonstrates the RAG ingestion pipeline:
Company Documents
↓
Load Documents
↓
Chunk Text
↓
HuggingFace Embeddings
↓
Pinecone Vector Database
python run.pyOpen:
http://127.0.0.1:8000
FastAPI API docs:
http://127.0.0.1:8000/docs
The evaluation harness compares the evidence-aware LangGraph workflow with a
basic retrieve-and-generate baseline using the same Pinecone retriever and LLM.
It records source selection, workflow traces, citations, and elapsed time for
each answer. The starter benchmark in
evaluation/questions.json is based on the
checked-in sample knowledge-base documents.
Index the sample documents into the same Pinecone index and namespace configured
in .env, then run the paired evaluation from the repository root:
python ingest_sample_kb.py
python -m scripts.evaluate_rag runThe run requires working OpenAI and Pinecone credentials; cases expected to use
web fallback also require a Tavily key. Raw answers and traces are written to
evaluation/results/latest.json. That folder is git-ignored because results
contain generated answers. Review the expected labels and answers against the
knowledge base you actually indexed before using the scores.
For each basic_rag and agentic_rag answer in the results file, annotate:
answer_correct:trueonly if the answer accurately covers the case'sexpected_answer_points; otherwisefalse.unsupported_claims:trueif the answer makes at least one factual claim unsupported by its retrieved or cited evidence; otherwisefalse. A refusal with no unsupported factual claims should befalse.
Calculate and save the report after annotation:
python -m scripts.evaluate_rag scoreThe report includes source-selection accuracy, web-fallback precision and
recall, human-scored answer accuracy, human-scored unsupported-claim rate, mean
latency, and the agentic-vs-baseline change. Answer-quality values remain N/A
until manually annotated; the unsupported-claim rate is a human-reviewed proxy,
not an automatic hallucination detector. Keep the raw run, annotations, dataset
version, and scoring rubric when quoting results. Include the benchmark size
and avoid generalizing from this small sample to all employee questions.
Measure document extraction separately using the checked-in
evaluation/ingestion_cases.json manifest:
python -m scripts.evaluate_ingestionEach file passes only when all its listed expected_text fragments are
present in extracted text. The current local manifest checks two PDFs from
uploads/, two checked-in Markdown files, and TXT/DOCX fixtures. This is a
format smoke test, not a statistically representative parsing benchmark or a
guarantee for every PDF/TXT/DOCX layout. Add more representative documents and
expected fragments before making broader claims. The report records case IDs
and aggregate counts, not extracted document text.
Run metric-calculation tests without calling external services:
python -m unittest discover -s tests -vAsk:
How do I connect to the company VPN from home?
Expected path:
Router → KB
Private KB Retrieval
KB Grade → GOOD
Generate from Private KB
Teaching point:
Trusted internal company knowledge is preferred. No public web call is necessary.
Ask:
Can IT support ask me to share my MFA code?
The internal handbook says employees must never share passwords or MFA codes with support personnel.
Expected path:
Router → KB
Private KB Retrieval
KB Grade → GOOD
Private KB Answer
Teaching point:
RAG allows the model to answer using company-specific information it was never trained on.
Ask:
What is the latest Microsoft Teams outage guidance?
Expected path when internal documents do not answer it:
Router → KB
Private KB Retrieval
KB Grade → WEAK
Tavily Search
Web Grade → GOOD
Web Answer
Teaching point:
Agentic RAG can choose another information source rather than blindly answering from irrelevant vectors.
Ask a deliberately vague question such as:
My work communication app is acting strange after the new update. What should I do?
If both KB and first external retrieval are weak, the workflow can rewrite the query and retry.
Teaching point:
Retrieval failure does not immediately mean failure. An agent can improve the search query and try again while using a retry guard to avoid loops.
Ask:
Hello!
Expected path:
Router → DIRECT
Direct Answer
Teaching point:
Not every message should trigger expensive vector search or web search.
The sidebar contains Add Company Document.
Use the ADMIN_API_KEY from .env.
Upload one of:
.pdf.txt.md.docx
The backend performs:
Upload
↓
Validate Type
↓
Load Text
↓
Recursive Chunking
↓
Embeddings
↓
Pinecone Indexing
↓
Immediately Available for Retrieval
This is useful in an FDE demonstration because the customer does not want to edit Python every time a new policy is published.
Health check.
Request:
{
"question": "How do I reset my company password?"
}Response contains:
{
"answer": "...",
"source_used": "private_kb",
"trace": [
"Router → KB",
"Private KB retrieval → 4 chunks",
"KB evidence grade → GOOD",
"Answer generation → PRIVATE KB"
],
"citations": []
}Protected by the X-Admin-Key request header.
Uploads a document and adds its chunks to Pinecone.
Contributions, bug reports, and suggestions are welcome. To contribute:
- Fork the repository.
- Create a feature branch (
git checkout -b feature/your-change). - Make your changes and commit them with a clear message.
- Push your branch and open a pull request describing your changes.
This project is licensed under the Apache License 2.0.