Evaluate your LLM's response with Prometheus and GPT4 💯
-
Updated
Apr 25, 2025 - Python
#
llm-as-evaluator
Here are 18 public repositories matching this topic...
Deliver safe & effective language models
nlp artificial-intelligence benchmarks benchmark-framework model-assessment ai-safety mlops responsible-ai ml-safety trustworthy-ai ethics-in-ai ml-testing large-language-models llm ai-testing llm-test llm-evaluation-toolkit llm-as-evaluator llm-testing
-
Updated
Aug 28, 2026 - Python
[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
benchmark regex reliability evaluation dataset gpt phi large-language-models llm cc-by-nc-nd-4 open-compass chatglm qwen lm-evaluation llm-as-a-judge llm-as-evaluator xfinder reliable-evaluation key-answer-extraction judge-model
-
Updated
Nov 14, 2025 - Python
This is the repo for the survey of Bias and Fairness in IR with LLMs.
information-retrieval recommender-systems bias ir fairness large-language-models llm chatgpt llm4rec llm4rs llm-as-a-judge llm-as-evaluator llm4ir
-
Updated
Sep 4, 2025
Code and data for "Timo: Towards Better Temporal Reasoning for Language Models" (COLM 2024)
temporal-reasoning sota-model llms rlhf rlaif llm-as-a-judge llm-as-evaluator self-critic-framework colm2024
-
Updated
Oct 23, 2024 - Python
Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"
nlp evaluation bias bias-detection llm llms llm-evaluation llms-benchmarking llm-as-judge llm-as-a-judge llm-as-evaluator status-published area-evaluation
-
Updated
Feb 16, 2024 - Jupyter Notebook
LLM-as-judge evals as Semantic Kernel Plugins
-
Updated
Feb 21, 2026 - C#
Starter Kit for Auto-Judge developers
-
Updated
Oct 8, 2026 - Python
An sdk and framework for evaluating and comparing multiple model outputs using configurable LLM-based jurors
python open-source eval evaluation-metrics evaluation-framework ai-evaluation evals llm-as-a-judge llm-as-evaluator ai-evaluation-tools ai-evaluation-metrics ai-evaluation-framework evaluation-harness ai-evals ai-eval llm-jury
-
Updated
Sep 6, 2026 - Python
PrometheusLLM is a unique transformer architecture inspired by dignity and recursion. This project aims to explore new frontiers in AI research and welcomes contributions from the community. 🐙🌟
deep-learning mcp evaluation pipelines tracing language-model self-organization cognitive-architecture hermeneutics philosophy-of-mind gpt4 llm llmops ollama litellm llm-as-evaluator autopoietic-systems prompt-logging
-
Updated
Oct 12, 2026 - Python
Use groq for evaluations
-
Updated
Jul 11, 2024 - Python
Evaluation system for computer-use agents that uses LLMs to assess agent performance on web browsing and interaction tasks. This judge system reads screenshots, agent trajectories, and final results to provide detailed scoring and feedback.
computer-vision evaluation evaluation-metrics performance-testing evaluation-framework web-agent llm-as-judge llm-as-a-judge llm-as-evaluator computer-use
-
Updated
Oct 28, 2025 - Python
A tiny, lightweight library for LLM-as-a-Judge evaluations on vLLM-hosted models.
-
Updated
Sep 29, 2025 - Python
Automated evaluation of llm generated responses on aws
-
Updated
Apr 11, 2025 - Python
project from UDACITY second module 'agentic workflow' as part of Agentic AI Nanodegree
-
Updated
Aug 16, 2025 - Python
Antibodies for LLMs hallucinations (grouping LLM as a judge, NLI, reward models)
-
Updated
Jun 13, 2024 - Python
Real-time voice agent with LLM tool routing (RAG / web search / direct), streaming STT→LLM→TTS, and mid-sentence barge-in. Includes a commit-tagged eval harness. Groq, Cartesia, LangGraph, Silero VAD.
text-to-speech speech-to-text whisper rag voice-ai voice-agent silero-vad chromadb llm-agents langgraph tavily llm-as-judge llm-as-evaluator cartesia
-
Updated
Nov 1, 2026 - Python
Python prompt engineering and AI evaluation toolkit with structured output, hallucination controls, JSON extraction and LLM-as-a-judge workflows.
python json json-api openai openai-api ai-evaluation llm prompt-engineering llm-tools llm-evaluation llm-agents llm-as-judge llm-as-a-judge llm-as-evaluator ai-evaluation-tools prompt-engineering-course prompt-engineering-tool prompt-engineering-tools ai-evaluations ai-evaluation-engineer
-
Updated
Oct 7, 2026 - Python
Add this topic to your repo
To associate your repository with the llm-as-evaluator topic, visit your repo's landing page and select "manage topics."