PYTHIA is a knowledge graph question-answering (KGQA) engine that is designed to function as a plug-and-play solution for general-purpose and specialized knowledge graphs (KG), without requiring any finetuning or modification.
- Knowledge Graph Agnostic. PYTHIA is a plug-and-play solution that can easily be deployed over any Knowledge Graph.
- Zero-shot. PYTHIA does not require any finetuning or training data to function.
- State of the Art performance. PYTHIA achieves new strong performance in popular question answering benchmarks, surpassing previous zero-shot solutions and closing the gap to finetuned systems.
- [Aug 12, 2026] PYTHIA oral presentation at KDD '26 !
- [Aug 11, 2026] PYTHIA poster presentation at KDD '26 !
- [Aug 8, 2026] Paper published and available in the ACM Digital Library !
- [May 16, 2026] PYTHIA: Universal Question Answering over Knowledge Graphs accepted as a full research paper at KDD '26 (Research Track) !
The repository is organized as follows:
_other_systems/contains the versions of Think-on-Graph (ToG), Paths-over-Graph (PoG), GRASP and LLM baselines used in the evaluation with instructions on how to reproduce our results.datasets/contains the QA datasets used in the evaluation.indices/is the default location for placing system indices.logs-and-results/contains logs, raw results and metrics for the evaluations of PYTHIA and other systems.src/contains the source code of PYTHIA.tools/contains helper tools for creating indices, file manipulation and other auxiliary functions.config.yamlcontains the configuration of PYTHIA.
Additional information is provided in each sub-directory.
We recommend the use of Conda for dependency handling. The existence of a GPU is also recommended to speed-up embedding calculation.
To create the environment:
conda env create -f pythia.ymlTo activate:
conda activate pythiaAdditionally, you will need to install SimString in the same environment. Instructions can be found in the official website.
PYTHIA utilizes indices for fast and accurate entity linking. You can either build your own or utilize our pre-built indices. Index folders must be placed in the index_dir specified in config.yaml
If you choose to use pre-built indices, unzip their contents in index_dir. By default this is a directory named indices/ at the project root.
Resulting structure:
indices/bestiary/
indices/dbpedia10/
indices/freebase/For knowledge graphs that don't have pre-built indices, or if you want more control over their contents, you can build your own indices. The following example is for building indices over Freebase.
a) Extract information about classes, entities and predicates from the knowledge graph source files. To do so, use tools/elements_extractor. Additional information can be found in the tools/ directory.
java -jar target/elements-extractor-1.0-SNAPSHOT-jar-with-dependencies.jar \
-i ~/freebase.nt \
-o ~/extracts_freebase/ \
-kg freebase \
-l http://www.w3.org/2000/01/rdf-schema#label \
-npl \
-d http://rdf.freebase.com/ns/common.topic.description \
-f -fcNote: The use of -Xmx -Xss flags might be required if the process runs out of memory for the given knowledge graph and arguments.
This command takes an RDF dump of Freebase (~/freebase.nt) and generates a folder with TSV files (~/extracts_freebase).
b) Generate class and entity indices. To do so, use tools/all_indices/generator.py.
python generator.py --input-dir ~/extracts_freebase --output-dir ~/indices_freebasec) Move generated files to index_dir. Take the outputs of both commands (that is, the contents of ~/extracts_freebase/ and ~/indices_freebase/) and put them in a new directory in the index_dir specified in config.yaml.
d) Compute class-predicate statistics. To do so, use tools/class_predicates_extractor.
java -jar target/class-predicates-extractor-1.0-SNAPSHOT-jar-with-dependencies.jar -i ~/freebase.nt -cf ~/extracts_freebase/classes.tsv -o {PATH_TO_FREEBASE_INDEX_DIR}Note: Make sure that after running this computation the output files are named classes_predicates.tsv and classes_predicates_no_literals.tsv.
e) A knowledge graph structure index will be automatically created when you first use your index. This might take some time, but all subsequent uses will be much faster.
To setup your LLM you need to provide an API key (and/or change model or provider). To use the default configuration simply set an OpenAI API key in config.yaml or in an environment variable named OPENAI_API_KEY.
a) PYTHIA requires a live RDF endpoint to execute queries. This endpoint should hold the same information as what was given for index creation. We recommend the use of GraphDB to simplify setup.
b) PYTHIA requires setting up a standalone graph search server. We split this from the body of PYTHIA for additional flexibility. Instructions for setting up the graph search server can be found in tools/graph_minimizer/. After setting up the graph search server its location must be given to PYTHIA via either setting it in config.yaml or setting the GRAPH_SEARCH_SERVER environment variable.
Note: You can download pre-built search indices. To build the search indices see the relevant README.md file.
c) PYTHIA requires a GoST endpoint to function. GoST is a utility built using Apache Jena for query modification and formatting (you can see supported actions in the source code). Before running PYTHIA see how to run GoST (tools/gost).
After having setup dependencies, indices and API key, you are ready to execute PYTHIA. In the following example we run PYTHIA over WebQSP from the project root directory (which is placed in our HOME directory on a Linux system).
python -m src.evaluation.qa_engine.pythia_generator \
--index_name freebase \
--dataset webqsp \
--dataset_path ~/pythia/datasets/webqsp/data/WebQSP.test_clean.json \
--bela --topic_entities_dataset ~/pythia/logs-and-results/nerd/baselines/bela_webqsp.json \
--output_dir ~/pythia-results/ \
--endpoint_server localhost:7200Note: The use of BELA is optional, but is the default configuration. In this example we use pre-calculated BELA topic entities for package deployment simplicity. Alternatively, BELA can be ran as a standalone server (see tools/bela_endpoint).
After PYTHIA finishes with the generation of the results we can run our evaluation scripts to calculate a number of metrics.
python -m src.evaluation.qa_engine.sparql_eval \
--dataset webqsp \
--dataset_path ~/pythia/datasets/webqsp/data/WebQSP.test_clean.json \
--generated_file ~/pythia-results/PYTHIA_webqsp_results.json \
--endpoint_server localhost:7200In this example, results can be found in ~/pythia-results/.
An often overlooked aspect of reproducibility is the version of the underlying knowledge graph. To avoid this problem, we note the exact versions and sources of our knowledge graphs.
| Dataset | Knowledge Graph Version | Source |
|---|---|---|
| WebQSP | Latest Freebase dump | https://developers.google.com/freebase |
| CWQ | Latest Freebase dump | https://developers.google.com/freebase |
| QALD 9 | DBpedia 2016-10 | https://downloads.dbpedia.org/wiki-archive/dbpedia-version-2016-10.html |
| LC-QuAD | DBpedia 2016-04 | https://downloads.dbpedia.org/wiki-archive/dbpedia-version-2016-04.html |
| QALD 10 | QALD 10 Wikidata dump | https://zenodo.org/records/7496690 |
| BESTIARY | Knowledge graph from the official repository | https://github.com/danrd/sparqlgen |
For our experiments we use GraphDB 10.6.3.
This repository contains the source code of PYTHIA and of prior works used in our experimental evaluation. We also provide pre-built knowledge graph indices, the exact versions of the datasets used and information about sourcing knowledge graph files. These should enable full reproducibility (with the caveat of randomness that is inherent in LLMs).
For running experiments with PYTHIA see this README.md file as well as the individual README.md files in the tools/ directory.
For running experiments with ToG, PoG and GRASP see _other_systems.
For running LLM baseline experiments see src/baselines.
In addition to our evaluated files, we include raw outputs for all systems evaluated, to allow future research efforts to re-evaluate outputs without needing to re-run systems.
🔍 If you want to learn more about PYTHIA consider:
- Reading our KDD '26 paper !
- Taking a look at our poster and presentation slides !
📜 If you use PYTHIA in your research, please consider citing our work:
Human Readable Format
Sergios-Anestis Kefalidis, Konstantinos Plas, and Manolis Koubarakis. 2026. PYTHIA: Universal Question Answering over Knowledge Graphs. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '26). Association for Computing Machinery, New York, NY, USA, 2285–2296. https://doi.org/10.1145/3770855.3817968
BibTex Format
@inproceedings{kefalidis2026pythia,
author = {Kefalidis, Sergios-Anestis and Plas, Konstantinos and Koubarakis, Manolis},
title = {PYTHIA: Universal Question Answering over Knowledge Graphs},
booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
series = {KDD '26},
year = {2026},
pages = {2285--2296},
numpages = {12},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
location = {Republic of Korea},
isbn = {9798400722592},
doi = {10.1145/3770855.3817968},
url = {[https://doi.org/10.1145/3770855.3817968](https://doi.org/10.1145/3770855.3817968)},
keywords = {knowledge graph question answering, kg-rag, text-to-sparql}
}
- Sergios-Anestis Kefalidis, Research Associate at the University of Athens, Greece
- Kostas Plas, Research Associate at the University of Athens, Greece
- Manolis Koubarakis, Professor at the University of Athens, Greece
This is a research project by the AI-Team of the Department of Informatics and Telecommunications at the University of Athens.