Managing Empirical Evidence: Application of the Evidence Evolution Framework

This repository demonstrates how to manage variance theories in software engineering (SE) research. It applies the evidence evolution framework to several cases of SE research agendas consisting of multiple pieces of evidence.

Artifact Content

This repository demonstrates how to manage scientific variance theories by defining clear relationships between individual pieces of evidence. An individual piece of empirical, quantitative evidence $e:=E(h, d, m)$ consists of three components.

1. Hypothesis $h$: A directed, acyclic graph connecting variables (nodes) with assumed causal relationships (edges).
2. Data $d$: A record of observations of all variables contained in the hypothesis $h$.
3. Analysis method $m$: An operation that infers whether one or more independent variables in hypothesis $h$ have a significant effect on the dependent variable in $h$.

Given an initial piece of evidence $e_1=E(h_1, d_1, m_1)$, follow-up studies can contribute related evidence that falls into one of three categories.

The full definition of empirical, quantitative evidence as well as the framework are described in detail in the scientific manuscript that this repository supports.

Artifact Structure

The repository contains the following directories and files:

├── material: directory of additional material
│   ├── dags: directory of basic directed, acyclic graphs (DAGs) used for demonstration
│   └── graphs: directory of custom graphs
│       ├── evolution: example of an evidence evolution graph
│       └── framework: visualization of the relationships between pieces of evidence
└── studies: collection of research agendas
    ├── requirements-quality: studies on the impact of passive voice on domain modeling
    └── text-coverage: studies on the impact of test coverage on test suite effectiveness

More elaborated case studies in the studies directory contains a separate README.md file that explains the context and involved studies better.

System Requirements and Setup

To view and edit the graphml files, install a graph editing tool like yEd from yWorks.

To execute the code provided for each study, ensure that you have R (version > 4.0) and RStudio installed on your machine. Then, execute the following steps to setup and integrate stan:

1. Install the rstan toolchain by following the instructions for Windows, Mac OS, or Linux respectively.
2. Restart RStudio and follow the instructions starting with the Installation of RStan
3. Install the latest version of stan by running the following commands

    install.packages("devtools")
    devtools::install_github("stan-dev/cmdstanr")
    cmdstanr::install_cmdstan()

4. Install all missing packages via install.packages(c("tidyverse", "patchwork", "ggdag", "ggstats", "lme4", "brms", "marginaleffects", "bayesplot", "effsize", "rcompanion", "lmtest")).
5. Create a folder called fits within each src/ directory such that brms has a location to place all Bayesian models.
6. Open the mvt-demo.Rproj file with RStudio, which will setup the environment correctly.

Usage

This repository is intended to demonstrate the use of the evidence evolution framework that helps managing variance theories. Once you are familiar with the fundamentals of this framework, browse the case studies provided in the studies directory. Each of them contains a detailed application of the framework.

Run the Rmd notebooks to interact with the data and analyses yourself, or view the html files (generated from the notebooks via knitr) for a pre-compiled version.

License

Copyright © 2026 Julian Frattini. This work (source code) is licensed under the MIT License.