Apparent

Analysing Physician-Patient Referral Network Topology

Apparent is a Python toolkit for analyzing patient referral flows within US healthcare systems using medical claims data (Medicare). We provide functionality for building and analyzing patient referral networks. In particular, we provide functionality to analyze these networks via discrete curvature and persistent homology, in hopes of supporting further research developments into using network analysis to improve efficiency and equity of the US healthcare system.

---

🔗 Prototype & Publication

• Prototype Tool: apparent.krv.ai
• Paper: Characterizing Physician Referral Networks with Ricci Curvature

---

🚀 Features

• Build Networks: Generate directed and undirected graphs representing referral relationships between physicians and patients.
• Describe Networks: Compute node- and edge-level features like degree, clustering coefficients, betweenness centrality, curvature measures, and persistence diagrams.
• Compare Networks: Measure pairwise similarity or distances between networks using curvature metrics and topological features.
• Embed Networks: Map networks into a lower-dimensional vector space for visualization and machine learning tasks.
• Cluster Networks: Group networks into clusters based on structural or functional similarity, leveraging techniques like k-means, hierarchical clustering, and DBSCAN.

---

⚙️ Installation

From Source

APPARENT uses uv as the package manager, which provides faster dependency resolution and installation.

Step 1: Clone the Repository

git clone https://github.com/aidos-lab/apparent.git
cd apparent

Step 2: Install Dependencies and Activate Virtual Environment

If you don't already have uv, install with pip:

pip install uv

To install dependencies, run:

uv sync

You'll notice this creates a .venv folder in the root directory.

We activate that new virtual environment as such:

source .venv/bin/activate

Step 3: Download the Database and Start a Local Server

from apparent.utils import download_and_launch_local_datasette

download_and_launch_local_datasette()

This downloads the ~3 GB SQLite database once (to data/), starts a local Datasette server at http://127.0.0.1:8001, and writes LOCAL_URL to .env so Apparent() picks it up automatically.

📚 Usage

Most actions can completed using the Apparent object, including the following functionality:

1. Pull Data: Extract data from our datasette tool (or local instance).
2. Build Networks: Construct physician-patient referral graphs from the extracted data.
3. Describe Networks: Compute network features such as curvature, centrality, and clustering coefficients.
4. Compare Networks: Analyze pairwise distances between networks using metrics like Forman curvature and Ollivier-Ricci curvature.
5. Embed Networks: Reduce dimensionality for visualization and machine learning.
6. Cluster Networks: Group similar networks using clustering algorithms like KMeans, DBSCAN, and hierarchical clustering.
7. Local Database: Download the SQL database and launch a local Datasette instance for environments with limited connectivity or firewall restrictions.

Quick Example

Here's a quick example for how you can pull specific Physician Referral Networks using apparent!

from apparent import Apparent

# Initialize Apparent (uses LOCAL_URL from .env, see Step 3)
A = Apparent()

# Example SQL query for fetching data
my_query = """
          SELECT
            hospital_atlas_data.hsa,
            hospital_atlas_data.year,
            hospital_atlas_data.latitude,
            hospital_atlas_data.longitude
          FROM
            hospital_atlas_data
          WHERE
            hospital_atlas_data.year = 2017
          LIMIT
            10;
        """

# Pull data from the database
A.pull(my_query)

# Build referral networks
A.build_networks()

# Compare networks based on Forman curvature
A.compare(measure="forman_curvature")

# Embed networks into a lower-dimensional space
A.embed()

# Cluster networks based on structural similarity
A.cluster_networks()

Working with a Local Database

If you're in an environment with connectivity issues, firewall restrictions, or need offline access, you can download the database and run a local Datasette instance.

from apparent import Apparent
from apparent.utils import download_and_launch_local_datasette, stop_local_datasette

# Download the database and start a local Datasette server
local = download_and_launch_local_datasette(verbose=True)

print(f"Local Datasette server running at: {local['url']}")
# Point Apparent at the local CSV endpoint
app = Apparent(base_url=local["csv_url"])

# Simple sample query that mirrors the test patterns
# This gets basic network info for small networks from 2017
query = """
    SELECT
    hospital_atlas_data.hsa,
    hospital_atlas_data.year,
    hospital_atlas_data.latitude,
    hospital_atlas_data.longitude
    FROM
    hospital_atlas_data
    WHERE
    hospital_atlas_data.year = 2017
    LIMIT
    10;
"""

app.pull(query)
print(f"Retrieved {len(app.data)} networks")
print(app.data.head())

# Stop the local Datasette server when done
stop_local_datasette(port=8001)

Note: If receiving a "sqlite3.DatabaseError: database disk image is malformed" error message, we recommend deleting the current version of the database. This can come from ungraceful shutdowns of python subprocesses. If this persists, consider launching the datasette directly with bash with the following command:

datasette /path/to/your/local/sqlFile.db --setting sql_time_limit_ms 500000 --setting max_returned_rows 200000 --setting allow_csv_stream off --reload`

🤝 Contributing

Contributions are welcome! To contribute:

1. Fork the repository.
2. Create a new branch (git checkout -b feature-name).
3. Commit changes (git commit -m 'Add feature').
4. Push to your branch (git push origin feature-name).
5. Open a Pull Request.

🧪 Testing

This project uses pytest for testing. The tests are divided into two categories: unit and integration.

Unit Tests

Unit tests have no external dependencies. They run automatically in CI on pushes to main and develop:

pytest -m unit

Integration Tests

You can run integration tests in two ways:

Option 1: Using the helper script

A script is provided to simplify running the integration tests. This script handles:

1. Downloading the raw dataset (under data/us_physician_referral_networks.db).
2. Launching a local Datasette server.
3. Executing the integration test suite.

Warning: The dataset is large (approximately 3 GB) and may take considerable time to download depending on your internet speed.

To execute the script, run the following command from the root directory:

bash tests/run-integration-tests.sh

Option 2: Using the Python API

You can also use the new Python API to set up the local database and run tests:

from apparent.utils import download_and_launch_local_datasette, stop_local_datasette
import subprocess

# Download DB and start Datasette
local = download_and_launch_local_datasette(
    db_path="data/us_physician_referral_networks.db",
    port=8001,
    update_env=True
)

# Run integration tests
subprocess.run(["python", "-m", "pytest", "tests/", "-v", "-m", "integration"])

# Stop the local Datasette server when done
stop_local_datasette(port=8001)

📝 License

This project is licensed under the BSD-3 License. See the LICENSE file for details.

📬 Contact

For questions, feedback, or collaboration opportunities please contact the AIDOS Lab.