Skip to content

Proposal: notebook joining IDC images to GDC clinical data for collections without a clinical table #117

Open

Description

@aaronkanzer

Several IDC collections whose patients are GDC cases have no clinical table in idc-index: cgci_htmcp_cc, cgci_htmcp_dlbcl, cgci_htmcp_lc, cgci_blgsp, hcmi_cmdc, cddp_eagle_1 and ccdi_mci. For these, the clinical data lives only in GDC.

I have a notebook that fetches those records from the public GDC cases endpoint and joins them to the IDC index on PatientID = submitter_id. You can run it on Scigantic, where the IDC index and the GDC tables are already mounted, or open it in Colab (source). For cgci_htmcp_cc, 211 of 211 patients match. It then shows H&E and p16 thumbnails labeled with FIGO stage, age and vital status. The Colab version uses only idc-index and the GDC API, so it needs no cloud project, and it follows the notebook conventions the CI checks.

I saw that draft #108 links IDC to GDC pathology reports. This notebook covers clinical records, so the two should not overlap, and I am happy to place it under related_external_data/ or fold it into that work if you prefer. Would you be interested in a PR?

Activity

  1. fedorov commented on Nov 10, 2026

    @fedorov
    Member

    @aaronkanzer thank you for reaching out about this!

    Management of clinical data for IDC collections is a complex topic. The tables that you will find exposed via idc-index originate for the most part from the spreadsheets that were deposited by the contributors along with the imaging data. We parse, import and host those because otherwise the effort to find and access them is too high.

    For the images contributed by projects that have their own procedures for managing and exposing clinical data, we never arrived at a satisfactory solution. Extracting and duplicating clinical data for such resources as GDC, HTAN or CCDI is problematic, since it would be difficult to keep it in sync. Not exposing it in IDC makes it difficult and potentially time-consuming for the users to find and access it....

    Another challenge - and one of the reasons why #108 is still in draft - is that I personally am not using those external resources in everyday work, and because of that it is difficult for me to decide if the approach taken is correct and what users would like to see....

    That being said, your contribution is very much welcomed! I looked over the notebook you referenced, and yes - you are welcome to contribute it to this repo! Also, if you have any comments about the notebooks in #108 - as perhaps a user more familiar with the use cases that need this clinical data - please add comments to that PR, or let me know if you think they are ready to be merged!

  2. aaronkanzer commented on Nov 13, 2026

    @aaronkanzer
    Author

    @aaronkanzer thank you for reaching out about this!

    Management of clinical data for IDC collections is a complex topic. The tables that you will find exposed via idc-index originate for the most part from the spreadsheets that were deposited by the contributors along with the imaging data. We parse, import and host those because otherwise the effort to find and access them is too high.

    For the images contributed by projects that have their own procedures for managing and exposing clinical data, we never arrived at a satisfactory solution. Extracting and duplicating clinical data for such resources as GDC, HTAN or CCDI is problematic, since it would be difficult to keep it in sync. Not exposing it in IDC makes it difficult and potentially time-consuming for the users to find and access it....

    Another challenge - and one of the reasons why #108 is still in draft - is that I personally am not using those external resources in everyday work, and because of that it is difficult for me to decide if the approach taken is correct and what users would like to see....

    That being said, your contribution is very much welcomed! I looked over the notebook you referenced, and yes - you are welcome to contribute it to this repo! Also, if you have any comments about the notebooks in #108 - as perhaps a user more familiar with the use cases that need this clinical data - please add comments to that PR, or let me know if you think they are ready to be merged!

    @fedorov thanks again for this and sorry again for the slow reply. That background on why the external clinical data isn't mirrored makes sense. I do think a notebook that fetches from GDC at run time is a good fit for that since nothing is duplicated and it stays in sync by construction.

    Just as a positive side-effect, I will mention that building this notebook is also what made the bug in idc-index#313 apparent. Also interactions on Scigantic by our users on the IDC data have uncovered some other improvements, might be great to just chat and discuss (my email is aaron@scigantic.com if you'd like to chat)

    I'll go through the #108 notebooks and leave comments there from the GDC side!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions