Ecosystem organization

[icweaver]

From today's meeting

Potential package categories:

• Measurements and units, e.g., statistical and systematic errors
• Time and reference frames, e.g., Astrometry.jl
• Virtual Observatory, e.g., VOTables.jl and data access
• Imaging, e.g., DJ9.jl, a Julia version of DS9
• Time series analysis, e.g., LombScargle.jl & Bretthorst.jl
• Spectral analysis, e.g., SpectralFitting.jl
• Radiative transfer, e.g., RTModels.jl, Korg.jl, etc.

[icweaver]

To discuss:

How should this merge with / replace the current organization scheme?

JuliaAstro
├── DOMAIN
│   ├── Solar System
│   ├── Exoplanets
│   ├── Stars
│   └── Galaxies and Cosmology
├── TECHNIQUES
│   ├── Photometry
│   ├── Astrometry
│   ├── Spectroscopy
│   ├── Time Series Analysis
│   └── Interferometry
├── STATISTICS
│   ├── Uncertainty
│   ├── Optimization
│   └── General
└── UTILITIES
    ├── Data I/O
    ├── Data Processing
    ├── Data Viz
    └── Units / Constants

[aditya-pandey-dev]

Hi @icweaver
I've been looking into Issue #167 my understanding for this issue is :- Right now, the JuliaAstro website organizes packages into four broad categories: Domain, Techniques, Statistics, and Utilities. The issue proposes a new set of categories (Measurements & Units, Time & Reference Frames, Virtual Observatory, Imaging, Time Series, Spectral Analysis, Radiative Transfer) — and the question is: how should this new scheme merge with or replace the existing one?
My approach is:-

1. Keep the top-level structure** (Domain / Techniques / Statistics / Utilities) since it's already intuitive and used on the live website
2. Add missing subcategories** under Techniques — specifically Imaging and Radiative Transfer, which are in the new proposal but currently have no dedicated home
3. Elevate Virtual Observatory** as its own subcategory under Utilities > Data I/O, since VOTables.jl and VirtualObservatory.jl deserve more visibility
4. Cross-list packages** that fit multiple categories (e.g., Korg.jl appears in both Spectroscopy and Radiative Transfer) rather than duplicating or moving them

But I have a question before i start

1. Is there a preferred way to handle packages from outside the JuliaAstro org (like Korg.jl, DSP.jl) — should they be treated differently from first-party packages?
2. Are there any ongoing discussions or WIP PRs about this that I should be aware of before starting?
3. Is the goal to only update ecosystem.md, or are there other config files (like _quarto.yml or similar) that control the navigation structure too?

[icweaver]

Hi Aditya, thanks for starting to take a look at our organization scheme! To answer your questions:

1. Luckily, this is handled already here:
   

   JuliaAstro.github.io/src/ecosystem.jl

   Line 23 in 1b35776

   write(io, " | `", org_name, "`\n\n")

   In other words, the org name for both JuliaAstro and external orgs are already parsed from their repo urls and included in the generated markdown pages.

2. No new discussion yet, but you are welcome to attend our next monthly community call to learn more and discuss!

3. ecosystem.md is automatically generated from src/JuliaAstroDocs.jl, so I would treat that as the main entry point for making site-wide changes. The nested data structure here specifies the categories and sub-categories used for each package:

   JuliaAstro.github.io/src/JuliaAstroDocs.jl

   Line 7 in 1b35776

   ecosystem() = (

   See this section of our dev docs for more.

As is usually the case, the hardest part is naming things, so this will be an evolving process. I'd recommend starting with a visual representation of your proposed categorization scheme in whichever platform you prefer, which can then be shared for discussion with our wider community. Here's an old example I put together when first establishing the current scheme, in case it is helpful.

[aditya-pandey-dev]

Hi @icweaver I've implemented the proposed changes to src/JuliaAstroDocs.jl. Here's a visual overview of the updated categorization scheme:
Diagram:https://excalidraw.com/#json=6LbPEl0xpsy3RTYAzSML3,OlpEae9xEErAh7oG5wRzCg

Summary of changes:
Added Imaging subcategory under Techniques — SAOImageDS9.jl, AstroImages.jl
Added Radiative Transfer subcategory under Techniques — Korg.jl
Added Virtual Observatory subcategory under Utilities — VirtualObservatory.jl, VOTables.jl

should i do some changes ?

[icweaver]

Hi Aditya, thank you for the diagram. I think this is a good starting point, but I think it would still be good to get some feedback from other JuliaAstro folks about continuing with this core organization scheme, or extending things to more categories as proposed earlier. I've added this as an agenda item to our next community call coming up this Wednesday.

[cgarling]

I think we should be resistant to adding more categories as smaller subdivisions make it harder to figure out where something you are looking for should be. E.g., Korg could go under radiative transfer or spectroscopy depending on whether the method or the application is being emphasized.

I have no problem with our current organization.

[icweaver]

Agreed. I think @barrettp recommended the other scheme, so I will leave it to them to follow up if they had any ideas to the contrary

[barrettp]

What scheme did I recommend?

Another approach is to think about how science is done. For example, an experimentalist/observer would start by looking for packages that provide data access, then data analysis, and finally modeling or simulations. So the categories might be utilities, data analysis/techniques, and modeling/simulations. Korg.jl would fall more into the latter category than the former.

[icweaver]

What scheme did I recommend?

This I believe: #167 (comment)

Open to discussing more

[barrettp]

The top level scheme is a good start. However, I would divide the TECHNIQUES category into two; namely, DATA ANALYSIS, and MODELING/SIMULATIONS.

Note that Interferometry is vague. I assume that you mean radio interferometry, at least for now.

[cgarling]

The top level scheme is a good start. However, I would divide the TECHNIQUES category into two; namely, DATA ANALYSIS, and MODELING/SIMULATIONS.

The way I see it, analyzing data often requires constructing models and minimizing the residual between the data and the model, so this subdivision seems somewhat arbitrary to me. E.g., Photometry.jl would probably fall under your "data analysis" group but PSF-fitting photometry relies on models of image point-spread functions, for example those provided by PSFModels.jl. Under this design, PSFModels.jl would fit most obviously under the "modeling/simulations" heading, but now these packages that are closely associated would be listed under different headings. I would like to avoid this.

I also don't think we have many packages that would fall under the modeling/simulations heading so I think this subdivision would be more confusing than helpful.

[barrettp]

I agree. However, what is data analysis? It is typically a combination of utilities (e.g.,data IO), modeling, and optimization. So modeling is actually a rather fundamental category. Whereas, data analysis and domains are based on the other three categories, which leads me to the issue of models.

I think it would be a good idea to have a models category so that we can assemble a list of all sorts of models, so that we do not have to keep reinventing the wheel. I'm initially thinking about radiative processes models, such as blackbody and synchrotron emission, but this could be extended to any group of models. It would be a valuable long term resource that those developing data analysis and modeling software could draw upon, particularly if they are well implemented and tested, like SpecialFunctions.jl