diff --git a/docs/advanced/index.rst b/docs/advanced/index.rst deleted file mode 100644 index 4aaf5b31..00000000 --- a/docs/advanced/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -Advanced Topics -=============== - -This section of the guide goes into more details on specific topics, or choices -the guide has made. You shouldn't need to read or understand anything in this -section to get up and running with a package, but you may want to refer to it if -you have questions. - - -.. toctree:: - versioning diff --git a/docs/advanced/versioning.rst b/docs/advanced/versioning.rst deleted file mode 100644 index 37f0fc9d..00000000 --- a/docs/advanced/versioning.rst +++ /dev/null @@ -1,72 +0,0 @@ -.. _versions: - -Specifying the Version of your Package -====================================== - -Synchronising the version information about your package between, the git -repository, the python source code and the python package metadata can be -complex. - -We strongly recommend using `setuptools_scm `__ to do this. -With a little effort and understanding it is possible to make the git history -the single source of truth for all your version information, for your releases -and any development installs. - -.. _setuptools-scm: - -``setuptools_scm`` ------------------- - -``setuptools_scm`` is an extension to the ``setuptools`` package, which performs two functions: - -* It calculates the version number from the git history. -* It uses the git repository to get a list of all files to include in the package. - -This section is focusing on the first one, the second is discussed in :ref:`data`. - -``setuptools_scm`` works by calling ``$ git describe`` which returns information -about the current version of the repository, based on distance away from the -last tag **on the current branch**. - -This means that with a little parsing, this information can be used to determine -the version of the package at the point where you build packages to release them on -PyPI (as described in :ref:`releasing`) or when someone accesses -``my_package.__version__``. - -When just considering these built packages ``setuptools_scm`` works by running -``git describe`` at the time you build the package and then saving the output -into the code (if configured) and metadata of the package. If you use the -default configuration of the template included with this guide, after installing -the package the contents of the ``my_package/_version.py`` file would be:: - - # coding: utf-8 - # file generated by setuptools_scm - # don't change, don't track in version control - __version__ = version = '0.1.dev0' - __version_tuple__ = version_tuple = (0, 1, 'dev0') - -This file is used to provide the ``__version__`` attribute of ``my_package``. - -.. _dev-versions: - -Dynamic Development Versions ----------------------------- - -The final piece of this puzzle is versions for packages installed in "editable" -mode from a git repository. By default when using ``setuptools_scm`` when you -run ``pip install -e ./`` in your git checkout, it will run ``git describe`` and -encode the current version into the ``_version.py`` file described above. The -problem with this approach is that the version number in ``_version.py`` will -not change, as you make new commits or even add new tags. - -The solution to this is to dynamically calculate the version on access of -``__version__`` with ``setuptools_scm``, however, you don't want to do this for -non-development installations as it's slow and requires ``setuptools_scm`` to -be installed. - -The solution to this implemented in the template (behind an opt-in option) is to -make a subpackage called `_dev` which is not included in the dists (it is -excluded in ``MANIFEST.in``) which invokes setuptools. This ``_dev`` package is -then invoked by the ``my_pacakge/version.py`` file, which will fall back to -reading the static version from ``my_package/_version.py`` file if it can't -access ``_dev`` (in the case where it's not a dev install). diff --git a/docs/ci.rst b/docs/ci.rst index bc9e7b4f..6708c165 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -1,149 +1,61 @@ .. _ci: -====================== Continuous Integration ====================== -Continuous Integration (CI) is the method by which software is tested and built before deployment to users. -A set of 'jobs' are defined in a ``.yml`` file, roughly taking the flow build - test - deploy. -Each run is built from a clean environment. -Workflows can be set to begin on triggers for example, a ``git push`` or a new tag. +The template defines a single workflow ``ci.yml`` which runs all tests, builds distributions and pushes to PyPI. +See :ref:`oa:ci` for CI fundamentals. + +This workflow makes heavy use of the OpenAstronomy workflows for :ref:`tox ` and publishing :ref:`pure Python packages ` and ones with :ref:`compiled extensions `. + + +Jobs +---- + +Core +^^^^ + +The core job is designed to gate the most expensive CI against a single test run. +This should normally be the newest version of Python under a Linux runner. + +Source dist verification +^^^^^^^^^^^^^^^^^^^^^^^^ -There are an array solutions for running CI, Open Astronomy recommends `GitHub Actions `__ for projects using GitHub. -GitHub Actions workflows are defined in the ``.github/workflows/`` folder at the root of the repo. -Open Astronomy maintains a `set of tools `__ to make configuring GitHub Actions easier. +This job verifies that the package creates a valid source distribution, which is a sanity check for packaging and releasing the package. -Examples -++++++++ -Testing -------- -In order GitHub Actions to run your workflow, it requires; an event to trigger the workflow, one or more jobs to complete and all steps must either run a script or trigger an action. -Looking at this in context: +Test +^^^^ -.. code-block:: yaml +The test job runs the rest of the tests once the ``core`` and ``sdist_verify`` jobs have succeeded. +The jobs you put here should **always pass** as your package will not release if this job fails, so any online jobs or other flakey builds should probably be in an extra job. - name: Run template tests +Documentation +^^^^^^^^^^^^^ - on: - push: - pull_request: - workflow_dispatch: +This job builds the documentation using sphinx, and error if any Sphinx warnings are emitted, which generally isn't the case on Read the Docs. - jobs: - test: - uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v1 - with: - envs: | - - linux: py311-test +Extra CI jobs +^^^^^^^^^^^^^ + +Using the ``extra_ci_jobs`` (:ref:`options_reference`) you can add the header for other jobs (such as online). +Enabling this option will reduce CI conflicts and enable the template to manage the version of the OpenAstronomy tox workflow. + +If you have a number of non-tox jobs or other things not managed by the OA workflows, it might be best to add them as a separate workflow file to minimise conflicts with the template updates. + +Building distributions +^^^^^^^^^^^^^^^^^^^^^^ + +The ``build_dists`` job creates wheels and the sdist for your package, which are then uploaded to PyPI upon a release. +If your package is a pure Python package (no compiled extensions) then a universal wheel and source dist will be built and tested in this step. +If you have compiled extensions, binary wheels for various platforms will be built and tested, using `cibuildwheel `__. -In this case the workflow is triggered on a push to a branch, a PR, or a manual trigger of the workflow (`workflow_dispatch`). -The second line of the job defines the name of the job, in this case ``test``, and uses Open Astronomy's pre-defined workflow to run the tests with tox. -The ``envs`` list, defines which tox environments will be run for that job. Publishing to PyPI ------------------- - -No Compiled Extensions -###################### - -Python packages should be published on `PyPI `__, which can be automated on CI. -This can improve security (as fewer people need access to publish on PyPI) and make it less effort for maintainers to publish a release. -When we are building and publishing releases to PyPI we only want this to happen on a `git tag `__, as opposed to on every commit. -However, if we only run the build job on tags, we never have a way to test that the job works before we tag a release of our package. -The OpenAstronomy publish workflows will (by default) only publish to PyPI on a tag which starts with `v` (`see here `__). -Therefore, we recommend running the workflow on both push to your default branch (`main`), on tags and on manual runs. - -.. code-block:: yaml - - on: - push: - branches: - - 'main' - tag: - workflow_dispatch: - - jobs: - publish: - uses: OpenAstronomy/github-actions-workflows/.github/workflows/publish_pure_python.yml@v1 - with: - test_extras: test - test_command: pytest --pyargs - secrets: - pypi_token: ${{ secrets.pypi_token }} - -Replace references to `` with the package to be published. - -To publish to PyPI we need a PyPI token, associated with your PyPI account. -`Instructions on creating up your key here `__. -The secret can be stored at `organisation `__ or `repo level `__ in GitHub settings. - -With Compiled Extensions -######################## - -Almost all packages on pypi also have environment specific binaries with all dependencies packaged. -It is expected that a package publishes both a pure python distribution and the binary, see `here for examples `__. - -In this case, in addition to the running the tests, the ``with`` block also includes targets. -'Targets' are the distributions which the binary will be built for, so in this case it would be linux and MacOS 64 bit. -The ``publish`` method from the Open Astronomy GitHub actions packages the module with the dependencies for the specific targets listed - -.. code-block:: yaml - - jobs: - publish: - uses: OpenAstronomy/github-actions-workflows/.github/workflows/publish.yml@v1 - with: - test_extras: test - test_command: pytest --pyargs test_package - targets: | - - linux - - cp3?-macosx_x86_64 - secrets: - pypi_token: ${{ secrets.pypi_token }} - - -.. sam, work your way to the full example use the sunkit example -.. https://github.com/sunpy/sunkit-instruments/blob/main/.github/workflows/ci.yml - -Putting it all together -####################### - -Combining the above steps reveals a total workflow, build, testing and publishing - -.. code-block:: yaml - - name: package_deployment - - on: - push: - tag: - - jobs: - test: - uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v1 - with: - envs: | - - linux: py311 - - publish_python: - uses: OpenAstronomy/github-actions-workflows/.github/workflows/publish_pure_python.yml@v1 - with: - test_extras: test - test_command: pytest --pyargs test_package - secrets: - pypi_token: ${{ secrets.pypi_token }} - - publish_binaries: - publish: - uses: OpenAstronomy/github-actions-workflows/.github/workflows/publish.yml@v1 - with: - test_extras: test - test_command: pytest --pyargs test_package - targets: | - - linux - - cp3?-macosx_x86_64 - secrets: - pypi_token: ${{ secrets.pypi_token }} - -The ``.github/workflows/`` directory may contain several workflows such as the above. -Each file may contain different workflows, with different triggers dependent on requirements. +^^^^^^^^^^^^^^^^^^ + +See also :ref:`oa:releasing` for release fundamentals and :ref:`releasing` for the template-specific setup. + +Notifications +^^^^^^^^^^^^^ + +If enabled with the ``matrix_room_id`` option then notifications of build status will be posted to the given matrix room. diff --git a/docs/conf.py b/docs/conf.py index 3f0987db..26d19898 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -4,21 +4,11 @@ # list see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html -# -- Path setup -------------------------------------------------------------- - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -# -# import os -# import sys -# sys.path.insert(0, os.path.abspath('.')) - # -- Project information ----------------------------------------------------- -project = 'OpenAstronomy Python Packaging Guide' -copyright = '2019, OpenAstronomy Developers' -author = 'OpenAstronomy Developers' +project = 'SunPy Package Template' +copyright = '2019, SunPy Developers' +author = 'SunPy Developers' # -- General configuration --------------------------------------------------- @@ -26,6 +16,8 @@ # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ + 'sphinx.ext.intersphinx', + 'sphinx.ext.todo', ] # Add any paths that contain templates here, relative to this directory. @@ -39,6 +31,17 @@ # Treat everything in single ` as a Python reference. default_role = 'py:obj' +# Render .. todo:: directives in the output. +todo_include_todos = True + +# -- Intersphinx mapping ----------------------------------------------------- + +intersphinx_mapping = { + 'oa': ('https://packaging-guide.openastronomy.org/en/latest/', None), + 'cookiecutter': ('https://cookiecutter.readthedocs.io/en/stable/', None), + 'oagha': ('https://github-actions-workflows.openastronomy.org/en/stable/', None), +} + # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for diff --git a/docs/data.rst b/docs/data.rst deleted file mode 100644 index fc9f666b..00000000 --- a/docs/data.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. _data: - -Including data in your package -============================== - -Using ``setuptools_scm`` to link your Python package to your git repository also makes including data easy. -By setting ``include_package_data = true`` in the ``[tool.setuptools]`` section of ``pyproject.toml``, ``setuptools_scm`` will automatically include all files tracked by git in your package. - -While this is useful for including required non-Python files, it's pretty common to have files that don't belong in your distribution in your git repository, such as continuious integration configurations, or even git config files. -It's possible to exclude certain files and directories which are tracked by git from being included in your built package by adding ``exclude`` or ``prune`` lines to the ``MANIFEST.in`` file in the root of the repository. -An example ``MANIFIEST.in`` file might look like:: - - # Exclude specific files - # All files which are tracked by git and not explicitly excluded here are included by setuptools_scm - # exclude whole directories from the package - prune .github - # exclude specific files from the package - exclude .mailmap - exclude .gitignore - exclude .gitattributes - exclude *.yml - exclude *.yaml - - -If you need to explicitly include a file, for example ``important.yaml``, when there is a more generic exclude you need to include that file *after* the more general exclude statement as the commands are processed in order. -See the `Python packaging guide `__ for more details. diff --git a/docs/docs.rst b/docs/docs.rst deleted file mode 100644 index 210b3abc..00000000 --- a/docs/docs.rst +++ /dev/null @@ -1,215 +0,0 @@ -.. _documentation: - -Documenting your Package -======================== - -There are two main ways to document your project, both of -which are essential: :ref:`docstrings` and :ref:`narrative`. - -.. _docstrings: - -Docstrings ----------- - -First, public functions, methods, and classes -in your package should include *docstrings*, which are strings -attached to those objects which the user can access interactively -using e.g. ``help()`` and which can also be retrieved by automated -tools. See `PEP 257 - Docstring Conventions `_ -for a high-level overview of what docstrings are. We recommend adopting -the `numpydoc `_ -format for docstrings. An example of such a docstring is:: - - def foo(var, long_var_name='hi'): - """A one-line summary that does not use variable names. - - Several sentences providing an extended description. Refer to - variables using back-ticks, e.g. `var`. - - Parameters - ---------- - var : int - The type above can either refer to an actual Python type - (e.g. ``int``), or describe the type of the variable in more - detail, e.g. ``(N,) ndarray`` or ``array_like``. - long_var_name : {'hi', 'ho'}, optional - Choices in brackets, default first when optional. - - Returns - ------- - out : type - Explanation of `out`. - """ - -These docstrings should be included in the Python files alongside the Python -objects they document. - -.. _narrative: - -Narrative Documentation ------------------------ - -Second, you should write a set of narrative documentation which functions as a -user guide, such as http://docs.astropy.org or http://docs.sunpy.org. For this -we recommend making use -of the `Sphinx `_ tool and storing your documentation -inside a ``docs`` directory. - -To set this up, first install the Sphinx package, then create a ``docs`` directory -and run ``sphinx-quickstart`` inside it:: - - $ mkdir docs - $ cd docs - $ sphinx-quickstart - -We recommend answering ``n`` for the question of whether to separate -source and build directories. Once you have run this, you should see the following files -inside your ``docs`` directory:: - - Makefile - conf.py - index.rst - make.bat - -The ``index.rst`` file is the root of your documentation. You can start writing content -in it and you can also start adding more ``.rst`` pages if needed. If you haven't used -Sphinx before, we recommend taking a look at their -`Getting Started `_ guide. - -.. _automodapi: - -Including docstrings in the narrative documentation ---------------------------------------------------- - -As part of the narrative documentation, it is also common practice to include an -Application programming interface (API) page which lists the available classes, -methods, and functions in your package. Thankfully, if you've defined your docstrings -as described in :ref:`docstrings`, then this can be automated using the -`sphinx-automodapi `_ -package. See the documentation of that package for more details, but briefly, -you will need to add ``'sphinx_automodapi.automodapi'`` to the ``extensions`` -variable in your ``conf.py`` file: - -.. code-block:: python - - extensions = ['sphinx_automodapi.automodapi'] - -In addition, if you use the numpydoc format for your docstrings, as recommended in :ref:`docstrings`, -you will need to include either ``'numpydoc'`` or ``'sphinx.ext.napoleon'`` in -the list of ``extensions`` (both packages provide a way to parse numpydoc-style -docstrings). If you use the numpydoc package, you will need to also include: - -.. code-block:: python - - numpydoc_show_class_members = False - -in your ``conf.py`` file. - -Declaring dependencies for documentation ----------------------------------------- - -To make it easier for contributors to get set up with the dependencies -required to build the documentation, as well as to make it easier to -configure automated builds (whether for :ref:`ReadTheDocs ` -or :ref:`tox `), you should define an ``[project.optional-dependencies]`` section in -your ``pyproject.toml`` file named ``docs`` which lists the dependencies -required to build the documentation (not including dependencies already -mentioned in ``dependencies``): - -.. code-block:: toml - - [project.optional-dependencies] - docs = [ - "sphinx", - "sphinx-automodapi", - "numpydoc", - ] - -This will then allow contributors to type:: - - pip install -e .[docs] - -to install the package in developer/editable mode along with the documentation -dependencies. - -.. _readthedocs: - -Setting up ReadTheDocs ----------------------- - -`ReadTheDocs `__ is a platform that will build -documentation with sphinx and will then host it, and is used by many of -the Scientific Python packages. The easiest way to configure the build -is to add a file called ``.readthedocs.yml`` to your package, and we -recommend starting off with: - -.. code-block:: yaml - - version: 2 - - build: - image: latest - tools: - python: 3.9 - - python: - install: - - method: pip - path: . - extra_requirements: - - docs - -Once you have added this to your repository, you can then import your -package into ReadTheDocs as described in `this guide -`_. - -.. _plot_directive: - -Add plots to your documentation -------------------------------- - -A plot is worth *many* words, and sometimes documentation -can demonstrate the uses and advantages of using a given -package much more efficiently than narrative docs. Matplotlib, -for example, has made this quite straightforward with the -`plot directive `_. - -To add a plot to your Sphinx documentation, add the following string to the -``extensions`` list in your ``docs/conf.py`` file: - -.. code-block:: python - - extensions = [ - ... # preserve your other extensions here, then add: - "matplotlib.sphinxext.plot_directive" - ] - -To make use of this extension, you will also need to add ``matplotlib`` to your -``tox.ini`` file: - -.. code-block:: ini - - deps = - # preserve your other deps here, then add: - matplotlib - -Now you can add plots to your Sphinx docs by adding a block like -the following to your narrative docs: - -.. code-block:: rst - - Here's a plot: - - .. plot:: - - import matplotlib.pyplot as plt - - x, y = [1, 2, 3], [4, 5, 6] - - plt.figure() - plt.plot(x, y) - -By default, sphinx and matplotlib will render the figure defined by the -Python code in the ``.. plot::`` block, without the source code. Full documentation -for the configuration settings for the plot directive can be found in the -`matplotlib docs `_. diff --git a/docs/extensions.rst b/docs/extensions.rst deleted file mode 100644 index 43c0db5e..00000000 --- a/docs/extensions.rst +++ /dev/null @@ -1,98 +0,0 @@ -.. _extensions: - -Compiled C/Cython extensions -============================ - -Python packages can include compiled extensions in a variety of languages, most -commonly C and `Cython `_ (Cython is a language close to -Python that can be automatically translated into C). An extension, once -compiled, looks just like a regular Python module/sub-module. - -There are some reasons why you might want to include compiled extensions -including for example to speed up code that is otherwise slow in Python, or -because you want to include an existing stable library without having to -re-implement it in Python. - -Defining extensions in ``setup.py`` ------------------------------------ - -To define an extension, we need to create an instance of -:class:`setuptools.Extension` inside the ``setup.py`` file. For a simple -case with a single ``.c`` file, this would look like:: - - from setuptools import Extension - ext = Extension(name='my_package.my_extension', - sources=['my_package/my_extension.c']) - -Here ``name`` is the final name the compiled extension will have, which means -that if the extension defines a function ``fast_function`` it can be imported -as:: - - from my_package.my_extension import fast_function - -The ``sources`` argument should be set to a list of source files to compile and -link together to create the extension, and the filenames should be relative to -the ``setup.py`` file. If your extension uses the Numpy C API, you should also -specify the Numpy include directory using:: - - ext = Extension(name='my_package.my_extension', - sources=['my_package/my_extension.c'], - include_dirs=[numpy.get_include()]) - -Several other options can be passed to set for example what -other libraries to link to, flags or macros to pass to the compiler, and so on. -For more information about these, see the:class:`Extension section -` in the Python documentation. - -Once your extension has been defined, you should pass a list of extensions -to the ``ext_modules`` keyword argument to the ``setup()`` function in the -``setup.py`` file:: - - setup(..., ext_modules=[ext]) - -If you want to build a Cython extension instead of a C extension, specify the -``.pyx`` file(s) in the ``sources`` argument:: - - ext = Extension(name='my_package.my_extension', - sources=['my_package/my_extension.pyx']) - -And make sure you also add ``cython`` to your ``pyproject.toml`` build-time -dependencies:: - - [build-system] - requires = [..., "cython"] - build-backend = 'setuptools.build_meta' - -Packages with many extensions ------------------------------ - -For packages with many extensions, you might want to consider using the -`extension-helpers `_ package. This -package serves two main purposes: - -* For single-file Cython extensions, it will automatically discover and - define these extensions. - -* For other extensions, it allows you to define extensions inside - ``setup_package.py`` files which can be anywhere in your package. These files - should contain a single function called ``get_extensions`` that returns a list - of extensions. The idea is then to make it easier to manage extensions for - large packages by placing the ``setup_package.py`` files close to the - extension code. - -To use extension-helpers, first, make sure it is included in your ``pyproject.toml`` -file as a build-time dependency:: - - [build-system] - requires = [..., "extension-helpers"] - build-backend = 'setuptools.build_meta' - -Then adjust your ``setup.py`` to include:: - - from extension_helpers.setup_helpers import get_extensions - - setup(..., ext_modules=get_extensions()) - -Finally, if needed, create ``setup_package.py`` files in sub-modules where you -have extensions, add a ``get_extensions()`` function, and make sure that it -returns a list of :class:`~setuptools.Extension` objects. diff --git a/docs/features.rst b/docs/features.rst new file mode 100644 index 00000000..d621c671 --- /dev/null +++ b/docs/features.rst @@ -0,0 +1,89 @@ +.. _features: + +Template Features +================= + +This page documents the key features the template provides beyond the :doc:`OpenAstronomy guide `, which do not have other dedicated sections of the documentation. +For file-by-file details, consult the generated package directly. + +Changelog Management with Towncrier +----------------------------------- + +The template uses `towncrier `__ to assemble ``CHANGELOG.rst`` from individual fragment files. +Configuration lives in the ``[tool.towncrier]`` table in ``pyproject.toml``. + +Fragments are stored in the ``changelog/`` directory and named ``.[.].rst``. +The supported fragment types are: + +* ``breaking`` -- Breaking Changes +* ``deprecation`` -- Deprecations +* ``removal`` -- Removals +* ``feature`` -- New Features +* ``bugfix`` -- Bug Fixes +* ``doc`` -- Documentation +* ``trivial`` -- Internal Changes + +On release, ``towncrier build`` collects all fragments, groups them by type, and writes the result to ``CHANGELOG.rst`` with links back to the originating PRs. +The rendered changelog is included in the documentation via `sphinx-changelog `__. + +.. note:: + + The ``[tool.towncrier]`` configuration lives in ``pyproject.toml`` rather than ``towncrier.toml`` because Gilesbot only reads ``pyproject.toml``. + + +Changelog Checks with Gilesbot +------------------------------ + +The template integrates `Gilesbot `__, a GitHub bot that checks changelog entries on every pull request. +Configuration lives in the ``[tool.gilesbot]`` table in ``pyproject.toml``. + +When a PR is opened, Gilesbot checks for a changelog fragment in the ``changelog/`` directory. +The ``verify_pr_number`` setting ensures the fragment's filename matches the PR number. +PRs that do not need a changelog entry can be labelled ``No Changelog Entry Needed`` to skip the check. + + +Oldest Dependencies Testing +--------------------------- + +The ``-oldestdeps`` tox factor tests the package against the minimum supported versions of its dependencies. +It uses the `minimum_dependencies `__ tool to generate a ``requirements-min.txt`` file pinned to the lower bounds declared in the package metadata, then installs and tests against those pins. + +This catches regressions caused by accidentally raising a dependency's lower bound without a corresponding ``requires-python`` or CI change. +The environment runs as part of the CI ``test`` job matrix (see :doc:`ci`). + + +Development Dependencies Testing +-------------------------------- + +The ``-devdeps`` tox factor tests the package against in-development versions of key dependencies. +It sets ``PIP_EXTRA_INDEX_URL`` to the astropy and scientific-python nightly wheel repositories, and pins ``numpy>=0.0.dev0`` so that the latest nightly numpy wheel is installed. + +For dependencies that do not publish nightly wheels, you can add a line such as ``devdeps: git+https://github.com/owner/repo`` to the ``deps`` section of ``tox.ini`` to build from source. + +This environment runs as part of the CI ``test`` job matrix (see :doc:`ci`). +It provides early warning of breakage from upstream API or behavior changes before they reach a stable release. + + +Coverage Reporting with Codecov +------------------------------- + +The template configures `Codecov `__ as a way to see coverage reports for PRs and commits. +The ``.codecov.yaml`` file configures options for the reporting; the ``tox.ini``, ``.coveragerc``, and ``ci.yml`` files also configure the behavior of the coverage calculation and reporting. + + +ReadTheDocs +----------- + +The template generates a ``.readthedocs.yaml`` configured to build documentation on ReadTheDocs using a conda environment defined in ``.rtd-environment.yml``. +This conda environment is used to install non-Python dependencies such as graphviz. +The Python dependencies are installed using the ``[docs]`` extra defined in ``pyproject.toml``. + +See also :ref:`oa:readthedocs` for ReadTheDocs fundamentals. + + +Label Synchronisation +--------------------- + +The template generates a ``label_sync.yml`` GitHub Actions workflow that synchronises the repository's issue labels against a canonical definition hosted at ``https://github.com/sunpy/.github/blob/main/labels.yml``. + +This ensures all SunPy packages share a consistent set of labels for triage, changelog enforcement, and CI automation. diff --git a/docs/index.rst b/docs/index.rst index 70ae5eb1..3ef453b8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,102 +1,35 @@ -Python Packaging Guide +SunPy Package Template ====================== -This guide is intended to explain modern Python packaging, it covers most of the core components of a modern package, and explains these components. It is broken up into the following sections: +The SunPy package template is a tool for creating, but primarily maintaining a Python package with the same layout, tooling and developer experience as the sunpy core package and other packages maintained by the SunPy Project. + +The template is built on top of the `OpenAstronomy Python Packaging Guide `__. +For an overview of the core concepts of maintaining a Python package start there. +These documentation pages will document the features added by the SunPy Template and the things you will need to setup to use it. .. toctree:: :maxdepth: 2 - minimal - docs - tests - tox - extensions - releasing - scripts - data + new_package_options + features + linting ci - advanced/index - -Using the Template -================== - -With this guide is a `cookiecutter `__ template which allows you to get started quickly with a package as described in this guide. - -To create a new package based on the template run: - -.. code-block:: console - - $ pip install cookiecutter cruft - $ cruft create https://github.com/sunpy/package-template - -and go through the steps offered in the cli naming your package and filling in your details. -Cruft is built on cookiecutter, and enables the updating of the template from the source. -This takes the form of pull requests to the repository that the new package is pushed to. -If a package already has a cookiecutter template, it can be linked to the parent repository using ``cruft link url-to-template``. - -To manually check whether the current environment matches with the template then ``cruft check`` will tell you what the current status is. -``cruft update`` will manually trigger an updating of the package to the template. - -If you would like to stick to simply the cookiecutter approach, the template still supports that functionality thusly: - -.. code-block:: console - - $ pip install cookiecutter - $ cookiecutter gh:sunpy/package-template -o ./output_directory - -This will create a new directory in your current directory named the same as the value of "packagename" you supplied. -Change into this directory and run ``git init`` to make it into a git repository, and make an initial commit. -This is required in order to have software versioning working for your package. - -The goal of the template is to quickly get you setup with the files described in the guide. -The template currently implements the following optional flags, all of which default to off: - -* ``include_example_code``: This option will fill your new package with some example functions to allow you to test it. -* ``use_compiled_extensions``: This turns on the features needed to support compiled extensions as described in :ref:`extensions`. -* ``enable_dynamic_dev_versions``: This enables a feature which ensures that ``my_package.__version__`` always returns the current git version as calculated by ``setuptools_scm`` when the package is installed as an editable install. See :ref:`dev-versions` for more details. -* ``include_cruft_update_github_repo``: This option adds a github workflow with pulls in the latest changes from the template every Monday morning and creates a PR against the repo which can then be accepted or closed. -* ``use_extended_ruff_linting``: This option flag enables the stricter ruff rules. Recommend `Y` on creation of a new project. - -Pre-commit -========== - -Pre-commit is configured through ``.pre-commit-config.yaml`` and can be installed locally and ran: - -.. code-block:: bash - - $ pre-commit run --all-files - -It also possible to use the tox environment to run it and and is integrated into the CI. - -Within ``.pre-commit-config.yaml``, there are several tools and each one is configured either within the ``.pre-commit-config.yaml`` or for larger tools like ruff, it is has a dedicated config file ``.ruff.toml`` and we strongly recommend using the full set of rules when you setup your package. - -Updating to a new version of the template -========================================= - -It will be simplest to updating a package to a newer template by waiting for the GitHub workflow to trigger. -This will trigger a pull request one can review and merge via GitHub's UI. - -If you do not want to wait, or want to do it manually, you will have to use cruft's CLI. -For this you will need to install cruft locally and then you can check the status of the package by running: - -.. code-block:: console - - $ cruft check - -This will let you know whether the repository is up to date or not. -From there, you can run: - -.. code-block:: console - - $ cruft update - -To update the repo and if there is a case conflicting files, ``.rej`` files will be created and you will have to manually deal with them and merge changes. + updates + releasing -If you need up update one of the variables in the package ``.cruft.json`` for example, changing a ``n`` to a ``y`` this can be done using: -.. code-block:: console +Topics Covered by the OpenAstronomy Guide +----------------------------------------- - $ cruft update --variables-to-update '{"use_extended_ruff_linting": "y"}' +The following topics are documented in the `OpenAstronomy Python Packaging Guide `__ and are not duplicated here. +Follow the cross-references to read about them. -This will work through the codebase and include the desired functionality without any further action. -Then you can commit and push the resulting changes. +* :ref:`oa:minimal` -- minimal package layout (``pyproject.toml``, ``setup.py``, ``MANIFEST.in``, ``LICENSE``, ``README``) +* :ref:`oa:documentation` -- documenting your package (docstrings, Sphinx, ReadTheDocs) +* :ref:`oa:testing` -- testing your package with pytest +* :ref:`oa:tox` -- running commands with tox +* :ref:`oa:extensions` -- compiled C/Cython extensions +* :ref:`oa:releasing` -- releasing your package (version numbers, sdist, PyPI) +* :ref:`oa:scripts` -- command-line scripts +* :ref:`oa:data` -- including data in your package +* :ref:`oa:versions` -- specifying the version of your package (setuptools_scm) diff --git a/docs/linting.rst b/docs/linting.rst new file mode 100644 index 00000000..27c3ca57 --- /dev/null +++ b/docs/linting.rst @@ -0,0 +1,74 @@ +.. _linting: + +Linting and Code Style +====================== + +The template provides a standardised linting and formatting toolchain via pre-commit. +All hooks are configured to exclude the ``data`` and ``extern`` subdirectories of the package. + + +Pre-commit +---------- + +`pre-commit `__ runs a collection of checks and formatters on every commit. +The template's configuration installs and runs the following tools automatically: + +* **zizmor** -- scans GitHub Actions workflow files for security issues. +* **ruff** -- lints and auto-fixes Python files (see below). +* **isort** -- sorts import statements (see below). +* **pre-commit-hooks** -- a suite of basic sanity checks: valid Python syntax, valid YAML/TOML, no trailing whitespace, no debug statements, no large files, consistent line endings, and end-of-file newlines. +* **codespell** -- checks for common spelling mistakes (see below). + +`pre-commit.ci `__ runs on pull requests to check that all pre-commit checks pass. +The automatic updates of the hook versions should be handled by the template PRs rather than update PRs from pre-commit.ci (which unfortunately can not be disabled). + +To run all hooks locally via tox do:: + + tox -e codestyle + +This installs the hooks and runs them across the whole repository with diffs shown on failure. + +To have pre-commit automatically run the checks when you make a commit locally run:: + + pre-commit install + + +Ruff +---- + +`Ruff `__ is a fast Python linter that replaces flake8, pyupgrade, isort (partially), and numerous other tools. +The template configures a base rule set covering pycodestyle errors and warnings, pyflakes, pyupgrade, and pytest-style rules. + +The ``use_extended_ruff_linting`` option (see :doc:`new_package_options`) enables additional rule sets for bugbear, blind-except, comprehensions, implicit namespace packages, print statements, return statements, tidy imports, pathlib usage, pandas idioms, pylint conventions and errors, flynt, numpy, performance, and ruff-specific checks. +This is recommended for new projects. + +Docstrings are checked against the numpy convention. +Per-file ignores relax import-ordering and unused-import rules in ``__init__.py`` and allow ``print`` in example scripts. + +See the ``.ruff.toml`` file in your repository for more information about what is configured. + + +isort +----- + +`isort `__ sorts Python imports into grouped sections. +The template defines a custom section order that separates astropy ecosystem packages (``astropy``, ``asdf``) and SunPy packages (``sunpy``) from generic third-party imports, so imports from the scientific Python ecosystem are visually distinct. + +.. note:: + + See https://github.com/sunpy/package-template/issues/230 for details on why we haven't (yet) replaced isort with ruff. + + +Codespell +--------- + +`Codespell `__ checks for common spelling mistakes in Python and RST files. +The template configures an ignore list of astronomy-domain terms that are commonly false positives (e.g. ``observ``, ``nd``, ``alog``). +Binary and data file formats are skipped. + + +Flake8 +------ + +`Flake8 `__ is configured as a legacy companion to ruff. +It is largely superseded by ruff but retained for compatibility with tools or editors that invoke it directly. diff --git a/docs/minimal.rst b/docs/minimal.rst deleted file mode 100644 index 59b5ca64..00000000 --- a/docs/minimal.rst +++ /dev/null @@ -1,296 +0,0 @@ -.. _minimal: - -Minimal package layout -====================== - -To start off, we will take a look at the minimal set of files you will need to -create an installable Python package. Once you have set these up, your package -directory should look like:: - - ├── LICENCE - ├── my_package - │   └── __init__.py - ├── pyproject.toml - ├── setup.py - ├── MANIFEST.in - └── README.rst - -where ``my_package`` is the name of your package. We will now take a look at all of -these files in turn. - -.. _license: - -``LICENSE`` ------------ - -Assuming that you are planning to make your package open source, the most -important file you will need to add to your package is an open source license. -Many packages in the scientific Python ecosystem use the `3-clause BSD license -`_ and we recommend following -this or using the `MIT license `_ -unless you have a good reason not to. - -To include the license in your package, create a file called LICENSE -and paste the license text into it, making sure that you update the -copyright year, authors, and any other required fields - -.. _readme: - -``README.rst`` --------------- - -Another important file to include is a README file, which briefly tells users -what the package is, and either gives some information about how to install/use -it or links to more extensive documentation. We recommend using the -`reStructuredText (rst) `_ format for -your README as this will ensure that the README gets rendered well online, e.g. -on `GitHub `_ or `GitLab `_ and on `PyPI -`_. - -.. _package_init: - -``my_package/__init__.py`` --------------------------- - -Python code for your package should live in a sub-directory that has the name -of the Python module you want your users to import. This module name should -be a valid Python variable name, so cannot start with numbers and cannot include -hyphens. Valid package names are ``example`` or ``my_package``. For the rest -of this guide, we will assume the name of the module is ``my_package``. - -Once you have created this directory, the first file to create in it should be a -file called ``__init__.py`` which will be the first code to be run when a user -imports your package. For now, the only information we will add to this file is -the version of the package, since users typically expect to be able to access -``my_package.__version__`` to find out the current package version. While you -could simply set e.g. - -.. code-block:: python - - __version__ = '1.2' - -in the ``__init__.py`` file, you then would need to make sure that the version -number is in sync with the version number defined in the :ref:`pyproject` file, -so a better approach is to put the following in your ``__init__.py`` file - -.. code-block:: python - - from importlib.metadata import version as _version, PackageNotFoundError - try: - __version__ = _version(__name__) - except PackageNotFoundError: - pass - -This will automatically set ``__version__`` to the global version of the package -declared in :ref:`pyproject` or set by the `setuptools_scm -`__ package (see :ref:`setup_py` and -:ref:`pyproject` for more details). - -.. _pyproject: - -``pyproject.toml`` ------------------- - -The ``pyproject.toml`` file is where we will define the metadata about the package. -At a minimum, this file should contain the ``[project]`` table (defined by -`PEP621 `_) and the ``[build-system]`` table -(defined by `PEP518 `__). - -``[project]`` -^^^^^^^^^^^^^ - -.. code-block:: toml - - [project] - name = "my-package" - description = "My package description" - readme = "README.rst" - authors = [ - { name = "Your Name", email = "your@email.com" } - ] - license = { text = "BSD 3-Clause License" } - dependencies = [ - "numpy", - "astropy>=3.2", - ] - dynamic = ["version"] - - [project.urls] - homepage = "https://link-to-your-project" - -The ``name`` field is the name your package will have on PyPI. It is not necessarily -the same as the module name, so in this case we've set the package name to -``my-package`` even though the module name is ``my_package``. However, aside from -the case where the package name has a hyphen and the module name has an underscore, -we strongly recommend making the package and the module name the same to avoid confusion. - -Note that the version of the package is **not** explicitly defined in the file above, -(rather, defined as ``dynamic``), because we are using the -`setuptools_scm `_ package to automatically -retrieve the latest version from Git tags. However, if you choose to not use that -package, you can explicitly set the version in the ``[project]`` section (and remove it -from the ``dynamic`` list): - -.. code-block:: toml - - [project] - version = "0.12" - -The ``description`` should be a short one-line sentence that will appear next to your package name -on `PyPI `_ when users search for packages. The ``readme`` -defines the ``README.rst`` file, which will be rendered nicely on the PyPI page for the package. - -Finally, the ``dependencies`` section is important since it is where you will -be declaring the dependencies for your package. The cleanest way to do this is -to specify one package per line, as shown above. You can optionally include version -restrictions if needed (as shown with ``astropy>=3.2`` above). If your package has no dependencies then you don't need this option. - -A complete list of keywords in ``[project]`` can be found in the `Python packaging documentation `_. - -``[build-system]`` -^^^^^^^^^^^^^^^^^^ - -In the previous section we discussed the ``dependencies`` which can -be used to declare run-time dependencies for the package, which are -dependencies that are needed for the package to import and run correctly. -However, your package may have dependencies that are needed to build the -package in the first place. For example, the :ref:`setup_py` file -will only run correctly if `setuptools `_ -is installed. - -The recommended way to specify build-time dependencies is to define the -``build-system`` table: - -.. code-block:: toml - - [build-system] - requires = ["setuptools>=45", "wheel", "setuptools_scm[toml]>=6.2"] - build-backend = 'setuptools.build_meta' - -If you choose to not use ``setuptools_scm``, you can remove it from this list. - -If you do want to use ``setuptools_scm`` you also want to add the following -block to enable and configure it:: - - [tool.setuptools_scm] - write_to = "my_package/_version.py" - -If your package has C extensions that interface with `Numpy `_, -you may also need to add Numpy to the above list - see :ref:`extensions` for -more details. - -A complete list of keywords in ``[build-system]`` can be found in `PEP518 `__. - -``[tool.setuptools]`` -^^^^^^^^^^^^^^^^^^^^^ - -.. code-block:: toml - - [tool.setuptools] - zip_safe = false - - [tool.setuptools.packages.find] - -The ``zip_safe`` option should be set to ``false`` unless you understand the -implications of setting it to ``true`` - this option is most relevant when -producing application bundles with Python packages. - -The ``packages.find`` line can be left as-is - this will automatically determine the -Python modules to install based on the presence of ``__init__.py`` files. - -A complete list of keywords in ``[tool.setuptools]`` can be found in the -`setuptools documentation `_. - -``[tool.setuptools_scm]`` -^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code-block:: toml - - [tool.setuptools_scm] - write_to = "my_package/version.py" - -The ``[tool.setuptools_scm]`` table indicates that we want to use the `setuptools_scm -`_ package to set the version -automatically based on git tags, which will produce version strings such as -``0.13`` for a stable release, or ``0.16.0.dev113+g3d1a8747`` for a developer -version. The ``write_to`` option is not necessary; it will write the parsed version -to a ``version.py`` with a ``__version__`` variable that can be imported by the -package itself. - -.. _setup_py: - -``setup.py`` ------------- - -The ``setup.py`` file used to be where project metadata was defined, before the -advent of ``setup.cfg`` and then PEP621 and PEP517 (``pyproject.toml``). -It is no longer necessary to include a ``setup.py`` file in your project, -unless you are building C extensions in your code. -However, it can increase compatibility with old versions of pip and other packaging tools. - -The minimal ``setup.py`` file is very simple: - -.. code-block:: python - - from setuptools import setup - - setup() - -.. _manifest: - -``MANIFEST.in`` ---------------- - -The last file needed for a minimal set-up is the ``MANIFEST.in`` file, -which declares which files should be included when you release your -package (see :ref:`releasing` for more details about how to do this). - -This file is simplified by using ``setuptools_scm``, as **everything** that is -git versioned will be included **by default**. There are likely to be things -you want to exclude, such as files generated by the documentation, to do this -add:: - - prune - -For example a minimal ``MANIFEST.in`` file for a package using ``setuptools_scm`` might look like - -.. code-block:: text - - prune build - prune docs/_build - prune docs/api - global-exclude *.pyc *.o - -which would exclude the autogenerated documentation folders and other build files from the distributions. - -If you have chosen not to use ``setuptools_scm``, then this file needs to list -files not in the module directory and other non-standard files. -So given the files we've seen above you would need to include:: - - include LICENSE - include README.rst - include pyproject.toml - -You can find out more about the syntax of this file in -`Specifying the files to distribute `_ -in the Python documentation. - - -Trying out your package ------------------------ - -Once you have committed all of the above files to your repository, you -can test out the package by running - -.. code-block:: shell - - pip install . - -from the root of the package. Once you have done this, you should be able to -start a Python session from a different directory and type e.g.:: - - >>> import my_package - >>> my_package.__version__ - 0.1.dev1+g25976ae - -.. TODO: mention about adding more files to package with functionality diff --git a/docs/new_package_options.rst b/docs/new_package_options.rst new file mode 100644 index 00000000..f2b1236f --- /dev/null +++ b/docs/new_package_options.rst @@ -0,0 +1,125 @@ +.. _new_package_and_options: + +Templating a New Package, and Available Options +=============================================== + +This section of the documentation covers the workflow of creating a new package from the template and also is a reference of all the available options. +We expect you to be familiar with Python package layout here, read the open astronomy guide if not. + +This package template is based on `cookiecutter `__ and uses `cruft `__ for incremental updates. +See the :doc:`cookiecutter:overview` for an introduction to the cookiecutter project. + +.. _new_package: + +Creating a New Package +---------------------- + +Using cruft +^^^^^^^^^^^ + +Firstly, install ``cruft`` with the package manager of your choice:: + + pip install cruft + +See :doc:`cookiecutter:installation` if you need to install cookiecutter separately. + +To create a new package with cruft run:: + + cruft create gh:sunpy/package-template + +This will use the main branch of the package-template repo and ask you a series of questions about how you want to customise your package. + + +.. _options_reference: + +Cookiecutter Options Reference +------------------------------ + +This is a complete reference of every option in ``cookiecutter.json``. +Options are grouped into required, optional feature flag, URL/metadata, and private categories. +See the :doc:`cookiecutter:overview` for background on how cookiecutter uses these values. + +Required options +^^^^^^^^^^^^^^^^ + +These options have no default (or a placeholder default) and must be answered when generating a package. + +* ``package_name`` -- The distribution name of the package as it will appear on PyPI. + May contain hyphens. + e.g. ``sunpy``. +* ``module_name`` -- The name of the importable Python module, i.e. what you will type after ``import``. + e.g. ``sunpy``. +* ``short_description`` -- A one-line description used in ``pyproject.toml`` and the generated ``README.rst``. +* ``author_name`` -- The name(s) of the package author(s), written into ``pyproject.toml`` and the license file. +* ``author_email`` -- The email address of the package author(s), written into ``pyproject.toml``. +* ``license`` -- The SPDX license for the package. + Choices: ``BSD 3-Clause`` (default), ``GNU GPL v3+``, ``Apache Software License 2.0``, ``BSD 2-Clause``, ``Other``. + The corresponding license text is copied to ``licenses/LICENSE.rst``. +* ``minimum_python_version`` -- The minimum supported Python version, written to ``requires-python`` in ``pyproject.toml`` and used in CI and tox configuration. + Choices: ``3.11`` (default), ``3.12``, ``3.13``, ``3.14``. + +Optional features +^^^^^^^^^^^^^^^^^ + +These are y/n that toggle features on or off. + +* ``use_compiled_extensions`` -- Enables Cython/C compiled extension support: adds ``extension-helpers``, ``cython``, and ``numpy`` to build dependencies; switches the CI publish job to build platform wheels via ``OpenAstronomy/github-actions-workflows``'s ``publish.yml``. + See also :ref:`oa:extensions`. +* ``enable_dynamic_dev_versions`` -- Enables dynamic development version calculation via ``setuptools_scm`` so that ``my_package.__version__`` reflects the current git state during editable installs. + Generates a ``_dev`` subpackage and ``version.py`` shim. + See :ref:`oa:versions`. +* ``include_example_code`` -- Generates example modules (``example_mod.py``, ``example_c.pyx`` when combined with ``use_compiled_extensions``), an example subpackage, data files, and tests. +* ``include_cruft_update_github_workflow`` -- Generates the per-repo ``sub_package_update.yml`` workflow that runs ``cruft update`` weekly and opens a PR. + Intended for affiliated packages outside the SunPy GitHub org; packages in the SunPy org are updated centrally. + See :doc:`updates`. +* ``use_pat_in_cruft_update_workflow`` -- When ``y``, the generated ``sub_package_update.yml`` uses a GitHub personal access token (PAT) with ``workflow`` scope (via the ``sub_package_update`` environment and ``WORKFLOWS_UPDATE_PAT`` secret) instead of the default ``GITHUB_TOKEN``. + Highly recommended to enable and configure this option when using the ``sub_package_update.yml`` workflow. + See :doc:`updates`. +* ``use_extended_ruff_linting`` -- Enables the extended ruff rule set in ``.ruff.toml`` (bugbear, print, pathlib, pandas, pylint, perf, ruff-specific, and more). + Recommended for new projects. + See :doc:`linting`. +* ``extra_ci_jobs`` -- A comma-separated list of extra tox environment names (e.g. ``online,threading``) to scaffold as additional CI jobs in ``ci.yml``. + The job name and tox env are emitted; the ``envs:`` body is left for the maintainer to fill in. + See :doc:`ci`. + +URLs and metadata +^^^^^^^^^^^^^^^^^ + +These options populate the ``[project.urls]`` table in ``pyproject.toml`` and links in the generated ``README.rst``. +Several derive defaults from ``github_repo``. + +* ``project_url`` -- Primary website for the project. + Leave blank to default to the SunPy homepage (``https://sunpy.org``). +* ``github_repo`` -- The GitHub repository in ``user/repo`` format (e.g. ``sunpy/sunpy``). + Leave blank if the project is not on GitHub. + Used to derive defaults for ``sourcecode_url`` and ``issue_tracker_url``. +* ``sourcecode_url`` -- URL for the source code. + Defaults to ``https://github.com/`` when ``github_repo`` is set. +* ``download_url`` -- PyPI address for the project. + Defaults to ``https://pypi.org/project/``. +* ``documentation_url`` -- URL to the rendered documentation. + No default. +* ``changelog_url`` -- URL to the changelog. + No default. +* ``issue_tracker_url`` -- URL to the issue tracker. + Defaults to ``https://github.com//issues/`` when ``github_repo`` is set. +* ``matrix_room_id`` -- A Matrix room ID (e.g. ``!example:matrix.org``). + When set, generates a ``notify`` job in ``ci.yml`` that posts CI summaries to the room. + See :doc:`ci`. + +Advanced / private options +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +These options are prefixed with an underscore, so cookiecutter does not prompt for them. +They can be set via ``--extra-context`` or by editing ``cookiecutter.json`` directly. +See :doc:`cookiecutter:advanced/private_variables`. + +* ``_sphinx_theme`` -- The Sphinx HTML theme used in the generated ``docs/conf.py``. + Default is ``sunpy`` (the ``sunpy-sphinx-theme`` package). +* ``_parent_project`` -- Name of a parent project, if any. + No default. +* ``_install_requires`` -- A string of runtime dependencies written into the ``[project] dependencies`` list in ``pyproject.toml``. + No default. +* ``_copy_without_render`` -- A list of paths that cookiecutter copies without Jinja rendering. + Defaults to ``docs/_templates``, ``docs/_static``, and ``.github/workflows/sub_package_update.yml``. + See :doc:`cookiecutter:advanced/copy_without_render`. diff --git a/docs/releasing.rst b/docs/releasing.rst index 7f9aab30..eb146b1a 100644 --- a/docs/releasing.rst +++ b/docs/releasing.rst @@ -1,101 +1,19 @@ .. _releasing: -Releasing Your Package -====================== +Configuring Automatic Releases +============================== -In this section we will describe how to take your package and publish a release to PyPI. +The :ref:`ci` setup of the package template configures builds and uploads to PyPI on tags. -There are a lot of permutations on how to release your package, and depending on -the size of your project you may need to build on this guide. The objective of -this is to provide you with the basic information you need to release something -built by following the rest of this guide. +There are a few steps required to configure GitHub and PyPI to support the automatic publishing of releases. -This section of the guide is assuming you configured `setuptools_scm -`__ in the :ref:`minimal` guide. If -you didn't you will need to update your ``pyproject.toml`` file as well as using -``git tag``. +Firstly, we need to create a `GitHub Environment `__ named ``pypi``, this should be configured to only apply to tags matching ``v*`` to minimize the possibility of someone publishing a bogus release. +No secrets are required to be configured. -Incrementing Version Numbers ----------------------------- +The package template uses `Trusted Publishing `__ to push to PyPI. +Following the PyPI documentation you need to go to ``https://pypi.org/manage/project//settings/publishing/`` and enter the details, this is probably: -When you are ready to release your package you need to give it a version number. -A version number for a release should generally be of the form ``X.Y.Z``, for -full details on versioning Python packages see `PEP 440 -`__. What meaning is conveyed by the -version numbers is up to you, there are multiple different thoughts on this, for -some examples see `Astropy -`__, `SunPy -`__ or `Semantic -Versioning `__. - -In this example we are going to release version ``0.1.0`` of our package -``my_package``. When doing releases it is common practice to use `git tags -`__ to identify the commit -the release relates to in the history. By using ``setuptools_scm`` these tags -become the reference for the version numbers of your Python package. This means -you only have to increment your version number using git. - -To mark a new release of your package in your git history run: - -.. code-block:: console - - $ git tag -a v0.1.0 -m "Release version 0.1.0" - -Here we use the convention of prepending release tags with ``v``. - -If you now import your package and print ``my_package.__version__`` it should say -``0.1.0``. - -Building Source Distributions ------------------------------ - -Now you have tagged your release, you need to build what is called a "source -distribution" to upload to `PyPI `__ or the Python Package -Index. This is the place where tools like ``pip`` download packages from and is -the primary place people will search for installable Python packages. - -The source distribution is a tarball of all the files needed by your package, -which includes everything in your ``my_package`` directory as well as everything -specified in your :ref:`manifest` file. - -As we have setup a package with a :ref:`pyproject` file, we recommend you use the -`build `__ package to build your -source distribution in the isolated environment specified in :ref:`pyproject`. -You can do this with: - -.. code-block:: console - - $ pip install build - $ python -m build --sdist --outdir dist . - -This is equivalent to running the legacy ``python setup.py sdist`` but ensures -that the state of your local environment does not affect the generated package. - -Publishing to PyPI ------------------- - -Now you have created the sdist to be uploaded to PyPI you can upload it with the -`twine `__ package: - -.. code-block:: console - - $ pip install twine - $ twine upload dist/my_package*.tar.gz - -This should ask you for your PyPI account details, and will create your project -on PyPI if it doesn't already exist. - -Releasing from Branches ------------------------ - -If your project is larger, you might want to create branches for each of your -major release versions to make it easy to continue to support those releases -with bug fixes while continuing development of your master branch. - -If you follow this pattern for your releases you will have to perform one extra -step when using ``setuptools_scm``, which is to also increment the version with -a tag on the master branch to indicate you have started to develop a new version -on your master branch. To do this at the point where you branch for your -upcoming release push a tag for ``vX.Ydev`` where ``X.Y`` is the version number -of the next major release e.g. if you just branched for 1.1 you would create a -``v1.2dev`` tag. +* **Owner**: ``sunpy`` (or other GitHub org if not sunpy) +* **Repository name**: The name of the GitHub repo. +* **Workflow name**: ``ci.yml`` +* **Environment name**: ``pypi`` diff --git a/docs/scripts.rst b/docs/scripts.rst deleted file mode 100644 index dca7afc2..00000000 --- a/docs/scripts.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. _scripts: - -Command-line scripts -==================== - -The recommended way to add command-line scripts to your package is to make use of the -setuptools ``console_scripts`` entry point, as described in the `setuptools -documentation `_. -If you want to add command-line arguments/flags to your script, we recommend using -`click `_ package which makes this much easier. diff --git a/docs/tests.rst b/docs/tests.rst deleted file mode 100644 index cfdfa680..00000000 --- a/docs/tests.rst +++ /dev/null @@ -1,102 +0,0 @@ -.. _testing: - -Testing your package -==================== - -While writing new functionality for your package, you should also make sure that -you write unit tests. We suggest using the `pytest `_ -framework for writing and running tests. - -Where to keep tests -------------------- - -We recommend placing the tests inside sub-folders called ``tests`` alongside the -Python code they are meant to test - for example, the layout of the package might -look like:: - - my_package/__init__.py - my_package/utils.py - my_package/tests/__init__.py - my_package/tests/test_utils.py - -The name of the test files should start with ``test_``. We recommend taking a look -at the `Getting Started `_ -guide for pytest for how to write tests, as well as the `Astropy guide on writing -tests `_. - -Running tests -------------- - -Assuming you have installed pytest, the easiest way to run the tests is to run the -``pytest`` command:: - - pytest my_package - -Note that if your package contains C extension, you will need to make sure the -extensions are compiled ahead of time - you can either do this with:: - - python setup.py build_ext --inplace - -or with:: - - pip install -e . - -Once you have done this, the pytest command should work. If your package defines -entry points, you will likely need to run the ``pip`` command if you are running -tests that rely on the entry points. - -Defining default pytest options -------------------------------- - -If you regularly need to run tests with the same command-line flags for your -package, or if you want to set options that are required for certain pytest -plugins, you can control these by adding a ``[tool.pytest.ini_options]`` section -to your ``pyproject.toml`` file; for example - -.. code-block:: toml - - [tool.pytest.ini_options] - addopts = "-v" - -will ensure that tests are always run in verbose mode. - -Running doctests ----------------- - -It is possible to use pytest to run doctests (blocks of code in docstrings or in -the .rst docs) as part of the test suite. If you want to do this, we recommend -using the `pytest-doctestplus ` -plugin. You can define the following options in your ``pyproject.toml`` file to make -sure that this option is always enabled - -.. code-block:: toml - - [tool.pytest.ini_options] - doctest_plus = "enabled" - addopts = "--doctest-rst" - -Declaring dependencies for tests --------------------------------- - -To make it easier for contributors to get set up with the dependencies -required to run the tests, as well as to make it easier to -configure automated builds (with e.g. :ref:`tox `), you should -define an ``[project.optional-dependencies]`` section in -your ``pyproject.toml`` file named ``test`` which lists the dependencies -required to run the tests (not including dependencies already -mentioned in ``dependencies``):: - - [project.optional-dependencies] - test = [ - "pytest", - "pytest-doctestplus", - ] - -This will then allow contributors to type - -.. code-block:: shell - - pip install -e ".[test]" - -to install the package in developer/editable mode along with the test -dependencies. diff --git a/docs/tox.rst b/docs/tox.rst deleted file mode 100644 index d670e492..00000000 --- a/docs/tox.rst +++ /dev/null @@ -1,237 +0,0 @@ -.. _tox: - -Running Commands with Tox -========================= - -`Tox `__ is a general purpose tool for -automating Python testing. We recommend using tox to specify the environments -in which your tests are run, both locally and on :ref:`ci` services. - -Getting Started with Tox: Running tests ---------------------------------------- - -The first thing to configure tox to do is to run the tests for a package. The -most minimal tox file for a package following this guide is: - -.. code-block:: ini - - [tox] - envlist = py38 - isolated_build = True - - [testenv] - extras = test - commands = pytest {posargs} - - -Let's dig into the sections of this file, the ``[tox]`` section is the `global -configuration -`__ for -the whole file. We use this to define ``envlist`` which is a list of all the -different builds configured in tox, here we set this to be a Python 3.8 -environment, we will expand on this shortly. The ``isolated_build`` -configuration option configures tox to build your source distribution in the -same manner as recommended in :ref:`releasing`. - -The ``[testenv]`` section describes settings common to all environments you -specify in the tox file (unless they are later overridden), here we default -the ``commands =`` option to run pytest. The ``{posargs}`` is a tox -`substitution -`__ which -passes extra arguments through to ``pytest``. -The ``extras = test`` line tells tox to install the ``optional-dependencies`` section -listed in ``pyproject.toml`` for running your test suite; this should include ``pytest``. - -To run your tests with tox run: - -.. code-block:: console - - $ tox -e py38 - -To pass arguments through to ``pytest`` use ``--`` here we tell pytest to -stop after the first failure. - -.. code-block:: console - - $ tox -e py38 -- -x - -Multiple builds -############### - -Tox allows configuration of multiple builds in a few different ways, the -easiest one is to specify multiple Python versions in the env list: - -.. code-block:: ini - - [tox] - envlist = py{37,38} - isolated_build = True - -This takes our one test configuration and makes a Python 3.7 and a Python 3.8 -environment that can be seen by listing all tox environments with: - -.. code-block:: console - - $ tox -l - py38 - py37 - -This feature is called `generative envlist `__ and can be used to create many build environments with minimal repetition. - -Named Environments -################## - -Using generative build environments you can define extra named environments -which can be useful for builds that need to specify specific dependencies or -settings. So far on this page we have assumed that all your dependencies are -specified in :ref:`pyproject`. You can extend or override this by using the -``deps =`` configuration option in tox. Here we define a named test -environment which installs the development version of numpy. - -.. code-block:: ini - - [tox] - envlist = py{37,38}{-numpydev,} - isolated_build = True - - [testenv] - extras = test - commands = pytest {posargs} - deps = - numpydev: git+https://github.com/numpy/numpy - - -the ``envlist`` is now more complex, the result of this the following: - -.. code-block:: console - - $ tox -l - py37-numpydev - py37 - py38-numpydev - py38 - -with the ``deps`` overridden for ``numpydev`` builds. - - -Environment variables -##################### - -It is often useful to set environment variables within the building and testing -environment prior to testing. Environment variables can be set within ``tox.ini`` -with: - -.. code-block:: ini - - [testenv] - # Pass through the following environment variables which may be needed for the CI - passenv = HOME, WINDIR, LC_ALL, LC_CTYPE, CC, CI, TRAVIS - - # Suppress display of matplotlib plots generated during docs build - setenv = MPLBACKEND=agg - -The variables listed after ``passenv`` will be preserved from the -environment that you used to run tox, while the ``setenv`` variables -are set within the testing environment. In the template, we have set the -``MPLBACKEND`` variable to the ``agg`` backend, which prevents matplotlib -from launching interactive plot displays when generating figures from the -matplotlib plot directive or pytest-mpl. For more on making use of this -feature, see :ref:`plot_directive`. - - -Building Documentation with tox -------------------------------- - -One common task which isn't running the test suite is building sphinx -documentation, documentation builds can be complex with a number of extra -dependencies or settings. In this section we will add a ``build_docs`` named -environment to tox. This section assumes you have already followed -:ref:`documentation`. - -.. code-block:: ini - - [testenv:build_docs] - extras = docs - commands = sphinx-build docs docs/_build/html -W -b html {posargs} - -This section installs the package extras for the documentation, which should -be a list of all your documentation dependencies and then sets the command to -be the `sphinx-build -`__ command to -build the docs and output them in the ``docs/_build/html`` folder relative to -the ``tox.ini`` file. - -You can now run your documentation with: - -.. code-block:: console - - $ tox -e build_docs - -you can pass through extra arguments to `sphinx-build -`__ because of -the ``{posargs}`` substitution. For example to force sphinx to ignore its -cache you can run: - -.. code-block:: console - - $ tox -e build_docs -- -aE - -Testing Packages with Compiled Extensions ------------------------------------------ - -As configured in this guide so far, tox will perform the following actions (all in the same directory as the ``tox.ini`` file): - -1. ``python setup.py sdist`` -2. Create a new virtualenv -3. Install the built sdist. -4. Run the commands listed in ``commands =``, which here we assume to be ``pytest``. - -(See https://tox.readthedocs.io/en/latest/index.html#system-overview for more details.) - -For packages laid out as described in this guide, i.e. with the Python -package in a directory in the root repo, i.e. ``astropy/``, this means that -when ``pytest`` is run, it will collect the tests from the local directory -(as desired), and all imports of the package i.e. ``astropy`` will be -imported from the local directory *not the installed sdist*. - -For pure python packages this generally isn't a problem, the contents of the -installed sdist and the local directory are the same (tox just made the sdist -from the local directory). However, for packages that include compiled -extensions, the installed package and the local directory are *not the same*. -The installed package has build the compiled extensions, and the local -directory does not. This means that unless you make some adjustments to the -package or the tox configuration compiled extensions will not work when -running pytest through tox as described above. - -There are two main ways to alleviate this issue: - -1. Move the Python package source code under a ``src/`` folder in the root of -the repo. This is a common package layout for Python projects, and it means -that you can not import your package relative to the git root, meaning it -will be imported from the installed sdist, see https://setuptools.readthedocs.io/en/latest/setuptools.html#using-a-src-layout for details. - -2. Configure tox to run ``pytest`` from a temporary directory so that the -local import does not work. With this method you make use of pytest's -`--pyargs flag -`__ -to run the tests against the installed version of the package. This ensures -that any compiled extensions are properly detected, but prevents things like -specifying paths to pytest from working. - -To configure tox to run ``pytest`` from a temporary directory do the -following in ``tox.ini``: - - -.. code-block:: ini - - [tox] - envlist = py38 - isolated_build = True - - [testenv] - changedir = tmp - extras = test - commands = pytest --pyargs packagename {posargs} - -replacing ``packagename`` with the name of your package as you import it, -i.e. ``astropy``. diff --git a/docs/updates.rst b/docs/updates.rst new file mode 100644 index 00000000..cf055170 --- /dev/null +++ b/docs/updates.rst @@ -0,0 +1,52 @@ +.. _updates: + +Template Updates +================ + +The package template uses `cruft `__ to apply incremental updates. +There are three main ways these updates are applied to your package: + +#. **Centralised updater** -- for packages in the SunPy GitHub org (runs from this template repository) +#. **Per-repo self-update** -- for affiliated packages outside the SunPy GitHub org (opt-in via ``include_cruft_update_github_workflow`` option) +#. **Manual cruft run** -- You can always run ``cruft update`` at the CLI to pull updates. + +The choice between the per-repo and centralised updater is based on whether your package lives in the SunPy GitHub organization. + + +Centralised updater (SunPy org packages) +---------------------------------------- + +If your package lives under the SunPy org, you should have it listed in the ``centralised_cruft_update.yml`` workflow file in the template repo. +Only SunPy org repos are supported for this because of GitHub token permissions. + +The workflow runs whenever there is an update to the template, if an existing update PR is open then it will be updated else, a new one will be opened. +If there are cruft update conflicts the PR will be opened as a draft. +The best way to fix cruft conflicts is to edit the files using the GitHub web UI, as generally they are easy to change. +It is always worth bearing in mind that the less you deviate from the template the less conflicts are likely to occur. + +Per-repo self-update +-------------------- + +If your repo isn't under the SunPy org then you can enable the ``include_cruft_update_github_workflow`` option. +This workflow runs weekly, or on demand, and pulls the latest changes into your package. +If this workflow fails, it should open an issue on your repo to help you remember to debug the failure. + +Finally, there is an option when manually triggering the workflow to specify variables to update as a json string, see `Updating Values of Template Variables `__ in the cruft documentation. + +Using a PAT for workflow file updates +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +It's highly recommended to configure this repo with a `GitHub PAT `__ with permissions to push to your repo including the ``workflow`` permission. +This is because frequent updates to the workflow files are pushed via the template and the default GitHub Actions ``GITHUB_TOKEN`` does not and can not have permissions to edit the workflow files. + +The use of a PAT is covered by the ``use_pat_in_cruft_update_workflow`` option, which defaults to on when using the per-repo update workflow. +This PAT should have read and write access for the ``Contents`` and ``Workflows`` permissions. + +You should then `create a GitHub environment `__ named ``sub_package_update``, configure it to only deploy on the main branch, and then add a ``WORKFLOWS_UPDATE_PAT`` secret. + + +Dependabot interaction +---------------------- + +The template also configures GitHub's dependabot for GitHub Actions updates, this is to ensure that any workflows and actions not managed by the template are also kept up to date. +Any workflows or actions (i.e. anything in ``ci.yml`` by default) should not be updated by dependabot because this will cause conflicts on the next cruft update run. +The best thing to do with dependabot is to wait for a package template update and then merge the Dependabot PR after as this should mean only non-template things are included in the dependabot update.