Release a package only for a change that counts, leaving out documentation, tests, examples and manifest metadata

AGENTS.md

@@ -267,11 +267,18 @@ touches all of them ([ci/packages.ps1](ci/packages.ps1),
 ## Versions are per package
 
 Every package is versioned and tagged on its own, `<package>/<version>`. A
-package is released when its folders changed since its last tag, or when a
-package it depends on is released. The examples packages are tested but never
-published. common-ci's publish carries one version; here it is only a label,
-and the release plan in `ci/packages.ps1` decides what is built, published
-and tagged. Never push a plain `4.5.x` tag: nothing here reads one.
+package is released when a file that counts in its folders changed since its
+last tag, or when a package it depends on is released. Documentation
+(`*.md`, `docs/`), tests (`tests/`, `test/`), examples (`examples/`) and
+`LICENSE` files do not count, nor do files a package's `Ignore` globs in
+`ci/packages.json` match, unless it sets `IgnoreDefaults` to false. A
+`setup.py`, `setup.cfg` or `pyproject.toml` whose change is only to the
+description, keywords, urls, classifiers, readme, authors or maintainers does
+not count either, which `ci/release-plan-filter.ps1` works out with Python.
+The examples packages are tested but never published. common-ci's publish
+carries one version, which here is only a label, and the release plan in
+`ci/packages.ps1` decides what is built, published and tagged. Never push a
+plain `4.5.x` tag, because nothing here reads one.
 
 `ci/packages.json` is the list of packages and what each depends on. Add a
 package there, with a `FirstVersion` if it has never been released, or it is

ci/packages.Tests.ps1

@@ -247,6 +247,267 @@ Describe 'Get-ReleasePlan' {
         $script:Repo = New-Repo @('p/new/a.py') @()
         { Get-ReleasePlan $script:Repo $graph } | Should -Throw '*FirstVersion*'
     }
+
+    It 'says how many files counted' {
+        $script:Repo = New-Repo $script:AllFiles ($script:AllTags + 'ipi/4.5.0')
+        Add-Commit $script:Repo @('device/dd/a.py', 'device/dd/b.py')
+        (@(Get-ReleasePlan $script:Repo $script:Graph) |
+            Where-Object Name -eq 'dd').Reason |
+            Should -Be 'changed since 4.5.96 (2 files)'
+    }
+
+    It 'releases nothing for a change only to documentation, tests or examples' {
+        $script:Repo = New-Repo $script:AllFiles ($script:AllTags + 'ipi/4.5.0')
+        Add-Commit $script:Repo @(
+            'device/dd/readme.md', 'device/dd/docs/guide.txt',
+            'device/dd/tests/test_a.py', 'device/dd/test/a.py',
+            'device/dd/examples/a.py', 'device/dd/LICENSE',
+            'pipeline/core/sub/notes.md')
+        @(Get-ReleasePlan $script:Repo $script:Graph).Count | Should -Be 0
+    }
+
+    It 'ignores documentation, tests and examples whatever their case' {
+        $script:Repo = New-Repo $script:AllFiles ($script:AllTags + 'ipi/4.5.0')
+        Add-Commit $script:Repo @(
+            'device/dd/README.MD', 'device/dd/Tests/a.py',
+            'device/dd/Examples/a.py')
+        @(Get-ReleasePlan $script:Repo $script:Graph).Count | Should -Be 0
+    }
+
+    It 'releases for the code changed beside the documentation' {
+        $script:Repo = New-Repo $script:AllFiles ($script:AllTags + 'ipi/4.5.0')
+        Add-Commit $script:Repo @('device/dd/readme.md', 'device/dd/a.py')
+        $plan = @(Get-ReleasePlan $script:Repo $script:Graph)
+        $plan.Name | Should -Be @('dd')
+        $plan[0].Reason | Should -Be 'changed since 4.5.96 (1 file)'
+    }
+
+    It 'leaves out the files a package''s Ignore globs match' {
+        $graph = @($script:Graph | ForEach-Object { $_.PSObject.Copy() })
+        ($graph | Where-Object Name -eq 'dd') |
+            Add-Member Ignore @('device/dd/tox.ini')
+        $script:Repo = New-Repo $script:AllFiles ($script:AllTags + 'ipi/4.5.0')
+        Add-Commit $script:Repo @('device/dd/tox.ini')
+        @(Get-ReleasePlan $script:Repo $graph).Count | Should -Be 0
+    }
+
+    It 'counts documentation for a package that turns the defaults off' {
+        $graph = @($script:Graph | ForEach-Object { $_.PSObject.Copy() })
+        ($graph | Where-Object Name -eq 'dd') | Add-Member IgnoreDefaults $false
+        $script:Repo = New-Repo $script:AllFiles ($script:AllTags + 'ipi/4.5.0')
+        Add-Commit $script:Repo @('device/dd/readme.md')
+        @(Get-ReleasePlan $script:Repo $graph).Name | Should -Be @('dd')
+    }
+
+    It 'refuses IgnoreDefaults given as text' {
+        $graph = @($script:Graph | ForEach-Object { $_.PSObject.Copy() })
+        ($graph | Where-Object Name -eq 'dd') | Add-Member IgnoreDefaults 'false'
+        $script:Repo = New-Repo $script:AllFiles ($script:AllTags + 'ipi/4.5.0')
+        Add-Commit $script:Repo @('device/dd/readme.md')
+        { Get-ReleasePlan $script:Repo $graph } |
+            Should -Throw '*IgnoreDefaults*true or false*'
+    }
+}
+
+Describe 'release-plan-filter.ps1' {
+    BeforeAll {
+        # A setup.py, setup.cfg and pyproject.toml like the ones here, with
+        # the metadata the filter leaves out and settings it does not. Line
+        # endings are made LF, whatever the checkout gave this file.
+        $script:SetupPy = @'
+import setuptools
+
+setuptools.setup(
+    name="dd",
+    version=read("version.txt"),
+    author="51Degrees Engineering",
+    author_email="engineering@51degrees.com",
+    project_urls={"Source": "https://github.com/51Degrees/51degrees-python"},
+    description=("Device detection."),
+    keywords=["51degrees", "device-detection"],
+    long_description=read("readme.md"),
+    long_description_content_type='text/markdown',
+    python_requires=">=3.10",
+    install_requires=["fiftyone_pipeline_core"],
+    classifiers=["Programming Language :: Python"],
+)
+'@.Replace("`r`n", "`n")
+        $script:SetupCfg = @'
+[metadata]
+name = dd
+description = Device detection.
+url = https://github.com/51Degrees/51degrees-python
+
+[options]
+install_requires =
+    fiftyone_pipeline_core
+'@.Replace("`r`n", "`n") + "`n"
+        $script:Pyproject = @'
+[build-system]
+requires = ["setuptools>=42", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "dd"
+description = "Device detection."
+keywords = [
+    "51degrees",
+    "device-detection",
+]
+dependencies = ["fiftyone_pipeline_core"]
+
+[project.urls]
+Source = "https://github.com/51Degrees/51degrees-python"
+'@.Replace("`r`n", "`n") + "`n"
+
+        # A repository with the given files, holding the filter as this
+        # repository does, with the tags of the small graph and the tag of
+        # dd on the commit that adds the files.
+        function New-FilterRepo([hashtable]$Files) {
+            $repo = New-Repo $script:AllFiles ($script:AllTags + 'ipi/4.5.0')
+            $null = New-Item -ItemType Directory -Force (Join-Path $repo 'ci')
+            foreach ($name in 'release-plan-filter.ps1', 'release-plan-filter.py') {
+                Copy-Item (Join-Path $PSScriptRoot $name) (Join-Path $repo 'ci')
+            }
+            Set-Files $repo $Files
+            git -C $repo tag -f 'dd/4.5.96' | Out-Null
+            return $repo
+        }
+
+        # Commits the files with exactly the text given.
+        function Set-Files([string]$Repo, [hashtable]$Files) {
+            foreach ($file in $Files.Keys) {
+                $full = Join-Path $Repo $file
+                $null = New-Item -ItemType Directory -Force (Split-Path $full)
+                [IO.File]::WriteAllText($full, $Files[$file])
+            }
+            git -C $Repo add -A
+            git -C $Repo -c user.name=test -c user.email=test@example.com `
+                commit --quiet --allow-empty -m 'Change'
+        }
+
+        function Get-DdPlan([string]$Repo) {
+            return @(Get-ReleasePlan $Repo $script:Graph |
+                Where-Object Name -eq 'dd')
+        }
+    }
+
+    It 'does not release for a setup.py change only to metadata' {
+        $script:Repo = New-FilterRepo @{ 'device/dd/setup.py' = $script:SetupPy }
+        $changed = $script:SetupPy
+        $changed = $changed.Replace('Device detection.', 'Device detection for Python.')
+        $changed = $changed.Replace('"device-detection"]', '"device-detection", "user-agent"]')
+        $changed = $changed.Replace('51degrees-python"}', '51degrees-python", "Docs": "x"}')
+        $changed = $changed.Replace('51Degrees Engineering', '51Degrees')
+        $changed = $changed.Replace('"Programming Language :: Python"]',
+            '"Programming Language :: Python", "Typing :: Typed"]')
+        $changed = $changed.Replace("'text/markdown'", "'text/x-rst'")
+        Set-Files $script:Repo @{ 'device/dd/setup.py' = $changed + "# A note.`n" }
+        (Get-DdPlan $script:Repo).Count | Should -Be 0
+    }
+
+    It 'releases for a setup.py change to anything else' {
+        $script:Repo = New-FilterRepo @{ 'device/dd/setup.py' = $script:SetupPy }
+        $changed = $script:SetupPy.Replace('["fiftyone_pipeline_core"]',
+            '["fiftyone_pipeline_core>=4.5.40"]')
+        Set-Files $script:Repo @{ 'device/dd/setup.py' = $changed }
+        $plan = Get-DdPlan $script:Repo
+        $plan.Count | Should -Be 1
+        $plan[0].Reason | Should -Be 'changed since 4.5.96 (1 file)'
+    }
+
+    It 'counts the code changed beside a setup.py change only to metadata' {
+        $script:Repo = New-FilterRepo @{ 'device/dd/setup.py' = $script:SetupPy }
+        Set-Files $script:Repo @{
+            'device/dd/setup.py' = $script:SetupPy.Replace(
+                'Device detection.', 'Device detection for Python.')
+            'device/dd/src/a.py' = 'print(1)'
+        }
+        (Get-DdPlan $script:Repo)[0].Reason |
+            Should -Be 'changed since 4.5.96 (1 file)'
+    }
+
+    It 'does not release for a setup.cfg change only to metadata' {
+        $script:Repo = New-FilterRepo @{ 'device/dd/setup.cfg' = $script:SetupCfg }
+        $changed = $script:SetupCfg.Replace('Device detection.', 'Device detection for Python.')
+        $changed = $changed.Replace('51degrees-python', '51degrees-python/tree/main')
+        Set-Files $script:Repo @{ 'device/dd/setup.cfg' = $changed }
+        (Get-DdPlan $script:Repo).Count | Should -Be 0
+    }
+
+    It 'releases for a setup.cfg change to its options' {
+        $script:Repo = New-FilterRepo @{ 'device/dd/setup.cfg' = $script:SetupCfg }
+        Set-Files $script:Repo @{ 'device/dd/setup.cfg' =
+            $script:SetupCfg + "    fiftyone_pipeline_engines`n" }
+        (Get-DdPlan $script:Repo).Count | Should -Be 1
+    }
+
+    It 'does not release for a pyproject.toml change only to metadata' {
+        $script:Repo = New-FilterRepo @{ 'device/dd/pyproject.toml' = $script:Pyproject }
+        $changed = $script:Pyproject.Replace('"Device detection."', '"Device detection for Python."')
+        $changed = $changed.Replace("    `"device-detection`",`n",
+            "    `"device-detection`",`n    `"user-agent`",`n")
+        $changed = $changed.Replace('Source = ', "Docs = `"x`"`nSource = ")
+        $changed += "`n[[project.authors]]`nname = `"51Degrees`"`n"
+        Set-Files $script:Repo @{ 'device/dd/pyproject.toml' = $changed }
+        (Get-DdPlan $script:Repo).Count | Should -Be 0
+    }
+
+    It 'does not release for a change only to the maintainers in <Case>' -ForEach @(
+        @{ Case = 'setup.py'; File = 'setup.py'; Start = 'SetupPy'
+           Was = 'author_email="engineering@51degrees.com",'
+           Changed = "author_email=`"engineering@51degrees.com`",`n    maintainer=`"51Degrees`",`n    maintainer_email=`"support@51degrees.com`"," }
+        @{ Case = 'setup.cfg'; File = 'setup.cfg'; Start = 'SetupCfg'
+           Was = "name = dd`n"
+           Changed = "name = dd`nmaintainer = 51Degrees`nmaintainer_email = support@51degrees.com`n" }
+        @{ Case = 'a pyproject.toml key'; File = 'pyproject.toml'; Start = 'Pyproject'
+           Was = 'dependencies = '
+           Changed = "maintainers = [{ name = `"51Degrees`" }]`ndependencies = " }
+        @{ Case = 'a pyproject.toml table'; File = 'pyproject.toml'; Start = 'Pyproject'
+           Was = '[project.urls]'
+           Changed = "[[project.maintainers]]`nname = `"51Degrees`"`n`n[project.urls]" }
+    ) {
+        $text = @{
+            SetupPy = $script:SetupPy
+            SetupCfg = $script:SetupCfg
+            Pyproject = $script:Pyproject
+        }[$Start]
+        $changed = $text.Replace($Was, $Changed)
+        $changed | Should -Not -Be $text
+        $script:Repo = New-FilterRepo @{ "device/dd/$File" = $text }
+        Set-Files $script:Repo @{ "device/dd/$File" = $changed }
+        (Get-DdPlan $script:Repo).Count | Should -Be 0
+    }
+
+    It 'releases for a pyproject.toml change to <Changed>' -ForEach @(
+        @{ Was = '["fiftyone_pipeline_core"]'; Changed = '["fiftyone_pipeline_core>=4.5"]' }
+        @{ Was = '"setuptools>=42"'; Changed = '"setuptools>=61"' }
+    ) {
+        $script:Repo = New-FilterRepo @{ 'device/dd/pyproject.toml' = $script:Pyproject }
+        Set-Files $script:Repo @{ 'device/dd/pyproject.toml' =
+            $script:Pyproject.Replace($Was, $Changed) }
+        (Get-DdPlan $script:Repo).Count | Should -Be 1
+    }
+
+    It 'releases for a manifest that is new since the last tag' {
+        $script:Repo = New-FilterRepo @{}
+        Set-Files $script:Repo @{ 'device/dd/setup.py' = $script:SetupPy }
+        (Get-DdPlan $script:Repo).Count | Should -Be 1
+    }
+
+    It 'releases for a manifest it cannot read' {
+        $script:Repo = New-FilterRepo @{ 'device/dd/setup.py' = $script:SetupPy }
+        Set-Files $script:Repo @{ 'device/dd/setup.py' = 'setup(' }
+        (Get-DdPlan $script:Repo).Count | Should -Be 1
+    }
+
+    It 'releases for a manifest that is not UTF-8' {
+        $script:Repo = New-FilterRepo @{ 'device/dd/setup.py' = $script:SetupPy }
+        [IO.File]::WriteAllBytes((Join-Path $script:Repo 'device/dd/setup.py'),
+            [byte[]](0x23, 0xff, 0xfe, 0x0a))
+        Set-Files $script:Repo @{}
+        (Get-DdPlan $script:Repo).Count | Should -Be 1
+    }
 }
 
 Describe 'Get-BuildPlan' {
@@ -320,6 +581,40 @@ Describe 'the real packages.json' {
         Get-ToxGaps (Split-Path $PSScriptRoot) $script:Real |
             Should -BeNullOrEmpty
     }
+
+    Context 'a change to a package''s tox.ini' {
+        BeforeAll {
+            $published = @($script:Real | Where-Object Publish -eq $true)
+            $script:RealTags = @($published | ForEach-Object { "$($_.Name)/4.5.1" })
+            $script:RealFiles = @($script:Real | ForEach-Object {
+                "$($_.Path)/src/a.py"
+                "$($_.Path)/tox.ini"
+            })
+        }
+
+        It 'releases nothing, whichever package it sits beside' {
+            $script:Repo = New-Repo $script:RealFiles $script:RealTags
+            Add-Commit $script:Repo ($script:Real | ForEach-Object { "$($_.Path)/tox.ini" })
+            @(Get-ReleasePlan $script:Repo $script:Real).Count | Should -Be 0
+        }
+
+        It 'still releases a package for a change to its source' {
+            $script:Repo = New-Repo $script:RealFiles $script:RealTags
+            Add-Commit $script:Repo @(
+                'pipeline/fiftyone_pipeline_core/tox.ini',
+                'pipeline/fiftyone_pipeline_core/src/a.py')
+            $plan = @(Get-ReleasePlan $script:Repo $script:Real)
+            $plan.Name | Should -Contain 'fiftyone_pipeline_core'
+            ($plan | Where-Object Name -eq 'fiftyone_pipeline_core').Reason |
+                Should -Be 'changed since 4.5.1 (1 file)'
+        }
+
+        It 'is ignored by every package, so one added later counts for nothing' {
+            foreach ($package in $script:Real) {
+                $package.Ignore | Should -Contain '**/tox.ini' -Because $package.Name
+            }
+        }
+    }
 }
 
 }