Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 31 additions & 5 deletions .github/workflows/publish-site.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,10 +79,36 @@ jobs:

# A Pages deployment can report success, be recorded as the active deployment, and
# still leave the site serving the previous one. Every step above went green for
# 1.0.0 while the download page offered 0.9.5, and nothing said so; re-running this
# workflow published the same files and fixed it. Reading the site back is the only
# place that failure is visible, so it fails the run that caused it rather than
# waiting to be noticed.
- name: Verify the site is serving this deployment
# 1.0.0 while the download page offered 0.9.5, and nothing said so. Reading the site
# back is the only place that failure is visible.
#
# It is neither rare nor random: it is what a release does to itself. A Pages
# deployment is identified by commit SHA - deploy-pages sends the SHA as
# pages_build_version and then polls a status keyed on that same SHA - and a release
# redeploys a commit Pages already holds, because the release tag, the pre-release
# tag and main are all one commit. The second deployment of that commit is reported
# as succeeded, since the status read back belongs to the first one, and is then
# never served. That is why release-triggered runs failed while pushes and manual
# runs passed, and why re-running by hand always fixed it.
#
# So the first look is deliberately short - a deployment that is going to appear is
# serving within about ten seconds, and a long wait here only delays the thing that
# actually fixes it - and the deploy that a person used to do by hand is done here.
- id: firstlook
name: Verify the site is serving this deployment
continue-on-error: true
shell: pwsh
run: ./scripts/verify-published-site.ps1 -Folder site -TimeoutSeconds 90

- id: redeployment
name: Deploy again
if: steps.firstlook.outcome == 'failure'
uses: actions/deploy-pages@v4

# The full wait this time. If the same files are still not served after a second
# deployment then something other than the collision is wrong, and the run says so
# rather than passing quietly.
- name: Verify the second deployment
if: steps.firstlook.outcome == 'failure'
shell: pwsh
run: ./scripts/verify-published-site.ps1 -Folder site
18 changes: 17 additions & 1 deletion docs/release-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,23 @@ from the live domain after deploying and fails if they are not the ones it just
(`scripts/verify-published-site.ps1`, which also runs by hand against any folder holding a
`CNAME` and the manifests). It compares against the deployed files rather than the newest
release, because a pre-release deployment leaves `stable.json` untouched and that is
correct. When it fails, the fix is **Run workflow** — the files are already right.
correct.

**Why that kept happening on releases, and what the workflow now does about it.** It was
not bad luck. A Pages deployment is identified by commit SHA: `deploy-pages` sends the SHA
as `pages_build_version` and then polls a deployment status keyed on that same SHA. A
release redeploys a commit Pages already holds — the release tag, the pre-release tag and
`main` are all one commit — so the second deployment of that commit reads the *first* one's
status, reports success within seconds, and is then never served. The numbers said so
plainly: pushes passed 9 out of 9 and manual runs 8 out of 8, while release-triggered runs
passed 7 out of 23, and the stalled deployment sat at `in_progress` for five minutes where
a healthy one reaches `success` in about ten seconds.

Deploying the same files again clears it, so the workflow does that itself rather than
waiting to be asked: a short first check, a second `deploy-pages` if the site is still
stale, then the full check. A run that fails now has failed twice, and the collision is not
the explanation — look under **Settings → Environments → `github-pages`** for a deployment
stuck in progress on that commit.

### Azure Pipelines

Expand Down
18 changes: 14 additions & 4 deletions scripts/verify-published-site.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,15 @@
deployment leaves stable.json untouched, which is correct, and a check written
against the newest release would fail on every one of them.

Read-only. It cannot repair a deployment - the remedy is to run the workflow again,
which publishes the same files - so its whole job is to make a silent staleness loud.
The usual cause is a release redeploying a commit Pages already holds: a Pages
deployment is identified by commit SHA, and the release tag, the pre-release tag and
main are all one commit, so the second deployment of it reports the first one's status
and is never served. The workflow answers that by deploying again, which is why this
runs twice there - once on a short timeout before the retry, once on the full one
after.

Read-only. It cannot repair a deployment - the remedy is to deploy the same files
again - so its whole job is to make a silent staleness loud.

.PARAMETER Folder
Folder holding the manifests that were deployed, and the CNAME naming where they were
Expand Down Expand Up @@ -113,6 +120,9 @@ throw @"
$BaseUrl is still serving an older deployment after $TimeoutSeconds seconds ($stale).

The deployment this run made succeeded, so the files are right and the site has not
picked them up. Run this workflow again - Actions -> Publish site -> Run workflow - and
it will publish the same files. That has been enough every time so far.
picked them up. Deploying the same files again clears it, and the workflow already does
that once by itself - so if this is the second attempt, the collision it works around is
not the whole story. Run it again by hand - Actions -> Publish site -> Run workflow - and
look for a deployment still in progress on this commit under the github-pages
environment.
"@