Updates to improve handling of 404s for bibliography entries

README.md

@@ -150,6 +150,8 @@ cd scripts
 
 This script retrieves the bibliography from Zotero, formats it appropriately, and places the individual JSON files in `static/data/bibItems/`.
 
+Entries marked as deleted in Zotero are recorded in `data/bibliography/removed.json`. The custom 404 page uses that list to recognize former bibliography URLs and show the removed entry's title. The file is merged, not replaced, so entries that Zotero has already purged are retained.
+
 ---
 
 ## Local Development
@@ -323,8 +325,8 @@ repository's GitHub Pages site:
 
 | Deployment | URL |
 |------------|-----|
-| Staging root | `https://interlisp.github.io/Interlisp.staging/` |
-| PR #123 preview | `https://interlisp.github.io/Interlisp.staging/pr-123/` |
+| Staging root | `https://interlisp.org/Interlisp.staging/` |
+| PR #123 preview | `https://interlisp.org/Interlisp.staging/pr-123/` |
 
 When a PR is opened or updated, the production workflow (`gh-pages.yml`)
 triggers the `deploy-preview` workflow in the staging repository, which:
@@ -358,6 +360,7 @@ The repository follows the standard [Hugo directory structure](https://gohugo.io
 | `assets/` | Custom global resources (CSS, JS, icons, SCSS) |
 | `config/` | Site configuration for different environments |
 | `content/en/` | All website content (English only) |
+| `data/` | Build-time data files |
 | `layouts/` | Hugo layout templates and overrides |
 | `static/` | Files copied directly to the built site |
 
@@ -395,6 +398,13 @@ The repository follows the standard [Hugo directory structure](https://gohugo.io
 
 </details>
 
+<details>
+<summary><strong>data/</strong> — Build-time data</summary>
+
+- `bibliography/removed.json` — Keys and titles of bibliography entries removed from Zotero (read by the 404 page)
+
+</details>
+
 <details>
 <summary><strong>static/</strong> — Static files</summary>
 
@@ -468,3 +478,4 @@ Modifying search scope requires updating the Google Custom Search engine setting
 This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
 
 Copyright (c) 2026 Interlisp.org
+

config/staging/hugo.yaml

@@ -1,4 +1,7 @@
-baseURL: https://interlisp.github.io/Interlisp.staging/
+baseURL: https://interlisp.org/Interlisp.staging/
+
+# Per-PR previews override this at build time so that links stay inside the
+# preview, e.g. https://interlisp.org/Interlisp.staging/pr-<N>/
 
 locale: en-us
 

content/en/history/_index.md

@@ -4,6 +4,7 @@ weight: 500
 type: docs
 aliases:
  - /medley/history/
+ - /doc/info/history.html
 ---
 
 ## The Short History

content/en/project/partners/_index.md

@@ -4,6 +4,7 @@ weight: 18
 type: docs
 aliases:
  - /medley/project/partners
+ - /about/partners/
 ---
 
 ### [Software Preservation Network](https://www.softwarepreservationnetwork.org/about/)

content/en/software/install-and-run/_index.md

@@ -7,6 +7,7 @@ aliases:
  - /developing/
  - /software/emu-compat/
  - /using/running/
+ - /running/running-with-docker
 weight: 10
 type: docs
 ---

content/en/software/install-and-run/windows/_index.md

@@ -3,6 +3,8 @@ title: Install and Run on Windows
 linkTitle: Windows
 weight: 40
 type: docs
+aliases:
+ - /medley/using/running/runnning-on-win
 ---
 
 There are two ways to install and run Medley on Windows. These two ways result in (nearly) identical Medley experiences, but differ in how Medley is installed and started.

content/en/software/using-medley/online-using.md

@@ -2,6 +2,8 @@
 title: Using Medley Online
 weight: 40
 type: docs
+aliases:
+ - /running/online/usingonline
 ---
 
 

data/bibliography/removed.json

@@ -0,0 +1,66 @@
+[
+  {
+    "key": "47P4BUHS",
+    "title": "Script for testing defstruct and the inspector with defstruct"
+  },
+  {
+    "key": "4R2EXMY4",
+    "title": "TEST APPRENTICE"
+  },
+  {
+    "key": "59UEUE22",
+    "title": "THE INTERLISP-D TESTING SYSTEM - USER GUIDE"
+  },
+  {
+    "key": "7KUMIRNC",
+    "title": "Special Files on <Test>ARs>"
+  },
+  {
+    "key": "D8B2SFFU",
+    "title": "Running AR Test Cases & Recording the Results"
+  },
+  {
+    "key": "ERU4YWDT",
+    "title": "Script for INSPECTALLFIELDSFLG"
+  },
+  {
+    "key": "F7XLXRG5",
+    "title": "Lyric Manual Test procedures"
+  },
+  {
+    "key": "JFGREFUI",
+    "title": "THE INTERLISP-D TESTING SYSTEM"
+  },
+  {
+    "key": "KHWFGC5M",
+    "title": "Debugger, Error System, and Unwinder test report"
+  },
+  {
+    "key": "LJQYU5HX",
+    "title": "Scripts for testing the record package"
+  },
+  {
+    "key": "RJLLRQDR",
+    "title": "SEdit test report"
+  },
+  {
+    "key": "SHRDJ9J4",
+    "title": "Testing Report"
+  },
+  {
+    "key": "SVIQ724G",
+    "title": "Test Utilities"
+  },
+  {
+    "key": "TAYYCYB8",
+    "title": "DEdit test report"
+  },
+  {
+    "key": "TTHB6JCX",
+    "title": "Testing the programmatic interface to INSPECTW"
+  },
+  {
+    "key": "UIS8C4AI",
+    "title": "Script to check code inspectors on frames"
+  }
+]

layouts/404.html

@@ -5,11 +5,36 @@
   </aside>
   <main class="col-12 col-md-9 col-xl-8 ps-md-5 pt-0" role="main">
     <div class="td-content mt-0 pt-0">
-      <h1 class="mt-0">Page Not Found</h1>
-      <header class="article-meta"></header>
-      <p>The page you were looking for does not exist. It may have been moved or removed.</p>
-      <p>You can use the search box or browse the site sections on the left to find what you are looking for.</p>
-      <p>Or <a href="{{ "" | relURL }}">return to the homepage</a>.</p>
+      <div id="not-found-generic">
+        <h1 class="mt-0">Page Not Found</h1>
+        <header class="article-meta"></header>
+        <p>The page you were looking for does not exist. It may have been moved or removed.</p>
+        <p>You can use the search box or browse the site sections on the left to find what you are looking for.</p>
+        <p>Or <a href="{{ "" | relURL }}">return to the homepage</a>.</p>
+      </div>
+
+      <div id="not-found-bibliography" class="d-none">
+        <h1 class="mt-0">Bibliography Entry Not Found</h1>
+        <header class="article-meta"></header>
+        <p>
+          The bibliography entry <code id="not-found-bib-key"></code><span id="not-found-bib-title-wrap"> &mdash; <cite id="not-found-bib-title"></cite></span>
+          is no longer available.
+        </p>
+        <p>
+          The Interlisp bibliography is generated from our
+          <a href="https://www.zotero.org/groups/2914042/interlisp">Zotero group library</a>.
+          Entries may be removed or reclassified when the library changes, which removes
+          the corresponding page from this site. If you followed a link from another
+          site, that link may be out of date.
+        </p>
+        <p>You can:</p>
+        <ul>
+          <li><a id="not-found-bib-browse" href="{{ "history/bibliography/" | relURL }}">Browse the bibliography</a></li>
+          <li><a id="not-found-bib-zotero" href="https://www.zotero.org/groups/2914042/interlisp" rel="noopener">View the item in Zotero</a> (the record may still be available)</li>
+          <li><a id="not-found-bib-intake" href="{{ "history/bibliography/intake/" | relURL }}">Report a missing or incorrect entry</a></li>
+        </ul>
+        <p>Or <a id="not-found-bib-home" href="{{ "" | relURL }}">return to the homepage</a>.</p>
+      </div>
     </div>
   </main>
   <aside class="d-none d-xl-block col-xl-2 td-sidebar-toc d-print-none pt-0 mt-0">
@@ -19,4 +44,70 @@ <h1 class="mt-0">Page Not Found</h1>
     </div>
   </aside>
 </div>
+
+{{- $bibData := index hugo.Data "bibliography" -}}
+{{- $removed := slice -}}
+{{- with $bibData -}}
+  {{- $removed = .removed | default (slice) -}}
+{{- end -}}
+<script type="application/json" id="bibliography-removed-data">{{ $removed | jsonify | safeJS }}</script>
+<script id="not-found-script">
+  (function () {
+    const path = window.location.pathname;
+
+    // Match the end of the path so this also works when the 404 page is
+    // served for a staging preview, e.g.
+    // /Interlisp.staging/pr-355/history/bibliography/<key>/.
+    // GitHub Pages supports only one custom 404 per site, and a preview can be
+    // served by the domain-level 404 page, so the build-time baseURL cannot be
+    // used to decide what kind of URL this is.
+    const match = path.match(/\/history\/bibliography\/([^/?#]+(?:\/[^/?#]+)*)\/?(?:[?#]|$)/);
+    if (!match) return;
+
+    const key = match[1].replace(/\/+$/, "");
+    // Everything before "history/", including any staging/PR prefix.
+    const siteRoot = path.slice(0, match.index + 1);
+
+    const generic = document.getElementById("not-found-generic");
+    const bib = document.getElementById("not-found-bibliography");
+    if (!generic || !bib) return;
+
+    generic.classList.add("d-none");
+    bib.classList.remove("d-none");
+
+    // Keep the links inside the same preview when the page is served from a
+    // staging path.
+    document.getElementById("not-found-bib-browse").href =
+      siteRoot + "history/bibliography/";
+    document.getElementById("not-found-bib-intake").href =
+      siteRoot + "history/bibliography/intake/";
+    document.getElementById("not-found-bib-home").href = siteRoot;
+
+    const displayKey = key.toUpperCase();
+    document.getElementById("not-found-bib-key").textContent = displayKey;
+    document.title = "Bibliography entry not found | The Medley Interlisp Project";
+
+    let removed = [];
+    const data = document.getElementById("bibliography-removed-data");
+    if (data) {
+      try {
+        removed = JSON.parse(data.textContent) || [];
+      } catch (e) {
+        removed = [];
+      }
+    }
+
+    const entry = removed.find(function (item) {
+      return String(item.key || "").toUpperCase() === displayKey;
+    });
+    if (entry && entry.title) {
+      document.getElementById("not-found-bib-title").textContent = entry.title;
+    } else {
+      document.getElementById("not-found-bib-title-wrap").classList.add("d-none");
+    }
+
+    document.getElementById("not-found-bib-zotero").href =
+      "https://www.zotero.org/groups/2914042/items/" + encodeURIComponent(displayKey);
+  })();
+</script>
 {{ end }}

layouts/_partials/hooks/head-end.html

@@ -8,4 +8,9 @@
 {{ with .Params.robots }}
 <meta name="robots" content="{{ . | htmlEscape }}">
 {{ end }}
+{{- /* The 404 page is served with an HTTP 404 status.  Tell crawlers not to
+       index it even if they ignore the status code. */ -}}
+{{ if eq .Kind "404" }}
+<meta name="robots" content="noindex, follow">
+{{ end }}
 {{ partial "bibliography-json-ld.html" . }}

scripts/update_bibliography.sh

@@ -355,6 +355,24 @@ if $debugFiles ; then
 fi
 
 showInfo 5 "Delete items that Zotero indicated as such."
+# Save the deleted entries before filtering them out.  The custom 404 page uses
+# this list to identify bibliography URLs that were removed and show their
+# titles.  Keep only keys and titles; the list is embedded in 404.html at build
+# time.  Merge with the existing file instead of replacing it: once Zotero
+# purges an item, the API no longer reports it as deleted, but its old URL can
+# still be requested by search engines and external links.
+REMOVED_DATA_FILE="./../data/bibliography/removed.json"
+mkdir -p "$(dirname "$REMOVED_DATA_FILE")"
+deletedItems=$(jq 'map(select(.deleted)) | map({key: .key, title: (.title // "")})' <<< "$items")
+if [[ -f "$REMOVED_DATA_FILE" ]]; then
+  # Newly deleted entries override older records with the same key.
+  jq --argjson deleted "$deletedItems" \
+    '(INDEX(.key) + ($deleted | INDEX(.key))) | [.[]] | sort_by(.key)' \
+    "$REMOVED_DATA_FILE" > "$REMOVED_DATA_FILE.tmp"
+else
+  printf '%s\n' "$deletedItems" > "$REMOVED_DATA_FILE.tmp"
+fi
+mv "$REMOVED_DATA_FILE.tmp" "$REMOVED_DATA_FILE"
 beforeCount=$(jq '. | length' <<< "$items")
 items=$(jq 'map(select(.deleted|not))' <<< "$items")
 afterCount=$(jq '. | length' <<< "$items")

staging-site-design.md

@@ -107,7 +107,7 @@ In the staging repo, `deploy-preview.yml`:
 
 1. The `build` job calls the same shared reusable workflow against the PR head
    commit with `hugo-environment: staging`, the per-PR baseURL
-   (`https://interlisp.github.io/Interlisp.staging/pr-<N>/`), and
+   (`https://interlisp.org/Interlisp.staging/pr-<N>/`), and
    `run-build-tests: false`.
 2. The `deploy` job downloads the `github-pages` artifact, extracts
    `artifact.tar`, and deploys the contents to the `pr-<N>/` subdirectory of
@@ -191,8 +191,8 @@ the staging repo.
 - **Pages:** staging repo serves from the `gh-pages` branch ("Deploy from a
   branch", not "GitHub Actions").
 - **Staging config:** `config/staging/hugo.yaml` sets
-  `baseURL: https://interlisp.github.io/Interlisp.staging/` and titles pages
-  "Staging Environment".
+  `baseURL: https://interlisp.org/Interlisp.staging/` and titles pages
+  "Staging Environment". Per-PR previews override the baseURL at build time.
 - **Repo pins:** both callers pin the shared workflow at `@main` of
   `Interlisp/shared-workflows`.
 - **`Interlisp/shared-workflows` `main`:** protected — all updates land via PR

static/IRM_files/content.htm

@@ -0,0 +1,3 @@
+<script type="text/javascript">
+	document.location = "/documentation/IRM.pdf"
+</script>