Skip to content

feat(payload): add content collections with sheet import and comparison - #46

Merged
maxitect merged 12 commits into
mainfrom
feat/33-replace-the-google-sheet-with-payload-for-site
Nov 12, 2026
Merged

maxitect merged 12 commits into
mainfrom
feat/33-replace-the-google-sheet-with-payload-for-site

Conversation

@maxitect

@maxitect maxitect commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds Payload with the collections, import and comparison scripts the sheet replacement needs, and makes the cached data functions read from Payload when DATABASE_URL is set. Production has no DATABASE_URL, so it keeps reading the sheet and nothing visitors see changes until the switchover.

Changes

  • Payload 3.93 base: users, media (images only, localised alt), en/fr localisation, GraphQL off, S3 storage mapped in one file (src/lib/payload/storage.ts, S3_* names only), admin and REST routes under src/app/(payload)/. Advent calendar live at /calendrier by 1 December #29 adds collections to src/payload.config.ts without touching the rest.
  • Collections tour-dates, music-items, lyrics (explicit order; lyrics status draft/published, public reads published only) and globals about-page, tour-page. Rules the sheet never had: tour date needs date and venue, links must be https:, a music src must be on the host of its type (YouTube, Spotify, Bandcamp), publishing a song needs lyrics.
  • src/lib/data/content.ts keeps the same cached functions and shapes, so pages and components are untouched. It picks payload.ts when DATABASE_URL is set and sheet.ts otherwise. Saving or deleting in the admin expires the cache tag straight away.
  • pnpm content:import (idempotent: upserts on date and venue / src / title and album, trims values, keeps sheet order, songs without lyrics become drafts) and pnpm content:compare (reads both sources, reports any difference, exit 1 if any).
  • netlify.toml: payload migrate runs before the build only when DATABASE_URL is set, so production is unaffected until the switchover.
  • docs/admin-guide.md for the band; docs/tickets/INSTRUCTIONS.md describes the two content sources.

Acceptance criteria

AC Result Method Evidence
After import: 11 tour dates, 7 music items, 17 songs (15 published), 2 pages in both languages; compare reports no difference PASS (local database, Neon preview and dev) cli, db AC1 below
/about, /tour, /music, /lyrics look the same, both languages, same order PASS browser AC2 below: 16 screenshots, sheet build against Payload build, 0 px different; deploy preview against production, only the Netlify preview drawer differs
/sitemap.xml, /llms.txt, .md pages list the same content PASS api AC3 below (one intended difference: trimmed trailing space)
No request fails during the switch (Netlify logs) Post-merge deploy Step 4 below
A band member adds a tour date and sees it on /tour within a minute, no redeploy PASS api AC5 below
A tour date with no date, or a non-https: ticket link, cannot be saved PASS api, db AC6 below
A signed-out visitor cannot open /admin or read drafts through the API PASS api AC7 below
After step 6 the site makes no request to Google Sheets Post-merge code Sheet code stays until step 6
Production build and behaviour unchanged without Payload env vars PASS cli AC9 below

Closes

Closes #33

Notes

  • Do not treat Replace the Google Sheet with Payload for site content #33 as finished on merge. The Closes line is required by the PR script; reopen the issue or track steps 2 to 6 below. This PR is switchover step 1 plus the code for step 4.
  • Neon preview and dev were reset to their parent, migrated and imported; content:compare prints "No difference." on both. The deploy preview builds and serves the imported content.
  • content:import and content:compare run with NODE_ENV=production, so they never push the schema in dev mode; a dev push leaves a marker that makes the next payload migrate stop at an interactive prompt.
  • While DATABASE_URL is unset, next.config.ts rewrites /admin/* and /api/* (except /api/send-emails) to a 404, as before Payload.
  • Storage credentials on deploy previews: not proven by an upload. The access key id matches the current one and the current key reads the preview bucket; a first image upload on the preview admin settles it.
  • Open question answered as proposed: the 2 songs without lyrics are imported as drafts. They stay hidden on the site.
  • database.md register: no departures. lyrics.status is a plain select, not versions.drafts, so required columns stay NOT NULL. The lyrics and tour-dates Payload _order columns come from orderable: true.
  • Tour dates keep an explicit order too so .md pages, which list in sheet order, stay identical. The /tour page still sorts by date.
  • /admin logs a dev-only Next warning about runtime data in generateMetadata() (Payload's template page). The page renders and the production build passes.
  • Hydration warning on /tour?isFrench=true is the existing client-side language switch, unrelated.
  • CSP allowances for /admin belong to Security hardening: headers, CMS URLs, email and account settings #34; robots.ts does not disallow /admin yet.
  • AGENTS.md is the notice next dev writes; committed to keep the tree clean.

Post-merge steps (production, not done here)

  1. Set on Netlify, production context: DATABASE_URL (Neon main, pooled), PAYLOAD_SECRET, S3_ENDPOINT, S3_REGION, S3_BUCKET, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY. Setting DATABASE_URL is what flips the site to Payload at the next deploy, so hold DATABASE_URL back until step 4 and set the others first. The build runs payload migrate once DATABASE_URL exists; or run it by hand against main before (step 1).
  2. Create the band admin accounts at /admin (first visit creates the first user); share docs/admin-guide.md.
  3. Import: with the main env loaded and SHEET_CSV_URL, run pnpm content:import, then pnpm content:compare until it prints "No difference."
  4. Ask the band to stop editing the sheet, re-run import and compare, then set DATABASE_URL on production and deploy. Watch the Netlify logs through the deploy. Roll back in Netlify if anything is wrong.
  5. After a week: remove src/lib/data/sheet.ts, src/utils/fetchSheet.ts, parseCsv.ts, parsePageCsv.ts, the import and compare scripts and SHEET_CSV_URL, then unpublish the sheet.

Review

Verdict: ALL CLEAR
Rounds: 2

Finding Severity Resolution
Content scripts dev-pushed the schema, so the next payload migrate hung on an interactive prompt critical fixed in 07de60b
A published song could not be saved in the fr locale (French lyrics required to publish) warning fixed in d11e0b2
Collections read with fallbackLocale: false, against database.md; French category came out undefined warning fixed in 3e61f66
Without Payload env vars, /admin and /api/* answered 200 with an error screen and 500 instead of 404 warning fixed in 27082fd
Deploy preview failed on the #32 spike schema in preview critical fixed: preview reset, migrated and imported; preview green
getMusicItems and getLyrics pair the en and fr queries by index suggestion open: ordering is stable on _order; left as is

Re-verified: AC1 (cli, db) PASS · AC2 (browser, 0 px on 16 local shots; preview vs production differs only by the Netlify drawer) PASS · AC6 (cli) PASS · AC7 (deploy: signed-out /api/lyrics?where[status][equals]=draft totalDocs 0, /api/lyrics 15 published, /admin shows the login) PASS · AC9 (cli: no-env build, /admin /api/users /api/lyrics 404, /api/send-emails GET 405 as on production; with DATABASE_URL, /admin/login 200, /api/users 403) PASS · check.sh PASS

Evidence

AC1

$ pnpm exec payload migrate           (local Postgres 17)
Migrated:  20261112_223859_initial (31ms)
$ pnpm content:import
about-page: updated
tour-page: updated
tour-dates: { created: 11 }
music-items: { created: 7 }
lyrics: { created: 17 }
$ pnpm content:import                 (second run, no duplicates)
tour-dates: { updated: 11 }
music-items: { updated: 7 }
lyrics: { updated: 17 }
$ psql -Atc "select count(*) from tour_dates; select count(*) from music_items; select count(*), count(*) filter (where status='published') from lyrics; select count(*) from about_page_locales"
11
7
17|15
2
$ pnpm content:compare
SAME  about page
SAME  tour page
SAME  tour dates (11)
SAME  music items (7)
SAME  lyrics (15)
No difference.

Negative: after adding a tour date and a music item, the same command printed DIFF tour dates, DIFF music items, 2 differ. and exited 1. The migration contains no jsonb or geometry column and no _rels, _texts or _numbers table for our collections (grep over the generated SQL: only payload_* tables).

AC2

Headless Chromium, reduced motion, full page, /about /tour /music /lyrics x en/fr x 1280/390 (16 shots).
Sheet build (no DATABASE_URL) vs Payload build (local Postgres, imported), next start on :4033:
$ compare -metric AE payload/<shot>.png sheet/<shot>.png
0 (0) on all 16; identical dimensions.
Deploy preview (Neon preview, imported) vs https://clarksbowlingclub.com (sheet):
identical dimensions on all 16; the only differing region is the Netlify preview drawer at the bottom.
Console errors other than blocked third-party embeds: 0.

AC3

Sheet-sourced server vs Payload-sourced server, after dropping Next's per-request ids and <lastmod>:
IDENTICAL llms.txt
IDENTICAL sitemap.xml
IDENTICAL about.md
IDENTICAL music.md
IDENTICAL lyrics.md
DIFF tour.md
$ diff pay/tour.md sheet/tour.md
7c7
< **30/04/2025** - Ninkasi Cordeliers, Lyon, FR
---
> **30/04/2025** - Ninkasi Cordeliers, Lyon, FR 
(the sheet's trailing space, trimmed on import as the ticket asks)

AC5

before: curl -s localhost:4033/tour | grep -c "Zenith Test Hall"   -> 0
POST /api/tour-dates (signed in)                                   -> CREATED id 12
after 2s: 1   after 4s: 1   after 6s: 1     (no redeploy, no restart)

AC6

no date:         'This field is required.' (eventDate)
http ticket:     'Must be a link starting with https://' (ticketUrl)
javascript link: 'Must be a link starting with https://' (locationUrl)
no venue:        'This field is required.' (venue)
valid https:     CREATED id 12
music src evil.com:        'Must be on www.youtube.com or www.youtube-nocookie.com'
music src http:            'Must be a link starting with https://'
spotify link, youtube type: 'Must be on www.youtube.com or www.youtube-nocookie.com'
publish song without lyrics: 'Required to publish'
$ psql -c "insert into tour_dates (venue,location) values ('a','b')"
DETAIL:  Failing row contains (13, null, null, a, b, null, null, ...)   (event_date NOT NULL)

AC7

GET /api/users                  -> 403
GET /api/lyrics?limit=0         -> totalDocs 15, drafts 0
GET /api/lyrics?where[status][equals]=draft -> totalDocs 0
POST /api/graphql               -> 404
POST /api/tour-dates unsigned   -> 403
GET /admin signed out           -> login form only

AC9

.env.local reduced to SHEET_CSV_URL and EMAIL_HOST; no DATABASE_URL, PAYLOAD_SECRET or S3_*:
$ sh -c 'if [ -n "$DATABASE_URL" ]; then pnpm exec payload migrate; fi && pnpm build'
Compiled successfully; 20/20 static pages; /about /music /lyrics /tour prerender (read the sheet).
$ DATABASE_URL=<local> .claude/skills/ticket/scripts/check.sh
CHECK PASS: lint typecheck build

AC9, guard (round 1 fix)

.env.local = SHEET_CSV_URL, EMAIL_HOST only; pnpm build && pnpm start
admin 404 | admin/login 404 | api/users 404 | api/lyrics 404 | api/users/me 404
about 200 | music 200 | sitemap.xml 200 | api/send-emails GET 405 (route still reached)

@netlify

netlify Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for clarksbowlingclub ready!

Name Link
🔨 Latest commit 27082fd
🔍 Latest deploy log https://app.netlify.com/projects/clarksbowlingclub/deploys/6af645a1b517f62d0d17bf7f
😎 Deploy Preview https://deploy-preview-46--clarksbowlingclub.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@maxitect
maxitect marked this pull request as ready for review November 12, 2026 23:11
@maxitect
maxitect merged commit 30f2859 into main Nov 12, 2026
5 checks passed
@maxitect
maxitect deleted the feat/33-replace-the-google-sheet-with-payload-for-site branch November 12, 2026 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace the Google Sheet with Payload for site content

1 participant