Ecency canonical to /@user/permlink, refresh app registry, validate data

[feruzm]

Prepares 1.4.0.

Ecency canonical

ecency and esteem now use https://ecency.com/@{username}/{permlink}.

ecency.com already serves a 308 from /{category}/@{author}/{permlink} to the short form, so the registry was telling every frontend to emit a canonical URL that permanently redirects. Links posted on-chain have already moved: in a sample of recent posts, /@author/permlink outnumbered the category form.

Registry accuracy

leofinance renamed to InLeo and repointed to inleo.io, which is where its post URLs resolve.

Added four apps that publish today but had no entry, so their posts canonicalised to whichever frontend happened to render them:

id	share of a 1,200-post sample	scheme
leothreads	7.0%	https://inleo.io/@{username}/{permlink}
liketu	3.6%	https://liketu.com/post/{username}/{permlink}
scrobble.life	3.3%	https://scrobble.life/p/{username}/{permlink}
waivio	0.5%	https://www.waivio.com/@{username}/{permlink}

scrobble.life and waivio were checked against live posts: /@user/permlink 308s to /p/... on the former, and the apex 301s to www on the latter.

Removed entries

Five entries pointed at domains that are no longer the app:

id	observed
clicktrackprofit	ctptalk.com and www.ctptalk.com both 301 to an unrelated domain that lands on a gambling site
proofofbrain	serves a parked-domain JS redirect with a signed token
eskateraleigh	eskateraleigh.com has no NS and no SOA record
stemsocial	stem.openhive.network 404s on every path; its CNAME target is an unpublished GitHub Pages site. Its #!/ scheme was never usable as a canonical URL anyway
inji	persistent 522

Removing the key rather than blanking url_scheme is deliberate: apps[identifier] becomes falsy and consumers fall back to their own scheme. That is correct behaviour for an app with no live home, and it works on old and new copies of the README recipe alike. clicktrackprofit is the urgent one, since canonical links pass ranking authority to whoever holds the domain now.

README recipe fixes

The documented canonical recipe threw a TypeError on any entry without a url_scheme, and beempy and steempress are both in the registry and in active use (beempy was 1.2% of the sample). It also assumed json_metadata was an object and app was a string, neither of which holds across all API methods and apps.

Old vs new, same inputs:

case	old	new
beempy (no url_scheme)	TypeError	falls back
steempress (no url_scheme)	TypeError	falls back
app written as an object	TypeError	resolves
json_metadata as a string (condenser_api)	wrong link	resolves
Ecency/3.1.4 (uppercase)	wrong link	resolves

The bad-actor and bad-domain recipes now use a Set, normalise input, and parse the URL instead of regexing it. Two consequences: new URL().hostname returns punycode, so homograph entries finally match, and walking parent domains catches login.phishing-site.tk against an entry for phishing-site.tk.

Data

• bad-domains.json was a strict subset of spaminator-domains.json. Pulled in the 49 domains it was missing, since bad-domains.json is the list the README tells developers to use.
• Added the punycode twin of the one homograph entry. The Unicode-only form could never match a parsed link.
• Removed two entries from spaminator-all.json that cannot be Hive accounts (names must start with a letter).
• Sorted and de-duplicated every list, so future community PRs produce readable diffs.

Tooling

scripts/validate.mjs runs on pull requests and before publish. It checks list shape, sorting, duplicates, casing, whitespace, good/bad overlap, homograph coverage, and apps.json placeholders, https, and hash fragments. Every guard was verified by mutating the data and confirming it fails; all 21 fire.

Workflows moved off checkout@v1, setup-node@v1 and Node 12. package.json gained a files allowlist, so .github and lockfiles no longer ship to npm.

Left for a separate decision

steemit still canonicalises Hive posts to steemit.com. Those are pre-fork posts that exist on both chains, so it is a community call rather than a data fix.

[feruzm]

All four fixed in a88eb05..HEAD. Thanks, #1 was mine and it was a bad one.

1. web.app (High). Confirmed and it is worse than a stale entry: web.app was not in bad-domains.json before this PR. It arrived through the spaminator sync in this same PR, and the parent-domain walk I added in this same PR is what turned it into a match for every Firebase-hosted site. Two changes that were each fine alone.

security-alert-portal.web.app was already listed separately, so the bare suffix was carrying no signal. Removed from bad-domains.json and spaminator-domains.json.

Since the entry came in from an upstream import, removing it alone would not stop the next one, so I checked all three lists against the Public Suffix List: web.app was the only hit. Added a guard for the class, backed by a vendored PSL snapshot (8,800 multi-label rules; single-label TLDs cannot be entries because every entry must contain a dot). It rejects web.app, github.io and co.uk. The snapshot is dev-only and outside the files allowlist, and scripts/update-public-suffix-list.mjs refreshes it.

2. Trailing dot (Medium). Confirmed: new URL("https://steemit24.cf./").hostname is "steemit24.cf.", which misses the entry. Fixed with .replace(/\.$/, "") before the www. strip.

3. Homepage check (Medium). Confirmed. Moved name and homepage above the optional-scheme branch. Verified both directions: an http:// homepage now fails whether or not the entry has a url_scheme.

4. Test script (Low). Confirmed. Dropped the scripts block rather than shipping dev tooling, since the PSL snapshot would have added 129 kB to every install. CI already invokes the validator directly and the README documents it for contributors.

Verification

Regression cases are run against the isBadDomain recipe extracted from README.md at test time, so the documented code cannot drift from what was tested:

link	before	now
https://legitimate-project.web.app/	true	false
https://my-startup.web.app	true	false
https://security-alert-portal.web.app/	true	true
https://steemit24.cf./	false	true
https://STEEMIT24.CF./	false	true
https://github.io/	n/a	false

Validator guards are now 25, each re-verified by mutating the data and confirming it fails; all 25 fire and the clean tree passes.

[feruzm]

All three fixed in 68c25a8. #1 was a good catch: I vendored a suffix list and then discarded the part of the format that carries the meaning.

1. PSL rule types (Medium). Confirmed, and wrong in both directions as you describe. ^[!*.]+ turned *.ck into ck, which my own "must contain a dot" filter then dropped, losing foo.ck entirely; and !www.ck became www.ck, marking a registrable domain as a suffix. 283 wildcard and 8 exception rules were affected.

The snapshot now stores all three rule types verbatim and validate.mjs implements the PSL algorithm: exception rules win outright, otherwise the longest matching rule prevails, with the implicit * as fallback.

One more problem in the same vendoring step: 459 upstream rules are unicode (公司.cn, aéroport.ci), and the data files store punycode, so those rules could never have matched an entry. The updater now converts at fetch time.

domain	before	now	why
foo.ck	false	true	wildcard *.ck
anything.ck	false	true	wildcard *.ck
www.ck	true	false	exception !www.ck
foo.kawasaki.jp	false	true	wildcard
city.kawasaki.jp	true	false	exception
a.foo.ck	false	false	registrable under foo.ck
xn--55qx5d.cn	false	true	punycode IDN rule

I also added four negative mutation cases (www.ck, city.kawasaki.jp, evil2.web.app, a.foo.ck all listed) so the guard cannot regress into rejecting registrable domains. A suffix check that over-rejects would push contributors to work around it, which is how the bare web.app would come back.

2. Non-string app name (Medium). Confirmed: {app: {name: 42}} threw raw?.split is not a function. json_metadata is arbitrary author-supplied JSON, so the type is now checked before any string method. Verified {app:{name:42}}, {app:{}}, {app:42}, {app:null}, {app:[]} and json_metadata: null — none throw, all fall back.

3. apps.json shape (Medium). Confirmed: [] passed and reported "0 apps". Now requires a non-null, non-array object, and an empty registry fails too. Worth noting apps.json of null was also broken but differently: it skipped the block and then hit Object.keys(null), failing with a stack trace rather than a message. Both give a clear failure now.

Verification

The PSL cases run against the algorithm read out of validate.mjs, and the canonical cases against the recipe extracted from README.md, so neither test can drift from the code it covers.

Guards are now 31 must-fail plus 4 must-pass, all re-verified by mutating the data; every one behaves correctly and the clean tree passes.

[feruzm]

Both fixed in e70af8f.

1. Non-string homepage (Medium). Confirmed: String(["https://example.com"]) is "https://example.com", so the array passed. The check now requires a string, then https, then a parseable URL.

Verified against array, object, number, boolean, null and "https://" with no host, all now rejected with the right message. A valid string and an absent homepage still pass, so entries like beempy are unaffected.

This was the last field checked by coercion. name and url_scheme already tested typeof, so nothing else in the registry has the same shape.

2. MPL 2.0 notice (Low). Confirmed, and worth fixing properly rather than pasting the text in. The notice is now read from the upstream file and carried into the generated snapshot, so a reworded upstream notice still lands, and the updater refuses to write if it cannot find one. Verified by serving it a copy of the list with the notice stripped:

guard FIRES: upstream licence notice not found, refusing to write

The generated file now opens with the upstream Exhibit A text followed by attribution:

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at https://mozilla.org/MPL/2.0/.
#
# Generated from https://publicsuffix.org/list/public_suffix_list.dat, punycode normalised.
# Licensed under MPL 2.0: https://mozilla.org/MPL/2.0/. Not part of the npm package.

README records the same, including that the snapshot sits outside the files allowlist. npm pack --dry-run confirms it: the tarball is the six data files plus README and package.json, so the published package stays MIT and MPL-free. MPL 2.0 is file-level copyleft, so the notice travelling with the file is all that is required, and the rest of the repository is unaffected.

Guards are now 34 must-fail plus 6 must-pass, all verified by mutation, plus the licence guard above.