fix: restore post comments through Reddit GraphQL by fxmfxmfx · Pull Request #603 · redlib-org/redlib

fxmfxmfx · 2026-10-24T13:03:57Z

Reddit now responds to anonymous comments JSON requests with an HTML 403, including the encoded-slash workaround. This makes every post page fail with “Failed to parse page JSON data”, while subreddit listings still work.

This change fetches post metadata through /by_id and loads comments using Android's PostComments persisted GraphQL operation with the existing anonymous OAuth client. The converter preserves the two-Listing shape consumed by post.rs, including rich text, flair, moderation flags, scores, replies, and focused comment context. Cursor links paginate both top-level comments and nested replies without an account or new configuration.

The converter and RTJSON renderer are adapted from polak14's implementation, with attribution retained. This patch keeps the current dependencies and Rust version, omits the Shreddit scraper fallback, and adds strict path/sort/cursor handling, safe HTML URLs, explicit HTTP/API errors, and focused-context pruning. In particular, nested cursors must not be combined with focusedCommentContext: Reddit returns DEPENDENCY_FAILURE for that combination.

Validation:

45 offline library tests passed, including 9 GraphQL conversion/request tests.
cargo fmt --all -- --check, cargo clippy --locked --all-targets -- -D warnings, and release build passed.
Live requests from the affected Debian LXC validated all supported sorts, focused comments/context, and opaque cursor continuation with the anonymous client.
End-to-end release-binary checks in the affected LXC passed: the reported post, focused permalink and context, top/new sorts, a large thread with 200 initial comments plus 186 new comments on the next page, and a nested continuation with 6 replies. The same verified binary was installed in production with rollback backup; the production revision and post rendering were checked after restart.
Existing external-network tests were excluded from the offline run. An earlier broader run failed the existing /r/gonwild fetching test with a Reddit transport error; 47 tests passed in that run.

Limits: each GraphQL request returns up to 200 comments at 10 levels of depth; continuation links load further comments/replies. Reddit's persisted operation hash and API behavior are external dependencies.

Related: #591, #602.

Reddit rejects anonymous comments JSON requests with an HTML 403, including encoded-slash paths. Fetch post metadata through by_id and transform the Android PostComments forest into the Listing shape used by the renderer. Keep sorting, focused comment context, nested replies, and cursor pagination. Escape rich text and reject unsafe URL schemes; report upstream HTTP/API failures without treating them as JSON syntax errors. Cache successful thread responses for 30 seconds and keep existing OAuth and post access checks. Adapt the converter and RTJSON renderer from polak14's a63460b implementation, without the unrelated dependency updates or Shreddit scraper fallback. Validation: nine GraphQL converter/request tests, cargo fmt and strict clippy; live anonymous GraphQL requests from the affected container validate sorts, focused comments, and cursor pagination. Co-authored-by: polak14 <76823966+polak14@users.noreply.github.com>

Reddit returns DEPENDENCY_FAILURE when PostComments combines an after cursor with focusedCommentContext. A cursor already identifies its subtree, so only send focused context for the initial permalink request. Cover this with request-variable tests and verify the failing nested cursor against the live API.

nomadlogic · 2026-10-24T19:09:15Z

just tested this out from a fresh checkout of your fork and built locally via cargo run --release and am seeing the same, or similar, issues with viewing threads.

$ cargo run --release
    Finished `release` profile [optimized] target(s) in 0.20s
     Running `target/release/redlib`
Starting Redlib...
Running Redlib v0.40.0 on [::]:8080!
 ERROR redlib::client > Got an invalid response from reddit expected value at line 1 column 1. Status code: 403 Forbidden
 ERROR redlib::utils  > Error page rendered: Failed to parse page JSON data: expected value at line 1 column 1

Reddit rejects anonymous comments JSON requests with an HTML 403, including encoded-slash paths. Fetch post metadata through by_id and transform the Android PostComments forest into the Listing shape used by the renderer. Keep sorting, focused comment context, nested replies, and cursor pagination. Escape rich text and reject unsafe URL schemes; report upstream HTTP/API failures without treating them as JSON syntax errors. Cache successful thread responses for 30 seconds and keep existing OAuth and post access checks. Adapt the converter and RTJSON renderer from polak14's a63460b implementation, without the unrelated dependency updates or Shreddit scraper fallback. Validation: nine GraphQL converter/request tests, cargo fmt and strict clippy; live anonymous GraphQL requests from the affected container validate sorts, focused comments, and cursor pagination. Co-authored-by: polak14 <76823966+polak14@users.noreply.github.com> [highda: resolved conflicts with the Tor commit in src/lib.rs, src/post.rs and templates/post.html] (cherry picked from commit d8824fa in redlib-org#603)

Reddit returns DEPENDENCY_FAILURE when PostComments combines an after cursor with focusedCommentContext. A cursor already identifies its subtree, so only send focused context for the initial permalink request. Cover this with request-variable tests and verify the failing nested cursor against the live API. (cherry picked from commit 6cd8366 in redlib-org#603)

Establishing a cold onion circuit can take longer than the 5 s authentication budget, so token requests over Tor timed out and retried. Allow 90 s in onion mode and keep the retry delay at 5 s. [highda: only the src/oauth.rs part of "fix: restore Reddit comments and onion API requests"; this fork takes the .json suffix fix from LucifersCircle and comments from upstream PR redlib-org#603 instead of the Shreddit scraper] (cherry picked from commit a86b3e037059553d08311000d77bf5b4512ecb06 in https://git.ptr.moe/baalajimaestro/redlib) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YTnpudqPXqMbtNkNsqFK8x

fxmfxmfx · 2026-10-25T02:33:21Z

@nomadlogic Thanks for testing. Could you confirm the branch and commit you built? The fork's default main currently points to upstream ba116c1; this PR's changes are on fix/graphql-comments, currently at 6cd8366.

To test the PR branch from a fresh checkout:

git clone --single-branch --branch fix/graphql-comments https://github.com/fxmfxmfx/redlib.git redlib-graphql
cd redlib-graphql
git rev-parse --short HEAD
cargo run --locked --release

This revision was tested on the affected instance with post pages, focused comments/context, top-level pagination, and nested reply pagination.

If the 403 persists on 6cd8366, please share the failing post URL, whether subreddit listings still load, and the full error including the endpoint path if available. The redlib::client JSON parse error can also come from the /by_id metadata request, so the endpoint will help distinguish that from the new GraphQL comments request.

nomadlogic · 2026-10-25T03:07:41Z

@nomadlogic Thanks for testing. Could you confirm the branch and commit you built? The fork's default main currently points to upstream ba116c1; this PR's changes are on fix/graphql-comments, currently at 6cd8366.

To test the PR branch from a fresh checkout:
git clone --single-branch --branch fix/graphql-comments https://github.com/fxmfxmfx/redlib.git redlib-graphql
cd redlib-graphql
git rev-parse --short HEAD
cargo run --locked --release
This revision was tested on the affected instance with post pages, focused comments/context, top-level pagination, and nested reply pagination.

If the 403 persists on 6cd8366, please share the failing post URL, whether subreddit listings still load, and the full error including the endpoint path if available. The redlib::client JSON parse error can also come from the /by_id metadata request, so the endpoint will help distinguish that from the new GraphQL comments request.

thanks! i missed that that this wasn't in the main branch. just tested again using the correct branch and can confirm it's working on my end.

freedit-dev · 2026-10-25T03:32:50Z

Thanks. It works well.

…t GraphQL redlib-org#603

evrial · 2026-10-25T13:05:17Z

compose file to build and run it: (pulls 3.8Gb of rust garbage)

services:
  redlib:
    build:
      context: https://github.com/redlib-org/redlib.git#refs/pull/603/head
      dockerfile: Dockerfile.alpine
    restart: unless-stopped
    container_name: redlib
    ports:
      - 8080:8080
    user: nobody
    read_only: true
    security_opt:
      - no-new-privileges:true
      # - seccomp=seccomp-redlib.json

…raphQL Reddit answers anonymous comments JSON requests with an HTML 403, the encoded-slash workaround in v0.40.0 included. Taken from redlib-org#603 (fxmfxmfx/redlib fix/graphql-comments at 6cd8366) ahead of an upstream merge.

Comments are back on thread pages without a Reddit login. **Why:** since 2026-10-23 Reddit 403s the thread JSON routes for app-only tokens, and it has closed self-service API apps (Responsible Builder Policy), so logging in is not available. The Android app's persisted `PostComments` GraphQL query on gql-fed still answers our anonymous token: verified 2026-10-25 from the NAS (134 and 198 comments on two threads). **What** (a port of upstream PR redlib-org#603; converter and RTJSON renderer adapted from polak14's a63460b, attribution kept): - `gql_comments.rs` / `rtjson.rs`: pure request builder, comment-forest → JSON-API Listing converter, and the richtext renderer (`content.html` is always empty for this query). Only http(s) and local links survive `url_attr`. - `client::gql`: one pipeline with `fetch_json` (`fetch_reddit`), same JSON cache. GraphQL `errors` become an Err, so they are never cached. gql-fed meters the token on its own budget (599 vs 99), so its headers never overwrite the session's REST counter (tested). - `thread.rs`: the post from `/by_id/` plus the comments, concurrent when the sort is known. A comments failure logs a warn and renders the post alone with the notice. - `comments.rs`: comment building moved out of `post.rs`, which keeps its fan-out at 9. "More replies" links carry the cursor, depth-10 continuations read "Continue this thread", and a "More comments" link follows the first 200. - Maintenance point: `POST_COMMENTS_HASH`. When Reddit retires it, threads fall back to post-only and the log names the GraphQL error. Tests: 116 (+19 new), synthetic fixtures only; the converter was also checked against a captured real response (134 comments, 2 continuations). Docs: decisions.md, fork.md (new section; redlib-org#603 marked ported), AGENTS.md. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

fix: restore post comments through Reddit GraphQL

…/graphql-comments redlib 0.40.0 and restore post comments through Reddit GraphQL

juliancalzada · 2026-10-27T09:54:52Z

Thanks! This fixed the issue on my local container, too. Apprecaite your efforts.

…t (0.43.2) (#54) The first review of upstream's **open PRs** (commits are unchanged at `ba116c1`), and the review process now covers them: `make upstream-review` lists open upstream PRs updated after `.upstream-pr-rev`. Record: `docs/fork.md` "Reviews"; decision in `docs/decisions.md`. ## Ported | Upstream | Bug here | Fix | |---|---|---| | redlib-org#506 CMAF video | live: 29 of 30 recent v.redd.it posts rendered `<video src="">` | `/vid/<id>/{dash,cmaf}/<size>`; the old `/vid/<id>/<size>` stays for pages sw.js cached | | redlib-org#560 profile share links | `/u/<name>/s/<id>` answered "Nothing here" | one `post::share_redirect` for `/r/`, `/u/`, `/user/`; `canonical_target` now only follows a resolved `/comments/` path (an echoed share link would loop) | | redlib-org#521 default post sort | `REDLIB_DEFAULT_POST_SORT` ignored until settings saved | `subreddit::listing_sort`, also used by the RSS feed (the PR missed it) | | redlib-org#603 retry | a GraphQL 200 with `errors` (field timeout) hid comments | retried once; never for `PersistedQueryNotFound` | | redlib-org#568 (intent) | footer "Code" linked upstream | this fork at the running sha (`instance_info::source_url`) | | redlib-org#396 | `/commits.atom`, `/instances.json` could panic | removed; nothing used them | ## Found while checking against live threads - Taken-down comments (`authorInfo: null`) linked `u/[unknown]` → `[deleted]`. - Reddit's `isInitiallyCollapsed` was ignored → `collapsed` (admin takedowns, low score). - AMA `qa` sort was served but missing from the sort menu (showed "Confidence") → `Q&A` option. - Checked fine: giphy (an `external-preview.redd.it` image), emotes (plain text), comment media (images only). ## Skipped redlib-org#599 (HTML scrape, worse on every axis), redlib-org#549, redlib-org#542, redlib-org#520, redlib-org#194, redlib-org#413, redlib-org#596, redlib-org#561, redlib-org#577, and feature/build PRs. Reasons in `docs/fork.md`. ## Verification Local instance vs reddit.box: CMAF posts render `/vid/…/cmaf/…` → 206 video/mp4; DASH and the legacy route proxy; footer link; `qa` selected on two AMAs; three real `/r/…/s/…` links redirect; `/u/foo/s/abcdefgh` reaches the share handler; a CryptoCurrency daily shows 0 `[unknown]` (2 on reddit.box) and Reddit-collapsed comments closed. ci-quick, ci-unit (118), ci-metrics (coverage 47.8 → 48.72, tightened). Not in this PR: 19 new RUSTSEC advisories (13 in zstd crates, 0 high/critical) appeared in the advisory DB; a dependency PR follows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Since 2026-10-28 Reddit answers every anonymous JSON API request (listings, about, by_id, users, search) and /svc/shreddit/ with a 403 "blocked by network security" page. client::json now answers those paths from the Android app's persisted GraphQL operations (gql_feed.rs), converted to the old JSON, so handlers and templates stay unchanged: - Feeds: SubredditFeedElements, PopularFeedElements, AllFeedSdui; combined feeds (r/a+b, the subscriptions front page) merge per-subreddit feeds. - Posts: PostsByIds; videos from packaged MP4s with sound, proxied at /packaged/ and /packaged-external/. - Subreddit info (ban/private/gated/quarantine reasons), user info, posts and comments, post and subreddit search, wiki pages and duplicates. - GraphQL's own rate limit (600 per token) drives the token rollover. Comments take the fixes of redlib-org#603 (adapted from our a63460b): strict post paths, comment sort mapping and the post's suggested sort, ?context=N, no focused thread with a cursor, safer link URLs, escaped text when richtext is missing, comment ID checks and a request timeout. The same input checks apply to the new paths. The shreddit scraper fallback is gone.

fxmfxmfx · 2026-10-29T10:34:02Z

Update: Redlib is currently unusable because Reddit is blocking anonymous JSON API requests, so pages and listings no longer load. This PR only addresses comments and is no longer sufficient to restore Redlib on its own.

polak14’s broader GraphQL patch aims to restore the affected endpoints and incorporates the comment fixes from this PR.

zImPatrick · 2026-10-29T11:02:41Z

Update: Redlib is currently unusable because Reddit is blocking anonymous JSON API requests, so pages and listings no longer load. This PR only addresses comments and is no longer sufficient to restore Redlib on its own.

polak14’s broader GraphQL patch aims to restore the affected endpoints and incorporates the comment fixes from this PR.

FYI you can just switch to another TLS fingerprint, it'll work again

Thats what I did for https://r.meowing.de for now

(hi reddit employees 👋)

Reddit's edge now answers most anonymous API requests with a 403 "blocked by network security" HTML page, which surfaced as "Failed to parse page JSON data: expected value at line 1 column 1" (redlib-org#608). - client: replace the Chrome145/Firefox147 TLS emulations, which Reddit now blocks outright, with OkHttp 4.9, Firefox Android 135 and Chrome 131 profiles that still get JSON. Shreddit fragments use Chrome 131/Windows. Log the chosen emulation. - client: when Reddit returns a non-JSON 403, start a new OAuth session (token and device ID) so a flagged session doesn't break the instance until restart. - user: /user/{name}/about.json is blocked, so build the profile from the user's hover card fragment (name, avatar, karma, cake day) and the profile subreddit's /r/u_{name}/about.json (title, description, banner, NSFW flag). Missing accounts are detected from the hover card. User post/comment listings still work through the JSON API (redlib-org#607). Comments come from the shreddit HTML loader merged from redlib-org#599. Fixes redlib-org#608 Refs redlib-org#607, redlib-org#598, redlib-org#600, redlib-org#602, redlib-org#599, redlib-org#603 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Since 2026-10-28 Reddit answers every anonymous JSON API request (listings, about, by_id, users, search) and /svc/shreddit/ with a 403 "blocked by network security" page. client::json now answers those paths from the Android app's persisted GraphQL operations (gql_feed.rs), converted to the old JSON, so handlers and templates stay unchanged: - Feeds: SubredditFeedElements, PopularFeedElements, AllFeedSdui; combined feeds (r/a+b, the subscriptions front page) merge per-subreddit feeds. - Posts: PostsByIds; videos from packaged MP4s with sound, proxied at /packaged/ and /packaged-external/. - Subreddit info (ban/private/gated/quarantine reasons), user info, posts and comments, post and subreddit search, wiki pages and duplicates. - GraphQL's own rate limit (600 per token) drives the token rollover. Comments take the fixes of redlib-org#603 (adapted from our a63460b): strict post paths, comment sort mapping and the post's suggested sort, ?context=N, no focused thread with a cursor, safer link URLs, escaped text when richtext is missing, comment ID checks and a request timeout. The same input checks apply to the new paths. The shreddit scraper fallback is gone.

Reddit now returns 403 for thread pages requested with anonymous tokens, including the %2f workaround. Port upstream PR redlib-org#603 (redlib-org/redlib): the post still comes from /by_id, and comments from the PostComments query the Android app uses, converted to the usual listing shape. Adjustments: dates parsed with `time` instead of adding chrono, a smaller cache (32 threads) for the Pi, the existing inline "More replies" links kept, and a "More comments" link for the next page of top-level comments. The JSON endpoint is still used when a script app is configured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fxmfxmfx and others added 2 commits October 24, 2026 22:50

itsjfx mentioned this pull request Oct 25, 2026

fix comments blocked by reddit corenting/eddrit#371

Open

This was referenced Oct 25, 2026

fix: load post comments via Reddit GraphQL (v2.3.8) mitchross/redlib#27

Merged

redlib: bump to 2.3.8 — post pages via GraphQL mitchross/talos-argocd-proxmox#3159

Merged

simonswine added a commit to simonswine/redlib that referenced this pull request Oct 25, 2026

Merge upstream PR redlib-org#603: restore post comments through Reddi…

63f91a1

…t GraphQL redlib-org#603

ButteredCats added a commit to ButteredCats/redlib that referenced this pull request Oct 25, 2026

Merge Redlib 0.40.0 and redlib-org#603 into catsarch

054b67d

ButteredCats mentioned this pull request Oct 25, 2026

[6/25] Rate limit part ??? #446

Open

agandhi4 mentioned this pull request Oct 26, 2026

Thread comments from the Android GraphQL query (0.43.0) agandhi4/redlib#49

Merged

EnkryptattuDev added a commit to EnkryptattuDev/redlib that referenced this pull request Oct 27, 2026

Merge pull request redlib-org#603 from fxmfxmfx:fix/graphql-comments

7fb87cb

fix: restore post comments through Reddit GraphQL

EnkryptattuDev added a commit to EnkryptattuDev/redlib that referenced this pull request Oct 27, 2026

Merge redlib 0.40.0 and pull request redlib-org#603 from fxmfxmfx:fix…

501935d

…/graphql-comments redlib 0.40.0 and restore post comments through Reddit GraphQL

agandhi4 mentioned this pull request Oct 28, 2026

Upstream open-PR review: CMAF video, profile share links, default sort (0.43.2) agandhi4/redlib#54

Merged

Navigation Menu

Repository navigation

fix: restore post comments through Reddit GraphQL - #603

fix: restore post comments through Reddit GraphQL#603
fxmfxmfx wants to merge 2 commits into
redlib-org:mainfrom
fxmfxmfx:fix/graphql-comments

fxmfxmfx commented Oct 24, 2026

Uh oh!

nomadlogic commented Oct 24, 2026

Uh oh!

fxmfxmfx commented Oct 25, 2026

Uh oh!

nomadlogic commented Oct 25, 2026

Uh oh!

freedit-dev commented Oct 25, 2026

Uh oh!

evrial commented Oct 25, 2026 •

edited

Loading

Uh oh!

juliancalzada commented Oct 27, 2026

Uh oh!

fxmfxmfx commented Oct 29, 2026

Uh oh!

zImPatrick commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

fxmfxmfx commented Oct 24, 2026

Uh oh!

nomadlogic commented Oct 24, 2026

Uh oh!

fxmfxmfx commented Oct 25, 2026

Uh oh!

nomadlogic commented Oct 25, 2026

Uh oh!

freedit-dev commented Oct 25, 2026

Uh oh!

evrial commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

juliancalzada commented Oct 27, 2026

Uh oh!

fxmfxmfx commented Oct 29, 2026

Uh oh!

zImPatrick commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

evrial commented Oct 25, 2026 •

edited

Loading