Skip to content

morca ui: robots.txt disallows every crawler, and pages say noindex - #542

Merged
winding-lines merged 1 commit into
mainfrom
ui-robots
Nov 18, 2026
Merged

winding-lines merged 1 commit into
mainfrom
ui-robots

Conversation

@winding-lines

Copy link
Copy Markdown
Contributor

Since 1.13.0 added a request log, prod's UI log shows that most traffic comes from crawlers: SemrushBot, Amazonbot, Meta's crawler, Reflectionbot and ElevenlabsCrawler, plus scrapers with browser User-Agents. They walk endless URLs (every run, and every snapshot × tab × view) and the data views, which take about 1.1 s each. The site's only robots.txt was Cloudflare's, which has comments but no rules.

  • /robots.txt: User-agent: * / Disallow: /, served before anything else, without a login.
  • Every answer carries X-Robots-Tag: noindex, nofollow.
  • tests/ui.sh checks both; docs/ui.md says so.

Tested locally (ouromo): test-ui ok.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6

@winding-lines
winding-lines merged commit 7807931 into main Nov 18, 2026
35 checks passed
@winding-lines
winding-lines deleted the ui-robots branch November 18, 2026 05:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant