8608fa27e6
Verified live 2026-09-11: an external reader's robots.txt-respecting fetch tool could load the repo root (README) but refused every actual file/source page once git.0ff.space went external, because Forgejo's built-in default robots.txt disallows /*/*/src/ and /*/*/raw/ -- exactly the paths that serve file content. Overrides it via the same customDir + tmpfiles L+ mechanism this file already uses for branding assets: identical to the default except those two lines are dropped. Everything else (blame, commits, diffs, actions, attachments) stays disallowed on purpose -- those are the actually expensive endpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016w25covboHfcYuLXBAvX3g
76 lines
2 KiB
Text
76 lines
2 KiB
Text
# Custom override of Forgejo's default robots.txt (docs/intent/... none yet,
|
|
# see forgejo.nix's systemd.tmpfiles.rules comment for how this is wired in).
|
|
# Identical to Forgejo's own default except /*/*/src/ and /*/*/raw/ are no
|
|
# longer disallowed -- those are exactly the paths that serve actual file
|
|
# content, and blocking them was silently defeating the point of making
|
|
# git.0ff.space reachable at all (an external reader's own robots.txt-
|
|
# respecting fetch tool refused every file/source page, verified live
|
|
# 2026-09-11). Everything else stays disallowed on purpose: blame/commits/
|
|
# diffs/actions/etc. are the genuinely expensive-to-crawl endpoints Forgejo
|
|
# was actually trying to protect against repeated automated hits.
|
|
|
|
User-agent: *
|
|
Disallow: /api/
|
|
Disallow: /avatars/
|
|
Disallow: /user/
|
|
Disallow: /swagger.*.json
|
|
Disallow: /explore/*?*
|
|
|
|
Disallow: /repo/create
|
|
Disallow: /repo/migrate
|
|
Disallow: /org/create
|
|
Disallow: /*/*/fork
|
|
|
|
Disallow: /*/*/watchers
|
|
Disallow: /*/*/stargazers
|
|
Disallow: /*/*/forks
|
|
|
|
Disallow: /*/*/blame/
|
|
Disallow: /*/*/commit/
|
|
Disallow: /*/*/commits/
|
|
Disallow: /*/*/media/
|
|
Disallow: /*/*/tags
|
|
Disallow: /*/*/graph
|
|
Disallow: /*/*/branches
|
|
Disallow: /*/*/compare
|
|
Disallow: /*/*/lastcommit/
|
|
Disallow: /*/*/rss/branch/
|
|
Disallow: /*/*/atom/branch/
|
|
|
|
Disallow: /*/*/activity
|
|
Disallow: /*/*/activity_author_data
|
|
|
|
Disallow: /*/*/actions
|
|
Disallow: /*/*/projects
|
|
Disallow: /*/*/labels
|
|
Disallow: /*/*/milestones
|
|
|
|
Disallow: /*/*/find/
|
|
Disallow: /*/*/tree-list/
|
|
Disallow: /*/*/search/
|
|
Disallow: /*/-/code
|
|
|
|
Disallow: /*/*/issues/new
|
|
Disallow: /*/*/pulls/*/files
|
|
Disallow: /*/*/pulls/*/commits
|
|
|
|
Disallow: /attachments/
|
|
Disallow: /*/*/attachments/
|
|
Disallow: /*/*/issues/*/attachments/
|
|
Disallow: /*/*/pulls/*/attachments/
|
|
Disallow: /*/*/releases/attachments
|
|
Disallow: /*/*/releases/download
|
|
|
|
Disallow: /*/*/archive/
|
|
Disallow: /*.bundle$
|
|
Disallow: /*.patch$
|
|
Disallow: /*.diff$
|
|
Disallow: /*.atom$
|
|
Disallow: /*.rss$
|
|
|
|
Disallow: /*lang=*
|
|
Disallow: /*redirect_to=*
|
|
Disallow: /*tab=*
|
|
Disallow: /*q=*
|
|
Disallow: /*sort=*
|
|
Disallow: /*repo-search-archived=*
|