Prevent indexing of non-stable docs versions and publish stable-only crawl policy#14800
Open
Prevent indexing of non-stable docs versions and publish stable-only crawl policy#14800
Conversation
Agent-Logs-Url: https://github.com/nextcloud/documentation/sessions/94393687-97e6-46db-ae6e-523576c970ab Co-authored-by: skjnldsv <14975046+skjnldsv@users.noreply.github.com>
github-actions
Bot
added
the
github_actions
Copilot
AI
changed the title
[WIP] Add meta robots tag to suppress indexing of non-stable builds
Prevent indexing of non-stable docs versions and publish crawl policy for stable-only SEO
May 7, 2026
skjnldsv
reviewed
May 7, 2026
| # Only the stable version should be indexed | ||
| Allow: /server/stable/ | ||
| Disallow: /server/ | ||
| Sitemap: https://docs.nextcloud.com/sitemap.xml |
Agent-Logs-Url: https://github.com/nextcloud/documentation/sessions/0af863bf-06c5-4928-81ea-59ff95c2e85a Co-authored-by: skjnldsv <14975046+skjnldsv@users.noreply.github.com>
Copilot
AI
changed the title
Prevent indexing of non-stable docs versions and publish crawl policy for stable-only SEO
Prevent indexing of non-stable docs versions and publish stable-only crawl policy
May 7, 2026
Copilot stopped work on behalf of
skjnldsv due to an error
May 7, 2026 16:50
skjnldsv
reviewed
May 7, 2026
| run: | | ||
| find "stage/${{ needs.stage-and-check.outputs.branch_name }}" -name '*.html' -print0 | while IFS= read -r -d '' f; do | ||
| perl -0pi -e 's{^\s*<link rel="canonical" href="https://docs\.nextcloud\.com/server/[^"]*" />\n}{}m' "$f" | ||
| perl -pi -e 's{<meta name="robots" content="noindex, follow" />\n?}{}g' "$f" |
Contributor
Author
There was a problem hiding this comment.
Good catch—this noindex removal in the link-check step was not actually needed for lychee (it checks links, and this meta tag does not affect link extraction). I removed it in 4b73b25 and kept only canonical stripping there. (No UI changes; workflow-only.)
Contributor
📖 Documentation PreviewNo RST documentation pages changed in this PR. Last updated: Thu, 07 May 2026 17:24:10 GMT |
Agent-Logs-Url: https://github.com/nextcloud/documentation/sessions/d832d9bc-33fb-4bda-bffa-7f69fde0afa6 Co-authored-by: skjnldsv <14975046+skjnldsv@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
☑️ Resolves
This sparked from a discussion in another ticket where users were seeing old versions docs in the first results of some search engines. Usually Google is smart enough, but sometimes it's not great.
I think we should maybe prevent indexing old docs and only let stable poke through ?
Search engines are indexing multiple version paths (
/latest,/stable,/N/), creating duplicate/outdated results when canonical tags are ignored. This change makes non-stable pages explicitly non-indexable and adds a root crawl policy that keeps onlyserver/stablecrawlable.Shared HTML meta policy
_shared_assets/templates/layout.htmlto inject:on generated Sphinx pages across manuals.
Deploy-time stable exception + robots policy
.github/workflows/sphinxbuild.yml(deploy job), after staged artifacts are applied:noindexfromserver/stable/**/*.html(stable remains indexable),robots.txt:User-agent: * Allow: /server/stable/ Disallow: /server/Link-check behavior
noindexstripping in link-check prep because it is unnecessary for lychee.🖼️ Screenshots
N/A (workflow + template-only changes; no rendered page layout/content changes)
✅ Checklist
codespellor similar and addressed any spelling issues