Migrate docs site to App Router on Next canary

[feedthejim]

Summary

• migrate the docs site from the Pages Router to the App Router
• upgrade to next@canary, enable Cache Components, and rely on Turbopack as the default build path
• replace the old MDX pipeline, move special endpoints to route handlers, and make the Sandpack/RSC path Turbopack-safe

What changed

• add App Router entrypoints for docs pages, errors, llms.txt, rss.xml, markdown API, 404, 500, not-found, and error
• add a shared content layer with use cache page-data helpers, cache tagging, and dev invalidation wiring for markdown/sidebar edits
• replace the old compileMDX / prepareMDX path with a server-first generateMDX pipeline and dedicated MDX plugins for TOC extraction, code metadata, max-width wrapping, and stable data-mdx-name annotations
• keep the existing client shell but refactor router integrations to next/navigation / useLinkStatus, plus explicit client islands for TOC, language list, challenges, examples, terminal copy UI, analytics, and error decoder behavior
• move root document concerns into the App Router layout, including theme bootstrapping, uwu mode, scroll restoration, Algolia preconnects, and analytics hooks
• replace build-time RSS generation with a route handler and move linting to flat-config ESLint
• remove the old Pages Router files once parity was restored

Sandpack / tooling notes

• generate raw RSC sources during the prebuild step instead of relying on raw-loader
• swap the browser lint path to eslint-linter-browserify and use a browser-safe alias for eslint-plugin-react-hooks
• add a small dev websocket watcher so content edits trigger cache invalidation and router.refresh() under Turbopack

Validation

• yarn build
• yarn lint (passes; existing no-img-element warnings remain)
• yarn deadlinks
• yarn test:eslint-local-rules

[github-actions]

Size changes

Details

📦 Next.js Bundle Analysis for react-dev

This analysis was generated by the Next.js Bundle Analysis action. 🤖

🎉 Global Bundle Size Decreased

Page	Size (compressed)
global	0 B (🟢 -115.14 KB)

Details

The global bundle is the javascript bundle that loads alongside every page. It is in its own category because its impact is much higher - an increase to its size means that every page on your website loads slower, and a decrease means every page loads faster.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

If you want further insight into what is behind the changes, give @next/bundle-analyzer a try!

[gaearon]

is this hack still needed and does it still work?

iirc this was to make Back gesture show previous content

[gaearon]

do we need this caching stuff now? feels like it's just reimplementing with 'use cache' gives us

[MaxwellCohen]

Thank you for doing this upgrade!
I did some quick performance testing on the useActionState page (this pr build on my machine vs react.dev) and saw a slight performance downgrade
LCP: .34s to .48s on local vs .2s to .3s on prod
INP: 138ms vs 144ms on local vs 112ms to 128ms on prod

I am also seeing a lot more JS being loaded, 2.9mb on local vs 1.9mb on prod, while both versions use about 980kb of JS. This suggests we have some tree-shaking/bundling issues. With Turbopack, I am not sure how to fix it because the magic comments do not appear to be supported yet.

localhost JS coverage report from Chrome

Production JS coverage report from Chrome