{"slug": "keeping-noindex-the-sitemap-and-the-image-sitemap-in-agreement-for-programmatic", "title": "Keeping noindex, the sitemap and the image sitemap in agreement for programmatic pages", "summary": "A developer maintaining LandscapeCraft, a Next.js App Router site with 40 programmatically generated design-style pages, documented how a single getStyleContent predicate keeps noindex tags, sitemap.xml and image-sitemap.xml in agreement for thin pages lacking long-form MDX content. Testing on an archived copy built with Next.js 16.3.4 on Node 22.12.0 reproduced the live site's 249 sitemap URLs and 199 image-sitemap entries, and commit history showed the noindex branch had never actually fired because every style shipped with a write-up.", "body_md": "Google's documentation for sitemaps says to \"include the URLs in your sitemap that you want to see in Google's search results\". Its documentation for the robots meta tag says the `noindex` rule means: \"Do not show this page, media, or resource in search results.\" A URL that carries both is giving two answers, and when the pages are generated from one array it is easy for them to drift apart without anyone noticing.\n\nLandscapeCraft (landscapecraft.ai) is a Next.js App Router site with 40 design-style pages generated from one TypeScript array. A style can have a long MDX write-up in `content/styles/`. A style without one falls back to a short generic template, and the site treats that page as thin: it renders `noindex, follow`, and its URL stays out of both `sitemap.xml` and `image-sitemap.xml`.\n\nThis post is about those three places: how the code keeps them in step, how I checked that it does, and where it can still drift. The code is from the repository as of 2026-11-02, and the live numbers are from the same day. Code comments below are translated from Chinese.\n\nThe helper is small:\n\n``` js\nexport function getStyleContent(slug: string): StyleContent | null {\n  const filePath = path.join(STYLES_CONTENT_DIR, `${slug}.mdx`);\n  if (!fs.existsSync(filePath)) return null;\n  const raw = fs.readFileSync(filePath, \"utf-8\");\n  const { content } = matter(raw);\n  return { slug, content };\n}\n```\n\nThe style page asks it for its metadata:\n\n```\n// app/styles/[slug]/page.tsx, inside generateMetadata()\n// Style pages without long-form content (they fall back to the generic template) stay out of the index\nnoIndex: !getStyleContent(slug),\n```\n\n`buildMetadata` turns that flag into `metadata.robots = { index: false, follow: true }`, which renders as `<meta name=\"robots\" content=\"noindex, follow\">`. `sitemap.ts` asks the same question:\n\n```\n// app/sitemap.ts\n// Style pages without a long-form MDX are rendered as noindex (see app/styles/[slug]/page.tsx),\n// so they must not appear in the sitemap.\nconst stylePages = ALL_STYLES.filter((s) => !!getStyleContent(s.slug)).map((s) => ({\n  url: `${SITE_URL}/styles/${s.slug}`,\n  // ...\n}));\n```\n\nand so does the image sitemap route:\n\n``` js\n// app/image-sitemap.xml/route.ts\nfor (const style of ALL_STYLES) {\n  // Style pages without long-form MDX are noindex; keep them out of the image sitemap\n  if (!getStyleContent(style.slug)) continue;\n  // ...\n}\n```\n\nNothing ties the three together except that predicate and a comment in each file. `robots.txt` is a fourth party: it disallows only `/api/`, on purpose. Google's robots meta documentation says that if a page is disallowed in `robots.txt`, its indexing rules \"will not be found and will therefore be ignored\", so blocking a noindex page there would defeat the noindex. `app/robots.ts` has a comment saying the same.\n\nWhat does a thin page look like here? The fallback is three headings (\"About … Design\", \"Key Design Elements\", \"Who Is This Style For?\") and four generic bullets such as \"Signature materials and textures\". The 40 MDX write-ups run from 765 to 2,265 words (median 1,104), counted after the front matter.\n\nBefore trusting the guard I wanted to see it do something, so I looked at the history first. The commit that added all three filters (2026-10-09) also added the last 12 MDX files: its parent had 33 styles and 21 write-ups, the commit itself 33 and 33. I checked each of the ten commits from that one to today, and none of them has a style without an MDX file. On 2026-11-02, 7 new styles arrived together with their 7 write-ups, making it 40 and 40. As far as the history shows, the `noindex` branch of the style page has never had a page to act on.\n\nSo I tested it on a copy. I exported the committed files with `git archive`, built them with Next.js 16.3.4 on Node 22.12.0 (macOS) and ran `next start`. The unmodified copy matched the live site: 249 URLs in `sitemap.xml` and 199 `<url>` entries in the image sitemap, 40 of each being style pages. Then I deleted one file, `content/styles/wabi-sabi-interior.mdx`, rebuilt and measured again:\n\n|  | all 40 write-ups | `wabi-sabi-interior.mdx` deleted | \n|---|---|---|\n| `/styles/wabi-sabi-interior` | `index, follow` (live site) | HTTP 200, `noindex, follow` , generic template | \n| URLs in `sitemap.xml` | 249 (40 style pages) | 248 (39) | \n| `<url>` entries in`image-sitemap.xml` | 199 (40) | 198 (39) | \n\nThe three move together because one predicate drives them. The page still answers 200 and keeps a canonical pointing at itself: `noindex` hides a page, it does not remove it.\n\nA second place uses the same idea and does fire right now. A blog category page is indexable only with at least 3 posts (`MIN_POSTS_FOR_INDEX`). The site has 39 posts in 4 of its 6 categories: 17, 13, 5 and 4. On 2026-11-02 the two empty categories, `diy-projects` and `trends`, return 200 with `noindex, follow` and are in neither sitemap. The other four are `index, follow` and in `sitemap.xml`. The image sitemap does not list category pages at all, so it agrees by leaving them out.\n\nThis rule is written in two shapes. The sitemap asks `getIndexableCategories()`, which keeps categories with `>= MIN_POSTS_FOR_INDEX` posts. The page asks `getPostsByCategory(category).length < MIN_POSTS_FOR_INDEX`. It is the same constant with opposite operators, in different files.\n\nThree things I found, with what I ran for each.\n\n**1. The filter covers one page type.** The site also has 69 `/ideas/<category>/<style>` pages. In the copy with `wabi-sabi-interior.mdx` deleted, its idea page `/ideas/bedroom/wabi-sabi-interior` stays `index, follow` and stays in both sitemaps, and it links to the noindex style page twice (the `/styles` index links to it once). I did not judge whether those idea pages are thin; the point is only that the rule is per page type, not per style.\n\nThe idea pages also show a smaller asymmetry. `sitemap.ts` lists every entry of each category's `recommendedStyleSlugs`. The image sitemap skips an entry whose style does not exist, and the page itself calls `notFound()` for one. Reading the code, a typo in a slug would put a URL that returns 404 into `sitemap.xml` and nowhere else. I did not try it. Today both files list the same 69 URLs.\n\n**2. The files are decided at different times.** The build output marks `/sitemap.xml` as static (○) and `/image-sitemap.xml` as dynamic (ƒ). That matches the Next.js docs: [Route Handlers are not cached by default](https://nextjs.org/docs/app/getting-started/route-handlers), while `sitemap.ts` stays static unless it uses request-time APIs. So the style pages' robots tag and `sitemap.xml` are fixed at build time, while the image sitemap is built on each uncached request by reading `content/` from the disk of the running server. The route sets `Cache-Control: public, max-age=3600, s-maxage=3600`.\n\nTo see what that means, I started the unmodified build with `next start` and renamed `content/` while it was running:\n\n|  | `content/` present | `content/` renamed | \n|---|---|---|\n| `sitemap.xml` | 249 URLs, 40 style pages | 249, 40 | \n| `/styles/japandi-interior` robots | `index, follow` | `index, follow` | \n| `image-sitemap.xml` | 199 entries, 40 style pages | 120, 0 | \n\nRenaming it back restored 199. The 79 missing entries are the 40 style pages and the 39 blog posts, which are also read from `content/` at request time. This is an artificial failure, not one I have seen in production. On the live site the image sitemap lists all 40 style pages, so the production runtime can see the folder right now. I did not look at how the site is deployed.\n\n**3. Nothing checks it.** `package.json` has `dev`, `build`, `start` and `lint`, and the repository has no test files. The agreement rests on three call sites and some comments.\n\nGoogle's [image sitemap documentation](https://developers.google.com/search/docs/crawling-indexing/sitemaps/image-sitemaps) requires only `<image:image>` and `<image:loc>`, and allows up to 1,000 images per `<url>`. It also says `caption`, `geo_location`, `title` and `license` were [deprecated in 2022](https://developers.google.com/search/blog/2022/05/spring-cleaning-sitemap-extensions): they have no effect on indexing and search features after 2022-08-06, though you can leave them in place \"without drawbacks\". This sitemap has 298 images, each with an `image:title` and an `image:caption`. That is 596 elements, 67,247 of 138,920 bytes (48.4%). The largest `<url>` is `/gallery` with 50 images, far below 1,000. Nothing is broken, but half the file is text Google no longer reads.\n\nI haven't made any of these changes.\n\n`isStyleIndexable(slug)` used by the metadata, `sitemap.ts` and the image sitemap. The category rule would get the same treatment.`sitemap.xml`, and no URL in either may be `noindex`. On 2026-11-02 the first holds (199 of 199 image sitemap URLs are in `lastmod`), and the second held for the nine pages I fetched. I did not fetch all 249.`export const dynamic = 'force-static'` is the documented switch for caching a GET handler).\nIf you generate `noindex`, a sitemap and an image sitemap from the same data, how do you test that they still agree? I would like to hear what has worked.\n\nThe pages this post is about are the style pages, for example the [Japandi Interior page](https://www.landscapecraft.ai/styles/japandi-interior?utm_source=devto&utm_medium=social&utm_campaign=article).\n\n*This article was drafted with help from an AI assistant and edited by me. The code excerpts come from the project's repository, and the measurements are from my own runs on 2026-11-02.*", "url": "https://wpnews.pro/news/keeping-noindex-the-sitemap-and-the-image-sitemap-in-agreement-for-programmatic", "canonical_source": "https://dev.to/picraftai/keeping-noindex-the-sitemap-and-the-image-sitemap-in-agreement-for-programmatic-pages-57o1", "published_at": "2026-11-02 14:48:00+00:00", "updated_at": "2026-11-02 14:53:53.180914+00:00", "lang": "en", "topics": ["generative-engine-optimization", "structured-data", "ai-search"], "entities": ["LandscapeCraft", "Google", "Next.js", "Node.js"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/keeping-noindex-the-sitemap-and-the-image-sitemap-in-agreement-for-programmatic", "markdown": "https://wpnews.pro/news/keeping-noindex-the-sitemap-and-the-image-sitemap-in-agreement-for-programmatic.md", "text": "https://wpnews.pro/news/keeping-noindex-the-sitemap-and-the-image-sitemap-in-agreement-for-programmatic.txt", "jsonld": "https://wpnews.pro/news/keeping-noindex-the-sitemap-and-the-image-sitemap-in-agreement-for-programmatic.jsonld"}}