Keeping noindex, the sitemap and the image sitemap in agreement for programmatic pages

A developer maintaining LandscapeCraft, a Next.js App Router site with 40 programmatically generated design-style pages, documented how a single getStyleContent predicate keeps noindex tags, sitemap.xml and image-sitemap.xml in agreement for thin pages lacking long-form MDX content. Testing on an archived copy built with Next.js 16.3.4 on Node 22.12.0 reproduced the live site's 249 sitemap URLs and 199 image-sitemap entries, and commit history showed the noindex branch had never actually fired because every style shipped with a write-up.

Google's documentation for sitemaps says to "include the URLs in your sitemap that you want to see in Google's search results". Its documentation for the robots meta tag says the noindex rule means: "Do not show this page, media, or resource in search results." A URL that carries both is giving two answers, and when the pages are generated from one array it is easy for them to drift apart without anyone noticing. LandscapeCraft landscapecraft.ai is a Next.js App Router site with 40 design-style pages generated from one TypeScript array. A style can have a long MDX write-up in content/styles/ . A style without one falls back to a short generic template, and the site treats that page as thin: it renders noindex, follow , and its URL stays out of both sitemap.xml and image-sitemap.xml . This post is about those three places: how the code keeps them in step, how I checked that it does, and where it can still drift. The code is from the repository as of 2026-11-02, and the live numbers are from the same day. Code comments below are translated from Chinese. The helper is small: js export function getStyleContent slug: string : StyleContent | null { const filePath = path.join STYLES CONTENT DIR, ${slug}.mdx ; if fs.existsSync filePath return null; const raw = fs.readFileSync filePath, "utf-8" ; const { content } = matter raw ; return { slug, content }; } The style page asks it for its metadata: // app/styles/ slug /page.tsx, inside generateMetadata // Style pages without long-form content they fall back to the generic template stay out of the index noIndex: getStyleContent slug , buildMetadata turns that flag into metadata.robots = { index: false, follow: true } , which renders as <meta name="robots" content="noindex, follow" . sitemap.ts asks the same question: // app/sitemap.ts // Style pages without a long-form MDX are rendered as noindex see app/styles/ slug /page.tsx , // so they must not appear in the sitemap. const stylePages = ALL STYLES.filter s = getStyleContent s.slug .map s = { url: ${SITE URL}/styles/${s.slug} , // ... } ; and so does the image sitemap route: js // app/image-sitemap.xml/route.ts for const style of ALL STYLES { // Style pages without long-form MDX are noindex; keep them out of the image sitemap if getStyleContent style.slug continue; // ... } Nothing ties the three together except that predicate and a comment in each file. robots.txt is a fourth party: it disallows only /api/ , on purpose. Google's robots meta documentation says that if a page is disallowed in robots.txt , its indexing rules "will not be found and will therefore be ignored", so blocking a noindex page there would defeat the noindex. app/robots.ts has a comment saying the same. What does a thin page look like here? The fallback is three headings "About … Design", "Key Design Elements", "Who Is This Style For?" and four generic bullets such as "Signature materials and textures". The 40 MDX write-ups run from 765 to 2,265 words median 1,104 , counted after the front matter. Before trusting the guard I wanted to see it do something, so I looked at the history first. The commit that added all three filters 2026-10-09 also added the last 12 MDX files: its parent had 33 styles and 21 write-ups, the commit itself 33 and 33. I checked each of the ten commits from that one to today, and none of them has a style without an MDX file. On 2026-11-02, 7 new styles arrived together with their 7 write-ups, making it 40 and 40. As far as the history shows, the noindex branch of the style page has never had a page to act on. So I tested it on a copy. I exported the committed files with git archive , built them with Next.js 16.3.4 on Node 22.12.0 macOS and ran next start . The unmodified copy matched the live site: 249 URLs in sitemap.xml and 199 <url entries in the image sitemap, 40 of each being style pages. Then I deleted one file, content/styles/wabi-sabi-interior.mdx , rebuilt and measured again: | | all 40 write-ups | wabi-sabi-interior.mdx deleted | |---|---|---| | /styles/wabi-sabi-interior | index, follow live site | HTTP 200, noindex, follow , generic template | | URLs in sitemap.xml | 249 40 style pages | 248 39 | | <url entries in image-sitemap.xml | 199 40 | 198 39 | The three move together because one predicate drives them. The page still answers 200 and keeps a canonical pointing at itself: noindex hides a page, it does not remove it. A second place uses the same idea and does fire right now. A blog category page is indexable only with at least 3 posts MIN POSTS FOR INDEX . The site has 39 posts in 4 of its 6 categories: 17, 13, 5 and 4. On 2026-11-02 the two empty categories, diy-projects and trends , return 200 with noindex, follow and are in neither sitemap. The other four are index, follow and in sitemap.xml . The image sitemap does not list category pages at all, so it agrees by leaving them out. This rule is written in two shapes. The sitemap asks getIndexableCategories , which keeps categories with = MIN POSTS FOR INDEX posts. The page asks getPostsByCategory category .length < MIN POSTS FOR INDEX . It is the same constant with opposite operators, in different files. Three things I found, with what I ran for each. 1. The filter covers one page type. The site also has 69 /ideas/<category /<style pages. In the copy with wabi-sabi-interior.mdx deleted, its idea page /ideas/bedroom/wabi-sabi-interior stays index, follow and stays in both sitemaps, and it links to the noindex style page twice the /styles index links to it once . I did not judge whether those idea pages are thin; the point is only that the rule is per page type, not per style. The idea pages also show a smaller asymmetry. sitemap.ts lists every entry of each category's recommendedStyleSlugs . The image sitemap skips an entry whose style does not exist, and the page itself calls notFound for one. Reading the code, a typo in a slug would put a URL that returns 404 into sitemap.xml and nowhere else. I did not try it. Today both files list the same 69 URLs. 2. The files are decided at different times. The build output marks /sitemap.xml as static ○ and /image-sitemap.xml as dynamic ƒ . That matches the Next.js docs: Route Handlers are not cached by default https://nextjs.org/docs/app/getting-started/route-handlers , while sitemap.ts stays static unless it uses request-time APIs. So the style pages' robots tag and sitemap.xml are fixed at build time, while the image sitemap is built on each uncached request by reading content/ from the disk of the running server. The route sets Cache-Control: public, max-age=3600, s-maxage=3600 . To see what that means, I started the unmodified build with next start and renamed content/ while it was running: | | content/ present | content/ renamed | |---|---|---| | sitemap.xml | 249 URLs, 40 style pages | 249, 40 | | /styles/japandi-interior robots | index, follow | index, follow | | image-sitemap.xml | 199 entries, 40 style pages | 120, 0 | Renaming it back restored 199. The 79 missing entries are the 40 style pages and the 39 blog posts, which are also read from content/ at request time. This is an artificial failure, not one I have seen in production. On the live site the image sitemap lists all 40 style pages, so the production runtime can see the folder right now. I did not look at how the site is deployed. 3. Nothing checks it. package.json has dev , build , start and lint , and the repository has no test files. The agreement rests on three call sites and some comments. Google's image sitemap documentation https://developers.google.com/search/docs/crawling-indexing/sitemaps/image-sitemaps requires only <image:image and <image:loc , and allows up to 1,000 images per <url . It also says caption , geo location , title and license were deprecated in 2022 https://developers.google.com/search/blog/2022/05/spring-cleaning-sitemap-extensions : they have no effect on indexing and search features after 2022-08-06, though you can leave them in place "without drawbacks". This sitemap has 298 images, each with an image:title and an image:caption . That is 596 elements, 67,247 of 138,920 bytes 48.4% . The largest <url is /gallery with 50 images, far below 1,000. Nothing is broken, but half the file is text Google no longer reads. I haven't made any of these changes. isStyleIndexable slug used by the metadata, sitemap.ts and the image sitemap. The category rule would get the same treatment. sitemap.xml , and no URL in either may be noindex . On 2026-11-02 the first holds 199 of 199 image sitemap URLs are in lastmod , and the second held for the nine pages I fetched. I did not fetch all 249. export const dynamic = 'force-static' is the documented switch for caching a GET handler . If you generate noindex , a sitemap and an image sitemap from the same data, how do you test that they still agree? I would like to hear what has worked. The pages this post is about are the style pages, for example the Japandi Interior page https://www.landscapecraft.ai/styles/japandi-interior?utm source=devto&utm medium=social&utm campaign=article . This article was drafted with help from an AI assistant and edited by me. The code excerpts come from the project's repository, and the measurements are from my own runs on 2026-11-02.