Skip to Content

Robots

Validate and generate robots.txt crawler rules.

@puzzle-ui/seo generates robots.txt rules and reports invalid directives before publication.

Install

npm install @puzzle-ui/seo react

React 19 is a peer dependency. Framework adapters do not require a framework package at runtime.

Quick start

import { generateRobotsTxt } from "@puzzle-ui/seo/robots";

const result = generateRobotsTxt({
  rules: [{ userAgent: "*", allow: ["/"], disallow: ["/admin/"] }],
  sitemap: "https://example.com/sitemap.xml",
});
if (result.issues.length) throw new Error(result.issues.map((issue) => issue.message).join("\n"));
const text = result.content;

Options

NameTypeDefaultDescription
rulesRobotsRule[]requiredGroups of crawler rules.
sitemapstring | string[]—Sitemap URL or URLs.
hoststring—Optional host directive.

Each RobotsRule requires userAgent as a string or string array and optionally accepts allow, disallow (arrays of paths), and numeric crawlDelay. validateRobotsOptions reports issues; serializeRobotsTxt returns text; generateRobotsTxt returns both text and issues. The server handlers turn invalid generation into a 500 response.

Robots rules are crawler instructions, not access control. Protect private resources with authentication and authorization. Page-level robots meta directives are separate; see Metadata and URLs.

Exports

@puzzle-ui/seo/robots exports validateRobotsOptions, serializeRobotsTxt, generateRobotsTxt and the RobotsDirective, RobotsRule, RobotsOptions, RobotsPolicy, RobotsResult, and RobotsValidationIssue types.