Robots Policy Advisor
Paste robots.txt or fetch from a URL - allow/deny explanation plus concrete crawl-policy suggestions.
Run this experiment yourself
Demos are not embedded on this site. Deploy a standalone copy on Vercel or run the experiment app locally.
Local development
cd apps/experiments/robots-policy-advisor pnpm install pnpm dev
Then open http://localhost:3010.
This is an experimental demo. Use it as a starting point for your own projects.
Robots Policy Advisor accepts pasted robots.txt or an optional fetch URL (auto-appends /robots.txt for origins), parses allow/deny locally, and explains the policy with suggestions via generateObject.
Features
- Paste or URL – fetch optional when paste empty
- Local parse hints – Allow / Disallow / Sitemap
- AI advice – allows, denies, suggestions, summary
Server Reference
POST /api/advise
{ "robotsText": "User-agent: *\nDisallow: /api/", "url": "https://example.com" }| Status | Cause |
|---|---|
503 | No AI provider |
400 | Validation / upstream / empty input |
Implementation Details
Pure helpers in logic.ts
Caps, parsers, and clamps live in the experiment folder and are covered by logic.test.ts.
Route Handler
POST /api/advise validates input, applies fetch/parse limits where relevant, then returns JSON from generateObject.
Use Cases
- Learn composition patterns for this lab’s stack.
- Prototype before wiring production auth and quotas.
- Deploy the standalone app as a reference implementation.
Limitations
- Demo-scale caps on input size and output counts.
- No auth / rate limits on public deploys.
- Model quality depends on input clarity.
- Treat outputs as drafts, not production truth.
Use in your project
Copy the Route Handler and logic.ts helpers from apps/experiments/robots-policy-advisor/. Wire your own persistence and auth as needed.
Deployment
Local Development
cd apps/experiments/robots-policy-advisor
pnpm install
pnpm devConfiguration
| Variable | Required | Purpose |
|---|---|---|
AI_GATEWAY_API_KEY | Yes (or OPENAI_API_KEY) | Preferred provider |
OPENAI_API_KEY | No | Fallback |
AI_MODEL | No | Model override |
Vercel / Next.js Features Used
- Route Handlers
- Zod
- AI SDK
generateObject