SEO and Crawler Hygiene

Executive Summary

Only the production custom domain should be indexable, and the EDS origin hostnames should never appear in anything a crawler can reach. This is a containment concern rather than an access control: it does not stop anyone reaching an origin, it stops search engines and casual readers discovering one.

Where Environment Access Control covers the layers that decide who may reach an environment, this page covers which URLs the outside world is told about.

Canonical Domain

Only rhea.pdcoe.dev should appear in search engine indexes. All SEO signals must point to this domain:

Signal Must Point To
Canonical tags https://rhea.pdcoe.dev/{path}
Sitemap entries https://rhea.pdcoe.dev/...
Internal links Relative paths (resolve to current domain)
Redirect targets https://rhea.pdcoe.dev/...
robots.txt Configured per site via config service
hreflang https://rhea.pdcoe.dev/...

robots.txt Per Environment

Configured via the Admin API config service per site.

Production (rhea)

User-agent: *
Allow: /
Sitemap: https://rhea.pdcoe.dev/sitemap.xml

Stage and Dev

User-agent: *
Disallow: /

What Must Be Avoided

To prevent crawlers and users from discovering .aem.live / .aem.page:

  • No browser-side fetch() calls to .aem.live or .aem.page
  • No canonical tags pointing to EDS origins
  • No sitemap entries using Adobe default hostnames
  • No redirects that expose .aem.live or .aem.page
  • No JavaScript, CSS, image, or JSON URLs referencing EDS origins
  • No preview links shared publicly
This is containment, not enforcement
None of the above is enforced by the access-control layers. A leaked origin URL stays reachable wherever access.site is not yet enabled — verified 2026-09-17: raw origins return 401 on rhea-stage but 200 on prod and dev. So these rules are the only thing keeping those hostnames out of an index today. See Environment Access Control for the lock-down rollout.

Verification

Check How
No origin hostnames in shipped code Grep the built output for aem.live / aem.page
Canonical points at the custom domain View source on a production page
Stage and dev are disallowed Fetch /robots.txt on each environment
Nothing unexpected is indexed site: search for the origin hostnames
On this page
h3
true