User-agent: *
Allow: /
Disallow: /dash
Disallow: /admin
Disallow: /offline
Disallow: /*/offline
Disallow: /open
Disallow: /*/open
Disallow: /share
Disallow: /*/share
Disallow: /search
Disallow: /*/search

# Everything here is public and static. There is nothing to hide from crawlers:
# no user content and no upload directory, because files never leave the
# visitor's browser.
#
# /dash and /dash/account are the first exception, and not because they are
# secret. They are a personal view (favourites and settings held in the
# visitor's own browser), so they are different for everyone and worth nothing
# in an index. Both pages also carry a noindex tag; this line saves the crawl.
#
# /admin is the operator console. It answers 404 to everybody but a signed-in
# admin (functions/admin/_middleware.js), so a crawler could never see it; this
# line just saves the request.
#
# /offline is what the service worker shows when a page is not cached and there
# is no connection. Indexing it would put "no connection" in a search result for
# a site that is up. The wildcard line covers the sixteen translated copies
# (/es/offline, /ar/offline, …), which exist so that losing your connection does
# not also lose you your language. All seventeen carry a noindex tag as well.
#
# /search is the site's own search results. Letting a crawler index those is how
# a site ends up competing with itself on thin, near-duplicate pages, and this
# site's entire traffic model is organic search, so that is the one penalty it
# cannot afford. Without a ?q= it is an empty form anyway. It carries noindex
# too, and the wildcard covers the sixteen translated copies.
#
# /open is where a file shared into the installed app lands, and /share is the
# path the share sheet POSTs to, handled entirely by the service worker, so it
# is not a page at all. Neither has anything to show a crawler: /open without a
# document is an empty state, and putting that in front of somebody searching
# for a tool would be a wasted result. /open carries noindex too.

Sitemap: https://pdfsansar.com/sitemap.xml