cleanup from improper Git merge
This commit is contained in:
parent
2495bec194
commit
2c5b05d02a
6 changed files with 129 additions and 13 deletions
30
robots.txt
30
robots.txt
|
|
@ -10,7 +10,7 @@ Disallow: /
|
|||
User-agent: Bingbot
|
||||
Disallow: /
|
||||
|
||||
# Marginalia.nu
|
||||
# Marginalia.nu can stay, I guess.
|
||||
|
||||
User-Agent: search.marginalia.nu
|
||||
Disallow:
|
||||
|
|
@ -20,7 +20,7 @@ Disallow:
|
|||
User-agent: Qwantify
|
||||
Disallow: /
|
||||
|
||||
# WayBack Machine
|
||||
# WayBack Machine (supposedly)
|
||||
|
||||
User-agent: ia_archiver
|
||||
Disallow: /
|
||||
|
|
@ -66,5 +66,31 @@ Disallow: /
|
|||
User-agent: SpiderLing
|
||||
Disallow: /
|
||||
|
||||
# And fuck your AI while I'm at it
|
||||
|
||||
User-agent: CCBot
|
||||
Disallow: /
|
||||
|
||||
User-agent: ChatGPT-User
|
||||
Disallow: /
|
||||
|
||||
User-agent: GPTBot
|
||||
Disallow: /
|
||||
|
||||
User-agent: Google-Extended
|
||||
Disallow: /
|
||||
|
||||
User-agent: anthropic-ai
|
||||
Disallow: /
|
||||
|
||||
User-agent: Omgilibot
|
||||
Disallow: /
|
||||
|
||||
User-agent: Omgili
|
||||
Disallow: /
|
||||
|
||||
User-agent: FacebookBot
|
||||
Disallow: /
|
||||
|
||||
User-agent: Bytespider
|
||||
Disallow: /
|
||||
Loading…
Add table
Add a link
Reference in a new issue