This website requires JavaScript.
Explore
Help
Sign In
lasers
/
nitter
mirror of
https://github.com/zedeus/nitter
Watch
1
Star
0
Fork
You've already forked nitter
0
Code
Issues
Projects
Releases
Wiki
Activity
6c4c691fe7
nitter
/
public
/
robots.txt
6 lines
74 B
Plaintext
Raw
Normal View
History
Unescape
Escape
Block search engines via robots.txt (#631) Prevents instances from being rate limited due to being senselessly crawled by search engines. Since there is no reason to index Nitter instances, simply block all robots. Notably, this does *not* affect link previews (e.g. in various chat software).
2022-06-04 17:48:25 +02:00
User-agent: *
Disallow: /
Add Crawl-delay to robots.txt
2023-01-09 00:12:07 +01:00
Crawl-delay: 1
Fix Twitterbot rule in robots.txt
2022-06-04 17:55:35 +02:00
User-agent: Twitterbot
Disallow: