4 Commits

Author	SHA1	Message	Date
Zed	6c4c691fe7	Add Crawl-delay to robots.txt	2023-01-09 00:12:07 +01:00
Zed	138826fb4f	Fix Twitterbot rule in robots.txt	2022-06-04 17:55:35 +02:00
Zed	dfb2519870	Explicitly allow Twitterbot to generate previews	2022-06-04 17:52:47 +02:00
minus	c543a1df8c	Block search engines via robots.txt (#631) ... Prevents instances from being rate limited due to being senselessly crawled by search engines. Since there is no reason to index Nitter instances, simply block all robots. Notably, this does *not* affect link previews (e.g. in various chat software).	2022-06-04 17:48:25 +02:00