From afbb700d2bd538ee3fee479d200ba160b5c8181c Mon Sep 17 00:00:00 2001
From: minus <minus@mnus.de>
Date: Sat, 4 Jun 2022 12:03:07 +0200
Subject: [PATCH] Block search engines via robots.txt

Prevents instances from being rate limited due to being senselessly
crawled by search engines. Since there is no reason to index Nitter
instances, simply block all robots. Notably, this does *not* affect link
previews (e.g. in various chat software).
---
 public/robots.txt | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 public/robots.txt

diff --git a/public/robots.txt b/public/robots.txt
new file mode 100644
index 0000000..1f53798
--- /dev/null
+++ b/public/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /