Add an optional robots.txt

Fix for https://gitlab.com/arkindex/requests/-/issues/284

• use a django page (no need to integrate with drf), set on /robots.txt
• Content-type should be text/plain
• this behaviour should be set by a new setting robots_txt_disallow: ["/aaa", "/xxx"]
• the payload generated would then be:

User-agent: *
Disallow: /aaa
Disallow: /xxx

By default this feature is disabled