Simon Willison: TILs on cloudflare

cloudflare Adding a robots.txt using Cloudflare workers - 2021-12-21

I got an unexpected traffic spike to https://russian-ira-facebook-ads.datasettes.com/ - which runs on Cloud Run - and decided to use robots.txt to block crawlers. …