We have an open API on the Forum, and people can and do set up bots to scrape the site for various reasons. This has been causing a few performance issues recently (and in fact for a fairly long time), so we have set up a separate environment for bots to use. This is exactly the same as the regular Forum, with all the same data, just running on different servers: https://forum-bots.effectivealtruism.org/
Also for the /graphql endpoint? Anyways, moved forum.nunosempere.com to that endpoint. But I’d expect this to be a small burden to others running their own stuff.
Yes this applies to all requests including /graphql. If the user agent of the request matches a known bot we will return a redirect to the forum-bots site. Some libraries (such as python requests and fetch in javascript) automatically follow redirects so hopefully some things will magically keep working, but this is not guaranteed.
I appreciate that this is annoying, and we didn’t really want to do it. But the site was being taken down by bots (for a few minutes) almost every day a couple of weeks ago so we finally felt this was necessary.
Also for the /graphql endpoint? Anyways, moved forum.nunosempere.com to that endpoint. But I’d expect this to be a small burden to others running their own stuff.
Yes this applies to all requests including /graphql. If the user agent of the request matches a known bot we will return a redirect to the forum-bots site. Some libraries (such as python requests and fetch in javascript) automatically follow redirects so hopefully some things will magically keep working, but this is not guaranteed.
I appreciate that this is annoying, and we didn’t really want to do it. But the site was being taken down by bots (for a few minutes) almost every day a couple of weeks ago so we finally felt this was necessary.
gg