#0 · ant · 2025-07-20 00:49:40
Many tilde.club members have placed robots.txt files in their ~/public_html/ directories, unaware that they have no effect, because crawlers consult robots.txt only at domain root[1]. Therefore, I have written a script for the consoliation of the individual users's robots.txt files into a unified server-level robots.txt.
This script is now installed and reguarly executed on the tilde.club machine, and everyone is welcome to verify whether their personal robots.txt is properly incorporated:
<http://tilde.club/robots.txt>
My latest version of the script is always available at:
<http://tilde.club/~ant/pub/botcons>
Bug requests and feature reports are welcome in the form of diff patches, English prose (or verse, for that matter) to my e-mail (ant [at] tilde.club) or in the #club channel on Newnet IRC[2].
1. <https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3>:
> The rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service.
2. <https://newnet.net/>
#1 · fenris · 2025-07-21 16:49:52
Replying to: ant
Wow, thanks a lot for the effort.
Log in to reply.