Google has added a bunch of latest supporting particulars to the robots.txt specification doc. Jamie Indigo seen these modifications and defined what was added embrace “specific particulars on the information for IDNs, IP-addresses, and port numbered hostnames.”
It appears these have been added on April seventh in keeping with the doc. Jamie posted the small print on Twitter saying “Robots.txt docs are full of latest delights due to @methode and @LizziHarvey’s on-going work to make robots protocol an web commonplace. The most recent additions embrace specific particulars on the information for IDNs, IP-addresses, and port numbered hostnames.”
Google final did an enormous replace to this doc in July 2019.
Discussion board dialogue at Twitter.