robots.txt unmaintained and lacks sitemaps

[jidanni]

Google is returning older versions of man pages instead of the current (v78) ones.

This is due to robots.txt not having been updated since 2015,

$ HEAD https://grass.osgeo.org/robots.txt
Last-Modified: Tue, 20 Jan 2015 11:14:30 GMT

$ GET https://grass.osgeo.org/robots.txt
Disallow: /grass61/manuals/
Disallow: /grass62/manuals/
Disallow: /grass63/manuals/
Disallow: /grass65/manuals/

And most importantly, the lack of sitemap entries.

​https://support.google.com/webmasters/answer/156184

[neteler]

I have added more entries to the existing robots.txt:

...
Disallow: /grass64/manuals/
Disallow: /grass65/manuals/
Disallow: /grass70/manuals/
Disallow: /grass71/manuals/
Disallow: /grass72/manuals/
Disallow: /grass73/manuals/
Disallow: /grass74/manuals/
Disallow: /grass75/manuals/
Disallow: /grass77/manuals/

Re: sitemap.xml: We are waiting for the initial draft of the new Web page (see ​https://grasswiki.osgeo.org/wiki/Website_migration_plan_2019). With the current oldish CMS is appears to be a "mission impossible"...

I hope that the SEO will then be way better than now.