mailman archives setup to discourage search engine crawlers?

[warmerdam]

I noticed that the OSGeo mailing list archives disallow indexing with the following code bit:

     <META NAME="robots" CONTENT="noindex,follow">

at this URL: ​http://lists.osgeo.org/pipermail/gdal-dev/

Could you remove the "noindex" bit, or at least forward this on to the owner of the archives in general so they can do so?

[warmerdam]

I have reviewed the "Archiving Options" page for mailman and didn't notice any obvious way of controlling this setting. Anyone else have ideas? Does the <META> tag in question really tell search engines not to catalog our archives?

[neteler]

Yes, noindex says to not archive (please change that...).

e.g. ​http://www.robotstxt.org/meta.html

​http://www.google.com/search?q=site:mail.python.org+mailman+meta+noindex

[jbirch]

Yes, but it's OK because the individual message pages say:

     <META NAME="robots" CONTENT="index,nofollow">

Mailman has it right; we only want the engines to index the "meat"

[warmerdam]

Thanks Jason, I think I understand now. Closing...