Need download stats tools installed on osgeo servers

[tmitchell]

As per ​this thread - I would like to have download stats and web access stats for osgeo servers. These stats were available on our previous set up and some projects, i.e. MapGuide depend on the metrics.

[warmerdam]

fixed "assigned to".

[warmerdam]

I've asked the guys from Indictrans if they could take care of installing this on download.osgeo.org.

[dreamil]

Kanhaiya has set up a stats page using some module in drupal. Assigning the ticket to him so that he can update the status along with URL for the new stats page.

[kanhaiya]

on request of me, Frank installed Graph stats on osgeo.org (in modules directory). For accessing the graphs for statistics of access log, go to this URL ​http://www.osgeo.org/admin/logs/graphs Let me know whether this is sufficient or want exactly like a awstats ? For awstats, i want access on osgeo.org server.

[kanhaiya]

Actually there is also another module called "Firestats". Demo link for this module is given below; ​http://demo.firestats.cc/firestats/ Check this and let me know whether this module fulfill our expectations or not.

[anonymous]

I'd suggest sticking with awstats instead of a drupal-based module, since then we can use 1 tool for all our servers.

[Omry]

FireStats is not Drupal based. in fact it also works in standalone mode, and supports (as of now) 7 ​platforms.

[dreamil]

I installed awstats on download.osgeo.org server. It can be seen using ​http://download.osgeo.org/awstats/cgi-bin/awstats.pl?config=www.download.osgeo.org Yet to fix the image/icon paths.

I would like to know:

1) on which URL we would like to host this? should it be logs.osgeo.org or download.osgeo.org/logs or something else? Depending on that we can configure apache alias/vhost. 2) how frequently should we update the stats? daily/hourly/something else?

[warmerdam]

Swapnil,

I don't see how we can have a logs.osgeo.org domain given that our logs are spread over several machines. I think this log stats service should just be something like download.osgeo.org/logs.

I would think running daily would be fine. If possible, can you incorporate some of the older log files in the statistics?

[dreamil]

Frank,

we could have links to various access logs of different OSGeo servers provided on logs.osgeo.org and for every individual server, $SERVERNAME/logs as the statistics page for that server. what do you say?

[warmerdam]

Swapnil,

In my opinion it would be sufficient to have a www.osge.org/logs or /stats page pointing to the various statistics pages for different servers. I don't really like the idea of setting up a new virtual subdomain just for logs.

[dreamil]

Hi

I have updated the stats and included the old stats also. Please check ​http://download.osgeo.org/logs/ Please check and verify. I have not yet added cron. If this looks ok, i will run cron to update this daily. Somebody help me in Apache RewriteRule so that the URL remains same after redirect (If it is possible at all)

bye

[warmerdam]

Swapnil,

The logs look fine. Please go ahead and setup a cron job.

What did you want the redirect for? Just to avoid having the cgi url show up? I don't think this is a pressing issue myself. I say we leave it as is.

Please update ​http://wiki.osgeo.org/index.php/Download_Server with the details about the stats generation. Once that is done, I think you can close this ticket.

Thanks!

[dreamil]

• Added Cronjob for updating stats daily 1am (/osgeo/scripts/update_logs.sh)

• Updated ​http://wiki.osgeo.org/index.php/Download_Server to include details of awstats installation

Closing the ticket. Check out ​http://download.osgeo.org/logs/ for updated logs!

[warmerdam]

Swapnil,

I notice that /demo/mapbuilder... is getting included in the statistics. I think we want to exclude that if possible. Looking around it seems that Matt Diez's community mapbuilder continuuim builds described at:

​http://wiki.osgeo.org/index.php/Community_Mapbuilder_Continuum_Builds

Are using the downloads logs for the nightly.communitymapbuilder.org virtual domain. Could you fix that up to have it's own logs in the /etc/httpd/conf/httpd.conf and notify Matt Diez?

[dreamil]

Frank, I have spearated access/error logs of mapbuilder to /etc/httpd/logs/mapbuilder_*_log

Do you also want to remove log of redirects of /geoserver and others from the download logs? If yes, I will have to see how to skip ProxyPass redirects. Otherwise, it is done.

How do we inform Matt about this? I dont have his email id. Otherwise tell me his nick on irc.

[warmerdam]

Swapnil,

A good search on mapbuilder and matt diez gives me:

Matthew.Diez@…

I don't really know the story on the /geoserver stuff. Is that related to the mapbuilder continuium stuff? Perhaps you could ask Matt, and get the wiki page for that updated appropriately. It certainly doens't seem like it belongs in the download logs.

I would like the download.osgeo.org logs to reflect what is actually downloaded as physical files from /osgeo/download.

[warmerdam]

Swapnil,

could we get this polished off so this ticket can be closed?

BTW, I have split up the configuration for the various sites on this system. Now they live in /etc/httpd/conf.d/sites.

[warmerdam]

Bump ... I'd like to get this finished off so we can close the ticket.

[tmitchell]

Looks like this only addresses the need for the download server stats, right? How about www.osgeo.org stats itself? Using the drupal modules for statistics is not, in my opinion, as useful as awstats.

I would like to get awstats up and running on our primary server and volunteer to take it on if no one else is already working on it.

[jlivni]

I've volunteered to make a slightly different awstats config file to allow for a summarized view of various mapguide downloads. Is it possible I can get some access to the server to get a copy of the current config file, and make a modified copy?

Thx,

-Josh

[tmitchell]

Looks like all of this is taken care of.

The last remaining need is to have a cron job set up for:

sudo perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=osgeo -update

Can a cron-master set this up for me?

[tmitchell]

oops, closing, my last comment was about the stats for main www.osgeo.org logs which is another issue altogether.

For the record, I've added a crontab entry for this on osgeo1 and also modified logrotate to run awstats update before rotating logs.

Added following lines

crontab:
 0 * * * *  /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=osgeo

/etc/logrotate.d/httpd:
   prerotate
     /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=osgeo
   endscript