Opened 13 years ago

Closed 12 years ago

#40 closed task (fixed)

Need download stats tools installed on osgeo servers

Reported by: tmitchell Owned by: dreamil
Priority: major Milestone:
Component: Systems Admin Keywords: download stats
Cc: warmerdam, robertbray, jlivni

Description

As per this thread - I would like to have download stats and web access stats for osgeo servers. These stats were available on our previous set up and some projects, i.e. MapGuide depend on the metrics.

Change History (25)

comment:1 Changed 13 years ago by warmerdam

Owner: changed from fwarmerdam to warmerdam

fixed "assigned to".

comment:2 Changed 13 years ago by warmerdam

Cc: dreamil added
Status: newassigned

I've asked the guys from Indictrans if they could take care of installing this on download.osgeo.org.

comment:3 Changed 13 years ago by dreamil

Owner: changed from warmerdam to kanhaiya
Status: assignednew

Kanhaiya has set up a stats page using some module in drupal. Assigning the ticket to him so that he can update the status along with URL for the new stats page.

comment:4 Changed 13 years ago by anonymous

Cc: warmerdam added

comment:5 Changed 13 years ago by kanhaiya

Owner: changed from kanhaiya to anonymous
Status: newassigned

on request of me, Frank installed Graph stats on osgeo.org (in modules directory). For accessing the graphs for statistics of access log, go to this URL http://www.osgeo.org/admin/logs/graphs Let me know whether this is sufficient or want exactly like a awstats ? For awstats, i want access on osgeo.org server.

comment:6 Changed 13 years ago by kanhaiya

Actually there is also another module called "Firestats". Demo link for this module is given below; http://demo.firestats.cc/firestats/ Check this and let me know whether this module fulfill our expectations or not.

comment:7 Changed 13 years ago by anonymous

Cc: robertbray added

comment:8 Changed 13 years ago by anonymous

I'd suggest sticking with awstats instead of a drupal-based module, since then we can use 1 tool for all our servers.

comment:9 Changed 13 years ago by Omry

FireStats is not Drupal based. in fact it also works in standalone mode, and supports (as of now) 7 platforms.

comment:10 Changed 13 years ago by dreamil

I installed awstats on download.osgeo.org server. It can be seen using http://download.osgeo.org/awstats/cgi-bin/awstats.pl?config=www.download.osgeo.org Yet to fix the image/icon paths.

I would like to know:

1) on which URL we would like to host this? should it be logs.osgeo.org or download.osgeo.org/logs or something else? Depending on that we can configure apache alias/vhost. 2) how frequently should we update the stats? daily/hourly/something else?

comment:11 Changed 13 years ago by warmerdam

Cc: dreamil removed
Owner: changed from anonymous to dreamil
Status: assignednew

Swapnil,

I don't see how we can have a logs.osgeo.org domain given that our logs are spread over several machines. I think this log stats service should just be something like download.osgeo.org/logs.

I would think running daily would be fine. If possible, can you incorporate some of the older log files in the statistics?

comment:12 Changed 13 years ago by dreamil

Status: newassigned

Frank,

we could have links to various access logs of different OSGeo servers provided on logs.osgeo.org and for every individual server, $SERVERNAME/logs as the statistics page for that server. what do you say?

comment:13 Changed 13 years ago by warmerdam

Swapnil,

In my opinion it would be sufficient to have a www.osge.org/logs or /stats page pointing to the various statistics pages for different servers. I don't really like the idea of setting up a new virtual subdomain just for logs.

comment:14 Changed 13 years ago by dreamil

Hi

I have updated the stats and included the old stats also. Please check http://download.osgeo.org/logs/ Please check and verify. I have not yet added cron. If this looks ok, i will run cron to update this daily. Somebody help me in Apache RewriteRule? so that the URL remains same after redirect (If it is possible at all)

bye

comment:15 Changed 13 years ago by warmerdam

Swapnil,

The logs look fine. Please go ahead and setup a cron job.

What did you want the redirect for? Just to avoid having the cgi url show up? I don't think this is a pressing issue myself. I say we leave it as is.

Please update http://wiki.osgeo.org/index.php/Download_Server with the details about the stats generation. Once that is done, I think you can close this ticket.

Thanks!

comment:16 Changed 13 years ago by dreamil

Resolution: fixed
Status: assignedclosed
  • Added Cronjob for updating stats daily 1am (/osgeo/scripts/update_logs.sh)

Closing the ticket. Check out http://download.osgeo.org/logs/ for updated logs!

comment:17 Changed 13 years ago by warmerdam

Resolution: fixed
Status: closedreopened

Swapnil,

I notice that /demo/mapbuilder... is getting included in the statistics. I think we want to exclude that if possible. Looking around it seems that Matt Diez's community mapbuilder continuuim builds described at:

http://wiki.osgeo.org/index.php/Community_Mapbuilder_Continuum_Builds

Are using the downloads logs for the nightly.communitymapbuilder.org virtual domain. Could you fix that up to have it's own logs in the /etc/httpd/conf/httpd.conf and notify Matt Diez?

comment:18 Changed 13 years ago by dreamil

Frank, I have spearated access/error logs of mapbuilder to /etc/httpd/logs/mapbuilder_*_log

Do you also want to remove log of redirects of /geoserver and others from the download logs? If yes, I will have to see how to skip ProxyPass? redirects. Otherwise, it is done.

How do we inform Matt about this? I dont have his email id. Otherwise tell me his nick on irc.

comment:19 Changed 13 years ago by warmerdam

Swapnil,

A good search on mapbuilder and matt diez gives me:

Matthew.Diez@…

I don't really know the story on the /geoserver stuff. Is that related to the mapbuilder continuium stuff? Perhaps you could ask Matt, and get the wiki page for that updated appropriately. It certainly doens't seem like it belongs in the download logs.

I would like the download.osgeo.org logs to reflect what is actually downloaded as physical files from /osgeo/download.

comment:20 Changed 12 years ago by warmerdam

Swapnil,

could we get this polished off so this ticket can be closed?

BTW, I have split up the configuration for the various sites on this system. Now they live in /etc/httpd/conf.d/sites.

comment:21 Changed 12 years ago by warmerdam

Bump ... I'd like to get this finished off so we can close the ticket.

comment:22 Changed 12 years ago by tmitchell

Looks like this only addresses the need for the download server stats, right? How about www.osgeo.org stats itself? Using the drupal modules for statistics is not, in my opinion, as useful as awstats.

I would like to get awstats up and running on our primary server and volunteer to take it on if no one else is already working on it.

comment:23 Changed 12 years ago by jlivni

Cc: jlivni added

I've volunteered to make a slightly different awstats config file to allow for a summarized view of various mapguide downloads. Is it possible I can get some access to the server to get a copy of the current config file, and make a modified copy?

Thx,

-Josh

comment:24 Changed 12 years ago by tmitchell

Looks like all of this is taken care of.

The last remaining need is to have a cron job set up for:

sudo perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=osgeo -update

Can a cron-master set this up for me?

comment:25 Changed 12 years ago by tmitchell

Resolution: fixed
Status: reopenedclosed

oops, closing, my last comment was about the stats for main www.osgeo.org logs which is another issue altogether.

For the record, I've added a crontab entry for this on osgeo1 and also modified logrotate to run awstats update before rotating logs.

Added following lines

crontab:
 0 * * * *  /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=osgeo

/etc/logrotate.d/httpd:
   prerotate
     /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=osgeo
   endscript

Note: See TracTickets for help on using tickets.