Trouble with awstats on download server

[tmitchell]

Traditional url for stats seems dead, one example:

​http://download.osgeo.org/cgi-bin/awstats.pl?month=09&year=2007&output=main&config=mapguide.download.osgeo.org&framename=index

Checking on the server and the apache2 ScriptAlias seems good:

sites-enabled/000-default: ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/

$ ls /usr/lib/cgi-bin/ -l total 552 -rwxr-xr-x 1 root root 548960 2008-12-10 04:11 awstats.pl

Any ideas? Marking as major since awstats seems completely down to me since the rebuild of the download server.

[tmitchell]

Thanks to Alex, I copied the scriptalias parameters for cgi-bin over from 000-default to download.... httpd config file. That seems to have done the trick. Now how to backload all the past httpd logs which have been set aside somehow.

[tmitchell]

Reopening as I struggle to ingest the previous awstats cache from /osgeo/download/logs. It seems the default file naming is awstatsmmddyy.txt instead of awstats072010.www.download.osgeo.org.txt, for example from previous set up. Not sure how to change that. Help! :)

[hamish]

Hi,

minor wish: could the awstats.pl cron job be run nice'd?

In the pre-osuosl days the awstats processes would dominate the xblade13's single cpu for 30min-1 hour each day, which tended to slow everything else down.

Out of curiosity I've restarted my log_cpu.sh 5 minute instrumenting on xblade13; in the last month before the xblades went down I had stopped it and posted the following to the SAC list, but it was a few kb over the size limit so never made it & I never bothered to repost.

Hi,

fyi, before the Projects VM gets well underway
and with the auth delays now sorted out, I
though I'd wrap up my logging of loads on
xblade13 and 14.

over the period xblade13 had an average load
  of 1%, and maxed out the CPU 1.5% of the
  time.  (5 minute averages, 5 minute sampling
  period)

over the period xblade14 had an average load
  of 52%, and maxed out the CPU 10.1% of the
  time.  (5 minute averages, 5 minute sampling
  period)


logging has now ceased.


regards,
Hamish

chart plots can be seen here:

​http://bambi.otago.ac.nz/hamish/osgeo/cpuload_5min.png ​http://bambi.otago.ac.nz/hamish/osgeo/xblade_cpu_histogram.png

cheers, Hamish

ps- if you have a choice, "yymmdd" instead of "ddmmyy" is nicer for sorted directory listings..

[Jeff McKenna]

bump

[Jeff McKenna]

Any ideas how we can move forward with this?

[neteler]

Did you try to run the setup (again)?

/usr/share/doc/awstats/examples/awstats_configure.pl

[wildintellect]

I haven't checked the server yet to see the current situation but here is the todo list:

1. confirm the configuration for the download.osgeo.org domain is setup correctly
2. create a cron job to run updates, it might be good to do it every hour or two to minimize the amount of time/resources, as hamish stated it should also be niced?(what level?)
3. stick a call to awstats into the logrotate script for apache to make sure it catches data before it gets rotated out. ​http://awstats.sourceforge.net/docs/awstats_faq.html#ROTATE

Now about the old data, best idea is to use a tool like mergelog to 1.combine all the old logs into 1 file, 2.then configure the awstats config to run this file 3.reconfigure awstats to point at the current file and update again

Then it would be up to date with full history. If I come across a better way to get the history implemented I'll let everyone know, but I won't have time to try this stuff for a couple of weeks.

[wildintellect]

Just an additional note, that awstats recommends an update timing of every 6 hours for a website with as much traffic as we have. This should reduce the time and load quite a bit, and I agree the cron job should be nice -n 19 ​http://awstats.sourceforge.net/docs/awstats_benchmark.html

[wildintellect]

Read over the config. Looks ok (might want to change log type 1 instead of 4, can't remember why it's better right now).

The cron job was not actually executing awstats with the proper config file. It was also set to run every 10 minutes which is a bit excessive. I've set it to run every 4 hours for now nice'd at level 19. This should be quick with a low load.

URL to access is ​http://download.osgeo.org/cgi-bin/awstats.pl?config=download.osgeo.org

I'll look into processing old logs in the next couple of weeks since I need to do this on some other servers too.

[wildintellect]

Old Logs are now restored. Changed apache logging to combined (instead of common) for more robust information. All up to date, not url is slightly different than I thought in last note. ​http://download.osgeo.org/cgi-bin/awstats.pl?config=www.download.osgeo.org

Please re-open if you notice anything isn't right.

[neteler]

I tried to see the entire year 2010 but

Error: Full year view has not been allowed from a browser (AllowFullYearView should be set to 3).

Setup ('/etc/awstats/awstats.www.download.osgeo.org.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory). 

Perhaps you could activate that?

[wildintellect]

Url is now ​http://download.osgeo.org/stats/ based on ticket:727