RE: [squid-users] Suggestion for log file analysis?

Here is my setup for what it's worth in deciding on your own.
I import the access.log file into a MySQL database every night and then use Crystal Reports to report on web usage. I run Squid on a Linux box but MySQL on Windows 2k Server so your setup may vary.
First I reformat the log file to my liking with a VB hack:
***snip***
                myLine = myLine.Replace(",", ";") 'remove any (,) to avoid problems later
                myFields = myLine.Split(" ")

                myDate = CDate("01-01-1970 00:00:00") 'Initialize date at Unix 0 point
                'First field should be a Unix date - turn it into a date by adding it as seconds to 1970
                intDate = CInt(myFields(0))
                myDate = myDate.AddSeconds(intDate)
                myDate = myDate.AddHours(-5) 'timezone
                myFields(0) = myDate.ToString("u")
                'Put the line back together
                newLine = Join(myFields, ",")

                'Remove duplicate (,)
                Dim replaceLine = newLine.Replace(",,", ",")
                While replaceLine <> newLine
                    newLine = replaceLine
                    replaceLine = newLine.Replace(",,", ",")
                End While

                'Split the domain and the page into separate fields
                Dim lastComma As Integer = 1
                Dim domainEnd As Integer
                For i As Int16 = 1 To 6
                    lastComma = newLine.IndexOf(",", lastComma + 1)
                Next
                domainEnd = newLine.IndexOf("/", lastComma)
                If Mid(newLine, domainEnd + 1, 1) = "/" Then
                    domainEnd = newLine.IndexOf("/", domainEnd + 2)
                End If
                If domainEnd = -1 Or newLine.IndexOf(",", lastComma + 1) < domainEnd Then
                    domainEnd = newLine.IndexOf(",", lastComma + 1)
                End If
                newLine = Left(newLine, lastComma) + Mid(newLine, lastComma + 1, domainEnd - lastComma) + "," + Mid(newLine, domainEnd + 1)
                'Use lastDateReached (stored in a separate file) to make sure we haven't already imported these fields
                If lastDateReached < myDate Then
                    'Write out the line we have edited
                    outFile.WriteLine(newLine)
                    lastDateReached = myDate
                End If
***snip***
Then I import the new file into MySQL:
"C:\Program Files\MySQL\MySQL Server 4.1\bin\mysqlimport" --user=***** --fields-optionally-enclosed-by=""" --fields-terminated-by=, --lines-terminated-by="\r\n" ***database*** \\***server***\***share***\***squid***access.log

Matthew Williams

-----Original Message-----
From: Michael St. Laurent [mailto:mikes@hartwellcorp.com]
Sent: Friday, February 16, 2007 6:52 PM
To: Squid Users
Subject: [squid-users] Suggestion for log file analysis?

I'm looking for suggestions about what to use for log file analysis.

We need something that can be used to drill down into the particular
pages a particular user was accessing during a given stretch of time. A
web interface would be nice but is not a requirement.

Squidalyser looks like it's close. I can't see how to specify a
date/time range for the query though.

--
Michael St. Laurent
Hartwell Corporation