[GH-ISSUE #27] Parsing old/missed log files? #20

New issue

Closed

opened 2026-05-05 14:59:32 -06:00 by gitea-mirror · 2 comments

gitea-mirror commented

2026-05-05 14:59:32 -06:00

Owner

Originally created by @l8gravely on GitHub (May 11, 2015).
Original GitHub issue: https://github.com/darold/sendmailanalyzer/issues/27

I've been starting to use sendmailanalyzer to watch a pair of DMZ relays and it's working well. But I messed up my logrotate and I think I missed a bunch of log files. What's the best way to go back and (re)parse my log files to bring everything back upto date?

I would assume that if I have a conf file, I could just do:

/etc/init.d/sendmailanalyzer stop
for log in ls -tr /var/log/dmz-relays*
do
/usr/local/sendmailanalyzer/sendmailanalyzer -b -f -i -l $log
done
/etc/init.d/sendmailanalyzer start

but when I did this, it didn't re-build the data, esp for the several days of data that I missed at one point. Would it be possible to either document the method to rebuild from scratch, or a way to skip checking dates? It's not a perfect solution unless you keep a DB somewhere of all parsed messages, which would get expensive.

It might be that the best way would be to go through each of the gzip'd logs cat them together into a single big file which is then sorted by date, then push it all through sendmailanalyzer again.

I'm doing the above now... with some tweaks, because I was able to blow away all my data/* directory and re-index from scratch since I only have three weeks worth of data to process..

Originally created by @l8gravely on GitHub (May 11, 2015). Original GitHub issue: https://github.com/darold/sendmailanalyzer/issues/27 I've been starting to use sendmailanalyzer to watch a pair of DMZ relays and it's working well. But I messed up my logrotate and I think I missed a bunch of log files. What's the best way to go back and (re)parse my log files to bring everything back upto date? I would assume that if I have a conf file, I could just do: /etc/init.d/sendmailanalyzer stop for log in `ls -tr /var/log/dmz-relays*` do /usr/local/sendmailanalyzer/sendmailanalyzer -b -f -i -l $log done /etc/init.d/sendmailanalyzer start but when I did this, it didn't re-build the data, esp for the several days of data that I missed at one point. Would it be possible to either document the method to rebuild from scratch, or a way to skip checking dates? It's not a perfect solution unless you keep a DB somewhere of all parsed messages, which would get expensive. It might be that the best way would be to go through each of the gzip'd logs cat them together into a single big file which is then sorted by date, then push it all through sendmailanalyzer again. I'm doing the above now... with some tweaks, because I was able to blow away all my data/\* directory and re-index from scratch since I only have three weeks worth of data to process..

gitea-mirror closed this issue

2026-05-05 14:59:36 -06:00

gitea-mirror commented

2026-05-05 14:59:40 -06:00

Author

Owner

@darold commented on GitHub (May 11, 2015):

Hi,

If you want to restart from scratch, the best way is to proceed as follow:

/etc/init.d/sendmailanalyzer stop
rm -rf /usr/local/sendmailanalyzer/data/*
for log in ls -tr /var/log/dmz-relays*
do
    /usr/local/sendmailanalyzer/sendmailanalyzer -b -f -i -l $log
done
/etc/init.d/sendmailanalyzer start

If you want to keep old data but just want to rewind for some days, you have to stop sendmailanalyzer then remove all data directories corresponding to these days and the data directories of the corresponding weeks and remove files cache.pm in the month and year directories.

Before reparsing all wanted log files, you need to remove file /usr/local/sendmailanalyzer/data/LAST_PARSED . And then:

for log in ls -tr /var/log/dmz-relays*
do
    /usr/local/sendmailanalyzer/sendmailanalyzer -b -f -i -l $log
done
/etc/init.d/sendmailanalyzer start

Data directory is build as follow for example:

/usr/local/sendmailanalyzer/data/
├── LAST_PARSED
└── smtp-gw-hostname
    └── #year
        ├── #month
        │   ├── #day
         ...
        │   └── cache.pm
        ├── cache.pm
        └── weeks
            ├── #week
             ...

Let me know if that's not clear enough.

Best regards

@darold commented on GitHub (May 11, 2015): Hi, If you want to restart from scratch, the best way is to proceed as follow: ``` /etc/init.d/sendmailanalyzer stop rm -rf /usr/local/sendmailanalyzer/data/* for log in ls -tr /var/log/dmz-relays* do /usr/local/sendmailanalyzer/sendmailanalyzer -b -f -i -l $log done /etc/init.d/sendmailanalyzer start ``` If you want to keep old data but just want to rewind for some days, you have to stop sendmailanalyzer then remove all data directories corresponding to these days and the data directories of the corresponding weeks and remove files cache.pm in the month and year directories. Before reparsing all wanted log files, you need to remove file /usr/local/sendmailanalyzer/data/LAST_PARSED . And then: ``` for log in ls -tr /var/log/dmz-relays* do /usr/local/sendmailanalyzer/sendmailanalyzer -b -f -i -l $log done /etc/init.d/sendmailanalyzer start ``` Data directory is build as follow for example: ``` /usr/local/sendmailanalyzer/data/ ├── LAST_PARSED └── smtp-gw-hostname └── #year ├── #month │ ├── #day ... │ └── cache.pm ├── cache.pm └── weeks ├── #week ... ``` Let me know if that's not clear enough. Best regards

gitea-mirror commented

2026-05-05 14:59:42 -06:00

Author

Owner

@l8gravely commented on GitHub (May 11, 2015):

Worked like a charm... Thanks for confirming that I was on the right track. Might be worth puting into the docs, esp since it seems like filter so you can reload just specific date(s) would be useful.

@l8gravely commented on GitHub (May 11, 2015): Worked like a charm... Thanks for confirming that I was on the right track. Might be worth puting into the docs, esp since it seems like filter so you can reload just specific date(s) would be useful.

gitea-mirror referenced this issue

2026-05-05 15:04:10 -06:00

[PR #20] [MERGED] Some documentation fixes #78