After spending some time troubleshooting this morning, I finally came across this gem hidden in the mail log: Clamav was having issues scanning emails, causing them to reject and delay.
This is what I found:
Mar 4 10:13:14 mail amavis[2280]: (02280-01) (!)connect to /var/run/clamav/clamd.ctl failed, attempt #1: Can't connect to a UNIX socket /var/run/clamav/clamd.ctl: No such file or directory Mar 4 10:13:15 mail amavis[2280]: (02280-01) (!)connect to /var/run/clamav/clamd.ctl failed, attempt #1: Can't connect to a UNIX socket /var/run/clamav/clamd.ctl: No such file or directory Mar 4 10:13:15 mail amavis[2280]: (02280-01) (!)ClamAV-clamd: All attempts (1) failed connecting to /var/run/clamav/clamd.ctl, retrying (2) Mar 4 10:13:21 mail amavis[2280]: (02280-01) (!)connect to /var/run/clamav/clamd.ctl failed, attempt #1: Can't connect to a UNIX socket /var/run/clamav/clamd.ctl: No such file or directory Mar 4 10:13:21 mail amavis[2280]: (02280-01) (!)ClamAV-clamd av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamav/clamd.ctl (All attempts (1) failed connecting to /var/run/clamav/clamd.ctl) at (eval 114) line 613.\n Mar 4 10:13:21 mail amavis[2280]: (02280-01) (!)WARN: all primary virus scanners failed, considering backups
After some searching online and poking around, I found that ClamAV wasn’t running:
ps -ef|grep clam clamav 1067 1 0 10:09 ? 00:00:00 /usr/bin/freshclam -d --foreground=true root 9656 2284 0 10:13 pts/0 00:00:00 grep --color=auto clam
Note it’s only freshclam (the update service). I went through and the logs for ClamAV were completely empty. This wasn’t going to be an easy find. Strace-ing and other logs, I finally just tried checking the status of the service:
Mar 04 10:15:45 mail systemd[1]: Starting Clam AntiVirus userspace daemon... Mar 04 10:15:45 mail mkdir[11848]: /bin/mkdir: cannot create directory ‘/run/clamav’: File exists Mar 04 10:15:45 mail systemd[1]: Started Clam AntiVirus userspace daemon. Mar 04 10:15:45 mail clamd[11862]: ERROR: Parse error at line 90: Unknown option AllowSupplementaryGroups Mar 04 10:15:45 mail clamd[11862]: ERROR: Can't open/parse the config file /etc/clamav/clamd.conf Mar 04 10:15:45 mail systemd[1]: clamav-daemon.service: Main process exited, code=exited, status=1/FAILURE Mar 04 10:15:45 mail systemd[1]: clamav-daemon.service: Unit entered failed state. Mar 04 10:15:45 mail systemd[1]: clamav-daemon.service: Failed with result 'exit-code'.
Coming full circle: I had added that config option per online troubleshooting. I removed then installed the ClamAV-daemon process again and removed the bad line and now processing is near instant on inbound emails (versus the 60+ second delay before). I’m just happy it all works. Cheers!