Introduction: Logs Are the Foundation of Troubleshooting
When a website returns a 502 error, when SSH login attempts spike at 3 AM, when disk space mysteriously fills up, or when a database connection starts timing out — the answer is almost always in the log files. Linux log files are the first place every experienced system administrator looks when something goes wrong, and knowing how to navigate them efficiently can mean the difference between a 5-minute fix and hours of blind guessing.
In this comprehensive guide, we will explore the Linux log file ecosystem: where every important log lives, what each log contains, how to read and search them effectively, and how to set up log rotation to prevent disk space problems. We will cover system logs, web server logs, database logs, authentication logs, and the systemd journal — with practical examples for real-world troubleshooting scenarios.
The Linux Log Directory Structure
Most Linux log files live under /var/log/. Here is a map of the most important log files and what they contain:
| Log File | What It Contains | When to Check |
|---|---|---|
/var/log/syslog | General system messages from all services | General troubleshooting, service failures |
/var/log/auth.log | Authentication events: SSH logins, sudo usage, PAM | Security audit, brute force detection |
/var/log/kern.log | Kernel messages: hardware, drivers, OOM killer | Hardware issues, memory problems |
/var/log/nginx/access.log | Every HTTP request to Nginx | Traffic analysis, 404 hunting |
/var/log/nginx/error.log | Nginx errors: config issues, upstream failures | 502/504 errors, PHP-FPM problems |
/var/log/apache2/access.log | Every HTTP request to Apache | Traffic analysis |
/var/log/apache2/error.log | Apache errors and PHP fatal errors | Application crashes, 500 errors |
/var/log/mysql/error.log | MySQL/MariaDB errors and warnings | Database connection issues, slow queries |
/var/log/mail.log | Postfix/Dovecot mail delivery logs | Email delivery problems |
/var/log/fail2ban.log | Fail2ban ban/unban actions | Intrusion detection, false positive bans |
/var/log/cron.log | Cron job execution records | Scheduled task failures |
/var/log/dpkg.log | Package installation/removal history | Post-update debugging |
Essential Log Reading Commands
tail: Watch Logs in Real Time
The tail -f command is the most used log reading command. It shows the last lines of a file and continues to display new lines as they are written — perfect for watching events unfold in real time.
$ tail -f /var/log/nginx/error.log
2026/03/17 08:15:23 [error] 1234#0: *5678 connect() failed (111: Connection refused)
while connecting to upstream, client: 192.168.1.100, server: example.com,
request: "GET /index.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/...
# Follow multiple log files simultaneously
$ tail -f /var/log/nginx/error.log /var/log/php8.3-fpm.log
# Show last 50 lines (default is 10)
$ tail -n 50 /var/log/syslog
grep: Search for Specific Patterns
$ grep ' 500 ' /var/log/nginx/access.log
# Find failed SSH login attempts
$ grep 'Failed password' /var/log/auth.log
Mar 17 03:42:15 server sshd[12345]: Failed password for invalid user admin from 185.234.xx.xx port 45678 ssh2
Mar 17 03:42:18 server sshd[12346]: Failed password for invalid user root from 185.234.xx.xx port 45679 ssh2
# Count occurrences
$ grep -c 'Failed password' /var/log/auth.log
4523
# Case-insensitive search with context
$ grep -i -B2 -A2 'error' /var/log/mysql/error.log
awk: Extract Specific Fields
$ awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
4523 185.234.219.47
2341 66.249.79.156
1829 54.36.148.77
987 192.168.1.100
# Top 10 most requested URLs
$ awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
# Top 10 response codes
$ awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
45230 200
3421 301
1234 404
89 500
45 502
The systemd Journal
Modern Linux distributions use systemd, which maintains its own binary journal alongside traditional text log files. The journalctl command provides powerful filtering capabilities:
$ journalctl -u nginx
$ journalctl -u php8.3-fpm
$ journalctl -u mysql
# Follow in real time (like tail -f)
$ journalctl -u nginx -f
# Show logs since a specific time
$ journalctl --since "2026-03-17 08:00:00"
$ journalctl --since "1 hour ago"
$ journalctl --since today
# Show only errors and above
$ journalctl -p err
# Show boot messages (kernel/hardware)
$ journalctl -b
Real-World Troubleshooting Scenarios
Scenario 1: Debugging a 502 Bad Gateway Error
A 502 error means Nginx received an invalid response from the upstream server (usually PHP-FPM). Here is the systematic debugging approach:
[error] connect() failed (111: Connection refused) while connecting to upstream
Mar 17 08:15:22 server php-fpm[1234]: WARNING: [pool www] child 5678 exited on signal 9 (SIGKILL) after 30.123 seconds
FATAL ERROR: Allowed memory size of 134217728 bytes exhausted
memory_limit in php.ini or optimize the application to use less memory.
Scenario 2: Detecting a Brute Force Attack
$ grep 'Failed password' /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -5
1523 185.234.219.47
892 192.241.xx.xx
456 45.33.xx.xx
234 104.236.xx.xx
123 159.89.xx.xx
# Check what usernames are being targeted
$ grep 'Failed password' /var/log/auth.log | awk '{print $9}' | sort | uniq -c | sort -rn | head -5
2341 root
892 admin
456 ubuntu
234 test
123 user
Scenario 3: Finding Why Disk Space Is Filling Up
$ du -sh /var/log/* | sort -rh | head -10
4.2G /var/log/nginx
1.8G /var/log/mail.log
800M /var/log/syslog
500M /var/log/auth.log
# Check if log rotation is working
$ ls -la /var/log/nginx/
-rw-r--r-- 1 root root 4.2G Mar 17 08:30 access.log
-rw-r--r-- 1 root root 12M Mar 10 00:00 access.log.1
Log Rotation with logrotate
Without log rotation, log files grow indefinitely and can fill your entire disk. The logrotate utility automatically rotates, compresses, and removes old log files on a schedule.
/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
postrotate
[ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
endscript
}
logrotate Directives Explained
| Directive | Meaning |
|---|---|
| daily | Rotate logs once per day (also: weekly, monthly) |
| rotate 14 | Keep 14 old log files before deleting |
| compress | Compress rotated logs with gzip (.gz) |
| delaycompress | Do not compress the most recent rotated file (in case processes still write to it) |
| missingok | Do not error if the log file does not exist |
| notifempty | Do not rotate empty log files |
| postrotate | Commands to run after rotation (e.g., signal the service to reopen log files) |
$ logrotate -d /etc/logrotate.d/nginx
# Force immediate rotation
$ logrotate -f /etc/logrotate.d/nginx
Per-Service Log Deep Dive
Nginx Access Log Format
Default combined format: IP - user [date] "method URI protocol" status bytes "referer" "user-agent"
192.168.1.100 - - [17/Mar/2026:08:30:15 +0000] "GET /page HTTP/2.0" 200 4523 "https://example.com/" "Mozilla/5.0..."
PHP-FPM Slow Log
Records PHP scripts that take longer than a configured threshold. Invaluable for finding performance bottlenecks.
Configure with: request_slowlog_timeout = 5s and slowlog = /var/log/php-fpm-slow.log in the pool config.
MySQL Error Log
Contains startup/shutdown messages, warnings about deprecated features, and critical errors like crashed tables or replication failures.
Location: /var/log/mysql/error.log or check SHOW VARIABLES LIKE 'log_error';
Mail Log
Tracks every email sent, received, bounced, or deferred. Each email gets a unique queue ID that you can trace through the entire delivery process.
grep 'QUEUE_ID' /var/log/mail.log to follow one email's journey.
Security Log Monitoring
The /var/log/auth.log file is critical for security monitoring. Here are the most important patterns to watch for:
| Pattern | Indicates | Action |
|---|---|---|
Failed password for root | SSH brute force attack | Verify fail2ban is active, consider key-only auth |
Failed password for invalid user | Dictionary attack on usernames | Fail2ban will auto-ban; review banned IPs |
Accepted publickey | Successful SSH key login | Normal — verify it is from known IPs |
session opened for user root by | Someone used sudo to become root | Audit who ran sudo and when |
COMMAND=/bin/rm | Sudo command executed | Review sudo command history for policy violations |
How Panelica Handles Logs
- Real-time streaming via WebSocket — equivalent to
tail -fin the browser - Per-domain logs — each domain has its own access.log and error.log
- Centralized log management — all service logs accessible from one interface
- No SSH required — routine log analysis from the web panel
- Automatic log rotation — configured for all services during installation
Conclusion
Linux log files are your most powerful troubleshooting tool. Knowing where each log lives, what it contains, and how to search it efficiently transforms you from a guessing administrator into a systematic debugger. Remember the golden rule: when something breaks, check the log first. Use tail -f for real-time monitoring, grep for pattern searching, awk for data extraction, and journalctl for systemd services. Set up proper log rotation to prevent disk space issues, and monitor authentication logs for security threats. With these skills, you can diagnose and resolve most server issues in minutes rather than hours.