Tutorial

Systemd Explained: Services, Timers, Logs, and Boot Management

April 23, 2026

Tutorial April 23, 2026

If you are running a modern Linux server, you are running systemd. It is the init system, the service manager, the logger, the timer scheduler, and much more. Yet many administrators only know the basics: systemctl start and systemctl stop. In this guide, we will go deep into systemd — from writing custom service unit files to replacing cron with timers, mastering journalctl for log analysis, and understanding boot targets. By the end, you will have the knowledge to manage any service on your server with confidence.

Why systemd matters: Before systemd, Linux distributions used SysVinit or Upstart, each with its own quirks. Systemd unified service management, logging, and scheduling under a single framework. Love it or hate it, knowing systemd is non-negotiable for modern server administration.

Understanding Unit Files

Everything in systemd is a unit. Services, timers, mount points, sockets — they are all defined by unit files. These are plain text INI-style configuration files that tell systemd what to run and how to run it.

Where Unit Files Live

Path	Purpose	Priority
`/etc/systemd/system/`	Admin-created and override units	Highest
`/run/systemd/system/`	Runtime-generated units	Medium
`/lib/systemd/system/`	Package-installed units	Lowest

When you create or modify a service, always work in /etc/systemd/system/. This ensures your changes survive package updates, which overwrite files in /lib/systemd/system/.

Anatomy of a Service Unit File

A service unit file has three main sections: [Unit], [Service], and [Install]. Let us walk through a complete, real-world example.

$ cat /etc/systemd/system/myapp.service [Unit] Description=My Go Application Server Documentation=https://docs.myapp.com After=network-online.target postgresql.service Wants=network-online.target Requires=postgresql.service [Service] Type=simple User=myapp Group=myapp WorkingDirectory=/opt/myapp ExecStartPre=/opt/myapp/bin/migrate ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=5 StandardOutput=journal StandardError=journal SyslogIdentifier=myapp LimitNOFILE=65535 Environment=GO_ENV=production EnvironmentFile=-/etc/myapp/env [Install] WantedBy=multi-user.target

Let us break down every important directive.

The [Unit] Section

This section describes the unit and defines its relationships with other units.

Directive	Purpose	Example
`Description`	Human-readable name shown in status output	`My Go Application Server`
`After`	Start this unit AFTER the listed units	`network-online.target postgresql.service`
`Before`	Start this unit BEFORE the listed units	`nginx.service`
`Requires`	Hard dependency — if the listed unit fails, this one stops too	`postgresql.service`
`Wants`	Soft dependency — try to start, but do not fail if it cannot	`network-online.target`

After vs. Requires: These are independent concepts. After=postgres controls ordering (start after postgres). Requires=postgres controls dependency (fail if postgres fails). You usually want both: After=postgresql.service AND Requires=postgresql.service.

The [Service] Section

This is where the real configuration happens. The Type directive is especially important.

Type=simple Most Common

The default. Systemd considers the service started as soon as the ExecStart process launches. Use this for applications that stay in the foreground.

Examples: Go binaries, Node.js apps, Python web servers

Type=forking

For daemons that fork a child process and then the parent exits. Systemd tracks the child via a PID file.

Examples: Apache httpd, traditional nginx, PHP-FPM

Type=oneshot

For commands that do one thing and exit. Systemd waits for the process to finish before considering the unit started.

Examples: Initialization scripts, migration runners, cleanup tasks

Type=notify

The service sends a notification to systemd when it is fully ready using sd_notify(). More precise startup tracking.

Examples: PostgreSQL, systemd-aware daemons

Other critical directives in the [Service] section include:

# Run a command before the main process starts ExecStartPre=/opt/myapp/bin/migrate # The main process ExecStart=/opt/myapp/bin/server # Graceful reload (send SIGHUP) ExecReload=/bin/kill -HUP $MAINPID # Restart policy Restart=on-failure # only restart on non-zero exit RestartSec=5 # wait 5 seconds between restarts # Security: run as unprivileged user User=myapp Group=myapp # Resource limits LimitNOFILE=65535 # max open file descriptors # Environment Environment=GO_ENV=production EnvironmentFile=-/etc/myapp/env # - means "ignore if missing"

The [Install] Section

This section controls what happens when you run systemctl enable. The most common directive is WantedBy=multi-user.target, which makes the service start at boot in normal multi-user mode.

Essential systemctl Commands

Here is every systemctl command you will use regularly, grouped by purpose.

Service Lifecycle

$ systemctl start myapp # Start the service $ systemctl stop myapp # Stop the service $ systemctl restart myapp # Stop + Start $ systemctl reload myapp # Reload config without downtime $ systemctl enable myapp # Start at boot $ systemctl disable myapp # Do not start at boot $ systemctl enable --now myapp # Enable AND start immediately

Status and Diagnostics

$ systemctl status myapp ● myapp.service - My Go Application Server Loaded: loaded (/etc/systemd/system/myapp.service; enabled) Active: active (running) since Mon 2026-03-17 08:00:01 UTC; 3h ago Main PID: 1234 (server) Tasks: 12 (limit: 4680) Memory: 128.5M CPU: 2min 34.567s CGroup: /system.slice/myapp.service └─1234 /opt/myapp/bin/server --config /etc/myapp/config.yaml $ systemctl is-active myapp active $ systemctl is-enabled myapp enabled $ systemctl list-units --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● redis.service loaded failed failed Redis Server

Advanced Operations

$ systemctl daemon-reload # Reload unit files after editing $ systemctl mask myapp # Completely prevent starting $ systemctl unmask myapp # Reverse mask $ systemctl cat myapp # Show the unit file contents $ systemctl show myapp # Show all properties $ systemctl edit myapp # Create an override snippet $ systemctl reset-failed # Clear failed state

Never forget daemon-reload: After editing any unit file in /etc/systemd/system/, you MUST run systemctl daemon-reload before starting or restarting the service. Systemd caches unit files in memory, so changes are invisible until you reload.

Timer Units: The Modern cron

Systemd timers are a powerful alternative to cron. They offer better logging, dependency management, and can trigger on events (not just time).

Creating a Timer

A timer requires two files: a .timer unit and a matching .service unit.

$ cat /etc/systemd/system/backup.timer [Unit] Description=Daily Backup Timer [Timer] OnCalendar=*-*-* 02:00:00 RandomizedDelaySec=900 Persistent=true [Install] WantedBy=timers.target $ cat /etc/systemd/system/backup.service [Unit] Description=Daily Backup Job [Service] Type=oneshot ExecStart=/opt/scripts/backup.sh User=backup StandardOutput=journal

Create the service unit that defines what to run (the backup script).

Create the timer unit that defines when to run it (daily at 2 AM with up to 15 min random delay).

Enable and start the timer (not the service): systemctl enable --now backup.timer

Timer vs. Cron Comparison

Feature	cron	systemd Timer
Logging	Must redirect to file manually	Automatic via journalctl
Dependencies	None	After=, Requires= supported
Missed runs	Lost forever	Persistent=true catches up
Random delay	Must script manually	RandomizedDelaySec= built-in
Resource limits	None	CPUQuota=, MemoryMax= supported
Setup complexity	One line in crontab	Two unit files required
Listing jobs	`crontab -l`	`systemctl list-timers`

OnCalendar Syntax

OnCalendar=*-*-* 02:00:00 # Every day at 2 AM OnCalendar=Mon *-*-* 09:00:00 # Every Monday at 9 AM OnCalendar=*-*-01 00:00:00 # First of every month OnCalendar=hourly # Shorthand for *-*-* *:00:00 OnCalendar=*-*-* *:*:00/30 # Every 30 seconds $ systemd-analyze calendar "Mon *-*-* 09:00:00" Original form: Mon *-*-* 09:00:00 Normalized form: Mon *-*-* 09:00:00 Next elapse: Mon 2026-03-23 09:00:00 UTC From now: 5 days left

The systemd-analyze calendar command is incredibly useful for testing your schedule expressions before deploying them.

Listing and Managing Timers

$ systemctl list-timers --all NEXT LEFT LAST PASSED UNIT Tue 2026-03-18 02:00:00 UTC 14h left Mon 2026-03-17 02:14:23 UTC 9h ago backup.timer Tue 2026-03-18 00:00:00 UTC 12h left Mon 2026-03-17 00:00:00 UTC 11h ago logrotate.timer Tue 2026-03-18 06:34:12 UTC 19h left Mon 2026-03-17 06:34:12 UTC 5h ago apt-daily.timer

Mastering journalctl

Systemd captures all service output (stdout and stderr) into the journal. journalctl is the tool to query it, and it is far more powerful than manually grepping through log files.

Basic Queries

$ journalctl -u nginx # All logs for nginx $ journalctl -u nginx --since "2 hours ago" $ journalctl -u nginx --since "2026-03-17" --until "2026-03-17 12:00" $ journalctl -u nginx -f # Follow (like tail -f) $ journalctl -u nginx -n 50 # Last 50 lines $ journalctl -u nginx -o json-pretty # JSON output

Filtering by Priority

$ journalctl -p err # Errors and above $ journalctl -p warning --since today $ journalctl -p crit -b # Critical since boot

Priority Level	Keyword	Description
0	`emerg`	System is unusable
1	`alert`	Immediate action required
2	`crit`	Critical conditions
3	`err`	Error conditions
4	`warning`	Warning conditions
5	`notice`	Normal but significant
6	`info`	Informational messages
7	`debug`	Debug-level messages

Journal Maintenance

$ journalctl --disk-usage Archived and active journals take up 2.4G in /var/log/journal $ journalctl --vacuum-size=500M # Shrink to 500MB $ journalctl --vacuum-time=2weeks # Remove logs older than 2 weeks

Boot Targets (Runlevels)

Systemd uses targets instead of the old SysVinit runlevels. A target is a grouping of units that defines a system state.

Target	Old Runlevel	Description
`rescue.target`	1	Single-user mode, minimal services
`multi-user.target`	3	Full multi-user, no GUI — Server default
`graphical.target`	5	Multi-user with GUI
`reboot.target`	6	System reboot
`poweroff.target`	0	System shutdown

$ systemctl get-default multi-user.target $ systemctl set-default multi-user.target $ systemctl isolate rescue.target # Switch to rescue mode NOW

Overriding Unit Files Without Editing Them

Never edit unit files in /lib/systemd/system/ directly — package updates will overwrite your changes. Instead, use drop-in overrides.

$ systemctl edit nginx # Creates /etc/systemd/system/nginx.service.d/override.conf [Service] LimitNOFILE=65535 Restart=always RestartSec=3 $ systemctl cat nginx # Shows base unit + all overrides

The override file only needs the directives you want to change. Everything else is inherited from the base unit file.

Practical Example: Creating a Complete Service

Let us put it all together by creating a service for a Node.js application that requires a database, restarts on failure, and has resource limits.

Create the unit file at /etc/systemd/system/nodeapp.service

[Unit] Description=Node.js Application After=network-online.target postgresql.service Wants=network-online.target Requires=postgresql.service [Service] Type=simple User=nodeapp Group=nodeapp WorkingDirectory=/opt/nodeapp ExecStart=/usr/bin/node /opt/nodeapp/server.js Restart=on-failure RestartSec=5 StartLimitIntervalSec=60 StartLimitBurst=3 StandardOutput=journal StandardError=journal SyslogIdentifier=nodeapp EnvironmentFile=/opt/nodeapp/.env LimitNOFILE=65535 MemoryMax=512M CPUQuota=80% [Install] WantedBy=multi-user.target

Reload, enable, and start:

$ systemctl daemon-reload $ systemctl enable --now nodeapp $ systemctl status nodeapp ● nodeapp.service - Node.js Application Active: active (running) since Mon 2026-03-17 12:00:01 UTC Main PID: 5678 (node) Memory: 87.3M (max: 512.0M) CPU: 1.234s

Verify logs: journalctl -u nodeapp -f

StartLimitIntervalSec and StartLimitBurst prevent restart loops. In our example, if the service fails 3 times within 60 seconds, systemd stops trying and marks it as failed. This protects your server from a crash loop consuming resources.

Debugging Failed Services

When a service refuses to start, follow this systematic debugging workflow:

systemctl status

→

journalctl -u

→

systemctl cat

→

ExecStart manual test

→

Fix & daemon-reload

$ systemctl status myapp # Check exit code and recent log $ journalctl -u myapp -n 30 # Read the last 30 log lines $ systemctl cat myapp # Verify the unit file $ /opt/myapp/bin/server # Run the command manually $ systemctl reset-failed myapp # Clear failed state after fix $ systemctl daemon-reload && systemctl restart myapp

Security Hardening with Systemd

Systemd provides built-in security features that isolate services without Docker or containers.

[Service] # Filesystem protection ProtectSystem=strict # Mount / as read-only ProtectHome=true # Hide /home from the service ReadWritePaths=/opt/myapp/data /var/log/myapp # Network isolation PrivateNetwork=false # Set true if no network needed # Privilege escalation protection NoNewPrivileges=true # Prevent setuid/setgid PrivateTmp=true # Isolated /tmp # Device access PrivateDevices=true # No access to physical devices DevicePolicy=closed

You can audit a service's security posture with:

$ systemd-analyze security myapp NAME DESCRIPTION EXPOSURE ✗ ProtectSystem= Service has full access... 0.1 ✓ NoNewPrivileges= Service cannot gain new... ... ✗ PrivateDevices= Service has access to... 0.1 Overall exposure level: 4.2 MEDIUM

Systemd and Panelica

How Panelica uses systemd: Panelica manages over 20 isolated services — nginx, PHP-FPM, MySQL, mail services, DNS, FTP, and more — all through systemd unit files. Its pn-service wrapper provides a simplified interface (pn-service restart nginx) while leveraging the full power of systemd under the hood. Each service has proper dependencies, resource limits, and restart policies configured, so you get enterprise-grade service management with single-command simplicity.

Summary

Unit files define services in three sections: [Unit], [Service], and [Install]
Always use /etc/systemd/system/ for custom units and overrides
Run systemctl daemon-reload after every unit file change
Use Type=simple for most modern applications
Replace cron with systemd timers for better logging and reliability
Master journalctl filters: -u, -p, --since, -f
Use security directives (ProtectSystem, NoNewPrivileges) to harden services
Follow the debug flow: status → journal → cat → manual test → fix

Systemd is a deep topic, but the knowledge in this guide covers what you will use 95% of the time. As you manage more complex server setups, you will appreciate the consistency and power that systemd brings to service management, logging, and scheduling.