Introduction: "Disk Full" — The #1 Server Emergency
It happens at the worst possible time. A production server grinds to a halt. Nginx returns 500 errors. MySQL refuses to write. PHP throws fatal errors. You SSH in, run df -h, and see it: 100% disk usage.
Disk management is not glamorous. It does not get demo videos or conference talks. But every serious server administrator has a war story about a disk that filled up and took down a service. And most of those stories could have been prevented with the knowledge in this guide.
This tutorial covers everything: checking disk usage with df and du, finding large files, partitioning with fdisk and gdisk, creating filesystems, and — most importantly — using LVM (Logical Volume Manager) to build flexible, resizable storage that grows with your needs.
By the end, you will have the complete toolkit to manage disk storage on any Linux server, from a $5 VPS to a multi-disk dedicated machine.
If you just need to find what is eating your disk space right now, our shorter guide to managing disk space covers du, df, and ncdu on their own. This one goes further into partitioning, LVM, RAID, and the disk-full emergency playbook.
1. Understanding Linux Storage
Before you can manage disk space, you need to understand how Linux thinks about storage.
Block Devices
In Linux, physical disks appear as block devices under /dev/:
- /dev/sda, /dev/sdb — SATA/SAS disks (and most cloud VPS disks)
- /dev/nvme0n1, /dev/nvme1n1 — NVMe SSDs
- /dev/vda, /dev/vdb — Virtual disks (KVM/QEMU hypervisors)
- /dev/hda — Legacy IDE disks (rarely seen today)
Partitions are numbered: /dev/sda1, /dev/sda2, /dev/nvme0n1p1, etc.
Partition Tables: MBR vs GPT
Every disk has a partition table that describes how the disk is divided:
- MBR (Master Boot Record) — The old standard. Limited to 4 primary partitions (or 3 primary + 1 extended). Maximum disk size: 2TB. Still common on older systems.
- GPT (GUID Partition Table) — The modern standard. Supports up to 128 partitions, disks larger than 2TB, and has built-in redundancy. Required for UEFI boot. Use this for all new systems.
Filesystems
A filesystem is the structure that organizes data on a partition. Choosing the right filesystem matters:
| Feature | ext4 | XFS | btrfs |
|---|---|---|---|
| Max volume size | 1 EiB | 8 EiB | 16 EiB |
| Max file size | 16 TiB | 8 EiB | 16 EiB |
| Journaling | Yes | Yes (metadata default) | CoW (no journal) |
| Snapshots | No (needs LVM) | No (needs LVM) | Yes (native) |
| Shrink online | No | No | Yes |
| Grow online | Yes | Yes | Yes |
| Checksums | No | No | Yes |
| Best for | General use, compatibility | Large files, databases | Advanced features, backups |
Practical advice: Use ext4 for simplicity and broad compatibility. Use XFS for large database servers or high-throughput storage. Use btrfs if you need native snapshots and checksums.
Mount Points: Directories as Disks
Linux does not use drive letters like Windows. Instead, filesystems are mounted at directories. The root filesystem / is the starting point. Other disks can be mounted anywhere in the tree:
/ <-- root filesystem (OS partition)
/boot <-- often a separate partition (512MB)
/home <-- user data (can be a separate disk)
/var <-- logs, caches, mail — often separate on servers
/opt <-- additional software
/mnt <-- conventional mount point for temporary mounts
/media <-- for removable media
/etc/fstab — Persistent Mounts
To mount a filesystem automatically at boot, add it to /etc/fstab:
# <device> <mount point> <type> <options> <dump> <pass>
UUID=a1b2c3d4-... / ext4 defaults,noatime 0 1
UUID=e5f6a7b8-... /var ext4 defaults,noatime 0 2
/dev/vg_data/lv_www /var/www ext4 defaults 0 2
Key points:
- Use UUIDs, not device names like
/dev/sdb1— device names change; UUIDs do not noatime— Skip updating file access times; significant performance improvement on SSDs- The last field (pass): 0 = skip fsck, 1 = check first (root only), 2 = check after root
- After editing fstab: run
mount -ato test without rebooting
Warning: A typo in /etc/fstab can prevent the system from booting. Always test with mount -a first.
2. Checking Disk Space with df
df (disk free) reports filesystem-level disk usage. It shows how much space each mounted filesystem is using.
Basic df Usage
# Human-readable sizes
df -h
# Sample output:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 18G 30G 38% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda2 100G 67G 28G 71% /var
/dev/sdb1 500G 234G 241G 49% /mnt/data
# Include filesystem type
df -hT
# Output adds a Type column:
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 50G 18G 30G 38% /
/dev/sda2 xfs 100G 67G 28G 71% /var
Inode Usage — The Hidden "Disk Full"
This catches many admins off guard. A filesystem can run out of inodes (file index entries) even when there is free space. This causes "No space left on device" errors despite df -h showing available space.
# Check inode usage
df -i
# Sample output:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 3276800 3276799 1 100% / <-- FULL! (inode exhaustion)
/dev/sda2 6553600 123456 6430144 2% /var
Inode exhaustion is usually caused by millions of tiny files — email queue directories, session files, or cache entries. Diagnose with:
find / -xdev -printf %hn | sort | uniq -c | sort -k 1 -rn | head -20
Useful df Flags
df -h # Human readable
df -hT # With filesystem type
df -i # Inode usage
df -h /var # Only the filesystem containing /var
df -h --total # Add a total row
df -x tmpfs # Exclude tmpfs filesystems
3. Checking Directory Sizes with du
du (disk usage) measures how much space directories and files consume. While df shows filesystem-level usage, du tells you exactly where the space is going.
Basic du Usage
# Total size of a directory
du -sh /var/log
# Output: 4.2G /var/log
# First-level breakdown
du -sh --max-depth=1 /var
Finding the Largest Directories
# Top 20 largest directories under /var
du -sh /var/* | sort -rh | head -20
# Sample output:
2.1G /var/lib
834M /var/log
512M /var/cache
234M /var/backups
Recursive Breakdown
# Drill down into /var/lib
du -sh /var/lib/* | sort -rh | head -10
# Output:
1.4G /var/lib/mysql
456M /var/lib/docker
123M /var/lib/postgresql
ncdu — Interactive Disk Usage
ncdu (NCurses Disk Usage) is the fastest way to find disk hogs on an unfamiliar server:
apt install ncdu
ncdu /
Navigate with arrow keys, press d to delete (with confirmation), ? for help.
4. Finding Large Files and Disk Hogs
find — Large File Detection
# Files larger than 100MB
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null
# Large log files
find /var/log -name "*.log" -size +50M 2>/dev/null
# Files not accessed in 30 days, larger than 100MB
find / -type f -atime +30 -size +100M 2>/dev/null
lsof — The Ghost Space Problem
When you delete a file, Linux removes the directory entry — but if a process still has the file open, the disk space is not released until that process closes it. This is the "I freed space but df still shows full" problem.
# Find deleted files still held open
lsof +L1
# Sample output -- Nginx writing to a deleted log file:
nginx 1234 www-data 5w REG 8,1 2147483648 0 /var/log/nginx/access.log (deleted)
# Fix: restart the process holding the file
systemctl restart nginx
Common Disk Hogs and Their Solutions
- /var/log/ — Log files accumulating. Fix: configure logrotate. Emergency:
journalctl --vacuum-size=500Morfind /var/log -name "*.gz" -delete - /var/cache/apt/ — Package cache. Fix:
apt clean - /tmp/ — Temp files. Fix:
find /tmp -type f -atime +7 -delete - /var/lib/docker/ — Unused Docker objects. Fix:
docker system prune -af - /var/spool/postfix/ — Mail queue. Fix:
postsuper -d ALL - Old kernels — Fix:
apt autoremove - Journal logs — Fix:
journalctl --vacuum-size=500M --vacuum-time=30d
5. Partitioning with fdisk and gdisk
Critical warning: Never partition a disk that is mounted and in active use. For running servers, use LVM to avoid downtime.
List All Partitions
# List all disks and partitions
fdisk -l
# Tree view with lsblk
lsblk
# Sample output:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 50G 0 disk
|--sda1 8:1 0 49G 0 part /
|--sda2 8:2 0 1G 0 part [SWAP]
sdb 8:16 0 500G 0 disk
fdisk
# Open fdisk on a new disk
fdisk /dev/sdb
# Key commands:
# n -- New partition
# d -- Delete partition
# p -- Print current partition table
# w -- Write changes and exit
# q -- Quit without saving
Creating a single partition on /dev/sdb:
Command: n
Partition type: p (primary)
Partition number: 1
First sector: [Enter for default]
Last sector: [Enter to use full disk]
Command: w
gdisk — GPT Partitioning
gdisk works identically to fdisk but always uses GPT. Use it for disks larger than 2TB or any new installation.
gdisk /dev/sdb
parted — Modern Alternative
# Create GPT partition table
parted /dev/sdb mklabel gpt
# Create a single partition using 100% of disk
parted /dev/sdb mkpart primary ext4 0% 100%
# List partitions
parted /dev/sdb print
6. Creating and Managing Filesystems
Creating Filesystems
# Create ext4 filesystem
mkfs.ext4 /dev/sdb1
# Create ext4 with a label
mkfs.ext4 -L data_disk /dev/sdb1
# Create XFS filesystem
mkfs.xfs /dev/sdb1
# Create btrfs filesystem
mkfs.btrfs /dev/sdb1
Mounting Filesystems
# Mount a partition
mount /dev/sdb1 /mnt/data
# Mount with options
mount -o noatime,data=writeback /dev/sdb1 /mnt/data
# Unmount
umount /mnt/data
# Lazy unmount (last resort)
umount -l /mnt/data
Finding UUIDs for fstab
# Get UUIDs of all block devices
blkid
# Sample output:
/dev/sda1: UUID="a1b2c3d4-e5f6-7890-abcd-ef1234567890" TYPE="ext4"
/dev/sdb1: UUID="f1e2d3c4-b5a6-9870-fedc-ba9876543210" TYPE="ext4"
Persistent Mount via /etc/fstab
# Add to /etc/fstab:
UUID=f1e2d3c4-b5a6-9870-fedc-ba9876543210 /mnt/data ext4 defaults,noatime 0 2
# Test without rebooting
mount -a
# Verify
df -h /mnt/data
Filesystem Information and Health
# ext4 filesystem info (inode count, mount count, reserved blocks %)
tune2fs -l /dev/sda1
# XFS filesystem info (must be mounted)
xfs_info /mnt/data
# Check and repair ext4 (unmounted only!)
fsck.ext4 -f /dev/sdb1
# Check XFS (unmounted)
xfs_repair /dev/sdb1
7. LVM — Logical Volume Manager
LVM is the single most important disk management tool for server administrators. It solves the fundamental limitation of traditional partitioning: once you create a partition, resizing it is painful and risky. With LVM, you can resize volumes online, span storage across multiple disks, and take instant snapshots — all without downtime.
Why LVM?
- Resize without downtime — Extend a running volume in seconds
- Span multiple disks — One logical volume across 4 physical disks
- Snapshots — Instant consistent backup point
- Thin provisioning — Allocate more than you have, expand as needed
- Flexibility — Move data between physical disks while online
LVM Architecture
+-------------------------------------------------------------+
| FILESYSTEM LAYER |
| ext4 / XFS / btrfs on top of Logical Volumes |
+-------------------------------------------------------------+
| LOGICAL VOLUMES (LV) |
| lv_root (50GB) lv_www (100GB) lv_db (200GB) |
+-------------------------------------------------------------+
| VOLUME GROUP (VG) |
| vg_main (total: 350GB pool) |
+-------------------------------------------------------------+
| PHYSICAL VOLUMES (PV) |
| /dev/sda1 (150GB) /dev/sdb1 (200GB) |
+-------------------------------------------------------------+
| PHYSICAL DISKS |
| /dev/sda (SSD) /dev/sdb (HDD) |
+-------------------------------------------------------------+
- Physical Volume (PV) — A disk or partition initialized for LVM via
pvcreate - Volume Group (VG) — A pool of storage from one or more PVs — one big virtual disk
- Logical Volume (LV) — A virtual partition carved from the VG; you create filesystems here
Installing LVM Tools
apt install lvm2
systemctl enable lvm2-monitor
Step-by-Step LVM Setup
# Step 1: Initialize physical volumes
pvcreate /dev/sdb /dev/sdc
# Step 2: Create a volume group
vgcreate vg_data /dev/sdb /dev/sdc
# Step 3: Create logical volumes
lvcreate -L 100G -n lv_www vg_data
lvcreate -L 200G -n lv_db vg_data
lvcreate -l 100%FREE -n lv_backup vg_data
# Step 4: Create filesystems
mkfs.ext4 /dev/vg_data/lv_www
mkfs.xfs /dev/vg_data/lv_db
mkfs.ext4 /dev/vg_data/lv_backup
# Step 5: Mount
mkdir -p /var/www /var/lib/mysql /var/backups
mount /dev/vg_data/lv_www /var/www
mount /dev/vg_data/lv_db /var/lib/mysql
mount /dev/vg_data/lv_backup /var/backups
# Step 6: Add to /etc/fstab
echo "/dev/vg_data/lv_www /var/www ext4 defaults 0 2" >> /etc/fstab
echo "/dev/vg_data/lv_db /var/lib/mysql xfs defaults 0 2" >> /etc/fstab
Checking LVM Status
# Quick overview
pvs
vgs
lvs
# Sample vgs output:
VG #PV #LV #SN Attr VSize VFree
vg_data 2 3 0 wz--n- 1000.00g 200.00g
# Detailed info
pvdisplay
vgdisplay vg_data
lvdisplay /dev/vg_data/lv_www
Extending a Logical Volume (Online, Zero Downtime)
# Check available space in the VG
vgs
# Extend the LV by 50GB
lvextend -L +50G /dev/vg_data/lv_www
# Resize the filesystem (ext4)
resize2fs /dev/vg_data/lv_www
# Resize XFS (must be mounted; XFS can only grow)
xfs_growfs /var/www
# Extend LV and resize filesystem in one step (ext4)
lvextend -r -L +50G /dev/vg_data/lv_www
Adding a New Disk to an Existing Volume Group
# Initialize new disk and add to VG
pvcreate /dev/sdd
vgextend vg_data /dev/sdd
# Now extend any LV
lvextend -r -L +100G /dev/vg_data/lv_www
Shrinking a Logical Volume (ext4 only — backup first!)
XFS cannot shrink. For ext4, shrinking must be done offline:
# WARNING: back up data first — not reversible without restore
# 1. Unmount
umount /var/www
# 2. Check filesystem
e2fsck -f /dev/vg_data/lv_www
# 3. Shrink filesystem FIRST (before shrinking the LV!)
resize2fs /dev/vg_data/lv_www 80G
# 4. Shrink the LV to match
lvreduce -L 80G /dev/vg_data/lv_www
# 5. Remount
mount /dev/vg_data/lv_www /var/www
LVM Snapshots
LVM snapshots are instant, space-efficient copies of a logical volume (Copy-on-Write). A snapshot of a 100GB volume might only use a few GB initially.
# Create a 10GB snapshot
lvcreate -L 10G -s -n snap_www /dev/vg_data/lv_www
# Mount read-only for backup
mount -o ro /dev/vg_data/snap_www /mnt/snapshot
rsync -av /mnt/snapshot/ /backups/www-backup/
umount /mnt/snapshot
# Remove snapshot when done
lvremove /dev/vg_data/snap_www
# Restore from snapshot (DESTRUCTIVE -- current LV state is replaced!)
umount /var/www
lvconvert --merge /dev/vg_data/snap_www
Panelica uses this same copy-on-write snapshot principle for its own backup consistency feature — though on BTRFS-formatted volumes rather than LVM, since BTRFS provides native snapshots without a separate volume-manager layer. The underlying idea is identical: capture a point-in-time state before running database dumps, so backups stay consistent even under load.
8. RAID Basics with mdadm
RAID (Redundant Array of Independent Disks) combines multiple physical disks for redundancy and/or performance. Note: RAID is not a backup — it protects against disk failure, not accidental deletion or filesystem corruption.
RAID Levels Comparison
| Level | Min Disks | Redundancy | Read Perf | Write Perf | Usable Space | Best For |
|---|---|---|---|---|---|---|
| RAID 0 | 2 | None (worse!) | Excellent | Excellent | 100% | Temp data, speed only |
| RAID 1 | 2 | 1 disk failure | Good | Normal | 50% | OS disk, boot volume |
| RAID 5 | 3 | 1 disk failure | Good | OK | (n-1)/n | General storage |
| RAID 6 | 4 | 2 disk failures | Good | Slower | (n-2)/n | Large arrays (>6 disks) |
| RAID 10 | 4 | 1 per mirror | Excellent | Good | 50% | Databases, high I/O |
Creating RAID Arrays with mdadm
apt install mdadm
# Create RAID 1 (mirror) from 2 disks
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
# Create RAID 5 from 3 disks
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1
# Check sync progress
watch cat /proc/mdstat
# Detailed array info
mdadm --detail /dev/md0
# Save config for boot-time reassembly
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
update-initramfs -u
Replacing a Failed Disk
# Check status -- [UU_] means one failed disk in 3-disk RAID 5
cat /proc/mdstat
# Mark failed disk as faulty (if not auto-detected)
mdadm --manage /dev/md0 --fail /dev/sdc1
# Remove it
mdadm --manage /dev/md0 --remove /dev/sdc1
# Replace disk and add new one
mdadm --manage /dev/md0 --add /dev/sdc1
# Monitor rebuild
watch cat /proc/mdstat
RAID + LVM: Best of Both Worlds
Many production setups combine RAID (hardware redundancy) with LVM (flexible volumes):
# Physical Disks -> mdadm RAID -> LVM PV -> VG -> LVs
# Create RAID 1
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
# Create LVM on top of RAID
pvcreate /dev/md0
vgcreate vg_data /dev/md0
lvcreate -L 200G -n lv_db vg_data
9. Disk I/O Monitoring
Disk space is only one dimension. I/O performance matters just as much — a disk at 50% capacity but 100% I/O utilization will still cause problems.
iostat — Per-Device I/O Statistics
apt install sysstat
# Real-time I/O stats, refresh every 1 second
iostat -x 1
# Key columns:
# r/s -- Reads per second
# w/s -- Writes per second
# rMB/s -- Read throughput (MB/s)
# wMB/s -- Write throughput (MB/s)
# r_await -- Average read latency (ms) -- should be <10ms for SSD
# w_await -- Average write latency (ms)
# %util -- Disk utilization -- above 80% is concerning
iotop — Per-Process I/O
apt install iotop
# Show active processes I/O only
iotop -o
Disk Speed Testing
# Quick sequential read test
hdparm -tT /dev/sda
# fio -- Professional I/O benchmarking
apt install fio
# Test random read IOPS (database workload)
fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 --group_reporting
# Test sequential write throughput (backup workload)
fio --name=seqwrite --ioengine=libaio --iodepth=16 --rw=write --bs=1M --direct=1 --size=4G --numjobs=1 --runtime=60 --group_reporting
Key I/O Metrics
- IOPS — Critical for databases. NVMe: 100K-500K. SATA SSD: 10K-100K. HDD: 100-200.
- Throughput (MB/s) — Critical for backups and file transfers.
- Latency (ms) — Below 1ms for NVMe, below 5ms for SATA SSD, 5-20ms for HDD.
- %util — Above 80% = saturation risk.
10. SSD-Specific Management
SSDs require different management than HDDs. Ignoring SSD-specific settings wastes performance and reduces drive lifespan.
TRIM — Keeping SSDs Fast
SSDs need TRIM to maintain performance over time. Without TRIM, the drive degrades as it fills with dirty blocks.
# Check TRIM support
lsblk --discard
# DISC-GRAN non-zero = TRIM supported
# Manual TRIM
fstrim -v /
fstrim -v /var
# Enable automatic weekly TRIM (recommended)
systemctl enable fstrim.timer
systemctl start fstrim.timer
# Check timer status
systemctl status fstrim.timer
Mount Options for SSDs
# /etc/fstab -- optimized for SSD
UUID=xxx / ext4 defaults,noatime,discard 0 1
# noatime -- Skip access time updates (major write reduction)
# discard -- Online TRIM (use fstrim.timer instead on busy servers)
Note: On busy database servers, prefer fstrim.timer (periodic batch TRIM) over the discard mount option. Continuous TRIM can cause I/O latency spikes under heavy write loads.
SSD Health Monitoring
apt install smartmontools
# Check overall health
smartctl -a /dev/sda
# Key metrics to watch:
# Reallocated_Sector_Ct -- Bad sectors moved to spares. Should be 0.
# Wear_Leveling_Count -- How worn the SSD is
# Total_LBAs_Written -- Total data written (TBW indicator)
# Power_On_Hours -- Drive age
# Run a quick self-test
smartctl -t short /dev/sda
smartctl -a /dev/sda # View results after ~2 minutes
11. Emergency: Disk Full Recovery
When you get the call at 2 AM — the website is down — and it is a disk full situation, you need to move fast. Here is the exact procedure.
Step 1: Confirm and Locate
# Which filesystem is full?
df -h
# Check inode exhaustion too
df -i
# Find the culprit
du -sh /var/* | sort -rh | head -20
du -sh /* | sort -rh | head -10
Step 2: Quick Wins
# 1. Vacuum journal logs (usually safe, frees 100MB-10GB)
journalctl --vacuum-size=100M
# 2. Clean apt package cache
apt clean
# 3. Remove old compressed log files
find /var/log -name "*.gz" -delete
find /var/log -name "*.1" -delete
# 4. Clear old temp files
find /tmp -type f -atime +7 -delete
find /var/tmp -type f -atime +30 -delete
# 5. Remove old kernels
apt autoremove
# Check: how much did we free?
df -h
Step 3: Ghost Space (If df Still Shows Full)
# Find deleted files still held open by processes
lsof +L1 | grep deleted
# Restart the offending process
systemctl restart nginx
systemctl restart php8.3-fpm
Step 4: Identify the Root Cause
# What filled the disk?
du -sh /var/log/* | sort -rh | head # Log explosion?
du -sh /var/lib/* | sort -rh | head # Database growth?
du -sh /home/* | sort -rh | head # User files?
du -sh /var/spool/* | sort -rh | head # Mail queue?
# Files created in the last 24 hours, larger than 50MB
find / -type f -newer /tmp -size +50M 2>/dev/null
Step 5: Long-Term Fix
- Log explosion → Configure logrotate:
/etc/logrotate.confand/etc/logrotate.d/ - Database growth → Check for large tables, binary log accumulation (
PURGE BINARY LOGS) - Docker bloat → Set
--log-max-size, enable periodicdocker system prune - Persistent fix → Add disk to LVM volume group and extend the logical volume — no downtime
- Monitoring → Alert at 75%, act at 85%, never see 100%
Quick Reference Cheat Sheet
Disk Information Commands
| Command | What It Shows |
|---|---|
df -h | Filesystem usage (human readable) |
df -hT | Filesystem usage with type |
df -i | Inode usage |
lsblk | Block devices tree view |
fdisk -l | All partitions |
blkid | UUID and filesystem type |
du -sh /path | Directory total size |
du -sh /* | sort -rh | Root-level breakdown sorted by size |
lsof +L1 | Deleted files still open |
iostat -x 1 | Real-time I/O stats per device |
iotop -o | Per-process I/O usage |
smartctl -a /dev/sda | Disk health (SMART data) |
LVM Commands
| Command | What It Does |
|---|---|
pvs / pvdisplay | List physical volumes |
vgs / vgdisplay | List volume groups |
lvs / lvdisplay | List logical volumes |
pvcreate /dev/sdb | Initialize disk as PV |
vgcreate vg0 /dev/sdb | Create volume group |
vgextend vg0 /dev/sdc | Add disk to VG |
lvcreate -L 50G -n lv0 vg0 | Create 50GB logical volume |
lvextend -r -L +20G /dev/vg0/lv0 | Extend LV and resize filesystem |
lvcreate -L 10G -s -n snap /dev/vg0/lv0 | Create snapshot |
lvremove /dev/vg0/lv0 | Remove logical volume |
Emergency Disk Recovery Commands
| Command | What It Frees |
|---|---|
journalctl --vacuum-size=100M | Journal logs down to 100MB |
apt clean | APT package download cache |
find /var/log -name "*.gz" -delete | Compressed old log files |
docker system prune -af | Unused Docker images/volumes/networks |
apt autoremove | Old kernels and unused packages |
postsuper -d ALL | Entire Postfix mail queue |
fstrim -v / | TRIM SSD (improves future write performance) |
Frequently Asked Questions
Can I extend an LVM logical volume without downtime?
Yes — lvextend followed by resize2fs (ext4) or xfs_growfs (XFS) both run on a mounted, live filesystem. This is one of the main reasons to put production data on LVM from day one rather than a raw partition.
Why does df show free space but I still get "No space left on device"?
You have likely run out of inodes, not disk blocks. Check with df -i — a filesystem with millions of tiny files (mail queues, session caches) can hit 100% inode usage while still showing free gigabytes in df -h.
Can an XFS filesystem be shrunk?
No. XFS can only grow. If you need to shrink a volume, you must back up the data, recreate a smaller filesystem, and restore — or use ext4 instead, which supports offline shrinking via resize2fs followed by lvreduce.
Is RAID a substitute for backups?
No. RAID protects against a physical disk failure, not against accidental deletion, ransomware, or filesystem corruption — a bad rm -rf replicates across every disk in the array just as fast as a good write does. RAID and backups solve different problems and you need both.
Conclusion: Prevention Over Recovery
The difference between an admin who gets woken up at 3 AM for disk-full emergencies and one who sleeps through the night is monitoring and proactive management.
The tools in this guide — df, du, LVM — are the foundation. But the real goal is to never need the emergency section. That means:
- Monitor continuously — Alert at 75%, take action at 85%, never see 100%
- Use LVM from the start — Extending a volume takes 30 seconds; repartitioning under load takes hours of risk
- Configure logrotate — Every server, every service that writes logs needs an entry in
/etc/logrotate.d/ - TRIM your SSDs — Enable
fstrim.timeronce and forget about it - Know your growth patterns — Databases grow, mail queues fill, Docker images accumulate
If you manage servers with Panelica, disk usage is monitored automatically — real-time per-partition graphs, per-user quota tracking, and alerts before you hit the wall. The panel also handles BTRFS-backed backup snapshots with consistency guarantees, so your backups capture a point-in-time state even while databases are running under load.
Whether you use Panelica or manage servers manually, the commands in this guide are the same. Master them, and disk full becomes a footnote in your career — not a recurring 3 AM crisis.