How to Fix the `No Space Left on Device` Issue on Ubuntu Server

The “No space left on device” error on an Ubuntu server is a common issue that occurs when the disk or file system becomes full, preventing the system from writing or creating new files. This problem can affect server performance and stability, especially when we are running applications or services that require disk space to function properly. In this guide, we’ll walk through various methods to diagnose and fix this issue.

1. Check Disk Usage with df Command

The first step to resolve the “No space left on device” issue is to check the available disk space. we can do this with the df command, which shows the disk usage statistics.

Run the following command to display the disk usage for all mounted file systems:

df -h

• -h makes the output human-readable (e.g., in GB or MB).
• The output will show the filesystem, size, used space, available space, and mount point.

Example output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       20G  19G   1G  95% /
/dev/sdb1       50G  10G  40G  20% /data

In this case, /dev/sda1 (the root filesystem) is 95% full, meaning there’s only 1GB of free space left. This could be the cause of the error.

2. Identify Large Files and Directories

If the disk is almost full, it’s crucial to identify which files or directories are consuming the most space. we can use the du (disk usage) command to find large files and directories.

To get a summary of disk usage by directory in the root filesystem:

sudo du -h --max-depth=1 /

• -h gives a human-readable format.
• --max-depth=1 limits the listing to the first level of directories.

This will give we an overview of which directories are taking up the most space.

If we need to explore a specific directory further, for example, /var, we can run:

sudo du -h --max-depth=1 /var

3. Check for Large Log Files

Log files often grow quickly and can occupy a significant amount of space if not managed correctly. A common location for logs is the /var/log directory.

To check the disk usage of files in /var/log, run:

sudo du -sh /var/log/*

Look for large log files such as syslog, kern.log, or nginx/*.log.

we can clear log files safely by truncating them. For example:

sudo truncate -s 0 /var/log/syslog
sudo truncate -s 0 /var/log/kern.log

Alternatively, we can use log rotation tools like logrotate to manage log file sizes automatically.

4. Delete Unnecessary Files

Check for unnecessary files that can be removed. These could be:

• Old backups
• Temporary files
• Unused software packages

To clean up old package files and cached data, we can use the following commands:

• Clean the APT cache:

  sudo apt-get clean
  

• Remove unused packages:

  sudo apt-get autoremove
  

• Remove orphaned packages:

  sudo apt-get autoclean
  

• Clean up temporary files: we can delete temporary files from the /tmp directory with:

  sudo rm -rf /tmp/*
  

5. Examine and Fix File System Issues

If wer disk usage appears normal but we are still seeing the “No space left on device” error, there could be issues with the file system itself, such as file system corruption or inode exhaustion.

Check Inodes Usage

Inodes are data structures that store metadata about files, and if we run out of inodes, the system cannot create new files even if there is free space. we can check inode usage with the df -i command:

df -i

Example output:

Filesystem      Inodes  IUsed  IFree IUse% Mounted on
/dev/sda1       1M     1M      0    100% /

If we see that inodes are 100% used, we will need to free up space by deleting files or directories with many small files.

Check and Repair File System

To check and repair the file system, use the fsck (file system check) tool. First, unmount the file system:

sudo umount /dev/sda1

Then run the fsck tool on the partition:

sudo fsck /dev/sda1

Follow the prompts to repair any errors found on the disk.

6. Resize the File System or Add More Disk Space

If the disk is genuinely full, and we’re unable to free enough space, consider resizing the file system or adding more disk space. If we’re using a virtual machine, we can resize the virtual disk in wer hypervisor and then extend the partition using gparted or resize2fs.

For instance, to resize the root partition, we can use resize2fs:

sudo resize2fs /dev/sda1

7. Monitor Disk Space Usage

To prevent future occurrences of this issue, it’s essential to monitor disk usage regularly. There are several monitoring tools available, including:

• df and du commands (manual checks)
• dstat or sysstat (for detailed disk activity)

8. Fix large log files using logrotate

Log files, especially those in /var/log, can grow rapidly if not managed properly. To prevent this from causing the “No space left on device” issue, we can configure Logrotate.

• Install logrotate

sudo apt-get install logrotate

• Logrotate Configuration

  • Global Configuration: Located in /etc/logrotate.conf, this file contains global settings that apply to all logs.
  • Individual Log Configurations: Located in /etc/logrotate.d/, this directory contains individual configuration files for specific log files or services (e.g., Apache, Nginx, MySQL, etc.).
  • Logrotate works by rotating logs based on predefined criteria, such as file size, age, or time. It can also compress logs to save space.

• Configure Logrotate for Specific Log Files

  • We can create or modify a configuration file for specific log files to rotate them automatically when they grow too large. Let’s configure log rotation for a large log file, such as /var/log/syslog.

• Configure Logrotate for Specific Log Files

  • We can create or modify a configuration file for specific log files to rotate them automatically when they grow too large. Let’s configure log rotation for a large log file, such as /var/log/syslog.

  • Create a Custom Logrotate File: we can create a new file inside /etc/logrotate.d/ to configure rotation for a specific log file.

    sudo vim /etc/logrotate.d/syslog
  

  • Logrotate Configuration: Here’s a sample configuration for rotating the syslog file

    /var/log/syslog {
        daily               # Rotate logs every day
        missingok           # Ignore errors if the log file is missing
        rotate 7            # Keep 7 days of log files
        compress            # Compress old logs to save space
        delaycompress       # Delay compression until the second rotation
        notifempty          # Do not rotate if the log file is empty
        create 0640 root root # Create new log with specified permissions and ownership
        size 50M            # Rotate the log file when it reaches 50MB in size
    }
    

    • daily: Rotates the log file every day.
    • rotate 7: Keeps the last 7 log files and removes older ones.
    • compress: Compresses old log files to save space.
    • delaycompress: Delays the compression of old logs until the second rotation.
    • size 50M: Rotates the log file when it reaches 50MB.
    • notifempty: Does not rotate the log if it’s empty.
    • create 0640 root root: Creates a new log file with specific permissions and ownership.

  • Test the Logrotate Configuration

    • Once we have configured logrotate for the large log files, we can test the configuration to make sure it’s working as expected

    sudo logrotate -d /etc/logrotate.conf
    

    • The -d option tells logrotate to run in debug mode, simulating the rotation process without actually performing it. Check for any errors or issues in the output.
    • If everything looks good, we can force logrotate to run and rotate the logs immediately:

    sudo logrotate -f /etc/logrotate.conf
    

  • Setting Up Cron for Logrotate:

    • Logrotate is typically configured to run automatically via a cron job. Check if logrotate is already scheduled by inspecting the cron directory

    sudo vim /etc/cron.daily/logrotate
    

    • This file should already exist, and logrotate will be run automatically once a day.
    • If we want to adjust how often logrotate runs, we can create custom cron jobs for more frequent rotations. For example, we could create a cron job that runs logrotate every hour if needed:

    sudo vim /etc/cron.hourly/logrotate
    

    Inside this file, add:

    #!/bin/sh
    /usr/sbin/logrotate /etc/logrotate.conf
    

    • Make the file executable

    sudo chmod +x /etc/cron.hourly/logrotate
    

  • Monitor and Adjust the Rotation Settings

    • After setting up logrotate, it’s important to monitor the system’s disk usage to ensure that log rotation is working properly. we can adjust the frequency, size limits, and number of retained log files as needed.
    • Additionally, we might want to periodically check the /var/log directory to ensure that old logs are being compressed and removed as expected.

Would you like to download 10+ free Django and Python books? Get them here