A few days after upgrading to 2.4.1, my disk filled up and now my mirror won't run. Sad face!

Preschool girl · Aug 4, 2018, 7:24 PM

Upgraded. Did all the config changes. Worked great. Went out of town. Came back to a blank mirror. Was confused to find the screen and RPi still working fine. SSH’ed in, and every pm2 command I tried gave me

[PM2] Spawning PM2 daemon with pm2_home=/home/pi/.pm2

Much googling ensued. Tried changing permission on a couple of files. Didn’t work. Finally decided it must mean my disk is full.

pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        13G   13G     0 100% /
devtmpfs        333M     0  333M   0% /dev
tmpfs           462M     0  462M   0% /dev/shm
tmpfs           462M   12M  450M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p6   68M   21M   47M  31% /boot
tmpfs            93M     0   93M   0% /run/user/1000

So suspicions confirmed I guess?

I used ncdu to try to figure out if something ballooned out of control and that’s when I realized I’m a complete beginner and have no idea what to look for. Here’s what I saw:

--- / ------------------------------------------------------------------------------------------------------------------------------
.   7.4 GiB [##########] /var                                                                                                       
    2.8 GiB [###       ] /usr
    1.0 GiB [#         ] /home
  945.6 MiB [#         ] /opt
  182.2 MiB [          ] /lib
   21.0 MiB [          ] /boot
.  11.9 MiB [          ] /run
    7.3 MiB [          ] /sbin
    6.8 MiB [          ] /bin
.   5.7 MiB [          ] /etc
. 228.0 KiB [          ] /tmp
!  16.0 KiB [          ] /lost+found
.  12.0 KiB [          ] /media
e   4.0 KiB [          ] /srv
!   4.0 KiB [          ] /root
e   4.0 KiB [          ] /mnt
.   0.0   B [          ] /sys
.   0.0   B [          ] /proc
    0.0   B [          ] /dev

Selecting that top directory (assuming the culprit is somewhere in there), I see this:

--- /var ---------------------------------------------------------------------------------------------------------------------------
                         /..
.   7.1 GiB [##########] /log                                                                                                       
. 145.6 MiB [          ] /cache
. 115.3 MiB [          ] /lib
  100.0 MiB [          ]  swap
  296.0 KiB [          ] /backups
.  68.0 KiB [          ] /tmp
.  28.0 KiB [          ] /spool
e   4.0 KiB [          ] /opt
e   4.0 KiB [          ] /mail
e   4.0 KiB [          ] /local
@   0.0   B [          ]  lock
@   0.0   B [          ]  run

Hm. Let’s try that top one again. Here’s what we get (truncated to the directories that take up any substantial space):

--- /var/log -----------------------------------------------------------------------------------------------------------------------
                         /..                                                                                                        
    2.4 GiB [##########]  kern.log
    1.8 GiB [#######   ]  messages
    1.4 GiB [#####     ]  messages.1
  534.7 MiB [##        ]  syslog.1
  533.4 MiB [##        ]  kern.log.1
  287.9 MiB [#         ]  syslog
   24.9 MiB [          ]  syslog.3.gz

And that’s where I’m stuck. Is my culprit in there? Can I delete it? How can I prevent this from happening in the future?

MadScientist · Aug 4, 2018, 8:08 PM

I think you should be able to delete everything from /var/log as these are just log files. But the question remains why the logs are inflated so much. Something gets triggered to write something in the logs. I wouldn’t know how to find out what it is as I am a total n00b myself (even after years of using Linux).

Preschool girl · Aug 4, 2018, 11:45 PM

Thanks for the response!

Based on that, I went ahead and deleted those guys. I did this:

pi@raspberrypi:~ $ sudo rm /var/log/kern* &>/dev/null
pi@raspberrypi:~ $ sudo rm /var/log/messages* &>/dev/null

…then rebooted and everything came back to life.

I’ve now got a little breathing room on my disk:

pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        13G   11G  1.4G  89% /
devtmpfs        333M     0  333M   0% /dev
tmpfs           462M     0  462M   0% /dev/shm
tmpfs           462M   12M  450M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p6   68M   21M   47M  31% /boot
tmpfs            93M     0   93M   0% /run/user/1000

And I guess I need to implement some kind of solution to “rotate my logs” if that’s the correct nomenclature?

The question remains: why did this happen? I know one of my modules is kind of noisy in the logs – like it updates once a minute and even though it’s working it sends an error message to the logs every time. But it’s hard to imagine that that one-line error would take up multiple gigs in a couple months. I dunno.

Maybe “log rotation” should be addressed in the config instructions?

Preschool girl · Dec 1, 2018, 2:39 AM

For future generations looking to solve this problem:

I believe one of my modules (MMM-CTA for the record) poops out a ton of log entries. Those fill up my SD card after a few weeks.

I ended up installing a pm2 module called pm2-logrotate. https://www.npmjs.com/package/pm2-logrotate

I’m setting it to rotate my var/log file before it gets too big. Hopefully this will solve future problems.

pi@raspberrypi:~ $ pm2 install pm2-logrotate
pi@raspberrypi:~ $ pm2 set pm2-logrotate:max_size 2G