For future generations looking to solve this problem:
I believe one of my modules (MMM-CTA for the record) poops out a ton of log entries. Those fill up my SD card after a few weeks.
I ended up installing a pm2 module called pm2-logrotate. https://www.npmjs.com/package/pm2-logrotate
I’m setting it to rotate my var/log file before it gets too big. Hopefully this will solve future problems.
pi@raspberrypi:~ $ pm2 install pm2-logrotate
pi@raspberrypi:~ $ pm2 set pm2-logrotate:max_size 2G
Thanks for the response!
Based on that, I went ahead and deleted those guys. I did this:
pi@raspberrypi:~ $ sudo rm /var/log/kern* &>/dev/null
pi@raspberrypi:~ $ sudo rm /var/log/messages* &>/dev/null
…then rebooted and everything came back to life.
I’ve now got a little breathing room on my disk:
pi@raspberrypi:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 13G 11G 1.4G 89% /
devtmpfs 333M 0 333M 0% /dev
tmpfs 462M 0 462M 0% /dev/shm
tmpfs 462M 12M 450M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 462M 0 462M 0% /sys/fs/cgroup
/dev/mmcblk0p6 68M 21M 47M 31% /boot
tmpfs 93M 0 93M 0% /run/user/1000
And I guess I need to implement some kind of solution to “rotate my logs” if that’s the correct nomenclature?
The question remains: why did this happen? I know one of my modules is kind of noisy in the logs – like it updates once a minute and even though it’s working it sends an error message to the logs every time. But it’s hard to imagine that that one-line error would take up multiple gigs in a couple months. I dunno.
Maybe “log rotation” should be addressed in the config instructions?
Upgraded. Did all the config changes. Worked great. Went out of town. Came back to a blank mirror. Was confused to find the screen and RPi still working fine. SSH’ed in, and every pm2 command I tried gave me
[PM2] Spawning PM2 daemon with pm2_home=/home/pi/.pm2
Much googling ensued. Tried changing permission on a couple of files. Didn’t work. Finally decided it must mean my disk is full.
pi@raspberrypi:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 13G 13G 0 100% /
devtmpfs 333M 0 333M 0% /dev
tmpfs 462M 0 462M 0% /dev/shm
tmpfs 462M 12M 450M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 462M 0 462M 0% /sys/fs/cgroup
/dev/mmcblk0p6 68M 21M 47M 31% /boot
tmpfs 93M 0 93M 0% /run/user/1000
So suspicions confirmed I guess?
I used ncdu to try to figure out if something ballooned out of control and that’s when I realized I’m a complete beginner and have no idea what to look for. Here’s what I saw:
--- / ------------------------------------------------------------------------------------------------------------------------------
. 7.4 GiB [##########] /var
2.8 GiB [### ] /usr
1.0 GiB [# ] /home
945.6 MiB [# ] /opt
182.2 MiB [ ] /lib
21.0 MiB [ ] /boot
. 11.9 MiB [ ] /run
7.3 MiB [ ] /sbin
6.8 MiB [ ] /bin
. 5.7 MiB [ ] /etc
. 228.0 KiB [ ] /tmp
! 16.0 KiB [ ] /lost+found
. 12.0 KiB [ ] /media
e 4.0 KiB [ ] /srv
! 4.0 KiB [ ] /root
e 4.0 KiB [ ] /mnt
. 0.0 B [ ] /sys
. 0.0 B [ ] /proc
0.0 B [ ] /dev
Selecting that top directory (assuming the culprit is somewhere in there), I see this:
--- /var ---------------------------------------------------------------------------------------------------------------------------
/..
. 7.1 GiB [##########] /log
. 145.6 MiB [ ] /cache
. 115.3 MiB [ ] /lib
100.0 MiB [ ] swap
296.0 KiB [ ] /backups
. 68.0 KiB [ ] /tmp
. 28.0 KiB [ ] /spool
e 4.0 KiB [ ] /opt
e 4.0 KiB [ ] /mail
e 4.0 KiB [ ] /local
@ 0.0 B [ ] lock
@ 0.0 B [ ] run
Hm. Let’s try that top one again. Here’s what we get (truncated to the directories that take up any substantial space):
--- /var/log -----------------------------------------------------------------------------------------------------------------------
/..
2.4 GiB [##########] kern.log
1.8 GiB [####### ] messages
1.4 GiB [##### ] messages.1
534.7 MiB [## ] syslog.1
533.4 MiB [## ] kern.log.1
287.9 MiB [# ] syslog
24.9 MiB [ ] syslog.3.gz
And that’s where I’m stuck. Is my culprit in there? Can I delete it? How can I prevent this from happening in the future?