Upgraded. Did all the config changes. Worked great. Went out of town. Came back to a blank mirror. Was confused to find the screen and RPi still working fine. SSH’ed in, and every pm2 command I tried gave me
[PM2] Spawning PM2 daemon with pm2_home=/home/pi/.pm2
Much googling ensued. Tried changing permission on a couple of files. Didn’t work. Finally decided it must mean my disk is full.
pi@raspberrypi:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 13G 13G 0 100% /
devtmpfs 333M 0 333M 0% /dev
tmpfs 462M 0 462M 0% /dev/shm
tmpfs 462M 12M 450M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 462M 0 462M 0% /sys/fs/cgroup
/dev/mmcblk0p6 68M 21M 47M 31% /boot
tmpfs 93M 0 93M 0% /run/user/1000
So suspicions confirmed I guess?
I used ncdu
to try to figure out if something ballooned out of control and that’s when I realized I’m a complete beginner and have no idea what to look for. Here’s what I saw:
--- / ------------------------------------------------------------------------------------------------------------------------------
. 7.4 GiB [##########] /var
2.8 GiB [### ] /usr
1.0 GiB [# ] /home
945.6 MiB [# ] /opt
182.2 MiB [ ] /lib
21.0 MiB [ ] /boot
. 11.9 MiB [ ] /run
7.3 MiB [ ] /sbin
6.8 MiB [ ] /bin
. 5.7 MiB [ ] /etc
. 228.0 KiB [ ] /tmp
! 16.0 KiB [ ] /lost+found
. 12.0 KiB [ ] /media
e 4.0 KiB [ ] /srv
! 4.0 KiB [ ] /root
e 4.0 KiB [ ] /mnt
. 0.0 B [ ] /sys
. 0.0 B [ ] /proc
0.0 B [ ] /dev
Selecting that top directory (assuming the culprit is somewhere in there), I see this:
--- /var ---------------------------------------------------------------------------------------------------------------------------
/..
. 7.1 GiB [##########] /log
. 145.6 MiB [ ] /cache
. 115.3 MiB [ ] /lib
100.0 MiB [ ] swap
296.0 KiB [ ] /backups
. 68.0 KiB [ ] /tmp
. 28.0 KiB [ ] /spool
e 4.0 KiB [ ] /opt
e 4.0 KiB [ ] /mail
e 4.0 KiB [ ] /local
@ 0.0 B [ ] lock
@ 0.0 B [ ] run
Hm. Let’s try that top one again. Here’s what we get (truncated to the directories that take up any substantial space):
--- /var/log -----------------------------------------------------------------------------------------------------------------------
/..
2.4 GiB [##########] kern.log
1.8 GiB [####### ] messages
1.4 GiB [##### ] messages.1
534.7 MiB [## ] syslog.1
533.4 MiB [## ] kern.log.1
287.9 MiB [# ] syslog
24.9 MiB [ ] syslog.3.gz
And that’s where I’m stuck. Is my culprit in there? Can I delete it? How can I prevent this from happening in the future?