VM Disk Full - 50 GiB in 4 months

sdetweil

@JohnGalt erase the files in the .pm2/logs folder, they are not the cause

JohnGalt

@sdetweil – I will do that. I was just down the rabbit hole, using df. It looks like a huge amount is taken up by /var/lib/apport/coredump. See below:

JohnGalt@MM-Dev-VM:~/.pm2/logs$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           794M  1.1M  793M   1% /run
/dev/sda2        49G   49G     0 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs         56K   27K   25K  52% /sys/firmware/efi/efivars
/dev/sda1       300M  7.8M  292M   3% /boot/efi
tmpfs           794M   92K  794M   1% /run/user/1000
JohnGalt@MM-Dev-VM:~/.pm2/logs$ cd ..
JohnGalt@MM-Dev-VM:~/.pm2$ cd ..
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var | sort -h | tail -n 10
4.0K    /var/metrics
4.0K    /var/opt
28K     /var/crash
40K     /var/spool
68K     /var/tmp
5.4M    /var/backups
49M     /var/cache
617M    /var/log
39G     /var
39G     /var/lib
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib | sort -h | tail -n 10
772K    /var/lib/systemd
2.0M    /var/lib/fwupd
3.8M    /var/lib/command-not-found
4.2M    /var/lib/aspell
7.5M    /var/lib/ubuntu-advantage
40M     /var/lib/swcatalog
92M     /var/lib/dpkg
276M    /var/lib/apt
38G     /var/lib/apport
39G     /var/lib
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib/apport | sort -h | tail -n 10
38G     /var/lib/apport
38G     /var/lib/apport/coredump
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib/apport/coredump | sort -h | tail -n 10
38G     /var/lib/apport/coredump
JohnGalt@MM-Dev-VM:~$

sdetweil

@JohnGalt next is who’s dump is it… electron?

JohnGalt

@sdetweil – true. I’m not sure I know to find that out…

JohnGalt

is that file [just] a record of the dump[s]?

Put differently, would it be safe to delete the file or directory?

[Edit]: On further recollection - this appears to be happening only on the two VMs running MM, and not on the two Raspberry Pis that I have MM running on.

Does this represent a clue as to what is happening?

sdetweil

@JohnGalt in the folder with the core dump files do

sudo file * >>~/somefile.txt

then ~/somefile.txt will have a line for each which should contain the failing code name

from
https://stackoverflow.com/questions/13308922/find-which-program-caused-a-core-dump-file

JohnGalt

@sdetweil

Well, there’s this under /var/lib/apport/coredump:

Name:
core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464

Size:
40,486,436,864

sdetweil

@JohnGalt well then… electron crashed…

which MM version are you on
cd ~/MagicMirror
grep version package.json

and for electron
grep version node_modules/electron/package.json

JohnGalt

@sdetweil

JohnGalt@MM-Dev-VM:~/MagicMirror$ grep version package.json
        "version": "2.35.0",
JohnGalt@MM-Dev-VM:~/MagicMirror$ grep version node_modules/electron/package.json
  "version": "41.2.0"

sdetweil

@JohnGalt if you hadn’t erased the logs, the pm2 log MIGHT have had a fault listed…

2.37 is the current release, 6 months newer… electron at 42.5.2

supposedly this could help find cause

 Analyze the Crash Dump
Because Electron code is pre-compiled, looking at a raw dump file yields only unreadable memory addresses. 
You need to symbolicate the dump using matching Electron debugging symbols.

The CLI Tool (electron-minidump)
    The easiest open-source way to inspect a .dmp file locally is using the electron-   minidump CLI tool. 
It automatically fetches the matching Electron symbols and outputs a readable stack trace.
Install the utility:
sudo  npm install -g electron-minidump

Analyze your file:
electron-minidump -f /path/to/your/file.dmp

JohnGalt

@sdetweil – OK, well… The program DID install (I was surprised , given what has been happening…). This resulted in an error. I specified that large file we identified earlier. Do let me know if I did the wrong thing. See results below.

While you are looking at this, I am going to fire up the other, older VM that also crashed and see if there is similar evidence there that might shed light.

JohnGalt@MM-Dev-VM:~/MagicMirror$ sudo  npm install -g electron-minidump
[sudo] password for JohnGalt:

added 29 packages in 34s

1 package is looking for funding
  run `npm fund` for details
npm notice
npm notice New major version of npm available! 10.9.4 -> 12.0.2
npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.0.2
npm notice To update run: npm install -g npm@12.0.2
npm notice
JohnGalt@MM-Dev-VM:~/MagicMirror$ electron-minidump -f /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
Error: EACCES: permission denied, open '/var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464'
    at async Object.open (node:internal/fs/promises:641:25)
    at async electronMinidump (/usr/local/lib/node_modules/electron-minidump/index.js:11:14) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464'
}
JohnGalt@MM-Dev-VM:~/MagicMirror$ sudo electron-minidump -f /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
Not a minidump file (MDMP header not found): /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
JohnGalt@MM-Dev-VM:~/MagicMirror$

sdetweil

@JohnGalt you need sudo in front of the mini dump. Line cause the file is in the /var folder