VM Disk Full - 50 GiB in 4 months

JohnGalt

@sdetweil – Thanks Sam. See below for the output of the file sizes there.

JohnGalt@MM-Dev-VM:~/.pm2/logs$ ls -laF
total 32988
drwxrwxr-x 2 al al     4096 Apr 12 11:24 ./
drwxrwxr-x 5 al al     4096 Aug 19 12:09 ../
-rw-rw-r-- 1 al al 27348992 Aug 17 23:33 MagicMirror-error.log
-rw-rw-r-- 1 al al  6413424 Aug 17 23:27 MagicMirror-out.log

sdetweil

@JohnGalt said:

27348992

error log is 27 meg… not gig

after that you would have to use something like ncdu

ncdu (NCurses Disk Usage) is a fast, text-based command-line tool used to analyze disk space.
It scans your file system and shows which folders and files use the most space, letting you easily navigate and delete large items directly inside an interactive terminal screen.
How to Install ncdu
Ubuntu / Debian: sudo apt install ncdu
CentOS / RHEL / Fedora: sudo dnf install ncdu or sudo yum install ncdu
macOS: brew install ncdu
Arch Linux: sudo pacman -S ncdu

Common Commands
ncdu — Scan the current folder.  sudo ncdu / --Scan your entire hard drive.
ncdu /home --Scan a specific folder.  
ncdu -x --Scan only the current disk (ignore mounted external drives).
Useful Keyboard Shortcuts
Up / Down arrows or j / k — Move through the list.
Enter / Right arrow — Open a folder to see files inside.
Left arrow — Go back to the parent folder.
d — Delete the highlighted file or folder.

JohnGalt

@sdetweil – Thanks, Sam. It looks like that won’t work - for the same reason as above. I need to give this some more thought… I can’t be the first person to encounter this issue. See below for the output of trying to install ncdu:

JohnGalt@MM-Dev-VM:~/.pm2/logs$ sudo apt install ncdu
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  ncdu
0 upgraded, 1 newly installed, 0 to remove and 33 not upgraded.
Need to get 50.7 kB of archives.
After this operation, 117 kB of additional disk space will be used.
E: You don't have enough free space in /var/cache/apt/archives/.
JohnGalt@MM-Dev-VM:~/.pm2/logs$

sdetweil

@JohnGalt erase the files in the .pm2/logs folder, they are not the cause

JohnGalt

@sdetweil – I will do that. I was just down the rabbit hole, using df. It looks like a huge amount is taken up by /var/lib/apport/coredump. See below:

JohnGalt@MM-Dev-VM:~/.pm2/logs$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           794M  1.1M  793M   1% /run
/dev/sda2        49G   49G     0 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs         56K   27K   25K  52% /sys/firmware/efi/efivars
/dev/sda1       300M  7.8M  292M   3% /boot/efi
tmpfs           794M   92K  794M   1% /run/user/1000
JohnGalt@MM-Dev-VM:~/.pm2/logs$ cd ..
JohnGalt@MM-Dev-VM:~/.pm2$ cd ..
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var | sort -h | tail -n 10
4.0K    /var/metrics
4.0K    /var/opt
28K     /var/crash
40K     /var/spool
68K     /var/tmp
5.4M    /var/backups
49M     /var/cache
617M    /var/log
39G     /var
39G     /var/lib
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib | sort -h | tail -n 10
772K    /var/lib/systemd
2.0M    /var/lib/fwupd
3.8M    /var/lib/command-not-found
4.2M    /var/lib/aspell
7.5M    /var/lib/ubuntu-advantage
40M     /var/lib/swcatalog
92M     /var/lib/dpkg
276M    /var/lib/apt
38G     /var/lib/apport
39G     /var/lib
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib/apport | sort -h | tail -n 10
38G     /var/lib/apport
38G     /var/lib/apport/coredump
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib/apport/coredump | sort -h | tail -n 10
38G     /var/lib/apport/coredump
JohnGalt@MM-Dev-VM:~$

sdetweil

@JohnGalt next is who’s dump is it… electron?

JohnGalt

@sdetweil – true. I’m not sure I know to find that out…

JohnGalt

is that file [just] a record of the dump[s]?

Put differently, would it be safe to delete the file or directory?

[Edit]: On further recollection - this appears to be happening only on the two VMs running MM, and not on the two Raspberry Pis that I have MM running on.

Does this represent a clue as to what is happening?

sdetweil

@JohnGalt in the folder with the core dump files do

sudo file * >>~/somefile.txt

then ~/somefile.txt will have a line for each which should contain the failing code name

from
https://stackoverflow.com/questions/13308922/find-which-program-caused-a-core-dump-file

JohnGalt

@sdetweil

Well, there’s this under /var/lib/apport/coredump:

Name:
core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464

Size:
40,486,436,864

sdetweil

@JohnGalt well then… electron crashed…

which MM version are you on
cd ~/MagicMirror
grep version package.json

and for electron
grep version node_modules/electron/package.json

JohnGalt

@sdetweil

JohnGalt@MM-Dev-VM:~/MagicMirror$ grep version package.json
        "version": "2.35.0",
JohnGalt@MM-Dev-VM:~/MagicMirror$ grep version node_modules/electron/package.json
  "version": "41.2.0"

sdetweil

@JohnGalt if you hadn’t erased the logs, the pm2 log MIGHT have had a fault listed…

2.37 is the current release, 6 months newer… electron at 42.5.2

supposedly this could help find cause

 Analyze the Crash Dump
Because Electron code is pre-compiled, looking at a raw dump file yields only unreadable memory addresses. 
You need to symbolicate the dump using matching Electron debugging symbols.

The CLI Tool (electron-minidump)
    The easiest open-source way to inspect a .dmp file locally is using the electron-   minidump CLI tool. 
It automatically fetches the matching Electron symbols and outputs a readable stack trace.
Install the utility:
sudo  npm install -g electron-minidump

Analyze your file:
electron-minidump -f /path/to/your/file.dmp

JohnGalt

@sdetweil – OK, well… The program DID install (I was surprised , given what has been happening…). This resulted in an error. I specified that large file we identified earlier. Do let me know if I did the wrong thing. See results below.

While you are looking at this, I am going to fire up the other, older VM that also crashed and see if there is similar evidence there that might shed light.

JohnGalt@MM-Dev-VM:~/MagicMirror$ sudo  npm install -g electron-minidump
[sudo] password for JohnGalt:

added 29 packages in 34s

1 package is looking for funding
  run `npm fund` for details
npm notice
npm notice New major version of npm available! 10.9.4 -> 12.0.2
npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.0.2
npm notice To update run: npm install -g npm@12.0.2
npm notice
JohnGalt@MM-Dev-VM:~/MagicMirror$ electron-minidump -f /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
Error: EACCES: permission denied, open '/var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464'
    at async Object.open (node:internal/fs/promises:641:25)
    at async electronMinidump (/usr/local/lib/node_modules/electron-minidump/index.js:11:14) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464'
}
JohnGalt@MM-Dev-VM:~/MagicMirror$ sudo electron-minidump -f /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
Not a minidump file (MDMP header not found): /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
JohnGalt@MM-Dev-VM:~/MagicMirror$

sdetweil

@JohnGalt you need sudo in front of the mini dump. Line cause the file is in the /var folder