VM Disk Full - 50 GiB in 4 months

[JohnGalt]

Query: What would cause a MagicMirror installed on a VM to fill up the 50 GiB VM virtual disk over 4 months?

I have an instance of MagicMirror installed in a Linux VM running under TrueNAS, which is mostly used to try out modules, configurations or layouts without disturbing the main MagicMirror. When I went to look at it yesterday I found the VM running, but not MM. PM2 status did not show it running, and it would [re]start. MagicMirror is the only thing installed on the VM, and running modules were newsfeed and default weather [5 instances],

Some searching tells me the virtual disk of 50 GiB is full! I have expanded the virtual disk size [resized the zvol], but am unable to extend the partition and filesystem as I get an error. I think it was along the lines of “you don’t have enough free space in /var/cache/apt/archives”.

At this point it appeared the best route would be to just build a new VM and install MM, but when I went to start making notes to record what I was doing - I found this same thing happened some months ago. The current VM is an upsized VM that similarly locked up.

Since starting over will not apparently solve the underlying root cause, I need some help. Thanks in advance.

[sdetweil]

@JohnGalt from MagicMirror the only thing could be the output of the console log. If debugging is enabled ( logLevel:[ “debug”]), calendar can generate a bunch of output

If this is a vm, logon and look at the

.pm2/logs folder

Get the file sizes

ls -laF

[JohnGalt]

@sdetweil – Thanks Sam. See below for the output of the file sizes there.

JohnGalt@MM-Dev-VM:~/.pm2/logs$ ls -laF
total 32988
drwxrwxr-x 2 al al     4096 Apr 12 11:24 ./
drwxrwxr-x 5 al al     4096 Aug 19 12:09 ../
-rw-rw-r-- 1 al al 27348992 Aug 17 23:33 MagicMirror-error.log
-rw-rw-r-- 1 al al  6413424 Aug 17 23:27 MagicMirror-out.log

[sdetweil]

@JohnGalt said:

27348992

error log is 27 meg… not gig

after that you would have to use something like ncdu

ncdu (NCurses Disk Usage) is a fast, text-based command-line tool used to analyze disk space.
It scans your file system and shows which folders and files use the most space, letting you easily navigate and delete large items directly inside an interactive terminal screen.
How to Install ncdu
Ubuntu / Debian: sudo apt install ncdu
CentOS / RHEL / Fedora: sudo dnf install ncdu or sudo yum install ncdu
macOS: brew install ncdu
Arch Linux: sudo pacman -S ncdu

Common Commands
ncdu — Scan the current folder.  sudo ncdu / --Scan your entire hard drive.
ncdu /home --Scan a specific folder.  
ncdu -x --Scan only the current disk (ignore mounted external drives).
Useful Keyboard Shortcuts
Up / Down arrows or j / k — Move through the list.
Enter / Right arrow — Open a folder to see files inside.
Left arrow — Go back to the parent folder.
d — Delete the highlighted file or folder.

[JohnGalt]

@sdetweil – Thanks, Sam. It looks like that won’t work - for the same reason as above. I need to give this some more thought… I can’t be the first person to encounter this issue. See below for the output of trying to install ncdu:

JohnGalt@MM-Dev-VM:~/.pm2/logs$ sudo apt install ncdu
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  ncdu
0 upgraded, 1 newly installed, 0 to remove and 33 not upgraded.
Need to get 50.7 kB of archives.
After this operation, 117 kB of additional disk space will be used.
E: You don't have enough free space in /var/cache/apt/archives/.
JohnGalt@MM-Dev-VM:~/.pm2/logs$

[sdetweil]

@JohnGalt erase the files in the .pm2/logs folder, they are not the cause

[JohnGalt]

@sdetweil – I will do that. I was just down the rabbit hole, using df. It looks like a huge amount is taken up by /var/lib/apport/coredump. See below:

JohnGalt@MM-Dev-VM:~/.pm2/logs$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           794M  1.1M  793M   1% /run
/dev/sda2        49G   49G     0 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs         56K   27K   25K  52% /sys/firmware/efi/efivars
/dev/sda1       300M  7.8M  292M   3% /boot/efi
tmpfs           794M   92K  794M   1% /run/user/1000
JohnGalt@MM-Dev-VM:~/.pm2/logs$ cd ..
JohnGalt@MM-Dev-VM:~/.pm2$ cd ..
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var | sort -h | tail -n 10
4.0K    /var/metrics
4.0K    /var/opt
28K     /var/crash
40K     /var/spool
68K     /var/tmp
5.4M    /var/backups
49M     /var/cache
617M    /var/log
39G     /var
39G     /var/lib
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib | sort -h | tail -n 10
772K    /var/lib/systemd
2.0M    /var/lib/fwupd
3.8M    /var/lib/command-not-found
4.2M    /var/lib/aspell
7.5M    /var/lib/ubuntu-advantage
40M     /var/lib/swcatalog
92M     /var/lib/dpkg
276M    /var/lib/apt
38G     /var/lib/apport
39G     /var/lib
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib/apport | sort -h | tail -n 10
38G     /var/lib/apport
38G     /var/lib/apport/coredump
JohnGalt@MM-Dev-VM:~$ sudo du -h --max-depth=1 /var/lib/apport/coredump | sort -h | tail -n 10
38G     /var/lib/apport/coredump
JohnGalt@MM-Dev-VM:~$

[sdetweil]

@JohnGalt next is who’s dump is it… electron?

[JohnGalt]

@sdetweil – true. I’m not sure I know to find that out…

[JohnGalt]

is that file [just] a record of the dump[s]?

Put differently, would it be safe to delete the file or directory?

[Edit]: On further recollection - this appears to be happening only on the two VMs running MM, and not on the two Raspberry Pis that I have MM running on.

Does this represent a clue as to what is happening?

[sdetweil]

@JohnGalt in the folder with the core dump files do

sudo file * >>~/somefile.txt

then ~/somefile.txt will have a line for each which should contain the failing code name

from
https://stackoverflow.com/questions/13308922/find-which-program-caused-a-core-dump-file

[JohnGalt]

@sdetweil

Well, there’s this under /var/lib/apport/coredump:

Name:
core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464

Size:
40,486,436,864

[sdetweil]

@JohnGalt well then… electron crashed…

which MM version are you on
cd ~/MagicMirror
grep version package.json

and for electron
grep version node_modules/electron/package.json

[JohnGalt]

@sdetweil

JohnGalt@MM-Dev-VM:~/MagicMirror$ grep version package.json
        "version": "2.35.0",
JohnGalt@MM-Dev-VM:~/MagicMirror$ grep version node_modules/electron/package.json
  "version": "41.2.0"

[sdetweil]

@JohnGalt if you hadn’t erased the logs, the pm2 log MIGHT have had a fault listed…

2.37 is the current release, 6 months newer… electron at 42.5.2

supposedly this could help find cause

 Analyze the Crash Dump
Because Electron code is pre-compiled, looking at a raw dump file yields only unreadable memory addresses. 
You need to symbolicate the dump using matching Electron debugging symbols.

The CLI Tool (electron-minidump)
    The easiest open-source way to inspect a .dmp file locally is using the electron-   minidump CLI tool. 
It automatically fetches the matching Electron symbols and outputs a readable stack trace.
Install the utility:
sudo  npm install -g electron-minidump

Analyze your file:
electron-minidump -f /path/to/your/file.dmp

[JohnGalt]

@sdetweil – OK, well… The program DID install (I was surprised , given what has been happening…). This resulted in an error. I specified that large file we identified earlier. Do let me know if I did the wrong thing. See results below.

While you are looking at this, I am going to fire up the other, older VM that also crashed and see if there is similar evidence there that might shed light.

JohnGalt@MM-Dev-VM:~/MagicMirror$ sudo  npm install -g electron-minidump
[sudo] password for JohnGalt:

added 29 packages in 34s

1 package is looking for funding
  run `npm fund` for details
npm notice
npm notice New major version of npm available! 10.9.4 -> 12.0.2
npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.0.2
npm notice To update run: npm install -g npm@12.0.2
npm notice
JohnGalt@MM-Dev-VM:~/MagicMirror$ electron-minidump -f /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
Error: EACCES: permission denied, open '/var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464'
    at async Object.open (node:internal/fs/promises:641:25)
    at async electronMinidump (/usr/local/lib/node_modules/electron-minidump/index.js:11:14) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464'
}
JohnGalt@MM-Dev-VM:~/MagicMirror$ sudo electron-minidump -f /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
Not a minidump file (MDMP header not found): /var/lib/apport/coredump/core._home_al_MagicMirror_node_modules_electron_dist_electron.1000.f1d1264a-f2d2-4a44-8b27-e2f3182e030f.380943.121091464
JohnGalt@MM-Dev-VM:~/MagicMirror$

[sdetweil]

@JohnGalt you need sudo in front of the mini dump. Line cause the file is in the /var folder