Upgraded, failed back, now mirror constantly restarts

sdetweil

look for the messages output from npm start
if using pm2 then pm2 logs --lines will show them

restart likely means pm2, as thats its job. keep app running
you could have kept the new folder too…

i would have liked to see the upgrade log…

ember1205

Error on my part… I still have the upgraded directory.

There is no info from the logs that I can tell that shows any sort of crash or anything. And the line from the npm logs just shows the run-start.sh command.

sdetweil

@ember1205 what device is this? p0?

the upgrade log is MagicMirror/installers/upgrade.log

also show output of
uname -a

ember1205

pi 3B+

uname -a output

Linux raspberrypi 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux

How do I post the log? Far too large to go into a post, and isn’t a support file type to attach.

sdetweil

@ember1205 email to me…

same userid at gmail

ember1205

Email sent…

sdetweil

@ember1205 how did u restore the changes to package.json? the script stashed the old package.json file so that the upgrade could continue

ember1205

Umm… git stash pop?

I renamed the upgraded install to “MM-Upgraded” and then renamed my copied directory back to “MagicMirror”. Neither of them works properly at this point and the copied directory was never touched during the upgrade process.

sdetweil

@ember1205 ok, can’t do that… poackage.json is version specific

u might not need it as that run-start.sh was for older configs…
restore with
git checkout package.json

MMM-ImagesPhotos update was because of the missing request lib… removed in 2.16 (you were on 2.13)

I dont see anything else in the upgrade that should be problematic…

try with the default package.json

ember1205

Forgive me, but “can’t do” what, exactly?

I have reverted to the exact software, install, and configuration that was in place before running the upgrade by renaming the upgraded directory and then renaming the backed up directory back to the original name. Unless there are files downloaded and stored -outside- of the MagicMirror directory, this should have ensured a complete reversion to the working files.

sdetweil

@ember1205 but I see a current version of node and npm, which would not have been available in 2.13 timeframe…

the old install will not run on the new node and npm , and a new npm install on the old folder

ember1205

The device is restarting for what appears to be out of memory / swap space issues. Every time the system crashes and restarts, /dev/shm is at zero available space.

This location appears to be where the processes are storing content from the image files that are being loaded and it’s apparently causing something to overrun the available space. I have added images, but that was a couple of weeks ago (before even applying the OS updates) and everything ran perfectly fine.

sdetweil

@ember1205 df -k will show you what is used

the top looks like this

Filesystem      1K-blocks       Used  Available Use% Mounted on
udev             24555800          0   24555800   0% /dev
tmpfs             4931772       4108    4927664   1% /run
/dev/nvme0n1p1 1921217360  889131676  944193092  49% /
tmpfs            24658844    1769768   22889076   8% /dev/shm
tmpfs                5120          4       5116   1% /run/lock
tmpfs            24658844          0   24658844   0% /sys/fs/cgroup

with appropriate names for your system

free -m will show u memory and swap space

ember1205

Pretty sure it’s an out of memory issue. Once it crashes, the free memory jumps way up then trends back down once it restarts.

I’ve also discovered that this seems to be purely a “client” problem as I can connect to the Pi from a remote machine using a web browser and it will run just fine without any hiccups while the local client on the Pi crashes about every 60-70 seconds which correlates to two load intervals for photos. Maybe the MMM-ImagesPhotos module isn’t releasing memory properly?

ember1205

After a LOT of different attempts at upgrades, fail-backs, new installs, etc., I believe I finally corrected the issue…

It appears that PM2 was either somehow corrupted during an update attempt or just simply too far out of date. Attempting to run PM2 with certain command options (like stop or even status) was resulting in errors indicating some sort of circular reference. After doing a forced upgrade to the latest version, it stopped showing these errors and the mirror is to running as expected.

I did make some additional changes that include copying the core MM directory off to another linux host where I now run the mirror as a server. The RPi is now purely a client and it loads noticeably quicker as a result.

sdetweil

@ember1205 thx. I will add checking pm2 to the upgrade script