"Out of memory" issues - where do I begin?

ember1205

UPDATE: At this point, I’m closing in on a full week of operating two mirrors without issue. I’ve changed to running Chromium instead of Electron and it’s working well all around. Here are some details of the operation in case it’s useful:

Pi 3 Model B (one UK, one China)
MM 2.12.0
Native modules configured: WeatherForecast, CurrentWeather (both weather modules set to different locations as the two Pi’s are in different locations), clock, calendar (2 instances, 2 different Google calendars), alert, updatenotification
Third-party modules: MMM-ImagesPhotos (@sdetweil version)
Customizations:
- Modified the WeatherForecast module according to this thread to get 5 days of forecast data back after recent MM update: https://forum.magicmirror.builders/topic/13187/weatherforecast-showing-only-two-days/31?_=1601303807634
- Converted to using Chromium instead of Electron locally as detailed in this thread
- Chromium update / notice disabled via the step from a couple of posts prior to this one
- Allow connections from local LAN addresses
Pi connected to a smart plug that turns on each day at 7AM, cron job to execute “init 0” each day at 11PM, smart plug turns off at 11:05PM

Prior to successfully switching over to Chromium, the devices would crash at least once every other day (at least one of them would crash during a 48 hour window), although it was much more common to see each one crash multiple times per day. There was no consistency to which one would crash, why, when, etc. The “crash” in question was a black screen on the mirror and no visible info being output although the mouse cursor would sometimes appear. I wrote a cron job that would run every five minutes, look up the location of the pm2 error logs, check to see if there was an “out of memory” error in the log, and do a restart of MM if there was (the restart consisted of shut down, log flush, and start). This job ran every five minutes and would silently exit if there was no error as the mirror was running ok.

Since the switchover, I have seen zero crashes on either device in almost seven days of operation.

Electron has been discussed ad nauseum as having a variety of shortcomings, bugs, issues, etc. - especially in the old version being used with MM. At this point, I have to wonder why it’s still not only the default but the ‘only’ browser that’s really discussed for use with MM. It would seem that it’s time to either forklift an upgrade to it within MM or switch to something else (at a minimum, at least provide a well-documented alternative).

I’m grateful to @sdetweil for his assistance with this, especially since it really took a calendar year almost to get to a point where it’s seemingly working as expected now.

sdetweil

@ember1205 sometimes upgrading one part causes forced upgrades to others and the collection brings more problems…

ember1205

@sdetweil I can appreciate that - I work for a software company, and know about inter-dependencies and the headaches they can cause.

MM is, in a very basic sense, a web server. Electron is a web browser. There shouldn’t be inter-dependencies between a web server and browser, and that seems to be apparent with Chromium not having an issue running against the MM server. So, I would hope that Electron could be brought forward or at least Chromium be documented as an solid alternative.

sdetweil

@ember1205 I got us moved from V3 to V6… we need to add another cycle… it didn’t work with V7 last time.

ember1205

@sdetweil v10 is out and listed as “stable”.

sdetweil

@ember1205 yeh, that and a quarter!!.. V7 was stable when we tried it last year too… running on PI is a whole different thing… they test on x86.

ember1205

Update… 30 full days since my last post about stability and have not experienced a single crash on either mirror. So, Electron is definitely the source of the problem.

Thank you to @sdetweil for the assistance in swapping over to Chromium and getting things to a stable state.

sdetweil

@ember1205 thank you for the feedback… there is a proposal to move to Electron V10

ember1205

Wanted to post what is likely a ‘final update’ to this thread. I was running for literally weeks if not months with zero issues. I even let the system run for at least a month with the “you’re out of date” banner across the top and not one crash.

I updated the mirror and it reverted itself back to using Electron (that in itself is infuriating) and it’s crashing again. So, not only is Electron the problem but it’s STILL a problem.

I’ve reverted my mirror back to the outdated code and will do another update and not allow it to change back to Electron so that it can run without crashing!

sdetweil

@ember1205 what do you mean ‘updated’? if u just did a git pull, then u updated to the same level of electron

new release coming this week (1/1) which moves up to electron 8. 5.3

if u want to test it today, use the develop branch

both still use only electron, you can reinsert run-start.sh to use chromium

ember1205

@sdetweil I used your upgrade script on my mirror. During the upgrade, it reverted the configuration back to using Electron instead of retaining the run-start.sh directive that would launch Chromium.

I wiped out the entire MM directory and copied back the working one that wasn’t upgraded. Unfortunately, it now seems that it is refusing to run the upgrade script and I’m wondering if it’s a certificate issue and I need to update the Pi itself first…

sdetweil

@ember1205 the upgrade doesn’t know about your local change to use run-start… but all u had to do was restore that one line in package.json

it now seems that it is refusing to run the upgrade script

show me the ~/MagicMirror/installers/upgrade.log

ember1205

@sdetweil This is all that it spits out on a test run (and MM no longer shows that it is out of date but it is):

Upgrade started - Mon Dec 28 09:07:08 EST 2020
system is Linux raspberrypi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux
the os is Distributor ID: Raspbian Description: Raspbian GNU/Linux 10 (buster) Release: 10 Codename: buster
doing test run = true, NO updates will be applied!

saving custom.css
Unable to determine upstream git repository
restoring custom.css
removing git alias
Upgrade ended - Mon Dec 28 09:07:09 EST 2020

I tried connecting to a couple of web sites via CURL and it throws errors about loading certificates and these sites should work fine. So, the combo of everything is why I’m suspecting that there is a certificate issue due to an older OS on the Pi. I’m doing an upgrade on it now…

sdetweil

@ember1205 when u made your backup, how did u do it? you may not have backed up the git repo

from the MagicMirror folder
do a git status
and ls .git (notice the leading dot)

ember1205

@sdetweil I honestly don’t recall if I renamed the original folder MM-backup/ and then did rsync back to MagicMirror/ or just did rsync to MM-backup/. My rsync would have been with flags ‘-vaPHSx’, so either way it should have gotten everything.

I’ll check for the .git directory…

ember1205

I can’t seem to get the system to tell me that it’s out of date again (even though I know it is). That’s ok for now, though… New problem…

Chromium is crashing, and I have a guess as to what might be going on. I have been searching and can not find anything related to a log file for Chromium, though, and I would need to be able to see that to understand if my suspicion is correct. Am I just completely missing it?