"Out of memory" issues - where do I begin?

ember1205

I have confirmed that only Electron crashes on the Pi itself. The MM server does not experience any hiccups and remote browser connections remain functional.

I at least now understand WHAT is crashing. Next steps might be to figure out WHY it’s crashing and potentially remedy it.

sdetweil

@ember1205 note that we are trailing release of electron quite significantly (we are on 6.1.7 and latest is 9.x.x) … you might try moving up to a different version, no guaranty that MM will run tho… or what the other impacts are…

as i said back a few, with a minor startup change, you can get chromium to be the viewer on the pi.

ember1205

I’ve made the change to Chromium - we’ll see how that does going forward.

I do recall reading plenty about the severe lag of Electron in terms of the version that’s part of MM, so that part wasn’t “news” really. What I genuinely don’t understand is why this specific, basic configuration continues to have issues for me while Electron mostly does what it needs to do for so many others without creating this sort of problem.

sdetweil

@ember1205 electron is a ‘single’ threaded app… but events happen that cause it to need to act like mutltiple threads.
there are lots of edge cases where the timing is problematic…

its not out of memory, its lost its way…

i wrote a plugin/module to display pics matches with cal entries, floating above the mirror screen… (pong like moving boxes)
electron does NOT like two windows being manipulated at the same time… took a long time to find that, always an out of memory error or hang… but not

ember1205

Good info.

I’m interested to see how for the MM will make it now using Chromium instead of Electron. Since I have two with the same setup, I’ll set both to use Chromium for an additional level of testing it out.

Why was Electron chosen as the browser up to this point? And, what potential downsides are there to Chromium?

sdetweil

@ember1205 >Why was Electron chosen as the browser up to this point? And, what potential downsides are there to Chromium?

i don’t know and i don’t know… was way before my time.

ember1205

Welp…

No change. System still crashes on the Pi using Chromium.

In looking at the most recent core file, it’s still showing that electron is what’s faulting.

There are no running processes of chromium and electron is still being started. Something is amiss with the configuration to where your run-start.sh script is either not being called or is not launching the correct browser for some reason.

I’ve made no changes to run-start.sh, I’ve edited package.json so that this line is gone:

"start": "DISPLAY=\"${DISPLAY:=:0}\" ./node_modules/.bin/electron js/electron.js",

And has been replaced with this one:

"start": "./run-start.sh",

config/config.js has the following setting:

var config = {
	address: "0.0.0.0", 	// Address to listen on, can be:
							// - "localhost", "127.0.0.1", "::1" to listen on loopback interfac
e
							// - another specific IPv4/6 to listen on a specific interface
							// - "0.0.0.0", "::" to listen on any interface
							// Default, when address config is left out or empty, is "localhost
"
	port: 8080,
	basePath: "/", 	// The URL path where MagicMirror is hosted. If you are using a Reverse proxy
					// you must set the sub path here. basePath must end with a /
	ipWhitelist: ["127.0.0.1", "192.168.192.0/24", "::1"], 	// Set [] to allow all IP addresses
															// 
or add a specific IPv4 of 192.168.1.5 :
															// 
["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.1.5"],
															// 
or IPv4 range of 192.168.3.0 --> 192.168.3.15 use CIDR format :
															// 
["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.3.0/28"],

	useHttps: false, 		// Support HTTPS or not, default "false" will use HTTP
	httpsPrivateKey: "", 	// HTTPS private key path, only require when useHttps is true
	httpsCertificate: "", 	// HTTPS Certificate path, only require when useHttps is true

	language: "en",
	logLevel: ["INFO", "LOG", "WARN", "ERROR"],
	timeFormat: 24,
	units: "metric",
	// serverOnly:  true/false/"local" ,
	serverOnly:"local",

ember1205

One tweak to the run-start.sh script seems to have fixed it. I changed this line:

serveronly=$(grep -v '^[[:blank:]]*//'  config/config.js | grep -i serveronly: | awk '{print tolower($2)}' | tr -d ,\"\'\\r)

To this:

serveronly=$(grep -v '^[[:blank:]]*//'  config/config.js | grep -i serveronly: | awk '{print tolower($1)}' | tr -d ,\"\'\\r)

The awk command seems to have been looking for a variable that wasn’t being passed and by changing the variable number that it was looking for, it no longer starts electron and now starts chromium.

sdetweil

@ember1205 hm… do this

cd ~/MagicMirror
./run-start.sh

you should see something like this

 ./run-start.sh 
[2020-09-22 13:13:51.584] [LOG] Starting MagicMirror: v2.13.0-develop
[2020-09-22 13:13:51.588] [LOG] Loading config ...
[2020-09-22 13:13:51.590] [LOG] Loading module helpers ...
[2020-09-22 13:13:51.590] [LOG] No helper found for module: alert.
[2020-09-22 13:13:51.602] [LOG] Initializing new module helper ...
[2020-09-22 13:13:51.602] [LOG] Module helper loaded: updatenotification
[2020-09-22 13:13:51.603] [LOG] No helper found for module: clock.
[2020-09-22 13:13:51.840] [LOG] Initializing new module helper ...
[2020-09-22 13:13:51.840] [LOG] Module helper loaded: calendar
[2020-09-22 13:13:51.841] [LOG] No helper found for module: compliments.
[2020-09-22 13:13:51.841] [LOG] No helper found for module: currentweather.
[2020-09-22 13:13:51.841] [LOG] No helper found for module: weatherforecast.
[2020-09-22 13:13:52.147] [LOG] Initializing new module helper ...
[2020-09-22 13:13:52.147] [LOG] Check MagicMirror version for node helper 'MMM-Ring' - Minimum version: 2.11.0 - Current version: 2.13.0-develop
[2020-09-22 13:13:52.147] [LOG] Version is ok!
[2020-09-22 13:13:52.148] [LOG] Module helper loaded: MMM-Ring
[2020-09-22 13:13:52.148] [LOG] All module helpers loaded.
[2020-09-22 13:13:52.186] [LOG] Starting server on port 8090 ...
[2020-09-22 13:13:52.187] [INFO] You're using a full whitelist configuration to allow for all IPs
[2020-09-22 13:13:52.190] [LOG] Server started ...
[2020-09-22 13:13:52.190] [LOG] Connecting socket for: updatenotification
[2020-09-22 13:13:52.190] [LOG] Connecting socket for: calendar
[2020-09-22 13:13:52.190] [LOG] Starting node helper for: calendar
[2020-09-22 13:13:52.191] [LOG] Connecting socket for: MMM-Ring
[2020-09-22 13:13:52.196] [LOG] MMM-Ring at (9/22/2020, 1:13:52 PM): Starting module: MMM-Ring
[2020-09-22 13:13:52.196] [LOG] Sockets connected & modules started ...
[2020-09-22 13:13:52.196] [LOG]
Ready to go! Please point your browser to: http://0.0.0.0:8090
Starting chromium browser now, have patience, it takes a minute

that last line is the key to running in split mode, server with chromium over the top…

to run it multiple times, you will have to kill the server process

 ps -ef | grep server

sam 3267 3266 1 13:13 pts/0 00:00:01 node serveronly

kill -9 3267

thx for the hint, the space after the serverOnly:

sdetweil

@ember1205 I just updated the script to handle with/without the space…

ember1205

Cool. I download the latest one and try it out.

Glad I can help you keep improving your code! lol

sdetweil

@ember1205 said in "Out of memory" issues - where do I begin?:

Glad I can help you keep improving your code! lol

me too… thought I had handled that… guess not

ember1205

Based on the directions you gave in this thread, the run-start.sh script won’t work. I didn’t take the time to look through the github content to see if this is covered there, but you have to do

chmod +x run-start.sh

Or the script won’t actually execute. Based on old habits, I was executing the chmod statement without even thinking about it. But, after downloading the newest script, I purposefully left it untouched and it wouldn’t run.

sdetweil

@ember1205 hm, thought it was uploaded with exec. will check this evening. have doctor appt shortly

thanks for the info

ember1205

I simply opened the github page, copied the link, and used wget to download it to the mirror. So, it’s entirely possible that the way I downloaded it was partly to blame. Figured I would mention it because I’m likely not the only one that might run into this, and at a minimum it could become a troubleshooting step if others run into issues.

sdetweil

@ember1205 thx. wget & curl won’t copy the execute permissions. when I install this on a pi 0 I chmod it to executable

ember1205

You can’t make this stuff up…

Mirror has been running for a couple of hours now without crashing as it has been doing intermittently in the past. However, there is now a “pop up” on the screen stating that Chromium couldn’t be updated and that I’m missing out on features and fixes.

I found what might be a fix in the way of potentially “configuring” Chromium to ignore updates, but I won’t know if this truly works until probably tomorrow at the earliest.

ember1205

I’m now about 2.5 days into two MM’s running without crashes using Chromium instead of Electron. I’ve also not seen a return of the pop up from Chromium about needing to be updated.

For those that might need to correct the update issue, here’ s what I did on my mirrors:

sudo touch /etc/chromium-browser/customizations/01-disable-update-check;echo CHROMIUM_FLAGS=\"\$\{CHROMIUM_FLAGS\} --check-for-update-interval=31536000\" | sudo tee /etc/chromium-browser/customizations/01-disable-update-check

sdetweil

@ember1205 cool. thx for that info. I will add it to my install script for pi0!!

ember1205

UPDATE: At this point, I’m closing in on a full week of operating two mirrors without issue. I’ve changed to running Chromium instead of Electron and it’s working well all around. Here are some details of the operation in case it’s useful:

Pi 3 Model B (one UK, one China)
MM 2.12.0
Native modules configured: WeatherForecast, CurrentWeather (both weather modules set to different locations as the two Pi’s are in different locations), clock, calendar (2 instances, 2 different Google calendars), alert, updatenotification
Third-party modules: MMM-ImagesPhotos (@sdetweil version)
Customizations:
- Modified the WeatherForecast module according to this thread to get 5 days of forecast data back after recent MM update: https://forum.magicmirror.builders/topic/13187/weatherforecast-showing-only-two-days/31?_=1601303807634
- Converted to using Chromium instead of Electron locally as detailed in this thread
- Chromium update / notice disabled via the step from a couple of posts prior to this one
- Allow connections from local LAN addresses
Pi connected to a smart plug that turns on each day at 7AM, cron job to execute “init 0” each day at 11PM, smart plug turns off at 11:05PM

Prior to successfully switching over to Chromium, the devices would crash at least once every other day (at least one of them would crash during a 48 hour window), although it was much more common to see each one crash multiple times per day. There was no consistency to which one would crash, why, when, etc. The “crash” in question was a black screen on the mirror and no visible info being output although the mouse cursor would sometimes appear. I wrote a cron job that would run every five minutes, look up the location of the pm2 error logs, check to see if there was an “out of memory” error in the log, and do a restart of MM if there was (the restart consisted of shut down, log flush, and start). This job ran every five minutes and would silently exit if there was no error as the mirror was running ok.

Since the switchover, I have seen zero crashes on either device in almost seven days of operation.

Electron has been discussed ad nauseum as having a variety of shortcomings, bugs, issues, etc. - especially in the old version being used with MM. At this point, I have to wonder why it’s still not only the default but the ‘only’ browser that’s really discussed for use with MM. It would seem that it’s time to either forklift an upgrade to it within MM or switch to something else (at a minimum, at least provide a well-documented alternative).

I’m grateful to @sdetweil for his assistance with this, especially since it really took a calendar year almost to get to a point where it’s seemingly working as expected now.