"Out of memory" issues - where do I begin?

[sdetweil]

@ember1205 i spent a year debugging an out of memory that only crashed one version of a module… another ran fine…

one line of code, my code, made the difference… but the problem ‘appeared’ to be a hang , suddenly the UI was dead…
randomnly displaying pics in the background.

not til I added a totally new logging approach did I find the actual problem, which was out of memory, which killed the UI, which made it LOOK hung…

[ember1205]

In my situation, the UI goes all black and the cursor appears (no mouse or keyboard connected). There’s a generic “out of memory” error entered into the log file. Restarting MM is the only way I’ve found to get it back up and running.

The photo module I’m running is your fork with the more recent changes to allow the blurring of the same image behind the full-view image. Any thoughts on anything that could/should be added to try and capture more in the logs? Is there an increased logging level I could be using in MM itself? Stack traces when it fails? Before it fails? Anything come to mind that might give me a new avenue?

The issue doesn’t seem to appear if I am not using the photo module. But, that could be because the other, more basic modules don’t use resources as intensively. We’ve compared notes on RPi, kernel, build, etc. and not found anything that seems to be materially different with the one exception being the specific image files in use.

[sdetweil]

@ember1205 i run that module every day all day and don’t see any problem, but… try this

edit MMM-ImagesPhotos.js

add the line shown below

      					img.style.top = result.targettop+"px";                    // line 271
                                        img=null      /// add this line
					// if another image was already displayed
					let c = self.fg.childElementCount;

[ember1205]

should it be:

img=null;

???

[sdetweil]

@ember1205 trailing ; is ok either way

[ember1205]

Updated and restarted the MM process. I will keep the three browsers hitting it and wait and see if it errors out again. Sometimes, this can literally take days while other times it can occur multiple times in an hour.

[sdetweil]

@ember1205 each browser is loading the images independantly… the server side just returns the list of images once for each

[ember1205]

So, the extra “stress” isn’t actually stress on the Pi at all?

[sdetweil]

@ember1205 well, the file system loading of the images will be thru the pi, but nothing else…all the display side is done in the browser (which is where it ran out of memory)

[ember1205]

Well, it isn’t a LOT of progress, but it’s progress.

It’s still crashing. But I managed to get a core dump. And it says that it’s Electron that’s crashing. The initial output from bt has me a little confused as I can’t tell if it didn’t load the symbol file correctly, or if the info just “unknown”. Here’s what I get from a bt command:

Core was generated by `/home/pi/MagicMirror/node_modules/electron/dist/electron --type=renderer --auto'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x02760514 in ?? ()
[Current thread is 1 (LWP 6298)]
(gdb) bt
#0  0x02760514 in ?? ()
#1  0x027603d8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Not really sure if there’s anywhere to go from here. :/

[sdetweil]

@ember1205 but the other browsers kept going?

i don’t think the symbols are available,certainly not for modules, if that is where the problem is
I would add MMM-Logging (my version) which merges the logs from browser and node_helpers into one log

https://github.com/sdetweil/MMM-Logging

[ember1205]

Hmmm. Not sure about other browsers since I had shut those down. I’ll wait until it crashes again and see if a secondary browser will keep running.

[ember1205]

I have confirmed that only Electron crashes on the Pi itself. The MM server does not experience any hiccups and remote browser connections remain functional.

I at least now understand WHAT is crashing. Next steps might be to figure out WHY it’s crashing and potentially remedy it.

[sdetweil]

@ember1205 note that we are trailing release of electron quite significantly (we are on 6.1.7 and latest is 9.x.x) … you might try moving up to a different version, no guaranty that MM will run tho… or what the other impacts are…

as i said back a few, with a minor startup change, you can get chromium to be the viewer on the pi.

[ember1205]

I’ve made the change to Chromium - we’ll see how that does going forward.

I do recall reading plenty about the severe lag of Electron in terms of the version that’s part of MM, so that part wasn’t “news” really. What I genuinely don’t understand is why this specific, basic configuration continues to have issues for me while Electron mostly does what it needs to do for so many others without creating this sort of problem.

[sdetweil]

@ember1205 electron is a ‘single’ threaded app… but events happen that cause it to need to act like mutltiple threads.
there are lots of edge cases where the timing is problematic…

its not out of memory, its lost its way…

i wrote a plugin/module to display pics matches with cal entries, floating above the mirror screen… (pong like moving boxes)
electron does NOT like two windows being manipulated at the same time… took a long time to find that, always an out of memory error or hang… but not

[ember1205]

Good info.

I’m interested to see how for the MM will make it now using Chromium instead of Electron. Since I have two with the same setup, I’ll set both to use Chromium for an additional level of testing it out.

Why was Electron chosen as the browser up to this point? And, what potential downsides are there to Chromium?

[sdetweil]

@ember1205 >Why was Electron chosen as the browser up to this point? And, what potential downsides are there to Chromium?

i don’t know and i don’t know… was way before my time.

[ember1205]

Welp…

No change. System still crashes on the Pi using Chromium.

In looking at the most recent core file, it’s still showing that electron is what’s faulting.

There are no running processes of chromium and electron is still being started. Something is amiss with the configuration to where your run-start.sh script is either not being called or is not launching the correct browser for some reason.

I’ve made no changes to run-start.sh, I’ve edited package.json so that this line is gone:

"start": "DISPLAY=\"${DISPLAY:=:0}\" ./node_modules/.bin/electron js/electron.js",

And has been replaced with this one:

"start": "./run-start.sh",

config/config.js has the following setting:

var config = {
	address: "0.0.0.0", 	// Address to listen on, can be:
							// - "localhost", "127.0.0.1", "::1" to listen on loopback interfac
e
							// - another specific IPv4/6 to listen on a specific interface
							// - "0.0.0.0", "::" to listen on any interface
							// Default, when address config is left out or empty, is "localhost
"
	port: 8080,
	basePath: "/", 	// The URL path where MagicMirror is hosted. If you are using a Reverse proxy
					// you must set the sub path here. basePath must end with a /
	ipWhitelist: ["127.0.0.1", "192.168.192.0/24", "::1"], 	// Set [] to allow all IP addresses
															// 
or add a specific IPv4 of 192.168.1.5 :
															// 
["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.1.5"],
															// 
or IPv4 range of 192.168.3.0 --> 192.168.3.15 use CIDR format :
															// 
["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.3.0/28"],

	useHttps: false, 		// Support HTTPS or not, default "false" will use HTTP
	httpsPrivateKey: "", 	// HTTPS private key path, only require when useHttps is true
	httpsCertificate: "", 	// HTTPS Certificate path, only require when useHttps is true

	language: "en",
	logLevel: ["INFO", "LOG", "WARN", "ERROR"],
	timeFormat: 24,
	units: "metric",
	// serverOnly:  true/false/"local" ,
	serverOnly:"local",

[ember1205]

One tweak to the run-start.sh script seems to have fixed it. I changed this line:

serveronly=$(grep -v '^[[:blank:]]*//'  config/config.js | grep -i serveronly: | awk '{print tolower($2)}' | tr -d ,\"\'\\r)

To this:

serveronly=$(grep -v '^[[:blank:]]*//'  config/config.js | grep -i serveronly: | awk '{print tolower($1)}' | tr -d ,\"\'\\r)

The awk command seems to have been looking for a variable that wasn’t being passed and by changing the variable number that it was looking for, it no longer starts electron and now starts chromium.