"Out of memory" issues - where do I begin?

[ember1205]

I used an iterative command for this because there are so many photos:

cd ~MagicMirror
find < path to images >/* -exec node_modules/.bin/electron file:///{} \;

For each file in that directory, it would launch the electron browser and open the image. Because electron continues to run until it is terminated, this puts one image at a time on the screen. After each one loaded, I would simply click “X” to close the browser and it would move on to the next image.

While I -did- see one specific image file appear to load but not actually show any content (initially), I believe this may have been an issue with the fact that I walked away from the process for about ten minutes. Expanding the browser window caused it to re-render and it displayed fine. Additionally, this is an image that I have personally seen on the screen many times with no apparent link to any crashes.

It seems that any image-specific issues are possibly cumulative since loading each one in turn starts with a fresh memory allocation as the browser exits after each image.

[ember1205]

Thought I would update this…

Still broken.

I have the latest MM as a fresh install on a brand new RPi with a new power adapter, using a brand new SDCard, connected to a different monitor, using a different HDMI cable, and it’s all plugged in to an outlet at an entirely different house in a different area of the state (different sections of the grid). I have the same few, basic modules installed and have added the photos module discussed here. It crashes and hangs randomly with “out of memory” errors just like the other one.

The only thing the same between the two is the photo collection. I’ve not found any consistency in the past on it failing with any specific photos so troubleshooting this seems like it’s going to be impossible.

[sdetweil]

@ember1205 debugging is indeed very difficult w these problems.

I wonder if running w chromium browser would produce different results.

easiest way to test
download the run-start.sh script from here
https://github.com/sdetweil/MagicMirror_scripts
into the MagicMirror folder
edit the package.json and change the start command to

"start":"./run-start.sh",

edit config.js
and set

 serverOnly:"local",

this will start mm in server mode, and use chromium to display instead of electron

[ember1205]

Would that be any different than running it in server mode and accessing it from a browser on my computer across the network? That would be simpler to set up and wouldn’t add more running software onto the Pi…

[sdetweil]

@ember1205 same, no new software on the pi. mm install installs everything.

can run as
full ---- how it is now
server only
server only w different browser ( already installed)
or
client only

[ember1205]

Understood that there isn’t new software installed, but this would mean no additional software actively running on the Pi (Electron would be stopped, and all browser software would be running from a different machine).

[sdetweil]

@ember1205 right. that doesn’t mean u fixed the out of memory problem. it’s possible.

I was working on one change at a time.

u can install the run-start.sh and change config.js to serverOnly: true, then launch browser from your pc

[ember1205]

Your suggestion incorporates multiple changes, though… Stopping Electron and Starting Chromium while also changing the operating mode. No?

I’ve changed the config.js to both

serverOnly: true,

and

serverOnly:"true",

and neither one prevents Electron from starting.

I’ve now also tried

serverOnly:true,

[sdetweil]

@ember1205 u need the run-start.sh script for that parm to work

besides, u said u were going to run in server only mode anyway. and then u will have to change your startup process too

[ember1205]

By adding an external browser, it crashed in only a few minutes. I’m going to let it keep running with the additional external browser to see if it causes it crash much more quickly on a more continuous basis.

If I can cause it to crash quickly on a repeated basis, it gives me an opportunity to try testing specific image files. It’s possible that this is a cumulative problem that is related to a small memory loss that gets added to on the load of additional errant files.

UPDATE: So much for that theory… I am currently running Electron plus an external instance of Chrome on Windows and Safari on MacOS. No “quick crashes.”

[sdetweil]

@ember1205 i spent a year debugging an out of memory that only crashed one version of a module… another ran fine…

one line of code, my code, made the difference… but the problem ‘appeared’ to be a hang , suddenly the UI was dead…
randomnly displaying pics in the background.

not til I added a totally new logging approach did I find the actual problem, which was out of memory, which killed the UI, which made it LOOK hung…

[ember1205]

In my situation, the UI goes all black and the cursor appears (no mouse or keyboard connected). There’s a generic “out of memory” error entered into the log file. Restarting MM is the only way I’ve found to get it back up and running.

The photo module I’m running is your fork with the more recent changes to allow the blurring of the same image behind the full-view image. Any thoughts on anything that could/should be added to try and capture more in the logs? Is there an increased logging level I could be using in MM itself? Stack traces when it fails? Before it fails? Anything come to mind that might give me a new avenue?

The issue doesn’t seem to appear if I am not using the photo module. But, that could be because the other, more basic modules don’t use resources as intensively. We’ve compared notes on RPi, kernel, build, etc. and not found anything that seems to be materially different with the one exception being the specific image files in use.

[sdetweil]

@ember1205 i run that module every day all day and don’t see any problem, but… try this

edit MMM-ImagesPhotos.js

add the line shown below

      					img.style.top = result.targettop+"px";                    // line 271
                                        img=null      /// add this line
					// if another image was already displayed
					let c = self.fg.childElementCount;

[ember1205]

should it be:

img=null;

???

[sdetweil]

@ember1205 trailing ; is ok either way

[ember1205]

Updated and restarted the MM process. I will keep the three browsers hitting it and wait and see if it errors out again. Sometimes, this can literally take days while other times it can occur multiple times in an hour.

[sdetweil]

@ember1205 each browser is loading the images independantly… the server side just returns the list of images once for each

[ember1205]

So, the extra “stress” isn’t actually stress on the Pi at all?

[sdetweil]

@ember1205 well, the file system loading of the images will be thru the pi, but nothing else…all the display side is done in the browser (which is where it ran out of memory)

[ember1205]

Well, it isn’t a LOT of progress, but it’s progress.

It’s still crashing. But I managed to get a core dump. And it says that it’s Electron that’s crashing. The initial output from bt has me a little confused as I can’t tell if it didn’t load the symbol file correctly, or if the info just “unknown”. Here’s what I get from a bt command:

Core was generated by `/home/pi/MagicMirror/node_modules/electron/dist/electron --type=renderer --auto'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x02760514 in ?? ()
[Current thread is 1 (LWP 6298)]
(gdb) bt
#0  0x02760514 in ?? ()
#1  0x027603d8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Not really sure if there’s anywhere to go from here. :/