UNSOLVED Core dump causing black screen

mlcampbe

Does anyone have an idea how to debug a core dump caused by a module or MM itself?

What I am seeing is that occasionally my mirror goes black and I originally thought it was the screensaver kicking in. However, I believe it is due to a core dump occurring.

From the pm2 logs I see this:

ATTENTION: default value of option force_s3tc_enable overridden by environment.
[16831:0413/111826.964220:FATAL:memory.cc(22)] Out of memory. size=79556608
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
[11331:0415/142933.736368:FATAL:memory.cc(22)] Out of memory. size=120422400
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
getrandom indicates that the entropy pool has not been initialized. Rather than continue with poor entropy, this process will block until entropy is available.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
[1700:0415/233524.137869:FATAL:memory.cc(22)] Out of memory. size=96002048
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
[6522:0416/101829.273887:FATAL:memory.cc(22)] Out of memory. size=155766784
ATTENTION: default value of option force_s3tc_enable overridden by environment.

I am not concerned about the force_s3tc_enable but notice that there are several out of memory errors. Based on the timestamp of the generated core file I can correlate these out of memory errors to within a minute or 2 of core file.

So, in the above I have a core dump from:
4-13-2019 @ 11:18am
5-15/2019 @ 14:28pm
4-15-2019 @ 23:35pm
4-16-2019 @ 10:18am

There does not look to be a pattern here so not sure where to go other than disabled modules 1-by-1 and wait.

Given I have the core file is there a way to read it and determine at least which module caused the error as that would help me try to narrow it down.

sdetweil

@mlcampbe said in Core dump causing black screen:

6522:0416/101829.273887:FATAL:memory.cc(22)] Out of memory. size=155766784

it is out of memory…

mlcampbe

Yeah I realize it is out of memory but what module is causing that? I have a core dump and I should be able to use gdb to see the stack trace to try to figure it out.

sdetweil

@mlcampbe no idea… I just caused a core dump on mine by comparing a whole structure to a string variable… (not on purpose) it didn’t tell me which module caused it

bhepler

@mlcampbe - It definitely sounds like there is a memory leak somewhere. Personally, I blame electron. Basically out of spite, not any technical reason.

I would recommend that you add a cron job that will reboot your mirror at some absurd hour in the morning. It should reset the memory allocation and prevent OOM errors. It’s not a fix, but it should get you close enough.

mlcampbe

Yeah that is a good idea and I might go that way as a last resort if I can’t identify which module is the culprit.

This morning I discovered the MMM-Logging module which prints date/time info into the logs and I am hoping that might help me match the coredump time to which module was doing something.

I also discovered a potential issue with the MMM-DailyBibleVerse module. I see that it has a getScripts function that loads the jquery-3.1.1.min.js file from its module directory. I am not sure if that is getting loaded over and over and thus eventually running out of memory but for now have removed that module for testing.

sdetweil

@mlcampbe getscripts is only called once per module

sdetweil

@mlcampbe see this https://www.javascriptjanuary.com/blog/nodejs-postmortem-debugging-for-fun-and-production

u need ulimit -c set to some non-0 number (number of core dumps)

0 by default

mlcampbe

@sdetweil I am already getting core dumps generated so I have the file. I had see the nodejs postmortem debugging and I can get a stack trace from the core file but so far I have not been able to identify which module was active at the time from it.

Doing more research shows that this may be related to MMM-WallPaper which I am using. I found https://github.com/kolbyjack/MMM-Wallpaper/issues/3 that seems to match my symptoms exactly. I’m removing that module and will test it a few days and see what happens.