Does anyone have an idea how to debug a core dump caused by a module or MM itself?

What I am seeing is that occasionally my mirror goes black and I originally thought it was the screensaver kicking in. However, I believe it is due to a core dump occurring.

From the pm2 logs I see this:

ATTENTION: default value of option force_s3tc_enable overridden by environment.
[16831:0413/111826.964220:FATAL:memory.cc(22)] Out of memory. size=79556608
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
[11331:0415/142933.736368:FATAL:memory.cc(22)] Out of memory. size=120422400
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
getrandom indicates that the entropy pool has not been initialized. Rather than continue with poor entropy, this process will block until entropy is available.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
[1700:0415/233524.137869:FATAL:memory.cc(22)] Out of memory. size=96002048
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
[6522:0416/101829.273887:FATAL:memory.cc(22)] Out of memory. size=155766784
ATTENTION: default value of option force_s3tc_enable overridden by environment.

I am not concerned about the force_s3tc_enable but notice that there are several out of memory errors. Based on the timestamp of the generated core file I can correlate these out of memory errors to within a minute or 2 of core file.

So, in the above I have a core dump from:
4-13-2019 @ 11:18am
5-15/2019 @ 14:28pm
4-15-2019 @ 23:35pm
4-16-2019 @ 10:18am

There does not look to be a pattern here so not sure where to go other than disabled modules 1-by-1 and wait.

Given I have the core file is there a way to read it and determine at least which module caused the error as that would help me try to narrow it down.

So did you just create a new project for the magic mirror and use that api key? I have done that and it says I am on a free trial. I had created an account months ago for another project. What will trigger being billed? Is it greater than x calls per month?

I was thinking about setting up MMM-MyCommute and it requires that I create an API key from google. It appears that I need to enable the Maps JavaScript API but from what I can tell that is not a free service. Can someone confirm that this does require payments?

Is there an alternate commute type module that is free?

It sounds like the pi does not have the desktop running. Did you happen to configure the ~/.config/lxsession/LXDE-pi/autostart and put lines in it? If so then there is a good chance that perhaps this file is taking precedence over the /etc/xdg/lxsession/LXDE-pi/autostart file. If that is the case then the entries to start the desktop may not get executed. Try removing/renaming the ~/.config/lxsession/LXDE-pi/autostart and reboot and see what happens. If you get the desktop GUI that way then you may want to add these lines to the ~/.config/lxsession/LXDE-pi/autostart file:

@lxpanel --profile LXDE-pi
@pcmanfm --desktop --profile LXDE-pi
@point-rpi

Sam, when I cleared the simpleGits array only then nothing was refreshing. That could have been where I was clearing it though. It seems that as you mentioned the configureModules was only getting called 1 time when initially loaded unless of course I accessed MM via the browser which called everything again.

For now I am happy with my solution as it is working for both auto refreshes via the updateInterval and for browser access from another machine. Its not any more code than you added but just done a different way. It may not be quite as efficient as it builds the simpleGits array everytime it is called but I don’t plan to call it more than once every 6 hrs anyway so that is not going to be a problem.

It is yet to be seen if this fixes the out of memory errors that were being seen though. I’ll have to leave it running for a day or 2 and check to see what happens.

@mlcampbe
I checked my logs this morning and with the updateInterval set to 1hr I see some fishy updates.

19:05 - checks all 7 modules 2 times
20:05 - checks all 7 modules 3 times
21:05 - checks all 7 modules 3 times
21:32 - checks all 7 modules 3 times
22:32 - checks all 7 modules 8 times
0:32 - checks all 7 modules 4 times
1:32 - checks all 7 modules 4 times

So there is something odd going on. After I restarted MM via the pm2 process I then access MM via browser on a local machine and it refreshes all modules and I can see that the number of times updateNotification checks the modules gets incremented by 1. Every time I refresh the browser it prints out my modules and clearly this scenario is causing the simpleGits array to be appended to and all modules to be checked n+1 times.

As a fix I added code to save the incoming module list (passed into socketNotificationReceived) like this:

        socketNotificationReceived: function (notification, payload) {
                if (notification === "CONFIG") {
                        this.config = payload;
                        savedPayload = payload;
                } else if(notification === "MODULES") {
                        this.configureModules(payload);
                        this.preformFetch();
                }
        },

Now the variable savedPayload is re-used at the bottom of the preformFetch:

                        sg.git.fetch().status(function(err, data) {
                                data.module = sg.module;
                                if (!err) {
                                        sg.git.log({"-1": null}, function(err, data2) {
                                                data.hash = data2.latest.hash;
                                                self.sendSocketNotification("STATUS", data);
                                        });
                                }
                        });
                });
                simpleGits = [];
                this.configureModules(savedPayload);
                this.scheduleNextFetch(this.config.updateInterval);
        },

That seems to work for browser refreshes as I can see that no matter how many times I refresh it there is only a single check of each module. I’ll monitor updateInterval refreshes to see what occurs.

Yeah I think I may have jump to a conclusion too fast. In my overnight test I had the updateInterval set to 6hrs so there was no overlap of events firing as I only have 6 modules installed.

The overnight test did show it adding to the simpleGits array but subsequent testing showed that if I cleared the array as I listed then there were no subsequent updates. That would go along with the fact that the array would only be built 1 time. I am not sure what was going on initially.

I’ve removed all my added code except the console.log message to see when the updates are called and what modules are updated. I’ll monitor it for a bit and see what happens.

Like many have commented on here it seems that the updateNotification module will often crash and cause the MM to display a black screen and the logs to show an out of memory error. The suggestion by many has been to disable the updateNotification module. I too had this problem and this week went about looking into what was going on. I found a post that indicated the problem could be due to the module firing off update checks for ALL installed modules simultaneously. That intrigued me to think about if it was possible to use the javascript setTimeout() function to add a delay between each modules update check. In updateNotification’s node_helper.js file I added the setTimeout() around the sg.git.fetch() call with a 60sec delay added for each module to be checked:

                var i = 1;
                simpleGits.forEach(function(sg) {
                      console.log("scheduling update of " + sg.module);
                      setTimeout(function(){
                        var time = new Date();
                        console.log(time.getHours() + ":" + time.getMinutes() + ":" + time.getSeconds() + "-->" + sg.module);
                        sg.git.fetch().status(function(err, data) {
.....
                     }, 60000*i);
                     i++;
                });

This did what I expected in that the modules were updated 1 at a time with a 60sec delay between each. However, as I had the debugging statement to print out when the module was scheduled and when it actually fired I noticed something odd.

On the first updateInterval all modules were updated as expected.
On the 2nd updateInterval each modules was updated 2 times.
On the 3rd updateInterval each modules was updated 3 times.
etc …

It appears that there is a bug in the updateNotification module in that it just keeps appending all of the modules to the end of an array and on the updateInterval it checks all rows in the array. So in my case the first update fires off 6 checks at the same time, the 2nd update 12 checks, the 3rd update 18 checks, …, the 20th check 120 checks. I can see how this could easily run into an out of memory condition if there are many modules to checked and all are fired off at the same time especially on a machine such as a pi with limited memory to start with.

As a test I have taken out my setTimeout code and added in a line to clear the contents of the simpleGits array after each update. For example I now have this:

                this.scheduleNextFetch(this.config.updateInterval);
                simpleGits.length = 0;
        },

        scheduleNextFetch: function(delay) {

Has anyone seen this or think this may be the cause of the updateNotification out of memory errors?

I found that the default update interval of the updatenotificaton module to be 10 minutes. That causes MM to check each of the installed modules every 10 minutes to determine if they are updated. While I don’t have usage issues for internet I found that 10 minutes seems way too often than need be. I adjusted mine back to 21600000 ms (6 hours).