Strange issue

richland007

Hi all i have my magic mirror running “flawlessly” with all my modules arranged nicely ready for final hardware install.
I have been leaving it running on my Rpi and monitor for a while before the whole final, on the wall install but something strange is happening … I have pm 2 running and MMM-watchdog and all seems to be going perfect until MM gets abruptly restarted and it happens quite often.

Where can i look for errors on what’s been causing this??.. also can i change any setting somewhere whether on pm2 or watchdog so it can be more “tolerating” and “forgiving”??

Any help will be greatly appreciated
Thank you
Denis

OceanK

@richland007

Hey mate!

Same issue here. I might suspect that the pi gets overloaded somehow. If I run less modules it’s not restarting.

I’m going to try a Intel NUC see what’s happening

richland007

Please some help!!! anybody!!
Where and how do i check for errors and whats casuing them and what module is the hardest and most tasking.
The strange thing is that my cpu usage is never more than 30% with 10 -20 % of free ram at all times!!

D

lolobyte

Incrase the swapfile?

evroom

Hi,

Investigating crashes and freezes, etc, can be very difficult.
When everything is running fine and you collect data, you will see that everything is more or less okay.
When it crashed or is frozen, you are too late.

Since a few weeks I had the problem that my MM was frozen and I could not enter any commands or a command took painstakingly long to return output.
A command like top was out of the question for example.
Using a script running as a cron job, I was able to identify a memory issue with electron.js.
I then decide to use a usb stick on which I created a swap device.
Since then my MM is not freezing anymore.
And I am able to see electron.js using up my memory and sometimes I see spikes in CPU usage.

Mind you, that crashing is something different then freezing, but the tools to use are more or less the same, provided the MM application or a MM module is not the culprit.

My script to check memory usage, very basic and simple:

pi@MagicPi:~ $ cat top.sh
#!/bin/sh

date >> /home/pi/top.txt
top -b -o +%MEM | head -n 22 >> /home/pi/top.txt
echo "=============================================" >> /home/pi/top.txt

The cron job:

00,05,10,15,20,25,30,35,40,45,50,55 * * * * /home/pi/top.sh >/dev/null 2>&1

%CPU will sort on CPU usage.

Concerning swap space, top will show the usage, but this gives a quick view:

$ free -h

Swap details via:

$ swapon -s

There are many more commands and settings to look at, but one has to start somewhere :-)

Good luck.

richland007

@evroom Thank you for your reply and i want to try all that you have written as i really feel it may help my case … but i am having a hard time decoding your steps.

Please walk me through it, on a step by step all the way to the end on how to even increase the swap file, and what to put on my scrip and make it executable and what is a cron job?? that will be my first one actually :)

Denis

evroom

@richland007
Since you managed to setup MM to the point that it is running, but crashing, I take it you know a little bit of Linux.
Otherwise it will be more like a Linux workshop.

Do you know how to use nano and/or vi to edit files?
Know what for example cat, tail and head does?
Just some Linux basics.

I do not want to sound like a smartarse and knowitall, am a hobbyist just like you.
Just to know the level of describing that is needed.

richland007

@evroom Hahah dont’ worry about coming out the wrong way … i know a little bit and i do know what nano and vi does …i use nano.
I am not familiar with cats and tails etc but i am a fast learner and i have set up quite a bit of MM modules some stubborn ones as well but i am no expert by all means.

I couldn’t read your first response at the beginning for some reason it wasn’t structured properly or my browser wasn’t reading it right now its a bit better but i still need some guidance

So i need to set up a script called “top” and make it executable right??
Than i have no idea what to do with cron :) !!!

Lets start somewhere :)

Denis

evroom

@richland007

Then let’s give it a try :slightly_smiling_face:

First create the script:

pi@MagicPi:~ $ cd
pi@MagicPi:~ $ nano top.sh

Copy-paste this:

!/bin/sh

date >> /home/pi/top.txt
top -b -o +%MEM | head -n 22 >> /home/pi/top.txt
top -b -o +%MEM | head -n 22 >> /home/pi/top.txt
echo "=============================================" >> /home/pi/top.txt

Make it executable:

pi@MagicPi:~ $ chmod 755 top.sh

Perhaps a good idea to prepare the output file in order to be sure the cron user can write in it:

pi@MagicPi:~ $ touch /home/pi/top.txt
pi@MagicPi:~ $ chmod 666 /home/pi/top.txt

Now we add a cronjob:

pi@MagicPi:~ $ crontab -e

Add this line at the end:

00,05,10,15,20,25,30,35,40,45,50,55 * * * * /home/pi/top.sh >/dev/null 2>&1

To verify:

pi@MagicPi:~ $ crontab -l

To verify if the cronjob is actually running:

pi@MagicPi:~ $ tail -F /var/log/syslog

To check if something comes in your output file:

pi@MagicPi:~ $ tail -F top.txt

The script will run every 5 minutes, every hour, every day.
It will output the top 15 MEMory usage and the top 15 CPU usage.

In your case, it will be interesting to look at the last entries before the crash.
Need to identify first if it is an CPU or MEM issue, or both.
Then we can modify the script.

Please do not post lengthy output.
Better we minimise the output by grepping and sorting, etc.

Success!

richland007

@evroom Ahhh you are a great man
I will set it up as soon as i get home
thanks in advance
Denis

evroom

Ha ha, not that great, my MM is as slow as a turtle again.
The used swap on my USB stick in the meanwhile is 1GB.
Cannot use any commands to investigate further.
Not very satisfactory. :frowning_face:

richland007

@evroom Mine is not slow it works “fine” it just keeps restarting
I am really curious to see the output of that file and see whats causing this behavior
D

vinp

If previous suggestions dont help, try removing the MMM-WatchDog module and see if it crashes anymore.

evroom

@richland007
I re-read your first post and your problem is more restarting then actual crashing.
Of course a restart can be caused by a crash …
MMM-WatchDog tends to restart MM, so I too would suggest to remove this module.
In principle all it does is sending a ping and restarting MM using pm2 when a timeout occurs.
I also would expect to see some lines in the pm2 logs on WatchDog.
Can you do ?:

pi@MagicPi:~ $ grep -i watchdog /home/pi/.pm2/logs/mm-out.log
pi@MagicPi:~ $ grep -i watchdog /home/pi/.pm2/logs/mm-error.log

What is alo a good package to install is sysstat:

pi@MagicPi:~ $ sudo apt-get install sysstat

This allows for commands like iostat and sar.

vmstat 10 10
iostat 10 10
sar 10 10

richland007

@evroom Ok so here is whats writen on that top.txt file

pi@SmartMirror:~ $ tail -F top.txt
  834 pi        20   0  155396  17884   8104 S   0.0  1.9  23:20.77 lxpanel
  582 root      20   0  211836  13576   2440 S   0.0  1.4 418:48.14 Xorg
  457 alexapi   20   0  218108  10768   4772 S  47.4  1.1 878:20.67 python
  721 pi        20   0  122032  10600   4792 S   0.0  1.1   8:34.40 PM2 v3.0.3+
  835 pi        20   0  154056   5184   4016 S   0.0  0.5   0:43.57 pcmanfm
18753 pi        20   0   47692   4896   3996 S   0.0  0.5   0:03.94 lxterminal
  829 pi        20   0   53536   4432   3556 S   0.0  0.5   0:13.22 openbox
  100 root      20   0   35220   3140   2980 S   0.0  0.3   2:16.35 systemd-jo+
 7236 pi        20   0    8112   3140   2712 R  15.8  0.3   0:00.07 top
=============================================
Sat Nov 17 11:25:01 CST 2018
top - 11:25:01 up 3 days, 15:28,  1 user,  load average: 0.56, 2.88, 4.35
Tasks: 135 total,   1 running,  89 sleeping,   0 stopped,   1 zombie
%Cpu(s): 21.4 us,  5.5 sy,  0.0 ni, 69.4 id,  2.3 wa,  0.0 hi,  1.5 si,  0.0 st
KiB Mem :   949452 total,   493928 free,   157456 used,   298068 buff/cache
KiB Swap:   949440 total,   777124 free,   172316 used.   727276 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  582 root      20   0  218840  24540  12364 S  10.5  2.6 419:19.31 Xorg
  502 root      20   0   45188  23400  14452 S  36.8  2.5  27:20.07 vncserver-+
  834 pi        20   0  155396  21680  10616 S   0.0  2.3  23:23.35 lxpanel
 7558 pi        20   0   77052  21140  18404 S   0.0  2.2   0:03.37 leafpad
  835 pi        20   0  154156  16420  13232 S   0.0  1.7   0:46.57 pcmanfm
  721 pi        20   0  122032  14004   7360 S   0.0  1.5   8:34.81 PM2 v3.0.3+
  457 alexapi   20   0  218108  10772   4772 S  36.8  1.1 880:07.93 python
  636 root      20   0   15416   9020   8820 S   0.0  1.0   3:55.73 vncagent
18753 pi        20   0   47692   8484   7164 S   0.0  0.9   0:04.22 lxterminal
  829 pi        20   0   53724   5888   4220 S   0.0  0.6   0:13.49 openbox
  953 pi        20   0   27236   5312   4812 S   0.0  0.6   0:03.05 vncserverui
  873 pi        20   0   27816   4328   3676 S   0.0  0.5   0:40.96 vncserverui
  804 pi        20   0   39468   3288   3000 S   0.0  0.3   0:02.79 gvfsd
  100 root      20   0   35220   3196   3036 S   0.0  0.3   2:16.50 systemd-jo+
 7603 pi        20   0    8112   3180   2752 R  10.5  0.3   0:00.05 top
top - 11:25:01 up 3 days, 15:28,  1 user,  load average: 0.56, 2.88, 4.35
Tasks: 135 total,   1 running,  89 sleeping,   0 stopped,   1 zombie
%Cpu(s): 21.4 us,  5.5 sy,  0.0 ni, 69.4 id,  2.3 wa,  0.0 hi,  1.5 si,  0.0 st
KiB Mem :   949452 total,   493776 free,   157600 used,   298076 buff/cache
KiB Swap:   949440 total,   777124 free,   172316 used.   727128 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  582 root      20   0  218840  24540  12364 S  11.1  2.6 419:19.33 Xorg
  502 root      20   0   45188  23400  14452 S  50.0  2.5  27:20.16 vncserver-+
  834 pi        20   0  155396  21680  10616 S   0.0  2.3  23:23.36 lxpanel
 7558 pi        20   0   77052  21140  18404 S   0.0  2.2   0:03.37 leafpad
  835 pi        20   0  154156  16420  13232 S   0.0  1.7   0:46.57 pcmanfm
  721 pi        20   0  122032  14004   7360 S   0.0  1.5   8:34.81 PM2 v3.0.3+
  457 alexapi   20   0  218108  10772   4772 S  61.1  1.1 880:08.05 python
  636 root      20   0   15416   9020   8820 S   0.0  1.0   3:55.73 vncagent
18753 pi        20   0   47692   8484   7164 S   0.0  0.9   0:04.24 lxterminal
  829 pi        20   0   53724   5888   4220 S   0.0  0.6   0:13.49 openbox
  953 pi        20   0   27236   5312   4812 S   0.0  0.6   0:03.05 vncserverui
  873 pi        20   0   27816   4328   3676 S   0.0  0.5   0:40.96 vncserverui
  804 pi        20   0   39468   3288   3000 S   0.0  0.3   0:02.79 gvfsd
  100 root      20   0   35220   3196   3036 S   0.0  0.3   2:16.50 systemd-jo+
 7608 pi        20   0    8112   3180   2752 R  22.2  0.3   0:00.07 top
=============================================

I use the command

tail -F top.txt

right?? i dont know how to read the above file so let me know if you see anything strange …i have increased the swap file to install opencv a while back for facial recognition to install rapidly but i do not use a usb …i could if i have to.

As far as taking out the watchdog…commenting it out of the config.js should do the trick or do i have to uninstall it form the modules??

Will pm2 still restart the MM if it crashes without the watchdog (it does if you do ctrl+q) ??

D

evroom

Okay, this calls for some more basics.

Print the whole file:
$ cat top.txt

Load the file in an editor:
$ nano top.txt
$ vi top.txt

I am old-school, so I use vi, but nano is more Word-like.

To show the last 50 lines:
$ tail -50 top.txt

To show the first 50 lines:
$ head -50 top.txt

Where of course 50 is just an example.

To show text that is appending:
$ tail -f top.txt

To show text that is appending and re-open the file when necessary (useful for rotating log files):
$ tail -F top.txt

On disabling a module see the next reply.

evroom

To disable MMM-WatchDog, edit the config.js file.
Locate:

module: 'MMM-WatchDog',

Edit this like this:

module: 'MMM-WatchDog',
disabled: true,

and restart MM:

$ pm2 restart mm

Use

pm2 list

to check if your application name actually is mm, or something else.

Later you can enable it again by updating config.js like this:

disabled: false,

This works for all modules.
When I test modules I normally leave the config lines and disable a module in this fashion.

evroom

@richland007
On pm2 and restarting I cannot say many.
On using ctrl+q neither, as I only access my MM via ssh.

Can you send the output of the following commands:

$ uname -a
$ swapon -s
$ free -h
$ cat /etc/dphys-swapfile | egrep -v ‘#|^$’
$ sudo service dphys-swapfile status

I just learned that swap is handled as a service.
Being old-school, this is something new to me :-)

evroom

@richland007
Concerning your top.txt, was that the output on the moment that MM was restarted ?
I do not see obvious memory issues.
Only that vncserver and python are CPU hungry, but within limits.
And I do not see any electron processes.

cowboysdude

@oceank The BIGGEST problem that nobody either wants to admit or doesn’t realize is that the pi wasn’t made to be pounded on this hard.

It’s a simple computer…when you start adding constant pulls for data and scrolling text, etc it over heats and overloads the pi.