Strange issue

richland007 · Nov 16, 2018, 8:25 PM

@evroom Mine is not slow it works “fine” it just keeps restarting
I am really curious to see the output of that file and see whats causing this behavior
D

vinp · Nov 17, 2018, 3:56 AM

If previous suggestions dont help, try removing the MMM-WatchDog module and see if it crashes anymore.

evroom · Nov 17, 2018, 5:34 PM

@richland007
I re-read your first post and your problem is more restarting then actual crashing.
Of course a restart can be caused by a crash …
MMM-WatchDog tends to restart MM, so I too would suggest to remove this module.
In principle all it does is sending a ping and restarting MM using pm2 when a timeout occurs.
I also would expect to see some lines in the pm2 logs on WatchDog.
Can you do ?:

pi@MagicPi:~ $ grep -i watchdog /home/pi/.pm2/logs/mm-out.log
pi@MagicPi:~ $ grep -i watchdog /home/pi/.pm2/logs/mm-error.log

What is alo a good package to install is sysstat:

pi@MagicPi:~ $ sudo apt-get install sysstat

This allows for commands like iostat and sar.

vmstat 10 10
iostat 10 10
sar 10 10

richland007 · Nov 17, 2018, 5:34 PM

@evroom Ok so here is whats writen on that top.txt file

pi@SmartMirror:~ $ tail -F top.txt
  834 pi        20   0  155396  17884   8104 S   0.0  1.9  23:20.77 lxpanel
  582 root      20   0  211836  13576   2440 S   0.0  1.4 418:48.14 Xorg
  457 alexapi   20   0  218108  10768   4772 S  47.4  1.1 878:20.67 python
  721 pi        20   0  122032  10600   4792 S   0.0  1.1   8:34.40 PM2 v3.0.3+
  835 pi        20   0  154056   5184   4016 S   0.0  0.5   0:43.57 pcmanfm
18753 pi        20   0   47692   4896   3996 S   0.0  0.5   0:03.94 lxterminal
  829 pi        20   0   53536   4432   3556 S   0.0  0.5   0:13.22 openbox
  100 root      20   0   35220   3140   2980 S   0.0  0.3   2:16.35 systemd-jo+
 7236 pi        20   0    8112   3140   2712 R  15.8  0.3   0:00.07 top
=============================================
Sat Nov 17 11:25:01 CST 2018
top - 11:25:01 up 3 days, 15:28,  1 user,  load average: 0.56, 2.88, 4.35
Tasks: 135 total,   1 running,  89 sleeping,   0 stopped,   1 zombie
%Cpu(s): 21.4 us,  5.5 sy,  0.0 ni, 69.4 id,  2.3 wa,  0.0 hi,  1.5 si,  0.0 st
KiB Mem :   949452 total,   493928 free,   157456 used,   298068 buff/cache
KiB Swap:   949440 total,   777124 free,   172316 used.   727276 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  582 root      20   0  218840  24540  12364 S  10.5  2.6 419:19.31 Xorg
  502 root      20   0   45188  23400  14452 S  36.8  2.5  27:20.07 vncserver-+
  834 pi        20   0  155396  21680  10616 S   0.0  2.3  23:23.35 lxpanel
 7558 pi        20   0   77052  21140  18404 S   0.0  2.2   0:03.37 leafpad
  835 pi        20   0  154156  16420  13232 S   0.0  1.7   0:46.57 pcmanfm
  721 pi        20   0  122032  14004   7360 S   0.0  1.5   8:34.81 PM2 v3.0.3+
  457 alexapi   20   0  218108  10772   4772 S  36.8  1.1 880:07.93 python
  636 root      20   0   15416   9020   8820 S   0.0  1.0   3:55.73 vncagent
18753 pi        20   0   47692   8484   7164 S   0.0  0.9   0:04.22 lxterminal
  829 pi        20   0   53724   5888   4220 S   0.0  0.6   0:13.49 openbox
  953 pi        20   0   27236   5312   4812 S   0.0  0.6   0:03.05 vncserverui
  873 pi        20   0   27816   4328   3676 S   0.0  0.5   0:40.96 vncserverui
  804 pi        20   0   39468   3288   3000 S   0.0  0.3   0:02.79 gvfsd
  100 root      20   0   35220   3196   3036 S   0.0  0.3   2:16.50 systemd-jo+
 7603 pi        20   0    8112   3180   2752 R  10.5  0.3   0:00.05 top
top - 11:25:01 up 3 days, 15:28,  1 user,  load average: 0.56, 2.88, 4.35
Tasks: 135 total,   1 running,  89 sleeping,   0 stopped,   1 zombie
%Cpu(s): 21.4 us,  5.5 sy,  0.0 ni, 69.4 id,  2.3 wa,  0.0 hi,  1.5 si,  0.0 st
KiB Mem :   949452 total,   493776 free,   157600 used,   298076 buff/cache
KiB Swap:   949440 total,   777124 free,   172316 used.   727128 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  582 root      20   0  218840  24540  12364 S  11.1  2.6 419:19.33 Xorg
  502 root      20   0   45188  23400  14452 S  50.0  2.5  27:20.16 vncserver-+
  834 pi        20   0  155396  21680  10616 S   0.0  2.3  23:23.36 lxpanel
 7558 pi        20   0   77052  21140  18404 S   0.0  2.2   0:03.37 leafpad
  835 pi        20   0  154156  16420  13232 S   0.0  1.7   0:46.57 pcmanfm
  721 pi        20   0  122032  14004   7360 S   0.0  1.5   8:34.81 PM2 v3.0.3+
  457 alexapi   20   0  218108  10772   4772 S  61.1  1.1 880:08.05 python
  636 root      20   0   15416   9020   8820 S   0.0  1.0   3:55.73 vncagent
18753 pi        20   0   47692   8484   7164 S   0.0  0.9   0:04.24 lxterminal
  829 pi        20   0   53724   5888   4220 S   0.0  0.6   0:13.49 openbox
  953 pi        20   0   27236   5312   4812 S   0.0  0.6   0:03.05 vncserverui
  873 pi        20   0   27816   4328   3676 S   0.0  0.5   0:40.96 vncserverui
  804 pi        20   0   39468   3288   3000 S   0.0  0.3   0:02.79 gvfsd
  100 root      20   0   35220   3196   3036 S   0.0  0.3   2:16.50 systemd-jo+
 7608 pi        20   0    8112   3180   2752 R  22.2  0.3   0:00.07 top
=============================================

I use the command

tail -F top.txt

right?? i dont know how to read the above file so let me know if you see anything strange …i have increased the swap file to install opencv a while back for facial recognition to install rapidly but i do not use a usb …i could if i have to.

As far as taking out the watchdog…commenting it out of the config.js should do the trick or do i have to uninstall it form the modules??

Will pm2 still restart the MM if it crashes without the watchdog (it does if you do ctrl+q) ??

D

evroom · Nov 17, 2018, 7:38 PM

Okay, this calls for some more basics.

Print the whole file:
$ cat top.txt

Load the file in an editor:
$ nano top.txt
$ vi top.txt

I am old-school, so I use vi, but nano is more Word-like.

To show the last 50 lines:
$ tail -50 top.txt

To show the first 50 lines:
$ head -50 top.txt

Where of course 50 is just an example.

To show text that is appending:
$ tail -f top.txt

To show text that is appending and re-open the file when necessary (useful for rotating log files):
$ tail -F top.txt

On disabling a module see the next reply.

evroom · Nov 17, 2018, 7:45 PM

To disable MMM-WatchDog, edit the config.js file.
Locate:

module: 'MMM-WatchDog',

Edit this like this:

module: 'MMM-WatchDog',
disabled: true,

and restart MM:

$ pm2 restart mm

Use

pm2 list

to check if your application name actually is mm, or something else.

Later you can enable it again by updating config.js like this:

disabled: false,

This works for all modules.
When I test modules I normally leave the config lines and disable a module in this fashion.

evroom · Nov 17, 2018, 8:07 PM

@richland007
On pm2 and restarting I cannot say many.
On using ctrl+q neither, as I only access my MM via ssh.

Can you send the output of the following commands:

$ uname -a
$ swapon -s
$ free -h
$ cat /etc/dphys-swapfile | egrep -v ‘#|^$’
$ sudo service dphys-swapfile status

I just learned that swap is handled as a service.
Being old-school, this is something new to me :-)

evroom · Nov 17, 2018, 8:14 PM

@richland007
Concerning your top.txt, was that the output on the moment that MM was restarted ?
I do not see obvious memory issues.
Only that vncserver and python are CPU hungry, but within limits.
And I do not see any electron processes.

cowboysdude · Nov 18, 2018, 2:18 AM

@oceank The BIGGEST problem that nobody either wants to admit or doesn’t realize is that the pi wasn’t made to be pounded on this hard.

It’s a simple computer…when you start adding constant pulls for data and scrolling text, etc it over heats and overloads the pi.

richland007 · Nov 19, 2018, 12:47 AM

@evroom I appologise on the delay at answering i was at work during the weekend

here is the autput from

pi@MagicPi:~ $ grep -i watchdog /home/pi/.pm2/logs/mm-out.log
pi@MagicPi:~ $ grep -i watchdog /home/pi/.pm2/logs/mm-error.log

a bunch of these almost every 10-15 minutes

Fri Nov 09 2018 09:52:50 GMT-0600 (CST) - WatchDog: Heartbeat timeout. Frontend might have crashed. Exit now.
Fri Nov 09 2018 10:06:24 GMT-0600 (CST) - WatchDog: Heartbeat timeout. Frontend might have crashed. Exit now.

WatchDog started. Maximum timeout: 20s.
Module helper loaded: MMM-WatchDog
Connecting socket for: MMM-WatchDog
Starting module helper: MMM-WatchDog
WatchDog started. Maximum timeout: 20s.
Module helper loaded: MMM-WatchDog
Connecting socket for: MMM-WatchDog
Starting module helper: MMM-WatchDog

Here is some sysstat output from the following commands

vmstat 10 10
iostat 10 10
sar 10 10

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0  63744 209036  34364 567792    0    6    82    12  778  312 18  3 78  1  0
 0  0  63744 209036  34372 567824    0    0     0     2 2431  278  4  0 96  0  0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.75    0.00    3.32    0.57    0.00   78.36

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0          13.75       311.58        45.87     821628     120969
zram0             1.77         0.95         6.13       2504      16164
zram1             1.73         0.78         6.12       2068      16132
zram2             1.70         0.74         6.08       1952      16028
zram3             1.74         0.94         6.02       2480      15880

06:44:46 PM     all      3.17      0.00      0.08      0.00      0.00     96.76
06:44:56 PM     all      4.12      0.00      0.25      0.03      0.00     95.60