rpi3 freezes every few days .. need help

nagaculun · Nov 5, 2019, 5:26 PM

First I thought it was one of the modules that load huge picture or video from google. But it happens as well with a very simple config.

I changed the power supply to 2.5A. But it still happens.

Any help ? What should I look for in the logs ?

Thanks

buzzkc · Nov 5, 2019, 6:10 PM

Run pm2 list to get your instance name

pm2 list

Run pm2 logs instanceNameHere to get your error logs.

pm2 logs mm

Let’s see if there are any out of memory errors or a module throwing an error.

nagaculun · Nov 6, 2019, 5:13 PM

Here are the output:

pi@MM:~ $ pm2 logs mm
[TAILING] Tailing last 15 lines for [mm] process (change the value with --lines option)
/home/pi/.pm2/logs/mm-error.log last 15 lines:
0|mm | npm WARN npm npm does not support Node.js v10.15.2
0|mm | npm WARN npm You should probably upgrade to a newer version of node as we
0|mm | npm WARN npm can’t make any promises that npm will work with this version.
0|mm | npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
0|mm | npm WARN npm You can find the latest version at https://nodejs.org/
0|mm | npm WARN npm npm does not support Node.js v10.15.2
0|mm | npm WARN npm You should probably upgrade to a newer version of node as we
0|mm | npm WARN npm can’t make any promises that npm will work with this version.
0|mm | npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
0|mm | npm WARN npm You can find the latest version at https://nodejs.org/
0|mm | npm WARN npm npm does not support Node.js v10.15.2
0|mm | npm WARN npm You should probably upgrade to a newer version of node as we
0|mm | npm WARN npm can’t make any promises that npm will work with this version.
0|mm | npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
0|mm | npm WARN npm You can find the latest version at https://nodejs.org/

/home/pi/.pm2/logs/mm-out.log last 15 lines:
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | There is no scanned photo currently.
0|mm | [GPHOTO] Scan finished : 3972
0|mm | [GPHOTO] Scan finished : 3972
0|mm | [GPHOTO] Scan finished : 3972
0|mm | [GPHOTO] Scan finished : 3972
0|mm | [GPHOTO] Scan finished : 3972

^C
pi@MM:~ $

Is the npm warning expected ?

nagaculun · Nov 6, 2019, 5:16 PM

It frozen again at 10:49:57. I grep that time and found this.
Kernel issue ?


root@MM:/var/log# cat error.l
Nov  5 10:49:34 MM kernel: [ 6438.728602] cache_from_obj: Wrong slab cache. shmem_inode_cache but object is from `�����
Nov  5 10:49:34 MM kernel: [ 6438.728616] ------------[ cut here ]------------
Nov  5 10:49:34 MM kernel: [ 6438.728649] WARNING: CPU: 3 PID: 0 at mm/slab.h:380 kmem_cache_free+0x1ec/0x238
Nov  5 10:49:34 MM kernel: [ 6438.728657] Modules linked in: fuse 8021q garp stp llc sg uas brcmfmac brcmutil sha256_generic cfg80211 raspberrypi_hwmon hwmon rfkill bcm2835_codec(C) bcm2835_v4l2(C) v4l2_mem2mem snd_bcm2835(C) bcm2835_mmal_vchiq(C) v4l2_common videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops snd_pcm snd_timer snd videobuf2_v4l2 videobuf2_common videodev media vc_sm_cma(C) uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables
Nov  5 10:49:34 MM kernel: [ 6438.728872] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G    B    C        4.19.75-v7+ #1270
Nov  5 10:49:34 MM kernel: [ 6438.728877] Hardware name: BCM2835
Nov  5 10:49:34 MM kernel: [ 6438.728906] [<80111fcc>] (unwind_backtrace) from [<8010d544>] (show_stack+0x20/0x24)
Nov  5 10:49:34 MM kernel: [ 6438.728926] [<8010d544>] (show_stack) from [<80819bc0>] (dump_stack+0xd4/0x118)
Nov  5 10:49:34 MM kernel: [ 6438.728945] [<80819bc0>] (dump_stack) from [<80120a88>] (__warn+0x104/0x11c)
Nov  5 10:49:34 MM kernel: [ 6438.728964] [<80120a88>] (__warn) from [<80120bd8>] (warn_slowpath_null+0x50/0x58)
Nov  5 10:49:34 MM kernel: [ 6438.728980] [<80120bd8>] (warn_slowpath_null) from [<802960d0>] (kmem_cache_free+0x1ec/0x238)
Nov  5 10:49:34 MM kernel: [ 6438.729000] [<802960d0>] (kmem_cache_free) from [<802585dc>] (shmem_destroy_callback+0x3c/0x4c)
Nov  5 10:49:34 MM kernel: [ 6438.729018] [<802585dc>] (shmem_destroy_callback) from [<8018e288>] (rcu_process_callbacks+0x2cc/0x674)
Nov  5 10:49:34 MM kernel: [ 6438.729038] [<8018e288>] (rcu_process_callbacks) from [<80102410>] (__do_softirq+0x190/0x3f0)
Nov  5 10:49:34 MM kernel: [ 6438.729057] [<80102410>] (__do_softirq) from [<80126a60>] (irq_exit+0xfc/0x120)
Nov  5 10:49:34 MM kernel: [ 6438.729077] [<80126a60>] (irq_exit) from [<8017f22c>] (__handle_domain_irq+0x70/0xc4)
Nov  5 10:49:34 MM kernel: [ 6438.729096] [<8017f22c>] (__handle_domain_irq) from [<801021b4>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
Nov  5 10:49:34 MM kernel: [ 6438.729112] [<801021b4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
Nov  5 10:49:34 MM kernel: [ 6438.729119] Exception stack(0xb9d51f38 to 0xb9d51f80)
Nov  5 10:49:34 MM kernel: [ 6438.729128] 1f20:                                                       80109a84 00000000
Nov  5 10:49:34 MM kernel: [ 6438.729142] 1f40: 40000093 40000093 80d04d70 b9d50000 80d04db8 00000008 80d8ed3e 410fd034
Nov  5 10:49:34 MM kernel: [ 6438.729155] 1f60: 00000000 b9d51f94 80d0517c b9d51f88 00000000 80109a88 40000013 ffffffff
Nov  5 10:49:34 MM kernel: [ 6438.729173] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
Nov  5 10:49:34 MM kernel: [ 6438.729193] [<80109a88>] (arch_cpu_idle) from [<80836c1c>] (default_idle_call+0x34/0x48)
Nov  5 10:49:34 MM kernel: [ 6438.729211] [<80836c1c>] (default_idle_call) from [<801526d8>] (do_idle+0xec/0x17c)
Nov  5 10:49:34 MM kernel: [ 6438.729232] [<801526d8>] (do_idle) from [<80152a28>] (cpu_startup_entry+0x28/0x2c)
Nov  5 10:49:34 MM kernel: [ 6438.729249] [<80152a28>] (cpu_startup_entry) from [<8010fde8>] (secondary_start_kernel+0x134/0x140)
Nov  5 10:49:34 MM kernel: [ 6438.729265] [<8010fde8>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
Nov  5 10:49:34 MM kernel: [ 6438.729276] ---[ end trace 44e2f56c32b8bf1c ]---
Nov  5 10:49:54 MM kernel: [ 6458.548472] cache_from_obj: Wrong slab cache. shmem_inode_cache but object is from `�����
Nov  5 10:49:54 MM kernel: [ 6458.548493] cache_from_obj: Wrong slab cache. shmem_inode_cache but object is from `�����
Nov  5 10:49:54 MM kernel: [ 6458.548505] cache_from_obj: Wrong slab cache. shmem_inode_cache but object is from `�����
Nov  5 10:49:54 MM kernel: [ 6458.548517] cache_from_obj: Wrong slab cache. shmem_inode_cache but object is from `�����
Nov  5 10:49:54 MM kernel: [ 6458.548529] cache_from_obj: Wrong slab cache. shmem_inode_cache but object is from `�����
Nov  5 10:49:57 MM kernel: [ 6460.970198] Unable to handle kernel paging request at virtual address 000064d4
Nov  5 10:49:57 MM kernel: [ 6460.977431] pgd = 31d1d4f4
Nov  5 10:49:57 MM kernel: [ 6460.980130] [000064d4] *pgd=208fc835, *pte=00000000, *ppte=00000000
Nov  5 10:49:57 MM kernel: [ 6460.986403] Internal error: Oops: 17 [#1] SMP ARM
Nov  5 10:49:57 MM kernel: [ 6460.991101] Modules linked in: fuse 8021q garp stp llc sg uas brcmfmac brcmutil sha256_generic cfg80211 raspberrypi_hwmon hwmon rfkill bcm2835_codec(C) bcm2835_v4l2(C) v4l2_mem2mem snd_bcm2835(C) bcm2835_mmal_vchiq(C) v4l2_common videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops snd_pcm snd_timer snd videobuf2_v4l2 videobuf2_common videodev media vc_sm_cma(C) uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables
Nov  5 10:49:57 MM kernel: [ 6461.028111] CPU: 0 PID: 1255 Comm: Chrome_IOThread Tainted: G    B   WC        4.19.75-v7+ #1270
Nov  5 10:49:57 MM kernel: [ 6461.036887] Hardware name: BCM2835
Nov  5 10:49:57 MM kernel: [ 6461.040292] PC is at ___slab_alloc.constprop.11+0x344/0x594
Nov  5 10:49:57 MM kernel: [ 6461.045859] LR is at ___slab_alloc.constprop.11+0x338/0x594
Nov  5 10:49:57 MM kernel: [ 6461.051424] pc : [<80294b14>]    lr : [<80294b08>]    psr: 60000093
Nov  5 10:49:57 MM kernel: [ 6461.057682] sp : a0a3dc40  ip : a0a3dc40  fp : a0a3dcb4
Nov  5 10:49:57 MM kernel: [ 6461.062898] r10: 00000000  r9 : b69cc000  r8 : b37cf508
Nov  5 10:49:57 MM kernel: [ 6461.068115] r7 : b37cf500  r6 : 00000000  r5 : 000064d0  r4 : ba67fffc
Nov  5 10:49:57 MM kernel: [ 6461.074634] r3 : b6a798c0  r2 : 000064d4  r1 : 00000000  r0 : 00000000
Nov  5 10:49:57 MM kernel: [ 6461.081154] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
Nov  5 10:49:57 MM kernel: [ 6461.088367] Control: 10c5383d  Table: 30f0406a  DAC: 00000055
Nov  5 10:49:57 MM kernel: [ 6461.094108] Process Chrome_IOThread (pid: 1255, stack limit = 0xdc362e5a)
Nov  5 10:49:57 MM kernel: [ 6461.100888] Stack: (0xa0a3dc40 to 0xa0a3e000)
Nov  5 10:49:57 MM kernel: [ 6461.105240] dc40: 00000000 80d8ee82 80d04d48 ff778840 80258618 006000c0 a0b364f0 a0a3de3c
Nov  5 10:49:57 MM kernel: [ 6461.113412] dc60: 0000001c 00000000 0000001c 80294d94 ff778840 00120012 006000c0 b69cc000
Nov  5 10:49:57 MM kernel: [ 6461.121583] dc80: b9cc8e40 f272aca4 a0a3dcb4 60000013 ff778840 80258618 006000c0 b69cc000
Nov  5 10:49:57 MM kernel: [ 6461.129755] dca0: b9cc8e40 000014ab a0a3dcdc a0a3dcb8 80294da8 802947dc ff778840 00000000
Nov  5 10:49:57 MM kernel: [ 6461.137927] dcc0: 006000c0 b69cc000 00000000 80258618 a0a3dd1c a0a3dce0 80295574 80294d70
Nov  5 10:49:57 MM kernel: [ 6461.146099] dce0: 00000003 80d0d640 a0a3dd14 a0a3dcf8 80201d10 b90bc1e0 b9731800 00000000
Nov  5 10:49:57 MM kernel: [ 6461.154271] dd00: 00000000 b9731800 b90bc1e0 00000000 a0a3dd2c a0a3dd20 80258618 80295374
Nov  5 10:49:57 MM kernel: [ 6461.162442] dd20: a0a3dd44 a0a3dd30 802c7ff4 802585f8 b90bc1e0 00008180 a0a3dd5c a0a3dd48
Nov  5 10:49:57 MM kernel: [ 6461.170613] dd40: 802ca594 802c7fd4 00000000 b90bc1e0 a0a3dd74 a0a3dd60 802ca5f4 802ca588
Nov  5 10:49:57 MM kernel: [ 6461.178786] dd60: 00000000 b90bc1e0 a0a3ddac a0a3dd78 80258c04 802ca5e4 a0a3dd9c a0a3dd88
Nov  5 10:49:57 MM kernel: [ 6461.186957] dd80: 80168384 80168264 b90bc1e0 b8c6b3b8 00000000 98e16000 94838e40 a0a3de90
Nov  5 10:49:57 MM kernel: [ 6461.195129] dda0: a0a3ddec a0a3ddb0 80258f54 80258bd0 00200000 00009a80 a0a3dde4 a0a3ddc8
Nov  5 10:49:57 MM kernel: [ 6461.203300] ddc0: 802c4e70 802c29fc 98e16000 000200c2 b8c6b3b8 00000000 a0a3df50 94838e40
Nov  5 10:49:57 MM kernel: [ 6461.211472] dde0: a0a3ddfc a0a3ddf0 802592b0 80258f2c a0a3de8c a0a3de00 802bb440 8025929c
Nov  5 10:49:57 MM kernel: [ 6461.219643] de00: 00000000 f272aca4 a0a3df44 98e16000 00000000 00008180 b90bc1e0 a0a3c000
Nov  5 10:49:57 MM kernel: [ 6461.227815] de20: 98e16000 000200c2 80d04d48 00000041 80d05688 00000040 00000001 00000006
Nov  5 10:49:57 MM kernel: [ 6461.235987] de40: 00001000 80d04d48 a9347c94 00000000 00010001 a0a3de54 a0a3de54 f272aca4
Nov  5 10:49:57 MM kernel: [ 6461.244159] de60: a9347c94 80d04d48 80d04d48 a0a3df50 a0a3de90 00000001 a0a3c000 00000142
Nov  5 10:49:57 MM kernel: [ 6461.252330] de80: a0a3df44 a0a3de90 802bcb80 802ba87c b96ef710 b8c6b3b8 3d1ed1df 0000001d
Nov  5 10:49:57 MM kernel: [ 6461.260502] dea0: b9fc6019 006000c0 00000000 b9a3a990 b90bc1e0 00000701 00000002 00000360
Nov  5 10:49:57 MM kernel: [ 6461.268673] dec0: 00000000 00000000 00000000 a0a3ded0 00000001 f272aca4 a0a3df40 0000005e
Nov  5 10:49:57 MM kernel: [ 6461.276845] dee0: b6a28e00 b0dda200 00000000 00002000 000200c2 00000100 a0a3df34 a0a3df08
Nov  5 10:49:57 MM kernel: [ 6461.285017] df00: b9fc6000 00000000 000200c2 00000002 ffffff9c ffffff9c b9fc6000 f272aca4
Nov  5 10:49:57 MM kernel: [ 6461.293188] df20: a0a3c000 0000005e 80d04d48 ffffff9c b9fc6000 801011c4 a0a3df94 a0a3df48
Nov  5 10:49:57 MM kernel: [ 6461.301360] df40: 802a8500 802bcb0c 00000000 00000000 000200c2 00008180 00000006 00000700
Nov  5 10:49:57 MM kernel: [ 6461.309531] df60: 00000001 f272aca4 0000020c 8fea2092 76f76968 00000000 00000142 801011c4
Nov  5 10:49:57 MM kernel: [ 6461.317703] df80: a0a3c000 00000142 a0a3dfa4 a0a3df98 802a85fc 802a83cc 00000000 a0a3dfa8
Nov  5 10:49:57 MM kernel: [ 6461.325874] dfa0: 80101000 802a85ec 8fea2092 76f76968 ffffff9c 069dc420 000200c2 00000180
Nov  5 10:49:57 MM kernel: [ 6461.334047] dfc0: 8fea2092 76f76968 00000000 00000142 00000000 0003a2f8 069dc420 00000000
Nov  5 10:49:57 MM kernel: [ 6461.342220] dfe0: 00000002 643b33a0 00000000 7491035c 80000010 ffffff9c 00000000 00000000
Nov  5 10:49:57 MM kernel: [ 6461.350405] [<80294b14>] (___slab_alloc.constprop.11) from [<80294da8>] (__slab_alloc.constprop.10+0x44/0x90)
Nov  5 10:49:57 MM kernel: [ 6461.360318] [<80294da8>] (__slab_alloc.constprop.10) from [<80295574>] (kmem_cache_alloc+0x20c/0x23c)
Nov  5 10:49:57 MM kernel: [ 6461.369536] [<80295574>] (kmem_cache_alloc) from [<80258618>] (shmem_alloc_inode+0x2c/0x38)
Nov  5 10:49:57 MM kernel: [ 6461.377885] [<80258618>] (shmem_alloc_inode) from [<802c7ff4>] (alloc_inode+0x2c/0xb4)
Nov  5 10:49:57 MM kernel: [ 6461.385801] [<802c7ff4>] (alloc_inode) from [<802ca594>] (new_inode_pseudo+0x18/0x5c)
Nov  5 10:49:57 MM kernel: [ 6461.393627] [<802ca594>] (new_inode_pseudo) from [<802ca5f4>] (new_inode+0x1c/0x30)
Nov  5 10:49:57 MM kernel: [ 6461.401278] [<802ca5f4>] (new_inode) from [<80258c04>] (shmem_get_inode+0x40/0x1b8)
Nov  5 10:49:57 MM kernel: [ 6461.408929] [<80258c04>] (shmem_get_inode) from [<80258f54>] (shmem_mknod+0x34/0xc8)
Nov  5 10:49:57 MM kernel: [ 6461.416667] [<80258f54>] (shmem_mknod) from [<802592b0>] (shmem_create+0x20/0x24)
Nov  5 10:49:57 MM kernel: [ 6461.424146] [<802592b0>] (shmem_create) from [<802bb440>] (path_openat+0xbd0/0x1024)
Nov  5 10:49:57 MM kernel: [ 6461.431886] [<802bb440>] (path_openat) from [<802bcb80>] (do_filp_open+0x80/0xec)
Nov  5 10:49:57 MM kernel: [ 6461.439368] [<802bcb80>] (do_filp_open) from [<802a8500>] (do_sys_open+0x140/0x1f0)
Nov  5 10:49:57 MM kernel: [ 6461.447022] [<802a8500>] (do_sys_open) from [<802a85fc>] (sys_openat+0x1c/0x20)
Nov  5 10:49:57 MM kernel: [ 6461.454330] [<802a85fc>] (sys_openat) from [<80101000>] (ret_fast_syscall+0x0/0x28)
Nov  5 10:49:57 MM kernel: [ 6461.461978] Exception stack(0xa0a3dfa8 to 0xa0a3dff0)
Nov  5 10:49:57 MM kernel: [ 6461.467024] dfa0:                   8fea2092 76f76968 ffffff9c 069dc420 000200c2 00000180
Nov  5 10:49:57 MM kernel: [ 6461.475196] dfc0: 8fea2092 76f76968 00000000 00000142 00000000 0003a2f8 069dc420 00000000
Nov  5 10:49:57 MM kernel: [ 6461.483368] dfe0: 00000002 643b33a0 00000000 7491035c
Nov  5 10:49:57 MM kernel: [ 6461.488416] Code: ebfeb53f e3500000 1affffb3 e2852004 (e5953004)
Nov  5 10:49:57 MM kernel: [ 6461.494507] ---[ end trace 44e2f56c32b8bf1d ]---
Nov  5 10:55:01 MM CRON[4379]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
root@MM:/var/log#

nagaculun · Nov 18, 2019, 1:43 AM

Anyone ?

sdetweil · Nov 18, 2019, 2:29 AM

@nagaculun sounds like the swap space might need to be enlarged

nagaculun · Nov 22, 2019, 10:44 PM

@sdetweil and how do I do that ?

Thanks

sdetweil · Nov 25, 2019, 2:21 AM

@nagaculun now now… google is your friend

see https://www.bitpi.co/2015/02/11/how-to-change-raspberry-pis-swapfile-size-on-rasbian/

Fozi · Nov 23, 2019, 8:31 AM

@nagaculun Are you using MMM-GoogleTrafficMap? This module has a severe memory leak. Increasing swap memory will not help, as it just shifts the freeze to a later point in time. Other option is to setup a cron job that reboots regularly the RPi.

sdetweil · Nov 25, 2019, 2:21 AM

@Fozi i have a fixed GoogleMapsTraffic module on my github.
I submitted it, but it hasn’t been picked up…

https://github.com/sdetweil/MMM-GoogleMapsTraffic