Every few hours I get "ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service"

sdetweil

@tve the 1st two errors in the stderr log are shown as a bug in electron.

nothing we can do.

no idea on the rest.

which os image did u use?

lsb_release -a

I am using the legacy release on a pi3b and it has been running mm for 2 weeks now, uninterrupted.

TvE

@sdetweil Sorry - I forgot that (latest OS :-))

pi@magicmirror:~/MagicMirror/logs $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye
pi@magicmirror:~/MagicMirror/logs $

I was not worried about the two first errors as they are always there (if they were serious I suspect that I would not have been able to launch MM at all)

The legacy release - what’s that (I should try that release and see if I can get the same positive results as you)

…googled…
It must be this you’re referring to?
https://www.tomshardware.com/news/raspberry-pi-releases-legacy-os

I gotta try that OS tomorow and se if that’s better

sdetweil

@tve https://www.raspberrypi.com/software/operating-systems/

scroll down

TvE

@sdetweil
Test is now started on legacy PI OS - so by tomorrow I’ll know if I have a stable setup

Goal 1 = 4 h (my shortest runtime with bullseye)
Goal 2 = 16 h (my longest runtime with bullseye)
Goal 3 = 24 h (if I get here I can expect that it’ll continue for ever :-) )

TvE

Hmm - still unstable

After just 1,5 hours I get the same error

[1229:0301/105243.836236:ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service.

Start test at 08:28
[01.03.2022 08:28.26.868] [LOG]   Starting MagicMirror: v2.18.0
...
Problem kicks in at 10:52
[01.03.2022 10:52.43.788] [LOG]   Shutting down server...

Could it really be problematic that I have configured both wlan as well as the eth interface??? (that would be hard to accept…)

Currently I am using eth, but prepared wlan as that’s what I’ll use when I “go to production”.

sdetweil

@tve well, address:“0.0.0.0”

means listen on ALL interfaces.

I use it on my pi w both enabled, but only Ethernet connected. ( and wifi not configured)

never has ‘crashed’

you could change to just the ethernet address

address:“192.168.?.?”

whatever that address is.

ifconfig

will show u
or
ip addr

TvE

@sdetweil OK - good point, that’s worth trying
New test has started only listening on one IP :-)

TvE

Hmm crashed after 6,5h with the same error…

NB.: My symptom is the same as being reported here (though I’m not using a docker container)
https://github.com/electron/electron/issues/31675

Is there any way of increasing the verbosity in any logs to get more debugging info?

I’m now perforing a new test:

Reverting to the default MM config
And tcpdump’ing the network traffic so I can correlate and maybe identify what triggers the network service crash

If it no longer crashes I’ll revert to my own config and retest (as it then must be related to that specific network traffic [no matter how unfair that seems]

TvE

Test with default config looks good 11h without a crash

[02.03.2022 07:43.20.529] [LOG]   Starting MagicMirror: v2.18.0
...
[02.03.2022 18:35.01.238] [INFO]  Calendar-Fetcher: Broadcasting 10 events.
And still running

Thus I must deduce that one of the two services I had configured was triggering the problem

I must enable my two changes individually to identify which is “bad”

sdetweil

@tve weird

TvE

@sdetweil Yep (but I get exposed to that type of behavior daily :-) )

I have also found the DEBUG setting in the conf file:

logLevel: ["INFO", "LOG", "WARN", "ERROR", "DEBUG"], // Add "DEBUG" for even more logging

And am tcpdumping the traffic as well

sudo tcpdump -i wlan0 -w tcpdump-wlan.pcap && sudo tcpdump -i eth0 -w tcpdump-eth.pcap &

So let’s see what a bisect approach to disabling the “services” leads to

sdetweil

@tve yeh, sorry on debug. lots of calendar stuff

TvE

@sdetweil Don’t worry about “debug” I just had to read the config in stead of skipping over that part

I have now reverted to my original config and will enable the different service in bisect style.

/*
TvE test log (on legacy OS)
Re-configured services resulting in crash:
  1 calendar
  2 openweather_current
  3 openweather_forecast
  4 news

A: Sample config
    => 11h and still OK

B: Enable DEBUG log level

C: Disable 2+3 (openweather) Enable 1+4
    => 14 h and still OK

D: Enable 2+3 (openweather) + disable 1+4 (news & calendar)
    => ongoing
*/

So in a few days I should know what’s triggering the issue…

TvE

Hmm so my latest config ran for more than 2 days.

I’m now reverting to the initial config (enabling all four config-changes) and then time will tell

TvE

@sdetweil This is really getting intersting - I have now re-enabled all the config changes so I’m back to my initial config and have been running for four days without a problem.

I even reproduced the problem on two differerent physical Pi’s (to eliminate HW) before starting to dissect the config…

So for now I can only conclude that something external was triggering these crashes!

I can now go back to the latest OS and start adding more stuff to my config (and keep my fingers crossed that I no longer encounter what-ever it was…)

Thank you for your help!

sdetweil

@tve interesting. keep us informed…

thgmirror

@TvE Do you use some wires at the GPIO? For example a PIR-sensor? Than may be interferences causes the issue when an electronic device is around your PI.

Or it is the temperature? Check it with
vcgencmd measure_temp

TvE

@sdetweil Will do!

@thgmirror Nope no PIR sensor (as in “not yet”)
My setup is as basic “Hello World!” as it can be (IMHO)
(The full config can be seen - as a diff to the sample - earlier in this thread)

The pi have been placed horisontally on my desk during all this testing.
It is in a cabinet though (the one from the “startet kit”).

I have measured the temperature (with my hand) and not been alarmed
Now I can validate my assumption based on your tip, thanx!

pi@mirror:~ $ vcgencmd measure_temp
temp=63.4'C

I have read that it’s possible to get add a cooling fin to the CPU, which is something I’ll look in to later (as I am also working on making a replacement for this guy, an image grabber [1].
You can no longer purchase it (and I can make more captures pr second on my pi) for 1/2 the price, I only need to add this thingy [2]
I have HelloWorld! for this, I “just” need to add

Capture to RAM disk in stead of USB
Auto upload to an ftp share (or maybe change this to scp…)
Maybe get the pi’s to netboot as well (as I need 10+ of the system)

But that’s completely irellevant to my MM problem ;-)

[1 ]https://www.epiphan.com/wp-content/uploads/2014/07/epiphan-vga2ethernet-brochure.pdf
[2] https://www.epiphan.com/products/avio-4k/

thgmirror

@tve 63.4 C? Could be less!
You can generate a simple protocol by using
crontab -e and enter

@reboot echo “---------- start ----------” >> proto.txt
*/5 * * * * date >> proto.txt
1-56/5 * * * * vcgencmd measure_temp >> proto.txt

which will write a timestamp and a celsius amount every 5 minutes…and an eyecatcher at every boot. Look after a while into the file.

TvE

@thgmirror A cron based measuring is a good approach I’ll keep that in mind if I start to suspect temperature (or when I perform a load test my grabber-PI).

For now I do not think it can be temperature related as it’s very reliable around 60 degrees centigrade

pi@mirror:~ $ vcgencmd measure_temp
temp=59.6'C