Read the statement by Michael Teeuw here.
Every few hours I get "ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service"
-
Hi,
My second post here - I found MagicMirror2 [MM]and think it’s awsome, hence I am trying to get it up and running.I have installed it (manually as descriped on GH) on a PI3 and it works - well only for some (4-10’ish) hours, then MM it stops and I see this as the root cause:
[5932:0227/150019.172517:ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service.I consider my setup really plain vanilla - as all I have done is
- Install latest version (as of a few days ago)
- NOT added any modules (yet)
- Only configured the out-of-the-box modules (so they present relevant data)
Is this a normal/expected behavior?
Meaning that one needs to also make a watchdog script that keeps relauncing MM in case it aborts like this or…?Searching for this error I see a lot of electron related hits, but only one hit in MM forum (which appears to be related to an added module, thus a different case than mine)
I launch MM manually via SSH
cd MagicMirror npm run start 1 > logs/log-stdout.log 2 > logs/log-err.log &STDOUT LOG:
> magicmirror@2.18.0 start > DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js [27.02.2022 10:41.09.077] [LOG] Starting MagicMirror: v2.18.0 [27.02.2022 10:41.09.095] [LOG] Loading config ... [27.02.2022 10:41.09.106] [LOG] Loading module helpers ... [27.02.2022 10:41.09.110] [LOG] No helper found for module: alert. [27.02.2022 10:41.09.144] [LOG] Initializing new module helper ... [27.02.2022 10:41.09.146] [LOG] Module helper loaded: updatenotification [27.02.2022 10:41.09.148] [LOG] No helper found for module: clock. [27.02.2022 10:41.09.460] [LOG] Initializing new module helper ... [27.02.2022 10:41.09.463] [LOG] Module helper loaded: calendar [27.02.2022 10:41.09.465] [LOG] No helper found for module: compliments. [27.02.2022 10:41.09.467] [LOG] No helper found for module: weather. [27.02.2022 10:41.09.530] [LOG] Initializing new module helper ... [27.02.2022 10:41.09.531] [LOG] Module helper loaded: newsfeed [27.02.2022 10:41.09.532] [LOG] All module helpers loaded. [27.02.2022 10:41.09.734] [LOG] Starting server on port 8080 ... [27.02.2022 10:41.09.763] [LOG] Server started ... [27.02.2022 10:41.09.765] [LOG] Connecting socket for: updatenotification [27.02.2022 10:41.09.766] [LOG] Starting module helper: updatenotification [27.02.2022 10:41.09.767] [LOG] Connecting socket for: calendar [27.02.2022 10:41.09.769] [LOG] Starting node helper for: calendar [27.02.2022 10:41.09.770] [LOG] Connecting socket for: newsfeed [27.02.2022 10:41.09.772] [LOG] Starting node helper for: newsfeed [27.02.2022 10:41.09.772] [LOG] Sockets connected & modules started ... [27.02.2022 10:41.10.311] [LOG] Launching application. [27.02.2022 10:41.13.786] [LOG] Create new calendarfetcher for url: http://calendars.icloud.com/holidays/dk_da.ics - Interval: 300000 [27.02.2022 10:41.13.863] [LOG] Create new newsfetcher for url: https://www.dr.dk/nyheder/service/feeds/senestenyt - Interval: 300000 [27.02.2022 10:41.13.941] [INFO] Checking git for module: default [27.02.2022 10:41.14.335] [INFO] Newsfeed-Fetcher: Broadcasting 20 items. ... [27.02.2022 14:57.47.128] [INFO] Calendar-Fetcher: Broadcasting 10 events. [27.02.2022 15:00.19.100] [LOG] Shutting down server... [27.02.2022 15:00.19.128] [LOG] Stopping module helper: updatenotification [27.02.2022 15:00.19.130] [LOG] Stopping module helper: calendar [27.02.2022 15:00.19.132] [LOG] Stopping module helper: newsfeedSTDERR LOG:
[27.02.2022 10:41.09.749] [WARN] ^[[33mYou're using a full whitelist configuration to allow for all IPs^[[39m [5963:0227/104111.264996:ERROR:viz_main_impl.cc(161)] Exiting GPU process due to errors during initialization [6009:0227/104111.601233:ERROR:sandbox_linux.cc(376)] InitializeSandbox() called with multiple threads in process gpu-process. [5932:0227/150019.172517:ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service. [5932:0227/150019.198860:ERROR:zygote_communication_linux.cc(276)] Failed to send GetTerminationStatus message to zygote [5932:0227/150019.210928:ERROR:zygote_communication_linux.cc(276)] Failed to send GetTerminationStatus message to zygoteThe version I am using:
pi@magicmirror:~/MagicMirror $ git log -1 commit a6cbc9f0ef46e4ba45e58e17f64f74410d8fe0b5 (HEAD -> master, tag: v2.18.0, origin/master, origin/HEAD) Merge: 222a5f37 be073daa Author: Michael Teeuw <michael@xonaymedia.nl> Date: Sat Jan 1 19:31:41 2022 +0100 Merge pull request #2761 from MichMich/develop release v2.18.0 pi@magicmirror:~/MagicMirror $ npm -v 8.3.1 pi@magicmirror:~/MagicMirror $ node -v v16.14.0 pi@magicmirror:~/MagicMirror $ ./node_modules/.bin/electron --version v16.0.5 pi@magicmirror:~/MagicMirror $The config I’m using:
pi@magicmirror:~/MagicMirror $ npm run config:check > magicmirror@2.18.0 config:check > node js/check_config.js [28.02.2022 08:37.10.501] [INFO] Checking file... /home/pi/MagicMirror/config/config.js [28.02.2022 08:37.10.639] [INFO] Your configuration file doesn't contain syntax errors :)pi@magicmirror:~/MagicMirror $ diff --suppress-common-lines config/config.js.sample config/config.js 11c11,12 < address: "localhost", // Address to listen on, can be: --- > //address: "localhost", // Address to listen on, can be: > address: "0.0.0.0", // Address to listen on, can be: 19c20,21 < ipWhitelist: ["127.0.0.1", "::ffff:127.0.0.1", "::1"], // Set [] to allow all IP addresses --- > //ipWhitelist: ["127.0.0.1", "::ffff:127.0.0.1", "::1"], // Set [] to allow all IP addresses > ipWhitelist: [], // Set [] to allow all IP addresses 29,30c31,34 < language: "en", < locale: "en-US", --- > //language: "en", > language: "da", > //locale: "en-US", > locale: "da-DK", 54c58 < header: "US Holidays", --- > header: "Helligdage", 60c64,65 < url: "webcal://www.calendarlabs.com/ical-calendar/ics/76/US_Holidays.ics" --- > //url: "webcal://www.calendarlabs.com/ical-calendar/ics/43/denmark-holidays.ics" > url: "webcal://calendars.icloud.com/holidays/dk_da.ics" 75,77c80,82 < location: "New York", < locationID: "5128581", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city < apiKey: "YOUR_OPENWEATHER_API_KEY" --- > location: "Copenhagen", > locationID: "2618425", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city > apiKey: "REDACTED" 83c88 < header: "Weather Forecast", --- > header: "Vejret", 87,89c92,94 < location: "New York", < locationID: "5128581", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city < apiKey: "YOUR_OPENWEATHER_API_KEY" --- > location: "Copenhagen", > locationID: "2618425", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city > apiKey: "REDACTED" 98,99c103,104 < title: "New York Times", < url: "https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml" --- > title: "Seneste nyt fra DR", > url: "https://www.dr.dk/nyheder/service/feeds/senestenyt" -
I initially posted this in the wrong categori (Modules -> Troubleshooting)
So if it looks familiar that’s why (sorry for the confusion…) -
@tve the 1st two errors in the stderr log are shown as a bug in electron.
nothing we can do.
no idea on the rest.
which os image did u use?
lsb_release -a
I am using the legacy release on a pi3b and it has been running mm for 2 weeks now, uninterrupted.
-
@sdetweil Sorry - I forgot that (latest OS :-))
pi@magicmirror:~/MagicMirror/logs $ lsb_release -a No LSB modules are available. Distributor ID: Raspbian Description: Raspbian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye pi@magicmirror:~/MagicMirror/logs $I was not worried about the two first errors as they are always there (if they were serious I suspect that I would not have been able to launch MM at all)
The legacy release - what’s that (I should try that release and see if I can get the same positive results as you)
…googled…
It must be this you’re referring to?
https://www.tomshardware.com/news/raspberry-pi-releases-legacy-osI gotta try that OS tomorow and se if that’s better
-
-
@sdetweil
Test is now started on legacy PI OS - so by tomorrow I’ll know if I have a stable setup- Goal 1 = 4 h (my shortest runtime with bullseye)
- Goal 2 = 16 h (my longest runtime with bullseye)
- Goal 3 = 24 h (if I get here I can expect that it’ll continue for ever :-) )
-
Hmm - still unstable
After just 1,5 hours I get the same error
[1229:0301/105243.836236:ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service.Start test at 08:28 [01.03.2022 08:28.26.868] [LOG] Starting MagicMirror: v2.18.0 ... Problem kicks in at 10:52 [01.03.2022 10:52.43.788] [LOG] Shutting down server...Could it really be problematic that I have configured both wlan as well as the eth interface??? (that would be hard to accept…)
Currently I am using eth, but prepared wlan as that’s what I’ll use when I “go to production”.
-
@tve well, address:“0.0.0.0”
means listen on ALL interfaces.
I use it on my pi w both enabled, but only Ethernet connected. ( and wifi not configured)
never has ‘crashed’
you could change to just the ethernet address
address:“192.168.?.?”
whatever that address is.
ifconfig
will show u
or
ip addr -
@sdetweil OK - good point, that’s worth trying
New test has started only listening on one IP :-) -
Hmm crashed after 6,5h with the same error…
NB.: My symptom is the same as being reported here (though I’m not using a docker container)
https://github.com/electron/electron/issues/31675Is there any way of increasing the verbosity in any logs to get more debugging info?
I’m now perforing a new test:
- Reverting to the default MM config
- And tcpdump’ing the network traffic so I can correlate and maybe identify what triggers the network service crash
If it no longer crashes I’ll revert to my own config and retest (as it then must be related to that specific network traffic [no matter how unfair that seems]
-
Test with default config looks good 11h without a crash
[02.03.2022 07:43.20.529] [LOG] Starting MagicMirror: v2.18.0 ... [02.03.2022 18:35.01.238] [INFO] Calendar-Fetcher: Broadcasting 10 events. And still runningThus I must deduce that one of the two services I had configured was triggering the problem
I must enable my two changes individually to identify which is “bad”
-
@tve weird
-
@sdetweil Yep (but I get exposed to that type of behavior daily :-) )
I have also found the DEBUG setting in the conf file:
logLevel: ["INFO", "LOG", "WARN", "ERROR", "DEBUG"], // Add "DEBUG" for even more loggingAnd am tcpdumping the traffic as well
sudo tcpdump -i wlan0 -w tcpdump-wlan.pcap && sudo tcpdump -i eth0 -w tcpdump-eth.pcap &So let’s see what a bisect approach to disabling the “services” leads to
-
@tve yeh, sorry on debug. lots of calendar stuff
-
@sdetweil Don’t worry about “debug” I just had to read the config in stead of skipping over that part
I have now reverted to my original config and will enable the different service in bisect style.
/* TvE test log (on legacy OS) Re-configured services resulting in crash: 1 calendar 2 openweather_current 3 openweather_forecast 4 news A: Sample config => 11h and still OK B: Enable DEBUG log level C: Disable 2+3 (openweather) Enable 1+4 => 14 h and still OK D: Enable 2+3 (openweather) + disable 1+4 (news & calendar) => ongoing */So in a few days I should know what’s triggering the issue…
-
Hmm so my latest config ran for more than 2 days.
I’m now reverting to the initial config (enabling all four config-changes) and then time will tell
-
@sdetweil This is really getting intersting - I have now re-enabled all the config changes so I’m back to my initial config and have been running for four days without a problem.
I even reproduced the problem on two differerent physical Pi’s (to eliminate HW) before starting to dissect the config…
So for now I can only conclude that something external was triggering these crashes!
I can now go back to the latest OS and start adding more stuff to my config (and keep my fingers crossed that I no longer encounter what-ever it was…)
Thank you for your help!
-
@tve interesting. keep us informed…
-
@TvE Do you use some wires at the GPIO? For example a PIR-sensor? Than may be interferences causes the issue when an electronic device is around your PI.
Or it is the temperature? Check it with
vcgencmd measure_temp -
@sdetweil Will do!
@thgmirror Nope no PIR sensor (as in “not yet”)
My setup is as basic “Hello World!” as it can be (IMHO)
(The full config can be seen - as a diff to the sample - earlier in this thread)The pi have been placed horisontally on my desk during all this testing.
It is in a cabinet though (the one from the “startet kit”).I have measured the temperature (with my hand) and not been alarmed
Now I can validate my assumption based on your tip, thanx!pi@mirror:~ $ vcgencmd measure_temp temp=63.4'CI have read that it’s possible to get add a cooling fin to the CPU, which is something I’ll look in to later (as I am also working on making a replacement for this guy, an image grabber [1].
You can no longer purchase it (and I can make more captures pr second on my pi) for 1/2 the price, I only need to add this thingy [2]
I have HelloWorld! for this, I “just” need to add- Capture to RAM disk in stead of USB
- Auto upload to an ftp share (or maybe change this to scp…)
- Maybe get the pi’s to netboot as well (as I need 10+ of the system)
But that’s completely irellevant to my MM problem ;-)
[1 ]https://www.epiphan.com/wp-content/uploads/2014/07/epiphan-vga2ethernet-brochure.pdf
[2] https://www.epiphan.com/products/avio-4k/
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login