Every few hours I get "ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service"

TvE

@sdetweil said in Every few hours I get "ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service":

@tve but see his post

https://forum.magicmirror.builders/topic/16485/temperature-of-a-rpi3

60 is bad if u can do 30!

Great post(!)

(And here I thought that 60 was the normal/expected working temperature)

I think I’ll add a passive solution (or maybe even an active - but then there is a fan that will die)
(I have some other - rather expensive - equipment that died due to a dead fan, and unfortunately there was no way to measure the temperature - bad design…)

I see these two obvious options:

The active solution might even add enough cooling even when the fan dies (as it seems that there is a passive “path for the heat” from the CPU
I should try both :-)

What is to gain?:
I am aware that the warmer the chip the shorter it’s life
(I have complained to a vendor that one of their products got so warm that I burn myself if I touch the enclosure, but they insist that it’s expected - even though I have had to get at least three of my nine devices replaced!
I think it’s due to a bad design - hence my heat complaint…)

The rule of thumb is something like this:

By increasing the device temperature by just 10°C, we have reduced the lifetime by over 2x

Sources:
https://jetcool.com/post/semiconductor-lifetime-how-temperature-affects-mean-time-to-failure-device-reliability/

https://www.electronics-cooling.com/2017/08/10c-increase-temperature-really-reduce-life-electronics-half/#:~:text=If one were to ask,reduces component life by half”.

BTW: I have not seen any more network-service-instance-crashes, it’s still going strong after 8 days, really weird (but good!)…

TvE

@thgmirror

63.4 C? Could be less!

I have now added one of these [1] to my RPI 3 and returned it to the enclosure (the “starter kit cabinet”).

The temperature is now 48-49 degrees centigrade after 30 min usage of MM.
So it reduces the temperature with ~14K - thats nice (if you don’t mind listening to a spinning fan which is actually fairly loud in at least 2 meters distance)…!

I’ll try passive cooling later…

Oh! - and I had a 14 day uptime before powering off (the original reason for this thread :-)

[1] https://uk.rs-online.com/web/p/raspberry-pi-hats-add-ons/2020449

sdetweil

@TvE said in Every few hours I get "ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service":

Oh! - and I had a 14 day uptime before powering off (the original reason for this thread

nice!!

TvE

@sdetweil Yeah - that’s perfect (and expected)
I really have (now stopped) wondering what happened initially :-)

@thgmirror
I made a quick test where I have removed the power to the fan to see how much that changes the temperature (aka a passive cooling test).

After ca. one hour the temperature rises to a max of 57-58 degrees centigrades
For the next hour the temperature is not rising any more
Thus a reduction of ~6K

NB.: I have now removed the lid to see how big (small) a difference that makes

TvE

So - without the lid I get another 2K as - after 15 min - the temperature stabilized around 54-55 degrees centigrades

* Passive cooling lid  on : 63 -> 58 =  -5
* Passive cooling lid off : 63 -> 55 =  -8
* Active  cooling lid  on : 63 -> 49 = -14
* Active  cooling lid off : 63 -> 42 = -21

Ivanov_d

@TvE I am in the same spot as you.

Raspberry Pi 3b+
Fresh installed Raspbian Bullseye 11
Fresh install of MM with vanilla config and modules - crashes every minute and restarts itself. Are you saying that the config might be the problem?

0|MagicMirror  | Launching application.
0|MagicMirror  | [18725:0322/102613.376542:ERROR:viz_main_impl.cc(161)] Exiting GPU process due to errors during initialization
0|MagicMirror  | [18766:0322/102613.700219:ERROR:sandbox_linux.cc(376)] InitializeSandbox() called with multiple threads in process gpu-process.
0|MagicMirror  | [22.03.2022 10:26.16.088] [LOG]   
0|MagicMirror  | Create new calendarfetcher for url: http://www.calendarlabs.com/ical-calendar/ics/76/US_Holidays.ics - Interval: 300000
0|MagicMirror  | [22.03.2022 10:26.16.175] [LOG]   
0|MagicMirror  | Create new newsfetcher for url: https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml - Interval: 300000
0|MagicMirror  | [22.03.2022 10:26.16.248] [INFO]  
0|MagicMirror  | Checking git for module: default
0|MagicMirror  | [22.03.2022 10:26.17.014] [INFO]  
0|MagicMirror  | Newsfeed-Fetcher: Broadcasting 61 items.
0|MagicMirror  | [22.03.2022 10:26.17.522] [INFO]  
0|MagicMirror  | Calendar-Fetcher: Broadcasting 10 events.
0|MagicMirror  | [22.03.2022 10:27.33.823] [LOG]   Shutting down server...
0|MagicMirror  | [22.03.2022 10:27.33.833] [LOG]   Stopping module helper: updatenotification
0|MagicMirror  | [22.03.2022 10:27.33.835] [LOG]   Stopping module helper: calendar
0|MagicMirror  | [22.03.2022 10:27.33.848] [LOG]   Stopping module helper: newsfeed
0|MagicMirror  | [18692:0322/102733.909781:ERROR:zygote_communication_linux.cc(276)] Failed to send GetTerminationStatus message to zygote
0|MagicMirror  | [18692:0322/102733.942607:ERROR:zygote_communication_linux.cc(276)] Failed to send GetTerminationStatus message to zygote
0|MagicMirror  | [18692:0322/102733.958362:ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service.
0|MagicMirror  | [18692:0322/102733.974458:ERROR:gpu_process_host.cc(962)] GPU process launch failed: error_code=1002
0|MagicMirror  | [18692:0322/102733.982862:ERROR:gpu_process_host.cc(962)] GPU process launch failed: error_code=1002
0|MagicMirror  | [18692:0322/102733.989821:ERROR:gpu_process_host.cc(962)] GPU process launch failed: error_code=1002
0|MagicMirror  | [18692:0322/102733.998527:ERROR:gpu_process_host.cc(962)] GPU process launch failed: error_code=1002
0|MagicMirror  | > magicmirror@2.18.0 start
0|MagicMirror  | > DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js
0|MagicMirror  | [22.03.2022 10:27.42.385] [LOG]   
0|MagicMirror  | Starting MagicMirror: v2.18.0
0|MagicMirror  | [22.03.2022 10:27.42.407] [LOG]   
0|MagicMirror  | Loading config ...
0|MagicMirror  | [22.03.2022 10:27.42.421] [LOG]   
0|MagicMirror  | Loading module helpers ...
0|MagicMirror  | [22.03.2022 10:27.42.427] [LOG]   
0|MagicMirror  | No helper found for module: alert.
0|MagicMirror  | [22.03.2022 10:27.42.463] [LOG]

thgmirror

@TvE That is my experimental setup:

TvE

@Ivanov_d
Maybe we’re observing the same - currently it’s hard for me to tell.

You surely have no heat issue (no additional cooling is needed - only to prolong the life of the RPI)
OS
As you hopefully have read in this thread I started with the same OS as you, then “downgraded” to the socalled “legacy” version and still saw the issues. then it suddenly stopped.
Thus I need to replace to the latest OS and retest

I have no logical reason to suspect the OS but I need to do the test to be 100% certain…

You can follow my test steps (using bisection - remove the handfull of external calls and see which triggers the problem)
My guess
So far I can only conclude that something external provoked the network stack to crash.
In the end of my testing () I had returned to my original config and no longer see problems
That something must relate to one of the two services I was using (part of the vanilla config)

UPDATE!!! - I just looked at my test and wow, this morning I had an error again (my test runs with the monitor turned off, thus I did not see this (I’m working on other tasks at the moment…).

Looking in the logs I see calendar related issues a few hours before the service stopped

My error occurred at [21.03.2022 22:39.02.219] in Denmark
Your error occurred at [22.03.2022 10:27.33.823] in US? (guess deduced from your config)
That way too close to just be a coincidence…

I don’t know for how long you have been running your system, but I will now start to suspect that the root cause lies in the calendar service and we both are affected by the same root cause

I wonder if we can find another source for those data

(I now regret that I turned of tcpdump for my latest testing…)

stdout

[21.03.2022 22:34.06.084] [INFO]  Newsfeed-Fetcher: Broadcasting 20 items.
[21.03.2022 22:39.02.219] [LOG]   Shutting down server...
[21.03.2022 22:39.02.225] [LOG]   Stopping module helper: updatenotification
[21.03.2022 22:39.02.226] [LOG]   Stopping module helper: calendar
[21.03.2022 22:39.02.227] [LOG]   Stopping module helper: newsfeed

err

[21.03.2022 16:43.52.620] [WARN]  ^[[33mYou're using a full whitelist configuration to allow for all IPs^[[39m
[1002:0321/164356.019064:ERROR:viz_main_impl.cc(161)] Exiting GPU process due to errors during initialization
[1040:0321/164356.860589:ERROR:sandbox_linux.cc(376)] InitializeSandbox() called with multiple threads in process gpu-process.
[21.03.2022 17:37.30.487] [ERROR] Calendar Error. Could not fetch calendar:  http://calendars.icloud.com/holidays/dk_da.ics FetchError: request to https://calendars.icloud.com/holidays/dk_da.ics failed, reason: connect ETIMEDOUT 17.248.150.146:443
    at ClientRequest.<anonymous> (/home/pi/MagicMirror/node_modules/node-fetch/lib/index.js:1483:11)
    at ClientRequest.emit (node:events:394:28)
    at TLSSocket.socketErrorListener (node:_http_client:447:9)
    at TLSSocket.emit (node:events:394:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  type: 'system',
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT'
}
[21.03.2022 17:44.40.539] [ERROR] Calendar Error. Could not fetch calendar:  http://calendars.icloud.com/holidays/dk_da.ics FetchError: request to https://calendars.icloud.com/holidays/dk_da.ics failed, reason: connect ETIMEDOUT 17.248.150.10:443
    at ClientRequest.<anonymous> (/home/pi/MagicMirror/node_modules/node-fetch/lib/index.js:1483:11)
    at ClientRequest.emit (node:events:394:28)
    at TLSSocket.socketErrorListener (node:_http_client:447:9)
    at TLSSocket.emit (node:events:394:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  type: 'system',
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT'
}
[958:0321/223902.258167:ERROR:zygote_communication_linux.cc(276)] Failed to send GetTerminationStatus message to zygote
[958:0321/223902.279012:ERROR:zygote_communication_linux.cc(276)] Failed to send GetTerminationStatus message to zygote

Ivanov_d

@TvE it’s been running for 25 hours so far:

pm2 status MagicMirror
┌───────────────┬────┬─────────┬──────┬──────┬────────┬─────────┬────────┬─────┬──────────┬──────┬──────────┐
│ App name      │ id │ version │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem      │ user │ watching │
├───────────────┼────┼─────────┼──────┼──────┼────────┼─────────┼────────┼─────┼──────────┼──────┼──────────┤
│   MagicMirror │ 0  │ 2.18.0  │ fork │ 2911 │ online │ 15      │ 25h    │ 0%  │ 2.7 MB   │ pi   │ disabled │
└───────────────┴────┴─────────┴──────┴──────┴────────┴─────────┴────────┴─────┴──────────┴──────┴──────────┘

I am still tweaking my config, however, my approach was the following:

I updated from Rapbian Buster to Bullseye
That broke my Raspberry Pi 3b+'s WiFi connection (seems like a known bug https://forums.raspberrypi.com/viewtopic.php?t=325484), so I removed connman package and installed network-manager and connected to the WiFi. If you install network-manager, you will also want to disable the MAC randomization:
To disable the WiFi MAC randomization, create the following file using the this command:

sudo nano /etc/NetworkManager/conf.d/100-disable-wifi-mac-randomization.conf

and paste the following content inside:

[connection]
wifi.mac-address-randomization=1
[device]
wifi.scan-rand-mac-address=no

Save the changes and continue.

I backed up my MM config and modules folder and started with a fresh install of MM 2.18
I started MM with the default config - no luck, it was crashing every minute
I started removing the default modules and installing the ones that I had before
I was observing the same behavior as you did - the default modules that needed network connection, when fetching data were breaking the network stack somehow … I edited the config to enter my calendar and newsfeed URLs and that seemed to fixed the issue. It might be something in the response of the default calendar and/or newsfeed … but that is just my speculation.

I also have pulled all the latest updates for every module that I had installed previously just to make sure that I am all up to date.

Ever since as you can see, my MM has been running steadily. Definitely not an SoC temp issue, since I never had anything than the stock aluminum radiators on the Pi and it has been running for 3-4 years 24/7 already.

I wouldn’t have decided to upgrade to bullseye if it was not a requirement for the new version of MMM-GoogleAssistant.

Anyways, I hope that this helps.

P.S. I am in Sofia, so the date/time in my log is GMT +02:00

Ivanov_d

@TvE I was jumping for joy way too soon. I took out my SD card to make a backup of it and when I put it back in, MagicMirror started restarting every other minute just like before. No changes were made whatsoever. I am now very confused …