Read the statement by Michael Teeuw here.
Every few hours I get "ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service"
-
@sdetweil OK - good point, that’s worth trying
New test has started only listening on one IP :-) -
Hmm crashed after 6,5h with the same error…
NB.: My symptom is the same as being reported here (though I’m not using a docker container)
https://github.com/electron/electron/issues/31675Is there any way of increasing the verbosity in any logs to get more debugging info?
I’m now perforing a new test:
- Reverting to the default MM config
- And tcpdump’ing the network traffic so I can correlate and maybe identify what triggers the network service crash
If it no longer crashes I’ll revert to my own config and retest (as it then must be related to that specific network traffic [no matter how unfair that seems]
-
Test with default config looks good 11h without a crash
[02.03.2022 07:43.20.529] [LOG] Starting MagicMirror: v2.18.0 ... [02.03.2022 18:35.01.238] [INFO] Calendar-Fetcher: Broadcasting 10 events. And still runningThus I must deduce that one of the two services I had configured was triggering the problem
I must enable my two changes individually to identify which is “bad”
-
@tve weird
-
@sdetweil Yep (but I get exposed to that type of behavior daily :-) )
I have also found the DEBUG setting in the conf file:
logLevel: ["INFO", "LOG", "WARN", "ERROR", "DEBUG"], // Add "DEBUG" for even more loggingAnd am tcpdumping the traffic as well
sudo tcpdump -i wlan0 -w tcpdump-wlan.pcap && sudo tcpdump -i eth0 -w tcpdump-eth.pcap &So let’s see what a bisect approach to disabling the “services” leads to
-
@tve yeh, sorry on debug. lots of calendar stuff
-
@sdetweil Don’t worry about “debug” I just had to read the config in stead of skipping over that part
I have now reverted to my original config and will enable the different service in bisect style.
/* TvE test log (on legacy OS) Re-configured services resulting in crash: 1 calendar 2 openweather_current 3 openweather_forecast 4 news A: Sample config => 11h and still OK B: Enable DEBUG log level C: Disable 2+3 (openweather) Enable 1+4 => 14 h and still OK D: Enable 2+3 (openweather) + disable 1+4 (news & calendar) => ongoing */So in a few days I should know what’s triggering the issue…
-
Hmm so my latest config ran for more than 2 days.
I’m now reverting to the initial config (enabling all four config-changes) and then time will tell
-
@sdetweil This is really getting intersting - I have now re-enabled all the config changes so I’m back to my initial config and have been running for four days without a problem.
I even reproduced the problem on two differerent physical Pi’s (to eliminate HW) before starting to dissect the config…
So for now I can only conclude that something external was triggering these crashes!
I can now go back to the latest OS and start adding more stuff to my config (and keep my fingers crossed that I no longer encounter what-ever it was…)
Thank you for your help!
-
@tve interesting. keep us informed…
-
@TvE Do you use some wires at the GPIO? For example a PIR-sensor? Than may be interferences causes the issue when an electronic device is around your PI.
Or it is the temperature? Check it with
vcgencmd measure_temp -
@sdetweil Will do!
@thgmirror Nope no PIR sensor (as in “not yet”)
My setup is as basic “Hello World!” as it can be (IMHO)
(The full config can be seen - as a diff to the sample - earlier in this thread)The pi have been placed horisontally on my desk during all this testing.
It is in a cabinet though (the one from the “startet kit”).I have measured the temperature (with my hand) and not been alarmed
Now I can validate my assumption based on your tip, thanx!pi@mirror:~ $ vcgencmd measure_temp temp=63.4'CI have read that it’s possible to get add a cooling fin to the CPU, which is something I’ll look in to later (as I am also working on making a replacement for this guy, an image grabber [1].
You can no longer purchase it (and I can make more captures pr second on my pi) for 1/2 the price, I only need to add this thingy [2]
I have HelloWorld! for this, I “just” need to add- Capture to RAM disk in stead of USB
- Auto upload to an ftp share (or maybe change this to scp…)
- Maybe get the pi’s to netboot as well (as I need 10+ of the system)
But that’s completely irellevant to my MM problem ;-)
[1 ]https://www.epiphan.com/wp-content/uploads/2014/07/epiphan-vga2ethernet-brochure.pdf
[2] https://www.epiphan.com/products/avio-4k/ -
@tve 63.4 C? Could be less!
You can generate a simple protocol by using
crontab -e and enter@reboot echo “---------- start ----------” >> proto.txt
*/5 * * * * date >> proto.txt
1-56/5 * * * * vcgencmd measure_temp >> proto.txtwhich will write a timestamp and a celsius amount every 5 minutes…and an eyecatcher at every boot. Look after a while into the file.
-
@thgmirror A cron based measuring is a good approach I’ll keep that in mind if I start to suspect temperature (or when I perform a load test my grabber-PI).
For now I do not think it can be temperature related as it’s very reliable around 60 degrees centigrade
pi@mirror:~ $ vcgencmd measure_temp temp=59.6'C -
@tve but see his post
https://forum.magicmirror.builders/topic/16485/temperature-of-a-rpi3
60 is bad if u can do 30!
-
@sdetweil said in Every few hours I get "ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service":
@tve but see his post
https://forum.magicmirror.builders/topic/16485/temperature-of-a-rpi3
60 is bad if u can do 30!
Great post(!)
(And here I thought that 60 was the normal/expected working temperature)
I think I’ll add a passive solution (or maybe even an active - but then there is a fan that will die)
(I have some other - rather expensive - equipment that died due to a dead fan, and unfortunately there was no way to measure the temperature - bad design…)I see these two obvious options:
- https://uk.rs-online.com/web/p/raspberry-pi-hats-add-ons/8679039
- https://uk.rs-online.com/web/p/raspberry-pi-hats-add-ons/2020449
The active solution might even add enough cooling even when the fan dies (as it seems that there is a passive “path for the heat” from the CPU
I should try both :-)What is to gain?:
I am aware that the warmer the chip the shorter it’s life
(I have complained to a vendor that one of their products got so warm that I burn myself if I touch the enclosure, but they insist that it’s expected - even though I have had to get at least three of my nine devices replaced!
I think it’s due to a bad design - hence my heat complaint…)The rule of thumb is something like this:
By increasing the device temperature by just 10°C, we have reduced the lifetime by over 2x
BTW: I have not seen any more network-service-instance-crashes, it’s still going strong after 8 days, really weird (but good!)…
-
63.4 C? Could be less!
I have now added one of these [1] to my RPI 3 and returned it to the enclosure (the “starter kit cabinet”).
The temperature is now 48-49 degrees centigrade after 30 min usage of MM.
So it reduces the temperature with ~14K - thats nice (if you don’t mind listening to a spinning fan which is actually fairly loud in at least 2 meters distance)…!I’ll try passive cooling later…
Oh! - and I had a 14 day uptime before powering off (the original reason for this thread :-)
[1] https://uk.rs-online.com/web/p/raspberry-pi-hats-add-ons/2020449
-
@TvE said in Every few hours I get "ERROR:network_service_instance_impl.cc(916)] Network service crashed, restarting service":
Oh! - and I had a 14 day uptime before powering off (the original reason for this thread
nice!!
-
@sdetweil Yeah - that’s perfect (and expected)
I really have (now stopped) wondering what happened initially :-)@thgmirror
I made a quick test where I have removed the power to the fan to see how much that changes the temperature (aka a passive cooling test).- After ca. one hour the temperature rises to a max of 57-58 degrees centigrades
- For the next hour the temperature is not rising any more
- Thus a reduction of ~6K
NB.: I have now removed the lid to see how big (small) a difference that makes
-
So - without the lid I get another 2K as - after 15 min - the temperature stabilized around 54-55 degrees centigrades
* Passive cooling lid on : 63 -> 58 = -5 * Passive cooling lid off : 63 -> 55 = -8 * Active cooling lid on : 63 -> 49 = -14 * Active cooling lid off : 63 -> 42 = -21
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login