MagicMirrorOS build intermittently not starting mm container (Version 2.30)

[sdetweil]

@scottwalsh can you show w MagicMirror running…

[scottwalsh]

An observation, if I power cycle the pi, there is a good chance of it not starting up with the errors in the original post.

Once leave it for five min for all the processes to start (and fail), then run docker compose up -d --force-recreate, it seems to be stable from then on and works well.

[scottwalsh]

@sdetweil said in MagicMirrorOS build intermittently not starting mm container (Version 2.30):

@scottwalsh can you show w MagicMirror running…

scottwalsh@calendar:/opt/mm/run $ 
scottwalsh@calendar:/opt/mm/run $ docker compose up -d --force-recreate
[+] Running 5/5
 ✔ Container watchtower  Started                                                                                                                                                                        42.0s 
 ✔ Container labwc       Started                                                                                                                                                                        41.9s 
 ✔ Container init        Started                                                                                                                                                                        42.2s 
 ✔ Container mmpm        Started                                                                                                                                                                        42.4s 
 ✔ Container mm          Started                                                                                                                                                                         5.3s 
scottwalsh@calendar:/opt/mm/run $ 
scottwalsh@calendar:/opt/mm/run $ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED         STATUS                   PORTS                                                                                                                                                 NAMES
a1cb1b400de1   karsten13/mmpm:latest          "/home/node/entrypoi…"   3 minutes ago   Up 2 minutes             0.0.0.0:6789->6789/tcp, :::6789->6789/tcp, 0.0.0.0:7890-7891->7890-7891/tcp, :::7890-7891->7890-7891/tcp, 0.0.0.0:8907->8907/tcp, :::8907->8907/tcp   mmpm
f7ec5d32f0bd   karsten13/magicmirror:latest   "/usr/bin/tini -- ./…"   3 minutes ago   Up About a minute                                                                                                                                                              mm
74193ae14fbf   karsten13/labwc:latest         "/entrypoint.sh"         3 minutes ago   Up 2 minutes                                                                                                                                                                   labwc
bcf30cd7bb10   containrrr/watchtower:latest   "/watchtower --inter…"   3 minutes ago   Up 2 minutes (healthy)   8080/tcp                                                                                                                                              watchtower
scottwalsh@calendar:/opt/mm/run $ 
scottwalsh@calendar:/opt/mm/run $ free -m
               total        used        free      shared  buff/cache   available
Mem:             425         236         134          36         141         189
Swap:           1023         296         727
scottwalsh@calendar:/opt/mm/run $

[sdetweil]

@scottwalsh probably swap file speed bound

[karsten13]

@scottwalsh

the script /opt/mm/install/install.sh is executed with every reboot.

As workaround you could add

sleep 10s
docker compose up -d --force-recreate

at the end (where you have to play with the sleep seconds).

[scottwalsh]

@karsten13 said in MagicMirrorOS build intermittently not starting mm container (Version 2.30):

the script /opt/mm/install/install.sh is executed with every reboot.

As workaround you could add

Is that executed on the OS boot, or by one of the containers as they start?

I was thinking of a workaround like that to introduce a pause. Given the low machine spec, it seems to be busy for about 5min after sshd starts, so will just try a 300 sec sleep.

[karsten13]

@scottwalsh

on OS boot

[scottwalsh]

As an update, based on the pointers from @KARSTEN13 and @SDETWEIL, have a workaround.

Had three issues going on.

1:
I found that on a graceful reboot (shutdown -r now) that it would generally start fine, but not on an ungraceful reboot (power cycle the machine).

Looks like when it was ungraceful reboot, the labwc container would show with uptime of (for example) five hours when ‘docker ps’, even though the machine was rebooted 20min ago after ‘docker compose up’ was executed by install.sh. Whereas the magicmirror container would have recently started.

I assume kind of lock file confusing things?

Workaround was to change the first ‘docker compose up’ install.sh to ‘docker compose up -d --force-recreate’

2:
I’d also included a pause and --force-recreate at the end of the install.sh and noted it was never executing as per @KARSTEN13’s suggestion.
Looked like the install.sh script was hanging at docker compose pull.
Commented that out.

3:
Lastly, the initial ‘docker compose up -d --force-recreate’ in install.sh would still fail often with the mm container restarting with logs as per the original post.

With a 7 min pause and then second ‘docker compose up -d --force-recreate’ it seems to start the second time well (3min wasn’t enough, 5min was better, 7min seems to work).

Workaround is a bit on the slow side, at some point will see if can refactor it a bit so that a pull for updates occurs.

Extract of resulting install.sh below.

_info "--> Pulling docker images and starting magicmirror"
# need sudo for docker here if docker was installed with this script

# use up so mm can start if there are already local images
_info "--> Pre Start Pause for 2min"
sleep 120

_info "--> Start and recreate"
$_sudo docker compose up -d --force-recreate

#_info "--> Pull Pause for 1 min"
#sleep 60
# pull new images
#_info "--> Image Pull"
#$_sudo docker compose pull

#_info "--> Restart Pause for 1 min"
#sleep 60
# restart (only if new images pulled)
#_info "--> Restart stopped"
#$_sudo docker compose up -d

_info "--> Final Pause for 7 min"
sleep 720

_info "--> Final  start and recreate"
$_sudo docker compose up -d --force-recreate

if [[ "$_sudo" == "sudo" ]]; then
  _info "--> Reboot needed, starting in 120 sec. (use ctrl-c to skip)"
  sleep 120
  sudo reboot now
fi

# cleanup
$_sudo docker image prune -f