What is your backup and restore method?

BillyTheKid9588 · Dec 18, 2020, 10:34 AM

My mirror configuration has taken some time to get right and i’m curious to know people’s back up and restore methods in case of system failure.

I’ve seen some backup the whole sd, some just the config file, i see advantages and disadvantages to both of these. If the sd is backed up, maybe the module code is out of date, if just the config file, there is the potential to lose Raspbian specific settings.

Curious to know how people approach this.

Snille · Dec 18, 2020, 11:49 AM

Hi there! Manual “backup” of the config, CSS and compliments files when I change them (just copying it to my server). Then when reinstalling, I’m using a script (that I can wget down to the new install) containing all modules to reinstall and it copy back the config, CSS and compliments from the backup location… :)

See script below…

#!/bin/bash

## ----------------------------------------------------------------------------
## Snilles automatic module clone and install script for MM2.
## ----------------------------------------------------------------------------

## Don't forget to add your public SSH key to your GIT profile!
## If you don't, nothing will be cloned!

## If you don't have an SSH key. This is how you get one.
## Generate SSH key(s).
# cd ~
# ssh-keygen -t rsa

## Press: Enter, Enter, Enter...

## See the public Key...
# cat /home/pi/.ssh/id_rsa.pub

## Copy the SSH public key and add it to your GIT SSH keys on GitHUB.

## ----------------------------------------------------------------------------
## Config below.
## ----------------------------------------------------------------------------

# System to install (what config to use).
## Housesystems Server Install
Toinstall='house'
## NUC install (On the actual Mirror, this is not used anymore)
#Toinstall='NUC'

# Real Name of the config file.
ConfigName='config.js'

## The Install Folder.
## Local install.
## On NUC
#User='/home/snille'
## On Raspberry Pi
#User='/home/pi'
## Install om Homesystems Server.
User='/var/www/html/magicmirror'

## The module install directory.
Moddir=$User'/MagicMirror/modules'

## Configuration and other files backup location (from previous installation).
DownloadFrom='https://yoursite.com/mmbackup/'

## Configuration file. 
Files[1]='config-'$Toinstall'.js'
## Where to put the Configuration file.
Dirs[1]=$User'/MagicMirror/config/'

## Custom CSS file.
Files[2]='custom.css'
## Where to put the CSS file.
Dirs[2]=$User'/MagicMirror/css/'

## Custom screen on/off scripts.
Files[3]='screen.sh'
## Where to put the screen script files.
Dirs[3]=$User'/'

## Not used moved to "screen.sh"
#Files[4]='turnoffscreen.sh'
#Dirs[4]=$User'/'
#Files[5]='turnonscreen.sh'
#Dirs[5]=$User'/'

## Custom Magic Mirror update script.
Files[6]='update.sh'
## Where to put the update script file.
Dirs[6]=$User'/'

## Custom compliment files.
Files[7]='compliments-Snille.json'
## Where to put the compliment file.
Dirs[7]=$User'/MagicMirror/modules/default/compliments/'

Files[8]='compliments-Camilla.json'
Dirs[8]=$User'/MagicMirror/modules/default/compliments/'

Files[9]='compliments-Louise.json'
Dirs[9]=$User'/MagicMirror/modules/default/compliments/'

#Files[10]='compliments-Martin.json'
#Dirs[10]=$User'/MagicMirror/modules/default/compliments/'

# Start file for the MagigMirror
Files[11]='MacigMirror.sh'
Dirs[11]=$User'/'

## Package file
Packfile='package.json'

## All the modules repos from Git.
Repos[1]='git@github.com:Snille/MMM-HideAll.git'
Repos[2]='git@github.com:matteodanelli/MMM-cryptocurrency.git'
Repos[3]='git@github.com:basknol/MMM-Globe.git'
Repos[4]='git@github.com:Snille/MMM-Modulebar.git'
Repos[5]='git@github.com:Snille/MMM-ModuleScheduler.git'
Repos[6]='git@github.com:CatoAntonsen/MMM-MotionEye.git'
Repos[7]='git@github.com:Gyran/MMM-plex-recently-added.git'
Repos[8]='git@github.com:Snille/MMM-Profilepicture.git'
Repos[9]='git@github.com:tosti007/MMM-ProfileSwitcher.git'
Repos[10]='git@github.com:Jopyth/MMM-Remote-Control.git'
Repos[11]='git@github.com:Snille/MMM-Sonos.git'
Repos[12]='git@github.com:brobergp/MMM-TextClock.git'
Repos[13]='git@github.com:tosti007/MMM-TouchNavigation.git'
Repos[14]='git@github.com:bugsounet/MMM-Tools.git'
Repos[15]='git@github.com:timdows/MMM-JsonTable.git'
Repos[16]='git@github.com:derekn/MMM-TautulliActivity.git'
#Repos[17]='git@github.com:Snille/MMM-IFTTT.git' # For development of the IFTTT module, the one below is used.
Repos[18]='git@github.com:jc21/MMM-IFTTT.git'
Repos[19]='git@github.com:NolanKingdon/MMM-MoonPhase.git'
Repos[20]='git@github.com:Snille/MMM-homeassistant-sensors.git'
#Repos[21]='git@github.com:schnibel/MMM-Memo.git' # Replaced by module below, I changed some things.
Repos[22]='git@github.com:Snille/MMM-Memo.git'
Repos[23]='git@github.com:almerica/MMM-ImageFit.git'
Repos[24]='git@github.com:Snille/MMM-Videoplayer.git'
Repos[25]='git@github.com:MartinGris/MMM-GoogleMaps-Tracking.git'

## ----------------------------------------------------------------------------

## Start!
cd "$Moddir"
for t in "${Repos[@]}"
	do
		git clone $t
		#echo "Cloning $t"
done

echo "Cloning Done."
echo "Now npm installing..."

for f in *;
	do
		if [[ -d $f ]]; then
			installmodule=$(basename $f)
			[[ $installmodule =~ ^(default|node_modules)$ ]] && continue
			cd "$installmodule"
			if [ -e "$Packfile" ]; then
#				npm install --production
				npm install
			fi
			cd ".."
		fi
	done

echo "Download and install done."
echo "Now restoring backup files from $DownloadFrom."

## Counts through the array of files (and uses the dirs to know where to put them).
for ((i=1; i<=${#Files[@]}; i++)); do
	if [[ $(wget $DownloadFrom${Files[i]} -O-) ]] 2>/dev/null
	then
		wget $DownloadFrom${Files[i]}
		printf 'Moving %s to %s...\n\n' "${Files[i]}" "${Dirs[i]}"
		mv ${Files[i]} ${Dirs[i]}
	fi
done

## Rename the config.
mv ${Dirs[1]}${Files[1]} ${Dirs[1]}$ConfigName

echo "Installation and restore done!"
echo "You can now start your Magic Mirror."

exit

swvalenti · Dec 19, 2020, 3:32 PM

@Snille Can you dumb this down a little bit for me please, trying to use this…I created key and added to github but not sure how to do the rest.

karsten13 · Dec 19, 2020, 10:47 PM

my approach …

The following script must be executed in the magicmirror folder and creates another script restore.sh. This result script should be saved (on my server it is under git control) and can be executed (again in the magicmirror folder) to restore the old system (or bringing the old config to a new system).

#!/bin/sh

base="$(cd "$(dirname "$0")" && pwd)"

restore="$base/restore.sh"
config="${1:-config/config.js}"
css="css/custom.css"
modules="modules"

# Tests
[ -f "$base/$config" ] || (echo "config.js does not exists" && exit 1)
[ -f "$base/$css" ] || (echo "custom.css does not exists" && exit 1)
[ -d "$base/$modules" ] || (echo "modules directory does not exists" && exit 1)


echo "#!/bin/sh" > $restore
echo "" >> $restore
echo "base=\"\$(cd \"\$(dirname \"\$0\")\" && pwd)\"" >> $restore

echo "" >> $restore

echo "mkdir -p \$base/config" >> $restore
echo "mkdir -p \$base/css" >> $restore
echo "mkdir -p \$base/modules" >> $restore

echo "" >> $restore

echo "cat > \$base/$config <<\"EOF\"" >> $restore
cat <$base/$config >> $restore
echo "EOF" >> $restore
echo "" >> $restore

echo "cat > \$base/$css <<\"EOF\"" >> $restore
cat <$base/$css >> $restore
echo "EOF" >> $restore
echo "" >> $restore

for dir in $(find "$modules" -maxdepth 1 -mindepth 1 -type d)
do
  [ -f "$dir/.git/config" ] && mods="$mods $(cat $dir/.git/config | grep 'url = ' | sed 's|.*url = ||g')"
done

for repo in $mods
do
  echo "cd \$base/$modules && git clone $repo" >> $restore
done

echo "" >> $restore

for repo in $mods
do
  moddir="$modules/$(echo $repo | sed -r 's|.*\/(.*)|\1|g;s|.git||g')"
  [ -f "$base/$moddir/package.json" ] && echo "cd \$base/$moddir && npm install" >> $restore
done

echo "Created restore script $restore" 

chmod +x $restore

swvalenti · Dec 19, 2020, 11:47 PM

@karsten13 Looks good but same thing I am not grasping how to implement this…sorry

Snille · Dec 20, 2020, 9:48 AM

@swvalenti Ok… :)
You need a “server” (a location where you can “wget” files from for this to work.
I have a specific backup directory on my own webserver, but you can use any server as long as you can wget from it.

Then the “Toinstall” part is just me having more the one config files in the backup dir.
Basically I have 2 different install possibility’s “house” and “NUC”.
So my backup config files are named “config-house.js” and config-NUC.js".
I enable the one I want to restore.

“ConfigName” is the actual name of the config file when it’s copied to it’s destination. This is probably always going to be “config.js” for the MM2. :)

“User” is in what directory the “MagicMirror” is located (when you cloned it).

“Moddir” is where your modules will be installed (this is probably always going to the same for MM2 as well).

“DownloadFrom” is where to download your backed up files from.

Then it’s just two "array"s with the files and folders with the backup files and where to copy them when restoring.
Files[1] = “file1-to-copy”
Dirs[1] = “Where/to/copy/file-1”
And so on…

Files[3] Is the script I use to set the screen to sleep and wake it up (Using MMM-Remote-Control).
Files[4] Is the script I use to automatically update the mirror and the modules (manually).

Files[7-9] It’s the “compliment-files” (used in the compliments module).

And Files[11] Is the MagicMirror start script I use (for starting the mirror with pm2).

“Packfile” Is the name of the file to look for when installing moduls. If it’s there, the npm install will be executed when installing the module.

Last but not least it’s all the git-repos for the modules I use in my mirror.
These will be cloned in the “Moddir” and installed (with npm if “Packfile” exists").

That’s about it. :)

Let me know if you have any more questions. :)

Lusbueb · Dec 22, 2020, 8:37 AM

i use this tool:

https://www.linux-tips-and-tricks.de/en/backup/

every night I make a full backup of all 4 Raspi as a TAR file to a NAS on my network. then i shrink and convert the TAR files into an IMG file (read Framps Linux-Tips-and-Tricks).

so I can always make a new SD card with my Win10 and Win32 Disk Imager.

swvalenti · Dec 20, 2020, 11:42 PM

@Lusbueb I have a Buffalo NAS LS I am trying to backup too but can’t figure out the right path to add to sudo nano /etc/fstab

BillyTheKid9588 · Dec 22, 2020, 8:37 AM

@Lusbueb this looks interesting.

Do you stop mm before the back up?

bhepler · Dec 23, 2020, 3:50 AM

The only information that is unique to my mirror is the module configuration. I just back up the config.js file whenever I make a change to the mirror configuration. If the mirror needs to be rebuilt, I can quickly build a slick Buster installation, SSH into the Pi, run the installation script, drop the backup config.js to the config folder and then do a half-dozen git clone commands.

Once the mirror is up & running, I can do the boring maintenance stuff (apt get update && apt get upgrade) and just let it run in the background.