Remote SSH Access to the Mirrors You Give Away

[bhepler]

This is where the magic happens. The goal here is to open a connection from the Pi to the C&C server and then have the C&C server use that connection to get command line access to the Pi. To do this, we open a reverse SSH connection from the Pi to the C&C server.

Add the Pi’s Public Key to the C&C Server
In order for the C&C server to recognize the Pi when it connects, the C&C server must have a copy of the Pi’s public key. Amazon has already created the file where it belongs for you, you just have to add to it.

Log into your C&C server and navigate to /home/ubuntu/.ssh. There is a file in this folder called authorized_keys and at present it contains the public key of your Home Keypair. Edit the file using nano authorized_keys, go to the end of the file and press enter to create a new line. On your home computer, open up the gift-mirror.pub file in WordPad or whatever and copy the contents to the authorized_keys file. Save your work!

Test Access via Key Exchange
First, let’s test that the key works. On your Pi, establish an SSH connection to the C&C server by entering ssh -i /home/pi/.ssh/gift-mirror.key ubuntu@mirror-tutorial.duckdns.org. This should open a connection to the C&C server, using the private key you specified with the -i switch for authorization. Progress: key works.

Test Reverse SSH Connection
Now we can open a reverse SSH connection to the C&C server. (better explanation can be found at SSH tunneling for fun and profit) Change that ssh command to add the -R flag and let’s add the ports that we’re forwarding for the reverse connection. Do you remember that port that you opened when you configured the security on your C&C server? This is where it goes.

ssh -i /home/pi/.ssh/gift-mirror.key -N -R [port number]:localhost:22 ubuntu@[mirror-DNS].duckdns.org -v

Substitute whatever port you’ve decided upon for the [port number] portion and whatever DNS name [mirror-DNS] you gave your C&C server at DuckDNS’s website. Once you hit enter, you will not get your command prompt back as you are still maintaining a connection with that user. It will stay open until you cancel it with Ctrl+c. But we still have to test it!

Open up new connection to your C&C server and log in normally. From here, you can reach the Pi by following the reverse SSH connection back to the gift mirror. You do so by opening an SSH connection to the localhost address on a specific port. Like so:
ssh pi@localhost -p [port number] which basically means “SSH into myself as the pi user on port [port number]”. You will be prompted for a password just as if you were logging into the Pi controlling your guest mirror, so go ahead and enter that password. You should get a prompt on your Pi. Progress: Reverse SSH works!

You can go ahead and press Ctrl+d on the C&C server to close the connection now. And type Ctrl+c on the guest mirror Pi to close the reverse SSH.

Set Up AutoSSH to Persist Connection
This part comes mostly from a lot of experimenting and reading Fun and Profit with Reverse SSH Tunnels, Setup autossh script on Raspian, and a few evenings of swearing at inanimate objects.

First, install autossh with sudo apt-get install autossh and let it install the program. Once it’s ready, we can test the reverse SSH using autossh instead of the normal ssh command. Autossh takes a couple of commands, but most everything is passed through to your existing ssh binary, so it uses the same switches. Enter this into your command line prompt on your gift Pi:
autossh -M 0 -N -i /home/pi/.ssh/gift-mirror.key -R [port number]:localhost:22 ubuntu@[mirror-DNS].duckdns.org -v

You should see a few lines of text giving you a status update as your Pi establishes a reverse SSH connection to your C&C server. Quick explanation of the new switches: -M 0 turns off the connection monitoring built into autossh, -N means "do not execute a command remotely on the C&C server, and -v is very verbose logging. When you enter that command, you should see something like this in return:

Authenticated to [mirror-DNS].duckdns.org ([C&C IP address]:22).
debug1: Remote connections from LOCALHOST:[port number] forwarded to local address localhost:22
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: remote forward success for: listen [port number], connect localhost:22
debug1: All remote forwarding requests processed

Once you have a connection, you can test the reverse SSH connection just like we did in the last step. If you can get to the Pi command prompt, you’re golden. So let’s give it a go! On the C&C server, try the following command:
ssh pi@localhost -p [port number]

You should get the same password challenge that you received when you opened the reverse SSH tunnel earlier. Which makes sense, as autossh wraps around the normal SSH capability. A real test would involved shutting down the C&C server via the AWS console, starting autossh and verifying it is retrying the connection and then starting up the C&C server. Ain’t nobody got time for that. Progress: autossh works.

Tweak SSH settings
Just to make sure the SSH connection behaves properly, we’re going to add a couple configuration variables to SSH and make sure it behaves. Edit the configuration file with this: sudo vi /etc/ssh/ssh_config.
Add the following variables to how SSH operates:

ServerAliveInterval 30
ServerAliveCountMax 3
StrictHostKeyChecking no

Save your work.

Set Autossh to Start Automatically
Okay people, home stretch. All that is left is to configure your gift mirror pi to automatically start autossh on boot. You’re going to create a service that will run autossh. That way the Pi’s operating system will handle starting it at reboot, restarting if it fails and all that.

Navigate to /etc/systemd/system and create a file that will define your service and add .service to the filename. Be sure to use sudo, because this is all root stuff. Like so:
sudo vi /etc/systemd/system/autossh.service
Once in the VI interface paste the following into the file:

[Unit]
Description=AutoSSH tunnel service - Mirror update, port 4348
After=network.target

[Service]
User=pi
Environment="AUTOSSH_GATETIME=0"
Environment="AUTOSSH_LOGFILE=/home/pi/autossh.log"
ExecStart=/usr/bin/autossh -M 0 -i /home/pi/.ssh/[gift-mirror.key] -N -R [port number]:localhost:22 ubuntu@[mirror-DNS].duckdns.org

[Install]
WantedBy=multi-user.target

Obviously, substitute your values for the name of the private key you use to connect to the C&C server, the port number and the DNS entry of your C&C server. Save you work!
Now come three commands in rapid succession: register the service, start the service, enable the service at boot:

sudo systemctl daemon-reload
sudo systemctl enable autossh.service
sudo service start autossh.service

That last command will give you a confirmation response that it has created a symlink. It will look like so: Created symlink /etc/systemd/system/multi-user.target.wants/autossh.service → /etc/systemd/system/autossh.service
Progress: AutoSSH will start on boot and continually try to connect to your C&C server.

[bhepler]

Okay, now we have all the pieces. How does it work in practice? The basic routine goes like this:

• Start your C&C server
• Give the C&C server a couple minutes to update the DNS entry
• Connect to your C&C server via SSH
• Follow the SSH connection down to the gift mirror
• make your changes, update the OS, whatever
• disconnect from the gift pi and then disconnect from the C&C server
• stop the C&C server via the cloud server interface

Thanks to the cloud provider, the C&C server only exists when you need to connect to your remote mirror(s). Once you’re done, the C&C server is destroyed and nobody can log into it when you’re not looking.

So, to connect to the remote mirror, first start your C&C server. SSH into the C&C server and then check to see which remote mirror(s) have connected to it. Run netstat -l and see what ports are open. For example, mine looks like this:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:domain        0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN
tcp        0      0 localhost:4548          0.0.0.0:*               LISTEN
udp        0      0 localhost:domain        0.0.0.0:*
udp        0      0 ip-172-31-34-37.:bootpc 0.0.0.0:*
raw6       0      0 [::]:ipv6-icmp          [::]:*                  7
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     SEQPACKET  LISTENING     12326    /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     23993    /run/user/1000/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     23997    /run/user/1000/gnupg/S.gpg-agent
...
[10 lines deleted for brevity]

The line tcp 0 0 localhost:4548 0.0.0.0:* LISTEN is the reverse SSH from one of my magic mirrors that has connected up to my C&C server. That gives me a port number (4548 in this case).

Connecting to the pi is easy: ssh pi@localhost -p 4548. If i need to connect to a different mirror, I look for a different port number (each mirror must use a different port to connect to your C&C server). I’ll get the prompt to enter the password to log into the Pi and then I’m issuing commands on the remote Pi.

When you’re done, remember to log out of the remote mirror and again out of your C&C server with Ctrl+d. And once you’re out of the C&C server, use the cloud provider console to stop the server so you don’t get charged for it just sitting there, burning cycles.

When I build new mirrors, I can configure them to connect to the same DNS entry which will be my C&C server a couple minutes after starting it. I only pay for as long as my C&C server is operational, which is something like $0.35/hour. Each mirror will still connect regardless of whether it is in a private home, university campus or corporate lobby.

[bhepler]

[THIS SPACE INTENTIONALLY LEFT BLANK]

[sdetweil]

@bhepler aws also provides generated keypairs.

I just created a server the other day to host an app. I have both ssh and scp’ed to that server from windows and my mirror and my Linux machine

[motdog]

I was planning on using https://www.dataplicity.com/ for my giveaway that I am working on. For the less technical of us this may be simpler and free

[bhepler]

If you have an easier way to accomplish this, I highly encourage you to write up a tutorial for us all.

[motdog]

@bhepler I have no idea at this point if it works, I just didn’t want to spend cash for this. In no way was I trying to be critical of your write up. Super detailed and if I had a reason to use a cloud provider or had a bunch to administer this would be perfect for me.

My apologies if I came across any other way.

[bhepler]

@motdog - It’s all good. There is a way to do this without spending any additional money. You’ll have to modify your residential gateway to forward a port, but it should be possible. It won’t work for university students or people who do not have control over their firewall. I’ll see about writing that up later as an addendum.

[motdog]

@bhepler ok thanks. The first one I am giving away is a college student thats why dataplicity seems like the way to go for me. Hopefully it works

[sojourner]

@bhepler Another cloud host option is google, who offers a free tier as well.

my universal caveat – As with all “free” things, TANSTAAFL