Using SSH tunnels

Why?

The situation is fairly common -- you have a system behind a firewall you wish to get access to over the Internet. A good example of this might be a digital video recorder -- you have the DVR set up in your office, but you want to monitor it remotely from home.

The brain-dead solution is to put a hole in the firewall, enable some kind of authentication on the device, and hope for the best. The reason this is brain-dead is most of these in-home devices were not designed with security in mind at all; in fact, most commercial devices are designed to work first, and then they think about security later. The security record of in home "Internet Of Things" devices is somewhere between comical and pathetic. The initial security is bad and the updates are non-existent. So let's just call this a really bad idea.

The usual accepted "good" answer is a VPN into your network. Commercial VPN solutions will scale to hundreds or thousands of users and offer a lot of management options. There are lots of problems there, however. VPNs typically fairly complicated to set up and require a bit of maintenance. They also have a security record that is not worthy of envy, and are generally maintainable only from within the network. One other problem with VPNs is they typically want to (virtually) move you and your computer inside the network they are protecting and disconnect you from any resources inside your home network. SSH tunnels do not try to isolate you from anything else. This may be bad or good, depending on your desires.

side track: the term VPN has been tossed around as a way for home users to anonomize their Internet activity -- you connect your home workstation to a VPN service, then the sites you are going to think you are in the VPN. That's not the kind of VPN I am talking about -- I am refering to the VPN that allows an outside machine into your internal network, supposedly securely.

SSH tunnels are a way to permit someone very secure access to systems within your network from the outside world. They are very good for individual people trying to get access to a network they own and maintain, for example, you being able to access resources within your home by remote. Scaling to ten users is not difficult. Scaling to a thousand users, well...management of those users is more difficult, by this point, you should probably be looking at a real VPN.

What is SSH?

SSH stands for "Secure Shell", which today for many people, doesn't help a lot. The bigger picture is that SSH provides a fully end-to-end encrypted communication channel between two computers. Common use includes getting "shell" (a text login prompt) on a remote computer. SSH replaced Telnet as a way to do this, as SSH sends NO unencrypted data over the wire -- encryption between the two ends is established before authentication (login). SSH was originally a college student project, which was released as free-to-use open source code. The person who initially developed it continued development as part of SSH.com as commercial software. The original source code was taken by the OpenBSD project, creating the OpenSSH project and evolved as well as ISC licensed open source code; free for all uses, commercial or not. Currently, OpenSSH is included with most major operating systems, even Windows 10, and should be considered the definitive standards bearer for OpenSSH.

SSH also provides a lot of additional functions besides just secured terminal sessions with its encrypted communications channel -- it also provides file transfer and remote command facilities. It also provides "tunnels" -- which is our topic here.

What's a tunnel?

IP addressing refresher

As you hopefully know, one IP address can serve multiple functions. A single machine can be logged into using Secure Shell, you can serve web pages, secured web pages, and lots of other functions. These are differentiated by "port" numbers -- for example, if you wish to talk to an unsecured web server at address 10.33.44.5, you do that over port 80 (by default). Secured web services would be at port 443, ssh at port 22.

In addition to the external IP address that most network-connected machines have, they also have an INTERNAL IP address called a "loopback" address, this is so a machine can easily talk to itself. This may sound pointless, as the machine could easily talk to its external IP address as well, but there are times when an application may not know what that external address is...or may not wish to make services available outside the box. A very standardized feature of the IP protocol is the "loopback" address of 127.0.0.1. This address is normally able to be identified by the DNS name "localhost".

I knew that. So what's a tunnel?

An SSH tunnel connects a port on your machine's "localhost interface" to a remote machine's service running on a port. The path is from your local machine to the system running an SSH service, and from that machine to the machine/service you are desiring to get to. That machine's service now appears to be running on your local machine's locahost IP address on a designated port. Let's look at an example:
SSH Tunnel example

We have a laptop on the outside world, that needs to get to some service on the system "Server" inside the home or office network. We have a machine labeled "SSH Jump Box" and the firewall has an inbound rule to permit the SSH service (port 22) on the SSH Jump Box to be exposed to the external Internet through the firewall.

Note that the SSH service is NOT running on the box labeled "Server" (it COULD be, but isn't in this case). Absolutely no changes in configuration to the "Server" need be made. Server will think the activity it sees from the remote user is coming from the SSH Jump Box machine's IP address. The machine that we SSH into could be the firewall, or another system. The only requirement is that the SSH Jump Box is able to access the target ports on the target system. However, more discussion on a GOOD selection later.

For the sake of discussion, let's assume that "Server" is running a web service on port 80. If the tunnel is established between port 8080 on the external system and port 80 on Server, the external user can access the server by pointing their web browser at http://localhost:8080.

Setting up a tunnel

Configuring the SSH Jump Box

On a standard OpenSSH configuration, literally nothing has to be done other than having SSH installed and listening. However, there are options for the SSH service (sshd) which are configured, typically, in /etc/ssh/sshd_config which could disable or alter behavior which might matter. Make sure "AllowTcpForwarding" has not been changed from the default of "Yes", at least for the user you will be using.

You need to create an account for the outside user to use during the SSH session. This account name is only for authenticating to the SSH Jump Box; it need have no relationship to the account name of any service the tunnel is used to access. Remember, this is a network-layer connection between the computer in the field and a computer inside your office, the network doesn't care what names you log in with. You can authenticate to the SSH Jump Box as "bob" and then use services via the tunnel as "mary" or any other name. Though keeping things consistent is not bad for humans.

Being that this box will be attached directly to the Internet, and people WILL be trying to break into it, I would suggest disabling password logins across the board and ONLY using SSH keys. I'd also suggest either disabling root logins or setting it to "prohibit-password". SSH keys are a topic I will hopefully describe in more detail in the future. If you don't understand keys and wish to use password logins, make sure they are very good passwords.

While a dedicated box for the SSH Jump Box is not required technically, it is a good idea for security. You need to make sure this box is very carefully managed, as it is directly attached to the Internet. If, for reasons unrelated to the SSH tunnel function, someone creates a user on this box with an easily guessed password, it can (and will) become an entry point into your network, often within minutes or hours.

Configuring your target system

There's basically nothing to do here!

Configuring your remote system

The user in the field trying to get into your network is where most of the configuration has to be done. One thing you might have noticed above is that for my web server example, rather than linking the local port 80 on the field user's computer to port 80 on the in-house web server, I used port 8080. This is for a couple reasons. First, and biggest, is that ports under 1024 require administrative access to bind to. This would require running as administrator in Windows or root in Unix, and that's a really bad idea. So, we bind to ports 1024 and above. The second reason is IF your field machine has a service on those ports already, you won't be able to bind a tunnel to that port. So, if your machine has a web server running, odds are port 80 and 443 are already busy. If it is a Windows machine, there's a possibility that port 3389 is in use by the remote desktop service. So IN GENERAL, I'm going to suggest altering the port number in a memorable way, for example port 8080 instead of 80, 2222 instead of port 22, etc. There's no rule on how this should be done, this is strictly for your use. You could start numbering sequentially at 1024, but you will have to keep track of that carefully.

You can also set up multiple tunnels, either to one machine or to multiple machines. You may wish to set up unique port numbers to track multiple machines, for example, 1443 for the HTTPS service on one system, 2443 for HTTPS on the second. One service may also require multiple ports to be forwarded.

Let's look at an example. We are going to make a few assumptions here:

The jump box can be reached from the DNS name MYNETWORK.COM. Note this could also be an IP address.
The user's login name is "sam"
The server we are looking for can be reached INSIDE the home/office network as "server.mynetwork.com". This name does not need to be resolvable from the general Internet, and it can be an IP address. It could also be an entry in the /etc/hosts file of the SSH jump box.
The first service we are interested in is Remote Desktop, running on port 3389. We are going to map that to 33389.
The second service we are interested in is an HTTP web server running on port 80, we are going map that to port 8080

Unix

First, we set up the SSH connection:

$ ssh -f -N -L 33389:server.mynetwork.com:3389 -L 8080:server.mynetwork.com:80 sam@mynetwork.com
$

You will note that if the connection works, you will just drop back to the command prompt. If you don't have key logins enabled OR your key wasn't accepted, you will get a password prompt. Assuming no error messages, you are now connected and the tunnel is established.

And again to review -- "mynetwork.com" has to resolve outside your network for the external computer to get to your SSH Jump Box. But, "server.mynetwork.com" does NOT need to be resolvable by the external computer -- those are commands being processed by the computer inside your network, only they need to understand how to resolve them.

That's a lot to type, so I'd suggest putting it into a script, with the lines wrapped to make things more readable and easier to update:

ssh -f -N -L 33389:server.mynetwork.com:3389 \
	  -L 8080:server.mynetwork.com:80 \
          sam@mynetwork.com

The command line options are:

-f : Drop into the background after authentication. Don't give me a command prompt on the remote machine. Maybe you want that remote command prompt, though. Optional.
-N : Do not execute a remote command. Again, if your purpose is just the tunnel, this is probably good...but maybe you want to do things remotely.
-L : Set up the tunnel. Command syntax there is
localport:tagethost:targetport. This is basically the reason you have read this far, so I'm going to assume this is required for your application. In this case, we set up two tunnels to the same machine.

You may have reason to use other SSH command line options.

Now, we point our web browser at http://localhost:8080 to see the web server or the remote desktop client to localhost:33389 for remote desktop services. Done!

MacOSX

I don't have a MacOSX system to test this with, but it's basically Unix under the covers, so I'd expect the Unix steps above will work just fine. Let me know if I'm wrong (and what the proper process is).

I have found out there is a "PuTTY for Macintosh", which is apparently a combination of PuTTY source code with the WINE Windows emulation libraries. It apprently works for SSH tunnels, and behaves and looks like PuTTY for Windows. I'm not sure I'm going to recommend it, but I have no reason to recommend against it, either. But I do prefer "native" tools, when possible.

Windows 10

For Windows, you have a few options.

Use Windows SSH: Microsoft has ported OpenSSH to Windows 10 and it works. It works ALMOST exactly as the Unix version from the CMD prompt, except it really doesn't background like the Unix version does and give you back the command prompt. But...no harm. You will have to install the SSH client from Microsoft, last I looked, it was not installed by default (this may have changed).
Use PuTTY: Until Microsoft provided a native port of OpenSSH, PuTTY was the primary way to use SSH on Windows. Don't let the "less than Version 1" version numbers scare you -- it's been around for over 20 years, and it works well. Also -- the link I provide here is the official source for PuTTY. There are a lot of other sources less official.
- Create and save a "session" that connects to your in-house SSH jump box. Include the key connectivity.
- Verify it works.
- Load the saved session, go to on the left-side tree, go to "Connection" -> "SSH" -> "Tunnels"
- Add the tunnels you want, one at a time. Source port is the port on the machine in the field that you are configuring PuTTY on. Destination is the in-house machine name/IP address (the IP address of the machine relative to the jump box, NOT the external IP), a ":", and the port number. So for our example, you would set:
```
Source port: 33389
Destination: server.mynetwork.com:3389

Source port: 8080
Destination: server.mynetwork.com:80
```
use PuTTY's PLINK: PLINK is a command-line version of SSH, which is much more scriptable than the PuTTY GUI. It works basically the same as the Unix, except you leave off the "-f". If you have PuTTY installed and want better command line usage, plink is a good option. If you don't have any other use for PuTTY, it is something of a toss-up between using PuTTY's PLINK or just installing MS's SSH.

Considerations

You will need to set up a tunnel for every port your remote service needs. For example, the Dell Remote Access Card (DRAC) requires several ports for all functionality. The primary port is port 443 used by a web browser, the others are all >1024, which is good, because it's a royal pain, if not impossible, to change how those other ports are accessed.

As indicated, you can have multiple tunnels going to multiple servers. For example, for my personal use, I manage two systems about 300km from where I live behind an SSH jump box. This command does what I need:

$ ssh -f -N -L 1443:ipmi1:443 -L 2443:ipmi2:443 nick@terminalserver

This command gives me access to the IPMI (remote management interface) on TWO different systems -- "ipmi1" and "ipmi2", depending on what port I point my browser at. HOWEVER, there's a problem. My browser has NO idea that port 1443 and port 2443 are two different systems, so the login state gets mangled. If I log into ipmi1 on port 1443, but then bring up another tab in the same browser for ipmi2 on 2443, my ipmi1 session gets logged off. One easy fix is simply use a different browser on each, or just expect to get logged out of the first when logging into the second. Different applications will handle this differently.

I can't think of a good way to deal with proper cert handling over a tunnel. Most certificates on web servers do not have "localhost" as a valid DNS name, and I really don't think they should, so if you access an HTTPS server over an SSH tunnel, you will get a certificate error. Roll with it. Rest assured that your SSH session provides all the encryption over the Internet you will need, probably much better than the HTTPS would otherwise. Different machines inside your network will have different certificates, so again, you will have issues flipping between them on the same browser.

Considerations for the SSH jump box

Yes, this has been mentioned before, but it is really important, so I'm going to leave it here, too. This box is directly connected to the Internet through your SSH firewall rule. So you might have a nice tight remote user login, but if you have another user on the same box, say, named "test" with the password, "test", or a weak password on the root account, you will probably have an undesired user inside your box within hours. Seriously. And that user will be able to use those same tunnels, or build their own. For this reason, I would suggest giving serious thought to a dedicated SSH jump box, with a bare minimum number of users, passwords disabled and key-only logins. If you do use an existing machine for this purpose, make sure all administrators of this machine understand the risks of adding a "test" user or trivial passwords.

You might want to consider moving the inbound port to the SSH jump box to something other than the expected port 22. THIS IS NOT A SECURITY IMPROVEMENT -- if this actually improves your security situation, you are doing something wrong. But, it will reduce the amount of bogus and hopefully failed login attempts you will find in your logs.

If your target system or firewall supports it, you might want to put some kind of rate limiter on your SSH connections. Real users will be doing very few connection attempts per minute, attackers will be trying lots of connections per minute. In OpenBSD's PF, you can do that with something along these lines in pf.conf:

pass in on egress inet proto tcp from any to any port 22 keep state \
  (max-src-conn-rate 15/5 overload  flush)

This shouldn't be a security improvement -- it isn't practical to brute-force a key authentication, but if you have other problems, it really is a security improvement. But then, I'd highly suggest fixing the real problems, not relying on this. Again, it does reduce the noise in your log files.

Any Unix with OpenSSH on it will work; you are probably thinking Linux, I'm a big fan of OpenBSD. Not a lot of processor, RAM or disk is needed for the jump box. I personally use a 1.1MHz Intel Celeron re-purposed HP T430 Thin Client. It's a little bigger than a hockey puck, it has 2G RAM, 16G flash storage, and draws about 4 watts of power. A recycled Pentium 4 with 512MB RAM will do fine for OpenBSD, Linux, you will probably want more RAM. If you have a lot of simultaneous users moving a lot of data, a couple cores and a little more RAM might be good, but for almost anything you would be expected to use this solution on, a recycled machine or a small VM is going to be sufficient.

These tunnels allow users to any system the jump box can access, and since the outside user can configure where they go, they may give the average administrator a bit of concern. However, a tractional VPN basically lets a user into your office directly, so there's not a lot of practical difference. You may wish to use a filter on the jump box to restrict what systems and ports the jump box can access.

Things I've used tunnels for

After Hurricane Katrina, I set up an SSH tunnel jump box for our users who had to relocate away from their office as part of an emergency "Work From Home" environment. We handed probably 20 or so Citrix users this way. Katrina happened over a weekend, by Wednesday, we had users back on-line in a remote user environment that literally hadn't existed two days earlier.
As I mentioned above, I have a couple security critical machines I remotely manage. They have IPMI remote console support, but IPMI is not very secure, and I don't trust attaching them to the Internet. So all the IPMI (and some Dell RAC systems) are hiding in a private network behind an OpenBSD box acting as an SSH jump box.
In house, I have a jump box which allows me access to my IMAP mail server from outside my home without directly exposing it to the Internet.
Some friends own a gas station, he wanted to be able to watch his video cameras by remote. I set them up with a PuTTY session and an icon on their desktop. Click the session icon, that would establish the tunnels. After that, call up their browser and go to a bookmark. The setup took a little skill, but the actual day-to-day operation was trivial and they had no problem using it.
Some old devices are desparately in need of security updates they will never get due to being abandoned products. Putting an SSH Jump Box between the internal network and these devices is a good way to make sure that only desired users can touch these boxes.

Holland Consulting home page
Contact Holland Consulting

since April 14, 2022