Using SSH tunnels

Why?

The situation is fairly common -- you have a system behind a firewall you wish to get access to over the Internet. A good example of this might be a digital video recorder -- you have the DVR set up in your office, but you want to monitor it remotely from home.

The brain-dead solution is to put a hole in the firewall, enable some kind of authentication on the device, and hope for the best. The reason this is brain-dead is most of these in-home devices were not designed with security in mind at all; in fact, most commercial devices are designed to work first, and then they think about security later. The security record of in home "Internet Of Things" devices is somewhere between comical and pathetic. The initial security is bad and the updates are non-existent. So let's just call this a really bad idea.

The usual accepted "good" answer is a VPN into your network. Commercial VPN solutions will scale to hundreds or thousands of users and offer a lot of management options. There are lots of problems there, however. VPNs typically fairly complicated to set up and require a bit of maintenance. They also have a security record that is not worthy of envy, and are generally maintainable only from within the network. One other problem with VPNs is they typically want to (virtually) move you and your computer inside the network they are protecting and disconnect you from any resources inside your home network. SSH tunnels do not try to isolate you from anything else. This may be bad or good, depending on your desires.

side track: the term VPN has been tossed around as a way for home users to anonomize their Internet activity -- you connect your home workstation to a VPN service, then the sites you are going to think you are in the VPN. That's not the kind of VPN I am talking about -- I am refering to the VPN that allows an outside machine into your internal network, supposedly securely.

SSH tunnels are a way to permit someone very secure access to systems within your network from the outside world. They are very good for individual people trying to get access to a network they own and maintain, for example, you being able to access resources within your home by remote. Scaling to ten users is not difficult. Scaling to a thousand users, well...management of those users is more difficult, by this point, you should probably be looking at a real VPN.

What is SSH?

SSH stands for "Secure Shell", which today for many people, doesn't help a lot. The bigger picture is that SSH provides a fully end-to-end encrypted communication channel between two computers. Common use includes getting "shell" (a text login prompt) on a remote computer. SSH replaced Telnet as a way to do this, as SSH sends NO unencrypted data over the wire -- encryption between the two ends is established before authentication (login). SSH was originally a college student project, which was released as free-to-use open source code. The person who initially developed it continued development as part of SSH.com as commercial software. The original source code was taken by the OpenBSD project, creating the OpenSSH project and evolved as well as ISC licensed open source code; free for all uses, commercial or not. Currently, OpenSSH is included with most major operating systems, even Windows 10, and should be considered the definitive standards bearer for OpenSSH.

SSH also provides a lot of additional functions besides just secured terminal sessions with its encrypted communications channel -- it also provides file transfer and remote command facilities. It also provides "tunnels" -- which is our topic here.

What's a tunnel?

IP addressing refresher

As you hopefully know, one IP address can serve multiple functions. A single machine can be logged into using Secure Shell, you can serve web pages, secured web pages, and lots of other functions. These are differentiated by "port" numbers -- for example, if you wish to talk to an unsecured web server at address 10.33.44.5, you do that over port 80 (by default). Secured web services would be at port 443, ssh at port 22.

In addition to the external IP address that most network-connected machines have, they also have an INTERNAL IP address called a "loopback" address, this is so a machine can easily talk to itself. This may sound pointless, as the machine could easily talk to its external IP address as well, but there are times when an application may not know what that external address is...or may not wish to make services available outside the box. A very standardized feature of the IP protocol is the "loopback" address of 127.0.0.1. This address is normally able to be identified by the DNS name "localhost".

I knew that. So what's a tunnel?

An SSH tunnel connects a port on your machine's "localhost interface" to a remote machine's service running on a port. The path is from your local machine to the system running an SSH service, and from that machine to the machine/service you are desiring to get to. That machine's service now appears to be running on your local machine's locahost IP address on a designated port. Let's look at an example:
SSH Tunnel example
We have a laptop on the outside world, that needs to get to some service on the system "Server" inside the home or office network. We have a machine labeled "SSH Jump Box" and the firewall has an inbound rule to permit the SSH service (port 22) on the SSH Jump Box to be exposed to the external Internet through the firewall.

Note that the SSH service is NOT running on the box labeled "Server" (it COULD be, but isn't in this case). Absolutely no changes in configuration to the "Server" need be made. Server will think the activity it sees from the remote user is coming from the SSH Jump Box machine's IP address. The machine that we SSH into could be the firewall, or another system. The only requirement is that the SSH Jump Box is able to access the target ports on the target system. However, more discussion on a GOOD selection later.

For the sake of discussion, let's assume that "Server" is running a web service on port 80. If the tunnel is established between port 8080 on the external system and port 80 on Server, the external user can access the server by pointing their web browser at http://localhost:8080.

Setting up a tunnel

Configuring the SSH Jump Box

On a standard OpenSSH configuration, literally nothing has to be done other than having SSH installed and listening. However, there are options for the SSH service (sshd) which are configured, typically, in /etc/ssh/sshd_config which could disable or alter behavior which might matter. Make sure "AllowTcpForwarding" has not been changed from the default of "Yes", at least for the user you will be using.

You need to create an account for the outside user to use during the SSH session. This account name is only for authenticating to the SSH Jump Box; it need have no relationship to the account name of any service the tunnel is used to access. Remember, this is a network-layer connection between the computer in the field and a computer inside your office, the network doesn't care what names you log in with. You can authenticate to the SSH Jump Box as "bob" and then use services via the tunnel as "mary" or any other name. Though keeping things consistent is not bad for humans.

Being that this box will be attached directly to the Internet, and people WILL be trying to break into it, I would suggest disabling password logins across the board and ONLY using SSH keys. I'd also suggest either disabling root logins or setting it to "prohibit-password". SSH keys are a topic I will hopefully describe in more detail in the future. If you don't understand keys and wish to use password logins, make sure they are very good passwords.

While a dedicated box for the SSH Jump Box is not required technically, it is a good idea for security. You need to make sure this box is very carefully managed, as it is directly attached to the Internet. If, for reasons unrelated to the SSH tunnel function, someone creates a user on this box with an easily guessed password, it can (and will) become an entry point into your network, often within minutes or hours.

Configuring your target system

There's basically nothing to do here!

Configuring your remote system

The user in the field trying to get into your network is where most of the configuration has to be done. One thing you might have noticed above is that for my web server example, rather than linking the local port 80 on the field user's computer to port 80 on the in-house web server, I used port 8080. This is for a couple reasons. First, and biggest, is that ports under 1024 require administrative access to bind to. This would require running as administrator in Windows or root in Unix, and that's a really bad idea. So, we bind to ports 1024 and above. The second reason is IF your field machine has a service on those ports already, you won't be able to bind a tunnel to that port. So, if your machine has a web server running, odds are port 80 and 443 are already busy. If it is a Windows machine, there's a possibility that port 3389 is in use by the remote desktop service. So IN GENERAL, I'm going to suggest altering the port number in a memorable way, for example port 8080 instead of 80, 2222 instead of port 22, etc. There's no rule on how this should be done, this is strictly for your use. You could start numbering sequentially at 1024, but you will have to keep track of that carefully.

You can also set up multiple tunnels, either to one machine or to multiple machines. You may wish to set up unique port numbers to track multiple machines, for example, 1443 for the HTTPS service on one system, 2443 for HTTPS on the second. One service may also require multiple ports to be forwarded.

Let's look at an example. We are going to make a few assumptions here:

Unix

First, we set up the SSH connection:
$ ssh -f -N -L 33389:server.mynetwork.com:3389 -L 8080:server.mynetwork.com:80 sam@mynetwork.com
$
You will note that if the connection works, you will just drop back to the command prompt. If you don't have key logins enabled OR your key wasn't accepted, you will get a password prompt. Assuming no error messages, you are now connected and the tunnel is established.

And again to review -- "mynetwork.com" has to resolve outside your network for the external computer to get to your SSH Jump Box. But, "server.mynetwork.com" does NOT need to be resolvable by the external computer -- those are commands being processed by the computer inside your network, only they need to understand how to resolve them.

That's a lot to type, so I'd suggest putting it into a script, with the lines wrapped to make things more readable and easier to update:

ssh -f -N -L 33389:server.mynetwork.com:3389 \
	  -L 8080:server.mynetwork.com:80 \
          sam@mynetwork.com

The command line options are:

You may have reason to use other SSH command line options.

Now, we point our web browser at http://localhost:8080 to see the web server or the remote desktop client to localhost:33389 for remote desktop services. Done!

MacOSX

Well, I don't have a MacOSX system to test this with, but it's basically Unix under the cover, so I'd expect the Unix steps above will work just fine. Let me know if I'm wrong (and what the proper process is).

Windows 10

For Windows, you have a few options.

Considerations

You will need to set up a tunnel for every port your remote service needs. For example, the Dell Remote Access Card (DRAC) requires several ports for all functionality. The primary port is port 443 used by a web browser, the others are all >1024, which is good, because it's a royal pain, if not impossible, to change how those other ports are accessed.

As indicated, you can have multiple tunnels going to multiple servers. For example, for my personal use, I manage two systems about 300km from where I live behind an SSH jump box. This command does what I need:

$ ssh -f -N -L 1443:ipmi1:443 -L 2443:ipmi2:443 nick@terminalserver
This command gives me access to the IPMI (remote management interface) on TWO different systems -- "ipmi1" and "ipmi2", depending on what port I point my browser at. HOWEVER, there's a problem. My browser has NO idea that port 1443 and port 2443 are two different systems, so the login state gets mangled. If I log into ipmi1 on port 1443, but then bring up another tab in the same browser for ipmi2 on 2443, my ipmi1 session gets logged off. One easy fix is simply use a different browser on each, or just expect to get logged out of the first when logging into the second. Different applications will handle this differently.

I can't think of a good way to deal with proper cert handling over a tunnel. Most certificates on web servers do not have "localhost" as a valid DNS name, and I really don't think they should, so if you access an HTTPS server over an SSH tunnel, you will get a certificate error. Roll with it. Rest assured that your SSH session provides all the encryption over the Internet you will need, probably much better than the HTTPS would otherwise. Different machines inside your network will have different certificates, so again, you will have issues flipping between them on the same browser.

Considerations for the SSH jump box

Yes, this has been mentioned before, but it is really important, so I'm going to leave it here, too. This box is directly connected to the Internet through your SSH firewall rule. So you might have a nice tight remote user login, but if you have another user on the same box, say, named "test" with the password, "test", or a weak password on the root account, you will probably have an undesired user inside your box within hours. Seriously. And that user will be able to use those same tunnels, or build their own. For this reason, I would suggest giving serious thought to a dedicated SSH jump box, with a bare minimum number of users, passwords disabled and key-only logins. If you do use an existing machine for this purpose, make sure all administrators of this machine understand the risks of adding a "test" user or trivial passwords.

You might want to consider moving the inbound port to the SSH jump box to something other than the expected port 22. THIS IS NOT A SECURITY IMPROVEMENT -- if this actually improves your security situation, you are doing something wrong. But, it will reduce the amount of bogus and hopefully failed login attempts you will find in your logs.

If your target system or firewall supports it, you might want to put some kind of rate limiter on your SSH connections. Real users will be doing very few connection attempts per minute, attackers will be trying lots of connections per minute. In OpenBSD's PF, you can do that with something along these lines in pf.conf:

pass in on egress inet proto tcp from any to any port 22 keep state \
  (max-src-conn-rate 15/5 overload  flush)
This shouldn't be a security improvement -- it isn't practical to brute-force a key authentication, but if you have other problems, it really is a security improvement. But then, I'd highly suggest fixing the real problems, not relying on this. Again, it does reduce the noise in your log files.

Any Unix with OpenSSH on it will work; you are probably thinking Linux, I'm a big fan of OpenBSD. Not a lot of processor, RAM or disk is needed for the jump box. I personally use a 1.1MHz Intel Celeron re-purposed HP T430 Thin Client. It's a little bigger than a hockey puck, it has 2G RAM, 16G flash storage, and draws about 4 watts of power. A recycled Pentium 4 with 512MB RAM will do fine for OpenBSD, Linux, you will probably want more RAM. If you have a lot of simultaneous users moving a lot of data, a couple cores and a little more RAM might be good, but for almost anything you would be expected to use this solution on, a recycled machine or a small VM is going to be sufficient.

These tunnels allow users to any system the jump box can access, and since the outside user can configure where they go, they may give the average administrator a bit of concern. However, a tractional VPN basically lets a user into your office directly, so there's not a lot of practical difference. You may wish to use a filter on the jump box to restrict what systems and ports the jump box can access.

Things I've used tunnels for

Holland Consulting home page
Contact Holland Consulting
 

since April 14, 2022

Copyright 2022, Nick Holland, Holland Consulting