A Visual Guide to SSH Tunnels (with labs)

Hello folks，Today I bring you an article from a buddy who is engaged in Cloud Native in the Netherlands. The highlights are as follows.

SSH is yet another example of an ancient technology that is still in wide use today. It may very well be that learning a couple of SSH tricks is more profitable in the long run than mastering a dozen Cloud Native tools destined to become deprecated next quarter.

One of my favorite parts of this technology is SSH Tunnels. With nothing but standard tools and often using just a single command, you can achieve the following:

• Access internal VPC endpoints through a public-facing EC2 instance.
• Open a port from the localhost of a development VM in the host's browser.
• Expose any local server from a home/private network to the outside world.

And more ～

But despite the fact that I use SSH Tunnels daily, it always takes me a while to figure out the right command. Should it be a Local or a Remote tunnel? What are the flags? Is it a local_port:remote_port or the other way around? So, I decided to finally wrap my head around it, and it resulted in a series of labs and a visual cheat sheet ：

Prerequisites

SSH Tunnels are about connecting hosts over the network, so every lab below expectedly involves multiple "machines". However, I'm too lazy to spin up full-blown instances, especially when containers can be used instead. That's why I ended up using just a single vagrant VM with Docker on it.

In theory, any Linux box with Docker Engine on it should do. However, running the below examples as-is with Docker Desktop won't be possible because the ability to access the machines containers by their IPs is assumed.

Alternatively, the labs can be done with Lima (QEMU + nerdctl + containerd + BuildKit), but don't forget to limactl shell bash first.

Every example requires a valid passphrase-less key pair on the host that is then mounted into the containers to simplify access management. If you don't have one, generating it is as simple as just ssh-keygen on the host.

Important: SSH daemons in the containers here are solely for educational purposes - containers in this post are meant to represent full-blown "machines" with SSH clients and servers on them. Beware that it's rarely a good idea to have SSH stuff in real-world containers!

Local Port Forwarding

Starting from the one that I use the most. Oftentimes, there might be a service listening on localhost or a private interface of a machine that I can only SSH to via its public IP. And I desperately need to access this port from the outside. A few typical examples:

• Accessing a database (MySQL, Postgres, Redis, etc) using a fancy UI tool from your laptop.
• Using your browser to access a web application exposed only to a private network.
• Accessing a container's port from your laptop without publishing it on the server's public interface.

All of the above use cases can be solved with a single ssh command:

ssh -L [local_addr:]local_port:remote_addr:remote_port [user@]sshd_addr

The -L flag indicates we're starting a local port forwarding. What it actually means is:

• On your machine, the SSH client will start listening on local_port (likely, on localhost, but it depends - check the GatewayPorts setting).
• Any traffic to this port will be forwarded to the remote_private_addr:remote_port on the machine you SSH-ed to.

Here is how it looks on a diagram:

Local Port Forwarding with a Bastion Host

It might not be obvious at first, but the ssh -L command allows forwarding a local port to a remote port on any machine, not only on the SSH server itself. Notice how the remote_addr and sshd_addr may or may not have the same value:

ssh -L [local_addr:]local_port:remote_addr:remote_port [user@]sshd_addr

Not sure how legitimate the use of the term bastion host is here, but that's how I visualize this scenario for myself:

I often use the above trick to call endpoints that are accessible from the bastion host but not from my laptop (e.g., using an EC2 instance with private and public interfaces to connect to an OpenSearch cluster deployed fully within a VPC).

Remote Port Forwarding

Another popular (but rather inverse) scenario is when you want to momentarily expose a local service to the outside world. Of course, for that, you'll need a public-facing ingress gateway server. But fear not! Any public-facing server with an SSH daemon on it can be used as such a gateway:

ssh -R [remote_addr:]remote_port:local_addr:local_port [user@]gateway_addr

The above command looks no more complicated than its ssh -L counterpart. But there is a pitfall...

By default, the above SSH tunnel will allow using only the gateway's localhost as the remote address. In other words, your local port will become accessible only from the inside of the gateway server itself, and highly likely it's not something you actually need. For instance, I typically want to use the gateway's public address as the remote address to expose my local services to the public Internet. For that, the SSH server needs to be configured with the GatewayPorts yes setting.

Here is what remote port forwarding can be used for:

• Exposing a dev service from your laptop to the public Internet for a demo.
• Hmm... I can think of a few esoteric examples, but I doubt it's worth sharing them here. Curious to hear what other people may use remote port forwarding for!

Here is how the remote port forwarding looks on a diagram:

Remote Port Forwarding from a Home/Private Network

Much like local port forwarding, remote port forwarding has its own bastion host mode. But this time, the machine with the SSH client (e.g., your dev laptop) plays the role of the bastion. In particular, it allows exposing ports from a home (or a private) network your laptop has the access to to the outside world through the ingress gateway:

ssh -R [remote_addr:]remote_port:local_addr:local_port [user@]gateway_addr

Looks almost identical to the simple remote SSH tunnel, but the local_addr:local_port pair becomes the address of a device in the home network. Here is how it can be depicted on a diagram:

Since I typically use my laptop as a thin client and the actual development happens on a home server, I rely on such a remote port forwarding when I need to expose a dev service from a home server to the public Internet, and the only machine with the gateway access is my thin laptop.

Summarizing

After doing all these labs and drawings, I noticed that:

• The word "local" can mean either the SSH client machine or an upstream host accessible from this machine.
• The word "remote" can mean either the SSH server machine (sshd) of an upstream host accessible from it.
• Local port forwarding (ssh -L) implies it's the ssh client that starts listening on a new port.
• Remote port forwarding (ssh -R) implies it's the sshd server that starts listening on an extra port.
• The mnemonics are "ssh -L local:remote" and "ssh -R remote:local" and it's always the left-hand side that opens a new port.

Happy reading!

- EOF -