SShing to unpredictable IPs behind a bastion

A pretty common security measure for cloud infrastructure is an SSH bastion: you have a bunch of servers you may need to SSH to at times, but you obviously don’t want all your servers open to the public Internet, so you designate one server as the SSH bastion, which can be appropriately hardened and is open to public SSH connections for proxying ssh connections into your private network.

I recently found myself needing to manage multiple environments (production and staging) like this, and transitioning a legacy configuration with a small number of static servers to a “cattle, not pets” approach where individual servers were coming and going all the time. Under the old system, because the list of servers was static ops people would give each server a DNS entry pointing at the private IP and share snippets for everyone to put in their ~/.ssh/config: a manual process that worked fine when servers were manually configured and long-lived but not one that fits comfortably in a world where the number of servers may be much higher and individual servers are being automatically provisioned or decommisioned all the time.

I could have also built automatic DNS assignments to the new servers and chosen predictable entries such that a wildcard Host block in ssh_config could match everything, but I was hoping to find a simpler approach. I think I’ve landed on a couple of lines in my ~/.ssh/config that will do what I need:

Match originalhost bastion.staging
  User myuser
  Port 12345 # publicly exposed servers shouldn't use default SSH port
  Hostname my.staging.bastion.domain

# ssh <private-address>.staging to hop via bastion
Match host *.staging
  User myuser
  ProxyCommand ssh bastion.staging -W $(echo "%h" | sed s/.staging//):%p

With this, to ssh to a server in the staging environment with IP 10.1.2.3, I can type ssh 10.1.2.3.staging. The same two blocks reproduced with the .prod suffix instead of .staging handle production.

This does mean remembering to add the suffix after copy-pasting a private IP and knowing which environment I’m targeting, so it’s not quite as smooth or idiot-proof as DNS entries would be. For a much larger infrastructure that extra effort would probably be worthwhile, but for the scale I expect this system to operate at for the forseeable future I think what I’ve landed on is a simpler and easier solution.