My SSH Config Looked Fine. ssh -G Said Nope.

I had a clean SSH guide ready to ship: key generation, chmod, ssh-copy-id, ~/.ssh/config, server hardening, the whole checklist.

Then I ran the boring command that should have been in the draft from the start:

ssh -G dev-web

And one of my diagrams immediately became a footgun.

TL;DR: if your SSH guide says what a Host block does, make ssh -G prove it before you publish it.

Quick answer: where should Host * go in ssh_config?

If Host * sets values that a specific host also sets, put the specific Host block first and put Host * later.

OpenSSH does not treat Host * like a CSS reset that can be overridden later. For many client options, the first obtained value wins. That means a broad Host * block can quietly prevent later per-host values like User or Port from taking effect.

Check the effective SSH config before you trust the prose:

ssh -G your-host-alias | awk '/^(hostname|user|port|identityfile|identitiesonly) /{print}'

If the output does not match the host you thought you configured, the article, runbook, or config snippet is not ready.

The ssh_config footgun: Host star is not CSS

The bad advice looked harmless:

Put the shared rules in Host *, then put exceptions below.

That sounds right if your mental model is CSS: broad rule first, specific rule later, later wins.

OpenSSH client config does not work like that for many options. For each parameter, the first obtained value is used. That means a broad Host * block can quietly eat settings that a later, more specific block was supposed to override.

Here is the tiny repro.

Host *
    Port 2222
    User common

Host dev-web
    Port 2200
    User developer

Now ask OpenSSH what it will actually use:

ssh -F common-first.conf -G dev-web |
  awk '/^(user|port) /{print}'

The output:

user common
port 2222

Nope.

The specific host did not win. The broad block got there first.

Now flip the order:

Host dev-web
    Port 2200
    User developer

Host *
    Port 2222
    User common

Same check:

ssh -F specific-first.conf -G dev-web |
  awk '/^(user|port) /{print}'

Output:

user developer
port 2200

That is the version I wanted.

This is why I no longer write "put common stuff at the top" as blanket SSH advice. I now say:

Put broad defaults at the end when they can collide with per-host settings.

Host * is still useful. I still use it for things like ServerAliveInterval. I just do not put override-sensitive defaults above host-specific blocks and hope vibes will handle it.

The smoke test I should have run first

The article was about a practical SSH setup. I wanted the examples to be copy-paste-safe, so I built a temp fixture instead of touching my real ~/.ssh.

ssh-keygen -t ed25519 \
  -f "$TMP/home/.ssh/projectA_ed25519" \
  -N '' \
  -C '[email protected] on testhost'

chmod 700 "$TMP/home/.ssh"
chmod 600 "$TMP/home/.ssh/config"
chmod 600 "$TMP/home/.ssh/projectA_ed25519"
chmod 644 "$TMP/home/.ssh/projectA_ed25519.pub"

stat -f '%Lp %N' \
  "$TMP/home/.ssh" \
  "$TMP/home/.ssh/config" \
  "$TMP/home/.ssh/projectA_ed25519" \
  "$TMP/home/.ssh/projectA_ed25519.pub"

What I needed to see:

ssh-keygen EXIT=0
700 /tmp/.../home/.ssh
600 /tmp/.../home/.ssh/config
600 /tmp/.../home/.ssh/projectA_ed25519
644 /tmp/.../home/.ssh/projectA_ed25519.pub

That is not exciting content. It is the kind of boring output that prevents a reader from losing an afternoon to UNPROTECTED PRIVATE KEY FILE.

That is the vibe I want in docs now: less "trust me bro," more receipts.

Use ssh -G to see the effective SSH config

For client-side SSH config, the receipt is usually ssh -G.

Here is a safe example config:

Host project-a
    HostName       127.0.0.1
    User           deploy
    Port           65000
    IdentityFile   /tmp/.../home/.ssh/projectA_ed25519
    IdentitiesOnly yes

Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3

Then:

ssh -F "$TMP/home/.ssh/config" -G project-a |
  awk '/^(hostname|user|port|identityfile|identitiesonly|serveraliveinterval|serveralivecountmax) /{print}'

Output:

user deploy
hostname 127.0.0.1
port 65000
identitiesonly yes
serveralivecountmax 3
serveraliveinterval 60
identityfile /tmp/.../home/.ssh/projectA_ed25519

This is the part that changed my writing process.

If a guide says "this alias connects as deploy on port 65000 with this key," ssh -G can prove that claim without hitting the network.

Server-side examples get a smoke test too

The server-side section had examples like this:

PasswordAuthentication no
PermitRootLogin no
PubkeyAuthentication yes
Port 65000

I did not need a production server to check whether the example was internally valid.

I used a temp host key and a temp sshd_config:

sshd -t -f "$TMP/sshd_config"
sshd -T -f "$TMP/sshd_config" |
  awk '/^(passwordauthentication|permitrootlogin|port|pubkeyauthentication) /{print}'

Output:

sshd -t EXIT=0
port 65000
permitrootlogin no
pubkeyauthentication yes
passwordauthentication no

Does that prove a real server rollout is safe? No.

It proves the snippet I am asking people to copy is not nonsense. That is a lower bar, but it is a bar worth clearing.

The delegation checks

Some commands need a remote server to fully succeed:

ssh-copy-id -i ~/.ssh/projectA_ed25519.pub -p 2222 [email protected]
scp project-a:/path/file .
rsync -avz project-a:/path/ ./path/
git clone project-a:org/repo.git

I did not spin up a full SSH server for this pass. I pointed the alias at 127.0.0.1:65000, where nothing was listening, and checked whether each tool delegated to SSH the way the article implied.

For scp, the failure was the expected kind of failure:

Executing: program /usr/bin/ssh host project-a, user (unspecified), command sftp
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 65000.
ssh: connect to host 127.0.0.1 port 65000: Connection refused

For Git:

Reading configuration data /tmp/.../home/.ssh/config
Applying options for project-a
Connecting to 127.0.0.1 [127.0.0.1] port 65000.
fatal: Could not read from remote repository.

The connection failed. Good. There was no server.

The important part was that the tools used the SSH alias and reached the intended host and port. That is the evidence the article needed.

My OpenSSH checklist for articles and runbooks

I used to think of SSH articles as setup notes.

Now I treat them like tiny integration tests.

If the doc contains an SSH config, the doc should also contain the command that proves how OpenSSH resolves it.

My shortlist is now:

# client config resolution
ssh -F ./example.conf -G host-alias

# server config syntax and effective values
sshd -t -f ./sshd_config
sshd -T -f ./sshd_config

# local file modes on macOS/BSD stat
stat -f '%Lp %N' ~/.ssh ~/.ssh/config ~/.ssh/id_ed25519 ~/.ssh/id_ed25519.pub

# local file modes on GNU/Linux
stat -c '%a %n' ~/.ssh ~/.ssh/config ~/.ssh/id_ed25519 ~/.ssh/id_ed25519.pub

That is the whole trick.

Not a framework. Not a yak shave. Just make OpenSSH tell you what it thinks before you tell humans what to think.

Try it on your own config

Pick one host alias from your current ~/.ssh/config and run this:

ssh -G your-host-alias |
  awk '/^(hostname|user|port|identityfile|identitiesonly) /{print}'

Then ask one question:

Is this the connection I thought I had configured?

If the answer is no, congrats. You found the bug before production did.

FAQ

Does Host * always need to be at the bottom?

No. Host * is safe when it sets defaults that do not collide with more specific host blocks. I put it later when it can collide with values such as User, Port, HostName, IdentityFile, or other per-host settings.

How do I check what SSH config is actually used?

Use ssh -G with the host alias:

ssh -G your-host-alias

For a shorter review, filter the output:

ssh -G your-host-alias |
  awk '/^(hostname|user|port|identityfile|identitiesonly) /{print}'

That prints the effective SSH config after OpenSSH has applied matching Host blocks.

Is ssh -G safe to run?

Yes. ssh -G prints the resolved client configuration and exits. It does not open an SSH session to the server.

Should I also test sshd_config?

Yes, if the article or runbook includes server-side SSH settings. Use sshd -t -f ./sshd_config for syntax and sshd -T -f ./sshd_config to inspect effective server settings. Depending on your environment, sshd -T may require privileges or a complete server config fixture.

What is the practical rule for SSH setup guides?

Do not stop at "this config looks right." Add the command that proves what OpenSSH will actually use. For client config, that command is usually ssh -G.

I am Sho Naka, writing as nomurasan. I publish implementation notes from teaching, automation, and small operational mistakes that turned into checklists.

AI helped with English editing and structure. The command evidence, corrections, and final responsibility are mine.