Published on 1 March 2018 by

Editor’s note: This post was originally published on Medium. We are republishing it with Abir’s permission.


Photo Credit: New Yorker

I remember the first time I used SSH. We were told on campus that we couldn’t telnet into servers anymore (PS I’m old). I had no idea why at the time (spoiler: it was hackers). Hard to believe this was over 20 years ago, and I’m still learning new SSH tricks (sidenote: I feel like I’ve been learning vi for 20 years too). It was only a few years ago I learned that I could save myself from entering a password by using public key authentication for SSH.

At first it sounded magical, but I worried the process would be really complicated. This deterred me from learning how to do it for the longest time.


Me avoiding learning SSH. Photo Credit: giphy

It turns out that it’s not complicated at all! However, a lot can go wrong and confusing error messages related to SSH drove me mad!

I’ve seen a lot of articles written about how to setup public key authentication for SSH, but I hadn’t seen articles about what to do if things go wrong. So I decided to write one.

The Basics

Let’s say I’ve got 3 machines:

  • professorx.local † - my personal Mac laptop
  • wolverine.vm † - centos 7 vm
  • deadpool.vm † - ubuntu 16.04 vm

I want to be able to easily access the centos and ubuntu machines from my Mac. In this example, I’ll want to be able to log in as the abir user on those machines.

The first thing I need to do create a SSH key on my laptop, professorx.local, using ssh-keygen. To make my life easier, I'm gonna leave the passphrase blank. If you're concerned about security (and you should be) you should not leave the passphrase blank. If you want to set a passphrase I recommend reading this article. For more information about public key authentication for ssh, check out the official documentation.

[[email protected]]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/abir/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/abir/.ssh/id_rsa.
Your public key has been saved in /Users/abir/.ssh/id_rsa.pub.
The key fingerprint is:
b9:67:64:cc:27:19:3a:d9:e0:65:41:bd:f9:cd:18:13 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
|         .o.     |
|           .. E  |
|        . +  o . |
|       . @ oo o  |
|        S O .. * |
|         = o  o o|
|        . o      |
|         o       |
|                 |
+-----------------+
[[email protected]]$ cat /Users/abir/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDsR/hrRCRCgJBFwOPhkcsZYFgvTL4GeeCPAdlCPbDbi3N9v8/mco27VF6Daxf8+nwUBW5gC5uuj1LbrrubdSHh6UniP1uzgLElI47jedB14sfs+wdtTvWgr+mWJIr/efNRqMY1CMqE1A7KXc9B2oMsVSo1UOKEfuRbdNW8DHkDof7L1dJ3ZmmP87Vx+r6wUMWHK5sVGb9k+69aikqEVJzifIaZDN6PbnPdKIDqKL3WThee1QXtSypb/3V2IH/nZKHwhEEZQH4FxA7Yz7K8G9a/wmGjvboqeLCfnw6vogDlQeHx7InGNuqaoaFP+8ccXi837RhgOhHXLHkjFCRjB/+n [email protected]

Once I’ve got the public key generated I need to append the contents of it to the .ssh/authorized_keys file on the remote machines. I have 3 ways to get that public key on those machines:

a. Copy and Paste
I can copy the contents of /Users/abir/.ssh/id_rsa.pub into my clipboard.

Pro Tip: If you're on a Mac, you can easily copy the contents of a file into the clipboard from the command line using the pbcopy command.

[[email protected]]$ pbcopy < ~/.ssh/id_rsa.pub

Now I need to log into wolverine, and take the output from the clipboard and paste it into /home/abir/.ssh/authorized_keys.

[[email protected]]$ echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDsR/hrRCRCgJBFwOPhkcsZYFgvTL4GeeCPAdlCPbDbi3N9v8/mco27VF6Daxf8+nwUBW5gC5uuj1LbrrubdSHh6UniP1uzgLElI47jedB14sfs+wdtTvWgr+mWJIr/efNRqMY1CMqE1A7KXc9B2oMsVSo1UOKEfuRbdNW8DHkDof7L1dJ3ZmmP87Vx+r6wUMWHK5sVGb9k+69aikqEVJzifIaZDN6PbnPdKIDqKL3WThee1QXtSypb/3V2IH/nZKHwhEEZQH4FxA7Yz7K8G9a/wmGjvboqeLCfnw6vogDlQeHx7InGNuqaoaFP+8ccXi837RhgOhHXLHkjFCRjE/+n [email protected] >> /home/abir/.ssh/authorized_keys

Caution: Copy and pasting the contents can be tricky and error prone.

b. Use scp

To simplify things, I could have also scp’d id_rsa.pub to wolverine, and then appended the contents of that file to/home/abir/.ssh/authorized_keys


a[email protected]]$ scp id_rsa.pub [email protected]:./id_rsa_professorx.pub password: **** [[email protected]]$ sudo cat ./id_rsa_professorx.pub >> /home/abir/.ssh/authorized_keys

c. Use ssh-copy-id
I could have also used the ssh-copy-id command

[[email protected]]$ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
password: ****

I should be able to ssh into wolverine.vm now.

[[email protected]]$ ssh [email protected]
The authenticity of host 'wolverine.vm (10.32.174.159)' can't be established.
ECDSA key fingerprint is 51:31:cb:67:ce:28:c8:62:87:3a:a6:b7:87:0a:c5:d7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'wolverine.vm,10.32.174.159' (ECDSA) to the list of known hosts.
Last login: Wed Nov  1 16:26:26 2017 from 10.0.9.38
[[email protected]]$

Woo hoo! Sounds easy, right? Here are common problems that I’ve run into in the past…

Problem #1: Unknown host

If you ping the machine you want to log into and you’re seeing an “Unknown host” error then the hostnames are not resolvable by the client.


[[email protected]]$ ping wolverine.vm -c 2 ping: cannot resolve wolverine.vm: Unknown host

Solution:

I can make wolverine.vm resolvable by appending an entry to my /etc/hosts file.

[[email protected]]$ sudo -- sh -c "echo '10.32.174.160  wolverine.vm wolverine' >> /etc/hosts"

Let’s try again.

[[email protected]]$ ping wolverine.vm -c 2
PING wolverine.vm (10.32.174.159): 56 data bytes
64 bytes from 10.32.174.159: icmp_seq=0 ttl=60 time=1.350 ms
64 bytes from 10.32.174.159: icmp_seq=1 ttl=60 time=2.356 ms
--- wolverine.vm ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.350/1.853/2.356/0.503 ms

Sweet!

Problem #2: Connection refused

If you try to ssh into the machine using a username and password but you’re getting “Connection refused” then:

  • the SSH daemon (sshd) is not running on the machine and/or
  • the port 22 (the default SSH port) isn’t open.

You can use ssh -v to see if the port is open. If the port wasn't open, the results would look something like this:

[[email protected]]$ ssh -v deadpool.vm
OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/abir/.ssh/config
debug1: /Users/abir/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to deadpool.vm [10.32.173.180] port 22.
debug1: connect to address 10.32.173.180 port 22: Connection refused
ssh: connect to host deadpool.vm port 22: Connection refused

Solution:

To resolve this issue you’ll need to install and configure openssh on the machine, and make sure the machine’s firewall has port 22 open. There are different ways to set firewall rules and install sshd on different operating systems so I’m not going into them here.

Problem #3: Permission Denied

If you try to ssh into the machine you could also get “Permission Denied” error.

[[email protected]]$ ssh [email protected]
Last login: Wed Nov  1 16:26:26 2017 from 10.0.9.38
[[email protected]]$
[[email protected]]$ ssh [email protected]
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Solution:

This most likely means that you don’t have your SSH creds set up correctly on the remote machine. To resolve this you’ll need to run through “The Basics” from above.

General Troubleshooting Tips

  1. Pass the -v flag to the ssh command to get more information about what might be happening.
  2. Try re-creating the public/private key. I had problems when I tried passing -t (type) and -b (bits) to ssh-keygen. When I tried executing simply ssh-keygen with no additional parameters and passed in parameters via stdin it worked fine.
  3. Sometimes the .ssh/known_hosts file on the client machine can cause issues. Try removing the remote machine entries from the file, and then try ssh’ing into those machines again.
  4. Make sure you generate the public key on the client (professorx.local in my case) and put it on the remote servers (wolverine.vm and deadpool.vm). A common mistake is to do the reverse: generate the key on the remote servers and place them on the client.
    Nerdy analogy: Professor X needs permission from Wolverine and Deadpool to read their minds. Think of the authorized_keys file as a list of people Deadpool and Wolverine let control their minds.
  5. Make sure you’ve got the right permissions set on the files in .sshfolder.
[[email protected]]$ chmod 700 ~/.ssh && chmod 600 ~/.ssh/*
[[email protected]]$   chmod 600 ~/.ssh/authorized_keys && chmod 700 ~/.ssh/

Once you’ve got public key authentication for SSH set up you can use a tool like Puppet Bolt to easily execute commands on those remote machines. For example, if I want to see the free disk space on wolverine.vm and deadpool.vm I could run:

[[email protected]]$ bolt command run 'df -h' --nodes wolverine.vm,deadpool.vm -u abir
wolverine.vm:
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        16G  2.6G   14G  17% /
devtmpfs        902M     0  902M   0% /dev
tmpfs           920M     0  920M   0% /dev/shm
tmpfs           920M   81M  840M   9% /run
tmpfs           920M     0  920M   0% /sys/fs/cgroup
tmpfs           184M     0  184M   0% /run/user/1000
deadpool.vm:
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        16G   12G  4.1G  75% /
devtmpfs        901M     0  901M   0% /dev
tmpfs           920M     0  920M   0% /dev/shm
tmpfs           920M   97M  824M  11% /run
tmpfs           920M     0  920M   0% /sys/fs/cgroup
tmpfs           184M     0  184M   0% /run/user/1000
Ran on 2 nodes in 0.40 seconds

† All Marvel characters and the distinctive likeness(es) thereof are Trademarks & Copyright © 1941–2017 Marvel Characters, Inc. All rights reserved. Please don’t sue me.

Abir Majumdar is a sales engineer at Puppet.

Share via:
Posted in:

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.