Published 2020-10-27. Last modified 2022-01-28.
Time to read: 4 minutes.

This page is part of the posts collection, categorized under AWS, Bash, Git.

For some reason the ssh certificates that AWS generated for me 3 years ago are no longer recognized by Ubuntu 20.10. This article shows how to create new certificates and push them to an AWS server that was just upgraded to Ubuntu 20.01, and now cannot be logged into. I decided to use OpenSSH to generate the new keypairs instead of AWS to generate the keypairs because the current problem stems from AWS-generated keys gradually becoming incompatible with OpenSSH servers.

This article describes the following:

1. Tracking down the problem
2. Create a new ssh certificate keypair.
3. Even though the system cannot accept logins, the new ssh public key must be copied to the ubuntu user’s ~/.ssh directory on the problem server. This is done by defining a user data script on the server instance prior to booting it.
4. Log into the problem server using the new certificates.
5. Complete the upgrade to XUbuntu 20.10.
6. Remove the user data script from the problem server instance.

Discovery

I was able to log into another of my machines (gojira), so first I wanted to know if the problem machines (va and france) had OpenSSH configured differently. I used the comm Linux utility to perform the comparison.

Shell

$ for x in cipher mac key kex; do
  comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort)
done

$ for x in cipher mac key kex; do
  comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort)
done

So the problem was not OpenSSH configuration per se. Next I wondered if the ssh connections to the problem machines were different somehow from the ssh connection to the working machine.

Shell

$ comm -3 <(ssh -Gva|sort) <(ssh -G gojira|sort)
hostname gojira
hostname va
        identityfile ~/.ssh/id_rsa
identityfile ~/.ssh/sslTest
        user mslinn
user ubuntu 

So the only differences were the hostnames and the keys offered.

One of the problem machines, france, resided on scaleway. I used the most recently available bootscript to launch the server and examined /var/log/auth.log. I found this: sshd[27025]: Unable to negotiate with 205.185.123.173 port 40555: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 [preauth]

This error message is produced by OpenSSH 7.0+. The release notes say “Support for the 1024-bit diffie-hellman-group1-sha1 key exchange is disabled by default at run-time. It may be re-enabled using the instructions at http://www.openssh.com/legacy.html”

So it seems that the version of OpenSSH installed with Ubuntu 20.10 rejects my old keys. The release notes for the new version of OpenSSH also indicate that OpenSSH 7.1 will be even stricter:

• “This focus of this release is primarily to deprecate weak, legacy and/or unsafe cryptography”
• “Refusing all RSA keys smaller than 1024 bits (the current minimum is 768 bits)”
• “Several ciphers will be disabled by default: blowfish-cbc, cast128-cbc, all arcfour variants and the rijndael-cbc aliases for AES.”
• “MD5-based HMAC algorithms will be disabled by default.”

Clearly, I need to generate better ssh keys.

The version of OpenSSH installed by Ubuntu 20.10 is 8.3:

Shell

$ sshd -V
unknown option -- V
OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f  31 Mar 2020
usage: sshd [-46DdeiqTt] [-C connection_spec] [-c host_cert_file]
            [-E log_file] [-f config_file] [-g login_grace_time]
            [-h host_key_file] [-o option] [-p port] [-u len]

$ ssh -V
OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f  31 Mar 2020 

Setup

AWS CLI

I prefer to use the AWS CLI instead of the web console. Installation instructions are here. This article uses the AWS CLI exclusively in favor of the AWS web console.

The code in the remainder of this article references an AWS EC2 instance, whose id I stored in AWS_PROBLEM_INSTANCE_ID. The previous article shows how I did that.

jq

I also use jq for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this:

Shell

$ yes | sudo apt install jq

Name the New Keypair

I wanted to make new ecdsa keys because this algorithm is the currently accepted best practice for commercial security concerns. ecdsa stands for Elliptic Curve Digital Signature Algorithm.

Unfortunately, AWS EC2 only accepts RSA keys. The name of the new key pair will be of the form ~/.ssh/rsa-YYYY-MM-DD.

Shell

$ AWS_KEY_PAIR_FILE="$HOME/.ssh/rsa-$( date '+%Y-%m-%d-%M-%S' )"

$ echo $AWS_KEY_PAIR_FILE
/home/mslinn/.ssh/rsa-2020-11-04-08-24-22 

The new public key will be called ~/.ssh/rsa-2020-11-04-08-24-22.pub and the new private key will be called ~/.ssh/rsa-2020-11-04-08-24-22.

Create a New Keypair

1. This is how I would have created a new ECDSA keypair, if AWS supported that type of encryption.
   Shell

   $ ssh-keygen -b 512 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -P "" -t ecdsa
   Generating public/private ecdsa key pair.
   Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22
   Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22.pub
   The key fingerprint is:
   SHA256:HEKjAA1GZxHbpwqjm85DXQpQEeIWrcjZ6fl84RHQaHE mslinn@mslinn.com
   The key’s randomart image is:
   +---[RSA 512]----+
   |=O*Bo.*E         |
   |+.=oo*.o         |
   |+o+.+.o..        |
   |o= o .o+ .       |
   | o+ +.  S        |
   |..o=.  o         |
   |o  .o . o        |
   |.+   o o         |
   |+o.   .          |
   +----[SHA256]-----+
   
   $ chmod 400 $AWS_KEY_PAIR_FILE

2. Instead, I created a new RSA keypair like this:
   Shell

   $ ssh-keygen -b 2048 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -m PEM -P "" -t rsa
   Generating public/private rsa key pair.
   Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22
   Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22.pub
   The key fingerprint is:
   SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com
   The key’s randomart image is:
   +---[RSA 2048]----+
   |  ooE  .*++o+**  |
   |   =.  ooXo=.B.. |
   |  o + o +.X.  =  |
   | o = * . =.o     |
   |. + = . S o      |
   |   o +   .       |
   |    . .          |
   |                 |
   |                 |
   +----[SHA256]-----+
   
   $ chmod 400 $AWS_KEY_PAIR_FILE

3. I would have liked to copy the keypair to the problem system using ssh-copy-id, but that only works when login is possible.
   Shell

   $ ssh-copy-id -i $AWS_KEY_PAIR_FILE ubuntu@$AWS_PROBLEM_IP

4. Instead, I decided to paste the public key into an AWS user data script and execute that script on the problem server the next time it booted. The purpose of the script is to copy the new public key that was just made to ~/.ssh/ on the problem server. This is the user data script I wrote to install the new public key, called rescue_ubuntu2010.sh:
   
   rescue_ubuntu2010.sh

   #!/bin/bash
   
   KEY_FILE_NAME=/home/ubuntu/.ssh/rsa-2020-11-03.pub
   
   cat > "$KEY_FILE_NAME" <<EOF
   ssh-rsa ABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFA== mslinn@mslinn.com
   EOF
   
   chown ubuntu: /home/ubuntu/.ssh/*
   chmod 400 "$KEY_FILE_NAME"
   cat "$KEY_FILE_NAME" >> /home/ubuntu/.ssh/authorized_keys
   

   The script runs on the problem server as root next time the system boots, and it reboots the server on the last line.
5. The script need to be converted into base 64, in a file called rescue_ubuntu2010.b64.
   Shell

   $ base64 rescue_ubuntu2010.sh > rescue_ubuntu2010.b64

6. The problem EC2 instance can be shut down like this:
   Shell

   $ aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID
   
   $ aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID

7. With the problem EC2 instance stopped, its user data was set to the base64-encoded version of the rescue script.
   Shell

   $ aws ec2 modify-instance-attribute \
     --instance-id $AWS_PROBLEM_INSTANCE_ID \
     --attribute userData \
     --value file://rescue_ubuntu2010.b64

8. Now the problem EC2 instance can be restarted. The script will add the new key to /home/ubuntu/.ssh/authorized_keys and login should be possible.
   Shell

   $ aws ec2 start-instances --instance-id $AWS_PROBLEM_INSTANCE_ID
       {
     "StartingInstances": [
         {
             "CurrentState": {
                 "Code": 0,
                 "Name": "pending"
             },
             "InstanceId": "i-d3b03954",
             "PreviousState": {
                 "Code": 80,
                 "Name": "stopped"
             }
         }
     ]
   }
   
   $ aws ec2 wait instance-running --instance-ids $AWS_PROBLEM_INSTANCE_ID

Reset User Data for Next Time

Next time the problem server is stopped, clear the user data so it is not provided the next time the server restarts.

Shell

$ aws ec2 modify-instance-attribute \
  --instance-id $AWS_PROBLEM_INSTANCE_ID \
  --user-data Value=

Share to Twitter, Hacker News, LinkedIn.