Published 2020-10-27.
Last modified 2022-01-28.
Time to read: 3 minutes.
For some reason the ssh
certificates that AWS generated for me 3 years ago are no longer recognized by Ubuntu 20.10.
This article shows how to create new certificates and push them to an AWS server that was just upgraded to Ubuntu 20.01,
and now cannot be logged into.
I decided to use OpenSSH to generate the new keypairs instead of AWS to generate the keypairs because
the current problem stems from
AWS-generated keys
gradually becoming incompatible with OpenSSH servers.
This article describes the following:
- Tracking down the problem
- Create a new
ssh
certificate keypair. - Even though the system cannot accept logins,
the new
ssh
public key must be copied to theubuntu
user’s~/.ssh
directory on the problem server. This is done by defining a user data script on the server instance prior to booting it. - Log into the problem server using the new certificates.
- Complete the upgrade to XUbuntu 20.10.
- Remove the user data script from the problem server instance.
Discovery
I was able to log into another of my machines (gojira
),
so first I wanted to know if the problem machines (va
and france
) had OpenSSH configured differently.
I used the comm Linux utility to perform the comparison.
$ for x in cipher mac key kex; do comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort) done $ for x in cipher mac key kex; do comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort) done
So the problem was not OpenSSH configuration per se. Next I wondered if the ssh connections to the problem machines were different somehow from the ssh connection to the working machine.
$ comm -3 <(ssh -Gva|sort) <(ssh -G gojira|sort) hostname gojira hostname va identityfile ~/.ssh/id_rsa identityfile ~/.ssh/sslTest user mslinn user ubuntu
So the only differences were the hostnames and the keys offered.
One of the problem machines, france
, resided on scaleway
.
I used the most recently available
bootscript
to launch the server and examined /var/log/auth.log
.
I found this:
sshd[27025]: Unable to negotiate with 205.185.123.173 port 40555: no matching key exchange method found.
Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 [preauth]
This error message is produced by OpenSSH 7.0+.
The release notes say
“Support for the 1024-bit diffie-hellman-group1-sha1 key exchange
is disabled by default at run-time. It may be re-enabled using
the instructions at
”
http://www.openssh.com/legacy.html
So it seems that the version of OpenSSH installed with Ubuntu 20.10 rejects my old keys. The release notes for the new version of OpenSSH also indicate that OpenSSH 7.1 will be even stricter:
- “This focus of this release is primarily to deprecate weak, legacy and/or unsafe cryptography”
- “Refusing all RSA keys smaller than 1024 bits (the current minimum is 768 bits)”
-
“Several ciphers will be disabled by default:
blowfish-cbc
,cast128-cbc
, allarcfour
variants and therijndael-cbc
aliases for AES.” -
“MD5-based
HMAC
algorithms will be disabled by default.”
Clearly, I need to generate better ssh
keys.
The version of OpenSSH installed by Ubuntu 20.10 is 8.3:
$ sshd -V unknown option -- V OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f 31 Mar 2020 usage: sshd [-46DdeiqTt] [-C connection_spec] [-c host_cert_file] [-E log_file] [-f config_file] [-g login_grace_time] [-h host_key_file] [-o option] [-p port] [-u len] $ ssh -V OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f 31 Mar 2020
Setup
AWS CLI
I prefer to use the AWS CLI instead of the web console. Installation instructions are here. This article uses the AWS CLI exclusively in favor of the AWS web console.
The code in the remainder of this article references an AWS EC2 instance,
whose id I stored in AWS_PROBLEM_INSTANCE_ID
.
The previous article
shows how I did that.
jq
I also use jq for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this:
$ yes | sudo apt install jq
Name the New Keypair
I wanted to make new ecdsa
keys because
this algorithm is the currently accepted best practice
for commercial security concerns.
ecdsa
stands for
Elliptic Curve Digital Signature Algorithm.
Unfortunately, AWS EC2 only accepts RSA
keys.
The name of the new key pair will be of the form ~/.ssh/rsa-YYYY-MM-DD
.
$ AWS_KEY_PAIR_FILE="$HOME/.ssh/rsa-$( date '+%Y-%m-%d-%M-%S' )" $ echo $AWS_KEY_PAIR_FILE /home/mslinn/.ssh/rsa-2020-11-04-08-24-22
The new public key will be called ~/.ssh/rsa-2020-11-04-08-24-22.pub
and
the new private key will be called ~/.ssh/rsa-2020-11-04-08-24-22
.
Create a New Keypair
- This is how I would have created a new
ECDSA
keypair, if AWS supported that type of encryption.Shell$ ssh-keygen -b 512 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -P "" -t ecdsa Generating public/private ecdsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22.pub The key fingerprint is: SHA256:HEKjAA1GZxHbpwqjm85DXQpQEeIWrcjZ6fl84RHQaHE mslinn@mslinn.com The key’s randomart image is: +---[RSA 512]----+ |=O*Bo.*E | |+.=oo*.o | |+o+.+.o.. | |o= o .o+ . | | o+ +. S | |..o=. o | |o .o . o | |.+ o o | |+o. . | +----[SHA256]-----+ $ chmod 400 $AWS_KEY_PAIR_FILE
-
Instead, I created a new
RSA
keypair like this:Shell$ ssh-keygen -b 2048 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -m PEM -P "" -t rsa Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key’s randomart image is: +---[RSA 2048]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+ $ chmod 400 $AWS_KEY_PAIR_FILE
-
I would have liked to copy the keypair to the problem system using
ssh-copy-id
, but that only works when login is possible.Shell$ ssh-copy-id -i $AWS_KEY_PAIR_FILE ubuntu@$AWS_PROBLEM_IP
-
Instead, I decided to paste the public key into an AWS user data script and execute that script on the problem server the next time it booted.
The purpose of the script is to copy the new public key that was just made to
~/.ssh/
on the problem server. This is the user data script I wrote to install the new public key, calledrescue_ubuntu2010.sh
:
#!/bin/bash KEY_FILE_NAME=/home/ubuntu/.ssh/rsa-2020-11-03.pub cat > "$KEY_FILE_NAME" <<EOF ssh-rsa ABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFA== mslinn@mslinn.com EOF chown ubuntu: /home/ubuntu/.ssh/* chmod 400 "$KEY_FILE_NAME" cat "$KEY_FILE_NAME" >> /home/ubuntu/.ssh/authorized_keys
The script runs on the problem server asroot
next time the system boots, and it reboots the server on the last line. -
The script need to be converted into base 64, in a file called
rescue_ubuntu2010.b64
.Shell$ base64 rescue_ubuntu2010.sh > rescue_ubuntu2010.b64
-
The problem EC2 instance can be shut down like this:
Shell
$ aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID $ aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID
-
With the problem EC2 instance stopped,
its user data was set to the base64-encoded version of the rescue script.
Shell
$ aws ec2 modify-instance-attribute \ --instance-id $AWS_PROBLEM_INSTANCE_ID \ --attribute userData \ --value file://rescue_ubuntu2010.b64
-
Now the problem EC2 instance can be restarted.
The script will add the new key to
/home/ubuntu/.ssh/authorized_keys
and login should be possible.Shell$ aws ec2 start-instances --instance-id $AWS_PROBLEM_INSTANCE_ID { "StartingInstances": [ { "CurrentState": { "Code": 0, "Name": "pending" }, "InstanceId": "i-d3b03954", "PreviousState": { "Code": 80, "Name": "stopped" } } ] } $ aws ec2 wait instance-running --instance-ids $AWS_PROBLEM_INSTANCE_ID
Reset User Data for Next Time
Next time the problem server is stopped, clear the user data so it is not provided the next time the server restarts.
$ aws ec2 modify-instance-attribute \
--instance-id $AWS_PROBLEM_INSTANCE_ID \
--user-data Value=