SSH communicator does not send SHA2 signatures #27768

ecky-l · 2021-02-13T16:28:15Z

Hello,

with the most recent version of CoreOS, 33.20210117.3.2, I cannot use the terraform ssh connection provisioner anymore. It used to work in prior versions.

Problem is that a SSH connection cannot be established by means of the tf connector, but it works with a simple “ssh core@”. Here are two log entries from the hosts syslog, acquired with journalctl while trying to login with either method:

When trying with terraform connector (failure):

Feb 13 14:27:10 k1.local.vlan audit[18899]: USER_LOGIN pid=18899 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="core" exe="/usr/sbin/sshd" hostname=? addr=192.168.56.1 terminal=ssh res=failed'

When trying with “ssh core@…” (success):

Feb 13 14:29:00 k1.local.vlan audit[20006]: USER_LOGIN pid=20006 uid=0 auid=1000 ses=12 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=? addr=192.168.56.1 terminal=/dev/pts/1 res=success'

The user is in both cases the same (core) and authentication is to be performed via RSA pub key from ssh-agent. Here is a simple terraform module to reproduce the case:

resource "null_resource" "copy" {
    connection {
        type = "ssh"
        host = "<some coreos host>"
        user = "core"
        timeout = "10m"
        agent = true
    }
    
    provisioner "file" {
        content = "blabla"
        destination = "ttt"
    }
}

when running “terraform init” and “terraform apply”, the apply hangs and dies. It works for any other host, which is not CoreOS.
When setting the TF_LOG to TRACE, there is a related message:

SSH authentication failed ([email protected]:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Some more context is below under "debug output".

There is some more info on the host's syslog, when I try to connect via the terraform connector:

userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
USER_ERR pid=4191 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=192.168.56.1 addr=192.168.56.1 terminal=ssh res=failed'

But I suspect that this is not fully relevant, since the simple ssh command works with the same key on that host, while the terraform connector doesn't. I suspect it is related to the concrete invocation of ssh from terraform in combination with the particular sshd configuration on this fedora coreos version...

Terraform Version

0.14

Debug Output (relevant parts only)

2021-02-13T16:44:20.301+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:20 [INFO] sleeping for 1s                                                            
2021-02-13T16:44:21.305+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:21 [DEBUG] Connecting to k1.local.vlan:22 for SSH                                    
2021-02-13T16:44:21.305+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:21 [DEBUG] Connection established. Handshaking for user core
2021-02-13T16:44:21.346+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:21 [WARN] SSH authentication failed ([email protected]:22): ssh: handshake failed: s
sh: unable to authenticate, attempted methods [none publickey], no supported methods remain                                                                                              
2021-02-13T16:44:21.346+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:21 [WARN] retryable error: SSH authentication failed ([email protected]:22): ssh: ha
ndshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain                                                                             
2021-02-13T16:44:21.346+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:21 [INFO] sleeping for 2s                                                            
2021-02-13T16:44:23.350+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:23 [DEBUG] Connecting to k1.local.vlan:22 for SSH                                    
2021-02-13T16:44:23.351+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:23 [DEBUG] Connection established. Handshaking for user core                         
2021-02-13T16:44:23.394+0100 [DEBUG] plugin.terraform: file-provisioner (internal) 2021/02/13 16:44:23 [WARN] SSH authentication failed ([email protected]:22): ssh: handshake failed: s
sh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Expected Behavior

It should be possible to connect to coreos hosts with terraform ssh connection and file provisioner (or any other provisioner respectively)

Actual Behavior

ssh connection is not possible with terraform connector, while it is possible by simple ssh command.

Steps to Reproduce

copy the code above into a main.tf, setup a simple coreos VM somewhere, then run

terraform init
terraform apply

References

See also my related forum post at Fedora Discussions.

The text was updated successfully, but these errors were encountered:

jbardin · 2021-02-16T13:35:18Z

Hi @ecky-l,

The logs you present here seem to indicate that the two ssh connections are using different keys for authentication. The log shows key type ssh-rsa not in PubkeyAcceptedKeyTypes, which implies that your ssh connection from the cli is not using an ssh-rsa key, while Terraform is.

Can you confirm what type of key is being chosen when using ssh on the cli?

How many keys are loaded in the agent? If you did not include the full logs from the authentication, there may have been too many key attempts by the client if the desired key was too far down the list.

Thanks!

ecky-l · 2021-02-16T15:13:54Z

Hello @jbardin ,

I have only one key in the agent and it is exactly the only one that is configured in the core account on the machine. There is another ed25519 key in my ~./ssh directory, which could theoretically be picked up for a login attempt, but its a password locked ssh-ed25519 key - and neither did I get a password prompt nor are there any complaints in the log about this key type. It is also definitely not in the agent.

If that matters: I use the ssh enabled gpg-agent with a yubikey, and the ssh key is a subkey of my gpg key. So the ssh-agent is in reality the gpg-agent and just appears as the ssh-agent. But that works otherwise without problems...

ecky-l · 2021-02-16T15:22:03Z

I just did a ssh -vvv ... to the machine and this is what is logged (relevant section only, I changed key hash to ******):

debug1: Next authentication method: publickey
debug1: Offering public key: /Users/eckhard/.ssh/id_rsa RSA SHA256:******* agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/eckhard/.ssh/id_rsa RSA SHA256:******* agent
debug3: sign_and_send_pubkey: RSA SHA256:*******
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).

ecky-l · 2021-02-16T15:37:43Z

Oh, maybe I should mention that I saved the public key, output of ssh-add -L, in a file ~/.ssh/id_rsa.pub. thats why it might appear that he tries to load ~/.ssh/id_rsa above, but recognizes its in the agent. Without this file, the same log entries look so:

debug1: Next authentication method: publickey
debug1: Offering public key: cardno:000614242388 RSA SHA256:****** agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: cardno:000614242388 RSA SHA256:****** agent
debug3: sign_and_send_pubkey: RSA SHA256:******
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).

But the issue is the same when I try with terraform.

jbardin · 2021-02-16T18:44:06Z

Thanks for additional information @ecky-l, but I think the key type ssh-rsa not in PubkeyAcceptedKeyTypes is still the primary issue here. Since the authentication only succeeds with rsa-sha2-512 using what is otherwise the same configuration, it's likely you're seeing: golang/go#37278 and golang/go#39885

Once this is resolved upstream, we can update the packages.

lorengordon · 2021-10-20T17:54:59Z

Perhaps another way to workaround this would be to use the now-supported ed25519 key-type option for the keypair?

https://aws.amazon.com/about-aws/whats-new/2021/08/amazon-ec2-customers-ed25519-keys-authentication/

lorengordon · 2021-10-20T18:19:21Z

And unfortunately, ed25519 is not fips-compliant anyway. So even that wouldn't actually help me here. Ok, time to follow the upstream golang issues...

alisdair · 2021-12-13T15:33:56Z

We're tracking this upstream issue in #30134, closing as duplicate.

github-actions · 2022-01-13T02:11:59Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

ecky-l added bug new new issue not yet triaged labels Feb 13, 2021

jbardin added the waiting-response An issue/pull request is waiting for a response from the community label Feb 16, 2021

jbardin self-assigned this Feb 16, 2021

ghost removed waiting-response An issue/pull request is waiting for a response from the community labels Feb 16, 2021

jbardin added dependencies Auto-pinning provisioner/file provisioner/remote-exec and removed new new issue not yet triaged labels Feb 16, 2021

jbardin changed the title ~~Cannot use the "file" provisioner with ssh on Fedora CoreOS hosts~~ SSH communicator does not send SHA2 signatures Feb 17, 2021

apparentlymart added the v0.14 label Apr 9, 2021

jbardin added v0.15 upstream and removed dependencies Auto-pinning labels May 5, 2021

jbardin mentioned this issue Jul 6, 2021

Terraform can't run remote-exec while normal SSH works as expected #29082

Closed

jbardin added v1.0 and removed v0.15 labels Jul 7, 2021

alisdair mentioned this issue Dec 10, 2021

SSH provisioner incompatible with Fedora 33, 34, 35 and CentOS Stream 9; also RHEL 8 with FIPS #30134

Closed

alisdair closed this as completed Dec 13, 2021

github-actions bot locked as resolved and limited conversation to collaborators Jan 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSH communicator does not send SHA2 signatures #27768

SSH communicator does not send SHA2 signatures #27768

ecky-l commented Feb 13, 2021 •

edited

Loading

jbardin commented Feb 16, 2021 •

edited

Loading

ecky-l commented Feb 16, 2021

ecky-l commented Feb 16, 2021

ecky-l commented Feb 16, 2021 •

edited

Loading

jbardin commented Feb 16, 2021 •

edited

Loading

lorengordon commented Oct 20, 2021

lorengordon commented Oct 20, 2021

alisdair commented Dec 13, 2021

github-actions bot commented Jan 13, 2022

SSH communicator does not send SHA2 signatures #27768

SSH communicator does not send SHA2 signatures #27768

Comments

ecky-l commented Feb 13, 2021 • edited Loading

Terraform Version

Debug Output (relevant parts only)

Expected Behavior

Actual Behavior

Steps to Reproduce

References

jbardin commented Feb 16, 2021 • edited Loading

ecky-l commented Feb 16, 2021

ecky-l commented Feb 16, 2021

ecky-l commented Feb 16, 2021 • edited Loading

jbardin commented Feb 16, 2021 • edited Loading

lorengordon commented Oct 20, 2021

lorengordon commented Oct 20, 2021

alisdair commented Dec 13, 2021

github-actions bot commented Jan 13, 2022

ecky-l commented Feb 13, 2021 •

edited

Loading

jbardin commented Feb 16, 2021 •

edited

Loading

ecky-l commented Feb 16, 2021 •

edited

Loading

jbardin commented Feb 16, 2021 •

edited

Loading