Occasional "Connection reset by peer" errors when mounting

Hi guys,

We are using efs-utils from within Docker containers spawned from AWS Batch. It works great, but occasionally we receive this error about 26 seconds after attempting to mount EFS over TLS:

mount.nfs4: Connection reset by peer

We are using the recommended mount command:

mount -t efs -o tls [EFS file system ID]:/ /mnt

This happens in ~0.2% of all mount attempts from all our VPCs. It's a particularly nasty issue because it seems to prevent the mount process from being killed cleanly. Since Apache Commons Exec 1.3's basic ExecuteWatchdog is not able to destroy it, the only remedy I have found is to terminate the EC2 instance.

Any ideas or insights would be greatly appreciated.

Thanks!

Cheers,
-Jon

Thanks for the feedback. We’ll have someone take a look and post a more detailed response once we better understand the issue.

@okeeffes

We are facing the same issue with AWS Batch and EFS. Any updates/workaround would be greatly appreciated.
@okeeffes @jonfryd
Thanks-
Kunal

This is still an issue for sure.

It has occurred three times today on production. We are using the latest version 1.12 of efs-utils.

I had the same issue. I kept getting "mount.nfs4: Connection reset by peer". After getting sure my EFS instances had the same security group as the EC2 ones it worked fine.

Hi there, is any update so far? I noticed the same issue sometimes, I am using the latest version amazon-efs-utils 1.27.1

Any updates please it's affecting critical production systems .

Seem to have run into the same issue -- it looks like sometimes it's not able to resolve the DNS name of the EFS endpoint at boot time (Amazon Linux 2 here). The issue is that it never appears to recover automatically, even when DNS is up later in the boot -- though manually running the mount command works successfully.

Output in the /var/log/amazon/efs/mount.log:

INFO - Starting TLS tunnel: "/usr/bin/stunnel /var/run/efs/stunnel-config.<REDACTED>"
INFO - Started TLS tunnel, pid: 2532
INFO - Starting TLS tunnel: "/usr/bin/stunnel /var/run/efs/stunnel-config.<REDACTED>"
INFO - Starting TLS tunnel: "/usr/bin/stunnel /var/run/efs/stunnel-config.<REDACTED>"
INFO - Started TLS tunnel, pid: 2537
INFO - Started TLS tunnel, pid: 2538
INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:<REDACTED> <REDACTED> -o rw,_netdev,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20112"
INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:<REDACTED> <REDACTED> -o rw,_netdev,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20390"
INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:<REDACTED> <REDACTED> -o rw,_netdev,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20319"
ERROR - Failed to mount <REDACTED>.efs.us-east-1.amazonaws.com at <REDACTED>: returncode=32, stderr="b'mount.nfs4: Connection reset by peer'"
ERROR - Failed to mount <REDACTED>.efs.us-east-1.amazonaws.com at <REDACTED>: returncode=32, stderr="b'mount.nfs4: Connection reset by peer'"
ERROR - Failed to mount <REDACTED>.efs.us-east-1.amazonaws.com at <REDACTED>: returncode=32, stderr="b'mount.nfs4: Connection reset by peer'"

Output from stunnel in the secure log around the same time:

Service [efs] accepted connection from 127.0.0.1:32956
Error resolving '<REDACTED>.efs.us-east-1.amazonaws.com': Neither nodename nor servname known (EAI_NONAME)
No host resolved
Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
Service [efs] accepted connection from 127.0.0.1:34048
Service [efs] accepted connection from 127.0.0.1:39250
Error resolving '<REDACTED>.efs.us-east-1.amazonaws.com': Neither nodename nor servname known (EAI_NONAME)
Error resolving '<REDACTED>.efs.us-east-1.amazonaws.com': Neither nodename nor servname known (EAI_NONAME)
No host resolved
Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
No host resolved
Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket

@gsfraley

For others getting this issue, first, check your security group configuration settings. Creating and managing security groups.

@gsfraley Are you using fstab to automatically mount at runtime? If so, make sure you're specifying _netdev option as well. See Using the EFS mount helper to automatically re-mount EFS file systems with some examples of fstab entries. This makes sure that the filesystem isn't trying to mount before the network systems are up. However, if this isn't the case for you or anyone else running into this issue, please follow up here.

We see this very occasionally when launching an ECS task on EC2, with two separate EFS mount points for the same EFS volume.

From CloudTrail:
Error response from daemon: create xxxxxxxxxxx: VolumeDriver.Create: mounting volume failed: Mount attempt 1/3 failed due to b'mount.nfs4: Connection reset by peer\n', wait 1 sec before next attempt. Mount attempt 2/3 failed due to b'mount.nfs4: Connection reset by peer\n', wait 1 sec before next attempt. b'mount.nfs4: Connection reset by peer'

We are using the latest ECS optimized Amazon Linux 2 AMI. The only customization to efs-utils we have made it to add region = eu-west-1 to the config as we were occasionally seeing issues where the region could not be found, e.g.
Error response from daemon: create xxxxxxxxxxx: VolumeDriver.Create: mounting volume failed: Error retrieving region. Please set the "region" parameter in the efs-utils configuration file.

Adding the region to the config file solved all those error messages, leaving only the connection reset message above. This happens less than the region error.

This config looks promising. I'll play with this:

efs-utils/dist/efs-utils.conf

Lines 54 to 55 in b6151f3

    
           retry_nfs_mount_command_count = 3 
        
           retry_nfs_mount_command_timeout_sec = 15

I forgot to feed back on this. Setting retry_nfs_mount_command_count = 15 in /etc/amazon/efs/efs-utils.conf resolved the issue for us.

We now have an error that occurs even less, but it's still kicking off alerts every week or so, which is annoying:
Error response from daemon: create xxx-xxx: VolumeDriver.Create: mounting volume failed: Unsuccessful retrieval of AWS security credentials at http://169.254.170.2/v2/credentials/xxx.

No clue what's causing this one.

@stewartcampbell

Hi, @stewartcampbell , just checking, are you still hitting the Unsuccessful retrieval of AWS security credentials issue on latest efs-utils version ?

That was quite some time and another job ago, but we eventually moved away from using EFS config inside ECS and just mounting EFS on our EC2 instances then bind mounting on ECS. Had too many weird and wonderful annoyances.

That approach was recommended by AWS after a 6 month running support ticket was going nowhere. Won't work for everyone's use case, but did for us at the time.

Given you have a workaround, I'm going to close this issue. If this comes up again, please feel free to open a new issue

Navigation Menu

Repository navigation

Occasional "Connection reset by peer" errors when mounting #32

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	retry_nfs_mount_command_count = 3
	retry_nfs_mount_command_timeout_sec = 15

Occasional "Connection reset by peer" errors when mounting #32

Description

Activity

okeeffes commented on Jul 31, 2019

kunupat commented on Sep 9, 2019

jonfryd commented on Sep 11, 2019

medinadato commented on Mar 4, 2020

andresdelgadillo commented on Sep 23, 2020

vibhor13 commented on Dec 24, 2021

gsfraley commented on Mar 17, 2022

RyanStan commented on Dec 19, 2022

stewartcampbell commented on Feb 28, 2023

stewartcampbell commented on Feb 28, 2023

stewartcampbell commented on May 26, 2023

wangnyue commented on Jun 2, 2026

stewartcampbell commented on Jun 2, 2026

samuhale commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions