Repository navigation
S3 input plugin does not work with IAM role WebIdentity #239
Description
Hi,
I'm using IAM role attached to the Service Account with WebIdentity in amazon EKS, but I'm not able to get logstash s3 input plugin working using the service account IAM role and not the node instance role.
As an input configuration we have the following:
s3 {
bucket => "${LOGBUCKET}"
prefix => "elb/xyz-app/"
region => "${AWS_REGION}"
add_field => { "context" => "xyz-app" }
tags => ["elblogs"]
additional_settings => {
"force_path_style" => true
"follow_redirects" => false
}
After removing the node role IAM polices for the S3 bucket we get the following error:
[2022-02-22T10:59:07,836][ERROR][logstash.inputs.s3 ][main][bc7c59a365d681e2cbd0d4cdac2804f2cd6088892917e7a15bccaade4e5cd31f] Unable to list objects in bucket {:exception=>Aws::S3::Errors::AccessDenied, :message=>"Access Denied", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/param_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/response_paging.rb:26:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/response_target.rb:21:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/request.rb:70:in send_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:207:in block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/request.rb:24:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/operations.rb:139:in all_batches'", "org/jruby/RubyEnumerator.java:396:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/collection.rb:18:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:143:in list_new_files'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:185:in process_files'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:133:in block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in interval'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:132:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:409:in inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:400:in block in start_input'"], :prefix=>"elb/xyz-app/"}
The Trust Policy on the Service Account role looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/BABCDEFAAABBB"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.eu-central-1.amazonaws.com/id/BABCDEFAAABBB": "sts.amazonaws.com",
"oidc.eks.eu-central-1.amazonaws.com/id/BABCDEFAAABBB": "system:serviceaccount:logging:logstash"
}
}
}
]
}
And the policies attached to the role:
{
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::logs-dev",
"arn:aws:s3:::logs-dev/*"
],
"Effect": "Allow",
"Sid": "AllowLogstashToGetS3Logs"
}
We also tried to add the role_arn in the S3 input configuration:
s3 {
bucket => "${LOGBUCKET}"
prefix => "elb/xyz-app/"
region => "${AWS_REGION}"
role_arn => "${SERVICE_ACCOUNT_ROLE_ARN}"
add_field => { "context" => "xyz-app" }
tags => ["elblogs"]
additional_settings => {
"force_path_style" => true
"follow_redirects" => false
}
But we get the the following weird error:
[2022-02-23T12:33:17,299][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Aws::STS::Errors::AccessDenied: User: arn:aws:sts::123456789:assumed-role/eks-dev-NodeInstanceRole-AAAI/i-xyz is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789:role/es-dev-ElasticSearch-LogstashRole-BBB>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/param_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/response_target.rb:21:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/request.rb:70:in send_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:207:in block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/assume_role_credentials.rb:49:in refresh'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/refreshing_credentials.rb:20:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/assume_role_credentials.rb:40:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-mixin-aws-5.0.0/lib/logstash/plugin_mixins/aws_config/v2.rb:67:in assume_role'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-mixin-aws-5.0.0/lib/logstash/plugin_mixins/aws_config/v2.rb:17:in aws_options_hash'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:439:in get_s3object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:106:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:390:in start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:315:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/beats.conf", "/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x35938dae run>"}
[2022-02-23T12:33:17,311][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create
Logstash (7.16.3)
logstash-input-s3 plugin (3.8.1)
Has anyone used before the sts:AssumeRoleWithWebIdentity and made it work?
Thank you
Activity
kares commentedon Feb 24, 2022
What and how are the AWS credentials set? I do not see any hints for that in the plugin configuration ..
mmujat commentedon Feb 24, 2022
I am using a service account token secret. I haven't declared any AWS credentials in the plugin configuration because I assumed the service account token will be used. But it seams it is not.
This works for logstash, I am able to push logs to AWS OpenSearch, but when it comes to getting logs from S3 I get an access denied.
ottramst commentedon Mar 7, 2022
Experiencing the exact same issues as described above.
Seems like this plugin uses a pretty outdated version of AWS SDK which doesn't support AWS IRSA
According to AWS documentation the SDK version must be >= 3.58.0, while this plugin uses 2.11.632
Source: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html