Error in logstash output http plugin

[GaneshbabuRamamoorthy]

Hi All,

I am trying to update new field to the existing documents using update by query through logstash and below is the configuration I have tried,

input {
   http_poller {
     urls => {
       version => "https://dev/auth/version"
     }
     request_timeout => 60
     schedule => { cron => "* * * * *"}
     codec => "plain"
     metadata_target => "http_poller_metadata"
     add_field => { "log_type" => "AUTH_DEV" }
   }
}
filter {
  if [log_type] == "AUTH_DEV" {
    mutate {
      add_field => {
        "version" => "%{message}"
      }
    remove_field => ["@version","http_poller_metadata","message"]
    }
  }
}
output {
  http {
    url => "https://localhost:9200/test/_update_by_query"
    headers => { "Authorization" => "Basic ZWxhc3RpYzpXTUBlbGDejia==" }
    http_method => "post"
    format => "message"
    content_type => "application/json"
    message => '{
            "script": {
              "source": "ctx._source.number= params.number",
              "lang": "painless",
              "params": {
              "number":{
                "number": "%{version}",
              }
             }
            },
            "query": {
              "term": {
                "app_module": "%{log_type}"
              }
             }
            }'
  }
  stdout {
    codec => rubydebug
  }
}

When I execute the below command we are getting HTTP Code 400 error in the logs,

/apps/logstash/bin/logstash -f /opt/logstash/fileinput/make.conf --path.settings=/opt/logstash --java-execution --path.data=/opt/logstash/SIEM/

[2022-09-14T19:20:52,897][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x4ef287de run>"}
[2022-09-14T19:20:54,859][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>1.95}
[2022-09-14T19:20:55,252][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-09-14T19:20:55,324][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>2.48}
[2022-09-14T19:20:55,362][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * *"}, :timeout=>nil}
[2022-09-14T19:20:55,427][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2022-09-14T19:20:55,568][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
/opt/logstash-7.16.3/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
[2022-09-14T19:21:01,200][ERROR][logstash.outputs.http    ] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"https://localhost:9200/test/_update_by_query", :event=>#<LogStash::Event:0x48341256>}
{
      "log_type" => "AUTH_DEV",
    "@timestamp" => 2022-09-14T11:21:00.869Z,
       "version" => "0.0.4-SNAPSHOT"
}

Though I verified the authorization token using curl in the same server and its working fine,

curl --request GET 'https://localhost:9200' --header 'Authorization: Basic ZWxhc3RpYzpXTUBlbGDejia==' --header 'Content-Type:application/json'

{
  "name" : "localhost",
  "cluster_name" : "logging-dev",
  "cluster_uuid" : "amsRwPiu-CaOYHvzama",
  "version" : {
    "number" : "7.16.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "dkfe4eab2297e949ecsdfsdd4629sdf",
    "build_date" : "2022-01-06T23:43:02.825887787Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Please share your thoughts on the issue and it would be really helpful.

Thanks,
Ganeshbabu R

[yaauie]

I believe that the downstream service is rejecting the request because the payload built from your message is not valid JSON; there is a trailing comma after your interpolated version.

               "number":{
-                "number": "%{version}",
+                "number": "%{version}"
               }

[muratkucuktepe]

I am also getting the same error: I have logstash 7.17 and try to delete an index in elasticsearch with http plugin in output.
Here is my setting:

output {
         http {
             id => "http_index_delete"
             url => "https://MYHOST:9200/%{[@metadata][index_to_delete]}"
             headers => ['Authorization', 'Basic myHASH']
             http_method => "delete"
             cacert => '/mycert.cer'
         }
}

And I got this error:

[2023-07-13T11:14:51,998][ERROR][logstash.outputs.http    ][main][http_index_delete] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"https://MYHOST:9200/article_first_idx", :event=>#<LogStash::Event:0xnnnnn>}

[biloocabba]

Hi, is there any update regarding this bug ? we are facing the same issue, we can't send notification to Teams channel using this HTTP plugin, we are getting the error below :

[HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400,

[dsohaliya-ontic]

Also getting same issue while pushing logs to Splunk using HTTP plugin. Please let us know if there are any updates..

[jsvd]

For those hitting these issues please upgrade this plugin to v5.6.1. It includes #142 which will log the HTTP response body, showing the reason why the endpoint rejected the request.

[Prashantraj45]

your message key is useless here as it sends every key present and does not read the value you specify. use ruby to form the query and remove the message key. it will work. also use json format , works better. example :

mutate {
add_field => {
"[query][bool][minimum_should_match]" => 1
}
}
code => '
should_array = []
fields = ["col1", "col2"]

  fields.each do |field|
    if event.get(field)
      # Convert the string to an actual array
      field_values = event.get(field).gsub("[", "").gsub("]", "").gsub("\"", "").split(", ")
      terms_query = { "terms" => { field => field_values } }
      should_array << terms_query
    end
  end

  event.set("[query][bool][should]", should_array)
'

mutate {
remove_field => ["message", "@Version", "@timestamp", "tags","col1","col2"]
}