This is caused by the default Elasticsearch limit for an Index of 1000 total fields. Run the below command from the ELK server directly to increase this limit to 100000
Error in Logstash logs:
[2018-04-04T16:38:46,241][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"index-name", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x25e996cf>], :response=>{"index"=>{"_index"=>"index-name", "_type"=>"doc", "_id"=>"h3OGkWIBjOPkI6XxYjvC", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] in index [index-name] has been exceeded"}}}}
The Fix
Run the below command from the ELK server (Or a server that connect to Elasticsearch on port 2000). There is no downtime doing this and no impact to users. Simply swap out index-name in the the below for the name of the index you need to raise the limit on:
curl -XPUT 'localhost:9200/index-name/_settings' -H 'Content-Type: application/json' -d' { "index" : { "mapping" : { "total_fields" : { "limit" : "100000" } } } }'