Fixing Index limit exceeded in ELK

This is caused by the default Elasticsearch limit for an Index of 1000 total fields. Run the below command from the ELK server directly to increase this limit to 100000

Error in Logstash logs:

[2018-04-04T16:38:46,241][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"index-name", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x25e996cf>], :response=>{"index"=>{"_index"=>"index-name", "_type"=>"doc", "_id"=>"h3OGkWIBjOPkI6XxYjvC", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] in index [index-name] has been exceeded"}}}}

The Fix

Run the below command from the ELK server (Or a server that connect to Elasticsearch on port 2000). There is no downtime doing this and no impact to users. Simply swap out index-name in the the below for the name of the index you need to raise the limit on:

curl -XPUT 'localhost:9200/index-name/_settings' -H 'Content-Type: application/json' -d'
{
"index" : {
"mapping" : {
"total_fields" : {
"limit" : "100000"
}
}
}
}'

Any comments or questions? Get in touch here or Email me at [email protected]