how to filter elasticsearch tail for a particular type and use regex

[satishmane]

Hi,

I have type and message fields in my elasticseach logstash entries.

I want to filter entries by type and then apply regular expression message field to tail only matching entries.
I tried this, but it does not work.

nodejs ./esTail.js --hostport=localost:9200 --index=logstash-* --context=type=test-component --regex='(Offset Report)' --regexflags="gm"

Above command should filter messages based matching type and then apply regex to show only entries that have "Offset Report" string in it.

Am I making any mistake here?

[edperry]

--context=key=value is used to replace variables in the default.search

{ "index": "{{index}}", "scroll": "30s", "sort": "_doc", "size" : {{ fetchsize }}, "body":{ "query": { "bool": { "must": [ { "range": { "@timestamp": { "gt": "{{from}}", "to": "now" } } } ],

so you would have to create a new default.search to handle {{type}}

"must": [ { "range": { "@timestamp": { "gt": "{{from}}", "to": "now" } } },{ "terms": { "type": "{{type}}" } } ]

by updating the search you should get the effect you want. (PS i did not test this search add) if it gives you problems I will look at in in the morning EST

[satishmane]

Hi,
thanks for quick reply. I tried as you suggested. I added terms as below in "must" section in default.search and ran again. I got error as below

"must": [
{
"range": {
"@timestamp": {
"gt": "{{from}}",
"to": "now"
}
}
},
{ "terms": {"type":"{{type}}"}}
]

ubuntu$>esTail$ node ./esTail.js --index=logstash-* --hostport=localhost:9200 --context=type='taskmanager.log'
[ '--context', 'type', 'taskmanager.log' ]
Connected to Elasticsearch cluster.
ERR:[parsing_exception] [terms] query does not support [type], with { line=1 & col=97 } :: {"path":"/logstash-*/_search","query":{"scroll":"30s","sort":"_doc","size":100},"body":"{"query":{"bool":{"must":[{"range":{"@timestamp":{"gt":"now-10m","to":"now"}}},{"terms":{"type":"taskmanager.log"}}],"must_not":[{"constant_score":{"filter":{"bool":{"must_not":{"exists":{"field":"message"}}}}}},{"constant_score":{"filter":{"bool":{"must_not":{"exists":{"field":"@timestamp"}}}}}}],"should":[]}},"sort":{"@timestamp":{"order":"asc"}}}","statusCode":400,"response":"{"error":{"root_cause":[{"type":"parsing_exception","reason":"[terms] query does not support [type]","line":1,"col":97}],"type":"parsing_exception","reason":"[terms] query does not support [type]","line":1,"col":97},"status":400}"}

[edperry]

my apology for the delay, it is "Term" not Terms, I just confirmed it looking at

"must": [ { "range": { "@timestamp": { "gt": "{{from}}", "to": "now" } } },{ "term": { "type": "{{type}}" } } ]

[satishmane]

Awasome. it worked. But regular expression to grep only entries with OffSet Report is not working. Please let me know if you are still online. OR I will play with it till you reply. thanks again

node ./esTail.js --index=logstash-* --hostport=localhost:9200 --context=type=taskmanager.log --regex='(Offset Report)' --regexflags="gm"

[edperry]

I would have to play with it myself, I wrote this back in elk 2.X days It would be going against the "message" field

So I think I would recommend that you just update the default.search (since we have already changed it) with your additional string
just add
"must": [ { "range": { "@timestamp": { "gt": "{{from}}", "to": "now" } } },{ "term": { "type": "{{type}}" } } ],"query_string" : { "default_field" : "message", "query" : "\"Offset Report\"" }

while it does not work as pretty as regex with yellow highlighting it will get you working till I can sit down and go though it see whats up.

you could even add --context=qs=Offset\ Report

to make it more dynamic. with this update

"must": [ { "range": { "@timestamp": { "gt": "{{from}}", "to": "now" } } },{ "term": { "type": "{{type}}" } } ],"query_string" : { "default_field" : "message", "query" : "{{ qs }}" }

[satishmane]

Thanks. Perfect. it works. Slight correction did on brackets

"must": [
{
"range": {
"@timestamp": {
"gt": "{{from}}",
"to": "now"
}
}
},
{ "term": {"type":"{{type}}"}},
{"query_string" : { "default_field" : "message", "query" : "{{ qs }}" }}
]

[satishmane]

Hi,

The date tagging done by elastic search index is different from the date in actual log message. for example, here is example of log entry. I want to filter based on date highlighted in bold color i.e. actual date of log message and not elastic index date. I tried applying regular expression as below, but it does not filter.

node ./esTail.js --index=logstash-* --hostport=localhost:9200 --context=type=taskmanager.log --context=qs=2018-05-02

2018-05-08T05:35:48.797Z: logstash-2018.05.08:2018-05-02 13:02:58,040 INFO

I tried, --from='now-10m'. but it does not show latest messages.

thanks
Satish

[edperry]

Well if there is a difference from Elasticsearch and your Log then you will have to check how your ingesting the data. esTail works on the ES date "@timestamp" in logstash you can use the "date" filter to take your log date and set the ES data to it. So much easier then worrying about regex's

Which is the more supportable way.

[satishmane]

thanks. agree. I realized after posting comment that ingestion need to be fixed

[eperry]

:) oh and on the regex item. That is not a "filter' it is a way of breaking out information your looking for so you don;t have to read though the entire line

nodejs ./esTail.js --hostport=hd1melk23lx.digital.hbc.com:9205 --index=apache-* --regex="(POST)"

So for example when the REGEX matches, it will echo the line AND another line with the matched data.

2018-05-08T13:18:02.754Z: bay-2018.05.08:10.250.169.4 - - [08/May/2018:09:18:02 -0400] "POST /XXXXXX/XXXXX HTTP/1.1" 200 373 1155  "-" "Solr[org.apache.solr.client.solrj.impl.CommonsHttpSolrServer] 1.0" ""
	regex: ["POST"]

so you could always grep the output for "regex"