Clever Geek Handbook

grokエクスポーターを使用して非構造化ログからプロメテウスメトリックを作成する方法

grok exporterに関する2件の投稿の翻訳があります。



最初の翻訳：grokエクスポーターを使用して非構造化ログからプロメテウスメトリックを作成する方法



grok exporterについて話します。この記事では、grokエクスポーターを使用して、非構造化ログからprometheusメトリックを生成する方法について説明します。





Grok ELK (ElasticSearch, Logstash, Kibana) Fabian Stäber grok exporter.



grok exporter => https://github.com/fstab/grok_exporter



1: Grok exporter



zip grok exporter https://github.com/fstab/grok_exporter/releases.



  1. ​ (releases) ( v0.2.7).
  2. ​ zip-, . — 64- Linux. .


wget https://github.com/fstab/grok_exporter/releases/download/v0.2.7/grok_exporter-0.2.7.linux-amd64.zip


  1. ​ .
  2. ​ , grok exporter.


[root@localhost grok_exporter-0.2.7.linux-amd64]# ./grok_exporter -config ./config.yml
Starting server on http://localhost.localdomain:9144/metrics


http://localhost.localdomain:9144/metrics.



2:



Grok exporter. , .



30.07.2016 04:33:03 10.3.4.1 user=Nijil message="logged in"
30.07.2016 06:47:03 10.3.4.2 user=Alex message="logged failed"
30.07.2016 06:55:03 10.3.4.2 user=Alex message="logged in"
30.07.2016 07:03:03 10.3.4.3 user=Alan message="logged in"
30.07.2016 07:37:03 10.3.4.1 user=Nijil message="logged out"
30.07.2016 08:47:03 10.3.4.2 user=Alex message="logged out"
30.07.2016 14:34:03 10.3.4.3 user=Alan message="logged out"


, . Prometheus .



1 , , config.xml, grok exporter. .



global:
    config_version: 2
input:
    type: file
    path: ./example/nijil.log  # Specify the location of the your log
    readall: true              # This should be True if you want to read whole log and False if you want to read only new lines.
grok:
    patterns_dir: ./patterns    
metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'

server:
    port: 9144


.



global:
    # Config version
input:
    # How to read log lines (file or stdin).
grok:
    # Available Grok patterns.
metrics:
    # How to map Grok fields to Prometheus metrics.
server:
    # How to expose the metrics via HTTP(S).


3: Grok exporter



, , Prometheus.



metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'


grok – %{SYNTAX:SEMANTIC}, SYNTAX — , , SEMANTIC – . %{HOSTNAME:instance}, HOSTNAME – grok, IP- , IP- ( ), . , SYNTAX , , IP- . , DATE, TIME, HOSTNAME, USER GREEDYDATA , , " " .



, , . , . , (SEMANTIC of the SYNTAX) . . . . (Counter), grok exporter , .



grok exporter ./grok_exporter -config ./config.yml . , user_activity, .



# TYPE user_activity counter
user_activity{user="Alan"} 2
user_activity{user="Alex"} 3
user_activity{user="Nijil"} 2


Prometheus . , prometheus, Prometheus .



: Apache grok exporter



https://www.robustperception.io/getting-metrics-from-apache-logs-using-the-grok-exporter



, .



, , , , . grok. , Apache, access.log:



x.x.x.x - - [20/Jan/2020:06:25:24 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16061 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16064 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET /blog/rss HTTP/1.1" 301 3478 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET / HTTP/1.1" 200 16065 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET /blog/feed HTTP/1.1" 200 3413 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET /feed HTTP/1.1" 200 6496 "-" "Emacs Elfeed 3.2.0"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"


:



wget https://github.com/fstab/grok_exporter/releases/download/v1.0.0.RC2/grok_exporter-1.0.0.RC2.linux-amd64.zip




unzip grok_exporter-*.zip
cd grok_exporter*amd64


:



cat << 'EOF' > config.yml
global:
    config_version: 2
input:
    type: file
    path: access.log
    readall: true
grok:
    patterns_dir: ./patterns
metrics:
    - type: counter
      name: apache_http_response_codes_total
      help: HTTP requests to Apache
      match: '%{COMBINEDAPACHELOG}'
      labels:
          method: '{{.verb}}'
          path: '{{.request}}'
          code: '{{.response}}'
server:
    port: 9144
EOF


grok exporter:



./grok_exporter -config config.yml


http://localhost:9144/metrics :



# HELP apache_http_response_codes_total HTTP requests to Apache
# TYPE apache_http_response_codes_total counter
apache_http_response_codes_total{code="200",method="GET",path="/"} 5
apache_http_response_codes_total{code="200",method="GET",path="/blog/feed"} 1
apache_http_response_codes_total{code="200",method="GET",path="/feed"} 1
apache_http_response_codes_total{code="301",method="GET",path="/blog/rss"} 1


readall, , .



Grok – , Logstash (Logstash — L ELK). , , Apache. Grok , , . COMMMONAPACHELOG , , 



COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)


. , . Go ( Prometheus alerting notification) .



, Grok , :



    - type: summary
      name: apache_http_response_bytes
      help: Size of HTTP responses
      match: '%{COMMONAPACHELOG}'
      value: '{{.bytes}}'


, , :



    - type: gauge 
      name: apache_http_last_request_seconds
      help: Timestamp of the last HTTP request
      match: '%{COMMONAPACHELOG}'
      value: '{{timestamp "02/Jan/2006:15:04:05 -0700" .timestamp}}'


(timestamp) grok exporter, time.Parse Golang. (divide), .



, Grok. , , , .





More articles:


All Articles