Fluent-BitのParserでログフォーマットを識別して適切なログ出力に整形する

Parserを使用していない場合

出力されたログは適切にParseされていない。

1
2
3

fluent-bit_1  | [0] 4fb66927922a: [1621578165.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1  | [0] 71d75b79d476: [1621578168.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1  | [0] 4fb66927922a: [1621578171.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]

既定のnginxのParserを使う

ParserでFILTERでParserを指定する方法が記載されている。

[SERVICE]
    Parsers_File /path/to/parsers.conf

[INPUT]
    Name dummy
    Tag  dummy.data
    Dummy {"data":"100 0.5 true This is example"}

[FILTER]
    Name parser
    Match dummy.*
    Key_Name data
    Parser dummy_test

[OUTPUT]
    Name stdout
    Match *

これを参考にFILTERを利用する設定を定義する。
parsers.confでは様々なParserが定義されており、今回はnginxを使用する。

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

Parsers_File parsers.confを設定することで、この既定のParserが利用可能になる。

[SERVICE]
    Log_Level       info
    Parsers_File    parsers.conf

[INPUT]
    Name    forward
    Listen  0.0.0.0
    Port    24224

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser nginx

[OUTPUT]
    Name stdout
    Match *

Name parserのFILTER定義を作成する。
対象となるのは現在logで出力されている部分だ。

"log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""

Parseされたenginxアクセスログ

nginxのログ部分はParseされて出力された。

web1_1        | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:35:35 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.23.0.1 - - [21/May/2021:07:35:36 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1  | [0] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [1] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [2] 9dfdc1a8b440: [1621582535.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [0] 5b4d36337166: [1621582536.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [1] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [2] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]

Parse対象以外の項目を残す

しかし,元々出力されえていたcontainer_idなどが消えてしまっている。
そこでReserve_Data Onを使用して元の項目を残す。ただし、Prase対象のlogは必要ないので、Preserve_Key Offで残さないようにする。

[SERVICE]
    Log_Level       info
    Parsers_File    parsers.conf

[INPUT]
    Name    forward
    Listen  0.0.0.0
    Port    24224

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser nginx
    Preserve_Key Off
    Reserve_Data On

[OUTPUT]
    Name stdout
    Match *

結果は以下。期待通りの項目が出力されている。

web1_1        | 172.23.0.1 - - [21/May/2021:07:47:03 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1  | [0] 9dfdc1a8b440: [1621583223.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]
fluent-bit_1  | [1] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]
fluent-bit_1  | [2] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "source"=>"stdout", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1"}]
fluent-bit_1  | [3] 9dfdc1a8b440: [1621583225.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]

Fluent-BitのParserでログフォーマットを識別して適切なログ出力に整形する

Parserを使用していない場合

既定のnginxのParserを使う

Parseされたenginxアクセスログ

Parse対象以外の項目を残す

nullpo

プロセスのメモリダンプをとる

KeyringでOSのパスワード管理機構を利用する

PythonでGmailを使ったメール送信

SMTPHandlerでログ出力をメール通知する

SlackのIncoming WebHooksを使う

Hexoを使った静的サイト作成

モダンWebホスティングサービスNetlify

Hexoの基本操作チュートリアル

Hexo Markdown

画像の引用

ハイパーリンク

カテゴリ・タグ

続きを読む

CI/CD