FluentBitの最小限の構成

INPUT FORWARD

INPUT/FORWARDにIPで待ち受けて、標準出力に出力する例がある。
OUTPUT/STDOUTで標準出力に出力できる。

1
2
3
4
5
6
7
8
9
10
[INPUT]
    Name              forward
    Listen            0.0.0.0
    Port              24224
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M

[OUTPUT]
    Name   stdout
    Match  *

PortはTCP port to listen for incoming connections.だ。

FILTER STDOUT

標準出力への出力はFILTERでも可能。
FILTER/STDOUTにFILTERを使った標準出力が示されている。

1
2
3
4
5
6
7
8
9
10
11
[INPUT]
    Name cpu
    Tag  cpu.local

[FILTER]
    Name  stdout
    Match *

[OUTPUT]
    Name  null
    Match *

OUTPUT/NULLは単純にイベントを破棄する。

Logging pipeline locally

Running a Logging Pipeline LocallyではDocker Composeの設定例がある。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
      - elasticsearch
  elasticsearch:
    image: elasticsearch:7.6.2
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node

Dockerのログを集約するサイドカー構成

docker-composeの設定例

以下がfluent-bitサイドカーとして設定した内容。
fluent-bitは24224ポートで待ち受け、web1、web2コンテナはLoggingDriverを使って待ち受けポートへログを送る。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
version: '3.5'
services:
  web1:
    image: nginx:latest
    ports:
      - "8888:80" 
    depends_on:
      - fluent-bit
    command: [nginx-debug, '-g', 'daemon off;']
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        fluentd-async-connect: "false"
  web2:
    image: nginx:latest
    ports:
      - "8889:80" 
    depends_on:
      - fluent-bit
    command: [nginx-debug, '-g', 'daemon off;']
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        fluentd-async-connect: "false"
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
        - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    ports:
      - "24224:24224"

fluent-bit.confは以下の通り。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[SERVICE]
    Log_Level       info

[INPUT]
    Name    forward
    Listen  0.0.0.0
    Port    24224

[FILTER]
    Name stdout
    Match *

[OUTPUT]
    Name null
    Match *

実行例

ログを見ると、web1、web2で出力しているログはfluent-bitへ送られ、fluent-bitのログとして出力されている。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
PS > docker-compose up
Docker Compose is now in the Docker CLI, try `docker compose up`

WARNING: Found orphan containers (stdout_web_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating stdout_fluent-bit_1 ... done
Creating stdout_web2_1         ... done
Creating stdout_web1_1         ... done
Attaching to stdout_fluent-bit_1, stdout_web2_1, stdout_web1_1
fluent-bit_1  | Fluent Bit v1.7.4
fluent-bit_1  | * Copyright (C) 2019-2021 The Fluent Bit Authors
fluent-bit_1  | * Copyright (C) 2015-2018 Treasure Data
fluent-bit_1  | * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
fluent-bit_1  | * https://fluentbit.io
fluent-bit_1  |
fluent-bit_1  | [2021/05/21 06:22:11] [Warning] [config] I cannot open /fluent-bit/etc/parser.conf file
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [engine] started (pid=1)
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [storage] version=1.1.1, initializing...
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [storage] in-memory
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
fluent-bit_1  | [2021/05/21 06:22:11] [error] [sp] could not initialize stream processor
fluent-bit_1  | [2021/05/21 06:22:11] [error] [engine] could not initialize stream processor
fluent-bit_1  | [2021/05/21 06:22:11] [Warning] [config] I cannot open /fluent-bit/etc/stream_processor.conf file
web1_1        | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web1_1        | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web1_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web1_1        | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web2_1        | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web2_1        | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web2_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web2_1        | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web2_1        | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web2_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web2_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
web2_1        | /docker-entrypoint.sh: Configuration complete; ready for start up
web1_1        | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web1_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web1_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"log"=>"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Configuration complete; ready for start up"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"log"=>"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh"}]
web1_1        | /docker-entrypoint.sh: Configuration complete; ready for start up
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 2021/05/21 06:22:33 [error] 31#31: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.24.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8889", referrer: "http://localhost:8889/"
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET /favicon.ico HTTP/1.1" 404 556 "http://localhost:8889/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.24.0.1 - - [21/May/2021:06:22:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:41 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.24.0.1 - - [21/May/2021:06:22:43 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.24.0.1 - - [21/May/2021:06:22:54 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Configuration complete; ready for start up", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578153.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578153.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET /favicon.ico HTTP/1.1" 404 556 "http://localhost:8889/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578153.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stderr", "log"=>"2021/05/21 06:22:33 [error] 31#31: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.24.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8889", referrer: "http://localhost:8889/""}]
fluent-bit_1  | [0] 71d75b79d476: [1621578158.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578161.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:41 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578163.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:43 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578165.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1  | [0] 71d75b79d476: [1621578168.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1  | [0] 4fb66927922a: [1621578171.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
Gracefully stopping... (press Ctrl+C again to force)
Stopping stdout_web1_1         ... done
Stopping stdout_web2_1         ... done
Stopping stdout_fluent-bit_1   ... done