AnsibleでUbuntuの初期設定を行う

 
カテゴリー Ansible Linux   タグ

AnsibleでUbuntuの設定を行う

プレイブックの実行

1
2
3
ansible-playbook unattended-upgrades.yml -i hosts
ansible-playbook install-ubuntu-packages.yml -i hosts
ansible-playbook ufw.yml -i hosts

インベントリファイル「hosts」

初期設定を行うinitグループを設定する。

  • ansible_user/ansible_passwordは対象Ubuntuのログインユーザ、パスワード
  • ansible_become_passwordはsudo時のパスワード
1
2
3
4
5
6
7
8
[ubuntu]
target

[ubuntu:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_port=※カスタマイズしたポート※
ansible_user=※ansible用ユーザ名※
ansible_ssh_private_key_file=~/.ssh/id_rsa

プレイブック「unattended-upgrades.yml」

セキュリティパッチや推奨パッチを自動インストールするUnattended-upgradesを設定する。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
---
- hosts: ubuntu

  vars:
  gather_facts: no
  become: true

  tasks:
    - name: Configure debconf
      debconf: name=unattended-upgrades question=unattended-upgrades/enable_auto_updates vtype=boolean value='true'

    - name: Install unattended-upgrades
      apt: name=unattended-upgrades

    - name: Configure unattended-upgrades
      command:
        cmd: dpkg-reconfigure -f noninteractive unattended-upgrades
        creates: /etc/apt/apt.conf.d/20auto-upgrades

    - name: Enable periodic for download and clean
      lineinfile:
        dest: /etc/apt/apt.conf.d/20auto-upgrades
        line: "{{ item }}"
      with_items:
        - 'APT::Periodic::Download-Upgradeable-Packages "1";'
        - 'APT::Periodic::AutocleanInterval "7";'

    - name: Enable upgrade recommend updates
      lineinfile:
        dest: /etc/apt/apt.conf.d/50unattended-upgrades
        regexp: '^//\t+\"\${distro_id}\:\${distro_codename}-updates\";'
        line: "\t\"${distro_id}:${distro_codename}-updates\";"

    - name: Enable automatic reboot
      lineinfile:
        dest: /etc/apt/apt.conf.d/50unattended-upgrades
        regexp: '^//Unattended-Upgrade::Automatic-Reboot \"false\";'
        line: "Unattended-Upgrade::Automatic-Reboot \"true\";"

    - name: Set automatic reboot time
      lineinfile:
        dest: /etc/apt/apt.conf.d/50unattended-upgrades
        regexp: '^//Unattended-Upgrade::Automatic-Reboot-Time \"02:00\";'
        line: "Unattended-Upgrade::Automatic-Reboot-Time \"04:00\";"

    - name: Enable remove unused kernel packages
      lineinfile:
        dest: /etc/apt/apt.conf.d/50unattended-upgrades
        regexp: '^//Unattended-Upgrade::Remove-Unused-Kernel-Packages \"true\";'
        line: "Unattended-Upgrade::Remove-Unused-Kernel-Packages \"true\";"

    - name: Enable remove unused dependencies
      lineinfile:
        dest: /etc/apt/apt.conf.d/50unattended-upgrades
        regexp: '^//Unattended-Upgrade::Remove-Unused-Dependencies \"false\";'
        line: "Unattended-Upgrade::Remove-Unused-Dependencies \"true\";"

    - name: Reboot the machine (Wait for 5 min)
      reboot:
        reboot_timeout: 300

プレイブック「install-ubuntu-packages.yml」

パッケージを追加する。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
---
- hosts: ubuntu

  vars:
  gather_facts: no
  become: true

  tasks:
    - name: Install net-tools
      apt:
        update_cache: yes
        name:
          - net-tools

    - name: Install curl
      apt:
        update_cache: yes
        name:
          - curl

    - name: Install tree
      apt:
        update_cache: yes
        name:
          - tree

    - name: Install htop
      apt:
        update_cache: yes
        name:
          - htop

プレイブック「ufw.yml」

ファイアウォールを有効化する。
デフォルトDENYでポート番号を変更したSSHポートのみ許可する

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
---
- hosts: ubuntu

  vars:
    - sshd_port: "{{ lookup('env', 'SSHD_PORT') }}"
  gather_facts: no
  become: true

  tasks:
    - name: Deny all access from any
      ufw:
        default: deny
        direction: incoming

    - name: Allow all access to port {{ sshd_port }} (SSH)
      ufw:
        rule: allow
        proto: tcp
        port: "{{ sshd_port }}"

    - name: Enable firewall
      ufw:
        state: enabled

コメント・シェア

AnsibleでUbuntuに専用ユーザで秘密鍵で接続可能にする

 
カテゴリー Ansible Linux   タグ

AnsibleでUbuntuに接続可能にする

プレイブックの実行

ansible-playbook add-user-ssh.yml -i hosts

インベントリファイル「hosts」

初期設定を行うinitグループを設定する。

  • ansible_user/ansible_passwordは対象Ubuntuのログインユーザ、パスワード
  • ansible_become_passwordはsudo時のパスワード
1
2
3
4
5
6
7
8
9
[init]
target

[init:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_port=22
ansible_user=※ユーザ名※
ansible_password=※パスワード※
ansible_become_password=※パスワード※

プレイブック「add-user-ssh.yml」

Ansible用のユーザ追加とSSHDの初期設定を行う。

  • lookup(‘env’,’XXXXX’)はAnsible実行ホストの環境変数を参照する
  • sudo実行可能なユーザとしてANSIBLE_USERNAMEで指定したユーザを追加する
  • 追加したユーザにSSH用の公開鍵を登録する
  • SSHDのポート番号変更やルートログイン禁止の設定を行いsshdをリスタートする
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
---
- hosts: init

  vars:
    - ansible_user_name: "{{ lookup('env', 'ANSIBLE_USERNAME') }}"
    - ansible_user_password: "{{ lookup('env', 'ANSIBLE_PASSWORD') }}"
    - sshd_port: "{{ lookup('env', 'SSHD_PORT') }}"
  gather_facts: no
  become: true

  tasks:
    - name: Add a new user named {{ ansible_user_name }}
      user:
        name: "{{ ansible_user_name }}"
        shell: /bin/bash
        password: "{{ ansible_user_password }}"
    
    - name: Add ansible user to the sudoers
      copy:
        dest: "/etc/sudoers.d/{{ ansible_user_name }}"
        content: "{{ ansible_user_name }}  ALL=(ALL)  NOPASSWD: ALL"
    
    - name: Deploy SSH Key
      authorized_key:
        user: "{{ ansible_user_name }}"
        key: "{{ lookup('file', '/root/.ssh/id_rsa.pub') }}"
        state: present

    - name: Change port to {{ sshd_port }}
      lineinfile:
        dest: /etc/ssh/sshd_config
        regexp: "^Port"
        insertafter: "^#Port"
        line: "Port {{ sshd_port }}"

    - name: Disable Root Login
      lineinfile:
        dest: /etc/ssh/sshd_config
        regexp: "^PermitRootLogin"
        line: "PermitRootLogin no"
        state: present
        backup: yes
      notify:
        - restart sshd

  handlers:
    - name: restart sshd
      service:
        name: sshd
        state: restarted

発生したエラーと対策

エラー「sudo: a password is required」

sudoのパスワードが指定されていない。

1
fatal: [target]: FAILED! => {"changed": false, "module_stderr": "Shared connection to target closed.\r\n", "module_stdout": "sudo: a password is required\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

ansible_become_passwordでsudoパスワードを指定する

エラー「/usr/bin/python: not found」

pythonが見つからない。

ansible_python_interpreter=/usr/bin/python3でpythonのパスを明示する

1
fatal: [target]: FAILED! => {"changed": false, "module_stderr": "Shared connection to target closed.\r\n", "module_stdout": "\r\n/bin/sh: 1: /usr/bin/python: not found\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}

コメント・シェア

Ansibleをdocker-composeで動かす

 
カテゴリー Ansible Container   タグ

dockerコンテナをansible実行ホストとして構成する

docker-composeで環境をカプセル化してGitでPlaybookとともに管理したい。

最終的な構成

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: '3.5'

services:
  tutorial:
    image: python:3-slim
    tty: true
    working_dir: /work
    entrypoint: ./entrypoint.sh
    command: /bin/bash
    volumes:
        - .:/work
    environment:
      ANSIBLE_CONFIG: .
      ANSIBLE_USERNAME: ${ANSIBLE_USERNAME}
      ANSIBLE_PASSWORD: ${ANSIBLE_PASSWORD}
    secrets:
      - secret_key
      - public_key

secrets:
  secret_key:
    file: ./.ssh/ubuntu.key
  public_key:
    file: ./.ssh/ubuntu.key.pub

entrypoint.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#!/bin/bash

#
# SSHキー
#
mkdir -p ~/.ssh
chown -R root:root ~/.ssh
chmod -R 0700 ~/.ssh
cp -ip /run/secrets/secret_key ~/.ssh/id_rsa
cp -ip /run/secrets/public_key ~/.ssh/id_rsa.pub
chmod -R 0600 ~/.ssh

#
# Ansibleインストール
#
apt-get install sshpass -y
pip install ansible

exec "$@"

ansible.cfg

1
2
[defaults]
host_key_checking = False

発生したエラーと対策

SecretsをつかってSSH用の公開鍵、秘密鍵を共有する

Secretsは**/run/secrets/**から参照できる
entrypoint.shでrootアカウントのキーとしてコピーし、パーミッションを設定する。

エラー「you must install the sshpass program」

ログイン時にパスワードを使用したログインを行う場合は、sshpassをインストールする必要がある。

1
fatal: [target]: FAILED! => {"msg": "to use the 'ssh' connection type with passwords, you must install the sshpass program"}

エラー「Please add this host’s fingerprint to your known_hosts file to manage this host.」

初回ログイン時などfingerprintの対話操作が含まれるのでエラーになってしまう。

1
fatal: [target]: FAILED! => {"msg": "Using a SSH password instead of a key is not possible because Host Key checking is enabled and sshpass does not support this.  Please add this host's fingerprint to your known_hosts file to manage this host."}

ansible.cfgをカレントディレクトリに作成しhost_key_checking = Falseを設定する。

1
2
[defaults]
host_key_checking = False

ansible.cfgが読み込まれない

docker-composeを実行するのがWindowsの場合、パーミッションが777になり読み込まれない。
export ANSIBLE_CONFIG=.のように環境変数を定義して明示的にansible.cfgの場所を明示することで読み込ませる。

Python3環境でaptでインストールしたAnsibleが動かない

aptで以下のようにインストールでき、ansibleコマンドも実行可能だが、妙なエラーが出る。

1
2
3
4
apt-get update
apt-get install -y software-properties-common
apt-add-repository --yes --update ppa:ansible/ansible
apt-get install -y ansible
1
fatal: [xxxx.xxxxxx.xxxx]: FAILED! => {"changed": false, "module_stderr": "Shared connection to xxxx.xxxxxx.xxxx closed.\r\n", "module_stdout": "\r\n/home/ansible/.ansible/tmp/ansible-tmp-1621777634.977247-71421390121516/AnsiballZ_user.py:17: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\r\n  import imp\r\nTraceback (most recent call last):\r\n  File \"/tmp/ansible_user_payload_frbeg21u/ansible_user_payload.zip/ansible/module_utils/basic.py\", line 279, in get_distribution\r\nAttributeError: module 'platform' has no attribute '_supported_dists'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1621777634.977247-71421390121516/AnsiballZ_user.py\", line 113, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1621777634.977247-71421390121516/AnsiballZ_user.py\", line 105, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1621777634.977247-71421390121516/AnsiballZ_user.py\", line 48, in invoke_module\r\n    imp.load_module('__main__', mod, module, MOD_DESC)\r\n  File \"/usr/lib/python3.8/imp.py\", line 234, in load_module\r\n    return load_source(name, filename, file)\r\n  File \"/usr/lib/python3.8/imp.py\", line 169, in load_source\r\n    module = _exec(spec, sys.modules[name])\r\n  File \"<frozen importlib._bootstrap>\", line 604, in _exec\r\n  File \"<frozen importlib._bootstrap_external>\", line 783, in exec_module\r\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\r\n  File \"/tmp/ansible_user_payload_frbeg21u/__main__.py\", line 2611, in <module>\r\n  File \"/tmp/ansible_user_payload_frbeg21u/__main__.py\", line 2516, in main\r\n  File \"/tmp/ansible_user_payload_frbeg21u/__main__.py\", line 403, in __new__\r\n  File \"/tmp/ansible_user_payload_frbeg21u/ansible_user_payload.zip/ansible/module_utils/basic.py\", line 337, in load_platform_subclass\r\n  File \"/tmp/ansible_user_payload_frbeg21u/ansible_user_payload.zip/ansible/module_utils/basic.py\", line 289, in get_distribution\r\nAttributeError: module 'platform' has no attribute 'dist'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

pipでインストールすればOK。

1
pip install ansible

コメント・シェア

Fluent-BitとFluentdのコンセプトと特徴を把握する

 
カテゴリー Container Logging   タグ

FluentdとFluent-Bit

Fluentd Fluent Bit
Scope Containers / Servers Embedded Linux / Containers / Servers
Language C & Ruby C
Memory ~40MB ~650KB
Performance High Performance High Performance
Dependencies Built as a Ruby Gem, it requires a certain number of gems. Zero dependencies, unless some special plugin requires them.
Plugins More than 1000 plugins available Around 70 plugins available
License Apache License v2.0 Apache License v2.0

Both Fluentd and Fluent Bit can work as Aggregators or Forwarders, they both can complement each other or use them as standalone solutions.

どちらも単独でログアグリゲーター＆フォワーダーとして機能する。

コンセプト

Event or Record

Internally, an Event always has two components (in an array form):

[TIMESTAMP, MESSAGE]

ログ行はイベントとして扱われ、イベントはタイムスタンプとメッセージのタプルで構成されている。

Filtering

In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering.

There are many use cases when Filtering is required like:

  • Append specific information to the Event like an IP address or metadata.
  • Select a specific piece of the Event content.
  • Drop Events that matches certain pattern.

フィルタリングによってイベントに大して、追加、変更、削除を行うことができる。

Tag

Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from.

INPUTでタグ付けを行い、FILTERやOUTPUTで利用する。

Timestamp

SECONDS.NANOSECONDS

タイムスタンプはSECONDSで示すUnixエポックタイムとNANOSECONDSで示されるナノ秒の組で表現される。

Match

Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule.

MATCHによって該当するタグを持つログを複数の宛先にルーティングするなどのルールを記述できる。

Structured Messages

{“project”: “Fluent Bit”, “created”: 1398289291}

Consider MessagePack as a binary version of JSON on steroids.

メッセージは構造化されたJSONデータ（JSONのバイナリ版であるMessagePack）を使う。

コメント・シェア

Fluent-Bitをdocker-composeでサイドカーとして使う

 
カテゴリー Container Logging   タグ

FluentBitの最小限の構成

INPUT FORWARD

INPUT/FORWARDにIPで待ち受けて、標準出力に出力する例がある。
OUTPUT/STDOUTで標準出力に出力できる。

1
2
3
4
5
6
7
8
9
10
[INPUT]
    Name              forward
    Listen            0.0.0.0
    Port              24224
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M

[OUTPUT]
    Name   stdout
    Match  *

PortはTCP port to listen for incoming connections.だ。

FILTER STDOUT

標準出力への出力はFILTERでも可能。
FILTER/STDOUTにFILTERを使った標準出力が示されている。

1
2
3
4
5
6
7
8
9
10
11
[INPUT]
    Name cpu
    Tag  cpu.local

[FILTER]
    Name  stdout
    Match *

[OUTPUT]
    Name  null
    Match *

OUTPUT/NULLは単純にイベントを破棄する。

Logging pipeline locally

Running a Logging Pipeline LocallyではDocker Composeの設定例がある。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
      - elasticsearch
  elasticsearch:
    image: elasticsearch:7.6.2
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node

Dockerのログを集約するサイドカー構成

docker-composeの設定例

以下がfluent-bitサイドカーとして設定した内容。
fluent-bitは24224ポートで待ち受け、web1、web2コンテナはLoggingDriverを使って待ち受けポートへログを送る。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
version: '3.5'
services:
  web1:
    image: nginx:latest
    ports:
      - "8888:80" 
    depends_on:
      - fluent-bit
    command: [nginx-debug, '-g', 'daemon off;']
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        fluentd-async-connect: "false"
  web2:
    image: nginx:latest
    ports:
      - "8889:80" 
    depends_on:
      - fluent-bit
    command: [nginx-debug, '-g', 'daemon off;']
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        fluentd-async-connect: "false"
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
        - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    ports:
      - "24224:24224"

fluent-bit.confは以下の通り。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[SERVICE]
    Log_Level       info

[INPUT]
    Name    forward
    Listen  0.0.0.0
    Port    24224

[FILTER]
    Name stdout
    Match *

[OUTPUT]
    Name null
    Match *

実行例

ログを見ると、web1、web2で出力しているログはfluent-bitへ送られ、fluent-bitのログとして出力されている。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
PS > docker-compose up
Docker Compose is now in the Docker CLI, try `docker compose up`

WARNING: Found orphan containers (stdout_web_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating stdout_fluent-bit_1 ... done
Creating stdout_web2_1         ... done
Creating stdout_web1_1         ... done
Attaching to stdout_fluent-bit_1, stdout_web2_1, stdout_web1_1
fluent-bit_1  | Fluent Bit v1.7.4
fluent-bit_1  | * Copyright (C) 2019-2021 The Fluent Bit Authors
fluent-bit_1  | * Copyright (C) 2015-2018 Treasure Data
fluent-bit_1  | * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
fluent-bit_1  | * https://fluentbit.io
fluent-bit_1  |
fluent-bit_1  | [2021/05/21 06:22:11] [Warning] [config] I cannot open /fluent-bit/etc/parser.conf file
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [engine] started (pid=1)
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [storage] version=1.1.1, initializing...
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [storage] in-memory
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
fluent-bit_1  | [2021/05/21 06:22:11] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
fluent-bit_1  | [2021/05/21 06:22:11] [error] [sp] could not initialize stream processor
fluent-bit_1  | [2021/05/21 06:22:11] [error] [engine] could not initialize stream processor
fluent-bit_1  | [2021/05/21 06:22:11] [Warning] [config] I cannot open /fluent-bit/etc/stream_processor.conf file
web1_1        | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web1_1        | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web1_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web1_1        | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web2_1        | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web2_1        | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web2_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web2_1        | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web2_1        | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web2_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web2_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
web2_1        | /docker-entrypoint.sh: Configuration complete; ready for start up
web1_1        | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web1_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web1_1        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"log"=>"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Configuration complete; ready for start up"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"log"=>"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh"}]
web1_1        | /docker-entrypoint.sh: Configuration complete; ready for start up
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 2021/05/21 06:22:33 [error] 31#31: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.24.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8889", referrer: "http://localhost:8889/"
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET /favicon.ico HTTP/1.1" 404 556 "http://localhost:8889/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.24.0.1 - - [21/May/2021:06:22:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:41 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.24.0.1 - - [21/May/2021:06:22:43 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.24.0.1 - - [21/May/2021:06:22:54 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578135.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Configuration complete; ready for start up", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578153.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578153.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET /favicon.ico HTTP/1.1" 404 556 "http://localhost:8889/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578153.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stderr", "log"=>"2021/05/21 06:22:33 [error] 31#31: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.24.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8889", referrer: "http://localhost:8889/""}]
fluent-bit_1  | [0] 71d75b79d476: [1621578158.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578161.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:41 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1  | [0] 71d75b79d476: [1621578163.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:43 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1  | [0] 4fb66927922a: [1621578165.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1  | [0] 71d75b79d476: [1621578168.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1  | [0] 4fb66927922a: [1621578171.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
Gracefully stopping... (press Ctrl+C again to force)
Stopping stdout_web1_1         ... done
Stopping stdout_web2_1         ... done
Stopping stdout_fluent-bit_1   ... done

コメント・シェア

EFKスタックをdocker-composeで構成するEFK Compose

 
カテゴリー Container Logging   タグ

EFK Compose

Docker Logging Efk Composeで、docker-composeを使ったELKスタックの例がある。

docker-compsoe.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
version: '2'
services:
  web:
    image: httpd
    ports:
      - "8080:80"
    links:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-address: "localhost:24224"
        fluentd-async-connect: "false"
        tag: httpd.access
    depends_on:
      - fluentd

  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    ports:
      - "24224:24224"
      - "24224:24224/udp"

  elasticsearch:
    image: elasticsearch:7.12.0
    expose:
      - 9200
    ports:
      - "9200:9200"

  kibana:
    image: kibana:7.12.0
    links:
      - "elasticsearch"
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

fluentd/conf/fluent.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# fluentd/conf/fluent.conf
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>
  <store>
    @type stdout
  </store>
</match>

fluentd/Dockerfile

1
2
3
# fluentd/Dockerfile
FROM fluent/fluentd:latest
RUN gem install fluent-plugin-elasticsearch --no-rdoc --no-ri

ElasticSeearchがcode78でエラーになる！？

EKF Composeを起動すると、ElasticSearchがcode 78エラーで停止してしまう。
least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configuredというメッセージが出ている

discoveryの設定が必要

このパラメーターはElastickSearchクラスタに関する設定
ElasticSearch7.X系以降はこの設定が必須になっているようだ。

パラメーター 意味
discovery.seed_hosts 設定ベースのseed_hosts
discovery.seed_providers ファイルベースのseed_hosts
cluster.initial_master_nodes クラスターのマスターノード 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,290Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "initialized" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,290Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "starting ..." }
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,309Z", "level": "INFO", "component": "o.e.x.s.c.PersistentCache", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "persistent cache index loaded" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,406Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "publish_address {192.168.32.2:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,524Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
elasticsearch_1  | ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
elasticsearch_1  | bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
elasticsearch_1  | ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,535Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "stopping ..." }
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,556Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "stopped" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,557Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "closing ..." }
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,569Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "closed" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,572Z", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "Native controller process has stopped - no new native processes can be started" }
ekf-stack_elasticsearch_1 exited with code 78

マスターノードのみ設定する

複数台構成ではないので、マスターノードのみ設定する。

1
2
3
4
5
6
7
8
elasticsearch:
  image: elasticsearch:7.12.0
  expose:
    - 9200
  ports:
    - "9200:9200"
  environment:
    - cluster.initial_master_nodes=elasticsearch

code 78はでなくなったがmaster not discovered yet,というメッセージが…

1
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:56:08,340Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "64a558eac1e4", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch] to bootstrap a cluster: have discovered [{64a558eac1e4}{2UB6bFD4QUSAu9rXFwZFhw}{q72ZliAYQOq5r_Zq7-vFxw}{172.19.0.2}{172.19.0.2:9300}{cdfhilmrstw}{ml.machine_memory=8348790784, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4177526784}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305] from hosts providers and [{64a558eac1e4}{2UB6bFD4QUSAu9rXFwZFhw}{q72ZliAYQOq5r_Zq7-vFxw}{172.19.0.2}{172.19.0.2:9300}{cdfhilmrstw}{ml.machine_memory=8348790784, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4177526784}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

ElasticSearch自体の応答はある。

ElasticSearch width=640

標準のノード名がランダムな値となっているので、ノード名とクラスター名を明示的に設定する。

1
2
3
4
5
6
7
8
9
10
elasticsearch:
  image: elasticsearch:7.12.0
  expose:
    - 9200
  ports:
    - "9200:9200"
  environment:
    - node.name=es-node
    - cluster.name=es-cluster
    - cluster.initial_master_nodes=es-node

設定後に各ポートにアクセスすると

ElasticSearch

ElasticSearch(localhost:9200)にアクセスすると、設定したノード名やクラスター名が表示されている。

ElasticSearch width=640

Kibana

Kibana(localhost:5601)へアクセスするとKibanaのUIが表示される。

Kibana width=640

Kibana width=640

Fluent-bitからがElasticSearchにせつぞくできていない・・・Faraday::Error::ConnectionFailed

正常にログが転送されると、KibanaのIndexManagementにfluentd-YYYYMMDDの形式でインデックスが登録される。
しかし、インデックスが存在しない状態…ログを確認すると、以下のエラーが出ている。

1
2
3
fluentd_1        | 2021-08-15 03:12:14 +0000 [warn]: #0 failed to flush the buffer. retry_time=13 next_retry_seconds=2021-08-15 04:17:15 +0000 chunk="5c98f797a0fdcfe5f8155507b4d141b4" error_class=NameError error="uninitialized constant Faraday::Error::ConnectionFailed"
fluentd_1        |   2021-08-15 03:12:14 +0000 [warn]: #0 suppressed same stacktrace
fluentd_1        | 2021-08-15 03:12:14.571214000 +0000 fluent.warn: {"retry_time":13,"next_retry_seconds":"2021-08-15 04:17:15 +0000","chunk":"5c98f797a0fdcfe5f8155507b4d141b4","error":"#<NameError: uninitialized constant Faraday::Error::ConnectionFailed>","message":"failed to flush the buffer. retry_time=13 next_retry_seconds=2021-08-15 04:17:15 +0000 chunk=\"5c98f797a0fdcfe5f8155507b4d141b4\" error_class=NameError error=\"uninitialized constant Faraday::Error::ConnectionFailed\""}

fluent.confではコンテナ名を指定する

いろいろ思考錯誤した結果、fluent.confのhostではコンテナ名を明示する必要がある。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# fluentd/conf/fluent.conf
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host es-node
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>
  <store>
    @type stdout
  </store>
</match>

Kibanaでインデックスるが作成されている

ログインすると前とは異なりいきなりログイン後画面に。

Kibana width=640

左上のメニューからStack Managemetを選択。

Kibana width=640

Index Managementを選択。fluentd-YYYYMMDDの形式でインデックスが登録されている。

Kibana width=640

コメント・シェア

Fluent-BitのParserでログフォーマットを識別して適切なログ出力に整形する

 
カテゴリー Container Logging   タグ

Parserを使用していない場合

出力されたログは適切にParseされていない。

1
2
3
fluent-bit_1  | [0] 4fb66927922a: [1621578165.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1  | [0] 71d75b79d476: [1621578168.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1  | [0] 4fb66927922a: [1621578171.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]

既定のnginxのParserを使う

ParserでFILTERでParserを指定する方法が記載されている。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[SERVICE]
    Parsers_File /path/to/parsers.conf

[INPUT]
    Name dummy
    Tag  dummy.data
    Dummy {"data":"100 0.5 true This is example"}

[FILTER]
    Name parser
    Match dummy.*
    Key_Name data
    Parser dummy_test

[OUTPUT]
    Name stdout
    Match *

これを参考にFILTERを利用する設定を定義する。
parsers.confでは様々なParserが定義されており、今回はnginxを使用する。

1
2
3
4
5
6
[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

Parsers_File parsers.confを設定することで、この既定のParserが利用可能になる。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[SERVICE]
    Log_Level       info
    Parsers_File    parsers.conf

[INPUT]
    Name    forward
    Listen  0.0.0.0
    Port    24224

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser nginx

[OUTPUT]
    Name stdout
    Match *

Name parserのFILTER定義を作成する。
対象となるのは現在logで出力されている部分だ。

1
"log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""

Parseされたenginxアクセスログ

nginxのログ部分はParseされて出力された。

1
2
3
4
5
6
7
8
9
10
11
12
web1_1        | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:35:35 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.23.0.1 - - [21/May/2021:07:35:36 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1        | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1  | [0] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [1] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [2] 9dfdc1a8b440: [1621582535.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [0] 5b4d36337166: [1621582536.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [1] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1  | [2] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]

Parse対象以外の項目を残す

しかし,元々出力されえていたcontainer_idなどが消えてしまっている。
そこでReserve_Data Onを使用して元の項目を残す。ただし、Prase対象のlogは必要ないので、Preserve_Key Offで残さないようにする。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[SERVICE]
    Log_Level       info
    Parsers_File    parsers.conf

[INPUT]
    Name    forward
    Listen  0.0.0.0
    Port    24224

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser nginx
    Preserve_Key Off
    Reserve_Data On

[OUTPUT]
    Name stdout
    Match *

結果は以下。期待通りの項目が出力されている。

1
2
3
4
5
6
7
8
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:03 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1  | [0] 9dfdc1a8b440: [1621583223.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]
fluent-bit_1  | [1] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]
fluent-bit_1  | [2] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "source"=>"stdout", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1"}]
fluent-bit_1  | [3] 9dfdc1a8b440: [1621583225.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]

コメント・シェア

バッチファイルでファイルリストのforル―プ

 
カテゴリー ShellScript   タグ

ファイルリストをforループで出力する

1
2
3
4
5
@echo off

for %%f　in (.\*) do (
    echo %%~nxf
)

%~nx変数でファイルパスを加工する

指定されたファイルパスをファイル名に加工する。
実はfor文の変数は加工する機能がある。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
%* バッチ スクリプト内では、すべての引数 (%1、%2、%3、%4、
    %5 など) を参照します。

バッチ パラメーター (%n) の置換は拡張されました。次のオプション構文
を使うことができます:

    %~1         - すべての引用句 (") を削除して、%1 を展開します。
    %~f1        - %1 を完全修飾パス名に展開します。
    %~d1        - %1 をドライブ文字だけに展開します。
    %~p1        - %1 をパスだけに展開します。
    %~n1        - %1 をファイル名だけに展開します。
    %~x1        - %1 をファイル拡張子だけに展開します。
    %~s1        - 展開されたパスは、短い名前だけを含みます。
    %~a1        - %1 をファイル属性に展開します。
    %~t1        - %1 をファイルの日付/時刻に展開します。
    %~z1        - %1 をファイルのサイズに展開します。
    %~$PATH:1   - PATH 環境変数に指定されているディレクトリを検索し、
                   最初に見つかった完全修飾名に %1 を展開します。
                   環境変数名が定義されていない場合、または
                   検索してもファイルが見つからなかった場合は、
                   この修飾子を指定すると空の文字列に展開されます。

修飾子を組み合わせて、複合結果を得ることもできます:

    %~dp1       - %1 をドライブ文字とパスだけに展開します。
    %~nx1       - %1 をファイル名と拡張子だけに展開します。
    %~dp$PATH:1 - PATH 環境変数に指定されているディレクトリを
                   検索して %1 を探し、最初に見つかったファイル
                   のドライブ文字とパスだけに展開します。
    %~ftza1     - %1 を DIR の出力行のように展開します。

上の例の %1 と PATH は、他の有効な値で置き換えることができ
ます。%~ 構文は有効な引数の数によって区切られます。%~ 修飾子
は %* と同時には使用できません。

コメント・シェア

Visual Studio Codeでまとめてタブインデント戻ししたい

 
カテゴリー Visual Studio Code   タグ

タブインデントを増やすとき

対象のコードブロックを選択して、TAB。

例えば、以下のコードで<meta>と<title>の２行を選択し、

1
2
3
4
<head>
<meta charset="utf-8">
<title>TEST PAGE</title>
</head>

TABをおすと

1
2
3
4
<head>
    <meta charset="utf-8">
    <title>TEST PAGE</title>
</head>

タブインデントを戻すときは

対象コードブロックを選択して、Shift + TAB！

同様に、<meta>と<title>の２行を選択し、

1
2
3
4
<head>
    <meta charset="utf-8">
    <title>TEST PAGE</title>
</head>

Shift + TABをおすと

1
2
3
4
<head>
<meta charset="utf-8">
<title>TEST PAGE</title>
</head>

コメント・シェア

Git for Windowsの文字コード変換

 
カテゴリー Git   タグ

Bashスクリプトの文字コードがおかしくなる

開発はWindows上で行っているが、実行はDockerを使用している。
この環境でgit pullするとBashスクリプトの改行コードがCRLFにかわり、実行エラーになってしまう事象が発生。

改行コードをかえているもの

gitのcore.autocrlf=trueが変換している。

core.autocrlfの挙動

ローカルリポジトリからのcheckout時とcommit時に変換している

Value checkout(IN) commit(OUT)
true LFからCRLF CRLFからLF
input ― CRLFからLF
false ― ―

どうするか？

リポジトリはLF統一にしたいので、inputを選択。

設定

1
2
3
4
5
6
7
8
9
PS > git config -l
...略...
core.autocrlf=true
...略...
PS > git config --global core.autocrlf input
PS > git config -l
...略...
core.autocrlf=input
...略...

コメント・シェア



nullpo

めも


募集中


Japan