简介： ELK是开源日志，收集集中日志。主要组件分为：elasticsearch、kibana、filebeat、logstash

组件名称 版本号
elasticsearch 8.7.0
kiabana 8.7.0
filebeat 8.7.0
logstash 8.7.0

1、创建安装目录、数据存放目录

1
2
3
sudo mkdir -p /usr/local/elastic

sudo mkdir -p /data/elastic

2、下载elasticsearch

1
sudo wget -P /usr/local/src https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.6.2-linux-x86_64.tar.gz

3、将解压的elasticsearch文件移动到安装目录中

1
sudo mv /usr/local/src/elasticsearch-8.6.2 /usr/local/elastic

4、修改elasticsearch的配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 单机安装的情况下使用下面配置
cat <<eof >/usr/local/elastic/elasticsearch-8.6.2/config/elasticsearch.yml
node.name: node04
path.data: /data/elastic/data
path.logs: /data/elastic/logs
bootstrap.memory_lock: false
network.host: 192.168.137.13
http.port: 9200
discovery.seed_hosts: ["node04"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["node04"]
http.host: 0.0.0.0
eof

5、添加到systemd系统服务中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cat <<eof |tee /usr/lib/systemd/system/elasticsearch.service
[Unit]
Description=Elasticsearch
After=network-online.target

[Service]
Type=forking
User=elastic
Group=elastic
PrivateTmp=true

Environment=ES_HOME=/usr/local/elastic/elasticsearch-8.6.2
Environment=ES_PATH_CONF=/usr/local/elastic/elasticsearch-8.6.2/config
Environment=PID_DIR=/usr/local/elastic/elasticsearch-8.6.2/elasticsearch.pid
WorkingDirectory=/usr/local/elastic/elasticsearch-8.6.2

ExecStart=/usr/local/elastic/elasticsearch-8.6.2/bin/elasticsearch -d
#ExecStart=/usr/local/elastic/elasticsearch-8.6.2/bin/elasticsearch -p \${PID_DIR} --quiet

LimitNOFILE=65535
LimitNPROC=65535
LimitAS=infinity
LimitFSIZE=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target
eof

6、下载kibana

1
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.6.2-linux-x86_64.tar.gz

7、解压

1
tar xf /usr/local/src/kibana-8.6.2.tar.gz -C /usr/local/src

8、修改配置文件

1
2
3
4
5
6
7
8
cat <<eof |tee /usr/local/elastic/kibana-8.7.0/config/kibana.yml
server.port: 5601
server.host: "192.168.137.13"
elasticsearch.hosts: ["https://192.168.137.13:9200"]
pid.file:  /usr/local/elastic/kibana-8.7.0/kibana.pid
i18n.locale: "zh-CN"
eof

9、添加到系统服务

1
2
3
4
5
6
7
8
9
10
11
12
13
cat <<eof | tee >/usr/lib/systemd/system/kibana.service
[Unit]
Description=filebeat
After=network-online.target

[Service]
User=elastic
Group=elastic
ExecStart=/usr/local/elastic/filebeat-8.7.0/filebeat -e -c /usr/local/elastic/filebeat-8.7.0/filebeat.yml

[Install]
WantedBy=multi-user.target
eof

10、启动服务

1
systemctl enable --now kibana.service

11、启用密码认证（修改elasticsearch文件）

12、下载filebeat

1
wget

13、解压

1
tar xf /usr/local/src/filebeat-8.7.0-linux-x86_64.tar.gz -C /usr/local/src

14、修改配置文件

1
2
3
4
5
6
7
8
9
10
11
cat <eof |tee /usr/local/elastic/filebeat-8.7.0/filebeat.yml 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "192.168.137.13:5601"
output.logstash:
  hosts: ["192.168.137.13:5044"]
eof

15、添加到系统服务中

1
2
3
4
5
6
7
8
9
10
11
12
13
cat <<eof |tee /usr/lib/systemd/system/filebeat.service
[Unit]
Description=filebeat
After=network-online.target

[Service]
User=elastic
Group=elastic
ExecStart=/usr/local/elastic/filebeat-8.7.0/filebeat -e -c /usr/local/elastic/filebeat-8.7.0/filebeat.yml

[Install]
WantedBy=multi-user.target
eof

16、下载logstash

1
wget

17、解压

1
tar xf /usr/local/src/logstash-8.7.0-linux-x86_64.tar.gz -C /usr/local/src

18、修改配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
cat <<eof |tee /usr/local/elastic/logstash-8.7.0/config/logstash.conf
input {
  beats {
    port => 5044
  }
  file {
    path => "/var/log/nginx/access.log"
    type => "access.log"
    start_position => "beginning"
    codec => json
  }
}

filter {
  grok {
    match => {
     "message" => '%{IP:client} - - \[%{HTTPDATE:time}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:bytes} "-" "%{DATA:user-agent}" "-"'
    }
  }
  date {
    match => [ "logdate", "MMM dd yyyy HH:mm:ss Z" ]
  }
  mutate {
    remove_field => [ "timestamp" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.137.13:9200"]
    #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    index => "nginx-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "nickqiu"
  }
}
eof

19、添加到系统服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat <<eof |tee /usr/lib/systemd/system/logstash.service
[Unit]
Description=logstash
After=network-online.target

[Service]
User=elastic
Group=elastic

ExecStart=/usr/local/elastic/logstash-8.7.0/bin/logstash -f /usr/local/elastic/logstash-8.7.0/config/logstash.conf

[Install]
WantedBy=multi-user.target
eof

20、heartbeat下载