Elasticsearch,Filebeat,Kibana部署,添加图表及elastalert报警
服务端安装Elasticsearch和Kibana（需要安装openjdk1.8以上）
安装方法：https://www.elastic.co
以Ubuntu为例：
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -sudo apt-get install apt-transport-httpsecho "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.listapt-get updateapt-get install elasticsearchapt-get install kibanaelasticsearch配置
cat /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearchpath.logs: /var/log/elasticsearchnetwork.host: 172.16.80.28http.port: 9200discovery.zen.ping.unicast.hosts: ["172.16.80.28","172.16.80.155"]
附件有文件elasticsearch.map
Kibana配置
cat /etc/kibana/kibana.yml
server.port: 5601server.host: "172.16.80.28"elasticsearch.url: "https://172.16.80.28:9200"Elastalert（需要Python2.7）
下载 https://github.com/Yelp/elastalert/releases
进入elastalert目录
pip install -r requirements.txtpython setup.py installelastalert-create-index
前台运行elastalert
python -m elastalert.elastalert --verbose --rule rules/my_rule.yaml
安装参考:
https://elastalert.readthedocs.io/en/latest/running_elastalert.html#downloading-and-configuring
https://blog.csdn.net/df007df/article/details/54773391
elastalert配置
cat /etc/elastalert/rules/my_rule.yaml
es_host: 172.16.80.28es_port: 9200name: filebeat rulestype: frequencyindex: filebeat*num_events: 5timeframe: hours: 1filter:- query: query_string: query: "message: *error*"alert:- "email"email:- "aaa@qq.com"- "bbb@qq.com"alert_text: "Ref Log https://172.16.80.28:5601/app/kibana"smtp_host: smtp.exmail.qq.comsmtp_port: 25smtp_ssl: falsesmtp_auth_file: /etc/elastalert/rules/smtp_auth_file.yamlfrom_addr: aaa@qq.com
cat /etc/elastalert/rules/smtp_auth_file.yaml
user: "aaa@qq.com"password: "邮箱密码"
cat /etc/elastalert/config.yaml
rules_folder: /etc/elastalert/rulesrun_every: minutes: 1buffer_time: minutes: 15es_host: 172.16.80.28es_port: 9200s_url_prefix: elasticsearchwriteback_index: elastalert_statusalert_time_limit: days: 2 elastalert配置注解rules_folder：用来加载下一阶段rule的设置，默认是example_rulesrun_every：用来设置定时向elasticsearch发送请求buffer_time：用来设置请求里时间字段的范围，默认是45分钟es_host：elasticsearch的host地址es_port：elasticsearch 对应的端口号use_ssl：可选的，选择是否用SSL连接es，true或者falsees_username：es认证的usernamees_password：es认证的passwordwriteback_index：elastalert产生的日志在elasticsearch中的创建的索引alert_time_limit：失败重试的时间限制
告警参考
https://blog.csdn.net/gamer_gyt/article/details/52917116
https://elastalert.readthedocs.io/en/latest/ruletypes.html
客户端安装Filebeat安装
以Ubuntu为例：
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -sudo apt-get install apt-transport-httpsecho "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.listapt-get updateapt-get install filebeatfilebeat配置
cat /etc/filebeat/filebeat.yml
filebeat.prospectors:- type: log enabled: true paths: - /var/log/nginx/*.log path: ${path.config}/modules.d/*.yml reload.enabled: falsesetup.template.settings: index.number_of_shards: 3setup.kibana: host: "172.16.80.28:5601"output.elasticsearch: hosts: ["172.16.80.28:9200"]EFK备注
日志位置 /var/lib/elasticsearch/nodes/
日志大小 暂时为345Mb
日志轮循 暂时不知道在哪设置
elasticsearch.map内容：
# uint mapping
{
"settings" : {
"index" : {
"number_of_replicas" : 1,
"number_of_shards" : 5
}
},
"mappings" : {
"values" : {
"properties" : {
"itemid" : {
"type" : "long"
},
"clock" : {
"format" : "epoch_second",
"type" : "date"
},
"value" : {
"type" : "long"
}
}
}
}
}
# dbl mapping
{
"settings" : {
"index" : {
"number_of_replicas" : 1,
"number_of_shards" : 5
}
},
"mappings" : {
"values" : {
"properties" : {
"itemid" : {
"type" : "long"
},
"clock" : {
"format" : "epoch_second",
"type" : "date"
},
"value" : {
"type" : "double"
}
}
}
}
}
# str mapping
{
"settings" : {
"index" : {
"number_of_replicas" : 1,
"number_of_shards" : 5
}
},
"mappings" : {
"values" : {
"properties" : {
"itemid" : {
"type" : "long"
},
"clock" : {
"format" : "epoch_second",
"type" : "date"
},
"value" : {
"fields" : {
"analyzed" : {
"index" : true,
"type" : "text",
"analyzer" : "standard"
}
},
"index" : false,
"type" : "text"
}
}
}
}
}
# text mapping
{
"settings" : {
"index" : {
"number_of_replicas" : 1,
"number_of_shards" : 5
}
},
"mappings" : {
"values" : {
"properties" : {
"itemid" : {
"type" : "long"
},
"clock" : {
"format" : "epoch_second",
"type" : "date"
},
"value" : {
"fields" : {
"analyzed" : {
"index" : true,
"type" : "text",
"analyzer" : "standard"
}
},
"index" : false,
"type" : "text"
}
}
}
}
}
# log mapping
{
"settings" : {
"index" : {
"number_of_replicas" : 1,
"number_of_shards" : 5
}
},
"mappings" : {
"values" : {
"properties" : {
"itemid" : {
"type" : "long"
},
"clock" : {
"format" : "epoch_second",
"type" : "date"
},
"value" : {
"fields" : {
"analyzed" : {
"index" : true,
"type" : "text",
"analyzer" : "standard"
}
},
"index" : false,
"type" : "text"
}
}
}
}
}