Filebeat6.4 quick start

brief introduction ：

Filebeat Is a lightweight log transmission tool , It has both input and output ends , Usually read data from the log file , Output to Logstash or Elasticsearch . Its function is to collect business The server Log , Output to a log system for centralized management .

Official website ： https://www.elastic.co/cn/products/beats/filebeat

install

System environment ：CentOS 6.x

Software version ：filebeat-6.4.0-x86_64.rpm

Download directly from the official website RPM package , Use rpm -ivh filebeat-6.4.0-x86_64.rpm Can be installed .

To configure • Master profile /etc/filebeat/filebeat.yml

###################### Filebeat Configuration Example #########################

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations.

# Every one here type Indicates that a log reading source is defined , This source is collection Nginx Access log - type: log

  enabled: true

  paths:     - /usr/log/nginx/access/access.log   fields_under_root: true   fields:     alilogtype: nginxacclog

# Collect the error log of a service - type: log

  enabled: true

  paths:     - /var/www/service/storage/logs/error.log   fields_under_root: true   fields:     alilogtype: service_error     serverip: ${serverip}

# Collect the error log of a service , And uses multi row merging - type: log

  enabled: true

  paths:     - /var/www/user_center/storage/logs/SERVER*.log   fields_under_root: true   fields:     alilogtype: usercenter_serverlog     serverip: ${serverip}

  multiline.pattern: '^\['   multiline.negate: true   multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:   # Glob pattern for configuration loading   path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading   reload.enabled: false

  # Period on which files under path should be checked for changes   #reload.period: 10s

#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------

# This part is used to configure log output to Elasticsearch Part of

#----------------------------- Logstash output --------------------------------

# Send logs to logstash The host 5044 port , Corresponding one logstash The host needs to be configured with input Monitor on 5044 （ The configuration process , Reference resources Logstash file ） output.logstash:   hosts: ["10.26.10.15:5044"]

Parameter interpretation ：

## The default value is log , Represents a log reading source type : log

## Whether the configuration is effective , If set to false The log of this configuration will not be collected enabled: true

## Log path to crawl , Write absolute path paths: /to/file.log

## fields Represents a custom field , Write the fields you want to add in the indented two spaces below . Such as ： alilogtype: usercenter_serverlog  Indicates that this field is added to each log output ,key：alilogtype , value：usercenter_serverlog Used to identify the category of the log source , Before transferring to the next layer logstash It can be classified and processed according to this field .   fields:     alilogtype: usercenter_serverlog ##    It means the same thing , Add a custom field ,key：serverip ,value: ${serverip} This value is the read system environment variable , If the environment variable is not defined in the system , So start filebeat I will make a mistake , Find this value .         serverip: ${serverip}

## Set system environment variables , create a file   /etc/profile.d/serverip.sh  Add content ： export serverip=`ifconfig eth0 | grep 'inet addr' | awk '{print $2}' | cut -d':' -f2` ## Here is the machine IP

## Multi row merge parameters , Regular expressions multiline.pattern: '^\[' ## true or false; The default is false, matching pattern Rows of are merged to the previous row ;true, Mismatch pattern Rows of are merged to the previous row multiline.negate: true ## after or before, Merge to the end or beginning of the previous line multiline.match: after

##  ['ERROR','WARN'] This attribute can be configured to collect only error Level and warn Level of logging , If multi row collection is configured , Be sure to put this configuration after multiple lines include_lines: ## ['DEBUG'] This attribute configuration does not collect DEBUG Level of logging , If multiple rows are configured This configuration should also be placed after multiple lines exclude_lines:

## Logstash host hosts:

## If set to TRUE And configured multiple logstash host , The output plug-in publishes load balancing events to all logstash host . If set to false, The output plug-in sends all events to a random host , If you choose unreachable, you will switch to another host . The default is false. loadbalance

## Each configured host publishes events to Logstash Number of workers . This is best used to enable load balancing mode . Example ： If you have 2 Two hosts and 3 Staff , Then there is 6 Staff started （ Each host 3 individual ）. worker