本文共 3565 字，大约阅读时间需要 11 分钟。
S3日志拉取
这里是S3插件的官方文档使用前，我们需要安装好S3插件我们因为从AWS的S3里拉取日志，需要在AWS上做一些配置：1.我们需要创建一个SQS的队列，然后授予每个桶SQS的权限。2.还需要给每个桶添加事件通知，这样S3桶一有更新，就会把更新扔到SQS里面。@type s3 aws_key_id XXXXXXXXXXXXXXXXXXXX aws_sec_key XXXXXXXXXXXXXXXXXXXX s3_bucket nx-rc-rancher-newelb-inner s3_region cn-northwest-1 tagelb-inner queue_name fluentd-s3 #这里只需要配置SQS的名字 @type s3 aws_key_id XXXXXXXXXXXXXXXXXXXX aws_sec_key XXXXXXXXXXXXXXXXXXXX s3_bucket nx-rc-rancher-newelb-com s3_region cn-northwest-1 queue_name fluentd-s3 @type s3 aws_key_id XXXXXXXXXXXXXXXXXXXX aws_sec_key XXXXXXXXXXXXXXXXXXXX s3_bucket nx-rc-www-newelb-com s3_region cn-northwest-1 queue_name fluentd-s3
因为fluentd是通过流的形式进行信息的过滤和处理，而且没有if语句，只能通过重写tag来筛选不同的日志
官方文档@type rewrite_tag_filter key message pattern /.yufuid.net:80/ tag elb-inner.net #匹配的就重写标签 @type parser #对日志就行分割和命名处理，fluentd的正则表达式可能需要自己写 key_name message reserve_data yes @type regexp expression /(? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+):(? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? "[^\"]+") (? "[^"]+") (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+)/ @type parser key_name message reserve_data yes @type regexp expression /(? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+):(? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? "[^\"]+") (? "[^"]+") (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+)/ @type parser key_name message reserve_data yes @type regexp expression /(? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+):(? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? "[^\"]+") (? "[^"]+") (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+) (? [^ ]+)/
@type elasticsearch host elasticsearchlog-lb.elasticsearch-log index_name fluentd-www-elb-%Y%m%d type_name fluentd-www-elb include_timestamp true #这里需配置时间字段，便于kibana的时间跟踪 ssl_verify false timekey 4s timekey_wait 1s #这里的刷新是把buffer的数据及时发送到ES，保持数据的实时性 @type elasticsearch host elasticsearchlog-lb.elasticsearch-log index_name fluentd-elb-%Y%m%d type_name fluentd-elb include_timestamp true ssl_verify false timekey 4s timekey_wait 1s @type elasticsearch host elasticsearchlog-lb.elasticsearch-log index_name fluentd-elb-%Y%m%d type_name fluentd-elb include_timestamp true ssl_verify false timekey 4s timekey_wait 1s
fluentd日志处理-安装配置（一）
Fluentd日志处理-tail拉取（三）Fluentd日志处理-插件使用和调试问题（四）转载于:https://blog.51cto.com/11078047/2316910