[log service CLS] Tencent cloud log4j/logback log collection best practices

One introduction

Log storage analysis plays an important role in application systems , Conventional ELK Too cumbersome for small teams , Maintenance trouble , Tencent cloud provides CLS Log collection and analysis system , Can pass LogListener To collect logs in a non intrusive way , Developers can also use API To collect logs （ There seems to be no... Available at present sdk To collect developer application logs , Or the author missed a part of the document ）, Official website documents for API There are relatively few best practice documents for collecting logs , The author combines Tencent cloud CLS and Java The two most common in the field log The tool implements code intrusion free business log reporting .

Two Log4j/Logback Knowledge preparation

log4j and logback yes Java The two most common open source in the development field log library , Built in many logs Appender class , such as Stdout,FileAppender, Developers can also define according to their specific needs Appender, stay log Configure custom in the configuration file Appender Class to implement custom log collection logic .

3、 ... and Architecture diagram

Log collection architecture diagram

Pictured above , Developers can choose to submit logs to Tencent cloud through the customized log collection module CLS.

Four Tencent cloud CLS Logical concept preparation

• Log set ： A log set corresponds to a project or application
• Journal topics ： A log topic corresponds to a type of application or service
• Log group ： A collection containing multiple logs
• Log partition ： A log topic can be divided into multiple topic partitions , But at least one partition （ It can improve the retrieval efficiency ）

5、 ... and preparation

1. Log in to Tencent cloud , obtain secretId,secretKey（ More important ）
2. Get into CLS Console
3. Create blog topics according to your personal needs （ Copy the theme ID spare ）

6、 ... and Best practices

The sample code address for this article ：Github Warehouse address （ For reference only ）

Technology selection

1. Java1.8+
2. Maven3.6+
3. log4j1.2.6
4. logback1.1.7

CLS Log escalation cloud API

1. The author of this article calls API To upload structured logs , Reference upload Structured log API file
2. cloud API Signature access reference Official website API Sign the document
3. cloud API Signature reference Official website signature demo
4. Upload structured logs API need google protobuf Convert the log contents to BP Format , Please check protobuf Introduce

establish maven project , The project structure is shown in the figure below

pom.xml Add the following dependencies

       <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.7</version>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-core</artifactId>
            <version>1.1.7</version>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-access</artifactId>
            <version>1.1.7</version>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.1.7</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.16</version>
        </dependency>
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.10</version>
        </dependency>
        <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>2.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.10</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>30.1.1-jre</version>
        </dependency>

Log4j Realize log reporting

Log4j Need to implement custom Appender Need to inherit org.apache.log4j.AppenderSkeleton,Log4j Log output is done through event driven , So in the core logic , We just need to get the information carried by the event , And call the cloud API, Upload the information carried in these events , The logic diagram is as follows

Scheme implementation steps

1 stay maven project resources Create under directory log4j.properties file （log4j The file name loaded by the log framework by default ）, The specific configuration is as follows

log4j.rootLogger=DEBUG,clsLogAppender, STDOUT
log4j.appender.clsLogAppender=com.github.cls.log4j.ClsLog4jAppender
log4j.appender.clsLogAppender.project=smartdoc
log4j.appender.clsLogAppender.region=ap-nanjing
log4j.appender.clsLogAppender.endpoint=ap-nanjing.cls.tencentcs.com
log4j.appender.clsLogAppender.secretId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
log4j.appender.clsLogAppender.secretKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
log4j.appender.clsLogAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.clsLogAppender.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
log4j.appender.clsLogAppender.topicId=b9ed1869-156a-4eb4-ae96-b0fa292cf5b5
log4j.appender.clsLogAppender.source=smart-doc
log4j.appender.clsLogAppender.timeFormat=yyyy-MM-dd'T'HH:mm:ssZ
log4j.appender.clsLogAppender.timeZone=UTC
log4j.appender.clsLogAppender.Threshold=INFO
#STDOUT
log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender
log4j.appender.STDOUT.layout=org.apache.log4j.PatternLayout
log4j.appender.STDOUT.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

In the above code ,clsLogAppender It belongs to what developers need to define appender The name of ,region,endpoint,secretId,secretKey,topicId These information fields need to be obtained from the console , And these fields need to be defined in the custom Appender in , Convenient from log4j.properties Get the custom configuration item from the file .

2 install proto; Installation steps refer to

https://cloud.tencent.com/document/product/614/16873#pb-.E7.BC.96.E8.AF.91.E7.A4.BA.E4.BE.8B

3 Define the log BP Structure

package com.github.cls;

message Log
{
    message Content
    {
        required string key   = 1; //  For each group of fields  key
        required string value = 2; //  For each group of fields  value
    }
    required int64   time     = 1; //  Time stamp ,UNIX Time format 
    repeated Content contents = 2; //  Multiple in one log kv Combine 
}

message LogTag
{
    required string key       = 1;
    required string value     = 2;
}

message LogGroup
{
    repeated Log    logs        = 1; //  Log array composed of multiple logs 
    optional string contextFlow = 2; //  It's not available at present 
    optional string filename    = 3; //  Log file name 
    optional string source      = 4; //  Source of a log , Generally, machines are used IP
    repeated LogTag logTags     = 5;
}

message LogGroupList
{
    repeated LogGroup logGroupList = 1; //  Log group list 
}

4 call protoc --java_out=./ ./cls.proto Generate BP Structure

5 The generated Java File copy to maven In Engineering （ In this example , My Java The structure name is Cls）

6 Custom log upload core logic （ Inherit org.apache.log4j.AppenderSkeleton class ）

7 The following abstract methods are implemented

@Override
protected void append(LoggingEvent event) {
        
        // 1  encapsulation BP Structure 
        Cls.LogGroupList logGroupList = Cls.LogGroupList.newBuilder().addLogGroupList(
                Cls.LogGroup.newBuilder().addLogs(
                        Cls.Log.newBuilder()
                                .setTime(event.timeStamp)
                                .addContents(Cls.Log.Content.newBuilder().setKey("exceptionTrace").setValue(String.valueOf(getThrowableStr(event))))
                                .addContents(Cls.Log.Content.newBuilder().setKey("level").setValue(String.valueOf(event.getLevel())))
                                .addContents(Cls.Log.Content.newBuilder().setKey("threadName").setValue(event.getThreadName()))
                                .addContents(Cls.Log.Content.newBuilder().setKey("time").setValue(String.valueOf(event.timeStamp)))
                                .addContents(Cls.Log.Content.newBuilder().setKey("message").setValue(String.valueOf(event.getMessage())))
                )
        ).build();
        
        //  Upload BP Structure 
        PostStructLogRequest req = client.newPostStructLogRequest(topicId, logGroupList);
        String result = client.send(req);
        
     }

In the above code client and PostStructLogRequest It is simply implemented by the author , Here is just the general logic , The details of implementation will not be repeated , Please refer to github Code example

8 Write a log test class

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;

public class Log4jAppenderExample {
    private static final Logger LOGGER = LogManager.getLogger(Log4jAppenderExample.class);
    public static void main(String[] args) {
        LOGGER.trace("cls log4j trace log");
        LOGGER.debug("cls log4j debug log");
        LOGGER.info("cls log4j info log");
        LOGGER.warn("cls log4j warn log");
        LOGGER.error("cls log4j error log", new RuntimeException("Runtime Exception"));
    }
}

9 Run test class

10 land CLS The console switches to the log retrieval menu , Here's the picture

11 If CLS The console displays the logs you printed in the application , Be accomplished

logback Report logs to the cloud

loback The implementation steps are the same as above , The difference is that developers are required to resources Definition under directory logback.xml In file , And configure Appender Custom properties for , as follows demo To configure

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{5} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="TencentCLS" class="com.github.cls.logback.ClsLogbackAppender">
        <project>smartdoc</project>
        <endpoint>ap-nanjing.cls.tencentcs.com</endpoint>
        <region>ap-nanjing</region>
        <secretId>AKIDQa1B6dIsc8OSntCuVpujKgHi8HAomOIE</secretId>
        <secretKey>vWqtiGadeXCbjcg39yZtwj2EJ7ewZv03</secretKey>
        <topicId>b9ed1869-156a-4eb4-ae96-b0fa292cf5b5</topicId>
        <source>smart-doc</source>
        <timeZone>UTC</timeZone>
        <timeFormat>yyyy-MM-dd'T'HH:mmZ</timeFormat>

        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{5} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="TencentCLS"/>

    </root>
</configuration>

remarks ：

1. com.github.cls.logback.ClsLogbackAppender It is self defined by the author logback Appender
2. Need to be in appender Configure properties in tags
3. logback The log reporting test classes are as follows

package com.github.cls.example;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LogbackAppenderExample {
    private static final Logger LOGGER = LoggerFactory.getLogger(LogbackAppenderExample.class);

    public static void main(String[] args) {
        LOGGER.trace("cls logback trace log");
        LOGGER.debug("cls logback debug log");
        LOGGER.info("cls logback info log");
        LOGGER.warn("cls logback warn log");
        LOGGER.error("cls logback error log", new RuntimeException("Runtime Exception"));
    }
}

7、 ... and summary

The above log reporting implementation scheme is relative to LogListener Speaking of , There is a performance gap , Because the above log collection method adopts the synchronous method for log reporting ,LogListener Using a listener , Make the production log party and the consumption log party decouple ; but LogListener Of is limited to CVM Log collection on the machine , Or other Tencent cloud container log collection , If the developer's application is in the self built computer room or other cloud platforms , But log management is in Tencent cloud CLS On , The above implementation scheme can solve this problem , And developers can customize the logic .