Design experience of log file IO system

Record the log file once IO System design experience

bytebuffer How to use it .

bytebuffer The essence of is caching , We can think of it as the concept of comparative physical layer , For example, if we don't plan the mapping access , Or the file index is not well designed , It is easy to create many file handles , Eventually, it is easy to cause the operating system inode Light consumption .

So I want to learn bytebuffer The key is to master Position and capacity Use skills .

secondly , Is to do a good job in file indexing and mapping range control , The common method is to segment the file segment, Then index the segment .

Of course, you can control the business scope, file recycling strategy and later use shell Write a compact The script of logic （ Compress large files into small files ） As compensation .

Use of locks

The place where we need to lock the whole system manually is the opening and closing of files .

If the file is opened and created without locking , It is easy to create many files , Even destroy our business logic , trigger BUG. This is ERROR Grade .

The second is to close the file and lock it , If the closed file is not locked , When recycling files or unmapping, it is easy to hold them because of concurrency " It's been shut down " Handle to the file , So as to throw the exception of missing handle . This is just WARN Grade .

For the part of business code , Locking is not recommended because it will reduce performance . Considering the document IO Less resources , therefore Multithreading + Synchronous write lock scheme , It's better to use one-way programming , Using thread pool to queue asynchronously . One is simple code , The difficulty of monitoring and maintenance is also low , The performance can be said to be quite . Also reduced CPU The cost of context switching .

The use of thread pools

Most of our projects use SpringBoot, So we decided to give the thread pool to Spring Bean trusteeship .

The second is to improve the isolation of business , At the same time, the performance problem is also considered , Because we have divided multiple files for writing according to business , therefore , We decided to give each document 【 Try to 】 Allocate a thread .

We applied for a set of thread pools , Each thread pool has a thread and a bounded synchronous blocking queue .

Then we do according to the file name hash, according to Hash To allocate thread pools .

In this way, basically every file has an independent working thread .

Later we can also consider using actuator Monitor the performance of thread pool .