Spark log analysis

1. summary

When the browser requests the server , If the access log is set on the server , The user's access records will be recorded . In the log , It usually contains a lot of information , But this information is not easy to use , Here we're going to talk about Apache Of access.log Log analysis , To further study Spark Program development under .

2. Suppose the demand

Suppose to provide us with a apache Of access.log file , According to business needs , We need to analyze and get the following requirements ：
1. Statistics of daily page visits
2. Count each different HTTP The number of accesses corresponding to the status
3. Statistics are different and independent IP Of visits
4. Count the number of visits to different pages

3. preparation

3.1 Log file download

Download the log for the specified analysis , Of course, you can also use your own real Apache journal , stay tomcat Of logs Directory , In order to make the analysis results more intuitive and obvious , It is recommended to use the download log .
Apache access.log Log download address ：http://labfile.oss.aliyuncs.com/courses/456/access.log

3.2 Log file format

After the downloaded log file is opened, it looks like this ：