Data-driven anomaly detection and early warning of item C in the May 1st mathematical modeling competition in 2021

2021 May 1st Mathematical Modeling Contest C topic Data driven anomaly detection and early warning

C topic Data driven anomaly detection and early warning

Original high-quality ideas Identify us The links below

High quality 、 original 、9.9 that will do Official account number 5 Discount sale  

link ：

https://mianbaoduo.com/o/bread/YZmVlJdw

C topic Data driven anomaly detection and early warning

Data description ：100 A sensor 24 Hour time series data （ No data name given ）

Data description diagram

 

 

 

Data processing ：100 A sensor , Do not know the data name , It is necessary to filter variables . The first calculation is 100 The Euclidean distance between columns is clustered and then filtered . This is the simplest , An effective plan .

to update 1： Data processing 100 Column data to do matlab Of cluster Dimension reduction ,matlab Of cluster The function seems to have failed （ Maybe the dimension is high ） I don't know why , Still use here R Language helps us do well , Directly send the data set with good dimensionality reduction .
to update 2：pdist Function can solve , No correlation coefficient . Be careful , The data dimensionality reduction here should not be standardized 、 normalization , Not much to explain . The results are placed below , Just select a few of each class as the new data set , The code and data results will be sent soon .

We use the code Get the... After data processing 10 Column （ Relatively independent data ） after , Take a brief look at their trends and characteristics ：

 

It can be seen that , The processed data has its own characteristics , It shows that our processing method is basically reliable （ For reference only ）.

According to the above time series （ chart ） It can be seen that , It basically conforms to the properties given in the title ： Such as the independent point of the picture in the lower right corner , The periodicity of the Chinese pictures on the .

 

Question 1 ： Establish risk abnormal data detection model （ Give an assessment of risk and non risk ）

 

According to the picture above , The above four properties of the time series of detection data are analyzed one by one

The following reference scheme is given ：

1. Non risk outliers detection

 

Be careful ！ There are some conceptual distinctions , The abnormal data of this problem has the same name as the abnormal value detection in machine learning , When you search for information, you should pay attention to distinguish , The abnormal data in this article refers to the logical abnormality of the sensor .

programme 1 ） The abnormal value is judged as non risk by a separate column （ It's not very realistic , According to the meaning of the topic , Just for learning ）

The title mentioned that there will be independent points in the time series , Is one of the characteristics of non risk . Therefore, the first step in the abnormal data monitoring model is to remove the abnormal points of individual columns , Use simple outlier detection .（ Single column outlier elimination ）

programme 2*） The global outliers judge that the outliers are non risk

Fluctuations in the sensor - Outliers , There may or may not be risks , Fluctuations caused by changes in the external environment , We think it is risk-free , If at some point ...........................