Feature preprocessing

2.4.1 What is feature preprocessing

Why normalize and standardize

Dimensionless

2.4.2 Normalization: For normalization, if there are abnormal points that affect the maximum and minimum values, the results will obviously change

Map the data to ([0,1] by default) by transforming the original data

Outliers: Maximum Minimum

2.4.3 Standardization: For standardization, if there are abnormal points, due to a certain amount of data, a small number of abnormal points have little effect on the average, so the variance changes are small

(x - mean)/std

Standard Deviation: Degree of Concentration

def minmax_demo():"""Normalized:return:"""# 1 Get datadata = pd.read_csv("lizi")data = data.iloc[:, :3]print("data:\n", data)# 2 Instantiate a converter class# transfer = MinMaxScaler()transfer = MinMaxScaler(feature_range=[2,3])# 3 Call fit_transformdata_new = transfer.fit_transform(data)print("data_new:\n", data_new)return Nonedef stand_demo():"""standardization:return:"""# 1 Get datadata = pd.read_csv("lizi")data = data.iloc[:, :3]print("data:\n", data)# 2 Instantiate a converter classtransfer = StandardScaler# 3 Use fit_transferdata_new = transfer.fit_transform(data)print("data_new:\n", data_new)return None

Because factor_returns.csv was not found, I don't know if it can be run or not

def variance_demo():"""Filter low variance features:return:"""# 1. Get datadata = pd.read_csv("factor_returns.csv")data = data.iloc[:, 1:-2]print("data:\n", data)# 2. Instantiate a convertertransfer = VarianceThreshold(threshold=10)# 3. Call fit_transformdata_new = transfer.fit_transform(data)print("data_new:\n", data_new, data_new.shape)return None

2.5.1 Dimensionality Reduction - Dimensionality Reduction

ndarray

Dimensions: the number of levels of nesting

2D array

Dimension reduction here: reduce the number of features

Effect: No correlation between features

2.5.1 Dimensionality reduction

Feature selection

Filter filter

Variance selection method: low variance feature filtering

Correlation coefficient - the degree of correlation between features

Value range: -1 ~1

The correlation between features is high:

1) Choose one of them

2) Weighted Summation

3) Principal Component Analysis

Embeded

def variance_demo():"""Filter low variance features:return:"""# 1. Get datadata = pd.read_csv("factor_returns.csv")data = data.iloc[:, 1:-2]print("data:\n", data)# 2. Instantiate a convertertransfer = VarianceThreshold(threshold=10)# 3. Call fit_transformdata_new = transfer.fit_transform(data)print("data_new:\n", data_new, data_new.shape)# Calculate the correlation coefficient between two variablesr1 = pearsonr(data["pe_ratio"], data["pb_ratio"])print("Correlation coefficient:\n", r1)r2 = pearsonr(data['revenue'], data['total_expense'])print("Correlation between revenue and total_expense:\n", r2)return None

Decision Tree Regularization Deep Learning

Principal component analysis:

2.6.1 What is Principal Component Analysis (PCA)

sklearn.decomposition.PCA(n_compinents=None)

n_components

def pca_demo():"""PCA:return:"""data = [[2,8,4,5], [6,3,0,8], [5,4,9,1]]# 1 Instantiate a converter classtransfer = PCA(n_components=2)# call fit_transform(data)data_new = transfer.fit_transform(data)print("data_new:\n", data_new)return None

Decimal means how much information is retained

Integer reduces to how many features

2.6.2 Case study of user preferences for item categories