Easy introduction to naturallanguageprocessing series topic 6 code practice -- spelling correction based on language model

Preface

This article mainly introduces the code practice , That is, spelling correction based on language model , Include task description 、 Implementation and application of syntax error correction .

One 、 Spelling correction task Overview

In the realization of QA System or retrieval system , User input is required , When the user enters a question , You can't expect him to input correctly , It is possible that the spelling of the words you typed is wrong . In a complete system , The background should be able to catch spelling errors in time , And correct , And then match the corrected result with the problem in the library . Here is a simple spelling error correction module , Automatically fix the wrong word .

The error correction module is based on Noisy Channel Model Noise channel model ：

c ∗ = argmax ⁡ c