2022 Tsinghua summer school notes L1_ NLP and bigmodel Foundation

2022 Tsinghua University large model cross Seminar

2022-6-27

L1

1 NLP part

• NLP Tasks
  • Part of speech tagging
  • Identification of named entities （ The phenomenon of omission ）
  • Community reduction （ pronouns ）
  • Note the dependencies between components
  • Automatic Chinese word segmentation

• Application ：

  • Use in search engines NLP
    • matching query and document The similarity , Given query after , You can give some advertisements
    • Judge the quality of documents
  • NLP combination knowledge graph
    • Fully extract 、 Using knowledge
  • Human assistant
  • translate （ Remove the language barrier ）
  • Use language as a perspective , To analyze society

• Word representation ：

  • Translate into the meaning of words that machines can understand
  • similarity and relation
    • disadvantages
      • Manual marking , Some new meanings are missing
      • Lack of subtle differences
      • Subjectivity
      • Data absorption
      • It requires a lot of labor

• one-hot

  • independent , Find a dimension corresponding to the word , The remaining dimensions are 0
  • By default, words are orthogonal , The similarity between any two words is 0
  • improvement ： The meaning of a word is related to its context
  • A word is represented by a common word in its context
  • disadvantages ：
    • Increase the space of words
    • For words that appear less frequently , The context is sparse , The result is not good

• word embedding

  • Establish a low dimensional vector space
  • Word2Vec

• Language Model

  • Modeling language , Predict according to the above
    • joint probability ： The probability of having a sequence to form a sentence
    • Conditional probability ： Predict the next word according to the existing sentence
  • How to complete ？
    • hypothesis ： A future word is only affected by the previous word
    • It can be divided , Get the relationship between joint probability and conditional probability

• N-gram Model

  • E.g, 4-gram(n-gram Only with the front of the word n-1 One word is about )
    $$P\left(w_j\mid \text{ never to late to }\right)=\frac{\mathrm{count}\left(\text{ too late to }{w}_j\right)}{\mathrm{count}(\text{ too late to })}$$
    （never Calculation is not included in this formula ）
  • disadvantages ：
    • The distance is short
    • The similarity between words is still not considered

• Neural Language Model

  • Distributed representation to construct
  • Put the front first 3 Each word is expressed as a low dimensional vector , Then put the low dimensional vectors together , Form a higher vector , Then you can use this vector to predict the next word .
  • All predictions are made through the representation of context .
  • Learn parameter settings through large models

2 Bigmodel

2.1 brief introduction

The mechanism and details of the pre training language model .

• Why PLM?
  
  • The effect of language understanding and language generation is very good
    • Increasing parameters
    • Add calculation
    • Increase calculation power
  • Example ：GPT-3
    • rich knowledge
    • zero/few-shot（ No label , Few samples ）
• Paradigms
  • Learn from unmarked data , Do pre training through some self supervised tasks , Get rich general knowledge from it . In specific application , Then introduce task related knowledge , To adjust the model .
  • word embddings
  • contextual word embddings
  • ELMo,ULMFiT
  • Transformer
• Typical Case
  • GPT
  • Bert

2.2 Demo

• Big model demo
  • GPT-3（ Q & a model ）
  • Code big model
  • DALL-E 2 Image generation
  • Search engine