Pytorch版本的BERT

一、Google BERT：

BERT地址：https://github.com/google-research/bert

pytorch版本的BERT：https://github.com/huggingface/pytorch-pretrained-BERT

使用要求：Python 3.5+  &  PyTorch0.4.1/1.0.0  &  pip install pytorch-pretrained-bert & 下载BERT-模型

二、BERT-模型

BERT-Base, Multilingual (Not recommended, use Multilingual Cased instead): 102 languages, 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Base, Chinese: Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parameters 中文模型【BERT-Base-Chinese】：https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese.tar.gz

三、简单介绍

1）Bidirectional Encoder Representations from Transformers

the first unsupervised, deeply bidirectional system for pre-training NLP，且上下文相关。 train a large model (12-layer to 24-layer Transformer) on a large corpus (Wikipedia + BookCorpus) for a long time (1M update steps) （曾经的 Semi-supervised Sequence Learning, Generative Pre-Training, ELMo, and ULMFit 只学到了单侧信息。）

2.）学习词的表示：BERT mask 了15%的word，如：

Input: the man went to the [MASK1] . he bought a [MASK2] of milk. Labels: [MASK1] = store; [MASK2] = gallon 3. ）学习句子间信息：

Sentence A: the man went to the store . Sentence B: he bought a gallon of milk . Label: IsNextSentence 四、BERT的使用

4.1、两种使用方式

1）Pre-training ：four days on 4 to 16 Cloud TPUs

BERT-Base, Chinese: Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parameters 2）fine-tuning ：a few hours on a GPU 

The fine-tuning examples which use BERT-Base should be able to run on a GPU that has at least 12GB of RAM using the hyperparameters given. 4.2、Out-of-memory 如何解决

when using a GPU with 12GB - 16GB of RAM, you are likely to encounter out-of-memory issues if you use the same hyperparameters described in the paper，调整以下参数：

max_seq_length: 训好的模型用512，可以调小 train_batch_size:  Model type, BERT-Base vs. BERT-Large: The BERT-Large model requires more memory. Optimizer: 训好的模型用Adam, requires a lot of extra memory for the m and v vectors. Switching to a more memory efficient optimizer can reduce memory usage, but can also affect the results.  4.3、Pytorch-BERT的使用

原始BERT是运行在TF上的，TF-BERT使用可参考：https://www.jianshu.com/p/bfd0148b292e

Pytorch版本BERT组成如下：

1）Eight Bert PyTorch models

BertModel - raw BERT Transformer model (fully pre-trained), BertForMaskedLM - BERT Transformer with the pre-trained masked language modeling head on top (fully pre-trained), BertForNextSentencePrediction - BERT Transformer with the pre-trained next sentence prediction classifier on top (fully pre-trained), BertForPreTraining - BERT Transformer with masked language modeling head and next sentence prediction classifier on top (fully pre-trained), BertForSequenceClassification - BERT Transformer with a sequence classification head on top (BERT Transformer is pre-trained, the sequence classification head is only initialized and has to be trained), BertForMultipleChoice - BERT Transformer with a multiple choice head on top (used for task like Swag) (BERT Transformer is pre-trained, the multiple choice classification head is only initialized and has to be trained), BertForTokenClassification - BERT Transformer with a token classification head on top (BERT Transformer is pre-trained, the token classification head is only initialized and has to be trained), BertForQuestionAnswering - BERT Transformer with a token classification head on top (BERT Transformer is pre-trained, the token classification head is only initialized and has to be trained). 2）Tokenizers for BERT (using word-piece) (in the tokenization.py file):

BasicTokenizer - basic tokenization (punctuation splitting, lower casing, etc.), WordpieceTokenizer - WordPiece tokenization, BertTokenizer - perform end-to-end tokenization, i.e. basic tokenization followed by WordPiece tokenization. 3）Optimizer for BERT (in the optimization.py file):

BertAdam - Bert version of Adam algorithm with weight decay fix, warmup and linear decay of the learning rate. 源码中： __init__(params, lr=required, warmup=-1, t_total=-1, schedule='warmup_linear', betas=(0.9, 0.999), e=1e-6, weight_decay=0.01, max_grad_norm=1.0, **kwargs) lr: learning rate warmup: portion of t_total for the warmup, -1  means no warmup. 【使用部分t_total热身】 t_total: total number of training steps for the learning rate schedule, -1  means constant learning rate of 1. (no warmup regardless of warmup setting). Default: -1【总训练步骤】 schedule: schedule to use for the warmup (see above). Can be `'warmup_linear'`, `'warmup_constant'`, `'warmup_cosine'`, `'none'`, `None` or a `_LRSchedule` object (see below). If `None` or `'none'`, learning rate is always kept constant. Default : `'warmup_linear'` eg：使用方式：train_optimi_step = int(train_iter_num / args.batch_size) * args.epochs optimizer = BertAdam([param for _, param in param_optimizer], lr=args.lr, warmup=0.1, t_total=train_optimi_step) 4）Five examples on how to use BERT (in the examples folder):

extract_features.py - Show how to extract hidden states from an instance of BertModel, run_classifier.py - Show how to fine-tune an instance of BertForSequenceClassification on GLUE's MRPC task, run_squad.py - Show how to fine-tune an instance of BertForQuestionAnswering on SQuAD v1.0 and SQuAD v2.0 tasks. run_swag.py - Show how to fine-tune an instance of BertForMultipleChoice on Swag task. run_lm_finetuning.py - Show how to fine-tune an instance of BertForPretraining on a target text corpus. 五、BERT的使用代码

使用Pytorch版本BERT使用方式如下：

1）First prepare a tokenized input with BertTokenizer

import torch from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM # 加载词典 pre-trained model tokenizer (vocabulary) tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') # Tokenized input text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]" tokenized_text = tokenizer.tokenize(text) # Mask a token that we will try to predict back with `BertForMaskedLM` masked_index = 8 tokenized_text[masked_index] = '[MASK]' assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]'] # 将 token 转为 vocabulary 索引 indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text) # 定义句子 A、B 索引 segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1] # 将 inputs 转为 PyTorch tensors tokens_tensor = torch.tensor([indexed_tokens]) segments_tensors = torch.tensor([segments_ids]) 2）use BertModel to get hidden states

# 加载模型 pre-trained model (weights) model = BertModel.from_pretrained('bert-base-uncased') model.eval() # GPU & put everything on cuda tokens_tensor = tokens_tensor.to('cuda') segments_tensors = segments_tensors.to('cuda') model.to('cuda') # 得到每一层的 hidden states  with torch.no_grad():     encoded_layers, _ = model(tokens_tensor, segments_tensors) # 模型 bert-base-uncased 有12层，所以 hidden states 也有12层 assert len(encoded_layers) == 12 3）use BertForMaskedLM

# 加载模型 pre-trained model (weights) model = BertForMaskedLM.from_pretrained('bert-base-uncased') model.eval() # cuda tokens_tensor = tokens_tensor.to('cuda') segments_tensors = segments_tensors.to('cuda') model.to('cuda') # Predict all tokens with torch.no_grad():     predictions = model(tokens_tensor, segments_tensors) # confirm we were able to predict 'henson' predicted_index = torch.argmax(predictions[0, masked_index]).item() predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0] assert predicted_token == 'henson' 1. BertModel 

输入: in modeling.py

input_ids: torch.LongTensor  [batch_size, sequence_length] with the word token indices in the vocabulary. token_type_ids: optional torch.LongTensor [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a sentence A and type 1 corresponds to a sentence B token. attention_mask: torch.LongTensor [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used when a batch has varying length sentences. output_all_encoded_layers: controls the content of the encoded_layers output as described below. Default: True. 输出:

encoded_layers: 取决于output_encoded_layers 参数: output_all_encoded_layers=True: 输出一列 encoded-hidden-states at the end of each attention block (12 full sequences for BERT-base, 24 for BERT-large), 每个 encoded-hidden-state= [batch_size, sequence_length, hidden_size] output_all_encoded_layers=False: 输出最后一个attention block对应的encoded-hidden-states，1个 [batch_size, sequence_length, hidden_size] pooled_output: torch.FloatTensor [batch_size, hidden_size] which is the output of a classifier pretrained on top of the hidden state associated [CLF] to train on the Next-Sentence task.