摘要:文章将详细讲解Keras实现经典的深度学习文本分类算法,包括LSTM、BiLSTM、BiLSTM+Attention和CNN、TextCNN。
本文分享自华为云社区《Keras深度学习中文文本分类万字总结(CNN、TextCNN、BiLSTM、注意力)》,作者: eastmount。
一.文本分类概述
文本分类旨在对文本集按照一定的分类体系或标准进行自动分类标记,属于一种基于分类体系的自动分类。文本分类最早可以追溯到上世纪50年代,那时主要通过专家定义规则来进行文本分类;80年代出现了利用知识工程建立的专家系统;90年代开始借助于机器学习方法,通过人工特征工程和浅层分类模型来进行文本分类。现在多采用词向量以及深度神经网络来进行文本分类。
牛亚峰老师将传统的文本分类流程归纳如下图所示。在传统的文本分类中,基本上大部分机器学习方法都在文本分类领域有所应用。主要包括:
Naive BayesKNNSVM集合类方法最大熵神经网络
利用Keras框架进行文本分类的基本流程如下:
步骤 1:文本的预处理,分词->去除停用词->统计选择top n的词做为特征词步骤 2:为每个特征词生成ID步骤 3:将文本转化成ID序列,并将左侧补齐步骤 4:训练集shuffle步骤 5:Embedding Layer 将词转化为词向量步骤 6:添加模型,构建神经网络结构步骤 7:训练模型步骤 8:得到准确率、召回率、F1值
注意,如果使用TFIDF而非词向量进行文档表示,则直接分词去停后生成TFIDF矩阵后输入模型。
深度学习文本分类方法包括:
卷积神经网络(TextCNN)循环神经网络(TextRNN)TextRNN+AttentionTextRCNN(TextRNN+CNN)BiLSTM+Attention迁移学习
推荐牛亚峰老师的文章:
基于 word2vec 和 CNN 的文本分类 :综述 & 实践
二.数据预处理及分词
这篇文章主要以代码为主,算法原理知识前面的文章和后续文章再继续补充。数据集如下图所示:
训练集:news_dataset_train.csv
游戏主题(10000)、体育主题(10000)、文化主题(10000)、财经主题(10000)测试集:news_dataset_test.csv
游戏主题(5000)、体育主题(5000)、文化主题(5000)、财经主题(5000)验证集:news_dataset_val.csv
游戏主题(5000)、体育主题(5000)、文化主题(5000)、财经主题(5000)
首先需要进行中文分词预处理,调用Jieba库实现。代码如下:
data_preprocess.py
# -*- coding:utf-8 -*-# By:Eastmount CSDN -03-19import csvimport pandas as pdimport numpy as npimport jiebaimport jieba.analyse#添加自定义词典和停用词典jieba.load_userdict("user_dict.txt")stop_list = pd.read_csv('stop_words.txt',engine='python',encoding='utf-8',delimiter="\n",names=['t'])['t'].tolist()#-----------------------------------------------------------------------#Jieba分词函数def txt_cut(juzi):return [w for w in jieba.lcut(juzi) if w not in stop_list]#-----------------------------------------------------------------------#中文分词读取文件def fenci(filename,result):#写入分词结果fw = open(result, "w", newline = '',encoding = 'gb18030')writer = csv.writer(fw) writer.writerow(['label','cutword'])#使用csv.DictReader读取文件中的信息labels = []contents = []with open(filename, "r", encoding="UTF-8") as f:reader = csv.DictReader(f)for row in reader:#数据元素获取labels.append(row['label'])content = row['content']#中文分词seglist = txt_cut(content)#空格拼接output = ' '.join(list(seglist))contents.append(output)#文件写入tlist = []tlist.append(row['label'])tlist.append(output)writer.writerow(tlist)print(labels[:5])print(contents[:5])fw.close()#-----------------------------------------------------------------------#主函数if __name__ == '__main__':fenci("news_dataset_train.csv", "news_dataset_train_fc.csv")fenci("news_dataset_test.csv", "news_dataset_test_fc.csv")fenci("news_dataset_val.csv", "news_dataset_val_fc.csv")
运行结果如下图所示:
接着我们尝试简单查看数据的长度分布情况及标签可视化。
data_show.py
# -*- coding: utf-8 -*-"""Created on -03-19@author: xiuzhang Eastmount CSDN"""import pandas as pdimport numpy as npfrom sklearn import metricsimport matplotlib.pyplot as pltimport seaborn as sns#---------------------------------------第一步 数据读取------------------------------------## 读取测数据集train_df = pd.read_csv("news_dataset_train_fc.csv")val_df = pd.read_csv("news_dataset_val_fc.csv")test_df = pd.read_csv("news_dataset_test_fc.csv")print(train_df.head())## 解决中文显示问题plt.rcParams['font.sans-serif'] = ['KaiTi'] #指定默认字体 SimHei黑体plt.rcParams['axes.unicode_minus'] = False #解决保存图像是负号'## 查看训练集都有哪些标签plt.figure()sns.countplot(train_df.label)plt.xlabel('Label',size = 10)plt.xticks(size = 10)plt.show()## 分析训练集中词组数量的分布print(train_df.cutwordnum.describe())plt.figure()plt.hist(train_df.cutwordnum,bins=100)plt.xlabel("词组长度", size = 12)plt.ylabel("频数", size = 12)plt.title("训练数据集")plt.show()
输出结果如下图所示,后面的文章我们会介绍论文如何绘制好看的图表。
注意,如果报错“UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xce in position 17: invalid continuation byte”,需要将CSV文件保存为UTF-8格式,如下图所示。
三.CNN中文文本分类
1.原理介绍
卷积神经网络(Convolutional Neural Networks, CNN)是一类包含卷积计算且具有深度结构的前馈神经网络(Feedforward Neural Networks),是深度学习(deep learning)的代表算法之一。它通常应用于图像识别和语音识等领域,并能给出更优秀的结果,也可以应用于视频分析、机器翻译、自然语言处理、药物发现等领域。著名的阿尔法狗让计算机看懂围棋就是基于卷积神经网络的。
卷积是指不在对每个像素做处理,而是对图片区域进行处理,这种做法加强了图片的连续性,看到的是一个图形而不是一个点,也加深了神经网络对图片的理解。通常卷积神经网络会依次经历“图片->卷积->持化->卷积->持化->结果传入两层全连接神经层->分类器”的过程,最终实现一个CNN的分类处理。
2.代码实现
Keras实现文本分类的CNN代码如下:
Keras_CNN_cnews.py
# -*- coding: utf-8 -*-
"""Created on -03-19@author: xiuzhang Eastmount CSDNCNN Model"""import osimport timeimport pickleimport pandas as pdimport numpy as npfrom sklearn import metricsimport matplotlib.pyplot as pltimport seaborn as snsimport tensorflow as tffrom sklearn.preprocessing import LabelEncoder,OneHotEncoderfrom keras.models import Modelfrom keras.layers import LSTM, Activation, Dense, Dropout, Input, Embeddingfrom keras.layers import Convolution1D, MaxPool1D, Flattenfrom keras.preprocessing.text import Tokenizerfrom keras.preprocessing import sequencefrom keras.callbacks import EarlyStoppingfrom keras.models import load_modelfrom keras.models import Sequential## GPU处理 读者如果是CPU注释该部分代码即可## 指定每个GPU进程中使用显存的上限 0.9表示可以使用GPU 90%的资源进行训练os.environ["CUDA_DEVICES_ORDER"] = "PCI_BUS_IS"os.environ["CUDA_VISIBLE_DEVICES"] = "0"gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8)sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))start = time.clock()#----------------------------第一步 数据读取----------------------------## 读取测数据集train_df = pd.read_csv("news_dataset_train_fc.csv")val_df = pd.read_csv("news_dataset_val_fc.csv")test_df = pd.read_csv("news_dataset_test_fc.csv")print(train_df.head())## 解决中文显示问题plt.rcParams['font.sans-serif'] = ['KaiTi'] #指定默认字体 SimHei黑体plt.rcParams['axes.unicode_minus'] = False #解决保存图像是负号'#--------------------------第二步 OneHotEncoder()编码--------------------## 对数据集的标签数据进行编码train_y = train_df.labelval_y = val_df.labeltest_y = test_df.labelprint("Label:")print(train_y[:10])le = LabelEncoder()train_y = le.fit_transform(train_y).reshape(-1,1)val_y = le.transform(val_y).reshape(-1,1)test_y = le.transform(test_y).reshape(-1,1)print("LabelEncoder")print(train_y[:10])print(len(train_y))## 对数据集的标签数据进行one-hot编码ohe = OneHotEncoder()train_y = ohe.fit_transform(train_y).toarray()val_y = ohe.transform(val_y).toarray()test_y = ohe.transform(test_y).toarray()print("OneHotEncoder:")print(train_y[:10])#-----------------------第三步 使用Tokenizer对词组进行编码--------------------max_words = 6000max_len = 600tok = Tokenizer(num_words=max_words) #最大词语数为6000print(train_df.cutword[:5])print(type(train_df.cutword))## 防止语料中存在数字str处理train_content = [str(a) for a in train_df.cutword.tolist()]val_content = [str(a) for a in val_df.cutword.tolist()]test_content = [str(a) for a in test_df.cutword.tolist()]tok.fit_on_texts(train_content)print(tok)#当创建Tokenizer对象后 使用fit_on_texts()函数识别每个词#tok.fit_on_texts(train_df.cutword)## 保存训练好的Tokenizer和导入with open('tok.pickle', 'wb') as handle: #savingpickle.dump(tok, handle, protocol=pickle.HIGHEST_PROTOCOL)with open('tok.pickle', 'rb') as handle: #loadingtok = pickle.load(handle)## 使用word_index属性查看每个词对应的编码## 使用word_counts属性查看每个词对应的频数for ii,iterm in enumerate(tok.word_index.items()):if ii < 10:print(iterm)else:breakprint("===================") for ii,iterm in enumerate(tok.word_counts.items()):if ii < 10:print(iterm)else:break#---------------------------第四步 数据转化为序列-----------------------------## 使用sequence.pad_sequences()将每个序列调整为相同的长度## 对每个词编码之后,每句新闻中的每个词就可以用对应的编码表示,即每条新闻可以转变成一个向量了train_seq = tok.texts_to_sequences(train_content)val_seq = tok.texts_to_sequences(val_content)test_seq = tok.texts_to_sequences(test_content)## 将每个序列调整为相同的长度train_seq_mat = sequence.pad_sequences(train_seq,maxlen=max_len)val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)test_seq_mat = sequence.pad_sequences(test_seq,maxlen=max_len)print("数据转换序列")print(train_seq_mat.shape)print(val_seq_mat.shape)print(test_seq_mat.shape)print(train_seq_mat[:2])#-------------------------------第五步 建立CNN模型--------------------------## 类别为4个num_labels = 4inputs = Input(name='inputs',shape=[max_len], dtype='float64')## 词嵌入使用预训练的词向量layer = Embedding(max_words+1, 128, input_length=max_len, trainable=False)(inputs)## 卷积层和池化层(词窗大小为3 128核)cnn = Convolution1D(128, 3, padding='same', strides = 1, activation='relu')(layer)cnn = MaxPool1D(pool_size=4)(cnn)## Dropout防止过拟合flat = Flatten()(cnn) drop = Dropout(0.3)(flat)## 全连接层main_output = Dense(num_labels, activation='softmax')(drop)model = Model(inputs=inputs, outputs=main_output)## 优化函数 评价指标model.summary()pile(loss="categorical_crossentropy",optimizer='adam',# RMSprop()metrics=["accuracy"])#-------------------------------第六步 模型训练和预测--------------------------## 先设置为train训练 再设置为test测试flag = "train"if flag == "train":print("模型训练")## 模型训练 当val-loss不再提升时停止训练 0.0001model_fit = model.fit(train_seq_mat, train_y, batch_size=128, epochs=10,validation_data=(val_seq_mat,val_y),callbacks=[EarlyStopping(monitor='val_loss',min_delta=0.0001)] )## 保存模型model.save('my_model.h5') del model # deletes the existing model## 计算时间elapsed = (time.clock() - start)print("Time used:", elapsed)print(model_fit.history)else:print("模型预测")## 导入已经训练好的模型model = load_model('my_model.h5')## 对测试集进行预测test_pre = model.predict(test_seq_mat)## 评价预测效果,计算混淆矩阵confm = metrics.confusion_matrix(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1))print(confm)## 混淆矩阵可视化Labname = ["体育", "文化", "财经", "游戏"]print(metrics.classification_report(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1)))plt.figure(figsize=(8,8))sns.heatmap(confm.T, square=True, annot=True,fmt='d', cbar=False, linewidths=.6,cmap="YlGnBu")plt.xlabel('True label',size = 14)plt.ylabel('Predicted label', size = 14)plt.xticks(np.arange(4)+0.5, Labname, size = 12)plt.yticks(np.arange(4)+0.5, Labname, size = 12)plt.savefig('result.png')plt.show()#----------------------------------第七 验证算法--------------------------## 使用tok对验证数据集重新预处理,并使用训练好的模型进行预测val_seq = tok.texts_to_sequences(val_df.cutword)## 将每个序列调整为相同的长度val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)## 对验证集进行预测val_pre = model.predict(val_seq_mat)print(metrics.classification_report(np.argmax(val_y,axis=1),np.argmax(val_pre,axis=1)))## 计算时间elapsed = (time.clock() - start)print("Time used:", elapsed)
GPU运行如下图所示。注意,如果您的电脑是CPU版本,只需要将上述代码第一部分注释掉即可,后面LSTM部分使用GPU对应的库函数。
训练输出模型如下图所示:
训练输出结果如下:
模型训练Train on 40000 samples, validate on 20000 samplesEpoch 1/1040000/40000 [==============================] - 15s 371us/step - loss: 1.1798 - acc: 0.4772 - val_loss: 0.9878 - val_acc: 0.5977Epoch 2/1040000/40000 [==============================] - 4s 93us/step - loss: 0.8681 - acc: 0.6612 - val_loss: 0.8167 - val_acc: 0.6746Epoch 3/1040000/40000 [==============================] - 4s 92us/step - loss: 0.7268 - acc: 0.7245 - val_loss: 0.7084 - val_acc: 0.7330Epoch 4/1040000/40000 [==============================] - 4s 93us/step - loss: 0.6369 - acc: 0.7643 - val_loss: 0.6462 - val_acc: 0.7617Epoch 5/1040000/40000 [==============================] - 4s 96us/step - loss: 0.5670 - acc: 0.7957 - val_loss: 0.5895 - val_acc: 0.7867Epoch 6/1040000/40000 [==============================] - 4s 92us/step - loss: 0.5074 - acc: 0.8226 - val_loss: 0.5530 - val_acc: 0.8018Epoch 7/1040000/40000 [==============================] - 4s 93us/step - loss: 0.4638 - acc: 0.8388 - val_loss: 0.5105 - val_acc: 0.8185Epoch 8/1040000/40000 [==============================] - 4s 93us/step - loss: 0.4241 - acc: 0.8545 - val_loss: 0.4836 - val_acc: 0.8304Epoch 9/1040000/40000 [==============================] - 4s 92us/step - loss: 0.3900 - acc: 0.8692 - val_loss: 0.4599 - val_acc: 0.8403Epoch 10/1040000/40000 [==============================] - 4s 93us/step - loss: 0.3657 - acc: 0.8761 - val_loss: 0.4472 - val_acc: 0.8457Time used: 52.203992899999996
预测及验证结果如下:
[[3928 472 264 336][ 115 4529 121 235][ 151 340 4279 230][ 145 593 195 4067]]precision recall f1-score support0 0.910.790.8450001 0.760.910.8350002 0.880.860.8750003 0.840.810.825000avg / total 0.850.840.8420000precision recall f1-score support0 0.900.770.8350001 0.780.920.8450002 0.880.850.8650003 0.840.850.855000avg / total 0.850.850.8520000
四.TextCNN中文文本分类
1.原理介绍
TextCNN 是利用卷积神经网络对文本进行分类的算法,由 Yoon Kim 于在 “Convolutional Neural Networks for Sentence Classification” 一文中提出的算法。
卷积神经网络的核心思想是捕捉局部特征,对于文本来说,局部特征就是由若干单词组成的滑动窗口,类似于N-gram。卷积神经网络的优势在于能够自动地对N-gram特征进行组合和筛选,获得不同抽象层次的语义信息。下图是该论文中用于文本分类的卷积神经网络模型架构。
另一篇TextCNN比较经典的论文是《A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification》,其模型结果如下图所示。主要用于文本分类任务的TextCNN结构描述,详细解释了TextCNN架构及词向量矩阵是如何做卷积的。
假设我们有一些句子需要对其进行分类。句子中每个词是由n维词向量组成的,也就是说输入矩阵大小为m*n,其中m为句子长度。CNN需要对输入样本进行卷积操作,对于文本数据,filter不再横向滑动,仅仅是向下移动,有点类似于N-gram在提取词与词间的局部相关性。
图中共有三种步长策略,分别是2、3、4,每个步长都有两个filter(实际训练时filter数量会很多)。在不同词窗上应用不同filter,最终得到6个卷积后的向量。然后对每一个向量进行最大化池化操作并拼接各个池化值,最终得到这个句子的特征表示,将这个句子向量丢给分类器进行分类,最终完成整个文本分类流程。
最后真心推荐下面这些大佬关于TextCNN的介绍,尤其是CSDN的Asia-Lee大佬,很喜欢他的文章,真心棒!
《Convolutional Neural Networks for Sentence Classification》《A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification》/asialee_bird/article/details/88813385深入TextCNN(一)详述CNN及TextCNN原理 - 知乎
2.代码实现
Keras实现文本分类的TextCNN代码如下:
Keras_TextCNN_cnews.py
# -*- coding: utf-8 -*-"""Created on -03-19@author: xiuzhang Eastmount CSDNTextCNN Model"""import osimport timeimport pickleimport pandas as pdimport numpy as npfrom sklearn import metricsimport matplotlib.pyplot as pltimport seaborn as snsimport tensorflow as tffrom sklearn.preprocessing import LabelEncoder,OneHotEncoderfrom keras.models import Modelfrom keras.layers import LSTM, Activation, Dense, Dropout, Input, Embeddingfrom keras.layers import Convolution1D, MaxPool1D, Flattenfrom keras.preprocessing.text import Tokenizerfrom keras.preprocessing import sequencefrom keras.callbacks import EarlyStoppingfrom keras.models import load_modelfrom keras.models import Sequentialfrom keras.layers.merge import concatenate## GPU处理 读者如果是CPU注释该部分代码即可## 指定每个GPU进程中使用显存的上限 0.9表示可以使用GPU 90%的资源进行训练os.environ["CUDA_DEVICES_ORDER"] = "PCI_BUS_IS"os.environ["CUDA_VISIBLE_DEVICES"] = "0"gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8)sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))start = time.clock()#----------------------------第一步 数据读取----------------------------## 读取测数据集train_df = pd.read_csv("news_dataset_train_fc.csv")val_df = pd.read_csv("news_dataset_val_fc.csv")test_df = pd.read_csv("news_dataset_test_fc.csv")## 解决中文显示问题plt.rcParams['font.sans-serif'] = ['KaiTi'] #指定默认字体 SimHei黑体plt.rcParams['axes.unicode_minus'] = False #解决保存图像是负号'#--------------------------第二步 OneHotEncoder()编码--------------------## 对数据集的标签数据进行编码train_y = train_df.labelval_y = val_df.labeltest_y = test_df.labelprint("Label:")print(train_y[:10])le = LabelEncoder()train_y = le.fit_transform(train_y).reshape(-1,1)val_y = le.transform(val_y).reshape(-1,1)test_y = le.transform(test_y).reshape(-1,1)print("LabelEncoder")print(train_y[:10])print(len(train_y))## 对数据集的标签数据进行one-hot编码ohe = OneHotEncoder()train_y = ohe.fit_transform(train_y).toarray()val_y = ohe.transform(val_y).toarray()test_y = ohe.transform(test_y).toarray()print("OneHotEncoder:")print(train_y[:10])#-----------------------第三步 使用Tokenizer对词组进行编码--------------------max_words = 6000max_len = 600tok = Tokenizer(num_words=max_words) #最大词语数为6000print(train_df.cutword[:5])print(type(train_df.cutword))## 防止语料中存在数字str处理train_content = [str(a) for a in train_df.cutword.tolist()]val_content = [str(a) for a in val_df.cutword.tolist()]test_content = [str(a) for a in test_df.cutword.tolist()]tok.fit_on_texts(train_content)print(tok)## 保存训练好的Tokenizer和导入with open('tok.pickle', 'wb') as handle: #savingpickle.dump(tok, handle, protocol=pickle.HIGHEST_PROTOCOL)with open('tok.pickle', 'rb') as handle: #loadingtok = pickle.load(handle)#---------------------------第四步 数据转化为序列-----------------------------train_seq = tok.texts_to_sequences(train_content)val_seq = tok.texts_to_sequences(val_content)test_seq = tok.texts_to_sequences(test_content)## 将每个序列调整为相同的长度train_seq_mat = sequence.pad_sequences(train_seq,maxlen=max_len)val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)test_seq_mat = sequence.pad_sequences(test_seq,maxlen=max_len)print("数据转换序列")print(train_seq_mat.shape)print(val_seq_mat.shape)print(test_seq_mat.shape)print(train_seq_mat[:2])#-------------------------------第五步 建立TextCNN模型--------------------------## 类别为4个num_labels = 4inputs = Input(name='inputs',shape=[max_len], dtype='float64')## 词嵌入使用预训练的词向量layer = Embedding(max_words+1, 256, input_length=max_len, trainable=False)(inputs)## 词窗大小分别为3,4,5cnn1 = Convolution1D(256, 3, padding='same', strides = 1, activation='relu')(layer)cnn1 = MaxPool1D(pool_size=4)(cnn1)cnn2 = Convolution1D(256, 4, padding='same', strides = 1, activation='relu')(layer)cnn2 = MaxPool1D(pool_size=4)(cnn2)cnn3 = Convolution1D(256, 5, padding='same', strides = 1, activation='relu')(layer)cnn3 = MaxPool1D(pool_size=4)(cnn3)# 合并三个模型的输出向量cnn = concatenate([cnn1,cnn2,cnn3], axis=-1)flat = Flatten()(cnn) drop = Dropout(0.2)(flat)main_output = Dense(num_labels, activation='softmax')(drop)model = Model(inputs=inputs, outputs=main_output)model.summary()pile(loss="categorical_crossentropy",optimizer='adam',# RMSprop()metrics=["accuracy"])#-------------------------------第六步 模型训练和预测--------------------------## 先设置为train训练 再设置为test测试flag = "train"if flag == "train":print("模型训练")## 模型训练 当val-loss不再提升时停止训练 0.0001model_fit = model.fit(train_seq_mat, train_y, batch_size=128, epochs=10,validation_data=(val_seq_mat,val_y),callbacks=[EarlyStopping(monitor='val_loss',min_delta=0.0001)] )model.save('my_model.h5') del modelelapsed = (time.clock() - start)print("Time used:", elapsed)print(model_fit.history)else:print("模型预测")## 导入已经训练好的模型model = load_model('my_model.h5')## 对测试集进行预测test_pre = model.predict(test_seq_mat)## 评价预测效果,计算混淆矩阵confm = metrics.confusion_matrix(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1))print(confm)## 混淆矩阵可视化Labname = ["体育", "文化", "财经", "游戏"]print(metrics.classification_report(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1)))plt.figure(figsize=(8,8))sns.heatmap(confm.T, square=True, annot=True,fmt='d', cbar=False, linewidths=.6,cmap="YlGnBu")plt.xlabel('True label',size = 14)plt.ylabel('Predicted label', size = 14)plt.xticks(np.arange(4)+0.5, Labname, size = 12)plt.yticks(np.arange(4)+0.5, Labname, size = 12)plt.savefig('result.png')plt.show()#----------------------------------第七 验证算法--------------------------## 使用tok对验证数据集重新预处理,并使用训练好的模型进行预测val_seq = tok.texts_to_sequences(val_df.cutword)## 将每个序列调整为相同的长度val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)## 对验证集进行预测val_pre = model.predict(val_seq_mat)print(metrics.classification_report(np.argmax(val_y,axis=1),np.argmax(val_pre,axis=1)))elapsed = (time.clock() - start)print("Time used:", elapsed)
训练模型如下所示:
__________________________________________________________________________________________________Layer (type)Output Shape Param #Connected to ==================================================================================================inputs (InputLayer) (None, 600)0 __________________________________________________________________________________________________embedding_1 (Embedding) (None, 600, 256)1536256inputs[0][0] __________________________________________________________________________________________________conv1d_1 (Conv1D)(None, 600, 256)196864embedding_1[0][0]__________________________________________________________________________________________________conv1d_2 (Conv1D)(None, 600, 256)262400embedding_1[0][0]__________________________________________________________________________________________________conv1d_3 (Conv1D)(None, 600, 256)327936embedding_1[0][0]__________________________________________________________________________________________________max_pooling1d_1 (MaxPooling1D) (None, 150, 256)0 conv1d_1[0][0] __________________________________________________________________________________________________max_pooling1d_2 (MaxPooling1D) (None, 150, 256)0 conv1d_2[0][0] __________________________________________________________________________________________________max_pooling1d_3 (MaxPooling1D) (None, 150, 256)0 conv1d_3[0][0] __________________________________________________________________________________________________concatenate_1 (Concatenate)(None, 150, 768)0 max_pooling1d_1[0][0] max_pooling1d_2[0][0] max_pooling1d_3[0][0] __________________________________________________________________________________________________flatten_1 (Flatten) (None, 115200) 0 concatenate_1[0][0] __________________________________________________________________________________________________dropout_1 (Dropout) (None, 115200) 0 flatten_1[0][0] __________________________________________________________________________________________________dense_1 (Dense) (None, 4) 460804dropout_1[0][0] ==================================================================================================Total params: 2,784,260Trainable params: 1,248,004Non-trainable params: 1,536,256__________________________________________________________________________________________________
预测结果如下:
[[4448 238 182 132][ 151 4572 124 153][ 185 176 4545 94][ 181 394 207 4218]]precision recall f1-score support0 0.900.890.8950001 0.850.910.8850002 0.900.910.9050003 0.920.840.885000avg / total 0.890.890.8920000precision recall f1-score support0 0.900.880.8950001 0.860.930.8950002 0.910.890.9050003 0.920.880.905000avg / total 0.900.900.9020000
五.LSTM中文文本分类
1.原理介绍
Long Short Term 网络(LSTM) 是一种RNN(Recurrent Neural Network)特殊的类型,可以学习长期依赖信息。LSTM 由Hochreiter & Schmidhuber (1997)提出,并在近期被Alex Graves进行了改良和推广。在很多问题,LSTM都取得相当巨大的成功,并得到了广泛的使用。
由于RNN存在梯度消失的问题,人们对于序列索引位置t的隐藏结构做了改进,通过一些技巧让隐藏结构复杂起来,来避免梯度消失的问题,这样的特殊RNN就是我们的LSTM。LSTM的全称是Long Short-Term Memory。LSTM由于其设计的特点,非常适合用于对时序数据的建模,如文本数据。LSTM的结构如下图:
LSTM 通过刻意的设计来避免长期依赖问题。记住长期的信息在实践中是 LSTM 的默认行为,而非需要付出很大代价才能获得的能力。LSTM是在普通的RNN上面做了一些改进,LSTM RNN多了三个控制器,即:
输入控制器输出控制器忘记控制器
左边多了个条主线,例如电影的主线剧情,而原本的RNN体系变成了分线剧情,并且三个控制器都在分线上。
输入控制器(write gate): 在输入input时设置一个gate,gate的作用是判断要不要写入这个input到我们的内存Memory中,它相当于一个参数,也是可以被训练的,这个参数就是用来控制要不要记住当下这个点。输出控制器(read gate): 在输出位置的gate,判断要不要读取现在的Memory。忘记控制器(forget gate): 处理位置的忘记控制器,判断要不要忘记之前的Memory。
LSTM工作原理为:如果分支内容对于最终结果十分重要,输入控制器会将这个分支内容按重要程度写入主线内容,再进行分析;如果分线内容改变了我们之前的想法,那么忘记控制器会将某些主线内容忘记,然后按比例替换新内容,所以主线内容的更新就取决于输入和忘记控制;最后的输出会基于主线内容和分线内容。通过这三个gate能够很好地控制我们的RNN,基于这些控制机制,LSTM是延缓记忆的良药,从而带来更好的结果。
2.代码实现
Keras实现文本分类的LSTM代码如下:
Keras_LSTM_cnews.py
"""Created on -03-19@author: xiuzhang Eastmount CSDNLSTM Model"""import osimport timeimport pickleimport pandas as pdimport numpy as npfrom sklearn import metricsimport matplotlib.pyplot as pltimport seaborn as snsimport tensorflow as tffrom sklearn.preprocessing import LabelEncoder,OneHotEncoderfrom keras.models import Modelfrom keras.layers import LSTM, Activation, Dense, Dropout, Input, Embeddingfrom keras.layers import Convolution1D, MaxPool1D, Flattenfrom keras.preprocessing.text import Tokenizerfrom keras.preprocessing import sequencefrom keras.callbacks import EarlyStoppingfrom keras.models import load_modelfrom keras.models import Sequential#GPU加速 CuDNNLSTM比LSTM快from keras.layers import CuDNNLSTM, CuDNNGRU## GPU处理 读者如果是CPU注释该部分代码即可## 指定每个GPU进程中使用显存的上限 0.9表示可以使用GPU 90%的资源进行训练os.environ["CUDA_DEVICES_ORDER"] = "PCI_BUS_IS"os.environ["CUDA_VISIBLE_DEVICES"] = "0"gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8)sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))start = time.clock()#----------------------------第一步 数据读取----------------------------## 读取测数据集train_df = pd.read_csv("news_dataset_train_fc.csv")val_df = pd.read_csv("news_dataset_val_fc.csv")test_df = pd.read_csv("news_dataset_test_fc.csv")print(train_df.head())## 解决中文显示问题plt.rcParams['font.sans-serif'] = ['KaiTi'] #指定默认字体 SimHei黑体plt.rcParams['axes.unicode_minus'] = False #解决保存图像是负号'#--------------------------第二步 OneHotEncoder()编码--------------------## 对数据集的标签数据进行编码train_y = train_df.labelval_y = val_df.labeltest_y = test_df.labelprint("Label:")print(train_y[:10])le = LabelEncoder()train_y = le.fit_transform(train_y).reshape(-1,1)val_y = le.transform(val_y).reshape(-1,1)test_y = le.transform(test_y).reshape(-1,1)print("LabelEncoder")print(train_y[:10])print(len(train_y))## 对数据集的标签数据进行one-hot编码ohe = OneHotEncoder()train_y = ohe.fit_transform(train_y).toarray()val_y = ohe.transform(val_y).toarray()test_y = ohe.transform(test_y).toarray()print("OneHotEncoder:")print(train_y[:10])#-----------------------第三步 使用Tokenizer对词组进行编码--------------------max_words = 6000max_len = 600tok = Tokenizer(num_words=max_words) #最大词语数为6000print(train_df.cutword[:5])print(type(train_df.cutword))## 防止语料中存在数字str处理train_content = [str(a) for a in train_df.cutword.tolist()]val_content = [str(a) for a in val_df.cutword.tolist()]test_content = [str(a) for a in test_df.cutword.tolist()]tok.fit_on_texts(train_content)print(tok)## 保存训练好的Tokenizer和导入with open('tok.pickle', 'wb') as handle: #savingpickle.dump(tok, handle, protocol=pickle.HIGHEST_PROTOCOL)with open('tok.pickle', 'rb') as handle: #loadingtok = pickle.load(handle)#---------------------------第四步 数据转化为序列-----------------------------train_seq = tok.texts_to_sequences(train_content)val_seq = tok.texts_to_sequences(val_content)test_seq = tok.texts_to_sequences(test_content)## 将每个序列调整为相同的长度train_seq_mat = sequence.pad_sequences(train_seq,maxlen=max_len)val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)test_seq_mat = sequence.pad_sequences(test_seq,maxlen=max_len)print("数据转换序列")print(train_seq_mat.shape)print(val_seq_mat.shape)print(test_seq_mat.shape)print(train_seq_mat[:2])#-------------------------------第五步 建立LSTM模型--------------------------## 定义LSTM模型inputs = Input(name='inputs',shape=[max_len],dtype='float64')## Embedding(词汇表大小,batch大小,每个新闻的词长)layer = Embedding(max_words+1, 128, input_length=max_len)(inputs)#layer = LSTM(128)(layer)layer = CuDNNLSTM(128)(layer)layer = Dense(128, activation="relu", name="FC1")(layer)layer = Dropout(0.1)(layer)layer = Dense(4, activation="softmax", name="FC2")(layer)model = Model(inputs=inputs, outputs=layer)model.summary()pile(loss="categorical_crossentropy",optimizer='adam', # RMSprop()metrics=["accuracy"])#-------------------------------第六步 模型训练和预测--------------------------## 先设置为train训练 再设置为test测试flag = "train"if flag == "train":print("模型训练")## 模型训练 当val-loss不再提升时停止训练 0.0001model_fit = model.fit(train_seq_mat, train_y, batch_size=128, epochs=10,validation_data=(val_seq_mat,val_y),callbacks=[EarlyStopping(monitor='val_loss',min_delta=0.0001)] )model.save('my_model.h5') del modelelapsed = (time.clock() - start)print("Time used:", elapsed)print(model_fit.history)else:print("模型预测")## 导入已经训练好的模型model = load_model('my_model.h5')## 对测试集进行预测test_pre = model.predict(test_seq_mat)## 评价预测效果,计算混淆矩阵confm = metrics.confusion_matrix(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1))print(confm)## 混淆矩阵可视化Labname = ["体育", "文化", "财经", "游戏"]print(metrics.classification_report(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1)))plt.figure(figsize=(8,8))sns.heatmap(confm.T, square=True, annot=True,fmt='d', cbar=False, linewidths=.6,cmap="YlGnBu")plt.xlabel('True label',size = 14)plt.ylabel('Predicted label', size = 14)plt.xticks(np.arange(4)+0.8, Labname, size = 12)plt.yticks(np.arange(4)+0.4, Labname, size = 12)plt.savefig('result.png')plt.show()#----------------------------------第七 验证算法--------------------------## 使用tok对验证数据集重新预处理,并使用训练好的模型进行预测val_seq = tok.texts_to_sequences(val_df.cutword)## 将每个序列调整为相同的长度val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)## 对验证集进行预测val_pre = model.predict(val_seq_mat)print(metrics.classification_report(np.argmax(val_y,axis=1),np.argmax(val_pre,axis=1)))elapsed = (time.clock() - start)print("Time used:", elapsed)
训练输出模型如下所示:
_________________________________________________________________Layer (type) Output Shape Param # =================================================================inputs (InputLayer)(None, 600)0 _________________________________________________________________embedding_1 (Embedding)(None, 600, 128)768128 _________________________________________________________________cu_dnnlstm_1 (CuDNNLSTM)(None, 128)132096 _________________________________________________________________FC1 (Dense) (None, 128)16512_________________________________________________________________dropout_1 (Dropout)(None, 128)0 _________________________________________________________________FC2 (Dense) (None, 4) 516 =================================================================Total params: 917,252Trainable params: 917,252Non-trainable params: 0
预测结果如下所示:
[[4539 153 188 120][ 47 4628 181 144][ 113 133 4697 57][ 101 292 157 4450]]precision recall f1-score support0 0.950.910.9350001 0.890.930.9150002 0.900.940.9250003 0.930.890.915000avg / total 0.920.920.9220000precision recall f1-score support0 0.960.890.9250001 0.890.940.9250002 0.900.930.9250003 0.940.920.935000avg / total 0.920.920.9220000
六.BiLSTM中文文本分类
1.原理介绍
BiLSTM是Bi-directional Long Short-Term Memory的缩写,是由前向LSTM与后向LSTM组合而成。它和LSTM在自然语言处理任务中都常被用来建模上下文信息。前向的LSTM与后向的LSTM结合成BiLSTM。比如,我们对“我爱中国”这句话进行编码,模型如图所示。
由于利用LSTM对句子进行建模还存在一个问题:无法编码从后到前的信息。在更细粒度的分类时,如对于强程度的褒义、弱程度的褒义、中性、弱程度的贬义、强程度的贬义的五分类任务需要注意情感词、程度词、否定词之间的交互。举一个例子,“这个餐厅脏得不行,没有隔壁好”,这里的“不行”是对“脏”的程度的一种修饰,通过BiLSTM可以更好的捕捉双向的语义依赖。
参考文章:详解BiLSTM及代码实现 - 知乎
2.代码实现
Keras实现文本分类的BiLSTM代码如下:
Keras_BiLSTM_cnews.py
"""Created on -03-19@author: xiuzhang Eastmount CSDNBiLSTM Model"""import osimport timeimport pickleimport pandas as pdimport numpy as npfrom sklearn import metricsimport matplotlib.pyplot as pltimport seaborn as snsimport tensorflow as tffrom sklearn.preprocessing import LabelEncoder,OneHotEncoderfrom keras.models import Modelfrom keras.layers import LSTM, Activation, Dense, Dropout, Input, Embeddingfrom keras.layers import Convolution1D, MaxPool1D, Flattenfrom keras.preprocessing.text import Tokenizerfrom keras.preprocessing import sequencefrom keras.callbacks import EarlyStoppingfrom keras.models import load_modelfrom keras.models import Sequential#GPU加速 CuDNNLSTM比LSTM快from keras.layers import CuDNNLSTM, CuDNNGRUfrom keras.layers import Bidirectional## GPU处理 读者如果是CPU注释该部分代码即可## 指定每个GPU进程中使用显存的上限 0.9表示可以使用GPU 90%的资源进行训练os.environ["CUDA_DEVICES_ORDER"] = "PCI_BUS_IS"os.environ["CUDA_VISIBLE_DEVICES"] = "0"gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8)sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))start = time.clock()#----------------------------第一步 数据读取----------------------------## 读取测数据集train_df = pd.read_csv("news_dataset_train_fc.csv")val_df = pd.read_csv("news_dataset_val_fc.csv")test_df = pd.read_csv("news_dataset_test_fc.csv")print(train_df.head())## 解决中文显示问题plt.rcParams['font.sans-serif'] = ['KaiTi'] #指定默认字体 SimHei黑体plt.rcParams['axes.unicode_minus'] = False #解决保存图像是负号'#--------------------------第二步 OneHotEncoder()编码--------------------## 对数据集的标签数据进行编码train_y = train_df.labelval_y = val_df.labeltest_y = test_df.labelprint("Label:")print(train_y[:10])le = LabelEncoder()train_y = le.fit_transform(train_y).reshape(-1,1)val_y = le.transform(val_y).reshape(-1,1)test_y = le.transform(test_y).reshape(-1,1)print("LabelEncoder")print(train_y[:10])print(len(train_y))## 对数据集的标签数据进行one-hot编码ohe = OneHotEncoder()train_y = ohe.fit_transform(train_y).toarray()val_y = ohe.transform(val_y).toarray()test_y = ohe.transform(test_y).toarray()print("OneHotEncoder:")print(train_y[:10])#-----------------------第三步 使用Tokenizer对词组进行编码--------------------max_words = 6000max_len = 600tok = Tokenizer(num_words=max_words) #最大词语数为6000print(train_df.cutword[:5])print(type(train_df.cutword))## 防止语料中存在数字str处理train_content = [str(a) for a in train_df.cutword.tolist()]val_content = [str(a) for a in val_df.cutword.tolist()]test_content = [str(a) for a in test_df.cutword.tolist()]tok.fit_on_texts(train_content)print(tok)## 保存训练好的Tokenizer和导入with open('tok.pickle', 'wb') as handle: #savingpickle.dump(tok, handle, protocol=pickle.HIGHEST_PROTOCOL)with open('tok.pickle', 'rb') as handle: #loadingtok = pickle.load(handle)#---------------------------第四步 数据转化为序列-----------------------------train_seq = tok.texts_to_sequences(train_content)val_seq = tok.texts_to_sequences(val_content)test_seq = tok.texts_to_sequences(test_content)## 将每个序列调整为相同的长度train_seq_mat = sequence.pad_sequences(train_seq,maxlen=max_len)val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)test_seq_mat = sequence.pad_sequences(test_seq,maxlen=max_len)print("数据转换序列")print(train_seq_mat.shape)print(val_seq_mat.shape)print(test_seq_mat.shape)print(train_seq_mat[:2])#-------------------------------第五步 建立BiLSTM模型--------------------------num_labels = 4model = Sequential()model.add(Embedding(max_words+1, 128, input_length=max_len))model.add(Bidirectional(CuDNNLSTM(128)))model.add(Dense(128, activation='relu'))model.add(Dropout(0.3))model.add(Dense(num_labels, activation='softmax'))model.summary()pile(loss="categorical_crossentropy",optimizer='adam', # RMSprop()metrics=["accuracy"])#-------------------------------第六步 模型训练和预测--------------------------## 先设置为train训练 再设置为test测试flag = "train"if flag == "train":print("模型训练")## 模型训练 当val-loss不再提升时停止训练 0.0001model_fit = model.fit(train_seq_mat, train_y, batch_size=128, epochs=10,validation_data=(val_seq_mat,val_y),callbacks=[EarlyStopping(monitor='val_loss',min_delta=0.0001)] )model.save('my_model.h5') del modelelapsed = (time.clock() - start)print("Time used:", elapsed)print(model_fit.history)else:print("模型预测")## 导入已经训练好的模型model = load_model('my_model.h5')## 对测试集进行预测test_pre = model.predict(test_seq_mat)## 评价预测效果,计算混淆矩阵confm = metrics.confusion_matrix(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1))print(confm)## 混淆矩阵可视化Labname = ["体育", "文化", "财经", "游戏"]print(metrics.classification_report(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1)))plt.figure(figsize=(8,8))sns.heatmap(confm.T, square=True, annot=True,fmt='d', cbar=False, linewidths=.6,cmap="YlGnBu")plt.xlabel('True label',size = 14)plt.ylabel('Predicted label', size = 14)plt.xticks(np.arange(4)+0.5, Labname, size = 12)plt.yticks(np.arange(4)+0.5, Labname, size = 12)plt.savefig('result.png')plt.show()#----------------------------------第七 验证算法--------------------------## 使用tok对验证数据集重新预处理,并使用训练好的模型进行预测val_seq = tok.texts_to_sequences(val_df.cutword)## 将每个序列调整为相同的长度val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)## 对验证集进行预测val_pre = model.predict(val_seq_mat)print(metrics.classification_report(np.argmax(val_y,axis=1),np.argmax(val_pre,axis=1)))elapsed = (time.clock() - start)print("Time used:", elapsed)
训练输出模型如下所示,GPU时间还是非常快。
_________________________________________________________________Layer (type) Output Shape Param # =================================================================embedding_1 (Embedding)(None, 600, 128)768128 _________________________________________________________________bidirectional_1 (Bidirection (None, 256)264192 _________________________________________________________________dense_1 (Dense) (None, 128)32896_________________________________________________________________dropout_1 (Dropout)(None, 128)0 _________________________________________________________________dense_2 (Dense) (None, 4) 516 =================================================================Total params: 1,065,732Trainable params: 1,065,732Non-trainable params: 0Train on 40000 samples, validate on 20000 samplesEpoch 1/1040000/40000 [==============================] - 23s 587us/step - loss: 0.5825 - acc: 0.8038 - val_loss: 0.2321 - val_acc: 0.9246Epoch 2/1040000/40000 [==============================] - 21s 521us/step - loss: 0.1433 - acc: 0.9542 - val_loss: 0.2422 - val_acc: 0.9228Time used: 52.763230400000005
预测结果如下图所示:
[[4593 143 113 151][ 81 4679 60 180][ 110 199 4590 101][ 73 254 82 4591]]precision recall f1-score support0 0.950.920.9350001 0.890.940.9150002 0.950.920.9350003 0.910.920.925000avg / total 0.920.920.9220000precision recall f1-score support0 0.940.900.9250001 0.890.950.9250002 0.950.900.9350003 0.910.940.935000avg / total 0.920.920.9220000
七.BiLSTM+Attention中文文本分类
1.原理介绍
Attention机制是模仿人类注意力而提出的一种解决问题的办法,简单地说就是从大量信息中快速筛选出高价值信息。主要用于解决LSTM/RNN模型输入序列较长的时候很难获得最终合理的向量表示问题,做法是保留LSTM的中间结果,用新的模型对其进行学习,并将其与输出进行关联,从而达到信息筛选的目的。
What is attention?
先简单描述一下attention机制是什么。相信做NLP的同学对这个机制不会很陌生,它在论文《Attention is all you need》中可以说是大放异彩,在machine translation任务中,帮助深度模型在性能上有了很大的提升,输出了当时最好的state-of-art model。当然该模型除了attention机制外,还用了很多有用的trick,以帮助提升模型性能。但是不能否认的时,这个模型的核心就是attention。
attention机制又称为注意力机制,顾名思义,是一种能让模型对重要信息重点关注并充分学习吸收的技术,它不算是一个完整的模型,应当是一种技术,能够作用于任何序列模型中。
Why attention?
为什么要引入attention机制。比如在seq2seq模型中,对于一段文本序列,我们通常要使用某种机制对该序列进行编码,通过降维等方式将其encode成一个固定长度的向量,用于输入到后面的全连接层。一般我们会使用CNN或者RNN(包括GRU或者LSTM)等模型来对序列数据进行编码,然后采用各种pooling或者对RNN直接取最后一个t时刻的hidden state作为句子的向量输出。
但这里会有一个问题:常规的编码方法,无法体现对一个句子序列中不同语素的关注程度,在自然语言中,一个句子中的不同部分是有不同含义和重要性的,比如上面的例子中:I hate this movie.如果做情感分析,明显对hate这个词语应当关注更多。当然是用CNN和RNN能够编码这种信息,但这种编码能力也是有上限的,对于较长的文本,模型效果不会再提升太多。
参考及推荐文章:Attention机制简单总结 - 知乎
Attention的应用领域非常广泛,文本、图片等都有应用。
文本:应用于seq2seq模型,最常见的应用是翻译图片:应用于卷积神经网络的图片提取语音
下图是一个比较经典的BiLSTM+Attention模型,也是我们接下来需要建立的模型。
2.代码实现
Keras实现文本分类的BiLSTM+Attention代码如下:
Keras_Attention_BiLSTM_cnews.py
"""Created on -03-19@author: xiuzhang Eastmount CSDNBiLSTM+Attention Model"""import osimport timeimport pickleimport pandas as pdimport numpy as npfrom sklearn import metricsimport matplotlib.pyplot as pltimport seaborn as snsimport tensorflow as tffrom sklearn.preprocessing import LabelEncoder,OneHotEncoderfrom keras.models import Modelfrom keras.layers import LSTM, Activation, Dense, Dropout, Input, Embeddingfrom keras.layers import Convolution1D, MaxPool1D, Flattenfrom keras.preprocessing.text import Tokenizerfrom keras.preprocessing import sequencefrom keras.callbacks import EarlyStoppingfrom keras.models import load_modelfrom keras.models import Sequential#GPU加速 CuDNNLSTM比LSTM快from keras.layers import CuDNNLSTM, CuDNNGRUfrom keras.layers import Bidirectional## GPU处理 读者如果是CPU注释该部分代码即可## 指定每个GPU进程中使用显存的上限 0.9表示可以使用GPU 90%的资源进行训练os.environ["CUDA_DEVICES_ORDER"] = "PCI_BUS_IS"os.environ["CUDA_VISIBLE_DEVICES"] = "0"gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8)sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))start = time.clock()#----------------------------第一步 数据读取----------------------------## 读取测数据集train_df = pd.read_csv("news_dataset_train_fc.csv")val_df = pd.read_csv("news_dataset_val_fc.csv")test_df = pd.read_csv("news_dataset_test_fc.csv")print(train_df.head())## 解决中文显示问题plt.rcParams['font.sans-serif'] = ['KaiTi'] #指定默认字体 SimHei黑体plt.rcParams['axes.unicode_minus'] = False #解决保存图像是负号'#--------------------------第二步 OneHotEncoder()编码--------------------## 对数据集的标签数据进行编码train_y = train_df.labelval_y = val_df.labeltest_y = test_df.labelprint("Label:")print(train_y[:10])le = LabelEncoder()train_y = le.fit_transform(train_y).reshape(-1,1)val_y = le.transform(val_y).reshape(-1,1)test_y = le.transform(test_y).reshape(-1,1)print("LabelEncoder")print(train_y[:10])print(len(train_y))## 对数据集的标签数据进行one-hot编码ohe = OneHotEncoder()train_y = ohe.fit_transform(train_y).toarray()val_y = ohe.transform(val_y).toarray()test_y = ohe.transform(test_y).toarray()print("OneHotEncoder:")print(train_y[:10])#-----------------------第三步 使用Tokenizer对词组进行编码--------------------max_words = 6000max_len = 600tok = Tokenizer(num_words=max_words) #最大词语数为6000print(train_df.cutword[:5])print(type(train_df.cutword))## 防止语料中存在数字str处理train_content = [str(a) for a in train_df.cutword.tolist()]val_content = [str(a) for a in val_df.cutword.tolist()]test_content = [str(a) for a in test_df.cutword.tolist()]tok.fit_on_texts(train_content)print(tok)## 保存训练好的Tokenizer和导入with open('tok.pickle', 'wb') as handle: #savingpickle.dump(tok, handle, protocol=pickle.HIGHEST_PROTOCOL)with open('tok.pickle', 'rb') as handle: #loadingtok = pickle.load(handle)#---------------------------第四步 数据转化为序列-----------------------------train_seq = tok.texts_to_sequences(train_content)val_seq = tok.texts_to_sequences(val_content)test_seq = tok.texts_to_sequences(test_content)## 将每个序列调整为相同的长度train_seq_mat = sequence.pad_sequences(train_seq,maxlen=max_len)val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)test_seq_mat = sequence.pad_sequences(test_seq,maxlen=max_len)print("数据转换序列")print(train_seq_mat.shape)print(val_seq_mat.shape)print(test_seq_mat.shape)print(train_seq_mat[:2])#---------------------------第五步 建立Attention机制----------------------"""由于Keras目前还没有现成的Attention层可以直接使用,我们需要自己来构建一个新的层函数。Keras自定义的函数主要分为四个部分,分别是:init:初始化一些需要的参数bulid:具体来定义权重是怎么样的call:核心部分,定义向量是如何进行运算的compute_output_shape:定义该层输出的大小推荐文章:/huanghaocs/article/details/95752379/p/2991"""# Hierarchical Model with Attentionfrom keras import initializersfrom keras import constraintsfrom keras import activationsfrom keras import regularizersfrom keras import backend as Kfrom keras.engine.topology import LayerK.clear_session()class AttentionLayer(Layer):def __init__(self, attention_size=None, **kwargs):self.attention_size = attention_sizesuper(AttentionLayer, self).__init__(**kwargs)def get_config(self):config = super().get_config()config['attention_size'] = self.attention_sizereturn configdef build(self, input_shape):assert len(input_shape) == 3self.time_steps = input_shape[1]hidden_size = input_shape[2]if self.attention_size is None:self.attention_size = hidden_sizeself.W = self.add_weight(name='att_weight', shape=(hidden_size, self.attention_size),initializer='uniform', trainable=True)self.b = self.add_weight(name='att_bias', shape=(self.attention_size,),initializer='uniform', trainable=True)self.V = self.add_weight(name='att_var', shape=(self.attention_size,),initializer='uniform', trainable=True)super(AttentionLayer, self).build(input_shape)def call(self, inputs):self.V = K.reshape(self.V, (-1, 1))H = K.tanh(K.dot(inputs, self.W) + self.b)score = K.softmax(K.dot(H, self.V), axis=1)outputs = K.sum(score * inputs, axis=1)return outputsdef compute_output_shape(self, input_shape):return input_shape[0], input_shape[2]#-------------------------------第六步 建立BiLSTM模型--------------------------## 定义BiLSTM模型## BiLSTM+Attentionnum_labels = 4inputs = Input(name='inputs',shape=[max_len],dtype='float64')layer = Embedding(max_words+1, 256, input_length=max_len)(inputs)#lstm = Bidirectional(LSTM(100, dropout=0.2, recurrent_dropout=0.1, return_sequences=True))(layer)bilstm = Bidirectional(CuDNNLSTM(128, return_sequences=True))(layer) #参数保持维度3layer = Dense(128, activation='relu')(bilstm)layer = Dropout(0.2)(layer)## 注意力机制attention = AttentionLayer(attention_size=50)(layer)output = Dense(num_labels, activation='softmax')(attention)model = Model(inputs=inputs, outputs=output)model.summary()pile(loss="categorical_crossentropy",optimizer='adam',# RMSprop()metrics=["accuracy"])#-------------------------------第七步 模型训练和预测--------------------------## 先设置为train训练 再设置为test测试flag = "test"if flag == "train":print("模型训练")## 模型训练 当val-loss不再提升时停止训练 0.0001model_fit = model.fit(train_seq_mat, train_y, batch_size=128, epochs=10,validation_data=(val_seq_mat,val_y),callbacks=[EarlyStopping(monitor='val_loss',min_delta=0.0001)] )## 保存模型model.save('my_model.h5') del model # deletes the existing model## 计算时间elapsed = (time.clock() - start)print("Time used:", elapsed)print(model_fit.history)else:print("模型预测")## 导入已经训练好的模型model = load_model('my_model.h5', custom_objects={'AttentionLayer': AttentionLayer(50)}, compile=False)## 对测试集进行预测test_pre = model.predict(test_seq_mat)## 评价预测效果,计算混淆矩阵confm = metrics.confusion_matrix(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1))print(confm)## 混淆矩阵可视化Labname = ["体育", "文化", "财经", "游戏"]print(metrics.classification_report(np.argmax(test_y,axis=1),np.argmax(test_pre,axis=1)))plt.figure(figsize=(8,8))sns.heatmap(confm.T, square=True, annot=True,fmt='d', cbar=False, linewidths=.6,cmap="YlGnBu")plt.xlabel('True label',size = 14)plt.ylabel('Predicted label', size = 14)plt.xticks(np.arange(4)+0.5, Labname, size = 12)plt.yticks(np.arange(4)+0.5, Labname, size = 12)plt.savefig('result.png')plt.show()#----------------------------------第七 验证算法--------------------------## 使用tok对验证数据集重新预处理,并使用训练好的模型进行预测val_seq = tok.texts_to_sequences(val_df.cutword)## 将每个序列调整为相同的长度val_seq_mat = sequence.pad_sequences(val_seq,maxlen=max_len)## 对验证集进行预测val_pre = model.predict(val_seq_mat)print(metrics.classification_report(np.argmax(val_y,axis=1),np.argmax(val_pre,axis=1)))## 计算时间elapsed = (time.clock() - start)print("Time used:", elapsed)
训练输出模型如下所示:
_________________________________________________________________Layer (type) Output Shape Param # =================================================================inputs (InputLayer)(None, 600)0 _________________________________________________________________embedding_1 (Embedding)(None, 600, 256)1536256 _________________________________________________________________bidirectional_1 (Bidirection (None, 600, 256)395264 _________________________________________________________________dense_1 (Dense) (None, 600, 128)32896_________________________________________________________________dropout_1 (Dropout)(None, 600, 128)0 _________________________________________________________________attention_layer_1 (Attention (None, 128)6500_________________________________________________________________dense_2 (Dense) (None, 4) 516 =================================================================Total params: 1,971,432Trainable params: 1,971,432Non-trainable params: 0
预测结果如下图所示:
[[4625 138 100 137][ 63 4692 77 168][ 129 190 4589 92][ 82 299 78 4541]]precision recall f1-score support0 0.940.930.9350001 0.880.940.9150002 0.950.920.9350003 0.920.910.915000avg / total 0.920.920.9220000precision recall f1-score support0 0.950.910.9350001 0.880.950.9150002 0.950.900.9250003 0.920.930.935000avg / total 0.920.920.9220000
点击关注,第一时间了解华为云新鲜技术~
如果觉得《万字总结Keras深度学习中文文本分类》对你有帮助,请点赞、收藏,并留下你的观点哦!