本文是中国大学慕课《机器学习》的“集成学习”章节的课后代码。
课程地址:
/course/WZU-1464096179
课程完整代码:
/fengdu78/WZU-machine-learning-course
代码修改并注释:黄海广,haiguang2000@
importwarningswarnings.filterwarnings("ignore")importpandasaspdfromsklearn.model_selectionimporttrain_test_split
生成数据
生成12000行的数据,训练集和测试集按照3:1划分
fromsklearn.datasetsimportmake_hastie_10_2data,target=make_hastie_10_2()
X_train,X_test,y_train,y_test=train_test_split(data,target,random_state=123)X_train.shape,X_test.shape
((9000, 10), (3000, 10))
模型对比
对比六大模型,都使用默认参数,因为数据是
fromsklearn.linear_modelimportLogisticRegressionfromsklearn.ensembleimportRandomForestClassifierfromsklearn.ensembleimportAdaBoostClassifierfromsklearn.ensembleimportGradientBoostingClassifierfromxgboostimportXGBClassifierfromlightgbmimportLGBMClassifierfromsklearn.model_selectionimportcross_val_scoreimporttimeclf1=LogisticRegression()clf2=RandomForestClassifier()clf3=AdaBoostClassifier()clf4=GradientBoostingClassifier()clf5=XGBClassifier()clf6=LGBMClassifier()forclf,labelinzip([clf1,clf2,clf3,clf4,clf5,clf6],['LogisticRegression','RandomForest','AdaBoost','GBDT','XGBoost','LightGBM']):start=time.time()scores=cross_val_score(clf,X_train,y_train,scoring='accuracy',cv=5)end=time.time()running_time=end-startprint("Accuracy:%0.8f (+/-%0.2f),耗时%0.2f秒。模型名称[%s]"%(scores.mean(),scores.std(),running_time,label))
Accuracy: 0.47488889 (+/- 0.00),耗时0.04秒。模型名称[Logistic Regression]Accuracy: 0.88966667 (+/- 0.01),耗时16.34秒。模型名称[Random Forest]Accuracy: 0.88311111 (+/- 0.00),耗时3.39秒。模型名称[AdaBoost]Accuracy: 0.91388889 (+/- 0.01),耗时13.14秒。模型名称[GBDT]Accuracy: 0.92977778 (+/- 0.00),耗时3.60秒。模型名称[XGBoost]Accuracy: 0.93188889 (+/- 0.01),耗时0.58秒。模型名称[LightGBM]
对比了六大模型,可以看出,逻辑回归速度最快,但准确率最低。而LightGBM,速度快,而且准确率最高,所以,现在处理结构化数据的时候,大部分都是用LightGBM算法。
XGBoost的使用
1.原生XGBoost的使用
importxgboostasxgb#记录程序运行时间importtimestart_time=time.time()#xgb矩阵赋值xgb_train=xgb.DMatrix(X_train,y_train)xgb_test=xgb.DMatrix(X_test,label=y_test)##参数params={'booster':'gbtree',#'silent':1,#设置成1则没有运行信息输出,最好是设置为0.#'nthread':7,#cpu线程数默认最大'eta':0.007,#如同学习率'min_child_weight':3,#这个参数默认是1,是每个叶子里面h的和至少是多少,对正负样本不均衡时的0-1分类而言#,假设 h 在0.01 附近,min_child_weight 为 1 意味着叶子节点中最少需要包含 100个样本。#这个参数非常影响结果,控制叶子节点中二阶导的和的最小值,该参数值越小,越容易 overfitting。'max_depth':6,#构建树的深度,越大越容易过拟合'gamma':0.1,#树的叶子节点上作进一步分区所需的最小损失减少,越大越保守,一般0.1、0.2这样子。'subsample':0.7,#随机采样训练样本'colsample_bytree':0.7,#生成树时进行的列采样'lambda':2,#控制模型复杂度的权重值的L2正则化项参数,参数越大,模型越不容易过拟合。#'alpha':0,#L1正则项参数#'scale_pos_weight':1, #如果取值大于0的话,在类别样本不平衡的情况下有助于快速收敛。#'objective':'multi:softmax',#多分类的问题#'num_class':10,#类别数,多分类与multisoftmax并用'seed':1000,#随机种子#'eval_metric':'auc'}plst=list(params.items())num_rounds=500#迭代次数watchlist=[(xgb_train,'train'),(xgb_test,'val')]
#训练模型并保存#early_stopping_rounds当设置的迭代次数较大时,early_stopping_rounds可在一定的迭代次数内准确率没有提升就停止训练model=xgb.train(plst,xgb_train,num_rounds,watchlist,early_stopping_rounds=100,)#model.save_model('./model/xgb.model')#用于存储训练出的模型print("bestbest_ntree_limit",model.best_ntree_limit)y_pred=model.predict(xgb_test,ntree_limit=model.best_ntree_limit)print('error=%f'%(sum(1foriinrange(len(y_pred))ifint(y_pred[i]>0.5)!=y_test[i])/float(len(y_pred))))#输出运行时长cost_time=time.time()-start_timeprint("xgboostsuccess!",'\n',"costtime:",cost_time,"(s)......")
[0]train-rmse:1.11000val-rmse:1.10422[1]train-rmse:1.10734val-rmse:1.10182[2]train-rmse:1.10465val-rmse:1.09932[3]train-rmse:1.10207val-rmse:1.09694
……
[497]train-rmse:0.62135val-rmse:0.68680[498]train-rmse:0.62096val-rmse:0.68650[499]train-rmse:0.62056val-rmse:0.68624best best_ntree_limit 500error=0.826667xgboost success! cost time: 3.5742645263671875 (s)......
2.使用scikit-learn接口
会改变的函数名是:
eta -> learning_rate
lambda -> reg_lambda
alpha -> reg_alpha
fromsklearn.model_selectionimporttrain_test_splitfromsklearnimportmetricsfromxgboostimportXGBClassifierclf=XGBClassifier(# silent=0, #设置成1则没有运行信息输出,最好是设置为0.是否在运行升级时打印消息。#nthread=4,#cpu线程数默认最大learning_rate=0.3,#如同学习率min_child_weight=1,#这个参数默认是1,是每个叶子里面h的和至少是多少,对正负样本不均衡时的0-1分类而言#,假设 h 在0.01 附近,min_child_weight 为 1 意味着叶子节点中最少需要包含 100个样本。#这个参数非常影响结果,控制叶子节点中二阶导的和的最小值,该参数值越小,越容易 overfitting。max_depth=6,#构建树的深度,越大越容易过拟合gamma=0,#树的叶子节点上作进一步分区所需的最小损失减少,越大越保守,一般0.1、0.2这样子。subsample=1,#随机采样训练样本训练实例的子采样比max_delta_step=0,#最大增量步长,我们允许每个树的权重估计。colsample_bytree=1,#生成树时进行的列采样reg_lambda=1,#控制模型复杂度的权重值的L2正则化项参数,参数越大,模型越不容易过拟合。#reg_alpha=0,#L1正则项参数#scale_pos_weight=1, #如果取值大于0的话,在类别样本不平衡的情况下有助于快速收敛。平衡正负权重#objective='multi:softmax',#多分类的问题指定学习任务和相应的学习目标#num_class=10,#类别数,多分类与multisoftmax并用n_estimators=100,#树的个数seed=1000#随机种子#eval_metric='auc')clf.fit(X_train,y_train)y_true,y_pred=y_test,clf.predict(X_test)print("Accuracy:%.4g"%metrics.accuracy_score(y_true,y_pred))
Accuracy : 0.936
LIghtGBM的使用
1.原生接口
importlightgbmaslgbfromsklearn.metricsimportmean_squared_error#加载你的数据#print('Loaddata...')#df_train=pd.read_csv('../regression/regression.train',header=None,sep='\t')#df_test=pd.read_csv('../regression/regression.test',header=None,sep='\t')##y_train=df_train[0].values#y_test=df_test[0].values#X_train=df_train.drop(0,axis=1).values#X_test=df_test.drop(0,axis=1).values#创建成lgb特征的数据集格式lgb_train=lgb.Dataset(X_train,y_train)#将数据保存到LightGBM二进制文件将使加载更快lgb_eval=lgb.Dataset(X_test,y_test,reference=lgb_train)#创建验证数据#将参数写成字典下形式params={'task':'train','boosting_type':'gbdt',#设置提升类型'objective':'regression',#目标函数'metric':{'l2','auc'},#评估函数'num_leaves':31,#叶子节点数'learning_rate':0.05,#学习速率'feature_fraction':0.9,#建树的特征选择比例'bagging_fraction':0.8,#建树的样本采样比例'bagging_freq':5,#k意味着每k次迭代执行bagging'verbose':1#<0显示致命的,=0显示错误(警告),>0显示信息}print('Starttraining...')#训练cvandtraingbm=lgb.train(params,lgb_train,num_boost_round=500,valid_sets=lgb_eval,early_stopping_rounds=5)#训练数据需要参数列表和数据集print('Savemodel...')gbm.save_model('model.txt')#训练后保存模型到文件print('Startpredicting...')#预测数据集y_pred=gbm.predict(X_test,num_iteration=gbm.best_iteration)#如果在训练期间启用了早期停止,可以通过best_iteration方式从最佳迭代中获得预测#评估模型print('error=%f'%(sum(1foriinrange(len(y_pred))ifint(y_pred[i]>0.5)!=y_test[i])/float(len(y_pred))))
Start training...[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000448 seconds.You can set `force_col_wise=true` to remove the overhead.[LightGBM] [Info] Total Bins 2550[LightGBM] [Info] Number of data points in the train set: 9000, number of used features: 10[LightGBM] [Info] Start training from score 0.012000[1]valid_0's auc: 0.814399valid_0's l2: 0.965563Training until validation scores don't improve for 5 rounds[2]valid_0's auc: 0.84729valid_0's l2: 0.934647[3]valid_0's auc: 0.872805valid_0's l2: 0.905265[4]valid_0's auc: 0.884117valid_0's l2: 0.877875[5]valid_0's auc: 0.895115valid_0's l2: 0.852189
……
[191]valid_0's auc: 0.982783valid_0's l2: 0.319851[192]valid_0's auc: 0.982751valid_0's l2: 0.319971[193]valid_0's auc: 0.982685valid_0's l2: 0.33Early stopping, best iteration is:[188]valid_0's auc: 0.982794valid_0's l2: 0.319746Save model...Start predicting...error=0.664000
2.scikit-learn接口
fromsklearnimportmetricsfromlightgbmimportLGBMClassifierclf=LGBMClassifier(boosting_type='gbdt',#提升树的类型gbdt,dart,goss,rfnum_leaves=31,#树的最大叶子数,对比xgboost一般为2^(max_depth)max_depth=-1,#最大树的深度learning_rate=0.1,#学习率n_estimators=100,#拟合的树的棵树,相当于训练轮数subsample_for_bin=200000,objective=None,class_weight=None,min_split_gain=0.0,#最小分割增益min_child_weight=0.001,#分支结点的最小权重min_child_samples=20,subsample=1.0,#训练样本采样率行subsample_freq=0,#子样本频率colsample_bytree=1.0,#训练特征采样率列reg_alpha=0.0,#L1正则化系数reg_lambda=0.0,#L2正则化系数random_state=None,n_jobs=-1,silent=True,)clf.fit(X_train,y_train,eval_metric='auc')#设置验证集合verbose=False不打印过程clf.fit(X_train,y_train)y_true,y_pred=y_test,clf.predict(X_test)print("Accuracy:%.4g"%metrics.accuracy_score(y_true,y_pred))
Accuracy : 0.927
参考
1.https://xgboost.readthedocs.io/
2.https://lightgbm.readthedocs.io/
3./q383700092/article/details/53763328?locationNum=9&fps=1
往期精彩回顾适合初学者入门人工智能的路线及资料下载机器学习及深度学习笔记等资料打印机器学习在线手册深度学习笔记专辑《统计学习方法》的代码复现专辑AI基础下载黄海广老师《机器学习课程》视频课黄海广老师《机器学习课程》711页完整版课件
本站qq群955171419,加入微信群请扫码:
如果觉得《【机器学习】随机森林 GBDT XGBoost LightGBM等集成学习代码练习》对你有帮助,请点赞、收藏,并留下你的观点哦!