深度残差网络+自适应参数化ReLU激活函数（调参记录8）

续上一篇：

深度残差网络+自适应参数化ReLU激活函数（调参记录7）

/dangqing1988/article/details/105670981

本文将层数设置得很少，只有两个残差模块，测试Adaptively Parametric ReLU（APReLU）激活函数在Cifar10图像集上的效果。

Keras代码如下：

#!/usr/bin/env python3# -*- coding: utf-8 -*-"""Created on Tue Apr 14 04:17:45 Implemented using TensorFlow 1.10.0 and Keras 2.2.1Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Shaojiang Dong, Michael Pecht,Deep Residual Networks with Adaptively Parametric Rectifier Linear Units for Fault Diagnosis, IEEE Transactions on Industrial Electronics, , DOI: 10.1109/TIE..2972458 @author: Minghang Zhao"""from __future__ import print_functionimport kerasimport numpy as npfrom keras.datasets import cifar10from keras.layers import Dense, Conv2D, BatchNormalization, Activation, Minimumfrom keras.layers import AveragePooling2D, Input, GlobalAveragePooling2D, Concatenate, Reshapefrom keras.regularizers import l2from keras import backend as Kfrom keras.models import Modelfrom keras import optimizersfrom keras.preprocessing.image import ImageDataGeneratorfrom keras.callbacks import LearningRateSchedulerK.set_learning_phase(1)# The data, split between train and test sets(x_train, y_train), (x_test, y_test) = cifar10.load_data()# Noised datax_train = x_train.astype('float32') / 255.x_test = x_test.astype('float32') / 255.x_test = x_test-np.mean(x_train)x_train = x_train-np.mean(x_train)print('x_train shape:', x_train.shape)print(x_train.shape[0], 'train samples')print(x_test.shape[0], 'test samples')# convert class vectors to binary class matricesy_train = keras.utils.to_categorical(y_train, 10)y_test = keras.utils.to_categorical(y_test, 10)# Schedule the learning rate, multiply 0.1 every 300 epochesdef scheduler(epoch):if epoch % 300 == 0 and epoch != 0:lr = K.get_value(model.optimizer.lr)K.set_value(model.optimizer.lr, lr * 0.1)print("lr changed to {}".format(lr * 0.1))return K.get_value(model.optimizer.lr)# An adaptively parametric rectifier linear unit (APReLU)def aprelu(inputs):# get the number of channelschannels = inputs.get_shape().as_list()[-1]# get a zero feature mapzeros_input = keras.layers.subtract([inputs, inputs])# get a feature map with only positive featurespos_input = Activation('relu')(inputs)# get a feature map with only negative featuresneg_input = Minimum()([inputs,zeros_input])# define a network to obtain the scaling coefficientsscales_p = GlobalAveragePooling2D()(pos_input)scales_n = GlobalAveragePooling2D()(neg_input)scales = Concatenate()([scales_n, scales_p])scales = Dense(channels, activation='linear', kernel_initializer='he_normal', kernel_regularizer=l2(1e-4))(scales)scales = BatchNormalization()(scales)scales = Activation('relu')(scales)scales = Dense(channels, activation='linear', kernel_initializer='he_normal', kernel_regularizer=l2(1e-4))(scales)scales = BatchNormalization()(scales)scales = Activation('sigmoid')(scales)scales = Reshape((1,1,channels))(scales)# apply a paramtetric reluneg_part = keras.layers.multiply([scales, neg_input])return keras.layers.add([pos_input, neg_part])# Residual Blockdef residual_block(incoming, nb_blocks, out_channels, downsample=False,downsample_strides=2):residual = incomingin_channels = incoming.get_shape().as_list()[-1]for i in range(nb_blocks):identity = residualif not downsample:downsample_strides = 1residual = BatchNormalization()(residual)residual = aprelu(residual)residual = Conv2D(out_channels, 3, strides=(downsample_strides, downsample_strides), padding='same', kernel_initializer='he_normal', kernel_regularizer=l2(1e-4))(residual)residual = BatchNormalization()(residual)residual = aprelu(residual)residual = Conv2D(out_channels, 3, padding='same', kernel_initializer='he_normal', kernel_regularizer=l2(1e-4))(residual)# Downsamplingif downsample_strides > 1:identity = AveragePooling2D(pool_size=(1,1), strides=(2,2))(identity)# Zero_padding to match channelsif in_channels != out_channels:zeros_identity = keras.layers.subtract([identity, identity])identity = keras.layers.concatenate([identity, zeros_identity])in_channels = out_channelsresidual = keras.layers.add([residual, identity])return residual# define and train a modelinputs = Input(shape=(32, 32, 3))net = Conv2D(16, 3, padding='same', kernel_initializer='he_normal', kernel_regularizer=l2(1e-4))(inputs)# net = residual_block(net, 9, 16, downsample=False)net = residual_block(net, 1, 32, downsample=True)# net = residual_block(net, 8, 32, downsample=False)net = residual_block(net, 1, 64, downsample=True)# net = residual_block(net, 8, 64, downsample=False)net = BatchNormalization()(net)net = aprelu(net)net = GlobalAveragePooling2D()(net)outputs = Dense(10, activation='softmax', kernel_initializer='he_normal', kernel_regularizer=l2(1e-4))(net)model = Model(inputs=inputs, outputs=outputs)sgd = optimizers.SGD(lr=0.1, decay=0., momentum=0.9, nesterov=True)pile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])# data augmentationdatagen = ImageDataGenerator(# randomly rotate images in the range (deg 0 to 180)rotation_range=30,# randomly flip imageshorizontal_flip=True,# randomly shift images horizontallywidth_shift_range=0.125,# randomly shift images verticallyheight_shift_range=0.125)reduce_lr = LearningRateScheduler(scheduler)# fit the model on the batches generated by datagen.flow().model.fit_generator(datagen.flow(x_train, y_train, batch_size=100),validation_data=(x_test, y_test), epochs=1000, verbose=1, callbacks=[reduce_lr], workers=4)# get resultsK.set_learning_phase(0)DRSN_train_score = model.evaluate(x_train, y_train, batch_size=100, verbose=0)print('Train loss:', DRSN_train_score[0])print('Train accuracy:', DRSN_train_score[1])DRSN_test_score = model.evaluate(x_test, y_test, batch_size=100, verbose=0)print('Test loss:', DRSN_test_score[0])print('Test accuracy:', DRSN_test_score[1])

实验结果如下：

Epoch 755/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2417 - acc: 0.9453 - val_loss: 0.5321 - val_acc: 0.8719Epoch 756/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2436 - acc: 0.9450 - val_loss: 0.5294 - val_acc: 0.8714Epoch 757/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2463 - acc: 0.9435 - val_loss: 0.5351 - val_acc: 0.8710Epoch 758/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2461 - acc: 0.9444 - val_loss: 0.5356 - val_acc: 0.8708Epoch 759/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2440 - acc: 0.9449 - val_loss: 0.5355 - val_acc: 0.8674Epoch 760/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2431 - acc: 0.9447 - val_loss: 0.5329 - val_acc: 0.8711Epoch 761/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2469 - acc: 0.9440 - val_loss: 0.5294 - val_acc: 0.8712Epoch 762/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2420 - acc: 0.9445 - val_loss: 0.5296 - val_acc: 0.8725Epoch 763/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2411 - acc: 0.9469 - val_loss: 0.5353 - val_acc: 0.8712Epoch 764/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2426 - acc: 0.9453 - val_loss: 0.5363 - val_acc: 0.8715Epoch 765/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2415 - acc: 0.9449 - val_loss: 0.5322 - val_acc: 0.8718Epoch 766/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2392 - acc: 0.9450 - val_loss: 0.5321 - val_acc: 0.8696Epoch 767/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2396 - acc: 0.9463 - val_loss: 0.5353 - val_acc: 0.8699Epoch 768/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2457 - acc: 0.9436 - val_loss: 0.5314 - val_acc: 0.8713Epoch 769/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2442 - acc: 0.9440 - val_loss: 0.5327 - val_acc: 0.8740Epoch 770/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2449 - acc: 0.9445 - val_loss: 0.5336 - val_acc: 0.8706Epoch 771/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2408 - acc: 0.9458 - val_loss: 0.5359 - val_acc: 0.8706Epoch 772/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2400 - acc: 0.9454 - val_loss: 0.5362 - val_acc: 0.8690Epoch 773/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2409 - acc: 0.9455 - val_loss: 0.5343 - val_acc: 0.8688Epoch 774/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2390 - acc: 0.9464 - val_loss: 0.5321 - val_acc: 0.8690Epoch 775/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2438 - acc: 0.9439 - val_loss: 0.5363 - val_acc: 0.8700Epoch 776/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2424 - acc: 0.9441 - val_loss: 0.5359 - val_acc: 0.8691Epoch 777/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2398 - acc: 0.9448 - val_loss: 0.5354 - val_acc: 0.8689Epoch 778/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2420 - acc: 0.9450 - val_loss: 0.5385 - val_acc: 0.8681Epoch 779/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2391 - acc: 0.9459 - val_loss: 0.5320 - val_acc: 0.8698Epoch 780/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2425 - acc: 0.9443 - val_loss: 0.5363 - val_acc: 0.8683Epoch 781/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2381 - acc: 0.9457 - val_loss: 0.5345 - val_acc: 0.8680Epoch 782/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2374 - acc: 0.9470 - val_loss: 0.5301 - val_acc: 0.8710Epoch 783/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2389 - acc: 0.9460 - val_loss: 0.5334 - val_acc: 0.8696Epoch 784/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2386 - acc: 0.9473 - val_loss: 0.5286 - val_acc: 0.8691Epoch 785/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2387 - acc: 0.9447 - val_loss: 0.5362 - val_acc: 0.8690Epoch 786/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2386 - acc: 0.9461 - val_loss: 0.5345 - val_acc: 0.8690Epoch 787/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2358 - acc: 0.9464 - val_loss: 0.5344 - val_acc: 0.8709Epoch 788/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2374 - acc: 0.9463 - val_loss: 0.5322 - val_acc: 0.8716Epoch 789/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2388 - acc: 0.9449 - val_loss: 0.5267 - val_acc: 0.8744Epoch 790/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2349 - acc: 0.9471 - val_loss: 0.5347 - val_acc: 0.8706Epoch 791/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2391 - acc: 0.9458 - val_loss: 0.5336 - val_acc: 0.8693Epoch 792/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2385 - acc: 0.9447 - val_loss: 0.5387 - val_acc: 0.8687Epoch 793/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2356 - acc: 0.9471 - val_loss: 0.5374 - val_acc: 0.8686Epoch 794/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2398 - acc: 0.9455 - val_loss: 0.5375 - val_acc: 0.8678Epoch 795/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2365 - acc: 0.9452 - val_loss: 0.5302 - val_acc: 0.8710Epoch 796/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2358 - acc: 0.9469 - val_loss: 0.5374 - val_acc: 0.8711Epoch 797/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2379 - acc: 0.9450 - val_loss: 0.5335 - val_acc: 0.8686Epoch 798/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2369 - acc: 0.9454 - val_loss: 0.5340 - val_acc: 0.8687Epoch 799/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2337 - acc: 0.9470 - val_loss: 0.5360 - val_acc: 0.8698Epoch 800/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2412 - acc: 0.9432 - val_loss: 0.5353 - val_acc: 0.8697Epoch 801/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2357 - acc: 0.9456 - val_loss: 0.5362 - val_acc: 0.8689Epoch 802/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2347 - acc: 0.9464 - val_loss: 0.5371 - val_acc: 0.8698Epoch 803/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2311 - acc: 0.9474 - val_loss: 0.5328 - val_acc: 0.8683Epoch 804/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2396 - acc: 0.9449 - val_loss: 0.5358 - val_acc: 0.8699Epoch 805/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2380 - acc: 0.9459 - val_loss: 0.5351 - val_acc: 0.8689Epoch 806/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2351 - acc: 0.9452 - val_loss: 0.5362 - val_acc: 0.8693Epoch 807/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2350 - acc: 0.9463 - val_loss: 0.5281 - val_acc: 0.8711Epoch 808/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2333 - acc: 0.9467 - val_loss: 0.5347 - val_acc: 0.8703Epoch 809/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2333 - acc: 0.9463 - val_loss: 0.5338 - val_acc: 0.8709Epoch 810/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2330 - acc: 0.9478 - val_loss: 0.5351 - val_acc: 0.8704Epoch 811/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2374 - acc: 0.9439 - val_loss: 0.5400 - val_acc: 0.8696Epoch 812/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2321 - acc: 0.9467 - val_loss: 0.5361 - val_acc: 0.8709Epoch 813/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2349 - acc: 0.9457 - val_loss: 0.5307 - val_acc: 0.8706Epoch 814/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2361 - acc: 0.9457 - val_loss: 0.5368 - val_acc: 0.8686Epoch 815/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2351 - acc: 0.9459 - val_loss: 0.5344 - val_acc: 0.8692Epoch 816/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2362 - acc: 0.9464 - val_loss: 0.5297 - val_acc: 0.8693Epoch 817/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2326 - acc: 0.9471 - val_loss: 0.5344 - val_acc: 0.8688Epoch 818/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2353 - acc: 0.9448 - val_loss: 0.5418 - val_acc: 0.8698Epoch 819/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2361 - acc: 0.9439 - val_loss: 0.5353 - val_acc: 0.8705Epoch 820/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2320 - acc: 0.9468 - val_loss: 0.5411 - val_acc: 0.8701Epoch 821/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2311 - acc: 0.9466 - val_loss: 0.5360 - val_acc: 0.8683Epoch 822/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2298 - acc: 0.9464 - val_loss: 0.5369 - val_acc: 0.8722Epoch 823/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2360 - acc: 0.9450 - val_loss: 0.5409 - val_acc: 0.8657Epoch 824/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2319 - acc: 0.9471 - val_loss: 0.5340 - val_acc: 0.8689Epoch 825/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2307 - acc: 0.9483 - val_loss: 0.5338 - val_acc: 0.8695Epoch 826/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2337 - acc: 0.9465 - val_loss: 0.5364 - val_acc: 0.8676Epoch 827/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2348 - acc: 0.9454 - val_loss: 0.5367 - val_acc: 0.8676Epoch 828/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2334 - acc: 0.9453 - val_loss: 0.5284 - val_acc: 0.8699Epoch 829/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2334 - acc: 0.9459 - val_loss: 0.5325 - val_acc: 0.8689Epoch 830/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2332 - acc: 0.9462 - val_loss: 0.5346 - val_acc: 0.8701Epoch 831/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2355 - acc: 0.9438 - val_loss: 0.5329 - val_acc: 0.8687Epoch 832/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2325 - acc: 0.9459 - val_loss: 0.5325 - val_acc: 0.8694Epoch 833/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2299 - acc: 0.9480 - val_loss: 0.5328 - val_acc: 0.8683Epoch 834/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2328 - acc: 0.9462 - val_loss: 0.5345 - val_acc: 0.8686Epoch 835/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2310 - acc: 0.9468 - val_loss: 0.5396 - val_acc: 0.8684Epoch 836/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2300 - acc: 0.9477 - val_loss: 0.5312 - val_acc: 0.8680Epoch 837/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2316 - acc: 0.9470 - val_loss: 0.5363 - val_acc: 0.8696Epoch 838/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2287 - acc: 0.9466 - val_loss: 0.5400 - val_acc: 0.8691Epoch 839/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2324 - acc: 0.9456 - val_loss: 0.5354 - val_acc: 0.8691Epoch 840/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2321 - acc: 0.9456 - val_loss: 0.5269 - val_acc: 0.8701Epoch 841/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2328 - acc: 0.9440 - val_loss: 0.5319 - val_acc: 0.8713Epoch 842/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2304 - acc: 0.9454 - val_loss: 0.5295 - val_acc: 0.8697Epoch 843/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2287 - acc: 0.9459 - val_loss: 0.5329 - val_acc: 0.8720Epoch 844/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2315 - acc: 0.9452 - val_loss: 0.5334 - val_acc: 0.8709Epoch 845/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2313 - acc: 0.9461 - val_loss: 0.5357 - val_acc: 0.8691Epoch 846/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2283 - acc: 0.9470 - val_loss: 0.5292 - val_acc: 0.8743Epoch 847/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2308 - acc: 0.9447 - val_loss: 0.5303 - val_acc: 0.8713Epoch 848/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2320 - acc: 0.9458 - val_loss: 0.5297 - val_acc: 0.8675Epoch 849/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2261 - acc: 0.9473 - val_loss: 0.5278 - val_acc: 0.8712Epoch 850/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2289 - acc: 0.9466 - val_loss: 0.5329 - val_acc: 0.8710Epoch 851/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2291 - acc: 0.9474 - val_loss: 0.5331 - val_acc: 0.8715Epoch 852/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2312 - acc: 0.9463 - val_loss: 0.5269 - val_acc: 0.8727Epoch 853/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2319 - acc: 0.9458 - val_loss: 0.5287 - val_acc: 0.8701Epoch 854/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2291 - acc: 0.9461 - val_loss: 0.5300 - val_acc: 0.8731Epoch 855/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2294 - acc: 0.9469 - val_loss: 0.5342 - val_acc: 0.8703Epoch 856/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2305 - acc: 0.9456 - val_loss: 0.5324 - val_acc: 0.8703Epoch 857/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2318 - acc: 0.9448 - val_loss: 0.5338 - val_acc: 0.8677Epoch 858/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2286 - acc: 0.9466 - val_loss: 0.5299 - val_acc: 0.8688Epoch 859/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2302 - acc: 0.9467 - val_loss: 0.5329 - val_acc: 0.8686Epoch 860/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2305 - acc: 0.9457 - val_loss: 0.5350 - val_acc: 0.8687Epoch 861/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2284 - acc: 0.9457 - val_loss: 0.5376 - val_acc: 0.8689Epoch 862/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2302 - acc: 0.9460 - val_loss: 0.5317 - val_acc: 0.8705Epoch 863/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2276 - acc: 0.9462 - val_loss: 0.5327 - val_acc: 0.8694Epoch 864/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2273 - acc: 0.9471 - val_loss: 0.5338 - val_acc: 0.8706Epoch 865/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2278 - acc: 0.9462 - val_loss: 0.5311 - val_acc: 0.8703Epoch 866/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2277 - acc: 0.9455 - val_loss: 0.5312 - val_acc: 0.8727Epoch 867/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2282 - acc: 0.9467 - val_loss: 0.5315 - val_acc: 0.8707Epoch 868/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2264 - acc: 0.9458 - val_loss: 0.5371 - val_acc: 0.8694Epoch 869/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2266 - acc: 0.9470 - val_loss: 0.5354 - val_acc: 0.8684Epoch 870/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2269 - acc: 0.9473 - val_loss: 0.5325 - val_acc: 0.8674Epoch 871/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2269 - acc: 0.9481 - val_loss: 0.5349 - val_acc: 0.8706 ETA: 12s - loss: 0.2107 - acc: 0.9536Epoch 872/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2230 - acc: 0.9476 - val_loss: 0.5338 - val_acc: 0.8709Epoch 873/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2291 - acc: 0.9455 - val_loss: 0.5291 - val_acc: 0.8692Epoch 874/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2271 - acc: 0.9459 - val_loss: 0.5248 - val_acc: 0.8708Epoch 875/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2307 - acc: 0.9456 - val_loss: 0.5285 - val_acc: 0.8705Epoch 876/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2288 - acc: 0.9452 - val_loss: 0.5331 - val_acc: 0.8694Epoch 877/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2296 - acc: 0.9455 - val_loss: 0.5343 - val_acc: 0.8671Epoch 878/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2263 - acc: 0.9469 - val_loss: 0.5348 - val_acc: 0.8703Epoch 879/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2274 - acc: 0.9462 - val_loss: 0.5333 - val_acc: 0.8678Epoch 880/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2234 - acc: 0.9483 - val_loss: 0.5364 - val_acc: 0.8684Epoch 881/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2257 - acc: 0.9469 - val_loss: 0.5303 - val_acc: 0.8693Epoch 882/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2256 - acc: 0.9474 - val_loss: 0.5301 - val_acc: 0.8691Epoch 883/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2280 - acc: 0.9457 - val_loss: 0.5285 - val_acc: 0.8670Epoch 884/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2256 - acc: 0.9479 - val_loss: 0.5231 - val_acc: 0.8709Epoch 885/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2270 - acc: 0.9444 - val_loss: 0.5311 - val_acc: 0.8693Epoch 886/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2281 - acc: 0.9458 - val_loss: 0.5255 - val_acc: 0.8707Epoch 887/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2280 - acc: 0.9457 - val_loss: 0.5289 - val_acc: 0.8727Epoch 888/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2240 - acc: 0.9470 - val_loss: 0.5342 - val_acc: 0.8698Epoch 889/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2264 - acc: 0.9464 - val_loss: 0.5375 - val_acc: 0.8672Epoch 890/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2261 - acc: 0.9469 - val_loss: 0.5372 - val_acc: 0.8686Epoch 891/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2214 - acc: 0.9472 - val_loss: 0.5297 - val_acc: 0.8692Epoch 892/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2239 - acc: 0.9473 - val_loss: 0.5325 - val_acc: 0.8705Epoch 893/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2236 - acc: 0.9470 - val_loss: 0.5261 - val_acc: 0.8673Epoch 894/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2269 - acc: 0.9465 - val_loss: 0.5368 - val_acc: 0.8674Epoch 895/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2242 - acc: 0.9469 - val_loss: 0.5361 - val_acc: 0.8684Epoch 896/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2241 - acc: 0.9470 - val_loss: 0.5322 - val_acc: 0.8689Epoch 897/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2239 - acc: 0.9466 - val_loss: 0.5413 - val_acc: 0.8645Epoch 898/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2239 - acc: 0.9467 - val_loss: 0.5379 - val_acc: 0.8674Epoch 899/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2287 - acc: 0.9460 - val_loss: 0.5365 - val_acc: 0.8663Epoch 900/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2216 - acc: 0.9482 - val_loss: 0.5382 - val_acc: 0.8695Epoch 901/1000lr changed to 9.999999310821295e-05500/500 [==============================] - 17s 34ms/step - loss: 0.2176 - acc: 0.9493 - val_loss: 0.5316 - val_acc: 0.8716Epoch 902/1000500/500 [==============================] - 17s 35ms/step - loss: 0.2117 - acc: 0.9507 - val_loss: 0.5298 - val_acc: 0.8711Epoch 903/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2125 - acc: 0.9514 - val_loss: 0.5297 - val_acc: 0.8705Epoch 904/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2120 - acc: 0.9520 - val_loss: 0.5282 - val_acc: 0.8713Epoch 905/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2125 - acc: 0.9510 - val_loss: 0.5284 - val_acc: 0.8710Epoch 906/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2103 - acc: 0.9520 - val_loss: 0.5281 - val_acc: 0.8719Epoch 907/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2075 - acc: 0.9528 - val_loss: 0.5279 - val_acc: 0.8719Epoch 908/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2074 - acc: 0.9526 - val_loss: 0.5284 - val_acc: 0.8713Epoch 909/1000500/500 [==============================] - 17s 35ms/step - loss: 0.2070 - acc: 0.9530 - val_loss: 0.5271 - val_acc: 0.8705Epoch 910/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2093 - acc: 0.9532 - val_loss: 0.5271 - val_acc: 0.8712Epoch 911/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2071 - acc: 0.9528 - val_loss: 0.5277 - val_acc: 0.8705Epoch 912/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2052 - acc: 0.9541 - val_loss: 0.5279 - val_acc: 0.8701Epoch 913/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2076 - acc: 0.9533 - val_loss: 0.5279 - val_acc: 0.8696Epoch 914/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2070 - acc: 0.9525 - val_loss: 0.5284 - val_acc: 0.8690Epoch 915/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2040 - acc: 0.9542 - val_loss: 0.5286 - val_acc: 0.8694Epoch 916/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2035 - acc: 0.9550 - val_loss: 0.5276 - val_acc: 0.8702Epoch 917/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2032 - acc: 0.9549 - val_loss: 0.5278 - val_acc: 0.8701Epoch 918/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2041 - acc: 0.9540 - val_loss: 0.5274 - val_acc: 0.8707Epoch 919/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2039 - acc: 0.9541 - val_loss: 0.5278 - val_acc: 0.8706Epoch 920/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2024 - acc: 0.9552 - val_loss: 0.5284 - val_acc: 0.8716Epoch 921/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2046 - acc: 0.9534 - val_loss: 0.5271 - val_acc: 0.8714Epoch 922/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2062 - acc: 0.9539 - val_loss: 0.5275 - val_acc: 0.8709Epoch 923/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2046 - acc: 0.9545 - val_loss: 0.5276 - val_acc: 0.8708Epoch 924/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2058 - acc: 0.9526 - val_loss: 0.5259 - val_acc: 0.8702Epoch 925/1000500/500 [==============================] - 17s 35ms/step - loss: 0.2047 - acc: 0.9537 - val_loss: 0.5269 - val_acc: 0.8700Epoch 926/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2053 - acc: 0.9533 - val_loss: 0.5270 - val_acc: 0.8712 ETA: 12s - loss: 0.2096 - acc: 0.9512Epoch 927/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2059 - acc: 0.9532 - val_loss: 0.5264 - val_acc: 0.8712Epoch 928/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2047 - acc: 0.9544 - val_loss: 0.5287 - val_acc: 0.8694Epoch 929/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2071 - acc: 0.9531 - val_loss: 0.5276 - val_acc: 0.8695Epoch 930/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2050 - acc: 0.9544 - val_loss: 0.5281 - val_acc: 0.8697Epoch 931/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2042 - acc: 0.9538 - val_loss: 0.5275 - val_acc: 0.8688Epoch 932/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9551 - val_loss: 0.5274 - val_acc: 0.8700Epoch 933/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2035 - acc: 0.9546 - val_loss: 0.5293 - val_acc: 0.8702Epoch 934/1000500/500 [==============================] - 17s 34ms/step - loss: 0.2042 - acc: 0.9547 - val_loss: 0.5289 - val_acc: 0.8695Epoch 935/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2026 - acc: 0.9550 - val_loss: 0.5290 - val_acc: 0.8707Epoch 936/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2031 - acc: 0.9545 - val_loss: 0.5307 - val_acc: 0.8696Epoch 937/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2034 - acc: 0.9549 - val_loss: 0.5291 - val_acc: 0.8699Epoch 938/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2042 - acc: 0.9534 - val_loss: 0.5273 - val_acc: 0.8713Epoch 939/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2067 - acc: 0.9536 - val_loss: 0.5276 - val_acc: 0.8707Epoch 940/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2035 - acc: 0.9544 - val_loss: 0.5285 - val_acc: 0.8706Epoch 941/1000500/500 [==============================] - 17s 34ms/step - loss: 0. - acc: 0.9553 - val_loss: 0.5285 - val_acc: 0.8704Epoch 942/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2026 - acc: 0.9549 - val_loss: 0.5278 - val_acc: 0.8715Epoch 943/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9557 - val_loss: 0.5292 - val_acc: 0.8713Epoch 944/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2029 - acc: 0.9543 - val_loss: 0.5300 - val_acc: 0.8696Epoch 945/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9548 - val_loss: 0.5295 - val_acc: 0.8702Epoch 946/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9553 - val_loss: 0.5305 - val_acc: 0.8683Epoch 947/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9549 - val_loss: 0.5295 - val_acc: 0.8685Epoch 948/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9557 - val_loss: 0.5292 - val_acc: 0.8694 ETA: 2s - loss: 0.1982 - acc: 0.9567Epoch 949/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2058 - acc: 0.9534 - val_loss: 0.5290 - val_acc: 0.8700Epoch 950/1000500/500 [==============================] - 17s 34ms/step - loss: 0. - acc: 0.9551 - val_loss: 0.5295 - val_acc: 0.8711Epoch 951/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9560 - val_loss: 0.5285 - val_acc: 0.8704Epoch 952/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9559 - val_loss: 0.5291 - val_acc: 0.8695Epoch 953/1000500/500 [==============================] - 16s 32ms/step - loss: 0.2033 - acc: 0.9546 - val_loss: 0.5310 - val_acc: 0.8703Epoch 954/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9540 - val_loss: 0.5318 - val_acc: 0.8704Epoch 955/1000500/500 [==============================] - 17s 33ms/step - loss: 0.1997 - acc: 0.9558 - val_loss: 0.5312 - val_acc: 0.8700Epoch 956/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9555 - val_loss: 0.5297 - val_acc: 0.8701Epoch 957/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2035 - acc: 0.9531 - val_loss: 0.5308 - val_acc: 0.8697Epoch 958/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9547 - val_loss: 0.5324 - val_acc: 0.8699Epoch 959/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9557 - val_loss: 0.5318 - val_acc: 0.8695Epoch 960/1000500/500 [==============================] - 16s 33ms/step - loss: 0.1996 - acc: 0.9558 - val_loss: 0.5311 - val_acc: 0.8690Epoch 961/1000500/500 [==============================] - 16s 32ms/step - loss: 0.1991 - acc: 0.9564 - val_loss: 0.5318 - val_acc: 0.8686Epoch 962/1000500/500 [==============================] - 16s 32ms/step - loss: 0. - acc: 0.9544 - val_loss: 0.5323 - val_acc: 0.8681Epoch 963/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9548 - val_loss: 0.5310 - val_acc: 0.8704Epoch 964/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2026 - acc: 0.9549 - val_loss: 0.5317 - val_acc: 0.8702Epoch 965/1000500/500 [==============================] - 17s 33ms/step - loss: 0.1990 - acc: 0.9565 - val_loss: 0.5312 - val_acc: 0.8708Epoch 966/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9535 - val_loss: 0.5303 - val_acc: 0.8706 ETA: 7s - loss: 0.2001 - acc: 0.9555Epoch 967/1000500/500 [==============================] - 17s 33ms/step - loss: 0.1996 - acc: 0.9568 - val_loss: 0.5307 - val_acc: 0.8700Epoch 968/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9556 - val_loss: 0.5313 - val_acc: 0.8699Epoch 969/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9553 - val_loss: 0.5307 - val_acc: 0.8693Epoch 970/1000500/500 [==============================] - 16s 32ms/step - loss: 0.1988 - acc: 0.9550 - val_loss: 0.5330 - val_acc: 0.8709Epoch 971/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2025 - acc: 0.9548 - val_loss: 0.5326 - val_acc: 0.8710Epoch 972/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2042 - acc: 0.9541 - val_loss: 0.5333 - val_acc: 0.8709Epoch 973/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9555 - val_loss: 0.5328 - val_acc: 0.8697Epoch 974/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9550 - val_loss: 0.5328 - val_acc: 0.8705Epoch 975/1000500/500 [==============================] - 17s 33ms/step - loss: 0.2047 - acc: 0.9535 - val_loss: 0.5337 - val_acc: 0.8700Epoch 976/1000500/500 [==============================] - 16s 33ms/step - loss: 0.1987 - acc: 0.9568 - val_loss: 0.5329 - val_acc: 0.8706Epoch 977/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9551 - val_loss: 0.5332 - val_acc: 0.8698Epoch 978/1000500/500 [==============================] - 16s 33ms/step - loss: 0.1967 - acc: 0.9574 - val_loss: 0.5329 - val_acc: 0.8700Epoch 979/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9550 - val_loss: 0.5333 - val_acc: 0.8692Epoch 980/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9546 - val_loss: 0.5333 - val_acc: 0.8690Epoch 981/1000500/500 [==============================] - 17s 33ms/step - loss: 0.1982 - acc: 0.9564 - val_loss: 0.5333 - val_acc: 0.8705Epoch 982/1000500/500 [==============================] - 17s 34ms/step - loss: 0.1995 - acc: 0.9567 - val_loss: 0.5330 - val_acc: 0.8702Epoch 983/1000500/500 [==============================] - 17s 33ms/step - loss: 0.1985 - acc: 0.9559 - val_loss: 0.5334 - val_acc: 0.8704Epoch 984/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9552 - val_loss: 0.5334 - val_acc: 0.8700Epoch 985/1000500/500 [==============================] - 16s 32ms/step - loss: 0. - acc: 0.9538 - val_loss: 0.5323 - val_acc: 0.8704Epoch 986/1000500/500 [==============================] - 16s 32ms/step - loss: 0.1995 - acc: 0.9563 - val_loss: 0.5331 - val_acc: 0.8694Epoch 987/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9560 - val_loss: 0.5336 - val_acc: 0.8694Epoch 988/1000500/500 [==============================] - 16s 33ms/step - loss: 0.1985 - acc: 0.9562 - val_loss: 0.5339 - val_acc: 0.8684Epoch 989/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9540 - val_loss: 0.5339 - val_acc: 0.8688Epoch 990/1000500/500 [==============================] - 17s 33ms/step - loss: 0.1997 - acc: 0.9562 - val_loss: 0.5347 - val_acc: 0.8686Epoch 991/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9547 - val_loss: 0.5339 - val_acc: 0.8699Epoch 992/1000500/500 [==============================] - 16s 32ms/step - loss: 0.1998 - acc: 0.9556 - val_loss: 0.5326 - val_acc: 0.8692Epoch 993/1000500/500 [==============================] - 16s 32ms/step - loss: 0. - acc: 0.9556 - val_loss: 0.5336 - val_acc: 0.8706Epoch 994/1000500/500 [==============================] - 16s 33ms/step - loss: 0. - acc: 0.9546 - val_loss: 0.5334 - val_acc: 0.8702Epoch 995/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2054 - acc: 0.9538 - val_loss: 0.5338 - val_acc: 0.8690Epoch 996/1000500/500 [==============================] - 17s 33ms/step - loss: 0.1983 - acc: 0.9563 - val_loss: 0.5346 - val_acc: 0.8709Epoch 997/1000500/500 [==============================] - 17s 33ms/step - loss: 0.1989 - acc: 0.9559 - val_loss: 0.5352 - val_acc: 0.8685Epoch 998/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9557 - val_loss: 0.5339 - val_acc: 0.8681Epoch 999/1000500/500 [==============================] - 17s 33ms/step - loss: 0. - acc: 0.9537 - val_loss: 0.5345 - val_acc: 0.8694Epoch 1000/1000500/500 [==============================] - 16s 33ms/step - loss: 0.2002 - acc: 0.9548 - val_loss: 0.5354 - val_acc: 0.8686Train loss: 0.16600286397337913Train accuracy: 0.968400007367134Test loss: 0.5354112640023232Test accuracy: 0.8685999995470047

训练集准确率比测试集高了接近10%，看来小网络也会过拟合。

到目前为止，测试准确率最高的，还是调参记录6里的93.23%。

/dangqing1988/article/details/105628681

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Shaojiang Dong, Michael Pecht, Deep Residual Networks with Adaptively Parametric Rectifier Linear Units for Fault Diagnosis, IEEE Transactions on Industrial Electronics, , DOI: 10.1109/TIE..2972458

/document/8998530

如果觉得《深度残差网络+自适应参数化ReLU激活函数（调参记录8）》对你有帮助，请点赞、收藏，并留下你的观点哦！