基于Paddle2.0实现视频分类模型GSM

· 前沿科技 · 202593

本文复现了CVPR 2020的GSM视频分类模型，用Paddle2.0在Diving48数据集实现。该模型通过分解3D卷积核实现时空域视频理解，参数量小。项目处理了数据集，将视频转帧、生成训练测试文件，经训练测试，最佳模型在测试集准确率达0.223。

项目背景

GSM是2020年CVPR上的一篇论文Gate-Shift Networks for Video Action Recognition中提出的视频分类模型。本项目即对其进行复现。

项目简介

本项目使用paddle2.0实现了视频动作分类GSM模型，并在Diving48数据集上进行了训练和验证。

Diving48数据集的划分是按数据集官方网站的划分方法进行划分的。

模型简介

Gate-Shift Networks for Video Action Recognition (GSM) 的核心是对进行时空域视频理解的3D卷积核进行分解的一种方法。该方法和已有方法的比较如下图所示。从图中可以看出，该模型跟其它网络模块相比具有参数量小的优点。

           

图1 GSM和其它网络结构比较图

模型细节

模型实现细节如下图所示。

           

图2 GSM实现细节图

GSM模块可以加在已有网络的不同部分，论文作者通过实验得出，将GSM模块加在下图所示的网络分支示例中具有较好的效果，即卷积核尺寸较小的分支中。

           

图3 GSM的plug-and-play示意图

论文原文：GSM: Gate-Shift Networks for Video Action Recognition

数据集介绍

本项目使用Diving48数据集进行训练和测试. 数据集官方网站为Diving48数据集

Diving48是跳水竞赛不同动作的细粒度视频数据集。该数据集包含48类不同的跳水标准动作，共18k左右个视频片段。由于不同的跳水动作可能在起跳、腾空和入水的三个不同阶段中的任一个阶段发生不同，需要模型学习视频中的长时间段间的特征关系。动作示例如下图所示：

           

图4 Diving48动作示意图

       

这48个类别中的每一个都是由三个不同的动作构成的，分别是起跳动作 (dive groups), 空中动作(somersaults and/or twists), 入水动作 (dive positions)。 下图展示了这48个类别的不同动作组成与互相间的关系：

           

图5 Diving48中不同动作间关系示意图

文件结构

文件名或文件夹名 功能
model 模型
best_model 训练好的模型
result 训练过程中的数据
dataset 数据文件夹
checkpoints_models 模型文件
pretrained_model 预训练模型
extract_frames_diving48.py 从视频中抽帧
process_dataset_diving.py 生成训练与测试的数据文件
dataset.py 自定义数据
datasets_video.py 读取数据
transforms.py 数据增强
train.py 模型训练
test.py 模型测试

模型实现及训练、验证

解压视频数据集

In [ ]

import osif not os.path.exists('dataset/Diving48'):
    os.makedirs('dataset/Diving48')

登录后复制
   

In [ ]

!tar -xzf data/data56350/Diving48_rgb.tar.gz -C dataset/Diving48

登录后复制
   

创建标签文件

仅在预测时使用

In [ ]

import jsonimport numpy as np

label_dir = {}
i = 0with open('data/data56350/Diving48_vocab.json', 'r') as f:
    vocab_list = json.load(f)    
for vocab in vocab_list:
    label_dir['_'.join(vocab)] = i
    i += 1np.save('work/label_dir.npy', label_dir)print(label_dir)

登录后复制
       

{'Back_15som_05Twis_FREE': 0, 'Back_15som_15Twis_FREE': 1, 'Back_15som_25Twis_FREE': 2, 'Back_15som_NoTwis_PIKE': 3, 'Back_15som_NoTwis_TUCK': 4, 'Back_25som_15Twis_PIKE': 5, 'Back_25som_25Twis_PIKE': 6, 'Back_25som_NoTwis_PIKE': 7, 'Back_25som_NoTwis_TUCK': 8, 'Back_2som_15Twis_FREE': 9, 'Back_2som_25Twis_FREE': 10, 'Back_35som_NoTwis_PIKE': 11, 'Back_35som_NoTwis_TUCK': 12, 'Back_3som_NoTwis_PIKE': 13, 'Back_3som_NoTwis_TUCK': 14, 'Back_Dive_NoTwis_PIKE': 15, 'Back_Dive_NoTwis_TUCK': 16, 'Forward_15som_1Twis_FREE': 17, 'Forward_15som_2Twis_FREE': 18, 'Forward_15som_NoTwis_PIKE': 19, 'Forward_1som_NoTwis_PIKE': 20, 'Forward_25som_1Twis_PIKE': 21, 'Forward_25som_2Twis_PIKE': 22, 'Forward_25som_3Twis_PIKE': 23, 'Forward_25som_NoTwis_PIKE': 24, 'Forward_25som_NoTwis_TUCK': 25, 'Forward_35som_NoTwis_PIKE': 26, 'Forward_35som_NoTwis_TUCK': 27, 'Forward_45som_NoTwis_TUCK': 28, 'Forward_Dive_NoTwis_PIKE': 29, 'Forward_Dive_NoTwis_STR': 30, 'Inward_15som_NoTwis_PIKE': 31, 'Inward_15som_NoTwis_TUCK': 32, 'Inward_25som_NoTwis_PIKE': 33, 'Inward_25som_NoTwis_TUCK': 34, 'Inward_35som_NoTwis_TUCK': 35, 'Inward_Dive_NoTwis_PIKE': 36, 'Reverse_15som_05Twis_FREE': 37, 'Reverse_15som_15Twis_FREE': 38, 'Reverse_15som_25Twis_FREE': 39, 'Reverse_15som_35Twis_FREE': 40, 'Reverse_15som_NoTwis_PIKE': 41, 'Reverse_25som_15Twis_PIKE': 42, 'Reverse_25som_NoTwis_PIKE': 43, 'Reverse_25som_NoTwis_TUCK': 44, 'Reverse_35som_NoTwis_TUCK': 45, 'Reverse_Dive_NoTwis_PIKE': 46, 'Reverse_Dive_NoTwis_TUCK': 47}

登录后复制
       

视频转为jpg格式图片

视频转换需要较长时间，不低于1个小时，建议转换到非data文件夹以便后续启动项目后可以直接使用。

In [10]

!python work/extract_frames_diving48.py

登录后复制
       

ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/nOlRwoxsDJ0_00555.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
  Duration: 00:00:03.18, start: 0.000000, bitrate: 838 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 830 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler[swscaler @ 0x2204380] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/nOlRwoxsDJ0_00555/frames%05d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=  191 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.18 bitrate=N/A    
video:703kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/OFxuiqI5G44_00406.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
  Duration: 00:00:03.77, start: 0.000000, bitrate: 1028 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1021 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler[swscaler @ 0x18f1400] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/OFxuiqI5G44_00406/frames%05d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=  226 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.76 bitrate=N/A    
video:862kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/LkluZoNfKu8_00121.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
  Duration: 00:00:09.40, start: 0.000000, bitrate: 816 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 814 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler[swscaler @ 0x1b7d3c0] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/LkluZoNfKu8_00121/frames%05d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=  235 fps=0.0 q=24.8 Lsize=N/A time=00:00:09.40 bitrate=N/A    
video:922kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/Q--70-ELqLw_00068.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
  Duration: 00:00:06.56, start: 0.000000, bitrate: 774 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 771 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler[swscaler @ 0x2108060] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/Q--70-ELqLw_00068/frames%05d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=  164 fps=0.0 q=24.8 Lsize=N/A time=00:00:06.56 bitrate=N/A    
video:606kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/_tigfCJFLZg_00205.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
  Duration: 00:00:04.00, start: 0.000000, bitrate: 1545 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1541 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler[swscaler @ 0x1d4e780] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/_tigfCJFLZg_00205/frames%05d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=  100 fps=0.0 q=24.8 Lsize=N/A time=00:00:04.00 bitrate=N/A    
video:479kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/sk8TafuB3lU_00696.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
  Duration: 00:00:01.27, start: 0.000000, bitrate: 1286 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1276 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler[swscaler @ 0x1f20b00] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/sk8TafuB3lU_00696/frames%05d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=   76 fps=0.0 q=24.8 Lsize=N/A time=00:00:01.26 bitrate=N/A    
video:199kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/xbQCwTHcGN8_00158.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
  Duration: 00:00:03.56, start: 0.000000, bitrate: 1580 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1576 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler[swscaler @ 0x1bbd3c0] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/xbQCwTHcGN8_00158/frames%05d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=   84 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.36 bitrate=N/A    
video:406kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exiting normally, received signal 2.
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/l6ZoMjD1UtA_00156.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
  Duration: 00:00:05.60, start: 0.000000, bitrate: 908 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 905 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler[swscaler @ 0x209c500] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/l6ZoMjD1UtA_00156/frames%05d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help

登录后复制
       

生成训练和测试的数据文本文件

In [ ]

!python work/process_dataset_diving.py

登录后复制
       

15026/15027

登录后复制
       

在之前训练基础上微调

In [ ]

!python work/train.py --epoch 5 --batch_size 32 --seg_num 16 --seg_len 1 --pretrain True --weights 'pretrained/inception_pretrained' \
--data_dir 'result/best_model.npz' --best_model_save 'best_model/best_model' --model_arc 'inception'

登录后复制
       

You are using Paddle compiled with TensorRT, but TensorRT dynamic library is not found. Ignore this if TensorRT is not needed.
W0119 22:07:28.029369 15355 device_context.cc:320] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0119 22:07:28.034546 15355 device_context.cc:330] device: 0, cuDNN Version: 7.6.
video number:15027
video number:1970
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/norm.py:634: UserWarning: When training, we now always track global mean and variance.
  "When training, we now always track global mean and variance.")
Loss at epoch 0 step 0: [3.657299], acc: [0.25]
Loss at epoch 0 step 200: [3.6588974], acc: [0.25]
Loss at epoch 0 step 400: [3.6196995], acc: [0.28125]
The best result is 0.22030456
Loss at epoch 1 step 0: [3.5962734], acc: [0.3125]
Loss at epoch 1 step 200: [3.5641773], acc: [0.34375]
Loss at epoch 1 step 400: [3.6840277], acc: [0.21875]
The best result is 0.22081219
Loss at epoch 2 step 0: [3.4678974], acc: [0.4375]
Loss at epoch 2 step 200: [3.4990504], acc: [0.40625]
Loss at epoch 2 step 400: [3.622469], acc: [0.28125]
The best result is 0.22335026
Loss at epoch 3 step 0: [3.6552052], acc: [0.25]
Loss at epoch 3 step 200: [3.5325003], acc: [0.375]
Loss at epoch 3 step 400: [3.6104255], acc: [0.28125]
Loss at epoch 4 step 0: [3.6843047], acc: [0.21875]
Loss at epoch 4 step 200: [3.6228292], acc: [0.28125]
Loss at epoch 4 step 400: [3.7475114], acc: [0.15625]
Final loss: [3.6923075]
time duration is  5594.123642206192

登录后复制
       

图示训练和验证过程

In [ ]

## 画图函数import numpy as npimport matplotlib.pyplot as pltdef draw_train_eval_process(title,step,train_reward, eval_reward, label_train_reward, label_eval_reward, color_train, color_eval, label_y):
    plt.title(title, fontsize=24)
    plt.xlabel("epochs", fontsize=20)
    plt.ylabel(label_y, fontsize=20)
    plt.plot(step, train_reward,color=color_train,label=label_train_reward) 
    plt.plot(step, eval_reward,color=color_eval,label=label_eval_reward) 
    plt.legend()
    plt.grid()
    plt.show()

登录后复制
   

In [ ]

result_gsm = np.load('result/best_model.npz')
gsm_train_rewards = result_gsm['all_train_rewards']
gsm_eval_rewards = result_gsm['all_test_rewards']
steps = [i for i in range(len(gsm_train_rewards))]
draw_train_eval_process("GSM training and eval",steps,gsm_train_rewards,gsm_eval_rewards, 
"GSM trainning accuracy", "GSM eval accuracy", "red","green", "gsm_train_eval_accuracy")

登录后复制
       

<Figure size 432x288 with 1 Axes>

登录后复制
               

用保存的最佳训练模型测试

In [ ]

!python work/test.py --weights 'best_model/best_model'

登录后复制
       

You are using Paddle compiled with TensorRT, but TensorRT dynamic library is not found. Ignore this if TensorRT is not needed.
[INFO: test.py:  132]: Namespace(batch_size=1, dataset='diving48', infer_topk=1, modality='RGB', model_arc='inception', seg_len=1, seg_num=16, weights='best_model/best_model')
W0120 00:08:33.052891 25454 device_context.cc:320] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0120 00:08:33.058135 25454 device_context.cc:330] device: 0, cuDNN Version: 7.6.
video number:1970
测试集准确率为:0.22335025668144226

登录后复制
       

以上就是基于Paddle2.0实现视频分类模型GSM的详细内容，更多请关注视野AI【www.syai.cn】。

相关推荐：

马斯克 SpaceXAI 首个编程智能体模型 Grok 4.5 发布：与 Cursor 联合训练，效率翻倍价格减半

感谢网友华南吴彦祖、对的时间点、huami、Alan_Tanis的线索投递！ 7月9日消息，SpaceXAI今日正式发布了其Grok4.5模型，这是该公司首个专门针对编程和智能体任务训练的模型。据介绍，该模型由SpaceXAI与Cursor联合完成训练，在提供前沿智能水平的同时，兼具领先的速度与成本...

造谣上海乐高乐园经营不善 相关违法人员被行拘

　　中新网上海7月7日电(记者李姝徵)“砸55亿美元打水漂？上海乐高乐园惨遭滑铁卢”“上海乐高乐园门可罗雀”，一粉丝数达65万的自媒体“大V”于多家网络平台发布不实信息。近日，上海金山警方破获一起抹黑上海乐高乐园的网络谣言案件，违法人员张某已被依法行政拘留。 　　今年3月，上海市公安局金山分局网安支...

纳米比亚总统访华 称赞中国将宏伟目标变为现实

视频：纳米比亚总统访华称赞中国将宏伟目标变为现实来源：新华社 　　纳米比亚共和国总统内通博·南迪－恩代特瓦7月6日在广州市参加纳米比亚-中国(广东)商务圆桌论坛时表示，她此访旨在从政治层面和经济层面进一步巩固两国及两国人民之间长期以来的友好关系。 　　恩代特瓦表示，她在抵达广东省后感受到，中国能为自...

柳州市动物园回应园区积水情况：大熊猫未受影响，状态良好

　　中新网7月8日电据“柳州动物园”微信公众号消息，柳州市动物园管理处8日发布说明称，7月8日，柳州市动物园关注到有网友发布题为“洪水中的柳州动物园：大熊猫蜀琳和明浜现状引关注，网友祈祷平安”的视频。该视频展示了园内部分区域存在积水画面，引发公众对大熊猫等动物安全的关切。 　　针对该视频反映的情况，...

“2026中国电影导演之夜”：《哪吒之魔童闹海》获年度影片

　　中新网山东烟台7月8日电由中国电影导演协会、中央广播电视总台央视频和山东省烟台市携手打造的“2026中国电影导演之夜”日前在山东省烟台市启幕。 　　活动以“中国电影·同心破浪”为主题，揭晓中国电影导演协会“2025年度荣誉”。 活动现场。主办方供图 　　导演蔡尚君凭借《日掛中天》获年度导演；饺子...

上汽名爵 MG 07 官宣 7 月 29 日预售：首搭 Momenta R7 世界模型，定位“年轻人的智能纯血轿跑”

感谢网友xiszero、热评钦定员、很宅很怕生的线索投递！ 7月6日消息，上汽MG今日宣布，上汽名爵MG07将在7月29日开启预售，新车定位“年轻人的智能纯血轿跑”。据介绍，新车尺寸为4886×1900×1485mm、轴距2825mm，前脸采用一体冲压蚌式机盖+曲面切割悬浮前大灯组设计，尾部采用掀背...

腾讯元宝接入混元 Hy3 模型，支持免费做文件

感谢网友艺术家李大丁、Autumn_Dream的线索投递！ 7月6日消息，腾讯今日发布混元Hy3模型，元宝也迎来再次升级。打开元宝深度思考或快速思考，选择Hy3模型就能立即体验，全部能力免费开放。据介绍，元宝现已支持查资料、写内容、做排版，然后直接把文件交给用户，还能输出可视化文档。不用会代码，一个...