[audio and video technology] video quality evaluation MSU vqmt & Netflix vmaf

Video quality evaluation tool experimental documents

The experimental tools are two full reference tools ：

• MSU VQMT
• NetFlix vmaf

Because it is widely used psnr To do video quality evaluation index , This experiment mainly tests and compares , Therefore, only psnr As an indicator of two tools

difference ：

• VQMT It's a commercial software , Now the free version can support HD, FullHD, 4k, 8k Resolution and Support for 200+ video & image formats

  Download link ：MSU Video Quality Measurement Tool (VMAF, PSNR, VQM, SSIM, NIQE, etc) (compression.ru)

  • Free and Pro The main differences are as follows ：（pro Expensive Edition ,1 device 999$, You can choose ProDemo, It's also a free version , But its official statement does not guarantee that the data is correct ）

  Features & Components	Free	ProDemo	Pro	PREMIUM
  GPU metrics	−	+	+	+
  HDR	−	+	+	+
  Windows Console	−	+	+	+
  Linux Console	−	+	+	+
  MAC (iOS) support	−	Planned	Planned	Planned
  Python Interface	−	+	+	+
  Unlimited installations	+	+	−	+
  Automatical activation	N/A	N/A	−	+
  Free updates	+	+	1 year	1 year
  Correct results (not DEMO)	+	−	+	+
  Usage in companies	−	+	+	+

• vmaf Only support yuv、y4m Format , Several parameters are also provided for selection （psnr, psnr_hvs, ssim, ms-ssim, ciede）

  Download link ：Releases · Netflix/vmaf (github.com)

Operation method ：

• VQMT There is a graphical interface to directly open video files for comparison

  

  It also has a variety of parameters that can be used , The optional parameters are as follows ：（ You can see that there are Netflix VMAF）

 Input the original video and reference video URL Or you can start the analysis after opening the file , The results of the analysis will also be automatically saved in the new version , Just can't do batch processing , need Pro Only version of the command line can support batch processing .

• vmaf There is no graphical interface , You need to use command-line tools to operate

  Command line guide vmaf/README.md at master · Netflix/vmaf (github.com)

  • Instructions ：

    vmaf -r E:\common\tang3.yuv -d E:\common\tang4.yuv -w 960 -h 960 -p 420 -b 8 -o output.log
    

  • Related parameters ：

    Reference .y4m or .yuv (-r/--reference) is required
    
    Usage: vmaf [options]
    
    Supported options:
     --reference/-r $path:      path to reference .y4m or .yuv
     --distorted/-d $path:      path to distorted .y4m or .yuv
     --width/-w $unsigned:      width
     --height/-h $unsigned:     height
     --pixel_format/-p: $string pixel format (420/422/444)
     --bitdepth/-b $unsigned:   bitdepth (8/10/12/16)
     --model/-m $params:        model parameters, colon ":" delimited
                                `path=` path to model file
                                `version=` built-in model version
                                `name=` name used in log (optional)
     --output/-o $path:         output file
     --xml:                     write output file as XML (default)
     --json:                    write output file as JSON
     --csv:                     write output file as CSV
     --sub:                     write output file as subtitle
     --threads $unsigned:       number of threads to use
     --feature $string:         additional feature
     
     
    Additional Metrics  Other optional parameters 
    psnr, psnr_hvs, ssim, ms-ssim, ciede
    
    A number of addtional metrics are supported. Enable these metrics with the --feature flag.
    
    # psnr, psnr_hvs, ssim, ms-ssim, ciede
    --feature psnr \
    --feature psnr_hvs \
    --feature float_ssim \
    --feature float_ms_ssim
    --feature ciede
    --feature cambi
    
    
     --cpumask: $bitmask        restrict permitted CPU instruction sets
     --frame_cnt $unsigned:     maximum number of frames to process
     --subsample: $unsigned     compute scores only every N frames
     --quiet/-q:                disable FPS meter when run in a TTY
     --no_prediction/-n:        no prediction, extract features only
     --version/-v:              print version and exit
    

Output ：

• VQMT

  • Y psnr ( Signal-to-noise ratio Peak signal-to-noise ratio);

  Convert picture to YCbCr Format , Then just calculate Y Component, that is, the brightness component PSNR, The typical peak signal-to-noise ratio in image compression is 30 To 40dB Between , The higher, the better .

  

  • Output report ：

    Metric: PSNR
    Color component: Y
    Files: constrait20fps.mp4 compared to: native_video.mp4
    
    # native_video.mp4
    0	48.089218
    1	48.097214
    2	40.957611
    3	48.148323
    4	47.362122
    ......
    ......
    10837	18.317478
    10838	18.341671
    10839	18.339325
    10840	18.337685
    10841	18.108231
    10842	18.247126
    10843	18.247503
    10844	18.245304
    total psnr	34.472847
    mean	47.133518
    harmonic mean	46.136684
    min. val	16.455748
    max. val	61.386471
    min. frame	3623
    max. frame	389
    std dev	4.385028
    variance	19.228470
    

• vmaf

  • No added feature：

  <VMAF version="2.3.1">
    <params qualityWidth="960" qualityHeight="960" />
    <fyi fps="20.94" />
    <frames>
      <frame frameNum="0" integer_adm2="0.978963" integer_adm_scale0="0.998125" integer_adm_scale1="0.995882" integer_adm_scale2="0.990478" integer_adm_scale3="0.956561" integer_motion2="0.000000" integer_motion="0.000000" integer_vif_scale0="0.549783" integer_vif_scale1="0.619952" integer_vif_scale2="0.648450" integer_vif_scale3="0.675513" vmaf="59.246770" />
      <frame frameNum="1" integer_adm2="0.991671" integer_adm_scale0="0.998279" integer_adm_scale1="0.994878" integer_adm_scale2="0.994040" integer_adm_scale3="0.986643" integer_motion2="0.852844" integer_motion="6.023301" integer_vif_scale0="0.815954" integer_vif_scale1="0.879384" integer_vif_scale2="0.898762" integer_vif_scale3="0.913492" vmaf="85.293165" />
  ......
  ......
   </frames>
    <pooled_metrics>
      <metric name="integer_adm2" min="0.413357" max="0.999776" mean="0.928834" harmonic_mean="0.923530" />
      <metric name="integer_adm_scale0" min="0.720963" max="1.001498" mean="0.978396" harmonic_mean="0.977443" />
      <metric name="integer_adm_scale1" min="0.340115" max="1.003616" mean="0.940059" harmonic_mean="0.933984" />
      <metric name="integer_adm_scale2" min="0.277522" max="0.999417" mean="0.926124" harmonic_mean="0.916255" />
      <metric name="integer_adm_scale3" min="0.325116" max="0.999697" mean="0.911883" harmonic_mean="0.906875" />
      <metric name="integer_motion2" min="0.000000" max="35.799564" mean="6.326431" harmonic_mean="3.413652" />
      <metric name="integer_motion" min="0.000000" max="67.485893" mean="7.641753" harmonic_mean="3.884414" />
      <metric name="integer_vif_scale0" min="0.096378" max="0.978172" mean="0.592144" harmonic_mean="0.571733" />
      <metric name="integer_vif_scale1" min="0.105987" max="0.990578" mean="0.660542" harmonic_mean="0.639917" />
      <metric name="integer_vif_scale2" min="0.104521" max="0.992963" mean="0.687723" harmonic_mean="0.666811" />
      <metric name="integer_vif_scale3" min="0.098240" max="0.995127" mean="0.710630" harmonic_mean="0.689127" />
      <metric name="vmaf" min="0.000000" max="96.823158" mean="63.415244" harmonic_mean="38.980743" />
    </pooled_metrics>
    <aggregate_metrics />
  </VMAF>
  

  • add to psnr:

  <VMAF version="e1d466c">
    <params qualityWidth="576" qualityHeight="324" />
    <fyi fps="41.98" />
    <frames>
      <frame frameNum="0" integer_adm2="0.962084" integer_adm_scale0="0.946338" integer_adm_scale1="0.939006" integer_adm_scale2="0.957474" integer_adm_scale3="0.980887" integer_motion2="0.000000" integer_motion="0.000000" integer_vif_scale0="0.505712" integer_vif_scale1="0.879061" integer_vif_scale2="0.937873" integer_vif_scale3="0.964301" psnr_y="34.760779" psnr_cb="39.229987" psnr_cr="41.349703" vmaf="83.856285" />
      ...
    </frames>
    <pooled_metrics>
      <metric name="integer_adm2" min="0.921008" max="0.962084" mean="0.934506" harmonic_mean="0.934463" />
      ...
      <metric name="psnr_y" min="29.640688" max="34.760779" mean="30.755064" harmonic_mean="30.727905" />
      <metric name="psnr_cb" min="38.019979" max="39.229987" mean="38.449441" harmonic_mean="38.447866" />
      <metric name="psnr_cr" min="40.649266" max="41.353846" mean="40.991910" harmonic_mean="40.991083" />
      <metric name="vmaf" min="71.176557" max="87.181420" mean="76.668905" harmonic_mean="76.510006" />
    </pooled_metrics>
    <aggregate_metrics />
  

Conclusion ：

On the use VQMT be relative to vmaf The reason for having a graphical interface is simpler , Functionally VQMT Bring me two surprises , The first point It can use vmaf metric, It is equivalent to covering vmaf The function of , Another point It is planned to provide flow support and more parameters in later versions , But I don't know if I can free　 Used in version .

Stream support

Allows VQMT read and estimate the quality of streamed  video

For videos with original videos and reference videos , After testing, the free version can cover the requirements , And the new free version 14.1 Of VQMT It can also support high resolution and various pixel formats .

Comprehensive evaluation ,vmaf You can use the command line for batch operations , This is one of its advantages , however VQMT Easy to use and powerful , And for the current use needs , No batch operation is required , So relatively speaking VQMT It is currently the best choice for full reference video quality evaluation .