Audio and video format introduction, encoding and decoding, audio and video synchronization

Supporting BiliBili video homepage ： Just now

Matching gitee Source warehouse ： Talent whale  

Introduction to audio and video formats

1. File format

• .mkv A container and file format , and AVI MP4 ASF similar matroska Official website agreement pdf Download address

• .mp4 A container format , full name MPEG-4 The first 14 part , agreement ISO/IEC 14496-14:2003 mpeg Official website ( slowly , Unable to download protocol document ) agreement pdf Download address After registration, you can download , No need to download points

Apple is original QuicTime Protocol address

• .mp3 An audio coding and compression format , Full name MPEG-1 or MPEG-2 Audio Layer III,ISO/IEC 13818-3 or ISO/IEC 11172-3 CSDN Download address You need to download points

• .jpg/.jpeg JFIF File format ,JPEG Compression method , agreement ISO/IEC 10918-1 agreement pdf Download address Download for free on the official website CSDN Download address You need to download points ( Consistent with the above file )

• .mjpg/.mjpeg Format RFC 2435 ITU T.81 Read online The download address is shown above .jpeg

2. Coding format

• mp3 MPEG-1 or MPEG-2 Audio Layer III ISO/IEC 13818-3 or ISO/IEC 11172-3 The protocol download address is shown above .mp3

• aac AAC It is standardized by the international organization for standardization and the International Electrotechnical Commission as MPEG-2 And MPEG-4 Part of the specification . .aac Use MPEG-2 Audio Transport Stream Containers MPEG-2 AAC and MPEG-4 AAC ISO/IEC 13818-7 or ISO/IEC 14496-3 (HE-AAC/aac+) 13818-7 Download address You need to download points 14496-3 Download address You need to download points

• h264 Full name MPEG-4 Part 10, Advanced Video Coding,ISO/IEC 14496-10:2004 agreement pdf Download address Download is slow , But it can be downloaded successfully

• mpeg2 ISO/IEC 13818-1 Audio and video synchronization and multiplexing 13818-1 Download address of Chinese version You need to download points ISO/IEC 13818-2 video 138181~3 Download address of Chinese version Need a download point 13818-2 English version download address You need to download points

• jpeg agreement ISO/IEC 10918-1 or ITU T.81 The download address is shown above .jpeg

3. Open source codec

• ffmpeg Official website Git Warehouse WiKi The compressed video data output becomes uncompressed color data , for example YUV420P,RGB wait ; The compressed audio data output becomes the uncompressed audio sample data , for example PCM data .

• bilibili Web player flv.js Github Warehouse

• bilibili Apple Android player ijkplayer Github Warehouse

• ffserver ffserver wiki Address

• ffplay ffplay Official website Use the FFmpeg libraries and the SDL library. MediaInfo View media file information Official website Source code

4. Reference material

• How to use ffplay Play the video ISO/IEC 14496-12：2015(ISO Format )  The official website is free to download
• ffplay read_thread Process of obtaining document information
• ffplay avformat_open_input Unpacking process
• fplay probe Probe file format flow
• mp4 File format details
• mkv File format details
• ffplay Video display process

5. ffplay Source code analysis

    //  Get file encapsulation format 
    main() --> stream_open() --> read_thread() --> avformat_open_input() --> init_input()
    --> av_probe_input_buffer2() --> av_probe_input_format2() --> av_probe_input_format3()
    --> for(){ score = fmt1->read_probe() } //  Select the packaging format , Such as mp4、mkv // AVInputFormat *fmt1
    // mp4 Of read_probe()  yes  mov_probe()

    //  decapsulation , Get basic flow information 
    main() --> stream_open() --> read_thread() --> avformat_open_input()
    --> s->iformat->read_header(s) //  Get basic flow information , Such as how much audio and video , What is the format of audio and video 
    // mp4 Of read_header()  yes  mov_read_header()

    //  Get the frame in the stream 
    --> ff_read_packet()
    --> s->iformat->read_packet(s, pkt)
    // mp4 Of read_packet()  yes  mov_read_packet()

    //  decode 
    AVCodec *codec
    video_thread() --> get_video_frame() --> decoder_decode_frame() --> packet_queue_get() --> avcodec_receive_frame()
    --> decode_receive_frame_internal() --> decode_simple_receive_frame() --> decode_simple_internal()
    --> avctx->codec->decode()
    ffmpeg/libavcodec/mpegaudiodec_float.c   decode_frame()
    ffmpeg/libavcodec/h264dec.c   h264_decode_frame()

    //  File operations 
    // The final read data interface  url.h URLProtocol  Read data to  AVIOContext
    main() --> stream_open() --> read_thread() --> avformat_alloc_context()
    --> avformat_get_context_defaults() --> io_open_default()
    --> ffio_open_whitelist() --> ffurl_open_whitelist()
    --> ffurl_connect() --> uc->prot->url_open()
    // The actual call is ：
    read_thread() --> avformat_open_input() --> init_input()
    --> s->io_open()
    // The read data is put into AVIOContext Medium buffer in 
    url_open            = file_open, //file.c
    ffurl_read()  ffio_geturlcontext()  AVIOContext in read_packet()

    // Data flow process 
    read_probe(AVProbeData.buf) <-- avio_read(AVIOContext.buf_ptr, 2k) <-- AVFormatContext.pb

    //h264 hevc transcoding 
    main() --> transcode() --> transcode_step() --> process_input() --> process_input_packet()
    --> do_streamcopy() --> output_packet() --> write_packet() --> av_interleaved_write_frame()
    --> write_packets_common() --> write_packet_common() --> write_packet() --> ff_raw_write_packet()
    --> avio_write() --> memcpy()
    //mpeg4 transcoding 
    main() --> transcode() --> transcode_step() --> process_input() --> process_input_packet()
    --> decode_video --> decode --> avcodec_send_packet --> avcodec_receive_frame
    --> av_bsf_send_packet
    main() --> transcode() --> transcode_step() --> reap_filters
    -->av_buffersink_get_frame_flags --> get_frame_internal
    --> ff_inlink_consume_frame --> ff_framequeue_take
    ( Take the frame out of the queue FFFrameQueue AVFilterLink AVFilterContex->inputs[0] OutputStream
    output_streams)
    //--> return_or_keep_frame --> av_frame_ref<frame.c>
    //--> do_video_out --> avcodec_send_frame --> do_encode
    //--> do_video_out--> avcodec_receive_packet<encode.c> --> av_packet_move_ref  After execution, you get mpeg4 Video streaming 

6. Additional web information

• mp4 Data flow structure 《[ Audio and video ]mp4 comparison m3u8 The reason for the slow loading of the first frame ？》
• mp4 Most of the box Detailed introduction 《mp4 The packaging format is different box Type explanation and IBP Frame calculation 》
• mkv Most of the element Detailed introduction 《【 Detailed explanation of multimedia packaging format 】---MKV》cluster/block 
• 《H.264 Simple analysis of coding format 》
• 《【H264/AVC Syntactic and semantic details 】( Two )：h264 Bitstream format and NALU Detailed explanation 1 》 
• 《H264 Bitstream sum Mp4 The structure, 》《H.264 Code stream structure analysis 》
• ！！！《H264—MP4 Format and in MP4 Extract in file H264 Of SPS、PPS And stream 》
• ！！！《MP4 Extract from H.264 stream 》
• ！！《【VS Development 】MP4 And H.264》
• 《 use ffmpeg from MP4 Extract from h.264 stream 》
• 《H264(NAL Introduction and I Frame judgment )》
• 《 distinguish mp4 In the format mdat Audio and video data in 》
• ！！！《MP4 mdat box analysis 》
• Test video download  《Mpeg MP4 Container》
• ！！！《mp4 all box Detailed explanation 》
• ！！！《H264 Detailed description of code stream 》
• 《H264 Code stream analysis 》
• AAC Detailed explanation of audio format
• 【 Detailed explanation of multimedia packaging format 】---MP4【4】
• mp4 File format analysis ffmpeg The command of ( Save stream )
• Matroska File parsing SimpleBlock
• FFmpeg The realization of simple player 5- Audio video synchronization
• Fmpeg The illustration ： Structure relation link graph
• FFMPEG】 About hard decoding and soft decoding
• FFmpeg Hardware decoding -HWAccelIntro

7. ffmpeg Source code analysis

    // Open file 
    main() --> ffmpeg_parse_options() --> open_files() -->
    --> open_input_file() --> avformat_open_input() --> AVFormatContext *ic
    // Processing flow 
    --> choose_decoder() --> avformat_find_stream_info()
    --> add_input_streams() --> transcode() --> transcode_step()
    --> transcode_step() --> transcode_from_filter()

8. Audio video synchronization

• About audio and video synchronization ffplay Analysis of the synchronization mode of
• How to realize audio and video synchronization (live555)
• Player technology sharing （3）： Synchronization of music and painting
• Principle and implementation of audio and video synchronization
• im —— Detailed audio and video synchronization technology
• Audio video synchronization ( Play ) principle
• WebRTC Audio and video synchronization details  
• Research on video decoding PTS(2)Mp4 Format ,AVI Format and MKV Format
• Research on video decoding PTS(1)TS Format and FLV Format
• TS The flow of PCR PTS DTS Timestamp calculation
• mkv block analysis
• How to be in H264 From the data PTS

mp4 mdat The sequence of video frames stored in is not stored in the display sequence , It is put in decoding order ！ But use elecardstreameyetools Tools to analyze h264 Stream file can be seen , Ordinary tools don't work ！ Then you can see ctts The value of is right , because ctts The value of cannot be negative , So the first one ctts Value is the overall offset value ！

9. dts/pts stay mp4 Access to documents

	mov.c
	//dts The calculation of 
	mov_read_packet() --> dts = timestamp; 	mov_read_trak() --> mov_build_index() --> timestamp = current_dts{in stsc box}; current_dts += stts_data.duration{in stts box} --> 
	// When adjustment is needed --> mov_fix_index() --> add_index_entry()

	//pts The calculation of 
	// No, b When the frame 
	pts = dts
	// Yes b When the frame 
	pts = dts + dts_shift + ctts_data.duration;
	dts_shift: mov_update_dts_shift() --> ctts_duration{ctts box Deposit dts pts Difference value }

10. Audio and video synchronization process

    /*  Audio playback process  */
    ffplay.c
    main() --> stream_open() --> read_thread() --> stream_component_open() --> audio_open() --> sdl_audio_callback()
    --> SDL_OpenAudioDevice() --> SDL_PauseAudioDevice()

    /*  Video playback process  */
    main() --> event_loop() --> refresh_loop_wait_event() --> video_refresh() --> video_display() --> video_image_display()

11. Time axis acquisition process of audio and video synchronization

	mov.c
		mp4 Unpack and get pkt->pts

	decode.c 
		ff_decode_frame_props()  frame->pts = pkt->pts;

	decoder_decode_frame() -->
		 Audio ：frame->pts // there frame->pts And the original mov.c in mp4 Of pkt->pts Agreement 
		 video ：frame->pts

	// obtain Frame in af->pts（float, In seconds ）
	audio_thread()  or video_thread()-->
		af->pts = (frame->pts == AV_NOPTS_VALUE) ? NAN : frame->pts * av_q2d(tb); // Convert from accumulated data amount to actual seconds of time axis 

	// obtain VideoState Medium audio_clock
	//=====================
	struct Frame --> double af->pts; double duration; // Measured as the current frame from 0 The relative number of seconds to start 
	audio_decode_frame() -->
	struct VideoState --> double audio_clock; // Measured from zero , The relative number of seconds when the current frame finishes playing （ floating-point , Accurate to milliseconds ）
		is->audio_clock = af->pts + (double) af->frame->nb_samples / af->frame->sample_rate; //l2493

	 Adjust the timeline time  // obtain Clock Medium pts
	//======================
	sdl_audio_callback() -->
		set_clock_at(&is->audclk, ...) --> Clock audclk; --> .pts --> get form "is->audio_clock"

12. Use ffmepg Medium libavcodec Codec library , Unpack and do it yourself , decode mpeg1 An official example of video streaming decode_video.cdecode_audio.c

13. Use sdl2 api, The official use case of playing audio and video by yourself （ Official examples are not very practical , Or the example on Baidu is useful ）

Playing Sounds

14 . some FFMPEG Transcoding command

• ffmpeg mp4 turn pcm, Play PCM ffmpeg -i 1.mp4 -vn -ar 44100 -ac 2 -f s16le out.pcm ffplay -ac 2 -ar 44100 -f s16le out.pcm

• ffmpeg mkv turn mp4 ffmpeg -i input.mkv -c:v copy -c:a copy output.mp4

• ffmpeg turn mpeg2 mpeg4 Pure video streaming

turn mpeg4 mpeg2: -vcodec mpeg4, -vcodec mpeg2video

Transcoding others ： -c:v copy out.m2v/m4v/h264/hevc/mpeg

【 Be careful ！】 To generate pure mpeg4 Video stream file , It needs to be done first ./configure --disable-muxer=ipod Again make, Cancel it ffmpeg Of ipod m4v Format , This format is actually mp4 then ffmpeg -i input.mp4 output.m4v direct ffmpeg -i in.mp4 -vcodec mpeg4 mpeg4.mp4 Generated mpeg4.mp4 take packet Copy directly into .m4v The file cannot be played While using ffmpeg -i mkv.mkv -i in.mp4 -c:v copy -c:a copy -map 0️0 -map 1️0 match_m4v.mp4 Generated mp4 File to directly copy the video stream into .m4v Can play after Generate m4v You need to specify an encoder when you file -vcodec mpeg4, direct copy Can't play

• Merge video and audio of two files:

ffmpeg -i s720-128_96-44100_2.mp4 -i a11025.aac -c:v copy -c:a aac -strict experimental -map 0️0 -map 1️0 output.mp4

ffmpeg -i 128_96_44100-movie.mp4 -i ShenTouNaiBa3.mp4 -c:v copy -c:a copy -map 0️0 -map 1️0 movie_test1.mp4

ffmpeg have not libxh264 and libmp3lame Compress movie -s 320*240 -ar 44100 -ac 1 or 2

• mp4 MDAT put in the head -movflags faststart
• Download SDL2-2.0.4.rar in csdn ffplay Use sdl2 Play audio and video

./configure;

make;

and make install or can install libsdl2-dev The ffplay can't play video:

make clean;

change -lW to -lSDL2 in EXTRALIBS of ffbuild/config.mak

Use media codes, add -L/usr/lib/i386-linux-gnu -lSDL2 in line 120 of scripts/gcclink.sh

• ffmpeg command Mp4 to yuv:

ffmpeg -i ~/video_files/s320.mp4 -s 640480 -pix_fmt yuv420p out.yuv ffplay -video_size 640480 -i out.yuv

• Intercept a certain length of audio and video from a certain time -ss 7 -t 1 // From 7 Seconds to intercept 1s