Build a minimalist gb28181 gatekeeper and gateway server, establish AI reasoning and 3D service scenarios, and then open source code (I)

1 Purpose

     Open source is not the use of existing open source systems , Instead, write your own system to open source .

     Why build minimalism GB service , In our company, the company has used nodejs ,go ,c# ,c++ And so on GB service , Many of these can be built , It's not like what many people say , must do tcp Can pass ,udp The same can be done , however GB Easy service , Can't the gateway be built very simply , Meet our simple needs ？

     The first is for video analysis , Use GB28181 After accessing the video , Followed by AI Services to meet simple needs

2、 analysis ps flow

The simplest analysis ps The flow method uses ffmpeg Directly analyze the outflow , The streaming media server does it by itself , Made a basic streaming media server , Open source in gitee, It's the first edition , It needs to be optimized by the readers themselves , The address is below
https://gitee.com/guanzhi0319/mediaserver
Setting up a server is an option , In addition, we can have a simpler way in the gateway
analysis ps Flow method

struct buffer_data {
    
    uint8_t* ptr;
    size_t size;
};
static int read_packet(void* opaque, uint8_t* buf, int buf_size)
{
    
    struct buffer_data* bd = (struct buffer_data*)opaque;
    buf_size = FFMIN(buf_size, bd->size);

    if (!buf_size)
        return AVERROR_EOF;
    printf("ptr:%p size:%zu\n", bd->ptr, bd->size);

    memcpy(buf, bd->ptr, buf_size);
    bd->ptr += buf_size;
    bd->size -= buf_size;
    return buf_size;
}
int main(int argc, char* argv[])
{
    
    AVFormatContext* fmt_ctx = NULL;
    AVIOContext* avio_ctx = NULL;
    uint8_t* buffer = NULL, * avio_ctx_buffer = NULL;
    size_t buffer_size, avio_ctx_buffer_size = 4096;
    char* input_filename = NULL;
    int ret = 0;
    struct buffer_data bd = {
     0 };

    input_filename =(char*)"D:/ps file/ps0.264";

    /* slurp file content into buffer */
    ret = av_file_map(input_filename, &buffer, &buffer_size, 0, NULL);
    if (ret < 0)
        return -1;

    /* fill opaque structure used by the AVIOContext read callback */
    bd.ptr = buffer;
    bd.size = buffer_size;
    if (!(fmt_ctx = avformat_alloc_context())) {
    
        ret = AVERROR(ENOMEM);
        return -1;
    }

    avio_ctx_buffer = (uint8_t*)av_malloc(avio_ctx_buffer_size);
    if (!avio_ctx_buffer) {
    
        ret = AVERROR(ENOMEM);
        return -1;
    }

    avio_ctx = avio_alloc_context(avio_ctx_buffer, avio_ctx_buffer_size,
        0, &bd, &read_packet, NULL, NULL);
    if (!avio_ctx) {
    
        ret = AVERROR(ENOMEM);
        return -1;
    }
    fmt_ctx->pb = avio_ctx;

    ret = avformat_open_input(&fmt_ctx, NULL, NULL, NULL);
    if (ret < 0) {
    
        std::cout << "Could not open input" << std::endl;
        return -1;
    }
    ret = avformat_find_stream_info(fmt_ctx, NULL);
    if (ret < 0) {
    
        std::cout << "Could not find stream information" << std::endl;
        return -1;
    }
    av_dump_format(fmt_ctx, 0, input_filename, 0);

    AVPacket pkt;
     av_read_frame(fmt_ctx, &pkt);

    avformat_close_input(&fmt_ctx);
    /* note: the internal buffer could have changed, and be != avio_ctx_buffer */
    if (avio_ctx) {
    
        av_freep(&avio_ctx->buffer);
        av_freep(&avio_ctx);
    }
    av_file_unmap(buffer, buffer_size);
    if (ret < 0) {
    
        std::cout << "Error occurred!" << std::endl;
        return 1;
    }
    return 0;
}

Enter a... In the above code ps file , It can output one frame , If ps Flow is standard , No problem .

2 Build a simple gateway

     Let's call this simple gateway, It's actually mediaserver, The above open source address already exists , After the video stream is pulled , We just use it to go to the cloud , And after the service terminal is connected to the stream , Send to AI The inference end performs video inference , And in 3d Insert our video service into the digital twin scene .
mediaserver Since we are responsible for transferring the agreement , In the Internet, that is gateway , It is also a router .

3 How simple enough ？

     This problem is more complicated , Because designing a product , We often fall into a dead end , Design into complexity .
1 Build a simple enough GB service
How simple is it? , only one c++ Program , But can bear 10k Concurrent , Can you do that? , Of course , There must be enough bandwidth to support .
2 GB Service and media network management .
If on the cloud , We can synthesize it directly , Then we need to combine the two , Use it directly c++ Program , I tested the following , Use the simplest and most simplified c++ The program has written the most basic code , Just a few hundred after compilation k The executable of , Do not rely on any other dynamic libraries . This will also be open source later .

4、 Play rtsp

establish rtsp There are certainly many ways to link , We can write by ourselves rtsp client , It's a way , Since the requirements here are the simplest , That's using ffmpeg Direct pull flow , The following code is an example .

int ffmpeg_rtsp_client()
{
    
    // Allocate an AVFormatContext
    AVFormatContext* format_ctx = avformat_alloc_context();
    // open rtsp: Open an input stream and read the header. The codecs are not opened
    const char* url = "rtsp://127.0.0.1/out.264";
    int ret = -1;
    AVDictionary* opts = NULL;
    av_dict_set(&opts, "rtsp_transport","tcp", 0);
    av_dict_set(&opts, "buffer_size", "1048576", 0);
    av_dict_set(&opts, "fpsprobesize", "2", 0);
    av_dict_set(&opts, "analyzeduration", "5000000", 0);

    // Set up   Maximum delay 
    av_dict_set(&opts, "max_delay", "500", 0);
    //rtmp、rtsp The delay is controlled to a minimum 
    av_dict_set(&opts, "fflags", "nobuffer", 0);
    // Set up   Blocking timeout , Otherwise, the connection may be blocked when the flow is disconnected 
    av_dict_set(&opts, "stimeout", "3000000", 0);


    ret = avformat_open_input(&format_ctx, url, nullptr, &opts);
    if (ret != 0) {
    
        fprintf(stderr, "fail to open url: %s, return value: %d\n", url, ret);
        return -1;
    }
    // Read packets of a media file to get stream information
    ret = avformat_find_stream_info(format_ctx, nullptr);
    if (ret < 0) {
    
        fprintf(stderr, "fail to get stream information: %d\n", ret);
        return -1;
    }
    // audio/video stream index
    int video_stream_index = -1;
    int audio_stream_index = -1;
    fprintf(stdout, "Number of elements in AVFormatContext.streams: %d\n", format_ctx->nb_streams);
    for (int i = 0; i < format_ctx->nb_streams; ++i) {
    
        const AVStream* stream = format_ctx->streams[i];
        fprintf(stdout, "type of the encoded data: %d\n", stream->codecpar->codec_id);
        if (stream->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) {
    
            video_stream_index = i;
            fprintf(stdout, "dimensions of the video frame in pixels: width: %d, height: %d, pixel format: %d\n",
                stream->codecpar->width, stream->codecpar->height, stream->codecpar->format);
        }
        else if (stream->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) {
    
            audio_stream_index = i;
            fprintf(stdout, "audio sample format: %d\n", stream->codecpar->format);
        }
    }
    if (video_stream_index == -1) {
    
        fprintf(stderr, "no video stream\n");
        return -1;
    }
    if (audio_stream_index == -1) {
    
        fprintf(stderr, "no audio stream\n");
    }
    int cnt = 0;
    AVPacket pkt;
    while (1) {
    
        if (++cnt > 100) break;
        ret = av_read_frame(format_ctx, &pkt);
        if (ret < 0) {
    
            fprintf(stderr, "error or end of file: %d\n", ret);
            continue;
        }
        if (pkt.stream_index == video_stream_index) {
    
            fprintf(stdout, "video stream, packet size: %d\n", pkt.size);
        }
        if (pkt.stream_index == audio_stream_index) {
    
            fprintf(stdout, "audio stream, packet size: %d\n", pkt.size);
        }
        av_packet_unref(&pkt);
    }
    avformat_free_context(format_ctx);
    return 0;
}

5 3d scene

Use threejs build ,threejs Simple enough , There are many examples , The details are written in the next chapter .

6 c# perhaps nodejs Establish gatekeeper

Use nodejs and c# The benefit is still simple enough , Use java It can be , But it is not as simple as the above two languages . establish http In service , If you use nodejs express, It is very fast and convenient . Use c# When I came to do it , It is also simple and fast enough , And cross platform , You can also directly compile the executable files of another platform on the platform , This and go Language is similar to , Grammar and java similar , The following is in gb28181 When building services , You need to use local IP Address , The code is as follows . When we go to the cloud, we need to use local IP Address, not Internet IP Address . Don't strange , Is true , The Internet address is only on the camera or nvr It uses .

class IPUtil
    {
    
        public static string IPV4()
        {
    
            string ipv4 = GetLocalIPv4(NetworkInterfaceType.Wireless80211);
            if (ipv4 == "")
            {
    
                ipv4 = GetLocalIPv4(NetworkInterfaceType.Ethernet);
                if (ipv4 == "")
                {
    
                    ipv4 = GetLoacalIPMaybeVirtualNetwork();
                }
            }
            return ipv4;
        }

        private static string GetLoacalIPMaybeVirtualNetwork()
        {
    
            string name = Dns.GetHostName();
            IPAddress[] ipadrlist = Dns.GetHostAddresses(name);
            foreach (IPAddress ipa in ipadrlist)
            {
    
                if (ipa.AddressFamily == AddressFamily.InterNetwork)
                {
    
                    return ipa.ToString();
                }
            }
            return " No network connection , Please link to the network and try again ！";
        }

        public static string GetLocalIPv4(NetworkInterfaceType _type)
        {
    
            string output = "";
            foreach (NetworkInterface item in NetworkInterface.GetAllNetworkInterfaces())
            {
    
                //Console.WriteLine(item.NetworkInterfaceType.ToString());
                if (item.NetworkInterfaceType == _type && item.OperationalStatus == OperationalStatus.Up)
                {
    
                    foreach (UnicastIPAddressInformation ip in item.GetIPProperties().UnicastAddresses)
                    {
    
                        if (ip.Address.AddressFamily == AddressFamily.InterNetwork)
                        {
    
                            output = ip.Address.ToString();
                        }
                    }
                }
            }
            return output;
        }
    }

7、 player

This next chapter talks about

8 Commercialization

Need one
1 The product manager
2 A front-end
3 A back end
4 One c++
5 One nodejs perhaps c# personnel
6 Project Freeman

After such a team is established , Need an experienced person to deal with the problem , We call this project following the free man , There is often such a person , It will reduce team members by 50% . This person needs to have enough experience .