Use BPF to count network traffic

This paper introduces the use of BPF Count network traffic . Network traffic is an important billing indicator of cloud products , The server can process millions of packets per second , This also requires efficient methods to count traffic , and BPF Technology originally handled as a network packet , Traffic processing designed and constructed to support this rate .

Use libpcap

stay BPF Times before , We can use libpcap Realization and tcpdump In a similar way , Capture network traffic and copy it to the user program for statistics .

Here is a simple example ：

#define ETHERNET_HEADER_LEN 14
#define MIN_IP_HEADER_LEN 20
#define IP_HL(ip) (((ip)->ihl) & 0x0f)

const char* dev = "br-7e20abc6df31";
const char* filter_expr = "src net 172.20.0.0/16";

int64_t traffic = 0;

// Reference:
//      https://www.tcpdump.org/manpages/
//      https://tools.ietf.org/html/rfc791
void traffic_stat() {
    // find the IPv4 network number and netmask for a device
    bpf_u_int32 net = 0;
    bpf_u_int32 mask = 0;
    char errbuf[PCAP_ERRBUF_SIZE] = {0};
    int ret = pcap_lookupnet(dev, &net, &mask, errbuf);
    if (ret == PCAP_ERROR) {
        fprintf(stderr, "pcap_lookupnet failed: %s\n", errbuf);
        exit(EXIT_FAILURE);
    }

    // open a device for capturing
    int promisc = 1;
    int timeout = 1000;   // in milliseconds
    const int SNAP_LEN = 64;
    auto handle = pcap_open_live(dev, SNAP_LEN, promisc, timeout, errbuf);
    if (!handle) {
        fprintf(stderr, "pcap_open_live failed: %s\n", errbuf);
        exit(EXIT_FAILURE);
    }

    // compile a filter expression
    struct bpf_program fp;
    int optimize = 1;
    ret = pcap_compile(handle, &fp, filter_expr, optimize, net);
    if (ret == PCAP_ERROR) {
        fprintf(stderr, "pcap_compile failed: %s\n", pcap_geterr(handle));
        exit(EXIT_FAILURE);
    }

    // set the filter
    ret = pcap_setfilter(handle, &fp);
    if (ret == PCAP_ERROR) {
        fprintf(stderr, "pcap_setfilter failed: %s\n", pcap_geterr(handle));
        exit(EXIT_FAILURE);
    }

    // process packets from a live capture
    int packet_count = -1;  // -1 means infinity
    pcap_loop(handle, packet_count, [](u_char* args, const struct pcap_pkthdr* header, const u_char* bytes) {

        auto ip_header = reinterpret_cast<iphdr*>(const_cast<u_char*>(bytes) + ETHERNET_HEADER_LEN);
        const int ip_header_len = IP_HL(ip_header) * 4;
        if (ip_header_len < MIN_IP_HEADER_LEN) {
            return;
        }

        auto len = ntohs(ip_header->tot_len);
        if (len <= 0) {
            return;
        }

        auto traffic = reinterpret_cast<int64_t*>(args);
        *traffic += len;

    }, reinterpret_cast<u_char*>(&traffic));

    // free a BPF program
    pcap_freecode(&fp);
    // close the capture device
    pcap_close(handle);
}

br-7e20abc6df31 It's using Docker Created bridge ,IP yes 172.20.0.1, Use src net 172.20.0.0/16 Expression to filter outbound traffic .

This program can do the right thing , Namely, statistical flow . The problem lies in , It needs to copy all the traffic flowing through the network card to the user program , Then the statistics are carried out , These copies are then discarded , A lot of system resources are wasted .

Big killer BPF

and BPF Obviously the perfect solution to this problem . All we need is to accumulate outbound traffic , If it can be executed in kernel mode , That means we don't need to copy network packets .

Here is a use libbpf Written program , Its function is similar to the above libpcap Consistent role ：

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

int ifindex = 0;
__u64 traffic = 0;

SEC("tp_btf/netif_receive_skb")
int BPF_PROG(netif_receive_skb, struct sk_buff *skb)
{
    if (skb->dev->ifindex == ifindex) {
        traffic += skb->data_len;
    }
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

here ,ifindex That is, the above network equipment br-7e20abc6df31 Interface index of .

In this program , We attached it to netif_receive_skb This tracepoint On , After the network card receives the packet , We judge whether it is the target device , if , Then accumulate the flow . No data copies , No context switch , A simple and efficient .

actual combat

Use dd Create a 512M Size file ：

$ dd if=/dev/zero of=/tmp/testfile bs=4096 count=131072

Create a new network device ：

$ sudo docker network create my-tc-net

Use the network created above to run a nginx The server , Provide file download ：

$ sudo docker run -d --rm \
        -p 10086:80 \
        -v /tmp/testfile:/home/data/testfile \
        -v $(PWD)/default.conf:/etc/nginx/conf.d/default.conf \
        --name my-nginx \
        --network my-tc-net \
        nginx:alpine

This is our server configuration ：

server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    location /downloads/ {
        alias               /home/data/;
    }
}

Run the above flow statistics program ：

$ sudo ./trafficstat

Download the file ：

$ curl http://localhost:10086/downloads/testfile --output testfile

View traffic statistics output ：

$ sudo ./trafficstat
...
traffic in bytes: 536878816
...

In line with expectations , Be accomplished . The source code of this article can be found in here find .

Conclusion

This article demonstrates the use of libbpf To write BPF Program , A scheme to achieve efficient statistics of network traffic in the kernel state .