TCP RTT measurement tips

TCP RTT Uncertainty has been criticized ：

• The receiving end may be turned on Delayed ACK ,Delay Delay is uncertain .
• The receiving end may be turned on LRO/GRO ,Merge Delay is uncertain .
• During retransmission, the original packet and retransmission packet cannot be distinguished .

The third point is QUIC The opposite of the textbook .

That's the case ？ How others teach you , Just take notes , Anything that doesn't match the notes is wrong ？

Tonight? , Let me demonstrate an accurate measurement RTT Methods , There must not be any in your notes , But you can add .

The interesting thing about this method is that it is in Recovery Measure in state . according to TCP standard , When receiving a discontinuous message, the receiving end must immediately ACK, This avoids Delayed ACK uncertainty .

It's easy to do ：

• Turn on timestamps Options , The opt Inside tsval Sure echo Come back , Use this field as the identification .
• For retransmission skb In the play anch = opts->tsval + 1 As tsval, Send , Record the current ts_us To skb.
• For other transmissions skb And Retransmission skb To avoid the anch As tsval, take anch + 1 As its tsval.
• observation ACK Of tsecr, scanning rtx queue, if ACK.tsecr == anch,now_us - skb.ts_us Is accurate RTT.

Here's a simple one POC Script ：

#!/usr/local/bin/stap -g
%{
    
#include <linux/tcp.h>
#include <net/tcp.h>

__u32 anch = 0;
__u32 skip = 0;

%}

function set_ts(tpp:long, opt:long)
%{
    
	struct tcp_sock *tp = (struct tcp_sock *)STAP_ARG_tpp;
	struct sock *sk = (struct sock *)tp;
	struct inet_connection_sock *icsk = inet_csk(sk);
	struct tcp_out_options *opts = (struct tcp_out_options *)STAP_ARG_opt;

	if (tp != NULL &&
	    ntohs(inet_sk(sk)->inet_dport) == 5001 &&
	    icsk->icsk_ca_state == TCP_CA_Recovery) {
    
		if (opts->tsval == skip) {
     //  To avoid the  anch
			opts->tsval ++; //  Don't avoid too much , Beware of the opposite end  PAWS  testing 
		} else if (skip == 0){
     //  Set up  anch  As an identifier 
			anch = opts->tsval; 
			opts->tsval = anch;
			skip = anch;
			STAP_PRINTF("set ts: %d\n", anch);
		}
	}
	if (tp != NULL && icsk->icsk_ca_state == TCP_CA_Open) {
    
		skip = 0;
		anch = 0;
	}
%}

function get_ts(skk:long)
%{
    
	struct sock *sk = (struct sock *)STAP_ARG_skk;
	struct tcp_sock *tp = (struct tcp_sock *)sk;
	struct inet_connection_sock *icsk = inet_csk(sk);

	if (ntohs(inet_sk(sk)->inet_dport) == 5001 &&
	    icsk->icsk_ca_state == TCP_CA_Recovery) {
    
		__u32 ecr = tp->rx_opt.rcv_tsecr + tp->tsoffset;
		if (skip == ecr) {
     //  Find the confirmation and type  anch  Of retransmission packets  ACK.
			u32 delta = tcp_time_stamp(tp) - ecr + tp->tsoffset;
			delta = delta * (USEC_PER_SEC / TCP_TS_HZ);
			STAP_PRINTF("echo ts: %d anch: %d rtt_us: %d srtt_us: %d\n", ecr, anch, delta, tp->srtt_us>>3);
			skip = 0;
			anch = 0;
		}
	}
%}

probe kernel.function("tcp_options_write")
{
    
	set_ts($tp, $opts);
}

probe kernel.function("tcp_ack_update_rtt")
{
    
	get_ts($sk);
}

Interestingly ,timestamps In measurement RTT The process does not involve any time-related operations , Only as an identifier . In the actual , TCP timestamps The accuracy is too low , Can't identify us, I always thought TCP Rely entirely on it to calculate RTT, But there is no ：

//  It's really useless  
static inline u32 tcp_skb_timestamp(const struct sk_buff *skb)
{
    
        return div_u64(skb->skb_mstamp_ns, NSEC_PER_SEC / TCP_TS_HZ);
}

The above script does not involve scanning rtx queue The logic of , Because it needs to be modified Linux TCP Source code , It's not something you can do in an hour or two at night .

TCP Transmission optimization , You can use this exact RTT：

• If accurate RTT And srtt_us The same does not increase significantly , Probability of non congestion packet loss , Can be appropriately radical , After retransmission, you can undo.
• Throughout Recovery state , This method can be used for accurate RTT measurement , And follow up with srtt_us The difference between the .
• Cooperate with the front half way RTT Jitter detection , even Data Half way jitter , It can also distinguish between jitter and congestion .

There are many ways to play …

But if you don't support timestamps What do I do ？

The goal is not to use timestamps, The goal is to need a field that can echo Come back for Data and ACK Just correspond .

obviously ,seq Can also be echo Come back , Take it echo What's back is ACK. To capture identification for accurate calculation RTT So as to judge whether there is congestion , Whether the final decision is undo, Need fancy retransmission ：

• if una～una + mss As a skb Transmit and be mark lost, Then retransmission una～una + fk(fk take 123 or 222 Equal flower value ).
• Record the current ts_us To retransmit skb.
• Retain una + fk ～ una + mss Is empty , So that the receiving end can reply in time ACK.
• observation ACK, scanning rtx queue, if ACK == una + fk == skb.end_seq,now_us - skb.ts_us Is accurate RTT.

Fancy retransmission can distinguish between original packet and retransmission packet ：

• if ACK Cover una + fk, Then ACK Non retransmission packet trigger , Probability of non congestion , Invalid retransmission , But the situation is complicated , Ignore .
• if ACK be equal to una + fk, Then ACK Triggered for retransmission packets , Calculate according to the above method RTT, And srtt_us Compare .

The following play is just like timestamps It is the same when it is opened .

Fancy transmission can also be more expensive , It will be introduced separately later , This article only relates to RTT Measurement related .

thus , We already have three messages ：

• Open In normal state srtt_us.
• Two half passes in any state ts The shaking of .
• Recovery State accurate RTT.

The last two of them are mined through additional ideas . I still have the old idea , Make full use of information , Use the existing information to the extreme and then complain about the lack of information .

This idea is very common , Given a mathematical function , You can see its image , But is that enough ？ Take its derivative , Take the derivative of it again , Take the derivative again … You can see its monotonicity , Concave and convex , Extremum and other properties .

It's sultry , High temperature yellow warning , It never rains , During the day, I used parson to operate the Exel table for a whole day , Do something else in the evening . Essay record .

Zhejiang Wenzhou leather shoes wet , It's not fat when it's raining .