| Message ID | CA+icZUXd8fN5BAZuCna=zpgSJcXivSG1S6cV1eMMJMWwL7RrLA@mail.gmail.com |
|---|---|
| State | RFC, archived |
| Delegated to: | David Miller |
| Headers | show |
On Mon, Jan 2, 2017 at 12:05 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote: > > Hi, > > I am trying to force the set of BBR congestion control as default. > My old linux-config uses CUBIC as default. > I want both BBR and CUBIC to be built but BBR shall be my default. > > I tried the below snippet. > > I refresh my new linux-config like this... > > $ MAKE="make V=1" ; COMPILER="mycompiler" ; MAKE_OPTS="CC=$COMPILER > HOSTCC=$COMPILER" ; yes "" | $MAKE $MAKE_OPTS oldconfig && $MAKE > $MAKE_OPTS silentoldconfig < /dev/null > > What am I doing wrong? Perhaps your build directory already has an old .config file that sets the DEFAULT_TCP_CONG to be "cubic", and your "make oldconfig" and "make silentoldconfig" are maintaining those lines from the old .config? If you want to start with your existing .config but then make BBR the default, then it might be simplest just to edit your .config directly: -CONFIG_DEFAULT_TCP_CONG="cubic" +CONFIG_DEFAULT_TCP_CONG="bbr" BTW, I presume you have seen the warning in the BBR commit message or tcp_bbr.c about ensuring that BBR is used with the "fq" qdisc: NOTE: BBR *must* be used with the fq qdisc ("man tc-fq") with pacing enabled, since pacing is integral to the BBR design and implementation. BBR without pacing would not function properly, and may incur unnecessary high packet loss rates. The BBR quick-start guide has some details about how to build and enable BBR and fq: https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md Hope that helps, neal
On Mon, Jan 2, 2017 at 7:17 PM, Neal Cardwell <ncardwell@google.com> wrote: > On Mon, Jan 2, 2017 at 12:05 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote: >> >> Hi, >> >> I am trying to force the set of BBR congestion control as default. >> My old linux-config uses CUBIC as default. >> I want both BBR and CUBIC to be built but BBR shall be my default. >> >> I tried the below snippet. >> >> I refresh my new linux-config like this... >> >> $ MAKE="make V=1" ; COMPILER="mycompiler" ; MAKE_OPTS="CC=$COMPILER >> HOSTCC=$COMPILER" ; yes "" | $MAKE $MAKE_OPTS oldconfig && $MAKE >> $MAKE_OPTS silentoldconfig < /dev/null >> >> What am I doing wrong? > > Perhaps your build directory already has an old .config file that sets > the DEFAULT_TCP_CONG to be "cubic", and your "make oldconfig" and > "make silentoldconfig" are maintaining those lines from the old > .config? > > If you want to start with your existing .config but then make BBR the > default, then it might be simplest just to edit your .config directly: > > -CONFIG_DEFAULT_TCP_CONG="cubic" > +CONFIG_DEFAULT_TCP_CONG="bbr" > Just to clarify... I can have both TCP_CONG cubic and bbr built and switch for example via sysctl? Which tc version is required? Here tc is from iproute (20121211-2~precise). Is that enough? > BTW, I presume you have seen the warning in the BBR commit message or > tcp_bbr.c about ensuring that BBR is used with the "fq" qdisc: > > NOTE: BBR *must* be used with the fq qdisc ("man tc-fq") with pacing > enabled, since pacing is integral to the BBR design and > implementation. BBR without pacing would not function properly, and > may incur unnecessary high packet loss rates. > > The BBR quick-start guide has some details about how to build and > enable BBR and fq: > > https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md > Hmmm... From [1] Section "Further reading"... egrep '(CONFIG_TCP_CONG_BBR|CONFIG_NET_SCH_FQ)=' .config then you see exactly the following lines: CONFIG_TCP_CONG_BBR=y CONFIG_NET_SCH_FQ=y Should CONFIG_TCP_CONG_BBR have a "select CONFIG_NET_SCH_FQ" in its Kconfig? That would be safer. [1] https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md > Hope that helps, Thanks you reply helped for further testings. - Sedat - P.S.: Note2myself: Enable NET_SCH_FQ $ ./scripts/diffconfig /boot/config-4.9.0-2-iniza-amd64 .config NET_SCH_FQ n -> y
On Mon, Jan 2, 2017 at 1:49 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote: > On Mon, Jan 2, 2017 at 7:17 PM, Neal Cardwell <ncardwell@google.com> wrote: >> On Mon, Jan 2, 2017 at 12:05 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote: >>> >>> Hi, >>> >>> I am trying to force the set of BBR congestion control as default. >>> My old linux-config uses CUBIC as default. >>> I want both BBR and CUBIC to be built but BBR shall be my default. >>> >>> I tried the below snippet. >>> >>> I refresh my new linux-config like this... >>> >>> $ MAKE="make V=1" ; COMPILER="mycompiler" ; MAKE_OPTS="CC=$COMPILER >>> HOSTCC=$COMPILER" ; yes "" | $MAKE $MAKE_OPTS oldconfig && $MAKE >>> $MAKE_OPTS silentoldconfig < /dev/null >>> >>> What am I doing wrong? >> >> Perhaps your build directory already has an old .config file that sets >> the DEFAULT_TCP_CONG to be "cubic", and your "make oldconfig" and >> "make silentoldconfig" are maintaining those lines from the old >> .config? >> >> If you want to start with your existing .config but then make BBR the >> default, then it might be simplest just to edit your .config directly: >> >> -CONFIG_DEFAULT_TCP_CONG="cubic" >> +CONFIG_DEFAULT_TCP_CONG="bbr" >> > > Just to clarify... > > I can have both TCP_CONG cubic and bbr built and switch for example via sysctl? Yes, you can set the default congestion control using sysctl, eg: sysctl net.ipv4.tcp_congestion_control=bbr And (as mentioned in the quick-start guide) on many Linux systems you can make this "sticky" after reboots with something like: sudo bash -c 'echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf' > Which tc version is required? > Here tc is from iproute (20121211-2~precise). > Is that enough? You should not need a particular version of tc to install "fq". In the BBR quick-start guide we recommend setting fq to be the default qdisc: sudo bash -c 'echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf' But if that doesn't work in your environment, then I believe you should be able to install the fq qdisc with any version of tc, with something like: tc qdisc replace dev eth0 root fq You just won't be able to set or view configuration options. >> BTW, I presume you have seen the warning in the BBR commit message or >> tcp_bbr.c about ensuring that BBR is used with the "fq" qdisc: >> >> NOTE: BBR *must* be used with the fq qdisc ("man tc-fq") with pacing >> enabled, since pacing is integral to the BBR design and >> implementation. BBR without pacing would not function properly, and >> may incur unnecessary high packet loss rates. >> >> The BBR quick-start guide has some details about how to build and >> enable BBR and fq: >> >> https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md >> > > Hmmm... > > From [1] Section "Further reading"... > > egrep '(CONFIG_TCP_CONG_BBR|CONFIG_NET_SCH_FQ)=' .config > > then you see exactly the following lines: > > CONFIG_TCP_CONG_BBR=y > CONFIG_NET_SCH_FQ=y > > Should CONFIG_TCP_CONG_BBR have a "select CONFIG_NET_SCH_FQ" in its Kconfig? > That would be safer. > > [1] https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md That would be a little safer, but not sufficient (since the qdisc still has to be configured to be in the transmit path somewhere). thanks, neal
On Mon, Jan 2, 2017 at 8:12 PM, Neal Cardwell <ncardwell@google.com> wrote: > On Mon, Jan 2, 2017 at 1:49 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote: >> On Mon, Jan 2, 2017 at 7:17 PM, Neal Cardwell <ncardwell@google.com> wrote: >>> On Mon, Jan 2, 2017 at 12:05 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote: >>>> >>>> Hi, >>>> >>>> I am trying to force the set of BBR congestion control as default. >>>> My old linux-config uses CUBIC as default. >>>> I want both BBR and CUBIC to be built but BBR shall be my default. >>>> >>>> I tried the below snippet. >>>> >>>> I refresh my new linux-config like this... >>>> >>>> $ MAKE="make V=1" ; COMPILER="mycompiler" ; MAKE_OPTS="CC=$COMPILER >>>> HOSTCC=$COMPILER" ; yes "" | $MAKE $MAKE_OPTS oldconfig && $MAKE >>>> $MAKE_OPTS silentoldconfig < /dev/null >>>> >>>> What am I doing wrong? >>> >>> Perhaps your build directory already has an old .config file that sets >>> the DEFAULT_TCP_CONG to be "cubic", and your "make oldconfig" and >>> "make silentoldconfig" are maintaining those lines from the old >>> .config? >>> >>> If you want to start with your existing .config but then make BBR the >>> default, then it might be simplest just to edit your .config directly: >>> >>> -CONFIG_DEFAULT_TCP_CONG="cubic" >>> +CONFIG_DEFAULT_TCP_CONG="bbr" >>> >> >> Just to clarify... >> >> I can have both TCP_CONG cubic and bbr built and switch for example via sysctl? > > Yes, you can set the default congestion control using sysctl, eg: > > sysctl net.ipv4.tcp_congestion_control=bbr > > And (as mentioned in the quick-start guide) on many Linux systems you > can make this "sticky" after reboots with something like: > > sudo bash -c 'echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf' > >> Which tc version is required? >> Here tc is from iproute (20121211-2~precise). >> Is that enough? > > You should not need a particular version of tc to install "fq". > > In the BBR quick-start guide we recommend setting fq to be the default qdisc: > > sudo bash -c 'echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf' > OK, this looks now good. # sysctl net.core.default_qdisc net.core.default_qdisc = fq # sysctl net.ipv4.tcp_congestion_control net.ipv4.tcp_congestion_control = bbr I wondered why my Internet connection has so many stalls. > But if that doesn't work in your environment, then I believe you > should be able to install the fq qdisc with any version of tc, with > something like: > > tc qdisc replace dev eth0 root fq > > You just won't be able to set or view configuration options. > >>> BTW, I presume you have seen the warning in the BBR commit message or >>> tcp_bbr.c about ensuring that BBR is used with the "fq" qdisc: >>> >>> NOTE: BBR *must* be used with the fq qdisc ("man tc-fq") with pacing >>> enabled, since pacing is integral to the BBR design and >>> implementation. BBR without pacing would not function properly, and >>> may incur unnecessary high packet loss rates. >>> >>> The BBR quick-start guide has some details about how to build and >>> enable BBR and fq: >>> >>> https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md >>> >> >> Hmmm... >> >> From [1] Section "Further reading"... >> >> egrep '(CONFIG_TCP_CONG_BBR|CONFIG_NET_SCH_FQ)=' .config >> >> then you see exactly the following lines: >> >> CONFIG_TCP_CONG_BBR=y >> CONFIG_NET_SCH_FQ=y >> >> Should CONFIG_TCP_CONG_BBR have a "select CONFIG_NET_SCH_FQ" in its Kconfig? >> That would be safer. >> >> [1] https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md > > That would be a little safer, but not sufficient (since the qdisc > still has to be configured to be in the transmit path somewhere). > Does BBR only work with fq-qdisc best? What about fq_codel? - Sedat -
On Mon, Jan 2, 2017 at 2:30 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote: > OK, this looks now good. Great. Glad to hear it! > Does BBR only work with fq-qdisc best? Yes. BBR is designed to work with pacing. And so far the "fq" qdisc is the only qdisc that offers pacing. So BBR currently needs the "fq" qdisc. In the future, other qdiscs (or even other layers of the stack) may offer pacing, in which case BBR could use those as well. > What about fq_codel? The "fq_codel" qdisc does not implement pacing, so it would not be sufficient. cheers, neal
On Tue, Jan 3, 2017 at 12:16 AM, Neal Cardwell <ncardwell@google.com> wrote: > On Mon, Jan 2, 2017 at 2:30 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote: >> OK, this looks now good. > > Great. Glad to hear it! > >> Does BBR only work with fq-qdisc best? > > Yes. BBR is designed to work with pacing. And so far the "fq" qdisc is > the only qdisc that offers pacing. So BBR currently needs the "fq" > qdisc. In the future, other qdiscs (or even other layers of the stack) > may offer pacing, in which case BBR could use those as well. > >> What about fq_codel? > > The "fq_codel" qdisc does not implement pacing, so it would not be sufficient. > Thanks, you helped a lot! More experimenting and investigating. [ LINUX-CONFIG ] I looked at config.gce [1] and set all relevant =y settings in my linux-config (XXX = most important). CONFIG_TCP_CONG_ADVANCED=y CONFIG_TCP_CONG_CUBIC=y CONFIG_TCP_CONG_BBR=y <--- XXX CONFIG_DEFAULT_CUBIC=y # CONFIG_DEFAULT_BBR is not set CONFIG_DEFAULT_TCP_CONG="cubic" CONFIG_NET_SCH_HTB=y CONFIG_NET_SCH_PRIO=y CONFIG_NET_SCH_DSMARK=y CONFIG_NET_SCH_CODEL=y CONFIG_NET_SCH_FQ_CODEL=y CONFIG_NET_SCH_FQ=y <--- XXX CONFIG_NET_SCH_FIFO=y [ SYSCTL KUNGFU NOT ENOUGH ] Just got more curious... Setting TCP_BBR and FQ_QDISC via sysctl seems not to be enough? # sysctl net.ipv4.tcp_congestion_control net.ipv4.tcp_congestion_control = bbr # sysctl net.core.default_qdisc net.core.default_qdisc = fq I use WLAN on my notebook for my Internet connection (my "router" is an AndroidAP sing HSPA mobile broadband)... # LC_ALL=C ifconfig wlan0 wlan0 Link encap:Ethernet HWaddr 88:53:2e:ac:c3:12 inet addr:192.168.43.103 Bcast:192.168.43.255 Mask:255.255.255.0 inet6 addr: fe80::8a53:2eff:feac:c312/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:73 errors:0 dropped:0 overruns:0 frame:0 TX packets:131 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:24210 (24.2 KB) TX bytes:20783 (20.7 KB) ...and tc shows pfifo_fast set... # tc qdisc show dev wlan0 qdisc mq 0: dev wlan0 root qdisc pfifo_fast 0: dev wlan0 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan0 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan0 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev wlan0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 It looks like I have to replace my $iface with FQ_QDISC explicitly? # tc qdisc replace dev wlan0 root fq # tc qdisc show dev wlan0 qdisc fq 8001: root refcnt 5 [Unknown qdisc, optlen=80] What means "Unknown qdisc"? [ TCP_BBR KCONFIG ] I looked at TCP_CONG_BBR Kconfig help text and it says... "It requires the fq ("Fair Queue") pacing packet scheduler." Again, please add a "select NET_SCH_FQ" line - this is very user-friendly. A reference to [2], also. [ DOCS ] I haven't seen "official" docs in Documentation directory (Linux-kernel sources). Do you plan an update :-)? [ Building as a module ] When CONFIG_TCP_CONG_BBR=m and running a 'modprobe -v tcp_bbr' I do not see a string with /path/to/tcp_bbr.ko loaded. Hope this feedback helps. - Sedat - [1] https://raw.githubusercontent.com/google/bbr/master/Documentation/config.gce [2] https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md
--- a/net/ipv4/Kconfig +++ b/net/ipv4/Kconfig @@ -642,7 +642,7 @@ config TCP_CONG_CDG config TCP_CONG_BBR tristate "BBR TCP" - default n + default y ---help--- BBR (Bottleneck Bandwidth and RTT) TCP congestion control aims to @@ -657,7 +657,7 @@ config TCP_CONG_BBR choice prompt "Default TCP congestion control" - default DEFAULT_CUBIC + default DEFAULT_BBR help Select the TCP congestion control that will be used by default for all connections. @@ -716,7 +716,7 @@ config DEFAULT_TCP_CONG default "dctcp" if DEFAULT_DCTCP default "cdg" if DEFAULT_CDG default "bbr" if DEFAULT_BBR - default "cubic" + default "bbr" config TCP_MD5SIG bool "TCP: MD5 Signature Option support (RFC2385)"