Patchwork HFSC classes going out of bounds, regression in recent kernels?

login
register
mail settings
Submitter Patrick McHardy
Date April 6, 2010, 3:37 p.m.
Message ID <4BBB5543.40607@trash.net>
Download mbox | patch
Permalink /patch/49525/
State Not Applicable
Delegated to: David Miller
Headers show

Comments

Patrick McHardy - April 6, 2010, 3:37 p.m.
Denys Fedorysychenko wrote:
> I notice on one of my QoS machines that HFSC start going out of bandwidth 
> limits. The most terrible thing - it happens suddenly, and if i just relaunch 
> QoS script - everything will work fine.

That sounds like there's an overflow somewhere.

> I'm not sure it is not my mistake, but most probably it is a bug.
> I can't tell for sure when it is happened, last kernel was on this machine 
> 2.6.28 i guess, or maybe even older.

Looking through the recent patches in this area, my prime suspect
is the attached patch. Does reverting it make any difference?
commit a4a710c4a7490587406462bf1d54504b7783d7d7
Author: Jarek Poplawski <jarkao2@gmail.com>
Date:   Mon Jun 8 22:05:13 2009 +0000

    pkt_sched: Change PSCHED_SHIFT from 10 to 6
    
    Change PSCHED_SHIFT from 10 to 6 to increase schedulers time
    resolution. This will increase 16x a number of (internal) ticks per
    nanosecond, and is needed to improve accuracy of schedulers based on
    rate tables, like HTB, TBF or CBQ, with rates above 100Mbit. It is
    assumed this change is safe for 32bit accounting of time diffs up
    to 2 minutes, which should be enough for common use (extremely low
    rate values may overflow, so get inaccurate instead). To make full
    use of this change an updated iproute2 will be needed. (But using
    older iproute2 should be safe too.)
    
    This change breaks ticks - microseconds similarity, so some minor code
    fixes might be needed. It is also planned to change naming adequately
    eg. to PSCHED_TICKS2NS() etc. in the near future.
    
    Reported-by: Antonio Almeida <vexwek@gmail.com>
    Tested-by: Antonio Almeida <vexwek@gmail.com>
    Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>
Denys Fedorysychenko - April 6, 2010, 3:45 p.m.
On Tuesday 06 April 2010 18:37:39 Patrick McHardy wrote:
> Denys Fedorysychenko wrote:
> > I notice on one of my QoS machines that HFSC start going out of bandwidth
> > limits. The most terrible thing - it happens suddenly, and if i just
> > relaunch QoS script - everything will work fine.
> 
> That sounds like there's an overflow somewhere.
> 
> > I'm not sure it is not my mistake, but most probably it is a bug.
> > I can't tell for sure when it is happened, last kernel was on this
> > machine 2.6.28 i guess, or maybe even older.
> 
> Looking through the recent patches in this area, my prime suspect
> is the attached patch. Does reverting it make any difference?
> 
I will try to upgrade soon, it is critical router, so probably i will do this 
tonight. 
I guess with reverting this patch also it will hurt shaper resolution on high 
speeds... not a case for me, but for other people.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Denys Fedorysychenko - April 11, 2010, 2:10 a.m.
On Tuesday 06 April 2010 18:37:39 Patrick McHardy wrote:
> Denys Fedorysychenko wrote:
> > I notice on one of my QoS machines that HFSC start going out of bandwidth
> > limits. The most terrible thing - it happens suddenly, and if i just
> > relaunch QoS script - everything will work fine.
> 
> That sounds like there's an overflow somewhere.
> 
> > I'm not sure it is not my mistake, but most probably it is a bug.
> > I can't tell for sure when it is happened, last kernel was on this
> > machine 2.6.28 i guess, or maybe even older.
> 
> Looking through the recent patches in this area, my prime suspect
> is the attached patch. Does reverting it make any difference?
> 
Hi, i made sure - bug not related to this patch. I try to patch (also had to 
test latest stable release, to make sure it is appearing on it), and each test 
taking 1 day. If i "restart" script to resetup HFSC - it works for a while 
fine. At next day, peak time period - it goes wild.

There is another thing also, i am going to try now, old kernel was 64-bit, but 
now 32bit, so i will try to shift to 64bit again.

It can be not regression, but a bug. I guess not many people use HFSC, even it 
is definitely better than HTB. Or i'm wrong?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Denys Fedorysychenko - May 20, 2010, 1:34 a.m.
I am trying to track down HFSC bug.
It seems, most probably it is related to PSCHED_SHIFT at the end, i am doing 
testing again. I will try to do complete clean build, maybe last time some .o 
was left or i forgot to do make clean.

SM_SHIFT in HFSC is calculated as 30 - PSCHED_SHIFT, and it is shifted too 
much (or not enough) with new changes (ISM_SHIFT seems wrong too). So it is 
most probably overflow or not enough resolution.
I will try to change PSCHED_SHIFT back to confirm that, and at least i found 
way to reproduce bug.

Additionally in sch_hfsc.c i notice mentioned that PSCHED_SHIFT 10 is tick per 
1024us, but i try to calculate their table (in source comments), it doesn't 
fit with my calculations based on 1024us/tick, but fits well with 1024 
nanosecond.

Is it was 1024ns per tick and now 64ns per tick? Or it is microseconds(us) ?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Michal Soltys - May 31, 2010, 5:53 p.m.
On 10-05-20 03:34, Denys Fedorysychenko wrote:
> I am trying to track down HFSC bug.
> It seems, most probably it is related to PSCHED_SHIFT at the end, i am doing
> testing again. I will try to do complete clean build, maybe last time some .o
> was left or i forgot to do make clean.
> 
> SM_SHIFT in HFSC is calculated as 30 - PSCHED_SHIFT, and it is shifted too
> much (or not enough) with new changes (ISM_SHIFT seems wrong too). So it is
> most probably overflow or not enough resolution.
> I will try to change PSCHED_SHIFT back to confirm that, and at least i found
> way to reproduce bug.
> 
> Additionally in sch_hfsc.c i notice mentioned that PSCHED_SHIFT 10 is tick per
> 1024us, but i try to calculate their table (in source comments), it doesn't
> fit with my calculations based on 1024us/tick, but fits well with 1024
> nanosecond.
> 
> Is it was 1024ns per tick and now 64ns per tick? Or it is microseconds(us) ?
> --

In the old table (when PSCHED_SHIFT was 10) in the hfsc file, the requirements 
to keep 4 decimal digits were 20 and 18 for SM_ and ISM_ respectively. 
For the reference:

 *  bits/sec      100Kbps     1Mbps     10Mbps     100Mbps    1Gbps
 *  ------------+-------------------------------------------------------
 *  bytes/1.024us 12.8e-3    128e-3     1280e-3    12800e-3   128000e-3
 *  1.024us/byte  78.125     7.8125     0.78125    0.078125   0.0078125
 *
 * So, for PSCHED_SHIFT 10 we need: SM_SHIFT 20, ISM_SHIFT 18.


Considering the table - and if I didn't miss anything - they were swapped.

psticks/byte in its "corner" case (0.0078125) requires 1e6, which corrsponds to 
ISM_SHIFT == 20.

Similary, corner case for bytes/pstick (0.0128) requires 1e5, where 
SM_SHIFT == 17 would be sufficient.

The above assuming pstick is 1.024us

Currently, with PSCHED_SHIFT == 6 (so pstick is 64ns), the define macros 
actually hit the spot, giving following values:

bytes/pstick: SM_SHIFT  24 (0.0008 @ 100kbit)
psticks/byte: ISM_SHIFT 14 (0.125  @ 1gbit)

But in generic case - the macros will not yield appropriate results - e.g. 
for PSCHED_SHIFT == 10, SM_SHIFT would be 20 and ISM_SHIFT would be 18 (not 
enough for 4 decimal digits).

On a related subject - it would probably be desirable to consider smaller 
speeds than 100kbit, as adsl links with mere 128kbit upstream total are 
nothing out of ordinary. If we considered 10kbit as the reference value, we 
would need SM_SHIFT 27 to cover 0.00008 @ 10kbit. Not sure if 10gbit is 
desirable too - ISM_SHIFT would need 17 then. All assuming that 4 decimal 
digits is what we really need.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/include/net/pkt_sched.h b/include/net/pkt_sched.h
index cd0e026..120935b 100644
--- a/include/net/pkt_sched.h
+++ b/include/net/pkt_sched.h
@@ -41,8 +41,8 @@  static inline void *qdisc_priv(struct Qdisc *q)
 typedef u64	psched_time_t;
 typedef long	psched_tdiff_t;
 
-/* Avoid doing 64 bit divide by 1000 */
-#define PSCHED_SHIFT			10
+/* Avoid doing 64 bit divide */
+#define PSCHED_SHIFT			6
 #define PSCHED_US2NS(x)			((s64)(x) << PSCHED_SHIFT)
 #define PSCHED_NS2US(x)			((x) >> PSCHED_SHIFT)