TCP and reordering

On Wed, 2012-11-28 at 09:16 -0800, Eric Dumazet wrote:
> On Wed, 2012-11-28 at 16:41 +0000, David Woodhouse wrote:
> 
> > Another fun issue with tunnelling protocols and BQL... packets tend to
> > *grow* as they get encapsulated. So you might end up calling
> > netdev_sent_queue() with a given size, then netdev_completed_queue()
> > with a bigger packet later...
> 
> Its the driver responsibility to maintain the coherent 'bytes' value for
> each transmitted/completed packet.
> 
> If a driver calls an external entity, it cannot possibly use BQL, unless
> doing an approximation (bytes becomes a fixed value)
> 
> BQL was really something to control/limit queueing on ethernet links,
> not for stacked devices, as stacked devices normally have no queue.

Oh, of *course*... this is why my kernel would panic if I attempted to
add BQL to BR2684. The ATM low-level driver was pushing a header onto
the skb, and I ended up calling netdev_completed_queue() with a larger
'bytes' value than the one I'd called netdev_sent_queue() with. Which
leads to a BUG(), which immediately results in a panic. A moderately
suboptimal failure mode, when a nasty warning and disabling BQL on this
interface might have been nicer.

However, *perhaps* it isn't so hard to get a consistent 'bytes' value.
This version appears to work... can we use something along these lines
in the general case? What if skb_is_nonlinear()?

This probably doesn't work for L2TP where it's going to be passed down
the whole stack and get a new network header and everything. But for
BR2684 is this at least a salvageable approach?

On Sun, 2012-12-02 at 01:30 +0000, David Woodhouse wrote:

> Oh, of *course*... this is why my kernel would panic if I attempted to
> add BQL to BR2684. The ATM low-level driver was pushing a header onto
> the skb, and I ended up calling netdev_completed_queue() with a larger
> 'bytes' value than the one I'd called netdev_sent_queue() with. Which
> leads to a BUG(), which immediately results in a panic. A moderately
> suboptimal failure mode, when a nasty warning and disabling BQL on this
> interface might have been nicer.
> 
> However, *perhaps* it isn't so hard to get a consistent 'bytes' value.
> This version appears to work... can we use something along these lines
> in the general case? What if skb_is_nonlinear()?
> 
> This probably doesn't work for L2TP where it's going to be passed down
> the whole stack and get a new network header and everything. But for
> BR2684 is this at least a salvageable approach?
> 
> --- net/atm/br2684.c~	2012-12-01 16:35:49.000000000 +0000
> +++ net/atm/br2684.c	2012-12-02 01:18:35.216607088 +0000
> @@ -180,6 +180,11 @@ static struct notifier_block atm_dev_not
>  	.notifier_call = atm_dev_event,
>  };
>  
> +static unsigned int skb_acct_len(struct sk_buff *skb)
> +{
> +	return skb_tail_pointer(skb) - skb_network_header(skb);
> +}
> +
>  /* chained vcc->pop function.  Check if we should wake the netif_queue */
>  static void br2684_pop(struct atm_vcc *vcc, struct sk_buff *skb)
>  {
> @@ -191,6 +196,8 @@ static void br2684_pop(struct atm_vcc *v
>  	/* If the queue space just went up from zero, wake */
>  	if (atomic_inc_return(&brvcc->qspace) == 1)
>  		netif_wake_queue(brvcc->device);
> +
> +	netdev_completed_queue(brvcc->device, 1, skb_acct_len(skb));
>  }
>  
>  /*
> @@ -265,6 +272,7 @@ static int br2684_xmit_vcc(struct sk_buf
>  			netif_wake_queue(brvcc->device);
>  	}
>  
> +	netdev_sent_queue(brvcc->device, skb_acct_len(skb));
>  	/* If this fails immediately, the skb will be freed and br2684_pop()
>  	   will wake the queue if appropriate. Just return an error so that
>  	   the stats are updated correctly */
> @@ -710,6 +718,7 @@ static int br2684_create(void __user *ar
>  		return err;
>  	}


Apparently this driver doesnt add any feature at alloc_netdev() time, so
it might work. (no frags in any skb)



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Sat, 2012-12-01 at 18:40 -0800, Eric Dumazet wrote:
> > +static unsigned int skb_acct_len(struct sk_buff *skb)
> > +{
> > +	return skb_tail_pointer(skb) - skb_network_header(skb);
> > +}
> 
> Apparently this driver doesnt add any feature at alloc_netdev() time, so
> it might work. (no frags in any skb)

Well... as long as no driver appends stuff to the *tail* of the skb. As
long as they only *prepend* headers for the device to use, we're fine.
But I think that's fairly safe, for now.

For PPP I think I'll end up keeping a tuple of
	{ skb, orig_len, orig_destructor, orig_sk }
for each skb which is in-flight. Since the queues of those should be
*short*, and should generally complete in-order, that shouldn't be too
much overhead. It'll only need a small static array of them per-channel.

(Before complaining about a *static* array holding metadata about items
on a queue, stop thinking of the PPP channel being a queue and start
thinking of it more like a ring buffer.)