[Privoxy-devel] TODO 157

[Fabian Keil]

Lee <ler762 at gmail.com> wrote:

> On 5/25/17, Fabian Keil <fk at fabiankeil.de> wrote:
> > Lee <ler762 at gmail.com> wrote:
> >  
> >> On 5/24/17, Fabian Keil <fk at fabiankeil.de> wrote:  
> >  
> >> > Lee <ler762 at gmail.com> wrote on ijbswa-developers@:  
 
> >> How do you feel about turning off filtering for things that most
> >> probably don't need it?  
>   <.. snip ..>
> >
> > Have you observed these file types to be frequently served
> > with a Content-Type that Privoxy considers to be filterable?  
> 
> remember this thread
>   https://sourceforge.net/p/ijbswa/mailman/message/26831156/
> The last time I remember looking at it was around that time & I forgot
> all about filtering not working on encrypted connections - I just kept
> adding to the "don't bother filtering this" list.
> 
> Is there an easy way to tell if something is being filtered?

Enabling "debug 64" and checking the log seems easy enough.
 
> > Obviously they could be changed but this is
> > unrelated to this commit.
> >
> > For reasonable buffer sizes I expect the performance impact
> > to be minimal so I don't consider this a priority.  
> 
> just curious - what do you consider a reasonable buffer size?
> I've had it set to 46720 for I don't know how long & haven't noticed
> any problems.

It depends on the connection. I don't see the point of using
a buffer that is so large that it never gets even close to
being full.

I'm usually using Privoxy with Tor. Given this read length distribution:

[...]
            8900 |                                         47       
            9000 |@                                        155      
            9100 |                                         9        
            9200 |                                         2        
            9300 |                                         3        
            9400 |                                         17       
            9500 |                                         7        
            9600 |                                         7        
            9700 |                                         2        
            9800 |                                         15       
            9900 |                                         12       
           10000 |@@@@@@@@                                 2321     
           20000 |                                         16       
           30000 |                                         2        
           40000 |                                         1        
           50000 |                                         1        
           60000 |                                         0        
   
it seems reasonable to me to use a receive-buffer-size below
20k. Using a bit more is unlikely to hurt but isn't likely to
noticeable improve things either.

Using a lot more (1m for example) seems unreasonable as
it clearly isn't needed.

> >> >     Things could be improved further by upwards scaling the buffer
> >> >     dynamically based on how much of the previous allocation
> >> >     was actually used.  
> 
> You might want to think about limiting how much you scale up the
> buffer each time.  If the user has window scaling & selective acks
> enabled you could see a very large amount of data suddenly become
> available.

While this could result in the upper bound being reached more quickly
I'm not sure it matters.
 
> >> Do you have any idea how often the headers include a 'Content-Length:
> >> nnn' line? Seems like allocation a larger buffer once would be
> >> better/faster than trying to figure it out on the fly.  
> >
> > I'm proposing to scale the buffer based on how much data
> > Privoxy could read from the network with on read_socket()
> > call. Whether or not a Content-Length header is available
> > isn't relevant for this to work.  
> 
> I'm just thinking it might be a pretty good hint as to what the
> initial buffer size should be.

I'm not sure this optimisation is necessary.

It also wouldn't work for the initial buffer size once the
buffer is used to read server headers as well.

The buffer scaling already done in add_to_iob() isn't fancy
either and seems to work well enough in practice.
 
> > I don't know. I made those tests in a bhyve vm.
> > While it has an emulated 10GB/s interface the tests
> > used the loopback interface and not the external network
> > or the Internet.  
> 
> Is it possible to have a dropped packet when you're using the loopback
> interface?

The system wasn't intentionally configured to drop packets
and it seems unlikely that the tests itself resulted in
package loss.

> Doing your testing on a vm seems like a good way to figure out where
> all the bottlenecks are, but it seems like you'd be missing most if
> not all of the nasty things that happen on the internet like dropped
> packets.

I'm aware of that.

I'm also intentionally using test scenarios that are likely to show
the impact of the patch set I'm testing.

Fabian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.privoxy.org/pipermail/privoxy-devel/attachments/20170527/105c96ce/attachment.bin>