[Privoxy-devel] TODO 157

[Fabian Keil]

Lee <ler762 at gmail.com> wrote:

> On 5/24/17, Fabian Keil <fk at fabiankeil.de> wrote:

> > Lee <ler762 at gmail.com> wrote on ijbswa-developers@:

> >> - what do you hope to gain making it a user-controlled size  
> >
> > The "best" size depends on the environment Privoxy runs in
> > and on the Privoxy admin's requirements so hard-coding it
> > complicates tuning.
> >
> > From the commit message in my local tree (which will change
> > before I commit it):  
> 
> in other words, you've already got a patch?

Yes. I intend to commit it in a couple of days.
 
> >     Add a receive-buffer-size directive
> >
> >     ... that can be used to set the size of the previously statically
> >     allocated buffer in handle_established_connection().
> >
> >     Increasing the buffer size increases Privoxy's memory usage but
> >     can lower the number of context switches and thereby reduce the
> >     cpu usage and potentially increase the throughput.
> >
> >     This is mostly relevant for fast network connections and
> >     large downloads that don't require filtering.  
> 
> for whatever it's worth - it makes a difference for me and I don't
> have a fast network connection.

It's supposed to mean that the impact is bigger when using
fast network connections.

> How do you feel about turning off filtering for things that most
> probably don't need it?
> I have this bit in my user.action:
> {-filter -deanimate-gifs}
> 
> /.*\.7z(\?|$)		# 7-Zip archive files
> /.*\.avi(\?|$)
> /.*\.cab(\?|$)
> /.*\.crl(\?|$)		# certificate revocation list
> /.*\.exe(\?|$)		# executable files
> /.*\.fid(\?|$)
> /.*\.flv(\?|$)
> /.*\.gz(\?|$)		# gnu zip
> /.*\.iso(\?|$)
> /.*\.mar(\?|$)		# dunno - firefox updates ask for
> something like # firefox-52.0.1esr-52.0.2esr.partial.mar
> /.*\.mov(\?|$)
> /.*\.mp3(\?|$)
> /.*\.mp4(\?|$)
> /.*\.mpeg(\?|$)
> /.*\.mpg(\?|$)
> /.*\.ogg(\?|$)
> /.*\.ogv(\?|$)
> /.*\.zip(\?|$)		# Zip archive files
> 
> .youtube.com/videoplayback\?

Have you observed these file types to be frequently served
with a Content-Type that Privoxy considers to be filterable?

> >     Currently BUFFER_SIZE is kept as default and lower limit.
> >
> >     We should probably change the default to 16384, though,
> >     while some users may further increase it to 32768 or 65536.  
> 
> take a look at   http://httparchive.org/interesting.php

Which of the graph do you consider relevant?

> Unless you're expecting privoxy to run on a 16 bit machine I'd set the
> default to 32KB

I don't see the connection here either.
 
> >     A dtrace command like:
> >     sudo dtrace -n 'syscall::read:return /execname == "privoxy"/ {
> > @[execname] = llquantize(arg0, 10, 0, 5, 20); @m = max(arg0)}'
> >     can be used to properly tune the receive-buffer-size.
> >
> >     If the buffer is too large it will increase Privoxy's memory
> >     footprint without any benefit. As the memory is (currently)
> >     cleared before using it, a buffer that is too large can
> >     actually reduce the throughput.  
> 
> Is it really necessary to clear the buffer?  Seems like no, but my
> ability to read & understand privoxy code leaves something to be
> desired.

Initialising the buffer makes it less likely that bugs result
in stack or heap data being leaked. Some code paths currently
depend on it. Obviously they could be changed but this is
unrelated to this commit.

For reasonable buffer sizes I expect the performance impact
to be minimal so I don't consider this a priority.

> >     Things could be improved further by upwards scaling the buffer
> >     dynamically based on how much of the previous allocation
> >     was actually used.  
> 
> Do you have any idea how often the headers include a 'Content-Length:
> nnn' line? Seems like allocation a larger buffer once would be
> better/faster than trying to figure it out on the fly.

I'm proposing to scale the buffer based on how much data
Privoxy could read from the network with on read_socket()
call. Whether or not a Content-Length header is available
isn't relevant for this to work.

If a large buffer isn't needed, allocating a large buffer
anyway is unlikely to help.

> >     Additionally the buffer should be referenced through
> >     csp and also be used for other receive-related functions.  
> 
> I'm lazy/ ignorant/ can't read code/ whatever.. take your pick :) but
> what other receive-related functions are there?  seems like
> jcc.c:handle_established_connection does it all.

You can grep for read_socket().

> >     Measured throughput when using four connections to
> >     constantly request a 10 MB file:
> >
> >     ~320 MB/s with the default
> >     ~400 MB/s with "receive-buffer-size    8192"
> >     ~490 MB/s with "receive-buffer-size   16384"
> >     ~610 MB/s with "receive-buffer-size   32768"
> >     ~700 MB/s with "receive-buffer-size   65536"
> >     ~755 MB/s with "receive-buffer-size  131072"
> >     ~795 MB/s with "receive-buffer-size  262144"
> >     ~804 MB/s with "receive-buffer-size  524288"
> >     ~798 MB/s with "receive-buffer-size 1048576"
> >     ~780 MB/s with "receive-buffer-size 2097152"  
> 
> Interesting test setup.  804MB = 6,432 Gigabits/sec which means the
> test was done with at least 10Gb/s links.
> 
> >     Sponsored by: Robert Klemme  
> 
> Robert Klemme has a 10Gb/s Internet connection!!?

I don't know. I made those tests in a bhyve vm.
While it has an emulated 10GB/s interface the tests
used the loopback interface and not the external network
or the Internet.

> >> https://www.gnu.org/software/gnu-c-manual/gnu-c-manual.html#Declaring-Arrays
> >>   Another GNU extension allows you to declare an array size using
> >> variables, rather than only constants.  
> >
> > I don't like GNU extensions in general but in this case a
> > stack-allocated array is also inappropriate as the "best" buffer size
> > for many environments would likely result in Privoxy reaching the
> > stack size limit when there are more than a couple of threads at the
> > same time.  
> 
> How do you make the buffer size allocation dynamic then?  Seems like
> handle_established_connection is only called once per request, so how
> do you change the buffer size after the initial allocation?

Once the buffer is allocated on the heap it can be resized
using realloc() or free'd and allocated again with the new size.

Fabian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.privoxy.org/pipermail/privoxy-devel/attachments/20170525/cac12e57/attachment.bin>