[Privoxy-devel] [privoxy-devel] TODO 157

[Fabian Keil]

CC and Reply-To set to privoxy-devel@ as some people have
no ijbswa-developers@ subscription (anymore).

Lee <ler762 at gmail.com> wrote on ijbswa-developers@:

> > Add #157: Add a directive to control the currently hardcoded
> > receive-buffer size  
> 
> Doesn't setting the receive buffer requires a setsockopt call with
> either SO_SNDBUF or SO_RCVBUF?  I don't see it.
> Or do you mean the
>   char buf[BUFFER_SIZE];
> in jcc.c:handle_established_connection changes ?  If that's it,

I mean the latter.
 
> - what do you hope to gain making it a user-controlled size

The "best" size depends on the environment Privoxy runs in
and on the Privoxy admin's requirements so hard-coding it
complicates tuning.

From the commit message in my local tree (which will change
before I commit it):

    Add a receive-buffer-size directive
    
    ... that can be used to set the size of the previously statically
    allocated buffer in handle_established_connection().
    
    Increasing the buffer size increases Privoxy's memory usage but
    can lower the number of context switches and thereby reduce the
    cpu usage and potentially increase the throughput.
    
    This is mostly relevant for fast network connections and
    large downloads that don't require filtering.
    
    Currently BUFFER_SIZE is kept as default and lower limit.
    
    We should probably change the default to 16384, though,
    while some users may further increase it to 32768 or 65536.
    
    A dtrace command like:
    sudo dtrace -n 'syscall::read:return /execname == "privoxy"/ { @[execname] = llquantize(arg0, 10, 0, 5, 20); @m = max(arg0)}'
    can be used to properly tune the receive-buffer-size.
    
    If the buffer is too large it will increase Privoxy's memory
    footprint without any benefit. As the memory is (currently)
    cleared before using it, a buffer that is too large can
    actually reduce the throughput.
    
    Things could be improved further by upwards scaling the buffer
    dynamically based on how much of the previous allocation
    was actually used.
    
    Additionally the buffer should be referenced through
    csp and also be used for other receive-related functions.
    
    Measured throughput when using four connections to
    constantly request a 10 MB file:
    
    ~320 MB/s with the default
    ~400 MB/s with "receive-buffer-size    8192"
    ~490 MB/s with "receive-buffer-size   16384"
    ~610 MB/s with "receive-buffer-size   32768"
    ~700 MB/s with "receive-buffer-size   65536"
    ~755 MB/s with "receive-buffer-size  131072"
    ~795 MB/s with "receive-buffer-size  262144"
    ~804 MB/s with "receive-buffer-size  524288"
    ~798 MB/s with "receive-buffer-size 1048576"
    ~780 MB/s with "receive-buffer-size 2097152"
    
    Sponsored by: Robert Klemme

> - how do you feel about using a GNU extension .. at least initially,
> to see if the change is worth the effort?

I already know that the change is worth the (quite moderate) effort.

> https://www.gnu.org/software/gnu-c-manual/gnu-c-manual.html#Declaring-Arrays
>   Another GNU extension allows you to declare an array size using
> variables, rather than only constants.

I don't like GNU extensions in general but in this case a stack-allocated
array is also inappropriate as the "best" buffer size for many environments
would likely result in Privoxy reaching the stack size limit when there are
more than a couple of threads at the same time.

Fabian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.privoxy.org/pipermail/privoxy-devel/attachments/20170524/4462e830/attachment.bin>