[Privoxy-devel] [privoxy-devel] TODO 157

[Lee]

On 5/24/17, Fabian Keil <fk at fabiankeil.de> wrote:
> CC and Reply-To set to privoxy-devel@ as some people have
> no ijbswa-developers@ subscription (anymore).
>
> Lee <ler762 at gmail.com> wrote on ijbswa-developers@:
>
>> > Add #157: Add a directive to control the currently hardcoded
>> > receive-buffer size
>>
>> Doesn't setting the receive buffer requires a setsockopt call with
>> either SO_SNDBUF or SO_RCVBUF?  I don't see it.
>> Or do you mean the
>>   char buf[BUFFER_SIZE];
>> in jcc.c:handle_established_connection changes ?  If that's it,
>
> I mean the latter.

OK.  I thought that was it until I'd been looking at the code for a
while and realized I wasn't sure.

>> - what do you hope to gain making it a user-controlled size
>
> The "best" size depends on the environment Privoxy runs in
> and on the Privoxy admin's requirements so hard-coding it
> complicates tuning.
>
> From the commit message in my local tree (which will change
> before I commit it):

in other words, you've already got a patch?

>     Add a receive-buffer-size directive
>
>     ... that can be used to set the size of the previously statically
>     allocated buffer in handle_established_connection().
>
>     Increasing the buffer size increases Privoxy's memory usage but
>     can lower the number of context switches and thereby reduce the
>     cpu usage and potentially increase the throughput.
>
>     This is mostly relevant for fast network connections and
>     large downloads that don't require filtering.

for whatever it's worth - it makes a difference for me and I don't
have a fast network connection.

How do you feel about turning off filtering for things that most
probably don't need it?
I have this bit in my user.action:
{-filter -deanimate-gifs}

/.*\.7z(\?|$)		# 7-Zip archive files
/.*\.avi(\?|$)
/.*\.cab(\?|$)
/.*\.crl(\?|$)		# certificate revocation list
/.*\.exe(\?|$)		# executable files
/.*\.fid(\?|$)
/.*\.flv(\?|$)
/.*\.gz(\?|$)		# gnu zip
/.*\.iso(\?|$)
/.*\.mar(\?|$)		# dunno - firefox updates ask for something like
			# firefox-52.0.1esr-52.0.2esr.partial.mar
/.*\.mov(\?|$)
/.*\.mp3(\?|$)
/.*\.mp4(\?|$)
/.*\.mpeg(\?|$)
/.*\.mpg(\?|$)
/.*\.ogg(\?|$)
/.*\.ogv(\?|$)
/.*\.zip(\?|$)		# Zip archive files

.youtube.com/videoplayback\?



>     Currently BUFFER_SIZE is kept as default and lower limit.
>
>     We should probably change the default to 16384, though,
>     while some users may further increase it to 32768 or 65536.

take a look at   http://httparchive.org/interesting.php
Unless you're expecting privoxy to run on a 16 bit machine I'd set the
default to 32KB

>     A dtrace command like:
>     sudo dtrace -n 'syscall::read:return /execname == "privoxy"/ {
> @[execname] = llquantize(arg0, 10, 0, 5, 20); @m = max(arg0)}'
>     can be used to properly tune the receive-buffer-size.
>
>     If the buffer is too large it will increase Privoxy's memory
>     footprint without any benefit. As the memory is (currently)
>     cleared before using it, a buffer that is too large can
>     actually reduce the throughput.

Is it really necessary to clear the buffer?  Seems like no, but my
ability to read & understand privoxy code leaves something to be
desired.

>     Things could be improved further by upwards scaling the buffer
>     dynamically based on how much of the previous allocation
>     was actually used.

Do you have any idea how often the headers include a 'Content-Length: nnn' line?
Seems like allocation a larger buffer once would be better/faster than
trying to figure it out on the fly.


>     Additionally the buffer should be referenced through
>     csp and also be used for other receive-related functions.

I'm lazy/ ignorant/ can't read code/ whatever.. take your pick :) but
what other receive-related functions are there?  seems like
jcc.c:handle_established_connection does it all.

>     Measured throughput when using four connections to
>     constantly request a 10 MB file:
>
>     ~320 MB/s with the default
>     ~400 MB/s with "receive-buffer-size    8192"
>     ~490 MB/s with "receive-buffer-size   16384"
>     ~610 MB/s with "receive-buffer-size   32768"
>     ~700 MB/s with "receive-buffer-size   65536"
>     ~755 MB/s with "receive-buffer-size  131072"
>     ~795 MB/s with "receive-buffer-size  262144"
>     ~804 MB/s with "receive-buffer-size  524288"
>     ~798 MB/s with "receive-buffer-size 1048576"
>     ~780 MB/s with "receive-buffer-size 2097152"

Interesting test setup.  804MB = 6,432 Gigabits/sec which means the
test was done with at least 10Gb/s links.

>     Sponsored by: Robert Klemme

Robert Klemme has a 10Gb/s Internet connection!!?

>> - how do you feel about using a GNU extension .. at least initially,
>> to see if the change is worth the effort?
>
> I already know that the change is worth the (quite moderate) effort.

Well.. a "quite moderate" effort for you is a serious stretch for me :)

This topic came up before, so what I did back then was add a
  #define TCP_BUFFER_SIZE xxx
in project.h & changed
  char buf[BUFFER_SIZE];
to
  char buf[TCP_BUFFER_SIZE];
in jcc.c:handle_established_connection and called it a day.

but after looking at the code again, handle_established_connection does a
  int max_bytes_to_read = sizeof(buf) - 1;
so now I'm thinking it should be xxx+1


>> https://www.gnu.org/software/gnu-c-manual/gnu-c-manual.html#Declaring-Arrays
>>   Another GNU extension allows you to declare an array size using
>> variables, rather than only constants.
>
> I don't like GNU extensions in general but in this case a stack-allocated
> array is also inappropriate as the "best" buffer size for many environments
> would likely result in Privoxy reaching the stack size limit when there are
> more than a couple of threads at the same time.

How do you make the buffer size allocation dynamic then?  Seems like
handle_established_connection is only called once per request, so how
do you change the buffer size after the initial allocation?

Regards,
Lee