[Privoxy-devel] any interest in a faster way to check blacklists? take 2

[Lee]

I started a thread about 1.5 years ago about creating action files
from host files.

The thread ended like this

>>> **DRAT**   For security reasons, Gmail does not allow you to use this
>>> type of file as it violates Google policy for executables and
>>> archives.
>>>
>>> In other words, goog doesn't allow .bat files in .zip files, so
>>> make-lightswitch.bat has been renamed to make-lightswitch.txt
>>
>> Looks like it didn't make it.
>
> *drat* again :(
>
> how about this
>   https://github.com/ler762/privoxy/tree/lee/contrib


So here's take 2

The scripts are at
  https://github.com/ler762/privoxy/tree/lee/contrib
hopefully I haven't left anything out.

make-lightswitch.sh takes the longest, so that's what I used for the
timing tests
( 6 lines of "./make-lightswitch.sh time" in a doit.sh script)

The times I got using the different privoxy "show url info" calls were

  print "GET http://config.privoxy.org/show-url-final-info?url=" url "
HTTP/1.1"  |& webserver
#   fastest -- calls merge_single_actions
Elapsed time: 493.00 seconds
Elapsed time: 505.00 seconds
Elapsed time: 482.00 seconds
Elapsed time: 493.00 seconds
Elapsed time: 482.00 seconds
Elapsed time: 500.00 seconds


  print "GET http://config.privoxy.org/show-url-info-x?url=" url "
HTTP/1.1"  |& webserver
#   faster -- calls merge_current_action
Elapsed time: 527.00 seconds
Elapsed time: 517.00 seconds
Elapsed time: 545.00 seconds
Elapsed time: 521.00 seconds
Elapsed time: 545.00 seconds
Elapsed time: 534.00 seconds


  print "GET http://config.privoxy.org/show-url-info?url=" url "
HTTP/1.1"  |& webserver
#   standard but slowest -- template from disk and calls merge_current_action
Elapsed time: 1016.00 seconds
Elapsed time: 1035.00 seconds
Elapsed time: 1036.00 seconds
Elapsed time: 1023.00 seconds
Elapsed time: 1013.00 seconds
Elapsed time: 1040.00 seconds


I think it'd be nice to add the "show-url-final-info" code to Privoxy
since it's the fastest, but I don't know if anyone else uses hostfile
blacklists to create action files, so maybe it's not worth it?

Another addition I'd like is not hitting the disk when repeating CGI calls:

$ ./make-lightswitch.sh
Elapsed time: 482.00 seconds

in jcc.c revert to always hitting the disk:
from
      if (continue_chatting && !(csp->flags & CSP_FLAG_CRUNCHED) &&
any_loaded_file_changed(csp))
to
      if (continue_chatting && any_loaded_file_changed(csp))

$ ./make-lightswitch.sh
Elapsed time: 581.00 seconds

put my "&& !(csp->flags & CSP_FLAG_CRUNCHED)" change back in:
$ ./make-lightswitch.sh
Elapsed time: 501.00 seconds


Any interest in merging the changes into Privoxy?

Lee