[Privoxy-devel] any interest in a faster way to check blacklists?

[Lee]

On 10/2/22, Fabian Keil  wrote:
> Lee  wrote on 2020-02-13 at 12:19:56:
>
> CC'ing Lee as Mailman unsubscribed gmail addresses a while
> ago due to Google bouncing mails again ...

*sigh*  the msg sent from your .de address has a
Why is this message in Spam? It's similar to messages that were
detected by our spam filters.
notice at the top even though it showed up in my inbox :\

I'm about ready to start paying for mail if they have a decent privacy
policy.  Or switch to some other free mail w/ a tolerable privacy
policy.  Anyone have any suggestions?

>> On 2/13/20, Fabian Keil   wrote:
>> > Lee   wrote:
>> >
>> >> On 2/7/20, Fabian Keil   wrote:
>> >> >> I'm not sure how to sum up..  Other than testing a new release, I
>> >> >> rarely run the privoxy regression test.  The only other program I
>> >> >> have
>> >> >> calling a privoxy cgi function is that turn a host file into an
>> >> >> action
>> >> >> file script that I run a couple of times a week.
>> >> >
>> >> > Is that program free software and available somewhere?
>> >>
>> >> See attached - a Windows .bat script calls a gawk script.  If you
>> >> really want, I can add a GPLv2 blurb in there somewhere.
>> >>
>> >> To do the timing tests I commented out the curl ... >
>> >> lightswitch-hosts.txt line so that I was always using the same input.
>> >> I didn't go as far as disconnecting the ethernet cable, which I'm
>> >> guessing is the main reason the times vary a bit.
>> >>
>> >>
>> >> **DRAT**   For security reasons, Gmail does not allow you to use this
>> >> type of file as it violates Google policy for executables and
>> >> archives.
>> >>
>> >> In other words, goog doesn't allow .bat files in .zip files, so
>> >> make-lightswitch.bat has been renamed to make-lightswitch.txt
>> >
>> > Looks like it didn't make it.
>>
>> *drat* again :(
>>
>> how about this
>>   https://github.com/ler762/privoxy/tree/lee/contrib
>
> I finally found the time to look at this thread and the commit
> ff0bf9b850d53 again and have additional comments.

Thank you!

> I still think a modified version of the commit would be nice to have
> but as it's intended to be used by scripts I'm not sure why the generated
> page has to be HTML.

Because my ignorance?  You get to the cgi page from http://p.p so the
result is html?

I was going for easy and the current show-url-info function returns
html so I copied that already working code and then tried to figure
out how to make it faster.

> Returning plain text may be even faster although
> I didn't benchmark it.

Possibly?  But I suspect returning text instead of text wrapped in
html it isn't going to be that much of an improvement..  at least if
what gets returned is small enough to fit into a single write from
privoxy and a single read from the user program.
  (i have a vague memory of seeing awk do multiple reads from privoxy
to get all the returned info from show-url-info.  And Privoxy doing
multiple reads to get the template into memory didn't help either..
Shrinking what privoxy wrote down to what awk could read in a single
IO request was a big improvement as was keeping the template in memory
instead of reading it in from disk)

Then again, I was pushing what I thought I was capable of, so returning just
  blocked yes/no
instead of the whole 'Final results:' info would probably be faster.
Maybe even consistently measurably faster :)  How much 'in-memory'
work needs to be removed before noticing a consistent improvement?

And _not_ checking to see if the action files have been modified makes
a noticeable difference - ie. jcc.c

@@ -5130,14 +5181,22 @@
             }
 #endif
          }
       }

-      if (continue_chatting && any_loaded_file_changed(csp))
+/*
+ *    don't check for action/filter file changes if processing cgi requests
+ *
+ *    XXX skip disk access only for show-url-final-info?
+ *        (ie. any_loaded_file_changed calling stat on all the config files)
+ *        or are other cgi calls expected to be called in a tight loop?
+ */
+      if (continue_chatting && !(csp->flags & CSP_FLAG_CRUNCHED) &&
any_loaded_file_changed(csp))
       {
          continue_chatting = 0;
          config_file_change_detected = 1;
+         log_error(LOG_LEVEL_INFO, "continue_chatting cleared,
config_file_change_detected set");
       }
 #ifdef FEATURE_HTTPS_INSPECTION
       if (continue_chatting && client_use_ssl(csp) &&
          csp->ssl_with_client_is_opened == 0)
       {

Again - blame the over-generalization on my ignorance/inability.  It
was a lot easier to convince myself of the correctness of
  "!(csp->flags & CSP_FLAG_CRUNCHED)"
than it was to figure out how to restrict the any_loaded_file_changed
test to only  happen if the user was calling cgi_show_url_info_x or
cgi_show_url_final_info


> I'm also not sure that it has to show up in the menu.

It doesn't, but why not?  In the menu makes it easier for the casual
user to discover vs. having to look thru the source code to figure out
that hidden goodies there are in the CGI functions.

Lee