[Privoxy-users] Privoxy-users Digest, Vol 41, Issue 7
Robert Klemme
rklemme at klemmeconsulting.com
Thu Jul 23 03:55:51 UTC 2020
I don't know the status of this project but it seems like the
privoxy-filter-test might be helpful in this case. If it is dormant, maybe
we should resurrect development?
https://www.fabiankeil.de/sourcecode/pft/
On Wed, Jul 22, 2020 at 10:15 PM <privoxy-users-request at lists.privoxy.org>
wrote:
> Send Privoxy-users mailing list submissions to
> privoxy-users at lists.privoxy.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.privoxy.org/mailman/listinfo/privoxy-users
> or, via email, send a message with subject or body 'help' to
> privoxy-users-request at lists.privoxy.org
>
> You can reach the person managing the list at
> privoxy-users-owner at lists.privoxy.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Privoxy-users digest..."
>
>
> Today's Topics:
>
> 1. external-filter example (U.Mutlu)
> 2. Re: external-filter example (U.Mutlu)
> 3. Re: external-filter example (Lee)
> 4. Re: external-filter example (U.Mutlu)
> 5. Re: external-filter example (U.Mutlu)
> 6. Re: external-filter example (Lee)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 22 Jul 2020 19:03:04 +0200
> From: "U.Mutlu" <um at mutluit.com>
> To: privoxy-users at lists.privoxy.org
> Subject: [Privoxy-users] external-filter example
> Message-ID: <5F187148.8030608 at mutluit.com>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Hi,
> as a newbie to privoxy I've built the developer version (v3.0.29) with
> ./configure --with-mbedtls and installed it. It works.
>
> I'm now trying to create an external-filter for experimenting. I did these
> steps:
>
> config:
> temporary-directory ./myTmp
> (and even created that dir manually in the config-dir and set the
> ownership
> appropriately)
>
> user.action:
> myfilters = +filter{html-annoyances} +filter{js-annoyances}
> +filter{all-popups}\
> +filter{webbugs} +filter{banners-by-size}\
> +external-filter{myTee}
>
> user.filter:
> EXTERNAL-FILTER: myTee MY TEE TEST
> /usr/bin/tee /tmp/myTee_Test.txt
>
> But privoxy won't start, writes this error to log:
> 2020-07-22 18:26:29.533 7f6ca8cfe700 Fatal error: can't load actions file
> '/usr/local/etc/privoxy/user.action': invalid alias line (64): myfilters =
> +filter{html-annoyances} +filter{js-annoyances} +filter{all-popups}
> +filter{webbugs} +filter{banners-by-size}
> +external-filter{myTee}
>
> What's the reason, and how to fix it? Thx.
>
> Btw, is the word "myfilters" in the config files a hardcoded special word
> or
> is it just a user-defined one?
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 22 Jul 2020 19:39:04 +0200
> From: "U.Mutlu" <um at mutluit.com>
> To: privoxy-users at lists.privoxy.org
> Subject: Re: [Privoxy-users] external-filter example
> Message-ID: <5F1879B8.3090706 at mutluit.com>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Update: sorry, had forgotten to build with "--enable-external-filters".
> After rebuilding and reinstalling it now starts ok,
> but the external-filter does not get invoked :-(
> How to fix that?
>
>
> U.Mutlu wrote on 07/22/2020 07:03 PM:
> > Hi,
> > as a newbie to privoxy I've built the developer version (v3.0.29) with
> > ./configure --with-mbedtls and installed it. It works.
> >
> > I'm now trying to create an external-filter for experimenting. I did
> these steps:
> >
> > config:
> > temporary-directory ./myTmp
> > (and even created that dir manually in the config-dir and set the
> ownership
> > appropriately)
> >
> > user.action:
> > myfilters = +filter{html-annoyances} +filter{js-annoyances}
> > +filter{all-popups}\
> > +filter{webbugs} +filter{banners-by-size}\
> > +external-filter{myTee}
> >
> > user.filter:
> > EXTERNAL-FILTER: myTee MY TEE TEST
> > /usr/bin/tee /tmp/myTee_Test.txt
> >
> > But privoxy won't start, writes this error to log:
> > 2020-07-22 18:26:29.533 7f6ca8cfe700 Fatal error: can't load actions file
> > '/usr/local/etc/privoxy/user.action': invalid alias line (64): myfilters
> =
> > +filter{html-annoyances} +filter{js-annoyances} +filter{all-popups}
> > +filter{webbugs} +filter{banners-by-size}
> +external-filter{myTee}
> >
> > What's the reason, and how to fix it? Thx.
> >
> > Btw, is the word "myfilters" in the config files a hardcoded special
> word or
> > is it just a user-defined one?
>
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 22 Jul 2020 15:29:07 -0400
> From: Lee <ler762 at gmail.com>
> To: "U.Mutlu" <um at mutluit.com>
> Cc: privoxy-users at lists.privoxy.org
> Subject: Re: [Privoxy-users] external-filter example
> Message-ID:
> <
> CAD8GWsttbjoSRosRAWRkPNtCOU353y92z+eKJzOfXnptHbF-ZA at mail.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
>
> On 7/22/20, U.Mutlu <um at mutluit.com> wrote:
> > Update: sorry, had forgotten to build with "--enable-external-filters".
> > After rebuilding and reinstalling it now starts ok,
> > but the external-filter does not get invoked :-(
> > How to fix that?
>
> I've never tried using external filters, so I'm not going to be a
> whole lot of help but
> https://www.privoxy.org/user-manual/actions-file.html#FILTER
> (Note: as of version 3.0.3 plain text documents are exempted from
> filtering, because web servers often use the text/plain MIME type for
> all files whose type they don't know.)
>
> Have you added
> debug 64 # debug regular expression filters
> to the config file to see if privoxy tries to run the filter?
>
> https://www.privoxy.org/gitweb/?p=privoxy.git;a=blob;f=filters.c;h=b490353b2d93c5b1cc40ab80a22cc717372f6cda;hb=HEAD#l1834
>
> If the filter isn't being called, maybe you need
> https://www.privoxy.org/user-manual/filter-file.html
> Enabled content filters are applied to any content whose "Content
> Type" header is recognised as a sign of text-based content, with the
> exception of text/plain. Use the force-text-mode action to also filter
> other content.
>
> > U.Mutlu wrote on 07/22/2020 07:03 PM:
> >> Hi,
> <.. snip ..>
> >> I'm now trying to create an external-filter for experimenting. I did
> these
> >> steps:
> <.. snip ..>
> >> user.action:
> >> myfilters = +filter{html-annoyances} +filter{js-annoyances}
> >> +filter{all-popups}\
> >> +filter{webbugs} +filter{banners-by-size}\
> >> +external-filter{myTee}
> <.. snip ..>
> >> Btw, is the word "myfilters" in the config files a hardcoded special
> word
> >> or is it just a user-defined one?
>
> It's a user-defined alias
> https://www.privoxy.org/user-manual/actions-file.html#ALIASES
>
> Lee
>
>
> ------------------------------
>
> Message: 4
> Date: Wed, 22 Jul 2020 23:06:46 +0200
> From: "U.Mutlu" <um at mutluit.com>
> To: privoxy-users at lists.privoxy.org
> Subject: Re: [Privoxy-users] external-filter example
> Message-ID: <5F18AA66.2020906 at mutluit.com>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Lee wrote on 07/22/2020 09:29 PM:
> > On 7/22/20, U.Mutlu <um at mutluit.com> wrote:
> >> Update: sorry, had forgotten to build with "--enable-external-filters".
> >> After rebuilding and reinstalling it now starts ok,
> >> but the external-filter does not get invoked :-(
> >> How to fix that?
> >
> > Have you added
> > debug 64 # debug regular expression filters
> > to the config file to see if privoxy tries to run the filter?
> >
> https://www.privoxy.org/gitweb/?p=privoxy.git;a=blob;f=filters.c;h=b490353b2d93c5b1cc40ab80a22cc717372f6cda;hb=HEAD#l1834
>
> Thx. Now tried with "debug 64" and it shows that it recognizes the said
> external-filter, but IMHO the action for it is not executed:
>
> ....
> 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter
> "user-agent"
> ("Tags the request with the complete User-Agent header.")
> 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Adding re_filter job
> 's@^User-Agent:.*@$0 at i' to filter user-agent succeeded.
> 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter
> "referer"
> ("Tags the request with the complete Referer header.")
> 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Adding re_filter job
> 's@^Referer:.*@$0 at i' to filter referer succeeded.
> 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter
> "content-type" ("Tags the request with the content type declared by the
> server.")
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> 's@^Content-Type:\s*([^;]+).*@$1 at i' to filter content-type succeeded.
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> "privoxy-control" ("Creates tags with the content of X-Privoxy-Control
> headers.")
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> 's@^X-Privoxy-Control:\s*@@i' to filter privoxy-control succeeded.
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> "privoxy-control" ("Removes X-Privoxy-Control headers.")
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> 's@^X-Privoxy-Control:.*@@i' to filter privoxy-control succeeded.
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> "privoxy-control" ("Creates tags with the content of X-Privoxy-Control
> headers.")
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> 's@^X-Privoxy-Control:\s*@@i' to filter privoxy-control succeeded.
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> "privoxy-control" ("Removes X-Privoxy-Control headers.")
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> 's@^X-Privoxy-Control:.*@@i' to filter privoxy-control succeeded.
> 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter "myTee"
> ("MY
> TEE TEST")
> ...
>
> There is no other log entry regarding this "myTee" filter.
> Ie. it seems the action for it does not get executed.
>
> > If the filter isn't being called, maybe you need
> > https://www.privoxy.org/user-manual/filter-file.html
> > Enabled content filters are applied to any content whose "Content
> > Type" header is recognised as a sign of text-based content, with the
> > exception of text/plain. Use the force-text-mode action to also filter
> > other content.
>
> But doing so seems to be dangerous according to the doc
> https://www.privoxy.org/user-manual/actions-file.html#FORCE-TEXT-MODE
> It says:
> "Warning - Think twice before activating this action. Filtering binary
> data
> with regular expressions can cause file damage."
>
>
> I just want to get a copy of the whole HTML that privoxy gets from the
> server,
> either the unmodified raw data, or the filtered version after
> privoxy applies the other defined filters.
>
> I think the HTML is always a text file, even if it contains binary data
> like picture images etc. (they are somehow text encoded I think using
> base64 encoding or so). Is this assumption wrong?
>
> If I read the doc right then the external-filter handler has to write
> the received data either verbatim (ie. no filtering), or a filtered
> version of the data, back to stdout, so privoxy can continue doing its job.
>
> A working example for such an external-filter would be helpful.
> Unfortunately I couldn't find any on the web :-(
>
>
>
>
>
> ------------------------------
>
> Message: 5
> Date: Thu, 23 Jul 2020 00:01:47 +0200
> From: "U.Mutlu" <um at mutluit.com>
> To: privoxy-users at lists.privoxy.org
> Subject: Re: [Privoxy-users] external-filter example
> Message-ID: <5F18B74B.1010305 at mutluit.com>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> It's funny that nothing of default.action, user.action, and my.action
> applies to the below URL [I'm using the default configs it has installed].
> (my.action and my.filter is just my experimental stuff with the said
> "myTee"
> external-filter (now moved from user.* to my.*)):
>
> http://config.privoxy.org/show-url-info?url=www.startpage.com
>
> Matches for http://www.startpage.com:
> In file: match-all.action View
> {+change-x-forwarded-for{block}
> +client-header-tagger{css-requests}
> +client-header-tagger{image-requests}
> +client-header-tagger{range-requests}
> +hide-from-header{block}
> +set-image-blocker{pattern} }
> /
> In file: default.action View
> (no matches in this file)
> In file: user.action View
> (no matches in this file)
> In file: my.action View
> (no matches in this file)
>
>
>
> U.Mutlu wrote on 07/22/2020 11:06 PM:
> > Lee wrote on 07/22/2020 09:29 PM:
> >> On 7/22/20, U.Mutlu <um at mutluit.com> wrote:
> >>> Update: sorry, had forgotten to build with "--enable-external-filters".
> >>> After rebuilding and reinstalling it now starts ok,
> >>> but the external-filter does not get invoked :-(
> >>> How to fix that?
> >>
> >> Have you added
> >> debug 64 # debug regular expression filters
> >> to the config file to see if privoxy tries to run the filter?
> >>
> https://www.privoxy.org/gitweb/?p=privoxy.git;a=blob;f=filters.c;h=b490353b2d93c5b1cc40ab80a22cc717372f6cda;hb=HEAD#l1834
> >>
> >
> > Thx. Now tried with "debug 64" and it shows that it recognizes the said
> > external-filter, but IMHO the action for it is not executed:
> >
> > ....
> > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter
> "user-agent"
> > ("Tags the request with the complete User-Agent header.")
> > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Adding re_filter job
> > 's@^User-Agent:.*@$0 at i' to filter user-agent succeeded.
> > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter
> "referer"
> > ("Tags the request with the complete Referer header.")
> > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Adding re_filter job
> > 's@^Referer:.*@$0 at i' to filter referer succeeded.
> > 2020-07-22 22:31:57.821 7f838df67700 Re-Filter: Reading in filter
> > "content-type" ("Tags the request with the content type declared by the
> server.")
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> > 's@^Content-Type:\s*([^;]+).*@$1 at i' to filter content-type succeeded.
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> > "privoxy-control" ("Creates tags with the content of X-Privoxy-Control
> headers.")
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> > 's@^X-Privoxy-Control:\s*@@i' to filter privoxy-control succeeded.
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> > "privoxy-control" ("Removes X-Privoxy-Control headers.")
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> > 's@^X-Privoxy-Control:.*@@i' to filter privoxy-control succeeded.
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> > "privoxy-control" ("Creates tags with the content of X-Privoxy-Control
> headers.")
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> > 's@^X-Privoxy-Control:\s*@@i' to filter privoxy-control succeeded.
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> > "privoxy-control" ("Removes X-Privoxy-Control headers.")
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Adding re_filter job
> > 's@^X-Privoxy-Control:.*@@i' to filter privoxy-control succeeded.
> > 2020-07-22 22:31:57.822 7f838df67700 Re-Filter: Reading in filter
> "myTee" ("MY
> > TEE TEST")
> > ...
> >
> > There is no other log entry regarding this "myTee" filter.
> > Ie. it seems the action for it does not get executed.
> >
> >> If the filter isn't being called, maybe you need
> >> https://www.privoxy.org/user-manual/filter-file.html
> >> Enabled content filters are applied to any content whose "Content
> >> Type" header is recognised as a sign of text-based content, with the
> >> exception of text/plain. Use the force-text-mode action to also filter
> >> other content.
> >
> > But doing so seems to be dangerous according to the doc
> > https://www.privoxy.org/user-manual/actions-file.html#FORCE-TEXT-MODE
> > It says:
> > "Warning - Think twice before activating this action. Filtering binary
> data
> > with regular expressions can cause file damage."
> >
> >
> > I just want to get a copy of the whole HTML that privoxy gets from the
> server,
> > either the unmodified raw data, or the filtered version after
> > privoxy applies the other defined filters.
> >
> > I think the HTML is always a text file, even if it contains binary data
> > like picture images etc. (they are somehow text encoded I think using
> > base64 encoding or so). Is this assumption wrong?
> >
> > If I read the doc right then the external-filter handler has to write
> > the received data either verbatim (ie. no filtering), or a filtered
> > version of the data, back to stdout, so privoxy can continue doing its
> job.
> >
> > A working example for such an external-filter would be helpful.
> > Unfortunately I couldn't find any on the web :-(
>
>
>
> ------------------------------
>
> Message: 6
> Date: Wed, 22 Jul 2020 23:15:18 -0400
> From: Lee <ler762 at gmail.com>
> To: "U.Mutlu" <um at mutluit.com>
> Cc: privoxy-users at lists.privoxy.org
> Subject: Re: [Privoxy-users] external-filter example
> Message-ID:
> <
> CAD8GWsuRNCLPV8MAcKFi5Yrqgs3WSd5ye7XwNd1r+NWYZ6sbmQ at mail.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
>
> On 7/22/20, U.Mutlu <um at mutluit.com> wrote:
> > Lee wrote on 07/22/2020 09:29 PM:
> >>
> >> Have you added
> >> debug 64 # debug regular expression filters
> >> to the config file to see if privoxy tries to run the filter?
> >
> > Thx. Now tried with "debug 64" and it shows that it recognizes the said
> > external-filter, but IMHO the action for it is not executed:
> <.. snip ..>
> correct, a filter being applied looks like
> 2020-07-22 21:38:06.979 000004a0 Re-Filter: filtering
> 'Accept-Encoding: gzip, deflate, br' (size 34) with
> 'no-brotli-accepted' ...
> 2020-07-22 21:38:06.979 000004a0 Re-Filter: ... produced 1 hits (new size
> 30).
>
> > There is no other log entry regarding this "myTee" filter.
> > Ie. it seems the action for it does not get executed.
>
> Did you add the filter call to your user.action? for example:
> A long time ago flamingtext made you wait for the pretty picture. I
> got annoyed, so I added a filter to remove the wait. In user.filter:
> #
> # flamingtest: removes image wait timer
> #
> # flamingtest.com - removes the timeout obstruction so you don't have
> to wait for your image
> FILTER: flamingtext no waiting
>
> s@<META HTTP-EQUIV=Refresh CONTENT="15; URL=@<META HTTP-EQUIV=Refresh
> CONTENT="1; URL=@ig
>
> Then activate the filter just for the flamingtext domain in user.actions:
> ## don't wait for flamingtext.com
> { +filter{flamingtext} }
> .flamingtext.com/
>
> and now, http://config.privoxy.org/show-url-info?url=flamingtext.com
> gives me
> In file: user.action
> {+filter{flamingtext} }
> .flamingtext.com/
> {+server-header-tagger{content-type} }
> /
> {-deanimate-gifs }
> .flamingtext.com/
>
> >> If the filter isn't being called, maybe you need
> >> https://www.privoxy.org/user-manual/filter-file.html
> >> Enabled content filters are applied to any content whose "Content
> >> Type" header is recognised as a sign of text-based content, with the
> >> exception of text/plain. Use the force-text-mode action to also filter
> >> other content.
> >
> > But doing so seems to be dangerous according to the doc
> > https://www.privoxy.org/user-manual/actions-file.html#FORCE-TEXT-MODE
> > It says:
> > "Warning - Think twice before activating this action. Filtering binary
> data
> > with regular expressions can cause file damage."
> >
> >
> > I just want to get a copy of the whole HTML that privoxy gets from the
> > server,
> > either the unmodified raw data, or the filtered version after
> > privoxy applies the other defined filters.
> >
> > I think the HTML is always a text file,
>
> oops, no, not at all
>
> > even if it contains binary data
> > like picture images etc. (they are somehow text encoded I think using
> > base64 encoding or so). Is this assumption wrong?
>
> it is wrong
>
> > If I read the doc right then the external-filter handler has to write
> > the received data either verbatim (ie. no filtering), or a filtered
> > version of the data, back to stdout, so privoxy can continue doing its
> job.
> >
> > A working example for such an external-filter would be helpful.
> > Unfortunately I couldn't find any on the web :-(
>
> How about
>
> http://config.privoxy.org/user-manual/actions-file.html#SERVER-HEADER-TAGGER
>
> Add this to your user.action:
> # Tag every request with the content type declared by the server
> {+server-header-tagger{content-type}}
> /
>
> # apply myTee filter to all text/html content
> { +external-filter{myTee} +force-text-mode}
> TAG:^text/html
>
> and the bit you have in user.filter should be something like
>
> EXTERNAL-FILTER: myTee make a copy of reveived text
> /usr/bin/tee myCopiedText
>
> maybe that will work?
>
> Lee
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Privoxy-users mailing list
> Privoxy-users at lists.privoxy.org
> https://lists.privoxy.org/mailman/listinfo/privoxy-users
>
>
> ------------------------------
>
> End of Privoxy-users Digest, Vol 41, Issue 7
> ********************************************
>
--
Robert Klemme Jr.
President, Klemme Consulting LLC.
RKlemme at Klemmeconsulting.com
More information about the Privoxy-users
mailing list