[Privoxy-devel] https-inspection challenges

[Roland Rosenfeld]

Hi all!

I did my first steps on https-inspection of privoxy 2.0.29 (git
8097d5c7) today, to find out whether it's useful and how it handles
bad certificates etc.

First minor issue I tripped on, is that the user.action still uses the
old label:
 {+enable-https-filtering}
which should be
 {+https-inspection}


Second I noted, that the cgi editor (eafu) currently does not support
https-inspection (is this intended or a bug?).


Next problem was, that I have a default upstream proxy defined.  This
resulted in an timeout of the client
 env HTTPS_PROXY=localhost:8118 curl --connect-timeout 10 https://badssl.com
and the following privoxy debug log:

2020-06-05 14:42:44.142 7f4aee1ed700 Connect: via [upstreamproxy]:8080 to: badssl.com:443
2020-06-05 14:42:44.142 7f4aee1ed700 Connect: Performing the TLS/SSL handshake with client. Hash of host: 060d2e7cc8f6e09e20799ba8491a76aa
2020-06-05 14:42:54.142 7f4aee1ed700 Error: medtls_ssl_handshake with client failed: SSL - The connection indicated an EOF
2020-06-05 14:42:54.142 7f4aee1ed700 Error: Failed to open a secure connection with the client
2020-06-05 14:42:54.142 7f4aee1ed700 Connect: Closing client socket 6. Keep-alive: 0. Socket alive: 0. Data available: 0. Configuration file change detected: 0. Requests received: 1.

After defining an exception of the upstream proxy
 forward .badssl.com .
it worked as expected.
(This works as a workaround, but I fear, that this is a bug in the new
https-inspection code, it should be possible to combine forwarding and
https-inspection, shouldn't it?)


After this I was able to reach the website and now I got the expected
cert error.  I uploaded the generated root cert as Authority to my
browser but now https-inspections seems to work for me.


I tried several broken certificates from https://badssl.com, most of
them are correctly detected, I'm only a little irritated about the
feedback the user gets, for example on https://expired.badssl.com/ I
see a valid SSL cert in the browser (generated by privoxy via my fake
CA) and some output that the certificate has expired with some info
about the cert.  That's okay, but I'd expect some hint on the output
page, that this page is generated by privoxy (maybe with a link to a
FAQ about https-inspection).


While most broken SSL sites were correctly detected, on
https://revoked.badssl.com/ privoxy doesn't notice that the
certificate has been revoked. Also https://pinning-test.badssl.com/ is
accepted, with bad HPKP.

Maybe you may want play more with badssl.com, which looks quite useful
for testing the https-inspection feature.


After grumbling that much about the https-inspection, don't get me
wrong: I think that it is really a great feature (much better than I
ever expected before, since I didn't had an idea how you could handle
broken server certs at all) and I like it very much.  It may be useful
in many situations not only for spam blocking but also for debugging
web pages etc.

I think that I will build the next Debian package with mbedtls enabled
and maybe with ca and certs directories preinstalled with correct
permissions and maybe a little README, that explains onboarding (what
directories have to exist with what permissions and how to create a CA
key/cert pair on Debian and where you will find a trusted-cas-file on
Debian etc.).

Greetings
Roland