[Privoxy-users] gitweb performance/errors

[Fabian Keil]

Fabian Keil via Privoxy-users <privoxy-users at lists.privoxy.org> wrote on 2026-01-25 at 19:01:40:

> tmknight88--- via Privoxy-users <privoxy-users at lists.privoxy.org> wrote on 2026-01-25 at 12:06:35:
> 
> > Hi, I wanted to report (assuming you are unaware) that the gitweb interface
> > of privoxy.git performance has degraded since approximately 23 Jan 2026.
> > For example, the summary page can take many seconds to load and frequently
> > times out with a generic nginx error message.
> 
> Thanks for the report.
> 
> I was already aware of the problem but haven't had time to look into solving it.
> 
> The robots.txt already disallows crawling of /gitweb and a bunch
> of obvious "AI" bots receive static error pages, but since about
> three days ago there are simply too many requests coming in from
> clients that ignore the robots.txt and don't use obvious "AI" bot
> User-Agent headers either.
> 
> Looks like we need more crawler counter measures.

I've added a cookie barrier which seems to keep the crawlers at bay for now.

Fabian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.privoxy.org/pipermail/privoxy-users/attachments/20260129/a8eada82/attachment.bin>