Add a receive-buffer-size directive
... which can be used to set the size of the previously statically
allocated buffer in handle_established_connection().
Increasing the buffer size increases Privoxy's memory usage but
can lower the number of context switches and thereby reduce the
cpu usage and potentially increase the throughput.
This is mostly relevant for fast network connections and
large downloads that don't require filtering.
Currently BUFFER_SIZE is kept as default and lower limit
but the default should be increased after some more testing.
A dtrace command like:
sudo dtrace -n 'syscall::read:return /execname == "privoxy"/ { @[execname] = llquantize(arg0, 10, 0, 5, 20); @m = max(arg0)}'
can be used to properly tune the receive-buffer-size.
If the buffer is too large it will increase Privoxy's memory
footprint without any benefit. As the memory is (currently)
cleared before using it, a buffer that is too large can
actually reduce the throughput.
Things could be improved further by upwards scaling the buffer
dynamically based on how much of the previous allocation
was actually used.
Additionally the buffer should be referenced through csp and
also be used for other receive-related functions.
Measured throughput when using four connections to
constantly request a 10 MB file:
~320 MB/s with the default
~400 MB/s with "receive-buffer-size 8192"
~490 MB/s with "receive-buffer-size 16384"
~610 MB/s with "receive-buffer-size 32768"
~700 MB/s with "receive-buffer-size 65536"
~755 MB/s with "receive-buffer-size 131072"
~795 MB/s with "receive-buffer-size 262144"
~804 MB/s with "receive-buffer-size 524288"
~798 MB/s with "receive-buffer-size
1048576"
~780 MB/s with "receive-buffer-size
2097152"
Sponsored by: Robert Klemme