Improve MultiHttpClient connection concurrency and reuse
authorAaron Schulz <aschulz@wikimedia.org>
Thu, 5 Sep 2019 15:30:36 +0000 (08:30 -0700)
committerMobrovac <mobrovac@wikimedia.org>
Tue, 10 Sep 2019 11:03:07 +0000 (11:03 +0000)
commit46531d62852239f620f7b7c0af1e5747a9006228
tree4517cea4131df07708fb445d7be532c1982c0bcc
parenta7f12ae596d611f5c84fbf85b150c90bf4dc4240
Improve MultiHttpClient connection concurrency and reuse

Use CURLMOPT_MAX_HOST_CONNECTIONS to enforce concurrent request limits.
This gives better concurrency than using na├»ve array_chunk() batches, which
were serialized and treated all URLs as pessimistically from the same host.

Allow connection reuse for multi-URL request batches. This avoids overhead
from reconnections and reduces the number of TIME_WAIT handles when many
batch operations happen in a short time frame. Previously, the use of the
CURLOPT_FORBID_REUSE flag meant that connections were cached but never
reused for multi-URL batches (only single-URL batches).

Connection limits can be verified by running large runMulti() batches
in an interactive shell and inspecting netstat for TCP connections.

Bug: T232128
Change-Id: I5c5f1eceb3fdb501a8f22f2b949756065f12379a
includes/libs/http/MultiHttpClient.php