.NET/C# multithreaded concurrency using the pit of HttpWebRequest

Little scum · Posted on 2021-12-20 15:21:04

When doing task scheduling, the task of scheduling is to simulate initiating an HTTP request, and find that when there are more tasks, the task scheduling seems to be like fake death.

The maximum default number of requested connections on a normal Windows system is 2, and on a server operating system, it defaults to 10. If you don't modify this concurrent connection limit,Then the number of http connections that the client can make at the same time is only 2 or 10。

System.Net.ServicePointManager.DefaultConnectionLimit This is the maximum number of connections that can get the current default setting, as shown in the figure below:

Login is visible.

Is there a default concurrency limit for HttpClient?

Prior to .Net 4.0, Http operations were relied upon by HttpWebRequest. It has a very conservative maximum of 2 concurrency limits for the same site by default, which often causes HttpWebRequests to not get the ideal speed by default (I guess this strategy frustrates many coders), and the value of App.config or ServicePointManager.DefaultConnectionLimit must be modified.

MS introduced an HttpClient class in .Net 4.5 to handle Http operations, and I thought HttpClient and HttpWebRequest followed the same policy. Today, when writing a multi-threaded download program, I used 10 concurrent connections and found that HttpClient does not have a concurrent number limit by default.

At first I thought it was. Net 4.5 removed this concurrency limit (after all, basically no one follows this standard now), and then rewrote the relevant code with WebRequest, and found that it was still 2 concurrency limit, and the value of ServicePointManager.DefaultConnectionLimit was also 2. That is, HttpClient is not controlled by the HttpWebRequest concurrency policy, and there is no system-level concurrency limit.

In addition, during testing, I found that HttpWebRequest does not default to a maximum of 2 concurrency for all addresses. For exampleThere is no concurrency limit for local HTTP address connections (http://localhost/*).。

C# multithreaded environment calls to HttpWebRequest concurrent connection limit

.net's HttpWebRequest or WebClient has a concurrent connection limit in the case of multithreading, which is 2 by default on desktop operating systems such as Windows XP and Windows 7, and 10 by default on server operating systems. If you don't change this concurrent connection limit, the number of HTTP connections that a client can make at a time is only 2 or 10. For some applications, such as browsers or spiders, the number of 2 or 10 concurrency is too small and greatly affects the performance of the application. This concurrent connection limit is due to the HTTP 1.0 and HTTP 1.1 standards that specify a maximum of 2 concurrent connections. However, mainstream browsers no longer follow this rule, but the .NET framework still follows this rule by default.

Many articles say that accessing HttpWebRequest asynchronously can improve concurrent performance, but I have tested that if you don't change the default number of concurrent connections, synchronous or asynchronous access performance is not good.

We can also set the maximum number of concurrent connections in app.config as follows:

Login is visible.

Reference article

The hyperlink login is visible.
The hyperlink login is visible.