Home Why can the C# HttpClient not call this URL (always times out)?
Reply: 0

Why can the C# HttpClient not call this URL (always times out)?

user9894
1#
user9894 Published in September 21, 2018, 8:04 am

I've been developing an application that determines information about web pages. One of the components of this involves making an HTTP GET request to a URL, grabbing the HTML and analysing it. This has worked fine with every URL I've thrown at it, apart from one...

The culprit is the .NET HttpClient that always seems to timeout requesting any URL within the problem domain. However, the same URL requested with a browser returns content within milliseconds. Nothing about the Headers seems unusual.

Upping the timeout simply results in it taking longer to bomb-out. I've tried minutes with the same result. I've tried various things, such as setting the User Agent string to that of Chrome but to no avail.

The domain in question is: http://careers.adidas-group.com Note the same site also runs on HTTPS at https://careers.adidas-group.com (it has a valid cert). Using either protocol results in the same error.

I can show the problem with a simple C# console app, shown below:

static void Main(string[] args)
{
    string url = "http://careers.adidas-group.com";

    var client = new HttpClient
    {
        Timeout = TimeSpan.FromSeconds(10)
    };

    using (var message = new HttpRequestMessage(HttpMethod.Get, url))
    {
        using (var httpResponse = Task.Run(() => client.SendAsync(message)).Result)
        {
            Console.WriteLine("{0}: {1}", httpResponse.StatusCode, httpResponse.ReasonPhrase);
        }
    }

    Console.ReadLine();
}

Note in the above example I set the timeout to 10 seconds, simply to expedite the problem - however, increasing the timeout makes no difference.

The same code with a different URL (such as https://stackoverflow.com/) runs fine.

Also note the code above is simplified to run as a Console App. My actual code runs properly asynchronously (using await) in an async MVC controller method - I'm just using Task.Run(() => ) to make it work with the context of a synchronous Main method in the example. But it makes no difference to the outcome. (The actual exception is a "Task was cancelled" but that appears to be a sympton of the time-out, rather than the actual issue).

Can anyone explain to me why this is happening (is it something about the server configuration?) and what, if anything, I can do to make HttpClient fulfill the request? Thanks.

share|improve this question
  • Could it be the useragent headers? – maccettura Feb 14 at 15:17
  • Happens with site test tool as well: redbot.org/?uri=http%3A%2F%2Fcareers.adidas-group.com%2F – Julian Reschke Feb 14 at 15:51
  • @JulianReschke No, I hadn't, but it's interesting the same timeout issue occurs using that tool. The server must be rejecting "bots", but I'm not clear on what criteria. – Dan Diplo Feb 14 at 15:55
  • Same result with curl – Max Feb 14 at 15:56
  • @maccettura That's was one of my first thoughts, but I've tried setting the user-agent string in the request to some popular browser agents, but it made no difference. – Dan Diplo Feb 14 at 16:04

2 Answers 2

active oldest votes
up vote 2 down vote
You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.358165 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO