Home Tornado's AsyncHTTPClient subclass
Reply: 2

Tornado's AsyncHTTPClient subclass

generationX
1#
generationX Published in 2018-01-11 15:56:25Z

(python 3.6.4 and tornado 4.5.3)

When using : http_client = tornado.httpclient.AsyncHTTPClient() the asynchronous http request fetching works fine.

But trying to define and use a subclass of AsyncHTTPClient got me into some kind of a deadlock when running the program (as opposed to the synchronous HTTPClient class where a subclass of it functioned well)

*Please correct me: If Tornado's AsynchHTTPClient class adheres to / inherits from the Configurable interface / abstract class , Then how it is possible to build objects from it? (Same rules as in Java?) . Is it that somehow something picks a default implementation for it among several built in ones?

"Configurable subclasses must define the class methods configurable_base and configurable_default, and use the instance method initialize instead of init. " - Will The default ctor in the case of inheritance will call super.init ? Is this the reason for the problem?

From the Documentation it appears that inheriting from AsyncHTTPClient is not recommended / not a valid method to use it:

    http_client = AsyncHTTPClient()
    http_client.fetch("http://www.google.com/", handle_response)

The constructor for this class is magic in several respects: It actually creates an instance of an implementation-specific subclass, and instances are reused as a kind of pseudo-singleton (one per .IOLoop). The keyword argument force_instance=True can be used to suppress this singleton behavior. Unless force_instance=True is used, no arguments should be passed to the AsyncHTTPClient constructor. The implementation subclass as well as arguments to its constructor can be set with the static method configure() All AsyncHTTPClient implementations support a defaults keyword argument, which can be used to set default values for HTTPRequest attributes. For example::

AsyncHTTPClient.configure(
    None, defaults=dict(user_agent="MyUserAgent"))
# or with force_instance:
client = AsyncHTTPClient(force_instance=True,
    defaults=dict(user_agent="MyUserAgent"))

additional questions:

1)Does buffering the response is a matter of choice?

3)When should I use the class tornado.web.RequestHandler ?


No errors at the moment but I'm not receiving a response after the actual fetch.

import sys
from tornado import ioloop, gen, httpclient

Under class SimpleAsyncHTTPClient(httpclient.AsyncHTTPClient):

#had to add this one (abstract function empty implementation? )
# I think that's the troublemaker
def fetch_impl(self, request, callback):
    pass

@gen.coroutine
def get(self, url):

    method = 'GET'
    print('send Async GET request ')
    res = yield self._fetch(url, method)
    print('after _fetch ...')
    return res

@gen.coroutine
def _fetch(self, url, method):
    print('send Asynchronous request ...')
    res = yield self.fetch(url, method=method)
    print('got a response')
    return res

Under a global:

@gen.coroutine

def ioloop_task():

yield gen.sleep(3)
url = 'http://google.com'
http_client = SimpleAsyncHTTPClient()

res = yield http_client.get(url)
res_code =  res.code
res_body =  res.body
print('return code: {}, body: {}....'.format(res_code, res_body[:60]))

print('do other stuff ....')
yield gen.sleep(2)

print(' task completed')
xyres
2#
xyres Reply to 2018-01-12 18:20:03Z

Since you've updated your question with additional details, and it has become very different from the original, I'd add another answer.

The magic behind AsyncHTTPClient

Consider this code:

http_client = AsyncHTTPClient()
print(http_client.__class__)

# Output:
<class 'tornado.simple_httpclient.SimpleAsyncHTTPClient'>

As you can see, http_client is an instance of SimpleAsyncHTTPClient, not AsyncHTTPClient. So, what's going on here?

If you look at the source code of AsyncHTTPClient, you'll see that it is inheriting the tornado.utils.Configurable class.

The most important piece of code in Configurable class is the __new__ method which is responsible for all the magic. If you look at its source code, you'll find that it will create an instance of whatever class is returned by the classmethod configurable_default.

Now, look at the source code of AsyncHTTPClient.configurable_default. It is returning the SimpleAsyncHTTPClient class, and that is why the object we created above (http_client), is the instance of SimpleAsyncHTTPClient, not of AsyncHTTPClient.

Finally, yes, you're right that you need to create the fetch_impl method in your subclass. Because AsyncHTTPClient will call the self.fetch_impl method. You can see this in this line in source code.

Though, fetch_impl hasn't been implemented in AsyncHTTPClient class, but it has been implemented in SimpleAsyncHTTPClient. You can find it here.


How to successfully subclass AsyncHTTPClient?

I'd start by a looking at the source code of SimpleAsyncHTTPClient and modify it to suit your needs.

xyres
3#
xyres Reply to 2018-01-12 07:32:33Z

But why? You can just use AsyncHTTPClient.

If you're doing this for learning, here's a general outline of the concepts:

  1. yield and await are for pausing a coroutine. But they're not the same. However, in Tornado, when the yield keyword is in a gen.coroutine decorated function, they both do the same things - that is suspending a coroutine until a future (or an awaitable) is complete.

  2. return is used for returning something from a function, I'm sure you already knew that.


When to use yield and when to use await?

If you're using Python older than 3.5, use @gen.coroutine ... yield syntax. For 3.5+, you should use the async def ... await syntax. Also, Tornado will drop support for older Python versions soon, so it's better to adhere to the new syntax.

Which brings me to point out that mixing these two doesn't seem a good idea. Get rid of the old syntax by replacing @gen.coroutine with async and yield with await.


A few comments about your code:

Let's take a look at the _fetch and get methods. Neither of them is returning anything.

To elaborate, in get method at res = await self._fetch(url, method), the res will be None because _fetch isn't returning anything.
And the same this is happening in ioloop_task at res = http_client.get(url). res will be None because get isn't returning anything.

To fix this, simply return the res object from both _fetch and get methods.

Also, you don't really need the _fetch method. You can do those things in the get method:

async def get(url):
    ...
    res = await self.fetch(url, method=method)
    return res

In fact, you don't even need the async ... await syntax either. This will work just as fine:

def get(url):
    ...
    return self.fetch(url, method=method)

Why the second code example will work fine, I leave it as an exercise for you to figure out.

You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.323813 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO