asyncio¶
New in version 2.0.
Scrapy has partial support for asyncio
. After you install the
asyncio reactor, you may use asyncio
and
asyncio
-powered libraries in any coroutine.
Installing the asyncio reactor¶
To enable asyncio
support, set the TWISTED_REACTOR
setting to
'twisted.internet.asyncioreactor.AsyncioSelectorReactor'
.
If you are using CrawlerRunner
, you also need to
install the AsyncioSelectorReactor
reactor manually. You can do that using
install_reactor()
:
install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')
Using custom asyncio loops¶
You can also use custom asyncio event loops with the asyncio reactor. Set the
ASYNCIO_EVENT_LOOP
setting to the import path of the desired event loop class to
use it instead of the default asyncio event loop.
Windows-specific notes¶
The Windows implementation of asyncio
can use two event loop
implementations:
SelectorEventLoop
, default before Python 3.8, required when using Twisted.ProactorEventLoop
, default since Python 3.8, cannot work with Twisted.
So on Python 3.8+ the event loop class needs to be changed.
Changed in version 2.6.0: The event loop class is changed automatically when you change the
TWISTED_REACTOR
setting or call
install_reactor()
.
To change the event loop class manually, call the following code before installing the reactor:
import asyncio
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
You can put this in the same function that installs the reactor, if you do that
yourself, or in some code that runs before the reactor is installed, e.g.
settings.py
.
Note
Other libraries you use may require
ProactorEventLoop
, e.g. because it supports
subprocesses (this is the case with playwright), so you cannot use
them together with Scrapy on Windows (but you should be able to use
them on WSL or native Linux).
Awaiting on Deferreds¶
When the asyncio reactor isn’t installed, you can await on Deferreds in the
coroutines directly. When it is installed, this is not possible anymore, due to
specifics of the Scrapy coroutine integration (the coroutines are wrapped into
asyncio.Future
objects, not into
Deferred
directly), and you need to wrap them into
Futures. Scrapy provides two helpers for this:
Tip
If you need to use these functions in code that aims to be compatible
with lower versions of Scrapy that do not provide these functions,
down to Scrapy 2.0 (earlier versions do not support
asyncio
), you can copy the implementation of these functions
into your own code.