Task: https://www.odoo.com/web#id=2081201&model=project.task
Branch: https://github.com/odoo-dev/odoo/tree/master-2081201-job_queue-juc
Pull Request: https://github.com/odoo/odoo/pull/41535
Odoo historically only supported two ways of processing tasks: sporadic HTTP
request and periodic CRON tasks. The first is the preferred way to exchange
information with the web application while the later is used to perform some
regular background batch operations.
Several models need to create sporadic background tasks assured to be processed
as soon as possible. The current strategy of doing so is to save the task in
some kind of queue later processed by a CRON. This strategy lacks reactivity due
to low frequency of CRON calls. e.g. (e)mail or marketing "send/start now"
button, payment transactions. Another strategy is to start the processing in a
brand new thread.
Those tasks are fine (to an extend) using a CRON because they do not need to
send back anything to the user. The user basically just fire-and-forget the
operation.
When it is required to communicate something back to the user, the CRONs become
useless because they lack a standard way to communicate back to the web
application. There is at the moment no other way then to do the heavy operation
right in the current HTTP Worker, blocking the UI of the user until it finishes
computing... or the process reaper kills the HTTP Worker because the time limit
was reached.
As a solution to the above problems, we introduce a new API to schedule and
process asynchronous tasks with optional web notification.
ir_async, a Job Queue.A new ir_async model has been created in the ORM, its job is to expose a (we
hope) simple backend-only API to synchronously create and asynchronously
execute background jobs on demand.
An example is worth a thousand docs, so here is how you create a background task:
class Foo(models.Model):
_name = 'foo'
def async_foo(self):
pass
def foo(self):
self.env['ir_async'].call(self.async_foo)
That's it, just pass a method bound to a recordset to the call static method
of the ir_async model, you have the guarantee the async_foo method will be
executed soon with the same environment (recordset, user and context). Here
the created task is anonymous, web notification are disabled. In other words
this is a pure backend only task similar to a CRON job.
The method signature is as follow:
def call(self, target, args=None, kwargs=None):
"""
partial-like API to create asynchronous jobs.
The method will be called later as ``target(*args, **kwargs)`` in a
new transaction using a copy of the current environment (user, context,
recordset). Arguments must be serializable in JSON.
The job's return value is stored in JSON in the ``payload`` field.
"""
Please note, beside the 5 common letters, the ir.async model in Odoo has
nothing to do with the asyncio standard library or the async/await
keywords.
To enable web-notification, use the call_notify static method. It behaves the
same as the above function apart that a message is sent via the longpolling bus
every time the job status changes. It also take an additional "description"
argument used to list the job in the systray.
def call_notify(self, description, method, *args, **kwargs):
"""
partial-like API to create asynchronous jobs with web notification.
"""
Notifications are:
created, the task has been enqueued ;processing, an async worker starts processing the task ;succeeded, the task finished with a result, the result is sent on the bus ;failed, the task finished due to an error, it is sent on the bus ;done, the task finished without error or someone called the completeir.async to notify the result of a succeeded task as beenAn example of a succeeded task is an export. When the export completes, the
XLSX file is saved in an attachment and the result is an ir.actions.act_url
to download the file. The frontend is expected to call the complete method
when the file has been successfully downloaded by the user.
class Foo(models.Model):
_name = 'foo'
def async_foo(self):
pass
def foo(self):
job = self.env['ir_async'].call_notify("foo", self.async_foo)
return {'asyncJobId': job.id}
In the frontend, you get back the jobId and start listening for events.
// const { bus } = require('web.core');
this.jobs = [];
bus.on('async_job', this, (job) => {
if (this.jobs.contains(job.id)) {
this.do_action(job.payload.result).then(() => {
this._rpc({model: 'ir.async', method: 'complete', args: [job.id]});
});
}
});
this._rpc({model: 'foo', method: 'foo'}).then(({asyncJobId}) => {
this.jobs.push(asyncJobId);
});
Alternatively, you may use the asyncRpc method of the new async_job service
if you don't need much control on the job. It takes the same arguments as a
normal rpc would do. The notable difference is the promise doesn't resolve with
the HTTP request but when the async task finishes.
this.call('async_job', 'asyncRpc', {
model: 'foo', method: 'foo'
}).then((job) => {
this.do_action(job.payload.result).then(() => {
this._rpc({model: 'ir.async', method: 'complete', args: [job.id]});
});
});
When the additional description is truthy, the task is listed in a new
systray menu. You may hide the task from the list but still get the notification
by setting a falsy value.
self.env['ir.async'].call("Employee XLSX export", self.async_foo)
You may read the source-code directly before it gets complicated with hackish
optimizations, rusty bugfixes and useless features. Just reading the three
commits bellow should give you a good gasp of what's inside the black box.
The solution is built on :
LISTEN/NOTIFY so servers only poll when there are enqueued jobs.SELECT FOR UPDATE SKIP LOCKED for job selection.bus.bus longpolling bus to send message from the server to the browser.There is no sphinx documentation yet, this mail, the task, the commit messages
and the source code are the documentation for now. Please consider the current
API is in a alpha-stage, it is going to change as I gather feedback.
Just reach me if you have any question. I'm Julien#4157 on discord, hiding in
the #rd-framework-py channel.
I suggest you increase the verbosity of a few loggers as the postgres
notifications are sometime hard of hearing (some notification don't wake up the
async workers). At worst you'll experience a 1-minute delay.
Here is the few arguments you can inject in your odoo-bin:
--log-handler odoo.service.server:DEBUG
--log-handler odoo.addons.base.models.ir_async:DEBUG
I would like to thanks both my team, the SaaS, the framework and the discuss
teams for their patience, help and reviews. If there are some architecture
non-sense or nasty stuff in the resulting source code, blame them :D
~The lazer guy