includes/job/README

   1 /*!
   2 \ingroup JobQueue
   3 \page jobqueue_design Job queue design
   4
   5 Notes on the Job queuing system architecture.
   6
   7 \section intro Introduction
   8
   9 The data model consist of the following main components:
  10
  11 * The Job object represents a particular deferred task that happens in the
  12   background. All jobs subclass the Job object and put the main logic in the
  13   function called run().
  14 * The JobQueue object represents a particular queue of jobs of a certain type.
  15   For example there may be a queue for email jobs and a queue for squid purge
  16   jobs.
  17
  18 Each job type has its own queue and is associated to a storage medium. One
  19 queue might save its jobs in redis while another one uses would use a database.
  20
  21 Storage medium are defined in a queue class. Before using it, you must
  22 define in $wgJobTypeConf a mapping of the job type to a queue class.
  23
  24 The factory class JobQueueGroup provides helper functions:
  25 - getting the queue for a given job
  26 - route new job insertions to the proper queue
  27
  28 The following queue classes are available:
  29 * JobQueueDB (stores jobs in the `job` table in a database)
  30
  31 All queue classes support some basic operations (though some may be no-ops):
  32 * enqueueing a batch of jobs
  33 * dequeueing a single job
  34 * acknowledging a job is completed
  35 * checking if the queue is empty
  36
  37 Some queue classes (like JobQueueDB) may dequeue jobs in random order while other
  38 queues might dequeue jobs in exact FIFO order. Callers should thus not assume jobs
  39 are executed in FIFO order.
  40
  41 Also note that not all queue classes will have the same reliability guarantees.
  42 In-memory queues may lose data when restarted depending on snapshot and journal
  43 settings (including journal fsync() frequency).  Some queue types may totally remove
  44 jobs when dequeued while leaving the ack() function as a no-op; if a job is
  45 dequeued by a job runner, which crashes before completion, the job will be
  46 lost. Some jobs, like purging squid caches after a template change, may not
  47 require durable queues, whereas other jobs might be more important.
  48
  49 Callers should also try to make jobs maintain correctness when executed twice.
  50 This is useful for queues that actually implement ack(), since they may recycle
  51 dequeued but un-acknowledged jobs back into the queue to be attempted again. If
  52 a runner dequeues a job, runs it, but then crashes before calling ack(), the
  53 job may be returned to the queue and run a second time. Jobs like cache purging can
  54 happen several times without any correctness problems. However, a pathological case
  55 would be if a bug causes the problem to systematically keep repeating. For example,
  56 a job may always throw a DB error at the end of run(). This problem is trickier to
  57 solve and more obnoxious for things like email jobs, for example. For such jobs,
  58 it might be useful to use a queue that does not retry jobs.