Backend Development - Queue API

In real Drupal applications, not every task should run immediately during a page request. Some operations are:

• Time-consuming
• Resource-intensive
• Dependent on external systems
• Better processed in the background

This is where the Queue API becomes essential.

The Queue API allows Drupal to:

• Defer processing
• Store tasks safely
• Process them later (often via cron)
• Scale heavy backend operations

This topic continues the learning-by-building approach using the same weeklydrupal_demo module.

By the end of this article, you will understand:

• What a queue is in Drupal
• When to use the Queue API
• How to create and push queue items
• How to process queue workers
• How queues integrate with cron
• Best practices for scalable backend processing

1. What Is a Queue

A queue is a list of tasks that need to be processed later.

Each task (called a queue item) contains data that a worker will process.

In simple terms:

Queue = storage for deferred work
Worker = code that processes queued tasks

2. When to Use the Queue API

Use the Queue API when:

• Importing thousands of records
• Sending large batches of emails
• Syncing data with external APIs
• Rebuilding search indexes
• Updating many entities

Avoid using queues for quick, lightweight operations.

3. How Drupal Queues Work

Basic flow:

1. Code creates a queue item
2. Item is stored in database
3. Cron runs queue workers
4. Worker processes items one by one

Drupal provides:

• Queue storage
• Queue workers
• Cron integration

4. Creating a Queue Item

Example: push an item into a queue.

$queue = \Drupal::queue('weeklydrupal_demo_queue');

$data = [
  'title' => 'Queued article',
  'rating' => 4,
];

$queue->createItem($data);

What is happening

• Drupal creates a queue storage entry
• $data is serialized and stored
• No processing happens yet

5. Creating a Queue Worker

Queue workers process queued items.

File path

weeklydrupal_demo/src/Plugin/QueueWorker/DemoQueueWorker.php

Code

<?php

namespace Drupal\weeklydrupal_demo\Plugin\QueueWorker;

use Drupal\Core\Queue\QueueWorkerBase;

/**
 * Processes weeklydrupal demo queue items.
 *
 * @QueueWorker(
 *   id = "weeklydrupal_demo_queue",
 *   title = @Translation("WeeklyDrupal Demo Queue"),
 *   cron = {"time" = 60}
 * )
 */
class DemoQueueWorker extends QueueWorkerBase {

  public function processItem($data) {

    // Example: create a node from queued data.
    $node = \Drupal::entityTypeManager()->getStorage('node')->create([
      'type' => 'article',
      'title' => $data['title'],
      'status' => 1,
    ]);

    $node->set('field_rating', $data['rating']);
    $node->save();
  }

}

6. Understanding the Annotation

@QueueWorker(
  id = "weeklydrupal_demo_queue",
  title = @Translation("WeeklyDrupal Demo Queue"),
  cron = {"time" = 60}
)

Explanation:

• id – Must match the queue name
• title – Human-readable label
• cron – Maximum processing time per cron run (in seconds)

Drupal discovers this worker automatically.

7. How Cron Processes Queues

When cron runs:

1. Drupal finds all queue workers
2. Processes items for each worker
3. Respects the time limit
4. Removes items after successful processing

If processing fails, items remain in the queue.

8. Handling Errors and Retries

If an exception is thrown in processItem():

• The item remains in the queue
• It will be retried later

Example:

if (empty($data['title'])) {
  throw new \Exception('Missing title');
}

This allows safe retry behavior.

9. Best Practices for Queue Workers

• Keep processing logic small and focused
• Avoid heavy memory usage
• Use dependency injection instead of static calls
• Log meaningful errors
• Ensure idempotency (safe re-processing)

Queues should be reliable and repeatable.

10. Using Services Inside Queue Workers

Instead of writing entity logic directly in the worker, prefer:

• Injecting a service
• Letting the service handle CRUD

This keeps architecture clean and testable.

11. Learning by Building: weeklydrupal_demo

In our demo module, a good exercise is:

• Create a route /weeklydrupal/demo/queue
• Controller pushes 10 items into queue
• Cron processes them
• Articles are created in background

This demonstrates:

• Routes
• Services
• Entity CRUD
• Queue processing
• Cron integration

12. Common Mistakes to Avoid

• Performing heavy work in page requests
• Not handling exceptions
• Writing non-idempotent logic
• Forgetting cron configuration

Queue API exists to protect performance and scalability.

13. Why Queue API Matters

Understanding the Queue API allows you to:

• Build scalable systems
• Handle large data imports
• Integrate external APIs safely
• Prevent slow page loads

Queue-based architecture is essential for enterprise Drupal projects.