Cross-Language Queues: Sending Jobs from Node.js to Laravel

One thing that I really like about Laravel is Queues. These allow you perform some slow running tasks asynchronously by pushing data into a queue, where Laravel can then pick those jobs up and process them at a later time in the background.

I think lots of devs use these, but fewer realise that you can push jobs into the queue from outside your Laravel application.

In this post I will show how you can push a Job from a NodeJS CloudFlare Worker into an AWS SQS Queue, which will then be picked up by our Laravel worker.

I have used this technique in the past for example to implement a view counter which we didn’t want to burden our main application with, but we still wanted to record the counted events in our main application’s database.

Pre-requisites

• SQS - In this example I’m using an SQS queue. You could do similar with a Redis queue, but SQS is far easier to work with from 3rd party services.
• CloudFlare - In this example I’m using a CloudFlare worker to push the data into the queue - other options are available.

Creating our Laravel Job

To get started we’ll create a Laravel job:

php artisan make:job SampleJob       

The job itself will be extremely basic, and just dumps out a message:

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;

class SampleJob implements ShouldQueue
{
    use Queueable;

    public function __construct(
        public string $message,
    )
    {
    }

    public function handle(): void
    {
        dump($this->message);
    }
}

So far, so boring…

Creating the CloudFlare Worker

The next thing we need is a CloudFlare Worker. We’ll need some extra dependencies, so we install them at the same time:

• @aws-sdk/client-sqs to allow us to connect to SQS,
• php-serialize to allow us to serialize a JS object into a string that PHP can deserialize,
• @cfworker/uuid to allow us to generate UUIDs.

Let’s go:

npm create cloudflare@latest -- cloudflare-worker
cd cloudflare-worker
npm install @aws-sdk/client-sqs php-serialize @cfworker/uuid

This will give us a directory cloudflare-worker with the basic scaffolding for the worker inside.

We’ll also create a .dev.vars file to hold our environment variables: Create a new file at cloudflare-worker/.dev.vars and add the following content:

AWS_ACCESS_KEY_ID={your access key}
AWS_SECRET_ACCESS_KEY={your secret}
AWS_REGION={your region}
SQS_QUEUE_URL={your queue url}

Next, we’ll create a SampleJob TypeScript class. The class name and its public properties must match our Laravel SampleJob exactly. Create a file at cloudflare-worker/src/SampleJob.ts

export class SampleJob {
    public message: string;

    public constructor(
        message: string,
    ) {
        this.message = message;
    }
}

Serializing the Job for Laravel

The next step is really where the magic happens: We need to be able to convert a SampleJob TypeScript object into a version that Laravel can understand and process.

When Laravel pushes a job into the queue, it json_encodes an associative array that contains all the data about the job. It contains some meta data (such as the maxTries and backoff). The main job details are contained with a data key that holds a PHP serialized representation of the job.

So we’ll add a method to our SampleJob class that will convert a SampleJob object into exactly such a string:

import {uuid} from "@cfworker/uuid";
import {serialize} from "php-serialize";

export class SampleJob {
    
    // [...]
    
    public toLaravelJob(): string {
        // we need this namespace to transform the SampleJob class name into the FQN for PHP
        const nameSpace: NameSpaceMap = {
            'App\\Jobs\\SampleJob': SampleJob,
        };

        return JSON.stringify({
            uuid: uuid(),
            displayName: 'App\\Jobs\\SampleJob',
            job: 'Illuminate\\Queue\\CallQueuedHandler@call',
            maxTries: null,
            maxExceptions: null,
            failOnTimeout: false,
            backoff: null,
            timeout: null,
            retryUntil: null,
            data: {
                commandName: 'App\\Jobs\\SampleJob',
                command: serialize(this, nameSpace),
            }
        })
    }
}

Creating the Worker Script

And finally, we create the actual worker script to push this job object into the queue. This will be at cloudflare-worker/src/index.ts:

import {SampleJob} from "./SampleJob";
import {SendMessageCommand, SQSClient} from "@aws-sdk/client-sqs";

export default {
	async fetch(request, env, ctx): Promise<Response> {
        // instantiate our job
        const job = new SampleJob('Hello World');

        // instantiate client
        const client = new SQSClient({
            region: env.AWS_REGION,
            credentials: {
                accessKeyId: env.AWS_ACCESS_KEY_ID,
                secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
            }
        });

        // push job to queue
        await client.send(new SendMessageCommand({
            QueueUrl: env.SQS_QUEUE_URL,
            MessageBody: job.toLaravelJob(),
        }));

        // return a response
        return new Response('OK');
    }
} satisfies ExportedHandler<Env>;

Testing the Setup

To test this setup, run npm run dev inside the cloudflare-worker directory and open the generated URL in a browser. Each time you visit the URL, a new job will be pushed into the queue. Then, by running php artisan queue:work in the terminal, you should see Hello World printed in the console shortly after.

Conclusion

This technique is useful when you have a Laravel application but need to process some parts of your system in a different language. The main challenge is creating a PHP-serialized representation of the job, but thanks to the php-serialize package for Node.js, it’s fairly straightforward. Even if you had to do it manually, it’s not particularly difficult.