Nuffing.com

It has been a lot of time since I started using Laravel Queues. During earlier days back than learning complex technologies on own was not easy and I somehow managed guts to learn them and apply in real-time projects.

Today I’m happy that I took that decision. While learning Laravel Framework I got introduced to a lot of intermediate and high-level programming concepts and one concept or say feature on this framework caught my attention was Queues because of its ability to run jobs or tasks in the background.

From the day I learnt queues in started to use it in each of the projects I do in laravel.

Table of Contents

What are Laravel Queues?

Consider you have a heavy processing task to be performed by application this may be generating large data for reporting, blasting emails for a large number of users or such notifying people when their task gets completed.

These are some of the examples and as they are time-consuming you cannot execute these task is normal manner such are on button click or page submit. So to properly handling these task laravel provides queue functionality where tasks data is storing in a database and then the queue will automatically detect any jobs and execute them in background.

Create a Fresh New Laravel Project

Laravel provides two different options for creating a fresh project.

Before you execute below command you must navigate to your directory where you would like to create this project and open the terminal from that particular path.

Using Laravel Installer

This installer is downloaded through composer. Before running this command composer must be installed on your computer.

composer global require laravel/installer
 

Then you can create a new project by typing below command.

laravel new your_project_name
 

Directly creating a project through composer command

This is another method and I always go through this method because it doesn’t need the additional installer to be installed other than a composer and also I can specify project version at the end.

composer create-project --prefer-dist laravel/laravel your_project_name "6.*"
 

At the end 6.* specifies which version you would like to install. You can visit Laravel versions and support policy page for more information.

Database Connections

In the root directory of the freshly created project, theirs an .env file which contains all the environment setting for the project.
There is an option for adding database connection which username and password.

DB_CONNECTION=mysql
 DB_HOST=127.0.0.1
 DB_PORT=3306
 DB_DATABASE=laravel_queues
 DB_USERNAME=root
 DB_PASSWORD=
 

laravel_queues is our database name and then goto PHPMyAdmin and create a database of laravel_queues name.

Queue Configuration and Settings

Since we are telling laravel to use our database as queue driver there is just a simple step to do.

That is to go to .env file and replace below code.

#Replace this line.
 QUEUE_CONNECTION=sync 
 
 #With this line.
 QUEUE_CONNECTION=database
 

Database Migration

Database migration takes place by creating a migration class inside the folder database/migrations. It contains the schema description of the table and can be migrated to the database.
But before start migration, you must inform laravel to create a migration class for queue table for this use below command.

php artisan queue:table
 

A migration file is created inside database/migrations folder.

Schema::create('jobs', function (Blueprint $table) {
 $table->bigIncrements('id');
 $table->string('queue')->index();
 $table->longText('payload');
 $table->unsignedTinyInteger('attempts');
 $table->unsignedInteger('reserved_at')->nullable();
 $table->unsignedInteger('available_at');
 $table->unsignedInteger('created_at');
 });
 

Now all these migrations must be migrated to the database for that run below command.

php artisan migrate
 

By switching to PHPMyAdmin you can see that tables are created in the database.

For testing, purpose let us also create two new tables they are countries_census, states_census.

php artisan make:migration create_countries_census_table --create=countries_census
 
 php artisan make:migration create_states_census_table --create=states_census
 

database/migrations/2019_08_19_000000_create_countries_census_table.php.

<?php
 
 use Illuminate\Database\Migrations\Migration;
 use Illuminate\Database\Schema\Blueprint;
 use Illuminate\Support\Facades\Schema;
 
 class create_countries_census_table extends Migration
 {
 /**
 * Run the migrations.
 * 
 * @return void
 */
 public function up()
 {
 Schema::create('countries_census', function (Blueprint $table) {
 $table->bigIncrements('country_id');
 $table->text('country_name');
 $table->unsignedInteger('total_population');
 $table->timestamp('updated_at')->nullable();
 });
 }
 
 /**
 * Reverse the migrations.
 *
 * @return void
 */
 public function down()
 {
 Schema::dropIfExists('countries_census');
 }
 }
 
 

database/migrations/2019_08_19_000000_create_states_census_table.php.

<?php
 
 use Illuminate\Database\Migrations\Migration;
 use Illuminate\Database\Schema\Blueprint;
 use Illuminate\Support\Facades\Schema;
 
 class create_states_census_table extends Migration
 {
 /**
 * Run the migrations.
 * 
 * @return void
 */
 public function up()
 {
 Schema::create('states_census', function (Blueprint $table) {
 $table->bigIncrements('state_id');
 $table->text('state_name');
 $table->unsignedInteger('country_id');
 $table->unsignedInteger('state_population');
 });
 }
 
 /**
 * Reverse the migrations.
 *
 * @return void
 */
 public function down()
 {
 Schema::dropIfExists('states_census');
 }
 }
 

php artisan migrate
 

Creating models for countries_census and states_census tables.

Laravel models are the representation of the database table which contains tables schematic information as well as one table to another table relationship description.

To create model use below command.

php artisan make:model CountryPopulationModel
 

In app/CountryPopulationModel.php a new model has been created.

<?php
 
 namespace App;
 
 use Illuminate\Database\Eloquent\Model;
 use Illuminate\Support\Facades\Log;
 
 class CountryPopulationModel extends Model
 {
 protected $table="countries_census";
 protected $primaryKey = 'country_id';
 protected $fillable = ['country_name', 'total_population'];
 
 public function states()
 {
 return $this->hasMany(CountryStatePopulationModel::class, 'country_id', 'country_id');
 }
 }
 

$table is database table name of than model.
$primaryKey is database table primary.
$fillable is columns of database table except for primary key.

Similarly, let us create a model for states_census.
In app/CountryStatePopulationModel.php.

<?php
 
 namespace App;
 
 use Illuminate\Database\Eloquent\Model;
 use Illuminate\Support\Facades\Log;
 
 class CountryStatePopulationModel extends Model
 {
 protected $table="states_census";
 protected $primaryKey = 'state_id';
 protected $fillable = ['country_id', 'state_name', 'state_population'];
 
 public function country()
 {
 return $this->belongsTo(CountryPopulationModel::class, 'country_id', 'country_id');
 }
 }
 

Dummy data is also added into both the tables and before running jobs, these tables look like below.

Creating a controller for handling logic

Command to create a new controller.

php artisan make:controller CountryController
 

app/Http/Controllers/CountryController.php

<?php
 
 namespace App\Http\Controllers;
 
 use App\CountryPopulationModel;
 use App\Jobs\ProcessCountriesPopulation;
 use Illuminate\Http\Request;
 use Illuminate\Support\Facades\Log;
 
 class CountryController extends Controller
 {
 public function process(Request $request){
 
 }
 }
 

Routing

routes/web.php

Route::get('countries-census/process', 'CountryController@process');
 

Creating New Job for Queue

Use php artisan the command for creating a new job class.

php artisan make:job ProcessCountriesPopulation
 

ProcessCountriesPopulation class is created inside the application root directory app/Jobs/ProcessCountriesPopulation.php path.
This job class will periodically check for population data for different countries, sum them and update the latest population of the country in countries_census table.

The ProcessCountriesPopulation job class looks like below.

<?php
 
 namespace App\Jobs;
 
 use App\CountryPopulationModel;
 use App\CountryStatePopulationModel;
 use Illuminate\Bus\Queueable;
 use Illuminate\Contracts\Queue\ShouldQueue;
 use Illuminate\Foundation\Bus\Dispatchable;
 use Illuminate\Queue\InteractsWithQueue;
 use Illuminate\Queue\SerializesModels;
 use Illuminate\Support\Facades\Log;
 
 class ProcessCountriesPopulation implements ShouldQueue
 {
 use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
 
 /**
 * Create a new job instance.
 *
 * @return void
 */
 public function __construct()
 {
 
 }
 
 /**
 * Execute the job.
 *
 * @return void
 */
 public function handle()
 {
 
 }
 } 
 

Pre-made methods __construct and handle are available.

<?php
 
 namespace App\Jobs;
 
 use App\CountryPopulationModel;
 use App\CountryStatePopulationModel;
 use Illuminate\Bus\Queueable;
 use Illuminate\Contracts\Queue\ShouldQueue;
 use Illuminate\Foundation\Bus\Dispatchable;
 use Illuminate\Queue\InteractsWithQueue;
 use Illuminate\Queue\SerializesModels;
 use Illuminate\Support\Facades\Log;
 
 class ProcessCountriesPopulation implements ShouldQueue
 {
 use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
 protected $country_census;
 
 /**
 * Create a new job instance.
 *
 * @return void
 */
 public function __construct(CountryPopulationModel $country)
 {
 Log::info('Entered Job ProcessCountriesPopulation __constructor method');
 $this->country_census = $country;
 }
 
 /**
 * Execute the job.
 *
 * @return void
 */
 public function handle()
 {
 Log::info('Entered Job ProcessCountriesPopulation handle method');
 $country_states_population_sum = CountryStatePopulationModel::where(["country_id" => $this->country_census["country_id"]])->sum('state_population');
 
 $country = CountryPopulationModel::find($this->country_census["country_id"]); 
 $country->total_population = $country_states_population_sum;
 $country->save();
 
 Log::info('Exited from Job ProcessCountriesPopulation handle method');
 }
 }
 

In the above job, the class constructor method __construct(CountryPopulationModel $country) takes an object which is of type CountryPopulationModel as an argument. This object is then assigned to a class variable $this->country_census.
Log::info logs the message in the log file which is placed in the path laravel_queues/storage/logs/laravel.log.

Dispatching Jobs to Queue

app/Http/Controllers/CountryController.php

<?php
 
 namespace App\Http\Controllers;
 
 use App\CountryPopulationModel;
 use App\Jobs\ProcessCountriesPopulation;
 use Illuminate\Http\Request;
 use Illuminate\Support\Facades\Log;
 
 class CountryController extends Controller
 {
 public function process(Request $request){
 Log::info('Entered Job CountryController process method');
 
 $country = CountryPopulationModel::find(101);
 
 ProcessCountriesPopulation::dispatch($country);
 
 Log::info('Exited Job CountryController process method');
 }
 }
 

In ProcessCountriesPopulation::dispatch($country); the static dispatch() method passed $country object to job class constructor.

Running Queue Workers on Development Machine

php artisan queue:work
 

This command will detect any pending jobs is jobs table and executes them one by one.

After going to route countries-census/process in browser. You can see that a new row is already been inserted in jobs table. And below log message appears in laravel_queues/storage/logs/laravel.log a file.

[2020-04-09 11:48:54] local.INFO: Entered Job CountryController process method 
 [2020-04-09 11:48:54] local.INFO: Entered Job ProcessCountriesPopulation __constructor method 
 [2020-04-09 11:48:54] local.INFO: Exited Job CountryController process method 
 [2020-04-09 11:48:55] local.INFO: Entered Job ProcessCountriesPopulation handle method 
 [2020-04-09 11:48:55] local.INFO: Exited from Job ProcessCountriesPopulation handle method 
 

Jobs Table before running queue.

When the job is processed queue automatically remove the completed job rows.

Population data inside the database table countries_census is also updated.

Processing of queued jobs.

Using Laravel Queues for sending mail

Here you’ll see a simple example of sending an email using queues.

For creating new email class.

php artisan make:mail WelcomeEmail
 

In laravel_queues/app/Mail/WelcomeEmail.php

public function build()
 {
 return $this->from('abc@gmail.com')->view('emails.welcome-email');
 }
 

For sending emails use Mail class and inside send() method pass the name of the email class you would like to send.

Mail::to("to-email@gmail.com")->send(new WelcomeEmail());
 

Running Queue Workers on Production Server

Use cron job option provided by servers and set the frequency to each minute. Which means that the cron job will execute the artisan console command schedule method each minute which is placed at the path laravel_queues/app/Console/Kernel.php.

I have created a personalized script while using executing job command inside console which I’ll be sharing with you.

$schedule->command('queue:restart')->everyFifteenMinutes()->withoutOverlapping();
 
 $schedule->command('queue:work --sleep=3 --tries=3')->everyMinute()->sendOutputTo(storage_path() . '/logs/queue-jobs.log')->withoutOverlapping();
 

First command queue:restart will restart the processing job listeners, every fifteen minutes. The withoutOverlapping method will prevent the same jobs from overlapping with each other.

Laravel has also provided a detailed overview of the Overlapping of tasks.

Next command queue:work will listen to new jobs every minute. You can also specify flags which have different purposes.
Flag --sleep will send queue to sleep mode for specified seconds once all jobs are processed.
Flag --tries will attempt to try executing queued for a specified number of times before sending them for failed_jobs.

How to handle failed jobs in Queues?

Queued jobs which were unable to execute even after many attempts are sent to failed_jobs the table. This table consists of information such as connection type, payload, exception information and date-time when a job has failed.

For re-attempting, these failed jobs use below command.

php artisan queue:retry all
 

Command queue:retry all will retrieve a list of all jobs that are failed and will try to execute them. You can also specify which job to executed by specifying its primary key number like php artisan queue:retry 2 this will execute the job with primary key 2.

List failed jobs

To see all the jobs which are failed use php artisan queue:failed command. This will display all the failed jobs in the terminal.

Queueing Jobs by Priorities

Queue executes jobs on default occurrence. But you can specify the priority of a job during dispatching stage.

dispatch((new ProcessCountriesPopulation($country))->onQueue('high'));
 

Conclusion

Time has come to say goodbye. Coming to the end of this post you have all the basic and intermediate information and examples of working and queues in the development and production stage. This post was regarding Laravel Jobs and Queues – Configuring, Sending Mail, Dispacting Jobs and were are happy to have taken time to go read this post. Support us by sharing this post which will help us grow and comment if you have any doubts we will reach you soon.

Popular Posts