Backup Performance Comparison: mysqldump vs MySQL Shell Utilities vs mydumper vs mysqlpump vs XtraBackup

https://www.percona.com/blog/wp-content/uploads/2021/12/MySQL-Backup-Performance-Comparison.pngMySQL Backup Performance Comparison

MySQL Backup Performance ComparisonIn this blog post, we will compare the performance of performing a backup from a MySQL database using mysqldump, MySQL Shell feature called Instance Dump, mysqlpump, mydumper, and Percona XtraBackup. All these available options are open source and free to use for the entire community.

To start, let’s see the results of the test.

Benchmark Results

The benchmark was run on an m5dn.8xlarge instance, with 128GB RAM, 32 vCPU, and 2xNVMe disks of 600GB (one for backup and the other one for MySQL data). The MySQL version was 8.0.26 and configured with 89Gb of buffer pool, 20Gb of redo log, and a sample database of 177 GB (more details below).

We can observe the results in the chart below:

MySQL Backup Results

And if we analyze the chart only for the multi-threaded options:

multi-threaded options

As we can see, for each software, I’ve run each command three times in order to experiment using 16, 32, and 64 threads. The exception for this is mysqldump, which does not have a parallel option and only runs in a single-threaded mode.

We can observe interesting outcomes:

  1. When using zstd compression, mydumper really shines in terms of performance. This option was added not long ago (MyDumper 0.11.3).
  2. When mydumper is using gzip, MySQL Shell is the fastest backup option.
  3. In 3rd we have Percona XtraBackup.
  4. mysqlpump is the 4th fastest followed closer by mydumper when using gzip.
  5. mysqldump is the classic old-school style to perform dumps and is the slowest of the four tools.
  6. In a server with more CPUs, the potential parallelism increases, giving even more advantage to the tools that can benefit from multiple threads.

Hardware and Software Specs

These are the specs of the benchmark:

  • 32 CPUs
  • 128GB Memory
  • 2x NVMe disks 600 GB
  • Centos 7.9
  • MySQL 8.0.26
  • MySQL shell 8.0.26
  • mydumper 0.11.5 – gzip
  • mydumper 0.11.5 – zstd
  • Xtrabackup 8.0.26

The my.cnf configuration:

[mysqld]
innodb_buffer_pool_size = 89G
innodb_log_file_size = 10G

Performance Test

For the test, I used sysbench to populate MySQL. To load the data, I choose the tpcc method:

$ ./tpcc.lua  --mysql-user=sysbench --mysql-password='sysbench' --mysql-db=percona --time=300 --threads=64 --report-interval=1 --tables=10 --scale=100 --db-driver=mysql prepare

Before starting the comparison, I ran mysqldump once and discarded the results to warm up the cache, otherwise our test would be biased because the first backup would have to fetch data from the disk and not the cache.

With everything set, I started the mysqldump with the following options:

$ time mysqldump --all-databases --max-allowed-packet=4294967295 --single-transaction -R --master-data=2 --flush-logs | gzip > /backup/dump.dmp.gz

For the Shell utility:

$ mysqlsh
MySQL JS > shell.connect('root@localhost:3306');
MySQL localhost:3306 ssl test JS > util.dumpInstance("/backup", {ocimds: true, compatibility: ["strip_restricted_grants","ignore_missing_pks"],threads: 16})

For mydumper:

$ time mydumper  --threads=16  --trx-consistency-only --events --routines --triggers --compress --outputdir /backup/ --logfile /backup/log.out --verbose=2

PS: To use zstd, there are no changes in the command line, but you need to download the zstd binaries.

For mysqlpump:

$ time mysqlpump --default-parallelism=16 --all-databases > backup.out

For xtrabackup:

$ time xtrabackup --backup --parallel=16 --compress --compress-threads=16 --datadir=/mysql_data/ --target-dir=/backup/

Analyzing the Results

And what do the results tell us?

Parallel methods have similar performance throughput. The mydumper tool cut the execution time by 50% when using zstd instead of gzip, so the compression method makes a big difference when using mydumper.

For the util.dumpInstance utility, one advantage is that the tool stores data in both binary and text format and uses zstd compression by default. Like mydumper, it uses multiple files to store the data and has a good compression ratio. 

XtraBackup got third place with a few seconds of difference from MySQL shell. The main advantage of XtraBackup is its flexibility, providing PITR and encryption for example. 

Next, mysqlpump is more efficient than mydumper with gzip, but only by a small margin. Both are logical backup methods and works in the same way. I tested mysqlpump with zstd compression, but the results were the same, hence the reason I didn’t add it to the chart. One possibility is because mysqlpump streams the data to a single file.

Lastly, for mysqldump, we can say that it has the most predictable behavior and has similar execution times with different runs. The lack of parallelism and compression is a disadvantage for mysqldump; however, since it was present in the earliest MySQL versions, based on Percona cases, it is still used as a logical backup method.

Please leave in the comments below what you thought about this blog post, if I missed something, or if it helped you. I will be glad to discuss it!

Useful Resources

Finally, you can reach us through the social networks, our forum, or access our material using the links presented below:

Planet MySQL

6 Data Structures Every Programmer Should Know

https://static1.makeuseofimages.com/wordpress/wp-content/uploads/2021/12/coding-pexels.jpg

The path to becoming a proficient and successful programmer is difficult, but one that is certainly achievable. Data structures are a core component that every programming student must master, and chances are you may have already learned or worked with some basic data structures such as arrays or lists.

Interviewers tend to prefer asking questions related to data structures, so if you’re preparing for a job interview, you’re going to need to brush up on your data structures knowledge. Read on as we list the most important data structures for programmers and job interviews.

1. Linked List

Linked lists are one of the most basic data structures and often the starting point for students in most data structures courses. Linked lists are linear data structures that allow sequential data access.

Elements within the linked list are stored in individual nodes that are connected (linked) using pointers. You can think of a linked list as a chain of nodes connected to each other via different pointers.

Related: An Introduction to Using Linked Lists in Java

Before we get into the specifics of the different types of linked lists, it is crucial to understand the structure and implementation of the individual node. Each node in a linked list has at least one pointer (doubly linked list nodes have two pointers) that connects it to the next node in the list and the data item itself.

MAKEUSEOF VIDEO OF THE DAY

Each linked list has a head and tail node. Single-linked list nodes only have one pointer that points to the next node in the chain. In addition to the next pointer, doubly linked list nodes have another pointer that points to the previous node in the chain.

Interview questions related to linked lists usually revolve around the insertion, searching, or deletion of a specific element. Insertion in a linked list takes O(1) time, but deletion and searching can take O(n) time in the worst case. So linked lists are not ideal.

2. Binary Tree

Binary trees are the most popular subset of the tree family data structure; elements in a binary tree are arranged in a hierarchy. Other types of trees include AVL, red-black, B trees, etc. Nodes of the binary tree contain the data element and two pointers to each child node.

Each parent node in a binary tree can have a maximum of two child nodes, and each child node, in turn, can be a parent to two nodes.

Related: A Beginner’s Guide to Binary Trees

A binary search tree (BST) stores data in a sorted order, where elements with a key-value less than the parent node are stored on the left, and elements with a key-value greater than the parent node are stored on the right.

Binary trees are commonly asked in interviews, so if you’re getting ready for an interview, you should know how to flatten a binary tree, look up a specific element, and more.

3. Hash Table

Hash tables or hash maps are a highly efficient data structure that stores data in an array format. Each data element is assigned a unique index value in a hash table, which allows for efficient searching and deletion.

The process of assigning or mapping keys in a hash map is called hashing. The more efficient the hash function, the better the efficiency of the hash table itself.

Each hash table stores data elements in a value-index pair.

Where value is the data to be stored, and index is the unique integer used for mapping the element into the table. Hash functions can be very complex or very simple, depending on the required efficiency of the hash table and how you will resolve collisions.

Collisions often arise when a hash function produces the same mapping for different elements; hash map collisions can be resolved in different ways, using open addressing or chaining.

Hash tables or hash maps have a variety of different applications, including cryptography. They are the first choice data structure when insertion or searching in constant O(1) time is required.

4. Stacks

Stacks are one of the simpler data structures and are pretty easy to master. A stack data structure is essentially any real-life stack (think of a stack of boxes or plates) and operates in a LIFO (Last In First Out) manner.

Stacks’ LIFO property means the element you inserted last will be accessed first. You cannot access elements below the top element in a stack without popping the elements above it.

Stacks have two primary operations—push and pop. Push is used to insert an element into the stack, and pop removes the topmost element from the stack.

They also have plenty of useful applications, so it is very common for interviewers to ask questions related to stacks. Knowing how to reverse a stack and evaluate expressions is quite essential.

5. Queues

Illustration of a queue data structure
Image Credit: Wikipedia

Queues are similar to stacks but operate in a FIFO (First In First Out) manner, meaning you can access the elements you inserted earlier. The queue data structure can be visualized as any real-life queue, where people are positioned based on their order of arrival.

Insertion operation of a queue is called enqueue, and deleting/removing an element from the beginning of the queue is referred to as dequeuing.

Related: A Beginner’s Guide to Understanding Queues and Priority Queues

Priority queues are an integral application of queues in many vital applications such as CPU scheduling. In a priority queue, elements are ordered according to their priority rather than the order of arrival.

6. Heaps

Heaps are a type of binary tree where nodes are arranged in ascending or descending order. In a Min Heap, the key value of the parent is equal to or less than that of its children, and the root node contains the minimum value of the entire heap.

Similarly, the root node of a Max Heap contains the maximum key value of the heap; you must retain the min/max heap property throughout the heap.

Related: Heaps vs. Stacks: What Sets Them Apart?

Heaps have plenty of applications due to their very efficient nature; primarily, priority queues are often implemented through heaps. They are also a core component in heapsort algorithms.

Learn Data Structures

Data structures can seem harrowing at first but devote enough time, and you’ll find them easy as pie.

They are a vital part of programming, and almost every project will require you to use them. Knowing what data structure is ideal for a given scenario is critical.

7 Algorithms Every Programmer Should Know

These algorithms are essential to every programmer’s workflow.

Read Next

About The Author

M. Fahad Khawaja
(84 Articles Published)

Fahad is a writer at MakeUseOf and is currently majoring in Computer Science. As an avid tech-writer he makes sure he stays updated with the latest technology. He finds himself particularly interested in football and technology.

More
From M. Fahad Khawaja

Subscribe to our newsletter

Join our newsletter for tech tips, reviews, free ebooks, and exclusive deals!

Click here to subscribe

MUO – Feed

5 PHP Date Functions and Their MySQL Equivalents

https://i0.wp.com/joshuaotwell.com/wp-content/uploads/2021/12/calendar-g738f0c8ad_640.png?w=640&ssl=1

Anyone working with data is going to encounter date values at some point. Dates are necessary for many reasons. If you’re a PHP/MySQL Developer, you have date function options for both the database and your programming language of choice. In this post, I cover 5 PHP date functions and their (somewhat) MySQL equivalents with examples of each. Continue reading…


Self-Promotion:

If you enjoy the content written here, by all means, share this blog and your favorite post(s) with others who may benefit from or like it as well. Since coffee is my favorite drink, you can even buy me one if you would like!


The Newsletter for PHP and MySQL Developers

PHP: date_format() | MySQL: DATE_FORMAT()

We always hope that the data comes to us in the format we need. In reality, we know better and that normally, data is formatted (if at all) in a way that we can’t use or don’t wish to use. With that being said, you can use either of these functions and put date data in pretty much any format you need.

Both functions have a myriad number of different formatting options and covering them all in one blog post would be convoluted at best. Instead, I’ll cover a couple of the common ones I am familiar with and leave the exploration up to you.

date_format()

Syntax PHP date_format():

date_format($object, $format)
  • Returns a date as specified by the format.
  • Both parameters are required.

Using the date_create() function, I make a practice date we can use for the examples:

$some_date = date_create('2021-12-04');
echo date_format($some_date, 'Y-d-m');

Echoing out in the browser returns:

2021-04-12

In order to format the date value to a full month name, 2-digit day (with suffix), and 4-digit year, we can use these format parameters:

echo date_format($some_date, 'F jS, Y');

Which returns:

December 4th, 2021

Oftentimes, you see dates in the 2-digit month 2-digit day, and 4-digit year format (MM/DD/YYYY). These format strings shown in the following example produce that format for you:

echo date_format($some_date, 'd/m/Y');
04/12/2021

(There are several format options for PHP’s date_format(). Visit the date_format() documentation to see them all.)

DATE_FORMAT()

Syntax MySQL DATE_FORMAT():

DATE_FORMAT(date, format)
  • Returns a date as specified by the format.
  • Both parameters are required.

For simplicity’s sake, I am using the ‘store’ table from the sakila practice database with this data:

mysql> SELECT last_update
-> FROM store;
+---------------------+
| last_update |
+---------------------+
| 2006-02-15 04:57:12 |
| 2006-02-15 04:57:12 |
+---------------------+
2 rows in set (0.00 sec)

For a 3-letter month abbreviation, a 2-digit day – with the suffix – and the full 4-digit year, we can use these format specifiers:

mysql> SELECT DATE_FORMAT(last_update, '%b %D, %Y') AS frmt
-> FROM store;
+----------------+
| frmt |
+----------------+
| Feb 15th, 2006 |
| Feb 15th, 2006 |
+----------------+
2 rows in set (0.00 sec

As you have seen in the 1st exploratory query on the ‘store’ table, the ‘last_update’ column date values are stored in a ‘YYYY-MM-DD’ format. If these values were not already stored as such, you can easily format them with the format specifiers shown in the next query:

mysql> SELECT DATE_FORMAT(last_update, '%Y-%m-%d') AS frmt
-> FROM store;
+------------+
| frmt |
+------------+
| 2006-02-15 |
| 2006-02-15 |
+------------+
2 rows in set (0.00 sec)

A common format you see is the ‘DD-MON-YY’ format (a lot of Oracle SQL dates are stored in this format). There are format specifiers for that pattern as well:

mysql> SELECT DATE_FORMAT(last_update, '%d-%b-%y') AS frmt 
-> FROM store;
+-----------+
| frmt |
+-----------+
| 15-Feb-06 |
| 15-Feb-06 |
+-----------+
2 rows in set (0.00 sec)

(Visit this MySQL DATE_FORMAT() resource for more information.)


PHP: date_diff() | MySQL: DATEDIFF()

PHP’s date_diff() function returns the difference between 2 DateTime objects. The difference can be a number of different values such as days, months, or even hours (and many more).

date_diff()

Syntax PHP date_diff()

date_diff($origin_date, $target_date)

I’ll create 2 dates and then call the format() method on the resulting object from the date_diff() function call:

$current_date = date_create('2021-12-04');
$xmas_date = date_create('2021-12-25');

$num_days = date_diff($current_date, $xmas_date);
echo $num_days->format('%R%a days');

The a format echoed in the browser returns:

+21 days

Using the m format, we can know the number of months difference between the date objects:

$current_date = date_create('2021-12-04');
$six_mnths_later = date_create('2021-06-25');

$num_mnths = date_diff($current_date, $six_mnths_later);
echo $num_mnths->format('%m months');
5 months

(Note: date_diff() is in the procedural style. Read more in the official date_diff() documentation.)

DATEDIFF()

MySQL’s DATEDIFF() function returns the number of days between 2 date values. This function is handy for calculating how many days an event happened in the past or will happen in the future (i.e., a birthday, holiday, anniversary, etc…).

Syntax MySQL DATEDIFF():

DATEDIFF(date_1, date_2)

At the time of writing this post, MySQL’s CURRENT_DATE() function returns this value in my local learning/development environment:

mysql> SELECT CURRENT_DATE();
+----------------+
| CURRENT_DATE() |
+----------------+
| 2021-12-04 |
+----------------+
1 row in set (0.00 sec)

Christmas is on December the 25th and we can calculate the remaining number of days until the holiday using DATEDIFF():

mysql> SELECT DATEDIFF('2021-12-25', CURRENT_DATE()) AS days_to_xmas;
+--------------+
| days_to_xmas |
+--------------+
| 21 |
+--------------+
1 row in set (0.00 sec)

Hope y’all were good this year so Santa visits!

(More information on DATEDIFF().)


You can support my content and this blog by tossing any spare change you have in my Tip Jar. I really appreciate your support!


PHP: date() | MySQL: NOW()

PHP’s date() function returns the current time, formatted according to the $format_string argument, if no $timestamp argument value is provided in the function call.

date()

Syntax PHP date():

date(string $format_string, $timestamp = null)

As of the time of writing this segment of the post, the date() function returns this date in my local development environment:

$current_date = date('Y-m-d');
echo $current_date;

2021-12-10

Just as several other example PHP date functions, there are many formatting options for date().

$current_date = date('F jS, Y');
echo $current_date;

December 10th, 2021

(Visit the PHP date() function documentation for more information on this function.)

NOW()

The MySQL NOW() function returns the current date and time. NOW() accepts no arguments.

Syntax MySQL NOW():

NOW()

Let’s see an example of NOW() in my local development environment:

mysql> SELECT NOW();
+---------------------+
| NOW() |
+---------------------+
| 2021-12-04 19:36:44 |
+---------------------+
1 row in set (0.00 sec)

The NOW() function is easy to use and provides a handy date/time value you can use to calculate information based off on well, now.

(Learn more about the NOW() function here.)


PHP: date_add() | MySQL: DATE_ADD()

If you have a date value that you want to add another unit (DAY, MONTH, YEAR, etc) to, both PHP’s date_add() and MySQL’s DATE_ADD() functions return this type of calculation.

date_add()

Syntax PHP date_add():

date_add($date_object, $interval_amount)

(Note: date_add() is in the procedural style.)

There are several date intervals you can pass to date_add() as the 2nd parameter. In this example, I specify ’21 days’ in the date_interval_create_from__date_string() function (which is the 2nd parameter for date_add()):

$some_date = date_create('2021-12-04');
date_add($some_date, date_interval_create_from_date_string('21 days'));
echo date_format($some_date,'Y-m-d');

Returns:

2021-12-25

Months are a perfectly valid date interval and in the following example, we calculate a date 3 months in the future:

$some_date = date_create('2021-12-04');
date_add($some_date, date_interval_create_from_date_string('3 months'));
echo date_format($some_date,'F jS, Y');

And in the browser we see:

March 4th, 2022

(See the online date_add() documentation for more information.)

DATE_ADD()

Syntax MySQL DATE_ADD():

DATE_ADD(date_value, INTERVAL value add_unit_value)

DATE_ADD() accepts a date or time INTERVAL value, adds that INTERVAL value to the specified date, and returns a date based on the calculation.

Adding a 6 DAY INTERVAL to the date ‘2021-12-25’ using DATE_ADD() returns the date ‘2021-12-31’ (New Year’s Eve):

mysql> SELECT DATE_ADD('2021-12-25', INTERVAL 6 DAY) AS new_yr_eve;
+------------+
| new_yr_eve |
+------------+
| 2021-12-31 |
+------------+
1 row in set (0.00 sec)

The INTERVAL value must be singular. A plural form returns an error:

mysql> SELECT DATE_ADD('2021-12-25', INTERVAL 6 DAYS) AS new_yr_eve;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DAYS) AS new_yr_eve' at line 1

You can add whole months as the INTERVAL value as well:

mysql> SELECT DATE_ADD('2021-12-25', INTERVAL 6 MONTH) AS half_way;
+------------+
| half_way |
+------------+
| 2022-06-25 |
+------------+
1 row in set (0.01 sec)

(Learn more about DATE_ADD() by visiting this site.)


Promotional: I am making Gmail HTML Email Signature Templates over in my Etsy shop. Make your emails stand out and pop with your very own.


PHP: date_sub() | MySQL: DATE_SUB()

With PHP’s date_sub() function, you have all the same capabilities you do with date_add() as far as calculations are concerned. Except with date_sub(), you are subtracting the date intervals.

date_sub()

Syntax PHP date_sub():

date_sub($date_object, $interval_amount)
$some_date = date_create('2021-12-04');
date_sub($some_date, date_interval_create_from_date_string('3 months'));
echo date_format($some_date,'F jS, Y');

This echo of date_format() on the $some_date object is now:

September 4th, 2021

And if we subtract 3 days using date_sub(), we get:

$some_date = date_create('2021-12-04');
date_sub($some_date, date_interval_create_from_date_string('3 days'));
echo date_format($some_date,'F jS, Y');
December 1st, 2021

DATE_SUB()

Syntax MySQL DATE_SUB():

DATE_SUB(date_value, INTERVAL value subtract_unit_value)

We can add an INTERVAL to a date/time value using DATE_ADD() and are returned a calculated date. On the same token, we can subtract an INTERVAL using DATE_SUB(). Just as with DATE_ADD(), the specified INTERVAL unit must be in the singular.

Again, I won’t cover every available INTERVAL unit. Visit any of the links throughout the post for more information.

mysql> SELECT DATE_SUB('2021-12-25', INTERVAL 1 WEEK) AS wk_bf_xmas;
+------------+
| wk_bf_xmas |
+------------+
| 2021-12-18 |
+------------+
1 row in set (0.02 sec)

For example, an INTERVAL value of ‘2 MONTH’ returns a date 2 months prior to the target date parameter:

mysql> SELECT DATE_SUB('2021-12-25', INTERVAL 2 MONTH) AS 2mnths_bf_xmas;
+----------------+
| 2mnths_bf_xmas |
+----------------+
| 2021-10-25 |
+----------------+
1 row in set (0.00 sec)

(Learn more about DATE_SUB() and the many INTERVAL values you can use in the function call.)


Are you a Medium member? If so, receive an email notification each time I publish a blog post if you prefer the Medium platform. Not a member? No worries! Use my sign-up link (I will get a commission at no extra cost to you) and join. I really enjoy reading all the great content there and I know you will too!!!


I hope you have learned something new about working with date values in PHP and MYSQL because I know I have. If you have any questions or see any mistakes in the code, please leave a comment below. Constructive comments help me better provide accurate information and are much appreciated. 🙏

Like what you have read? See anything incorrect? Please comment below and thank you for reading!!!

A Call To Action!

Thank you for taking the time to read this post. I truly hope you discovered something interesting and enlightening. Please share your findings here, with someone else you know who would get the same value out of it as well.

Visit the Portfolio-Projects page to see blog posts/technical writing I have completed for clients.

To receive email notifications (Never Spam) from this blog (“Digital Owl’s Prose”) for the latest blog posts as they are published, please subscribe (of your own volition) by clicking the ‘Click To Subscribe!’ button in the sidebar on the homepage! (Feel free at any time to review the Digital Owl’s Prose Privacy Policy Page for any questions you may have about: email updates, opt-in, opt-out, contact forms, etc…)

Be sure and visit the “Best Of” page for a collection of my best blog posts.


Josh Otwell has a passion to study and grow as a SQL Developer and blogger. Other favorite activities find him with his nose buried in a good book, article, or the Linux command line. Among those, he shares a love of tabletop RPG games, reading fantasy novels, and spending time with his wife and two daughters.

Disclaimer: The examples presented in this post are hypothetical ideas of how to achieve similar types of results. They are not the utmost best solution(s). The majority, if not all, of the examples provided, are performed on a personal development/learning workstation environment and should not be considered production quality or ready. Your particular goals and needs may vary. Use those practices that best benefit your needs and goals. Opinions are my own.

The Newsletter for PHP and MySQL Developers

How can I help you?

  • Are you thinking of starting up a blog? I use WordPress for my blog. Let’s both save money on the plans offered. 💸
  • Grab a Gmail HTML Email Signature template from my Etsy shop and make your emails pop and stand out. ✉
  • Need hosting for your next web application or WordPress site? I use and highly recommend Hostinger. They have great pricing and service.
  • I enjoy reading Refind: The essence of the web, every morning in your inbox. Subscribe for free. Help me get a premium subscription by signing up yourself with my referral link.
  • Grab a free pack of mobile Creator wallpapers.

Disclosure: Some of the services and products links in this post are affiliate links. At no additional cost to you, should you make a purchase by clicking through one of them, I will receive a commission.

The post 5 PHP Date Functions and Their MySQL Equivalents appeared first on Digital Owl’s Prose.

Planet MySQL

Offering Outgoing Webhooks with Laravel, Eloquent & Twig


Webhooks are a common integration mechanism between systems. A small detail I don’t enjoy about them is
their ambiguity, so let me start by specifying what I’ll be talking about. At my organization we call
them Outgoing Webhooks when we are a Webhook Provider and we’re sending out our system’s data out.
And we call Incoming Webhooks when we’re the receiver of data from other systems. This post will describe
my journey around being a Webhook Provider.

The Context

When talking about providing real-time data out of the system, we have a few scope definition established
upfront. It’s a process that goes well with Pub/Sub systems where we internally notify that an event
happened and have a subscriber loop through all webhooks configured to send out the event.
Webhooks can have different forms depending on whether the project is developer-facing like Algolia, GitHub,
AWS or Netlify. In my context, the goal is to provide Outgoing Webhooks for business people that are
looking to save Engineering time and hook things up themselves as easily and simple as possible. This means
that it’s more important to be able to send out data to off-the-shelf API solutions. Nobody is going to be
writing custom code on the receiving end to be able to receive my webhook calls. The business value is
that users can request our system to send one of their other system a piece of data in real time.

The Stack

The ingredients for this project are:

  • Pub/Sub
  • HTTP Client
  • Payload Transformation
  • 3rd-party Authentication

For the sake of understanding, let’s give a concrete goal for the pub/sub: Everytime a user logs in, we should
be able to notify another system. The event here may be meaningless in a lot of contexts, but the
implementation for other events would roughly be the same. With the event published, we can then
prepare an HTTP Client so that we may perform an authentication on a 3rd party system and send
an API call providing the data configured. Here the most powerful tech decision I made was to support
OAuth2. This means a business user can log into our platform, fill out an OAuth2 Form which looks
exactly like Postman OAuth2 configuration and configure an endpoint from a system like Microsoft Dynamics
to be the receiver of the Webhook call. Microsoft Dynamics doesn’t need to be prepared to be a Webhook
Receiver from my small company. Their public native REST HTTP APIs will do.

The Flow

We first identify which part of our system is responsible for fulfilling the event and then publish
that as an event. We may use Laravel Pub/Sub or something else. In my case we opted for AWS SNS.
Once the subscriber kicks in we need to load all webhooks stored in our database and trigger one by one.
This is a simple data query and looping.

    $webhooks = $this->configuration->newQuery()->where('enabled', 1)->get();

    foreach ($webhooks as $webhook) {
        $this->runEventThroughWebhook($event, $webhook);
    }

For a great User Experience, I like to create a record of the webhook execution prior to starting the
actual execution. This means that we can wrap the execution on an eager try/catch and awlays
update the delivery execution with how it went. If something goes wrong on the remote server,
we can record the 400/500 so that our users can beware of what’s happening.

    private function runEventThroughWebhook(EloquentModel $event, WebhookConfiguration $webhook)
    {
        $delivery = $this->delivery->newQuery()->create([
            'webhook_id' => $webhook->id,
            'status' => 'processing',
            'request_method' => $webhook->method,
            'request_url' => $webhook->url,
        ]);

        $bag = $this->executeWebhook($webhook, $event);

        $delivery->update([
            'status' => $bag->status,
            'request_body' => $bag->body,
            'response_status' => $bag->responseStatus,
            'response_body' => $bag->responseBody,
        ]);
    }

To actually execute the call to an external system, we’ll need an HTTP Client. The beauty here is to
either factory a clean standard HTTP Client or, if needed, factory an HTTP Client with an OAuth2 Bearer
Token already configured. We can do that roughly in the following way:

        if ($webhook->relationLoaded('oauth2') && $webhook->oauth2) {
            try {
                $client = $this->httpFactory->oauth2($webhook->oauth2);
            } catch (RequestException $exception) {
                return WebhookExecutionBag::auth($exception->response);
            }
        } else {
            $client = $this->httpFactory->client();
        }

Here we’re checking if there is a OAuth2 relation on the webhook configured and if so we try to
load a Bearer Token to be used in the Authorization Header. But if the authentication fails, we can
already return a failed webhook delivery. If there is no OAuth2 configured, we can deliver a regular
API call.

If a customer wants our system to make an API call with a signature for verification (much like GitHub does)
we can easily allow for that with the following snippet:

if ($webhook->secret) {
    $headers['X-ACME-SIGNATURE'] = hash_hmac('sha256', $body, $webhook->secret);
}

PHP’s hash_hmac will compute a signature of the array $body using the user’s secret if they provided one.
Here I stored the user provided secret encrypted with AWS Secret.

The Body of the Request

The most interesting part is the body of the request. Here I combined Eloquent with Twig. The choice of
Twig as opposed to Blade is because Twig has an Array Environment
and a secure Sandbox.
Eloquent is great for it because we can write Accessor methods that will act as the data source and
transform the data into an array so that Twig can parse it. Here is a sample of a Request Body:

{
  "email": "",
  "country": "",
  "login_at": "",
  "organization": ""
}

This is what we store as the body of the webhook request. A frontend application can offer some
dropdown options and build this JSON automatically for the users. In the backend we will parse it
with the following script:

    private function body(string $template, EloquentModel $event): string
    {
        $loader = new \Twig\Loader\ArrayLoader(['template' => $template]);

        $twig = new \Twig\Environment($loader);

        $policy = $this->twigSecurityPolicy();

        $twig->addExtension(new \Twig\Extension\SandboxExtension($policy));

        return $twig->render('template', ['event' => $event]);
    }

    private function twigSecurityPolicy(): TwigSecurityPolicyForWebhooks
    {
        $tags = [];

        $filters = [];

        $methods = [];

        $properties = []; 

        $functions = [];

        $policy = new \Twig\Sandbox\SecurityPolicy($tags, $filters, $methods, $properties, $functions);

        return new TwigSecurityPolicyForWebhooks($policy);
    }

Twig’s ArrayLoader is perfect for inline/user-provided template and unfortunately Laravel Blade
is very tied to the file system so this is what led me to choose Twig over Blade. The Security Policies
will disallow any attempt at remote code execution or scripting attack from users. In order for this out
to work we only need the Eloquent model to have attributes called email or accessors such as the following:

public function getTeamNameAttribute() {
    return $this->team->name;
}

The last important bit is the TwigSecurityPolicyForWebhooks. I wrote it because Twig doesn’t have any
sort of wildcard * character to allow property access.

final class TwigSecurityPolicyForWebhooks implements SecurityPolicyInterface
{
    public function __construct(private SecurityPolicy $policy) {}

    public function checkSecurity($tags, $filters, $functions): void
    {
        $this->policy->checkSecurity($this, $filters, $functions);
    }

    public function checkMethodAllowed($obj, $method): void
    {
        $this->policy->checkMethodAllowed($obj, $method);
    }

    public function checkPropertyAllowed($obj, $method): void
    {
        
        if ($obj instanceof \Illuminate\Database\Eloquent\Model) {
            return;
        }

        $this->policy->checkPropertyAllowed($obj, $method);
    }
}

With all of this setup, all we have left to do is to actually send out the webhook call and record
any result from it. The next snippet represents that portion:

try {
    $response = $client->withHeaders($headers)->send($webhook->method, $webhook->url, ['body' => $body]);
} catch (HttpClientException|GuzzleException $t) {
    return WebhookExecutionBag::exception($body, $t);
}

return WebhookExecutionBag::processed($body, $response);

Conclusion

I enjoyed working on this project A LOT. It combines a lot of simple and straight-forward tech to give
a huge business benefit with easy drag-and-drop system integration. Users are able to pick virtually any
API out there and call them from our system in real-time as things happens. The webhook configuration
consist of allowing users to define the Body of the Request, custom headers, an endpoint and merge tags
for body transformation. Our users can call an API that expects a URL token, Basic Auth, OAuth2 or custom
headers. We are also able to sign the body of the request in case the receiving end wants/is able to
validate it. Twig security policies protect us against code injection and our users are able to build
their request body as the target system expects it. And finally the Webhook Delivery list will
always be up-to-date with every API call we made and their response status, timestamp and any relevant
diagnostic information.

This is an extremely simple and powerful webhook provider implementation that empower businesses to
seamlessly integrate data flows without having to write code.

As always, hit me up on Twitter with any
questions.

Cheers.

Laravel News Links

If Modern Apps Ran on macOS 9

https://theawesomer.com/photos/2021/12/modern_apps_on_macos_9_t.jpg

If Modern Apps Ran on macOS 9

Link

macOS X and its successors have been around since 2001, but those of us who used Apple’s computers in the 20th century remember even earlier versions of the operating system. Designer Michael Feeney imagines what it might have been like if today’s apps ran on the more primitive user interface of macOS 9.

The Awesomer

Laravel – Redirecting HTTP to HTTPS

https://www.jhanley.com/wp-content/uploads/2021/06/pexels-pixabay-33153-scaled.jpg

Introduction

Once you have an SSL certificate configured, the next step is to redirect unencrypted traffic. There are several methods of doing this. Within your application (Laravel), by the web server (Apache or Nginx) or by the frontend (load balancer). This article will redirect HTTP requests to HTTPS in Laravel using middleware.

If you are also deploying a frontend load balancer, configure both HTTP and HTTPS frontends. In most cases, you will forward traffic from the load balancer to the backend (Laravel) via HTTP and not by HTTPS. This is called SSL Offloading. This means your Laravel middleware must detect the protocol (HTTP or HTTPS) that the client connected to the load balancer and ignore the protocol that the load balancer is using to connect to the backend. Otherwise, the middleware will detect HTTP even if the client connected to the load balancer using HTTPS, and the client will go into a redirect loop.

In this article, I will use yourdomain.com. Replace with your domain name.

Laravel middleware only supports files served by routes. Files that are not served by Laravel, such as /js/app.js will NOT be redirected. This is one of the reasons I like to have HTTP Redirection as several layers (load balancer, web server, application framework). Another reason is to ensure that more than one service layer enforces HTTP Redirection.

Configure .env

This article supports two environments, development and production. The development settings will not redirect HTTP to HTTPS. The production environment will redirect. The environment will be detected by the APP_ENV setting.

Production Configuration:

  • APP_ENV=production
  • APP_DEBUG=false
  • APP_URL=https://yourdomain.com

Development Configuration:

  • APP_ENV=local
  • APP_DEBUG=true
  • APP_URL=http://localhost:8000

The application environment labels local and production are used to enable/disable certain features in Laravel.

Initial Testing

Open a web browser and connect to your site via HTTP: http://yourdomain.com. Verify that your site loads correctly, and you are not redirected to HTTPS. Note: some TLD domains such as .dev automatically redirect in browsers. If this is the case for you, use the curl command method below.

Open a command prompt and run this command:

curl I http://yourdomain.com

We are interested in the first part of the output which is the HTTP status code. If HTTP redirection is disabled, you should receive a 200 response:

For this article, we want a 200 response so that we can implement and test HTTP redirection.

If HTTP redirection is enabled, then you will receive a 3xx response with an HTTP Location header:

HTTP/1.1 302 Found

....

Location: https://yourdomain.com

Before continuing, disable redirects in your web server or frontend (load balancer). Save your changes, so that you can reenable redirection at the frontend or at the webserver.

Note: browsers tend to cache HTTP redirects. You might need to disable the browser cache.

Disable Chrome Cache

  • Open the Chrome Developer Tools (F12).
  • Go to the Network tab and make sure Disable cache is ticked.
  • Keep the Developer Tools open while testing.

Create the Middleware

Using artisan create the middleware template:

php artisan make:middleware HttpRedirect

This creates the file app/Http/Middleware/HttpRedirect.php.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

<?php

 

namespace App\Http\Middleware;

 

use Closure;

use Illuminate\Http\Request;

 

class HttpRedirect

{

    /**

     * Handle an incoming request.

     *

     * @param  \Illuminate\Http\Request  $request

     * @param  \Closure  $next

     * @return mixed

     */

    public function handle(Request $request, Closure $next)

    {

        return $next($request);

    }

}

Near the top of the file add:

use Illuminate\Support\Facades\App;

Modify the function handle(). Note the following features:

  • Check if the request is using HTTP: !$request->secure()
  • Check if the environment is production: App::environment('production')
  • If both requirements are met, redirect the client to the same URI using HTTPS. Otherwise, proceed to the next handler.

    public function handle(Request $request, Closure $next)

    {

        if (!$request>secure() && App::environment(‘production’) {

                return redirect()>secure($request>getRequestUri());

        }

 

        return $next($request);

    }

The above redirect will return the HTTP code 302. For permanent HTTP to HTTPS redirects, return HTTP code 301 (permanent redirect):

return redirect()>secure($request>getRequestUri(), 301);

If you have development, staging and production environments and you want HTTP redirection for both staging and production:

    public function handle(Request $request, Closure $next)

    {

        if (!$request>secure() && App::environment([‘staging’, ‘production’])) {

                return redirect()>secure($request>getRequestUri(), 301);

        }

 

        return $next($request);

    }

Edit App/Http/Kernel.php and add the middleware to $middleware:

    protected $middleware = [

        ...

        \App\Http\Middleware\HttpRedirect::class,

Clear the configuration:

php artisan optimize:clear

Supporting Proxy Frontends

If you are using a load balancer that connects to your Laravel backend using HTTP, detect the HTTP header X-Forwarded-Proto. Use this code for the handle() function instead:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

    public function handle(Request $request, Closure $next)

    {

        // If the client connected to the frontend (load balancer) via https

        // redirection is not necessary

 

        if ($request>headers>has(‘X-Forwarded-Proto’)) {

            if (strcmp($request>header(‘X-Forwarded-Proto’), ‘https’) === 0) {

                return $next($request);

            }

        }

 

        if (!$request>secure() && App::environment([‘staging’, ‘production’])) {

                return redirect()>secure($request>getRequestUri(), 301);

        }

 

        return $next($request);

    }

Warning

If your Laravel application does not have a proxy (load balancer) accepting traffic, do not add the proxy code. A smart hacker could manually add the header X-Forwarded-Proto and bypass the HTTP Redirect feature.

If you allow your Laravel backend to be accessed from a load balancer and directly from the Internet, add logic to only process the X-Forwarded-Proto header if the request arrives from a known frontend. Google Cloud HTTP(S) Load Balancers use the 130.211.0.0/22 and 35.191.0.0/16 IP address ranges.

Additional Options

The above middleware will redirect requests that are handled by Laravel routes. I also recommend that Laravel always generate content using HTTPS based URLs. Examples are JavaScript and CSS references.

Edit app/Providers/AppServiceProvider.php

Near the top add:

use Illuminate\Support\Facades\App;

use URL;

Add the following code to the boot function:

    public function boot()

    {

        if (App::environment([‘staging’, ‘production’])) {

            URL::forceScheme(‘https’);

        }

    }

Summary

I prefer to implement multiple layers of security. When implementing HTTP Redirection, I try to implement this feature at each service layer. Starting with the backend (Laravel), then with the web server (Apache or Nginx), and finally at the load balancer. Sometimes mistakes are made, and one layer might disable HTTP Redirection. By enabling this feature in more than one service, I have a higher confidence level that clients’ data is and remains encrypted.

Photography Credits

I write free articles about technology. Recently, I learned about Pexels.com which provides free images. The image in this article is courtesy of Pixabay at Pexels.

I design software for enterprise-class systems and data centers. My background is 30+ years in storage (SCSI, FC, iSCSI, disk arrays, imaging) virtualization. 20+ years in identity, security, and forensics.

For the past 14+ years, I have been working in the cloud (AWS, Azure, Google, Alibaba, IBM, Oracle) designing hybrid and multi-cloud software solutions. I am an MVP/GDE with several.

Related Posts

Laravel News Links