A Guide To Command-Line Data Manipulation

http://cloud.netlifyusercontent.com/assets/344dbf88-fdf9-42bb-adb4-46f01eedd629/9865d051-a600-4de7-a961-8b39a9226757/guide-command-line-data-manipulation-cli-miller.jpg

Allow me to preface this article by saying that I’m not a terminal person. I don’t use Vim. I find sed, grep, and awk convoluted and counter-intuitive. I prefer seeing my files in a nice UI. Despite all that, I got into the habit of reaching for command-line interfaces (CLIs) when I had small, dedicated tasks to complete. Why? I’ll explain all of that below. In this article, you’ll also learn how to use a CLI tool named Miller to manipulate data from CSV, TSV and/or JSON files.

Why Use The Command Line?

Everything that I’m showing here can be done with regular code. You can load the file, parse the CSV data, and then transform it using regular JavaScript, Python, or any other language. But there are a few reasons why I reach out for command-line interfaces (CLIs) whenever I need to transform data:

  • Easier to read.
    It is faster (for me) to write a script in JavaScript or Python for my usual data processing. But, a script can be confusing to come back to. In my experience, command-line manipulations are harder to write initially but easier to read afterward.
  • Easier to reproduce.
    Thanks to package managers like Homebrew, CLIs are much easier to install than they used to be. No need to figure out the correct version of Node.js or Python, the package manager takes care of that for you.
  • Ages well.
    Compared to modern programming languages, CLIs are old. They change a lot more slowly than languages and frameworks.

What Is Miller?

The main reason I love Miller is that it’s a standalone tool. There are many great tools for data manipulation, but every other tool I found was part of a specific ecosystem. The tools written in Python required knowing how to use pip and virtual environments; for those written in Rust, it was cargo, and so on.

On top of that, it’s fast. The data files are streamed, not held in memory, which means that you can perform operations on large files without freezing your computer.

As a bonus, Miller is actively maintained, John Kerl really keeps on top of PRs and issues. As a developer, I always get a satisfying feeling when I see a neat and maintained open-source project with great documentation.

Installation

  • Linux: apt-get install miller or Homebrew.
  • macOS: brew install miller using Homebrew.
  • Windows: choco install miller using Chocolatey.

That’s it, and you should now have the mlr command available in your terminal.

Run mlr help topics to see if it worked. This will give you instructions to navigate the built-in documentation. You shouldn’t need it, though; that’s what this tutorial is for!

How mlr Works

Miller commands work the following way:

mlr [input/output file formats] [verbs] [file]

Example: mlr --csv filter '$color != "red"' example.csv

Let’s deconstruct:

  • --csv specifies the input file format. It’s a CSV file.
  • filter is what we’re doing on the file, called a “verb” in the documentation. In this case, we’re filtering every row that doesn’t have the field color set to "red". There are many other verbs like sort and cut that we’ll explore later.
  • example.csv is the file that we’re manipulating.

Operations Overview

We can use those verbs to run specific operations on your data. There’s a lot we can do. Let’s explore.

Data

I’ll be using a dataset of IMDb ratings for American TV dramas created by The Economist. You can download it here or find it in the repo for this article.

Note: For the sake of brevity, I’ve renamed the file from mlr --csv head ./IMDb_Economist_tv_ratings.csv to tv_ratings.csv.

Above, I mentioned that every command contains a specific operation or verb. Let’s learn our first one, called head. What it does is show you the beginning of the file (the “head”) rather than print the entire file in the console.

You can run the following command:

`mlr --csv head ./tv_ratings.csv`

And this is the output you’ll see:

titleId,seasonNumber,title,date,av_rating,share,genres
tt2879552,1,11.22.63,2016-03-10,8.489,0.51,"Drama,Mystery,Sci-Fi"
tt3148266,1,12 Monkeys,2015-02-27,8.3407,0.46,"Adventure,Drama,Mystery"
tt3148266,2,12 Monkeys,2016-05-30,8.8196,0.25,"Adventure,Drama,Mystery"
tt3148266,3,12 Monkeys,2017-05-19,9.0369,0.19,"Adventure,Drama,Mystery"
tt3148266,4,12 Monkeys,2018-06-26,9.1363,0.38,"Adventure,Drama,Mystery"
tt1837492,1,13 Reasons Why,2017-03-31,8.437,2.38,"Drama,Mystery"
tt1837492,2,13 Reasons Why,2018-05-18,7.5089,2.19,"Drama,Mystery"
tt0285331,1,24,2002-02-16,8.5641,6.67,"Action,Crime,Drama"
tt0285331,2,24,2003-02-09,8.7028,7.13,"Action,Crime,Drama"
tt0285331,3,24,2004-02-09,8.7173,5.88,"Action,Crime,Drama"

This is a bit hard to read, so let’s make it easier on the eye by adding --opprint.

mlr --csv --opprint head ./tv_ratings.csv

The resulting output will be the following:

titleId   seasonNumber title            date          av_rating   share   genres
tt2879552      1       11.22.63         2016-03-10    8.489       0.51    Drama,Mystery,Sci-Fi
tt3148266      1       12 Monkeys       2015-02-27    8.3407      0.46    Adventure,Drama,Mystery
tt3148266      2       12 Monkeys       2016-05-30    8.8196      0.25    Adventure,Drama,Mystery
tt3148266      3       12 Monkeys       2017-05-19    9.0369      0.19    Adventure,Drama,Mystery
tt3148266      4       12 Monkeys       2018-06-26    9.1363      0.38    Adventure,Drama,Mystery
tt1837492      1       13 Reasons Why   2017-03-31    8.437       2.38    Drama,Mystery
tt1837492      2       13 Reasons Why   2018-05-18    7.5089      2.19    Drama,Mystery
tt0285331      1       24               2002-02-16    8.5641      6.67    Action,Crime,Drama
tt0285331      2       24               2003-02-09    8.7028      7.13    Action,Crime,Drama
tt0285331      3       24               2004-02-09    8.7173      5.88    Action,Crime,Drama

Much better, isn’t it?

Note: Rather than typing --csv --opprint every time, we can use the --c2p option, which is a shortcut.

Chaining

That’s where the fun begins. Rather than run multiple commands, we can chain the verbs together by using the then keyword.

Remove columns

You can see that there’s a titleId column that isn’t very useful. Let’s get rid of it using the cut verb.

mlr --c2p cut -x -f titleId then head ./tv_ratings.csv

It gives you the following output:

seasonNumber  title            date         av_rating   share    genres
     1      11.22.63          2016-03-10    8.489       0.51     Drama,Mystery,Sci-Fi
     1      12 Monkeys        2015-02-27    8.3407      0.46     Adventure,Drama,Mystery
     2      12 Monkeys        2016-05-30    8.8196      0.25     Adventure,Drama,Mystery
     3      12 Monkeys        2017-05-19    9.0369      0.19     Adventure,Drama,Mystery
     4      12 Monkeys        2018-06-26    9.1363      0.38     Adventure,Drama,Mystery
     1      13 Reasons Why    2017-03-31    8.437       2.38     Drama,Mystery
     2      13 Reasons Why    2018-05-18    7.5089      2.19     Drama,Mystery
     1      24                2002-02-16    8.5641      6.67     Action,Crime,Drama
     2      24                2003-02-09    8.7028      7.13     Action,Crime,Drama
     3      24                2004-02-09    8.7173      5.88     Action,Crime,Drama

Fun Fact

This is how I first learned about Miller! I was playing with a CSV dataset for https://details.town/ that had a useless column, and I looked up “how to remove a column from CSV command line.” I discovered Miller, loved it, and then pitched an article to Smashing magazine. Now here we are!

Filter

This is the verb that I first showed earlier. We can remove all the rows that don’t match a specific expression, letting us clean our data with only a few characters.

If we only want the rating of the first seasons of every series in the dataset, this is how you do it:

mlr --c2p filter '$seasonNumber == 1' then head ./tv_ratings.csv

Sorting

We can sort our data based on a specific column like it would be in a UI like Excel or macOS Numbers. Here’s how you would sort your data based on the series with the highest rating:

mlr --c2p sort -nr av_rating then head ./tv_ratings.csv

The resulting output will be the following:

titleId   seasonNumber title                         date         av_rating  share   genres
tt0098887      1       Parenthood                    1990-11-13   9.6824     1.68    Comedy,Drama
tt0106028      6       Homicide: Life on the Street  1997-12-05   9.6        0.13    Crime,Drama,Mystery
tt0108968      5       Touched by an Angel           1998-11-15   9.6        0.08    Drama,Family,Fantasy
tt0903747      5       Breaking Bad                  2013-02-20   9.554      18.95   Crime,Drama,Thriller
tt0944947      6       Game of Thrones               2016-05-25   9.4943     15.18   Action,Adventure,Drama
tt3398228      5       BoJack Horseman               2018-09-14   9.4738     0.45    Animation,Comedy,Drama
tt0103352      3       Are You Afraid of the Dark?   1994-02-23   9.4349     2.6     Drama,Family,Fantasy
tt0944947      4       Game of Thrones               2014-05-09   9.4282     11.07   Action,Adventure,Drama
tt0976014      4       Greek                         2011-03-07   9.4        0.01    Comedy,Drama
tt0090466      4       L.A. Law                      1990-04-05   9.4        0.1     Drama

We can see that Parenthood, from 1990, has the highest rating on IMDb — who knew!

Saving Our Operations

By default, Miller only prints your processed data to the console. If we want to save it to another CSV file, we can use the > operator.

If we wanted to save our sorted data to a new CSV file, this is what the command would look like:

mlr --csv sort -nr av_rating ./tv_ratings.csv > sorted.csv

Convert CSV To JSON

Most of the time, you don’t use CSV data directly in your application. You convert it to a format that is easier to read or doesn’t require additional dependencies, like JSON.

Miller gives you the --c2j option to convert your data from CSV to JSON. Here’s how to do this for our sorted data:

mlr --c2j sort -nr av_rating ./tv_ratings.csv > sorted.json

Case study: Top 5 Athletes With Highest Number Of Medals In Rio 2016

Let’s apply everything we learned above to a real-world use case. Let’s say that you have a detailed dataset of every athlete who participated in the 2016 Olympic games in Rio, and you want to know who the 5 with the highest number of medals are.

First, download the athlete data as a CSV, then save it in a file named athletes.csv.

Let’s open up the following file:

mlr --c2p head ./athletes.csv

The resulting output will be something like the following:

id        name                nationality sex    date_of_birth height weight sport      gold silver bronze info
736041664 A Jesus Garcia      ESP         male   1969-10-17    1.72    64     athletics    0    0      0      -
532037425 A Lam Shin          KOR         female 1986-09-23    1.68    56     fencing      0    0      0      -
435962603 Aaron Brown         CAN         male   1992-05-27    1.98    79     athletics    0    0      1      -
521041435 Aaron Cook          MDA         male   1991-01-02    1.83    80     taekwondo    0    0      0      -
33922579  Aaron Gate          NZL         male   1990-11-26    1.81    71     cycling      0    0      0      -
173071782 Aaron Royle         AUS         male   1990-01-26    1.80    67     triathlon    0    0      0      -
266237702 Aaron Russell       USA         male   1993-06-04    2.05    98     volleyball   0    0      1      -
382571888 Aaron Younger       AUS         male   1991-09-25    1.93    100    aquatics     0    0      0      -
87689776  Aauri Lorena Bokesa ESP         female 1988-12-14    1.80    62     athletics    0    0      0      -

Optional: Clean Up The File

The CSV file has a few fields we don’t need. Let’s clean it up by removing the info , id , weight, and date_of_birth columns.

mlr --csv -I cut -x -f id,info,weight,date_of_birth athletes.csv

Now we can move to our original problem: we want to find who won the highest number of medals. We have how many of each medal (bronze, silver, and gold) the athletes won, but not the total number of medals per athlete.

Let’s compute a new value called medals which corresponds to this total number (bronze, silver, and gold added together).

mlr --c2p put '$medals=$bronze+$silver+$gold' then head ./athletes.csv

It gives you the following output:

name                 nationality   sex      height  sport        gold silver bronze medals
A Jesus Garcia       ESP           male     1.72    athletics      0    0      0      0
A Lam Shin           KOR           female   1.68    fencing        0    0      0      0
Aaron Brown          CAN           male     1.98    athletics      0    0      1      1
Aaron Cook           MDA           male     1.83    taekwondo      0    0      0      0
Aaron Gate           NZL           male     1.81    cycling        0    0      0      0
Aaron Royle          AUS           male     1.80    triathlon      0    0      0      0
Aaron Russell        USA           male     2.05    volleyball     0    0      1      1
Aaron Younger        AUS           male     1.93    aquatics       0    0      0      0
Aauri Lorena Bokesa  ESP           female   1.80    athletics      0    0      0      0
Ababel Yeshaneh      ETH           female   1.65    athletics      0    0      0      0

Sort by the highest number of medals by adding a sort.

mlr --c2p put '$medals=$bronze+$silver+$gold' \
    then sort -nr medals \
    then head ./athletes.csv

Respectively, the resulting output will be the following:

name              nationality  sex     height  sport       gold silver bronze medals
Michael Phelps    USA          male    1.94    aquatics      5    1      0      6
Katie Ledecky     USA          female  1.83    aquatics      4    1      0      5
Simone Biles      USA          female  1.45    gymnastics    4    0      1      5
Emma McKeon       AUS          female  1.80    aquatics      1    2      1      4
Katinka Hosszu    HUN          female  1.75    aquatics      3    1      0      4
Madeline Dirado   USA          female  1.76    aquatics      2    1      1      4
Nathan Adrian     USA          male    1.99    aquatics      2    0      2      4
Penny Oleksiak    CAN          female  1.86    aquatics      1    1      2      4
Simone Manuel     USA          female  1.78    aquatics      2    2      0      4
Alexandra Raisman USA          female  1.58    gymnastics    1    2      0      3

Restrict to the top 5 by adding -n 5 to your head operation.

mlr --c2p put '$medals=$bronze+$silver+$gold' \
    then sort -nr medals \
    then head -n 5 ./athletes.csv

You will end up with the following file:

name             nationality  sex      height  sport        gold silver bronze medals
Michael Phelps   USA          male     1.94    aquatics       5     1      0      6
Katie Ledecky    USA          female   1.83    aquatics       4     1      0      5
Simone Biles     USA          female   1.45    gymnastics     4     0      1      5
Emma McKeon      AUS          female   1.80    aquatics       1     2      1      4
Katinka Hosszu   HUN          female   1.75    aquatics       3     1      0      4

As a final step, let’s convert this into a JSON file with the --c2j option.

Here is our final command:

mlr --c2j put '$medals=$bronze+$silver+$gold' \
    then sort -nr medals \
    then head -n 5 ./athletes.csv > top5.json

With a single command, we’ve computed new data, sorted the result, truncated it, and converted it to JSON.

[
  {
    "name": "Michael Phelps",
    "nationality": "USA",
    "sex": "male",
    "height": 1.94,
    "weight": 90,
    "sport": "aquatics",
    "gold": 5,
    "silver": 1,
    "bronze": 0,
    "medals": 6
  }
  // Other entries omitted for brevity.
]

Bonus: If you wanted to show the top 5 women, you could add a filter.

mlr --c2p put '$medals=$bronze+$silver+$gold' then sort -nr medals then filter '$sex == "female"' then head -n 5 ./athletes.csv

Respectively, you would end up with the following output:

name              nationality   sex       height   sport        gold silver bronze medals
Katie Ledecky     USA           female    1.83     aquatics       4    1      0      5
Simone Biles      USA           female    1.45     gymnastics     4    0      1      5
Emma McKeon       AUS           female    1.80     aquatics       1    2      1      4
Katinka Hosszu    HUN           female    1.75     aquatics       3    1      0      4
Madeline Dirado   USA           female    1.76     aquatics       2    1      1      4

Conclusion

I hope this article showed you how versatile Miller is and gave you a taste of the power of command-line tools. Feel free to scourge the internet for the best CLI next time you find yourself writing yet another random script.

Resources

Further Reading on Smashing Magazine

Smashing Magazine

If You’ve Ever Used LastPass, You Should Change All Your Passwords Now

https://static1.makeuseofimages.com/wordpress/wp-content/uploads/2022/12/open-padlock-hanging-on-a-wire-fence-with-snow-falling-in-the-background.jpg

Personal details and password vaults containing the sign-in credentials of millions of users are now in the hands of criminals. If you’ve ever used the password manager, LastPass, you should change all of your passwords for everything, now. And you should immediately take further measures to protect yourself.

What Happened in the 2022 LastPass Data Breach?

LastPass is a password management service which operates on a "freemium" model. Users can store all of their passwords and logins for online services with LastPass, and access them through the web interface, through browser add-ons, and through dedicated smartphone apps.

Passwords are stored in "vaults", which are protected by a single master password.

In August 2022, LastPass announced that criminals had used a compromised developer account to access the LastPass development environment, source code, and technical information.

Further details were released in November 2022, when LastPass added that some customer data had been disclosed.

The true severity of the breach was revealed on December 22, when a LastPass blog post noted that criminals had used some of the information obtained in the earlier attack to steal backup data including customer names, addresses and phone numbers, email addresses, IP addresses, and partial credit card numbers. Additionally, they managed to steal user password vaults containing unencrypted website URLs and site names, as well as encrypted usernames and passwords.

Is It Difficult for Criminals to Crack Your LastPass Master Password?

Theoretically, yes, hackers should find it difficult to crack your master password. The LastPass blog post notes that if you use their default recommended settings, "it would take millions of years to guess your master password using generally-available password-cracking technology."

LastPass requires the master password to be a minimum of 12 characters, and recommends "that you never reuse your master password on other websites."

However, LastPass is unique among password management services in that it allows users to set a password hint to remind them of their master password should they lose it.

Effectively, this encourages users to use dictionary words and phrases as part of their password, rather than a truly random strong password. No password hint is going to help if your password is "lVoT=.N]4CmU".

The LastPass password vaults have been in the hands of criminals for some time now, and even though they’re encrypted, they will eventually be subject to brute force attacks.

Attackers will find their work easier thanks to the existence of massive databases of commonly used passwords. You can download a 17GB password list comprising the 613 million most common passwords from haveibeenpwned, for instance. Other password and credential lists are available on the dark web.

To try each of the half billion most common keys against an individual vault would take minutes, and although relatively few would be the required 12 characters, it’s likely that cybercriminals will be able to easily break into a good proportion of vaults.

Add to that the fact that computing power increases year-on-year, and that motivated criminals can use distributed networks to help with the effort; "millions of years" doesn’t seem feasible for the majority of accounts.

Does the LastPass Breach Just Affect Passwords?

While the headline news is that criminals can take their time to break into your LastPass vault, they can take advantage of you in other ways by using your name, address, phone number, email address, IP address, and partial credit card number.

These can be used for a number of nefarious purposes including spearphishing attacks against you and your contacts, identity theft, taking out credit and loans in your name, and SIM swap attacks.

How Can You Protect Yourself After the LastPass Data Breaches?

You should assume that within a few years, your master password will be compromised and all the passwords contained within will be known to criminals. You should change them now, and use unique passwords you have never used before, and which aren’t in any of the commonly used password lists.

With regard to the other data criminals obtained from LastPass, you should freeze your credit, and engage a credit monitoring service to monitor any new card or loan applications in your name. If you’re able to change your phone number without too much inconvenience, you should do that too.

Take Responsibility for Your Own Security

It’s easy to blame LastPass for the data breaches which saw your password vaults and personal details fall into the hands of criminals, but password management services that secure your life and help you generate unique combos are still the best way to secure your online life.

One way to make it more difficult for would-be thieves to get hold of your vital data is to host a password manager on your own hardware. It’s cheap, easy to do, and some solutions, such as VaultWarden, can even be deployed on a Raspberry Pi Zero.

MUO – Feed

3 Seconds of Every TNG Episode Is a Wild Way to Experience Star Trek

https://i.kinja-img.com/gawker-media/image/upload/c_fill,f_auto,fl_progressive,g_center,h_675,pg_1,q_80,w_1200/b46c3e910048e66c46e9610ba8e6fad3.png

There’s a lot of Star Trek, and a lot of it is very watchable—especially in the iconic sophomore series The Next Generation, which helped truly catapult the franchise into the pop culture stratosphere. Asking a Trek neophyte to dive into over five days of TV is a daunting task, however. So why not just give them an  appetizer of everything?

This incredible mash-up of all 178 episodes of Star Trek: The Next Generation by Sentinel of Something condenses the seven seasons of boldly going done by Captain Jean-Luc Picard and the crew of the U.S.S. Enterprise into a little over nine minutes… by giving you three random seconds of every episode. It’s unhinged and it’s perfect.

3 Seconds of Every Star Trek: TNG Episode

There’s an artistry to the consideration here. Do you pick an iconic visual, a perfect, but short enough line of dialogue, a joke, a sad moment, or a shot of action? Just how do you distill an entire episode of TNG, from the very best to the very worst, in just three seconds? The answer is that you not take Star Trek seriously, so what you get is three manic seconds of out-of-context weirdness, 178 times in a row.

Okay, it’s probably not helpful to a Star Trek newbie looking to shave some time off of a marathon. But for TNG fans, it’s a delightfully zany whirlwind trip through one of the best sci-fi TV shows of all time.


Want more io9 news? Check out when to expect the latest Marvel, Star Wars, and Star Trek releases, what’s next for the DC Universe on film and TV, and everything you need to know about the future of Doctor Who.

Gizmodo

A quick guide on why you should use Laravel Octane to scale your app

https://laravelnews.s3.amazonaws.com/images/og-image—laravel.png

Laravel has gained a lot of attention in recent years, though it has been a charm in the eyes of product owners for small-scale web applications. However, developing a large-scale application with Laravel still seems like a hard nut to crack. In this blog post, we have covered the latest addition to the Laravel framework, Laravel Octane, and the buzz Octane to scale Laravel applications has created around the market to fill the scalability gap of Laravel experienced by product owners

Introduction

Laravel possesses a simple and decent syntax that allows developers to make their web application development easier. The only drawback which led Laravel to lack on every front is Scalability which is why Laravel is not among the top most preferred languages among developers. After the introduction of Octane, this perception shifted slightly in favor of Laravel and its community. But, before we move to our Topic of Octane to Laravel Applications. Let us first understand What Laravel Octane is and How does it work.

What is Laravel Octane?

Laravel Octane is an open-source package introduced by Taylor Otwell in his Laracon Online Talk, with its release date on May 12, 2021. This package serves the Laravel application and uses the web server Swoole and RoadRunner to help improve performance. It is initialized with the first request, stored in memory, and does not reinitialize at subsequent requests. It possesses 3.3k Stars and 234 Forks on GitHub.

Octane Key Features

The key features of Octane include several features that allow it to present better performance than the traditional Laravel; given below are a few such features:

  • It makes your application extremely fast and efficient
  • It saves resources compared to conventional Laravel applications.

How does Laravel Octane work?

Octane Laravel is built on top of the traditional Laravel and PHP version 8.0 and above. It is not a separate language but is an addition to the standard Laravel. Octane works on its servers via Swoole and RoadRunner, which the traditional Apache and Nginx should redirect for incoming traffic.

Octane Open Swoole

It is a software development kit that allows product owners to create efficient and scalable web applications. Open Swoole is a fork of the Swoole framework initiated in late 2021 concerning the internal conflicts within the developer community of swoole.

Octane Roadrunner

It is an open-source, high-performance, serviceable PHP application server that is effective and efficient in replacing the traditional Nginx setups delivering better performance and flexibility.

We can also mention that though Swoole and RoadRunner still use worker processes for all requests, they only serve the first request to boot the framework (dependency containers), and the remaining ones are from the framework’s bootstrapped version. This collectively makes your application perform impressively fast from your existing application.

Octane Roadrunner

Octane Laravel catches the bootstrapped Laravel framework in RAM with the first request. After that, all the requests use the already bootstrapped framework from RAM instead of doing the whole reading of files from the hard disk and re-bootstrapping the framework, making the Laravel application faster and improving its performance to a significant extent.

Laravel with Apache vs Laravel Octane Benchmarks

According to the creator of Laravel, Taylor Otwell, after the introduction of Apache and Octane, there is almost no requirement to choose Lumen for new projects. As we know that Lumen is a Laravel-based micro-framework intentionally devoid of a few components, making it fast. Also, using the Laravel Octane as its server is a fast better option than Lumen. To understand that better, let us look at a few benchmarks.

In this benchmark, we took in a total of three modes Laravel Octane vs Laravel Apache Web Server vs Laravel built-in server:

  • Laravel with Octane
  • Laravel with Apache Web Server
  • Laravel with its in-built server.

In this benchmarking, we used the tool wrk with the given below configuration:

1wrk -tl -c50 URL

The first attest is performed on the Just Hello World HTML, and the results that we have received are as follows:

Mode Requests Handled in 10-Seconds Requests Per Second (RPS)
Laravel with Octane 2667 266 rps
Laravel with Apache 1210 121 rps
Laravel with its in-built Server 705 70 rps

These, Laravel Octane benchmark metrics are enough to give you insight that Octane Laravel is far faster in terms of performance than the traditional Apache-Server Laravel.

Why Scale your Laravel Application with Octane?

Octane catches your data and objects in the memory, which contributes to the improvement in the performance of your Laravel application by upto 30%. This improvement in the performance is observed due to the eliminating of disk reads, process forks, and various other overheads that come to function whensoever the framework is booted up on every request, which allows the Laravel application to manage thousands of connections quickly, thereby contributing to the increase in traffic or the need to be highly available. Also, this is a productive method that too with a little expense.

Additionally, a minor or no change in the codebase is required under certain circumstances. But it is related to how your application is structured.

What to Look for Before Scaling your Laravel Application with Octane (Prerequisites)

Octane can improve your web application development performance to a great extent, but that doesn’t mean that it is sufficiently capable of improving the performance of your web application and providing you with the freedom to scale.

It involves prerequisites like PHP version 8.0 and adopting the stateful programming convention. Therefore, before you finalize using Octane, the first and foremost step is to update your PHP version and install it. Though it seems a simple task, the severe hiccup with this is that numerous websites that work on Laravel are using an older version of Laravel, which does not support the Octane and hence needs to be updated with the latest Laravel version.

Also, many product owners, whether experienced or fresher, are not familiar with the stateful framework, which again requires efforts to understand it before they finalize to implement Laravel Octane within your web application to scale.

Other Necessary Factors

In addition to the above-given prerequisites, a few other factors must be considered to make a scalable web application. Let s look at those factors given below:

Refactoring Implications

For applications using session state or a few middleware types, some prior changes to the code are required for using the Octane. Also, you should add support for Swoole coroutine-based sessions and remove middlewares that aren’t compatible with Swoole.

The rest of the changes that need to be made involve the Dependency Injection Container. You can refer to the Laravel Official Document of Laravel for more details on How to do it.

Handling Code Changes

The traditional Laravel requires you to start or stop the entire Laravel application instance whensoever a change is made. However, contrary to this, with Octane, you get a convenient way to monitor your codebase automatically and restart your Octane server without any manual interventions, which allows you to make the developments easily by altering the code at a single instance or in one place instead of restarting the whole application insta\nce every time the codebase is updated.

Are there any other Options to Scale Laravel applications?

Yes, there is always another solution to a problem. Though Laravel Octane is a great tool to resolve the issue of scaling within the traditional Laravel application, it is still not the only option; plenty of other options can contribute to your scaling needs. Let us have a look at a few of them:

Horizontal Scaling

The first comes horizontal scaling, a simple yet least recommended method to scale your Laravel application. The reason behind this is that though it is efficient, it becomes more and more complex and expensive as the running costs keep increasing and more difficult as you migrate to a different server. Further, it is still a limited solution as you cannot scale on demand with this setup.

Load Balancing (with AWS or Without AWS)

For scaling your Laravel Application on AWS, you can automatically scale the applications to a limited extent for applications with the same MySQL database. These servers thereafter use a single Amazon RDS or Amazon Relational Database.

Though this seems a viable option to scale compared to Horizontal scaling, the hidden costs involved here are not transparent. Also, it is less effective than Octane Laravel. Still, the Significant benefit is that it allows you to scale without any alteration needed in your existing codebase, as with Laravel Octane.

Laravel Vapor and Octane

Laravel Vapor is an auto-scaling, serverless deployment platform for Laravel. Vapor allows for scalability and simplicity of the serverless. When combined with Octane, the Laravel Octane Vapor combination provides autoscaling and faster load times at any scale. However, to understand it, even more, let us take an example from the official Laravel Documentation.

Vapor project on the us-west-1 region of Amazon. This Vapor application is configured with 1 GB of RAM and an RDS MySQL instance (db.t2.micro) with 1 VCPU and 1Gib RAM.

Now, looking at the API endpoint that gets a user from the database. With Octane, its endpoint becomes 7x faster. It uses 44% less memory:

Before Vapor’s Octane integration

1Time Taken by Request: 39.40 ms, Memory Usage: 169 MB

2Time Taken by Request: 40.20 ms, Memory Usage: 169 MB

3Time Taken by Request: 37.71 ms, Memory Usage: 169 MB

4Time Taken by Request: 42.16 ms, Memory Usage: 169 MB

5Time Taken by Request: 40.60 ms, Memory Usage: 169 MB

6Time Taken by Request: 45.75 ms, Memory Usage: 169 MB

After Vapor’s Octane integration

1Time Taken by Request: 6.78 ms, Memory Usage: 112 MB

2Time Taken by Request: 6.64 ms, Memory Usage: 112 MB

3Time Taken by Request: 6.67 ms, Memory Usage: 112 MB

4Time Taken by Request: 6.38 ms, Memory Usage: 112 MB

5Time Taken by Request: 6.75 ms, Memory Usage: 112 MB

6Time Taken by Request: 6.47 ms, Memory Usage: 112 MB

While the “login” route which renders a static template. Octane at this endpoint has made it almost 3x faster. It uses 35% less memory:

Before Vapor’s Octane integration

1Time Taken by Request: 11.32 ms, Memory Usage: 165 MB

2Time Taken by Request: 11.35 ms, Memory Usage: 165 MB

3Time Taken by Request: 11.29 ms, Memory Usage: 165 MB

4Time Taken by Request: 11.29 ms, Memory Usage: 165 MB

5Time Taken by Request: 11.36 ms, Memory Usage: 165 MB

6Time Taken by Request: 11.43 ms, Memory Usage: 165 MB

After Vapor’s Octane integration

1Time Taken by Request: 4.89 ms, Memory Usage: 108 MB

2Time Taken by Request: 4.89 ms, Memory Usage: 108 MB

3Time Taken by Request: 4.83 ms, Memory Usage: 108 MB

4Time Taken by Request: 4.66 ms, Memory Usage: 108 MB

5Time Taken by Request: 4.79 ms, Memory Usage: 108 MB

6Time Taken by Request: 4.91 ms, Memory Usage: 108 MB

Thus, we can infer that Octane decreases the duration of requests as well as memory usage. Also, adding the AWS applies 1ms billing granularity on Lambda, so you need to pay comparatively less for your HTTP function. Hence, the combination of Laravel Octane Vapor is a great choice for making a Laravel Octane Vapor web application with scalability.

Conclusion

This is what we have on scaling your Laravel application using Laravel Octane. Though it is a decent option to scale your existing Laravel application, it shows significant improvement in terms of performance in comparison to the traditional Laravel application. Still, its incompatibility with the existing codebase of many traditional Laravel applications is a setback. It can be a good choice if you wish to improve the performance of your application, but if you plan on saving an already struggling application, then Octane Laravel is not the best option. You must do thorough research before making the final call.

Additionally, when you have opted for Octane, make sure to use the Octane Compatible packages you add to your project to get the ideal application compatible with Octane if you are still hesitant about making the final call or scaling your Laravel application. A Laravel Development Company like Bacancy can assist you in the same; our expert professionals are always available to guide and walk alongside you through your entire web application development process.

Laravel News

The Rimfire Report: Why Your 10/22 Rifle is Inaccurate

https://www.thefirearmblog.com/blog/wp-content/uploads/2022/12/keyhole-targetfeat-180×180.png

The Rimfire Report: Why Your 10/22 Rifle is InaccurateHello everyone and welcome back to another edition of The Rimfire Report! This ongoing series is all about the rimfire firearm world and its various firearms, shooting sports, ammunition types, and history! Last week on The Rimfire Report we talked about the historical significance of what was collectively known as gallery guns – basically versions of rifles […]

Read More …

The post The Rimfire Report: Why Your 10/22 Rifle is Inaccurate appeared first on The Firearm Blog.

The Firearm Blog