As you may know, sometimes MySQL can be memory-hungry. Of course having data in memory is always better than disk… RAM is still much faster than any SSD disk.

This is the reason why we recommended to have the working set as much as possible in memory (I assume you are using InnoDB of course).

Also this why you don’t want to use Swap for MySQL, but don’t forget that a slow MySQL is always better than no MySQL at all, so don’t forget to setup a Swap partition but try to avoid using it. In fact, I saw many people just removing the Swap partition… and then OOM Killer did its job… and mysqld is often its first victim.

MySQL allocates buffers and caches to improve performance of database operations. That process is explained in details in the manual.

In this article series, I will provide you some information to check MySQL’s memory consumption and what configuration settings or actions can be made to understand and control the memory usage of MySQL.

We will start the series by the Operating System.

Operating System

In the OS level, there are some commands we can use to understand MySQL’s memory usage.

Memory Usage

You can check mysqld‘s memory usage from the command line:

# ps -eo size,pid,user,command --sort -size | grep [m]ysqld \
  | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' \ 
  |cut -d "" -f2 | cut -d "-" -f1

1841.10 Mb /usr/sbin/mysqld 
   0.46 Mb /bin/sh /usr/bin/mysqld_safe

top can also be used to verify this.

For top 3.2:

# top -ba -n1 -p $(pidof mysqld) | grep PID -A 1
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                
 1752 mysql     20   0 1943m 664m  15m S  0.0 11.1  19:55.13 mysqld                                                                                                                                 
# top -ba -n1 -m -p $(pidof mysqld) | grep PID -A 1
  PID USER      PR  NI  USED  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                
 1752 mysql     20   0  664m 664m  15m S  2.0 11.1  19:55.17 mysqld

For more recent top, you can use top -b -o %MEM -n1 -p $(pidof mysqld) | grep PID -A 1

VIRT represents the total amount of virtual memory used by mysql. It includes all code, data and shared libraries plus pages that have eventually been swapped out.

USED reports the sum of process rss (resident set size, the portion of memory occupied by a process that is held in RAM) and swap total count.

We will see later what we can check from MySQL client.

SWAP

So we see that this can eventually include the swapped pages too. Let’s check if mysqld is using the swap, and the first thing to do is to check is the machine has some information in swap already:

# free -m
             total       used       free     shared    buffers     cached
Mem:          5965       4433       1532        128        454       2359
-/+ buffers/cache:       1619       4346
Swap:         2045         30       2015

We can see that a little amount of swap is used (30MB), is it by MySQL ? Let’s verify:

# cat /proc/$(pidof mysqld)/status | grep Swap
VmSwap:	       0 kB

Great, mysqld si not swapping. In case you really want to know which processes have swapped, run the following command:

for i in $(ls -d /proc/[0-9]*) 
do  
   out=$(grep Swap $i/status 2>/dev/null)
   if [ "x$(echo $out | awk '{print $2}')" != "x0" ] && [ "x$(echo $out | awk '{print $2}')" != "x" ]
   then    
	  echo "$(ps -p $(echo $i | cut -d'/' -f3) \
         | tail -n 1 | awk '{print $4'}): $(echo $out | awk '{print $2 $3}')" 
   fi
done

Of course the pages in the swap could have been there for a long time already and never been used since…  to be sure, I recommend to use vmstat and verify the columns si and so (a trending system is highly recommended):

# vmstat  1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0  31252 1391672 505840 2523844    0    0     2    57    5    2  3  1 96  0  0	
 1  0  31252 1392664 505840 2523844    0    0     0   328  358  437  6  1 92  1  0	
 0  0  31252 1390820 505856 2523932    0    0     0  2024 1312 2818 28  3 67  2  0	
 0  0  31252 1391440 505860 2523980    0    0     0   596  484  931  1  1 98  1  0	
 0  0  31252 1391440 505860 2523980    0    0     0  1964  500  970  0  1 96  3  0	
 0  0  31252 1391440 505860 2523980    0    0     0    72  255  392  0  0 98  2  0	
 0  0  31252 1391440 505860 2523980    0    0     0     0  222  376  0  0 99  0  0	
 0  0  31252 1391440 505908 2524096    0    0     0  3592 1468 2095 34  6 52  9  0	
 0  0  31252 1391688 505928 2524092    0    0     0  1356  709 1179 12  1 85  2  0	
 0  0  31252 1390696 505928 2524092    0    0     0   152  350  950  4  6 90  1  0

On this server, we can see that mysqld is not using the swap, but if it was the case and some free RAM was still available, what could have been done ?

If this was the case, you must check 2 direct causes:

1. swappiness
2. numa

Swappiness

The swappiness parameter controls the tendency of the kernel to move processes out of physical memory and put them onto the swap disk partition. As I explained earlier,  disks are much slower than RAM, therefore this leads to slower response times for system and applications if processes are too aggressively moved out of memory. A high swappiness value means that the kernel will be more apt to unmap mapped pages. A low swappiness value means the opposite, the kernel will be less apt to unmap mapped pages. This means that the higher is the swappiness value, the more the system will swap !

The default value (60) is too high for a dedicated MySQL Server and should be reduced. Pay attention that with older Linux kernels (prior 2.6.32), 0 meant that the kernel should avoid swapping processes out of physical memory for as long as possible. Now the same value totally avoid swap to be used. I recommend to set it to 1 or 5.

# sysctl -w vn.swappinness=1

Numa

For servers having multiple NUMA cores, the recommendation is to set the NUMA mode to interleaved which balances memory allocation to all nodes. MySQL 8.0 supports NUMA for InnoDB. You just need to enable it in your configuration: innodb_numa_interleave = 1

To check if you have multiple NUMA nodes, you can use numactl -H

These are two different output:

# numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64379 MB
node 0 free: 2637 MB
node distances:
node 0 
0: 10

# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 2 4 6
node 0 size: 8182 MB
node 0 free: 221 MB
node 1 cpus: 8 10 12 14
node 1 size: 8192 MB
node 1 free: 49 MB
node 2 cpus: 9 11 13 15
node 2 size: 8192 MB
node 2 free: 4234 MB
node 3 cpus: 1 3 5 7
node 3 size: 8192 MB
node 3 free: 5454 MB
node distances:
node 0 1 2 3 
0: 10 16 16 16 
1: 16 10 16 16 
2: 16 16 10 16 
3: 16 16 16 10

We can see that when there are multiple NUMA nodes (right column), by default the memory is not spread equally between all those nodes. This can lead to more swapping. Check these two nice articles from Jeremy Cole explaining this behavior:

• http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
• http://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-mysql/

Filesystem Cache

Another point we can check from the OS is the filesystem cache.

By default, Linux will use the filesystem cache for all I/O accesses (this is one of the reason why using MyISAM is not recommended, as this storage engine relies on the FS cache and can lead in loosing data as Linux sync those writes up to every 10sec). Of course as you are using InnoDB, with O_DIRECT as innodb_flush_method, MySQL will bypass the filesystem cache (InnoDB has already enough optimized caches anyway and one extra is not necessary). InnoDB will then not use any FS Cache Memory for the data files (*.ibd).

But there are of course other files used in MySQL that will still use the FS Cache. Let’s check this example:

# ~fred/dbsake fincore binlog.000017
binlog.000017: total_pages=120841 cached=50556 percent=41.84

# ls -lh binlog.000017 
-rw-r----- 1 mysql mysql 473M Sep 18 07:17 binlog.000017

# free -m
             total       used       free     shared    buffers     cached
Mem:          5965       4608       1356        128        435       2456
-/+ buffers/cache:       1716       4249
Swap:         2045         30       2015

# ~fred/dbsake uncache binlog.000017 
Uncached binlog.000017
# free -m
             total       used       free     shared    buffers     cached
Mem:          5965       4413       1552        128        435       2259
-/+ buffers/cache:       1718       4247
Swap:         2045         30       2015

Some explanations. I started checking how much of one binary log was present in the filesystem cache (using dbsake fincore), and we could see that 42% of 473M were using the RAM as FS cache. Then I forced an unmap of those pages in the cache (using fincore uncache) and finally, you could see that we freed +/- 195MB of RAM.

You could be surprised to see which logs or datafiles are using the FS cache (making a file copy for example). I really encourage you to verify this

The next article will be about what can be seen from MySQL’s side and what are the best configuration practices.

via Planet MySQL
MySQL and Memory: a love story (part 1)

On Aug. 3, 2018, Gov. John Kasich signed Senate Bill 220, also known as the Ohio Data Protection Act. Under the Act, eligible organizations may rely on their conformance to certain cybersecurity frameworks as an affirmative defense against tort claims in data breach litigation. The Act is intended to provide organizations with a legal incentive to implement written cybersecurity programs. 
In order to qualify for this new defense, the organization must implement a written cybersecurity program…

via Columbus Business News – Local Columbus News | Business First of Columbus
New Ohio law incentivizes businesses that comply with cybersecurity programs

We’ve finally made it to the most intimidating of Eloquent relationships: many to many polymorphic. Don’t worry; it’s far simpler to understand than you might initially think. In this lesson, we’ll construct a standard many-to-many relationship: a user can "like" a post. Then, we’ll throw a wrench into the mix. What if a user can also "like" a comment? How might we restructure our existing structure to allow for this?
via Laracasts
Many to Many Polymorphic Relations

We have all known that person, the one who monopolizes the conversations or seems to always have a flattering story to tell – always about themselves. Maybe, we listen politely for a bit and make a break for it as soon as we possibly can. Perhaps, we suffer silently while inwardly rolling our eyes.

But what if that person is you? Would you even know it?

Identifying narcissism in others is one thing. There are some glaring signs that we can all agree upon. However, identifying it in yourself can be more difficult.

Most narcissists have no idea that they fall into this category. They often view themselves as unique, special, outgoing or justifiably confident. What they don’t realize is that those views taken to the extreme can cause many problems in their relationships, work or even their family.

Why Narcissism Is Dangerous

Although narcissism can be a dangerous trait, it exists in all of us to some degree. In its most mild and most common form, it can be looked at as a desire to feel special. We can all relate to that on some level. Wanting to feel like there is something that makes us unique, interesting or desirable is completely normal.

For some, however, this goes far beyond the general desire to stand out. For certain people, narcissism can become so extreme that they can no longer relate to anyone around them. They feel they are in a category of greatness no one can understand. They may also assume that they are so unique that rules don’t apply to them or that they are more interesting and have far more to offer than anyone else.

The narcissist may become so intent on proving their superiority that they lie, manipulate or hurt those around them. They do these things in order to maintain the feeling that they are better and more worthy of admiration than anyone else. While these behaviors are destructive in and of themselves, they also have an impact that goes far beyond the day-to-day.

Relationships for someone with extreme narcissistic behavior are often either short-lived or unhealthy. A healthy relationship balances the needs of both partners with each respecting and appreciating the other. When your focus is completely on yourself, this isn’t possible. People in relationships with narcissists will often be used as a prop for the narcissist’s ego. This should prompt a swift end to a relationship because it often results in emotional and mental abuse.

Recognizing Narcissism

This isn’t really a fair question because almost everyone will answer no. None of us wants to feel like our behavior rises to that level. It is possible, however, that you are or someone you love is exhibiting narcissistic tendencies.

Consider the following questions when recognizing narcissism:

• Do you often feel like your decision-making skills are better than others?
• Do you override what others want because you are sure your desires/interests/plans are better?
• Are your achievements over-embellished?
• Do you often think, “they will thank me for this later” when you are doing things?
• Do you feel like you have an undiscovered talent and you are just biding your time until people see it?
• Are you a good listener or are you just waiting for someone to stop talking so that you can talk?
• Do you feel like you have a way to get around things you don’t like and that the rules don’t really apply to you?
• Do you expect to receive special treatment above others in the same situation?
• Are you looking forward to talking about your achievements?
• Do you always have an “I can do you one better” kind of story when others have talked about something?
• Do you reject criticism and feedback as being stupid or a product of other people’s jealousy of you?

If the answer to more than a couple of these questions is yes, you or the person you answered for could likely be classified as a narcissist.

Narcissism can be a disabling trait. It destroys friendships, romantic relationships, careers, and causes a great deal of personal pain and conflict. When a person finally comes to terms with the truth, it can be a difficult road to recovery.

If you recognize these behaviors in yourself or someone else, it is time to initiate change. For a narcissist, change is a big step in the right direction.

That change is possible but it won’t happen overnight. So, if you or someone you love is trying to make positive alterations, be patient. With time, effort, and a good support system it can be done.

The post How To Tell If You Or Someone You Love Is A Narcissist appeared first on Dumb Little Man.

via Dumb Little Man – Tips for Life
How To Tell If You Or Someone You Love Is A Narcissist

class PostViewModel extends ViewModel
{
    public $indexUrl = null;

    public function __construct(User $user, Post $post = null)
    {
        $this->user = $user;
        $this->post = $post;

        $this->indexUrl = action([PostsController::class, 'index']); 
    }

    public function post(): Post
    {
        return $this->post ?? new Post();
    }

    public function categories(): Collection
    {
        return Category::canBeUsedBy($this->user)->get();
    }
}

class PostsController
{
    public function create()
    {
        $viewModel = new PostViewModel(
            current_user()
        );

        return view('blog.form', $viewModel);
    }

    public function edit(Post $post)
    {
        $viewModel = new PostViewModel(
            current_user(), 
            $post
        );

        return view('blog.form', $viewModel);
    }
}

<input type="text" value="" />
<input type="text" value="" />

<select>
    @foreach ($categories as $category)
        <option value=""></option>
    @endforeach
</select>

<a href="">Back</a>

class PostsController
{
    public function update(Request $request, Post $post)
    {
        // …

        return (new PostViewModel($post))->view('post.form');
    }
}

class PostViewModel extends ViewModel
{
    public function formatDate(Carbon $date): string
    {
        return $date->format('Y-m-d');
    }
}

via Laravel News
Laravel View Models

For many, many years we’ve been writing about the ridiculousness of the Congressional Research Service’s reports being kept secret. If you don’t know, CRS is a sort of in-house think tank for Congress, that does, careful, thoughtful, non-partisan research on a variety of topics (sometimes tasked by members of Congress, sometimes of its own volition). The reports are usually quite thorough and free of political nonsense. Since the reports are created by the federal government, they are technically in the public domain, but many in Congress (including many who work at CRS itself) have long resisted requests to make those works public. Instead, we were left with relying on members of Congress themselves to occasionally (and selectively) share reports with the public, rather than giving everyone access to the reports.

Every year or so, there were efforts made to make all of that research available to the public, and it kept getting rejected. Two years ago, two members of Congress agreed to share all of the reports they had access to with a private site put together by some activists and think tanks, creating EveryCRSReport.com, which was a useful step forward. At the very least, we’ve now had two years to show that, when these reports are made public, the world does not collapse (many people within CRS feared that making the reports public would lead to more political pressure).

Earlier this year, in the Consolidated Appropriations Act of 2018, there was a nice little line item to officially make CRS reports publicly available.

And, this week, it has come to pass. As announced by Librarian of Congress Carla Hayden, there is now an official site to find CRS reports at crsreports.congress.gov. It appears that the available catalog is still limited, but they’re hoping to expand backwards to add older reports to the system (a few quick test searches only shows fairly recent reports). But all new reports will be added to the database.

The result is a new public website for CRS reports based on the same search functionality that Congress uses – designed to be as user friendly as possible – that allows reports to be found by common keywords. We believe the site will be intuitive for the public to use and will also be easily updated with enhancements made to the congressional site in the future.

Moving forward, all new or updated reports will be added to the website as they are made available to Congress. The Library is also working to make available the back catalog of previously published reports as expeditiously as possible.

This is a big deal. The public pays over $100 million every year to have this research done, and all of it is in the public domain. Starting now, we can actually read most of it, and don’t need to rely on leaks to find this useful, credible research.

Permalink | Comments | Email This Story

via Techdirt
Congressional Research Service Reports Now Officially Publicly Available

by Scott Adams

via Dilbert Daily Strip
Comic for September 17, 2018

Amazon.com is investigating internal leaks as it fights to root out fake reviews and other seller scams from its website, the company told WSJ. From the report:

Employees of Amazon, primarily with the aid of intermediaries, are offering internal data and other confidential information that can give an edge to independent merchants selling their products on the site, according to sellers who have been offered and purchased the data, brokers who provide it and people familiar with internal investigations. The practice, which violates company policy, is particularly pronounced in China, according to some of these people, because the number of sellers there is skyrocketing. As well, Amazon employees in China have relatively small salaries, which may embolden them to take risks. In exchange for payments ranging from roughly $80 to more than $2,000, brokers for Amazon employees in Shenzhen are offering internal sales metrics and reviewers’ email addresses, as well as a service to delete negative reviews and restore banned Amazon accounts, the people said.

Amazon is investigating a number of cases involving employees, including some in the U.S., suspected of accepting these bribes, according to people familiar with the matter. An internal probe began in May after Eric Broussard, Amazon’s vice president who oversees international marketplaces, was tipped off to the practice in China, according to people familiar with the matter. Amazon has since shuffled the roles of key executives in China to try to root out the bribery, one of these people said.

via Slashdot
Amazon Says It is Investigating Claims That Its Employees Are Taking Bribes To Sell Internal Data To Merchants To Help Them Increase Their Sales on the Website

Until very recently, very few people actually liked the enterprise software they were forced to use at work, from the technology managers grumbling about extortionist vendors to the end users grappling with user interfaces designed by engineers. That all started to change once the industry began to realize that software could be provided as a service.

As browsers and internet connections became more reliable, software-as-a-service emerged as the modern way to deliver enterprise applications, and the industry hasn’t looked back. Mobile apps accelerated this shift to an even greater degree, and now hundreds of dedicated apps for the different parts of running a business — from sales and marketing to operations and finance — are available for companies to adopt on a trial basis, rather than locking themselves into three-year contracts with indifferent vendors hawking huge, sprawling software suites that companies still had to manage themselves.

But the inherent bargain with this new world is that the SaaS provider is now the one responsible for all of the technical complexity associated with delivering these applications to the end user. And expectations for performance and uptime go up every year alongside demand for new features and potentially game-changing technologies like machine learning.

Doing this right is hard. That’s why we invited five experts who have worked inside the technology organizations behind some of the most widely used and liked enterprise software applications of the day to speak at our 2018 GeekWire Cloud Tech Summit this past June.

Learn what forward-thinking SaaS operations are doing to keep up with customer demand in the sessions below.

Milena Talavera, Senior Engineering Manager, Slack

Slack’s workplace collaboration software has been widely adopted among the tech set, but it has faced scaling issues as more and more users demand more and more of its product. Talavera explained how Slack developed Flannel, an edge caching system that helps the company deliver some of its main product features to end users without delay.

Mark Nelson, EVP of Product Development, Tableau

The “consumerization of IT” is another way to describe some of the changes that have taken place in the enterprise software market as SaaS has taken over. Nelson walked attendees through the history of this shift, and how engineering teams need to have a different mindset to deliver SaaS products compared to traditional IT products.

Susan Galbraith, Sr. Manager of Software Development, Smartsheet

Push notifications are both an absolute necessity and an easy way to drive your users crazy; it’s easy to run into problems getting the balance right. Galbraith showed attendees how Smartsheet built its notification system, and how it supported the back end of the system while using data to get the front end just right.

Bob Karaban, VP of Engineering, SkyKick

Given the scale at which modern SaaS apps can operate, automating as much of one’s technical infrastructure as possible is a smart idea. Karaban discussed how SkyKick has used automation to improve performance and reliability as user demands grow.

Catherine Renwick, VP Platform Engineering at Workday

Workday’s SaaS human resources products are used startups and big companies alike, and the most personal part of a company’s relationships with its employees demands a certain approach. Renwick revealed how Workday’s technical organization focuses on delivering its service as reliably as possible while absorbing feedback from customer behavior on what it needs to improve.

via GeekWire
SaaS at scale: Here’s how five cloud tech experts are building enterprise software empires

Link

Looking for a fun NERF toy that nobody else on the block has? The Q has you set, with this sweet bow setup that automatically loads the next foam projectile from its magazine. The DIY build requires PVC pipe, rubber bands, plywood, and some miscellaneous other hardware.

via The Awesomer
DIY NERF Bow