Author: mike

When we are hired for a MySQL performance audit, we are expected to review the MySQL configuration and to suggest improvements. Many people are surprised because in most cases, we only suggest to change a few settings even though hundreds of options are available. The goal of this post is to give you a list of some of the most critical settings.We already made such suggestions in the past here on this blog a few years ago, but things have changed a lot in the MySQL world since then!Before we start…Even experienced people can make mistakes that can cause a lot of trouble. So before blindly applying the recommendations of this post, please keep in mind the following items:Change one setting at a time! This is the only way to estimate if a change is beneficial.Most settings can be changed at runtime with SET GLOBAL. It is very handy and it allows you to quickly revert the change if it creates any problem. But in the end, you want the setting to be adjusted permanently in the configuration file.A change in the configuration is not visible even after a MySQL restart? Did you use the correct configuration file? Did you put the setting in the right section? (all settings in this post belong to the [mysqld] section)The server refuses to start after a change: did you use the correct unit? For instance, innodb_buffer_pool_size should be set in MB while max_connection is dimensionless.Do not allow duplicate settings in the configuration file. If you want to keep track of the changes, use version control.Don’t do naive math, like “my new server has 2x RAM, I’ll just make all the values 2x the previous ones”.Basic settingsHere are 3 settings that you should always look at. If you do not, you are very likely to run into problems very quickly.innodb_buffer_pool_size: this is the #1 setting to look at for any installation using InnoDB. The buffer pool is where data and indexes are cached: having it as large as possible will ensure you use memory and not disks for most read operations. Typical values are 5-6GB (8GB RAM), 20-25GB (32GB RAM), 100-120GB (128GB RAM).innodb_log_file_size: this is the size of the redo logs. The redo logs are used to make sure writes are fast and durable and also during crash recovery. Up to MySQL 5.1, it was hard to adjust, as you wanted both large redo logs for good performance and small redo logs for fast crash recovery. Fortunately crash recovery performance has improved a lot since MySQL 5.5 so you can now have good write performance and fast crash recovery. Until MySQL 5.5 the total redo log size was limited to 4GB (the default is to have 2 log files). This has been lifted in MySQL 5.6.Starting with innodb_log_file_size = 512M (giving 1GB of redo logs) should give you plenty of room for writes. If you know your application is write-intensive and you are using MySQL 5.6, you can start with innodb_log_file_size = 4G.max_connections: if you are often facing the ‘Too many connections’ error, max_connections is too low. It is very frequent that because the application does not close connections to the database correctly, you need much more than the default 151 connections. The main drawback of high values for max_connections (like 1000 or more) is that the server will become unresponsive if for any reason it has to run 1000 or more active transactions. Using a connection pool at the application level or a thread pool at the MySQL level can help here.InnoDB settingsInnoDB has been the default storage engine since MySQL 5.5 and it is much more frequently used than any other storage engine. That’s why it should be configured carefully.innodb_file_per_table: this setting will tell InnoDB if it should store data and indexes in the shared tablespace (innodb_file_per_table = OFF) or in a separate .ibd file for each table (innodb_file_per_table= ON). Having a file per table allows you to reclaim space when dropping, truncating or rebuilding a table. It is also needed for some advanced features such as compression. However it does not provide any performance benefit. The main scenario when you do NOT want file per table is when you have a very high number of tables (say 10k+).With MySQL 5.6, the default value is ON so you have nothing to do in most cases. For previous versions, you should set it to ON prior to loading data as it has an effect on newly created tables only.innodb_flush_log_at_trx_commit: the default setting of 1 means that InnoDB is fully ACID compliant. It is the best value when your primary concern is data safety, for instance on a master. However it can have a significant overhead on systems with slow disks because of the extra fsyncs that are needed to flush each change to the redo logs. Setting it to 2 is a bit less reliable because committed transactions will be flushed to the redo logs only once a second, but that can be acceptable on some situations for a master and that is definitely a good value for a replica. 0 is even faster but you are more likely to lose some data in case of a crash: it is only a good value for a replica.innodb_flush_method: this setting controls how data and logs are flushed to disk. Popular values are O_DIRECT when you have a hardware RAID controller with a battery-protected write-back cache and fdatasync (default value) for most other scenarios. sysbench is a good tool to help you choose between the 2 values.innodb_log_buffer_size: this is the size of the buffer for transactions that have not been committed yet. The default value (1MB) is usually fine but as soon as you have transactions with large blob/text fields, the buffer can fill up very quickly and trigger extra I/O load. Look at the Innodb_log_waits status variable and if it is not 0, increase innodb_log_buffer_size.Other settingsquery_cache_size: the query cache is a well known bottleneck that can be seen even when concurrency is moderate. The best option is to disable it from day 1 by setting query_cache_size = 0 (now the default on MySQL 5.6) and to use other ways to speed up read queries: good indexing, adding replicas to spread the read load or using an external cache (memcache or redis for instance). If you have already built your application with the query cache enabled and if you have never noticed any problem, the query cache may be beneficial for you. So you should be cautious if you decide to disable it.log_bin: enabling binary logging is mandatory if you want the server to act as a replication master. If so, don’t forget to also set server_id to a unique value. It is also useful for a single server when you want to be able to do point-in-time recovery: restore your latest backup and apply the binary logs. Once created, binary log files are kept forever. So if you do not want to run out of disk space, you should either purge old files with PURGE BINARY LOGS or set expire_logs_days to specify after how many days the logs will be automatically purged.Binary logging however is not free, so if you do not need for instance on a replica that is not a master, it is recommended to keep it disabled.skip_name_resolve: when a client connects, the server will perform hostname resolution, and when DNS is slow, establishing the connection will become slow as well. It is therefore recommended to start the server with skip-name-resolve to disable all DNS lookups. The only limitation is that the GRANT statements must then use IP addresses only, so be careful when adding this setting to an existing system.ConclusionThere are of course other settings that can make a difference depending on your workload or your hardware: low memory and fast disks, high concurrency, write-intensive workloads for instance are cases when you will need specific tuning. However the goal here is to allow you to quickly get a sane MySQL configuration without spending too much time on changing non-essential MySQL settings or on reading documentation to understand which settings do matter to you.The post 10 MySQL settings to tune after installation appeared first on MySQL Performance Blog.
via Planet MySQL
10 MySQL settings to tune after installation

via Gizmodo
Watch Steve Jobs Show Off the Mac in Footage Unseen Since 1984

VentureBeat is one of the many outlets featuring recently surfaced video of Steve Jobs doing an early demo of the Macintosh, 30 years ago. I remember first seeing one of these Macs in 1984 at a tiny computer store in bustling downtown Westminster, Maryland, and mostly hogging it while other customers (or, I should say, actual customers) tapped their feet impatiently.

Read more of this story at Slashdot.

via Slashdot
Watch Steve Jobs Demo the Mac, In 1984

Now that Canon has two semi-professional full-frame cameras, the EOS 5D Mark III and EOS 6D, naturally photographers wonder which is the best model for them. As a former Technical Editor of EOS Magazine (the best magazine Canon owners could possible buy) I would like to share my insights. But first, if you’re in the […]

The post Review: Comparison Canon 5D MarkIII vs the Canon 6D by Andrew S. Gibson appeared first on Digital Photography School.

via Digital Photography School
Review: Comparison Canon 5D MarkIII vs the Canon 6D

When it’s a super cold day or you have aching joints, nothing quite beats the soothing heat a microwave heat bag can deliver. With some basic sewing skills, you can make your own right at home.

Lizziecharlton over at Instructables has put together a step-by-step guide to creating your own DIY microwave heat bag. All you need some cotton material, thread, and some filing for the bag. Fillings can include rice, wheat, feed corn, beans, or several other items. Optionally, you can add some essential oils if you like your bag to have a scent when heated. That, plus being able to choose a fabric that suits your tastes, is what elevates these bags over their store-bought counterparts.

Hit up the full post at Instructables to see how to make one for yourself.

Making A Microwave Heat Bag | Instructables

via Lifehacker
Make a DIY Microwave Heat Bag

Dear Lifehacker,
I like to keep up on current events, but I simply don’t have the time to read every news story every day. Do you have any suggestions for keeping up on the news when I don’t have a lot of time?

Sincerely,
Just the News

Dear JN,
It’s true, keeping up to date on current events is basically a full time job. RSS readers are great for people who have the time to go through them, but they’re not that good for just getting a summary of world news. Thankfully, you actually have a few great options for keeping up on the news without spending a lot of time.

Find a Daily Summary You Like

If you’re a fan of watching or listening to the news, most news outlets have an hourly summary they run throughout the day that includes all of the most important news jammed into a quick, five to 10 minute slot. This is a great way to easily catch up on everything you miss.

You have a few different choices for news outlets depending on the type of news you prefer. Here are a few bulletin services that offer always up-to-date news summaries:

• NPR’s Hourly Newscast runs through the world news in about five minutes.
• BBC’s hourly bulletin runs through the daily headlines from around the world.
• ABC’s News Pop features daily video with news updates and ABC News Radio does the same.
• Fox’s 5 Minute Newscast runs through the headlines each day.

Of course, the above picks aren’t the only options, but they do offer simple, always updating news summaries so you can keep yourself up to date without spending time digging into it. Mac users might want to check out Hourly News, an app that stuffs the above hourly newscasts into your menu bar. Most news services also have podcasts you can subscribe to and integrate into your daily playlists.

Use Wikipedia’s Current Events Portal

While Wikipedia isn’t exactly the most accurate source of news information, it is a surprisingly good way to quickly get a synopsis of what’s going on in the world.

You have two different ways to do this. As we’ve mentioned before, Wikipedia’s Current Events page is an incredibly simple way to drop in, see what’s going on in the news, and read more if you’re interested. If you need to just get caught up on what you’ve missed while you were on vacation or otherwise off the grid, type the month and year into Wikipedia’s search and you’ll get a synopsis of all the important news that happened that month.

Use a Service that Sums Up the News for You

If you’re looking for just a quick and easy to read summary of the news in digestible little bits, your smartphone can help you out quite a bit.

We like Circa as a service that condenses all the important news from a ton of different sources, and then summarizes the main points for you. Circa does a good job of keeping things unbiased with a variety of sources so you get a pretty good overall view of what’s happening in the world in a short amount of time. The popular digest-style apps like Flipboard are also good for this as well, although they’ll cater to you specifically so you might miss some of the broader news out there.

If an app isn’t your thing, Skim That does the same thing by just sending you a daily email with news summaries.

Curate Your Social Media Feeds

Two very simple places where you probably already get the news can be made a little better. Both Twitter and Facebook are great sources for popular news articles, but they need a little work to make them usable.

Twitter’s great as a replacement to RSS feeds if you use lists. Tested’s Will Smith uses Twitter lists as a means to get curated news and it works really well. Just add a handful of news outlets to a list, and periodically check in on that list to see what’s going on. Facebook doesn’t have quite the same filtering ability as Twitter lists, but when you like a news station, you’re usually shown the biggest, best, and most important news stories of the day.

Information overload is usually bad enough as it is, but it’s possible to keep up on the news without overwhelming yourself.

Good luck,
Lifehacker

Photos by Charles Taylor and Morrison77.

via Lifehacker
How Can I Keep Up on News When I Don’t Have a Lot of Time?

MySQL is the little engine that could. It powers sites like Facebook, YouTube, Twitter, and thousands of blogs, CMSes, and e-commerce sites. Its value to the world and to the development community could be measured in the hundreds of billions, and yet it’s free, and you can use it just by downloading it. Almost every programming language has drivers for it and it can run on so many operating systems and architectures, there’s really no limit on it.
Yet there’s a dark side. MySQL is full of gotchas and bugs, and it lacks features that sometimes call into question its status as a real database. The documentation is often open-ended and confusing, with gaps in key parts. If you want to run it, you have the option of using it on Linux, Mac, Solaris, or Windows and every hosting company or provider like Amazon AWS has their own managed service, each with its own quirks and limitations. The user community has also produced thousands of projects, with varying degrees of completeness and support.
A simple question like “how do I back my system up?” has at least 5 good answers, each with their own advocates, tutorials, and scripts. In short, the MySQL world is a crazy bazaar that is at times overwhelming.
I want to cut through confusion
I’m writing a book to help take away the confusion and provide an opinionated guide that cuts through the confusion and provides you knowledge you need to get back to your other job(s). Your boss, your team, and your customers don’t want to hear “I’m researching 5 possible solutions.” They want to hear, “I have a great solution and I’ve implemented it.” This book is your fast-track through the most common issues I’ve encountered in MySQL.
Every time I read about team giving up on MySQL because MongoDB is easier, I cry a little bit. MySQL is so simple and easy, it’s been tested and debugged over decades, and there’s so much community support from utilities, scripts, blogs, and tutorials. I want to convince you that with a little knowledge, MySQL is easier than Mongo, and it will help you and your company make money.
Another trouble is with proprietary databases. Many people are under the impression that Oracle or SQL Server have the corner on performance, that they are not bounded by the laws of computer science. While Oracle and SQL Server have pretty awesome technology that lets them sort out horrific queries, with good schema design and well-written queries, MySQL can churn out tens of thousands of requests a second (and some benchmarks put that up to hundreds of thousands), which for most workloads is plenty.
My background is that I’m a senior DBA who’s worked on MySQL with small teams. I came from a Microsoft background and decided I wanted to go over to Linux/MySQL side of the house. I’m now writing the book I wish I had when I first started. I want to empower MySQL users who like me are under pressure but who want to take those steps that will pay off in the end, like good schema design and a solid backup and recovery plan.
I started writing when I saw a pattern of confusion and lack of knowledge among my coworkers or people on StackOverflow, Reddit, and HackerNews. I’ve helped my team understand and use MySQL, and I think I can help you too.
Throughout my career, I’ve made mistakes that I want to give you a chance to avoid. I’ll also guide you through those mistakes that you can recover from (writing a badly performing query) to those that might be a career limiting move (forgetting to monitor a production server).
Audience
The first audience for this book is those who’ve used MySQL but want to get a solid foundation. There are some things that you must do if you’re developing on MySQL or if you’re in charge of a MySQL server, and I want to give you a simple guide to the essentials, as well as pointers to further research if you want to go deeper.
Another value I want to add is pep talks throughout urging you not to give into short-term laziness when it buys you long-term trouble. I won’t nag you, but I want to give you motivation to do those things that don’t have an immediate payout but in the long run are well worth it.
This guide was also meant for entrepreneurs who are focused on building a product. I created a blueprint for using MySQL effectively for you to read quickly, apply what you’ve learned, and get back to work. I want to save you time and sleepless nights.
The benefit
Why should you get this book? Two main reasons: it will reduce the risks of using MySQL while maximizing its power. Effective backups, scalable queries, and well-defined schema save you costs down the road. The end result is that you play less for your database (yes, even though MySQL is free, it still has a price). On the other hand, it will increase your revenue. I’m basing this off of a simple principle that the faster you can release a feature, the faster you can sell it to customers, and the more revenue you get. This book will show you those things you can do to complete your features faster. What’s faster than fast? How about no time at all. I want to show you the tools that are already complete, ready for your use.
I’m not going to claim I know everything about MySQL and this guide will be the end all, be all of MySQL books. I’m still learning, but I wanted to provide a way for others to benefit from what I’ve already learned. If you’ve read this far, sign-up below and get updates on the book.
Want to hear more about my book and what I’ve learned about MySQL? You can subscribe below and I’ll send you updates. I’ll also send out stuff on MySQL like tips and tutorials.
Email
Your Name
 No spam
via Planet MySQL
MySQL Simplified

I’ve written a short course on MySQL backups. It’s really a MySQL backup starter kit, giving you the basics of what you need to make sure your system is protected. The real gem in this course is XtraBackup, which is a tool that allows you to backup your MySQL server without blocking other transactions. You see, MySQL doesn’t have that out of the box. You have to spend a bunch of money with Oracle to get that otherwise. Hot backups are pretty necessary if you care about uptime, so I’m excited about XtraBackup.
Here are the topics I cover:
Simple backups
Even though XtraBackup is a great tool, no MySQL user’s knowledge would be complete without knowing how to take a mysqldump. Simple backups are just useful for moving data from server to server, or perhaps migrating your data to, say, PostgreSQL.
Disaster Recovery Plans
Okay, so this isn’t the sexiest topic, but a plan for what to do during a disaster is a great investment. Trust me, you will have a disaster, even if you’re a small shop (Go here, Ctrl+f for MySQL). You need a plan and you need to try it out at least once. Give me twenty minutes and I’ll save you heartache.
XtraBackup
This is the big gun. Learn how to use the most powerful MySQL backup tool so you can have a site without any downtime. If you don’t know about this tool, you’re missing out. This is a lifesaver for MySQL users.
Testing backups
Although it’s almost as boring as a recovery plan, you must test your backups. Get tips on how to do that with speed and with confidence. I know that if something’s hard or it’s confusing, people will just avoid doing it. Let me clear that up and get you going.
Sign-up below!
Want to see how Facebook, Imgur, and Pinterest do their backups? Take my 4-week course on MySQL backups. You’ll learn backup basics and how to protect your system from the most dangerous threats. Then you’ll learn how the big boys do their backups.
In just a few lessons, you’ll know to backup your system like a pro.
Email
Your Name
 No spam
via Planet MySQL
A Course on MySQL Backups

As expected, the Privacy and Civil Liberties Oversight Board (PCLOB) has come out with it’s quite scathing report concerning the federal government’s interpretation of Section 215 of the PATRIOT Act. The full report is quite readable and well worth reading, no matter how familiar you are with the program. If you’re not familiar, it lays out all the details. If you are familiar, it still may fill in a number of useful gaps as well. While the full recommendations and conclusions were not supported unanimously by the board, the majority did agree that not only is the program unconstitutional, but that it involves a gross misinterpretation of the law. The executive summary makes the point pretty clearly:

Section 215 is designed to enable the FBI to acquire records that a business has in its possession, as part of an FBI investigation, when those records are relevant to the investigation. Yet the operation of the NSA’s bulk telephone records program bears almost no resemblance to that description. While the Board believes that this program has been conducted in good faith to vigorously pursue the government’s counterterrorism mission and appreciates the government’s efforts to bring the program under the oversight of the FISA court, the Board concludes that Section 215 does not provide an adequate legal basis to support the program.

There are four grounds upon which we find that the telephone records program fails to comply with Section 215. First, the telephone records acquired under the program have no connection to any specific FBI investigation at the time of their collection. Second, because the records are collected in bulk — potentially encompassing all telephone calling records across the nation — they cannot be regarded as “relevant” to any FBI investigation as required by the statute without redefining the word relevant in a manner that is circular, unlimited in scope, and out of step with the case law from analogous legal contexts involving the production of records. Third, the program operates by putting telephone companies under an obligation to furnish new calling records on a daily basis as they are generated (instead of turning over records already in their possession) — an approach lacking foundation in the statute and one that is inconsistent with FISA as a whole. Fourth, the statute permits only the FBI to obtain items for use in its investigations; it does not authorize the NSA to collect anything.

In addition, we conclude that the program violates the Electronic Communications Privacy Act. That statute prohibits telephone companies from sharing customer records with the government except in response to specific enumerated circumstances, which do not include Section 215 orders.

Finally, we do not agree that the program can be considered statutorily authorized because Congress twice delayed the expiration of Section 215 during the operation of the program without amending the statute. The “reenactment doctrine,” under which Congress is presumed to have adopted settled administrative or judicial interpretations of a statute, does not trump the plain meaning of a law, and cannot save an administrative or judicial interpretation that contradicts the statute itself. Moreover, the circumstances presented here differ in pivotal ways from any in which the reenactment doctrine has ever been applied, and applying the doctrine would undermine the public’s ability to know what the law is and hold their elected representatives accountable for their legislative choices.

Basically, in those four short paragraphs, the PCLOB dismantles nearly all of the arguments that people have put forth to support the bulk collection of metadata, and make it clear that the DOJ, NSA and FISC are clearly twisting the plain language meaning of Section 215 to support what is ultimately an unconstitutional program.

On that front, the report notes clearly the Constitutional issues:

The NSA’s telephone records program also raises concerns under both the First and Fourth Amendments to the United States Constitution. We explore these concerns and explain that while government officials are entitled to rely on existing Supreme Court doctrine in formulating policy, the existing doctrine does not fully answer whether the Section 215 telephone records program is constitutionally sound. In particular, the scope and duration of the program are beyond anything ever before confronted by the courts, and as a result of technological developments, the government possesses capabilities to collect, store, and analyze data not available when existing Supreme Court doctrine was developed. Without seeking to predict the direction of changes in Supreme Court doctrine, the Board urges as a policy matter that the government consider how to preserve underlying constitutional guarantees in the face of modern communications technology and surveillance capabilities.

While the PCLOB repeatedly states it believes that the government acted in good faith, it nonetheless finds the program immensely troubling. The idea that collecting all phone metadata is okay simply is not supported by what the law itself actually says:

Notably, Section 215 requires that records sought be relevant to “an” authorized investigation. Elsewhere, the statute similarly describes the records that can be obtained under its auspices as those sought “for an investigation.” The use of the singular noun in these passages signals an expectation that the records are being sought for use in a specific, identified investigation. This interpretation is reinforced by the requirement that the FISA court make specific findings about the investigation for which the records are sought — that it is supported by a factual predicate, conducted according to guidelines approved by the Attorney General, and not based solely upon activities protected by the First Amendment when conducted of a U.S. person.

[….] The government’s approach, in short, has been to declare that the calling records being sought are relevant to all of the investigations cited in its applications. This approach, at minimum, is in deep tension with the statutory requirement that items obtained through a Section 215 order be sought for “an investigation,” not for the purpose of enhancing the government’s counterterrorism capabilities generally. Declaring that the calling records are relevant to every counterterrorism investigation cited by the government is little different, in practical terms, from simply declaring that they are relevant to counterterrorism in general.

That is particularly so when the number of calling records sought is not limited by reference to the facts of any specific investigation. At its core, the approach boils down to the proposition that essentially all telephone records are relevant to essentially all international terrorism investigations. The Board does not believe that this approach comports with a fair reading of the statute.

Moreover, this approach undermines the value of an important statutory limitation on the government’s collection of records under Section 215. The statute provides that records cannot be obtained for a “threat assessment,” meaning those FBI investigatory activities that “do not require a particular factual predicate.”201 By excluding threat assessments from the types of investigations that can justify an order, Congress directed that Section 215 not be used to facilitate the broad and comparatively untethered investigatory probing that is characteristic of such assessments. But by collecting the nation’s calling records en masse, under an expansive theory of their relevance to multiple investigations, the NSA’s program undercuts one of the functions of the “threat assessment” exclusion: ensuring that records are not acquired by the government without some reason to suspect a connection between those records and a specific, predicated terrorism investigation. While the rules governing the program limit the use of telephone records to searches that are prompted by a specific investigation, the relevance requirement in Section 215 restricts the acquisition of records by the government.

The PCLOB clearly sees through the feds’ ridiculous re-interpretation of the word "relevant" as well — calling it "untenable" and "dangerously overbroad."

The government has argued, and the FISA court has agreed, that essentially the entire nation’s calling records are “relevant” to every counterterrorism investigation cited in the government’s applications to the court. This position is untenable. Moreover, the interpretation of Section 215 adopted by the FISA court is dangerously overbroad, leading to the implication that virtually all information may be relevant to counterterrorism and therefore subject to collection by the government.

Later, the report argues that the government’s interpretation "is circular and deprives the word ‘relevant’ of any interpretive value."

All records become relevant to an investigation, under this reasoning, because the government has developed an investigative tool that functions by collecting all records to enable later searching. The implication of this reasoning is that if the government develops an effective means of searching through everything in order to find something, then everything becomes relevant to its investigations. The word “relevant” becomes limited only by the government’s technological capacity to ingest information and sift through it efficiently.

The PCLOB also totally debunks the line trotted out by numerous NSA defenders that this program is no different than a grand jury subpoena. Not so, says the board:

To determine what might be the outer limits of a grand jury subpoena, we have examined both the cases cited by the government and others. There has never been a grand jury subpoena as broad as the FISA court’s Section 215 orders. And contrary to the government’s suggestion, the case law does not hold that the breadth of a grand jury subpoena is unlimited, but rather that a subpoena must be designed to address the circumstances of a specific investigation.

One decision, In re Grand Jury Proceedings, merely explains that district courts assessing the relevance of subpoenaed materials should not proceed “document-by-document,” but should instead evaluate whether each “broad category” of requested materials could contain possibly relevant documents. The former approach would “unduly disrupt the grand jury’s broad investigatory powers” and force the government “to justify the relevancy of hundreds or thousands (or more) of individual documents, which it has not yet even seen[.]” Often the government “is not in a position to establish the relevancy with respect to specific documents,” because “it may not know the precise content of the requested documents” and “it may not know precisely what information is or is not relevant at the grand jury investigative stage.” Accepting the “incidental” production of irrelevant documents, when measured by the hundreds or thousands, does not support the legitimacy of the Section 215 calling records program, in which the NSA potentially collects billions of records per day with full knowledge that virtually all of them are irrelevant.

It goes on to point to a number of other cases and how the government’s interpretation of them is simply bogus.

It also points out that screaming "but… but… terrorism!" is no excuse either:

Finally, the heightened importance of counterterrorism investigations, as compared with typical law enforcement matters, does not alter the equation. Items either are relevant to an investigation or they are not — the significance of that investigation is a separate matter. No matter how critical national security investigations are, therefore, some articulable principle must connect the items sought to those investigations, or else the word “relevant” is robbed of meaning. Congress added a relevance requirement to Section 215 in 2006 knowing full well that the statute governs national security investigations. It cannot, therefore, have meant for the importance of such investigations to efface that requirement entirely.

There’s also an interesting tidbit, noting that Section 215 was designed specifically and solely for the FBI, not the NSA — yet it is used here by the NSA (who then may share the info with lots of other agencies):

Section 215 expressly allows only the FBI to acquire records and other tangible things that are relevant to its foreign intelligence and counterterrorism investigations. Its text makes unmistakably clear the connection between this limitation and the overall design of the statute. Applications to the FISA court must be made by the director of the FBI or a subordinate. The records sought must be relevant to an authorized FBI investigation. Records produced in response to an order are to be “made available to,” “obtained” by, and “received by” the FBI.

[….] Under the bulk telephone records program, however, the FBI does not receive any records in response to the FISA court’s orders. While FBI officials sign every application seeking to renew the program, the calling records produced in response to the court’s orders are never “made available to the Federal Bureau of Investigation” or “received by the Federal Bureau of Investigation,” as called for by the statute.335 Instead, the FISA court’s orders specifically direct telephone companies to “produce to NSA” their calling records — thwarting congressional intentions regarding the role each agency is to play in counterterrorism efforts that involve the collection of information within the United States about Americans.

In compliance with the FISA court’s orders, telephone companies that are subject to this program transmit their calling records to the NSA. The records are not delivered to the FBI and are never passed on to the FBI by the NSA. Instead, the NSA stores the records in its own databases, conducts its own analysis of them, and provides reports to various federal agencies — including but not limited to the FBI — with information about telephone communications that “the NSA concludes have counterterrorism value.”

In fact, the PCLOB notes, the FISC orders on this program actually prohibit the NSA from giving much of the information to the FBI, despite the fact that the law is only designed to be used by the FBI.

There’s another section detailing how the FISA Court more or less ignores ECPA (the Electronic Communications Privacy Act) which the bulk metadata collection program clearly violates. The report notes that the FISC more or less admits this, and then says that Congress couldn’t really have meant to say what the ECPA law says.

The FISA court concluded that its orders authorizing the NSA’s program were consistent with ECPA. In reaching this conclusion, the court first determined that the terms of Section 215 and ECPA were in tension. Both statutes could not both be given “their full, literal effect,” wrote the court, because Section 215 authorizes the production of “any tangible things,” and applying the prohibitions of ECPA would limit the meaning of the word “any.”

Instead, the PCLOB gives a fairly compelling argument for why the FISC is just wrong on this:

As the FISA court acknowledged, the very statute that created Section 215, the Patriot Act, also amended ECPA “in ways that seemingly re-affirmed that communications service providers could divulge records to the government only in specified circumstances” — without including FISA court orders issued under Section 215. The fact that the same statute both created Section 215 and amended ECPA, but without adding an exception to ECPA for Section 215 orders, undermines the notion that ECPA and Section 215 are in conflict, and provides an additional basis for strictly adhering to ECPA’s prohibitions by not inferring unwritten exceptions to those prohibitions. It also demonstrates that another fundamental canon of statutory construction applies here — that the inclusion of some implies the exclusion of others not mentioned. “Where there is an express exception, it comprises the only limitation on the operation of the statute and no other exceptions will be implied.” Congress did not add an exception to ECPA for Section 215 orders, even though it amended ECPA in other ways at the same time that it created Section 215. That omission should be respected.

As for the claim that because Congress re-enacted Section 215, it clearly approves of the bizarre reinterpretation of it by the FISC and the executive branch, the PCLOB rejects this, claiming it, too, is a "novel proposition" reinterpreting the "reenactment doctrine" beyond its intended purpose. And part of that, of course, is the fact that the FISA and NSA/DOJ interpretations were all kept really secret from Congress:

The “reenactment doctrine” does not trump the plain meaning of a law, but rather is one of many interpretive tools that come into play when statutory ambiguity demands an inquiry into congressional intent. Reenactment, in other words, “cannot save” an administrative or judicial interpretation that contradicts the requirements of the statute itself. And for the many reasons explained above, any interpretation of Section 215 that would authorize the NSA’s telephone records program is irreconcilable with the plain words of the statute, its manifest purpose, and its role within FISA as a whole.

Even if Section 215 were sufficiently ambiguous to justify an inquiry into congressional intent, the circumstances presented here are unlike any in which the reenactment doctrine has ever been applied — and the differences are pivotal. First, there was no judicial interpretation of Section 215 of which Congress could have been aware in 2010 or 2011: at that time the FISA court had never issued any opinion explaining the legal rationale for the NSA’s program under Section 215, but had merely signed orders authorizing the program. Second, even if the FISA court’s orders, combined with the government’s applications to the court, are viewed as an “interpretation” of Section 215, members of Congress may have been prohibited from reading those orders and those applications (except for members of the intelligence and judiciary committees) by operation of committee rules. Thus, to apply the reenactment doctrine here, Senators and Congressmen must be presumed to have adopted an “interpretation” that they had no ability to read for themselves. Third, even if being apprised of the NSA’s program were equivalent to being made aware of a judicial interpretation of a statute, applying the reenactment doctrine is improper where members of Congress must try to comprehend a secret legal interpretation without the aid of their staffs or outside experts and advocates. That scenario robs lawmakers of a meaningful opportunity to gauge the legitimacy and implications of the legal interpretation in question. Fourth, Congress did not reenact Section 215 at all in 2010 and 2011, but merely delayed its expiration. To our knowledge, no court has applied the reenactment doctrine under a combination of circumstances remotely like this.

Oh, and then there’s this:

Finally, even if Section 215 were ambiguous about whether it authorizes the NSA’s bulk collection of telephone records, and even if the reenactment doctrine could be extended to the novel circumstances presented here, doing so would undermine the ability of the American public to know what the law is, and to hold their elected representatives accountable for their legislative choices. Applying the reenactment doctrine to legitimize the government’s interpretation of Section 215, therefore, is both unsupported by legal precedent and unacceptable as a matter of democratic accountability.

In other words, no, you can’t have secret laws and secret interpretations.

Moving on to the constitutional questions, the PCLOB takes a look at the 4th Amendment and the third party doctrine. Unlike many knee jerk NSA defenders, the PCLOB notes that there are significant problems with applying the infamous Smith v. Maryland ruling to the bulk metadata collection program:

[Smith v. Maryland] does not provide a good fit for the telephone records program, particularly in light of rapid technological changes and in light of the nationwide, ongoing nature of the program. The NSA’s Section 215 program gathers significantly more information about each telephone call and about far more people than did the pen register surveillance approved in Smith (essentially everyone in the country who uses a phone) and it has collected that data now for nearly eight years without interruption.441 In contrast, the pen register approved in Smith v. Maryland compiled only a list of the numbers dialed from Michael Lee Smith’s telephone. It did not show whether any of his attempted calls were actually completed — thus it did not reveal whether he engaged in any telephone conversations at all. Naturally, therefore, the device also did not indicate the duration of any conversations. Furthermore, the pen register provided no information about incoming telephone calls placed to Smith’s home, only the outbound calls dialed from his telephone.

The pen register was in operation for no more than two days. And finally, the device recorded only the dialing information of one person: Smith himself. The police had no computerized ability to aggregate Smith’s dialing records with those of other individuals and gain additional insight from that analysis.

In contrast, for each of the millions of telephone numbers covered by the NSA’s Section 215 program, the agency obtains a record of all incoming and outgoing calls, the duration of those calls, and the precise time of day when they occurred. When the agency targets a telephone number for analysis, the same information for every telephone number with which the original number has had contact, and every telephone number in contact with any of those numbers. And, subject to regular program renewal by the FISA court, it collects these records every day, without interruption, and retains them for a five year time period. Sweeping up this vast swath of information, the government has explained, allows the NSA to use “sophisticated analytic tools” to “discover connections between individuals” and reveal “chains of communication” — a broader power than simply learning the telephone numbers dialed by a single targeted individual.

To illustrate the greater scope of the NSA’s program, the pen register discussed in Smith might have shown that, during the time that Michael Lee Smith’s telephone was monitored, he dialed another number three times in a single day. That information could have simply evinced three failed attempts to reach the other number. The NSA’s collection program, however, would show not only whether each attempted call connected but also the precise duration and time of each call. It also would reveal whether and when the other telephone number called Smith and the length and time of any such calls. Because the NSA collects records continuously and stores them for five years, it would be in a position to see how frequently those two numbers contacted each other during the preceding five years and the pattern of their contact. And because the agency would have full access to the calling records of the other telephone number as well, it could examine the activity of that other number and see, for instance, whether it ever communicated with any of the same numbers as Smith over a five-year period, or what numbers it communicated with around the time of its calls with Smith. The agency could then do the same thing for every other number that Smith had communicated with in the past five years, employing what it calls contact-chaining analysis. It could then go further and analyze the complete calling records of every number that was called by any of the numbers that ever communicated with Smith — going three “hops” from the original number.

But, that’s not all. The report (like many others) slams the Supreme Court’s reasoning in Smith, quotes "the leading academic treatise" and even third party doctrine supporter Orin Kerr, highlighting how almost no serious scholar thinks the Supreme Court’s reasoning in Smith v. Maryland makes much sense. It quotes numerous other Supreme Court justices and other courts who find the majority ruling in Smith to be profoundly nonsense, and a dangerous attack on the 4th Amendment. And then points out why all of those critics were right:

The implications of this all-or-nothing approach to privacy have grown since the 1970s, as Americans increasingly must share personal information with companies in order to avail themselves of services and products that have become typical features of modern living. Another major criticism of the third-party doctrine, which has gained increased salience in light of these developments, challenges the notion that a customer of such companies, simply by “revealing his affairs to another,” truly chooses to risk “that the information will be conveyed by that person to the Government.” This criticism rejects the idea that conducting business that is essential to contemporary life represents a voluntary decision to lay bare the details of one’s habits to governmental scrutiny.

“For all practical purposes,” Justice Brennan observed in his Miller dissent, “the disclosure by individuals or business firms of their financial affairs to a bank is not entirely volitional, since it is impossible to participate in the economic life of contemporary society without maintaining a bank account.”

Moving on to the First Amendment, the PCLOB also notes serious questions about whether or not the bulk metadata collection violates the prohibition on Congress not to pass laws that infringe on free speech and free association. Citing the NAACP v. Alabama case that clearly stated that having a government reveal groups and associations would violate the First Amendment, the PCLOB takes issue with the collection of so much metadata, that clearly reveals who people associate with:

Although the NSA’s telephone records program does not include an overt disclosure requirement of the type evaluated in such cases as NAACP v. Alabama, its operation similarly results in the compulsory disclosure of information about individuals’ associations to the government. Like the government’s collection of membership lists, its bulk collection of telephone records makes that information available for government analysis and can create a chilling effect on those whose records are being collected.

[….] By indefinitely collecting information about all Americans’ telephone calls, the NSA’s telephone records program clearly implicates the First Amendment freedoms of speech and association. The connections revealed by the extensive database of telephone records gathered under the program will necessarily include relationships established among individuals and groups for political, religious, and other expressive purposes. Compelled disclosure to the government of information revealing these associations can have a chilling effect on the exercise of First Amendment rights.

There’s much more in the report worth reading, but those are many of the highlights. Honestly, much of it could be turned into the legal briefs that could eventually be used in court against the program.

Next up, the PCLOB will be releasing a report looking at Section 702 of the FISA Amendments Act and programs like PRISM that fit under it. I imagine that will be equally interesting.

Permalink | Comments | Email This Story

via Techdirt.
Civil Liberties Board Completely Destroys Arguments For Bulk Metadata Collection: Program Is Both Illegal And Unconstitutional