LDAP with auth_pam and PHP to authenticate against MySQL

In the quest to secure MySQL as well as ease the number of complicated passwords to remember, many organizations are looking into external authentication, especially using LDAP. For free and open source, Percona’s PAM authentication plugin is the standard option.

tl;dr is I go through how to compile php-cli for use with auth_pam plugin.

Background

There are two plugins that can be used. From the documentation, the two plugins are:

  • Full PAM plugin called auth_pam. This plugin uses dialog.so. It fully supports the PAM protocol with arbitrary communication between client and server.
  • Oracle-compatible PAM called auth_pam_compat. This plugin uses mysql_clear_password which is a part of Oracle MySQL client. It also has some limitations, such as, it supports only one password input. You must use -p option in order to pass the password to auth_pam_compat.

Percona’s MySQL client supports both plugins natively. That is, you can use auth_pam or auth_pam_compat and use the “mysql” tool (or “mysqldump”, or mysql_upgrade, etc.) and you are good to go. Given the choice, we would all use auth_pam, under which clients DO NOT use mysql_clear_password.

Not all clients support auth_pam, which is the main problem. Workarounds have called for using auth_pam_compat over SSL, which is a perfectly reasonable way to handle the risk of cleartext passwords – encrypt the connection.

However, what if you want to use auth_pam?

The problem with auth_pam

Back in 2013, Percona posted about how to install and configure auth_pam and auth_pam_compat. I will not rehash that setup, except to say that most organizations no longer use /etc/shadow, so the setup involves getting the correct /etc/pam.d/mysqld in place on the server.

That article has this gem:

As of now, only Percona Server’s mysql client and an older version of HeidiSQL(version 7), a GUI MySQL client for Windows, are able to authenticate over PAM via the auth_pam plugin by default.

So, if you try to connect to MySQL using Perl, PHP, Ruby, Python and the like, you will receive this error: “Client does not support authentication protocol requested by server; consider upgrading MySQL client.”

Fast forward 4 years, to now, and this is still an issue. Happily, the article goes on to explain how to recompile clients to get them to work:

The good news is that if the client uses libmysqlclient library to connect via MySQL, you can recompile the client’s source code to use the libmysqlclient library of Percona Server to make it compatible. This involves installing Percona Server development library, compiler tools, and development libraries followed by compiling and installing the client’s source code.

And, it helpfully goes step by step on how to recompile perl-DBD-mysql to get it working with LDAP authentication (as well as without – it still works for users who do not use LDAP).

But what if you are using PHP to connect to MySQL?

PHP and auth_pam

If you try to connect, you get this error:
SQLSTATE[HY000] [2054] The server requested authentication method unknown to the client

So let us try to mirror the perl recompilation process in PHP.

Step 1

“Install Percona yum repository and Percona Server development library.” This is not a problem, do what you need to do to install Percona-Server-devel for your version.

Step 2

Install a package manager so you can build a package – optional, but useful, if you ever want to have this new client without having to recompile. As in the example, I chose the RPM package manager, so I installed rpm-build.

Step 3

Download and install the source RPM for the client package. This is where I started running into trouble. What I did not realize was that PHP does not divide out its packages like Perl does. Well, it does, but php-mysqlnd is compiled as part of the core, even though it is a separate package.

Downloading the main PHP package

So I downloaded the source RPM for PHP at http://ift.tt/2i2s1cC, and installed it into the sources directory:
cd SRPMS
wget http://ift.tt/2fHR7Nh
cd ../SOURCES
rpm -Uvh ../SRPMS/php-7.0.22-2.remi.src.rpm

This unpacks a main file, php-7.0.22.tar.xz, plus a bunch of supplemental files (like patches, etc).

What it does NOT contain is a spec file, which is critical for building the packages.

Getting a spec file

I searched around and found one at http://ift.tt/2i2s20a – this is for 7.0.21, so beware of using different versions of spec files and source code. Once that was done, I changed the mysql lines to /usr/bin/mysql_config as per Choosing a MySQL library. Note that I went with the “not recommended” library, but in this case, we WANT to compile with libmysqlclient.

Compiling php-cli, not php-mysqlnd

In addition, I discovered that compiling php-mysqlnd with the new libraries did not work. Perhaps it was something I did wrong, as at that point I was still compiling the whole PHP package and every module in it.

However, what I *did* discover is that if I recompiled the php-cli package with libmysqlclient, I was able to get a connection via PHP using LDAP authentication, via a tool written by someone else – with no changes to the tool.

Final spec file

So here is the spec file I eventually came up with. I welcome any optimizations to be made!

Step 4

“Install compilers and dependencies”.
On my host I had to do a bunch of installations to get the requirements installed (your mileage may vary), including the Percona Server package for the /usr/lib64/mysql/plugin/dialog.so file:
yum install Percona-Server-server-55-5.5.55-rel38.8.el6.x86_64 libtool systemtap-sdt-devel unixODBC-devel

Step 5

“Build the RPM file”. Such an easy step, but it took about a week of back and forth with building the RPM file (which configures, tests and packages up everything), so I went between this step and updating the spec file a lot.

cd rpmbuild/SPECS/
rpmbuild -bb rpmbuild/SPECS/php-cli.spec

Then I installed my PHP file and tested it, and it worked!
# rpm -e php-cli –nodeps
# rpm -Uvh /root/rpmbuild/RPMS/x86_64/php70u-cli-7.0.22-2.ius.el6.x86_64.rpm –nodeps
Preparing… ########################################### [100%]
1:php70u-cli ########################################### [100%]

I hope you have similar success, and if you have updates to the spec files and lists of packages to install, please let me know!

via Planet MySQL
LDAP with auth_pam and PHP to authenticate against MySQL

Synology launches ‘budget-friendly’ 4-bay NAS that can handle up to 40TB of storage

Data storage solutions manufacturer Synology has added a new, lower-cost NAS to its DiskStation j line that has a maximum capacity of 40TB, and is aimed at the home user and photography enthusiast.

The DS418j has 4 disk bays, each of which can accommodate HDDs or SSDs of up to 10TB. Data is controlled via a new 64-bit dual-core processor and the device has been fitted with 1GB of DDR4 RAM—doubling the memory of the previous model.

Synology claims the DS418j can read at 112MB/s and write at 87MB/s encrypted and in RAID 5, providing fast data storage and retrieval, and allowing high resolution media streaming and synchronisation in domestic or small office environments. The case has a single RJ-45 1GbE LAN port and a pair of USB 3.0 sockets, and can support wireless access via a dongle.

The DS418j comes with a two-year warranty and is available now for £284/€312/$330. For more information see the Synology website.

Press Release

Synology® Introduces DiskStation DS418j

Powerful entry-level 4-bay NAS for home data backup and multimedia streaming

Synology® Inc. today launched the new DiskStation DS418j, a budget-friendly 4-bay NAS server designed to help home and individual users to manage, protect and share data effectively.

DS418j is powered by a brand-new 64-bit dual-core CPU and delivers an outstanding encrypted file transfer performance at over 112 MB/s reading and 87 MB/s writing under a RAID 5 configuration in a Windows® environment. Equipped with a 1GB DDR4 memory, which is twice the size of its predecessor, and over 40TB single volume raw capacity support, DS418j brings flexible storage management in addition to smooth performance.

"In the digital era, photos, videos, and digital assets are being generated faster than ever. For home and individual users, it is essential to have a private storage solution that can satisfy the needs for both large storage capacity and secure data sharing," said Michael Wang, Product Manager at Synology® Inc. "Combining hardware innovations and rich applications, the 4-bay DS418j allows users to enjoy cloud synchronization and multimedia streaming at a competitive price."

Designed with user experience and energy efficiency in mind, DS418j is equipped with adjustable front LED indicators, allowing users to precisely control and schedule the brightness in four levels. In addition, the model consumes only 21.22 watts in full operation and as little as 8.97 watts in HDD hibernation.

DS418j runs on DiskStation Manager (DSM), one of the most advanced and intuitive operating systems for network-attached storage devices which offers a wide range of applications from backup to multimedia for home and personal use. With DSM, Synology® has received numerous media accolades, including PC Mag Readers’ Choice for seven years in a row.

For more information, please visit http://ift.tt/2vHTgid

Availability
Synology DiskStation DS418j is available worldwide immediately.

via News: Digital Photography Review (dpreview.com)
Synology launches ‘budget-friendly’ 4-bay NAS that can handle up to 40TB of storage

Watch: Quick DIY PVC Bow Without Heat or Power Tools

This video gets right down to it, showing us one way to make a good, useful bow using PVC, some fiberglass rods, and some paracord — without any heat or power tools.

It’s super simple, and clearly works well. He uses fiberglass “driveway markers” slipped into the 3/4″ PVC to give it more strength and spring.

He says its draw weight is about 35 pounds.

After about the halfway point, he starts to wander and there’s nothing else about this bow build… so it really is a quick little video. Check it out.

The post Watch: Quick DIY PVC Bow Without Heat or Power Tools appeared first on AllOutdoor.com.

via All Outdoor
Watch: Quick DIY PVC Bow Without Heat or Power Tools

Nintendo faces lawsuit over the Switch’s detachable controllers

The Nintendo Switch certainly isn’t the first gaming tablet, but is it directly riffing on others’ ideas? Gamevice thinks so. The accessory maker is suing Nintendo for allegedly violating a patent for concepts used in the Wikipad, its gaming-oriented Android slate, as well as its namesake add-on controllers for phones and tablets. According to the suit, the Switch and its removable Joy-Con controllers are too close to Gamevice’s vision of a combination of detachable game controller and a device with a "flexible bridge section." Not surprisingly, the lawsuit calls for both damages and a ban on Switch sales.

Gamevice declined to comment on the lawsuit. We’ve reached out to Nintendo as well, and will let you know if it has a response.

In some ways, the lawsuit is an epilogue to a long-running story. The Wikipad team started out with grand plans for tablet gaming in 2012, when it promised elaborate features as glasses-free 3D and game streaming. However, it didn’t work out that way: delays and a rethink prompted a change in design, and while the Wikipad did receive some acclaim, Gamevice eventually dropped it in favor of its add-ons for phones and tablets. Although Nintendo’s Switch clearly has some differences (it’s intended more as a hybrid TV and portable console, for one thing), it’s effectively showing what could have been if the Wikipad had taken off.

Source: RPX Insight

via Engadget
Nintendo faces lawsuit over the Switch’s detachable controllers

Heal thyself: Skin-zapping chip aims to reprogram cells for tissue repair


reader comments
12

With a jolt from a tiny chip, humdrum skin cells may transform into medical mavericks.

A small electrical pulse blasts open tiny pores in cells and zaps in fragments of DNA or RNA loaded in the chip’s nanochannels. Those genetic deliveries then effectively reprogram the skin cells to act like other types of cells and repair damaged tissue. In early experiments on mice, researchers coaxed skin cells to act like brain cells. They also restored blood flow to a rodent’s injured limb by prompting skin cells to grow into new blood vessels.

The technology, published this week in Nature Nanotechnology, is still a long way from confirmed clinical applications in humans. But, the Ohio State researchers behind the chip are optimistic that it may one day perform myriad medical feats—including healing severe injuries, restoring diseased organs, erasing brain damage, and even turning back the clock on aging tissues.

The researchers, led by regenerative medicine expert Chandan Sen and biomolecular engineer L. James Lee, expect to begin clinical trials next year.

“The concept is very simple,” Lee said in a press statement. “As a matter of fact, we were even surprised how it worked so well. In my lab, we have ongoing research trying to understand the mechanism and do even better. So, this is the beginning, more to come.”

Their concept is similar to other cell-based regenerative therapies under development, but it skips some pesky steps. Some other methods explored by researchers—and dubious clinics—involve harvesting adult cells from patients, reprograming them to revert to stem cells, then injecting those cells back into patients, where they develop into a needed cell type.

But this setup has snags. Researchers often use viruses to deliver the genetic elements that reprogram the cells, which have caused cancer in some animal studies. The method also requires a lot of manipulation of cells in lab, adding complications. It’s unclear if the suspect stem cell clinics are even successful at reprogramming cells.

Doctored cells

The method used by Lee, Sen, and colleagues ditches the need for a virus and for any cellular handling. The electrical pulse opens pores in cells that allow for direct genetic delivery—a process called electroporation. The researchers skipped the need to make stem cells by using preexisting methods of converting one cell type directly into a different one. Generally, this works by introducing bits of genetic material that code for gene regulators key to a specific cell type. Once delivered, these regulators can switch genes on or off so cells can start acting like the different cell type. Such a method has been worked out for creating liver, brain, and vascular cells from other cell types.

Finally, the researchers’ method also all takes place on a patch of skin on a living subject, potentially directly where it’s needed—no cell harvesting or lab manipulations are required. (That said, the researchers note that future therapies could use skin patches to generate specific cell types that can then be transferred to other locations in the body if needed.)

So far, the researchers have dabbled with making brain cells and vasculature cells from skin cells. In early experiments, their direct delivery proved effective at converting the cells. The researcher verified that the converted cells mirrored normal brain and vasculature cells’ gene expression profiles—the pattern of genes they have turned on and off.

In their ultimate test, the researchers severed leg arteries in a handful of mice. Then a researcher placed over the injuries nanochips loaded with genetic ingredients for converting skin cells to vasculature cells. The conversion reached cells deep within the skin layers. After a week, the researchers saw more blood flow and less tissue death in the treated mice compared with control animals that weren’t treated.

Much work still needs to be done to test the idea and prove it’s effective for certain treatments. But the researchers are optimistic. They conclude in the study that the technology “has the potential to ultimately enable the use of a patient’s own tissue as a prolific immunosurveilled bioreactor.”

Nature Nanotechnology, 2017. DOI: 10.1038/nnano.2017.134  (About DOIs).

via Ars Technica
Heal thyself: Skin-zapping chip aims to reprogram cells for tissue repair

Tracking Internal Marketing Campaigns With Google Analytics

Editor’s Note: This article is targeted at readers experienced in using Google Analytics. If you’re new to Analytics, the following guide might be challenging.

Many websites use internal advertising in the form of banners or personalized product recommendations to bring additional products and services to the attention of visitors and to increase conversions and leads.

Example: Some internal advertising on Amazon.com1
Example: Some internal advertising on smashingmagazine.com (View large version2)

Naturally, the performance and effectiveness of internal marketing campaigns should be assessed, too, as this is one of the most powerful instruments for generating more leads, more conversions and more revenue on your website. In many cases, web analysts use Google Analytics’ UTM campaign parameters3 to track internal advertising.

For those of you who are not familiar with UTM parameters, this is what an internal link could look like if you add UTM parameters:

http://ift.tt/2wwFKvn

The problem: UTM parameters are intended to be used in external campaigns (for example, promoted posts on Facebook). Unfortunately, they are not suitable for tracking internal campaigns. In this article, I’ll explain why you would corrupt your Google Analytics data when using UTM parameters for internal tracking purposes.

The solution: I have developed a set of URL parameters (called ITM parameters) that make tracking internal marketing just as simple and convenient as the familiar UTM parameters. Set-up takes a little time, but then you’d have a means of tracking internal marketing that both is easy to use and delivers accurate data. This article presents the solution and includes a precise description of all the necessary steps.

Collecting Data, The Powerful Way Link

Did you know that CSS can be used for collecting statistics? Indeed, there’s even a CSS-only approach for tracking UI interactions using Google Analytics. Read more →4

Can Internal Marketing Campaigns Be Tracked With UTM Parameters? Link

Well, yes and no. There’s no technical problem with using UTM parameters for internal links as well. However, in the context of web analytics, it is definitely not advisable! As soon as a visitor clicks on an internal link with UTM parameters, their current session in Google Analytics is ended and a new session is started with the source and medium information from the UTM parameters. This has disagreeable consequences for your data in Google Analytics:

  • The number of sessions counted increases artificially, because every click on an internal link with UTM parameters results in a new session.

Clicks on an internal link with UTM parameters results in a new session5
Clicks on an internal link with UTM parameters result in a new session. (View large version6)

  • A conversion can no longer be traced back to the original, external source of the traffic, because the new session uses the source and medium information from the UTM parameters, which makes it more difficult to quantify the contribution of various external traffic sources to your conversions.

New session uses the source and medium information from the UTM parameters7
New session uses the source and medium information from the UTM parameters. (View large version8)

  • If a new visitor clicks on an internal link with UTM parameters right on the landing page, this immediately results in a new session. The upshot of this is that the bounce rate for visitors from external sources increases artificially.

Naturally, you want to avoid these negative consequences, because they reduce the quality of your website’s analytics.

Nevertheless, the use of UTM parameters to track banners ads and product recommendations is widespread. For one thing, many web analysts are unaware of the problem described above. But the deciding factor is something else: The UTM parameters are easy to use, very flexible and accessible, even for the less technically adept among us, which is why they are so often misused for internal campaigns.

In my blog post “Tracking Teasers and Internal Links in Google Analytics9” (in German only, unfortunately), I’ve already covered how clicks on internal banner ads can be monitored and evaluated using event tracking or enhanced e-commerce tracking.

In practice, I’ve discovered that setting up tracking is too much for many people and/or they don’t follow through with it systematically. As a result, they fall back on using the UTM parameters for internal purposes.

So, I asked myself the following question: If the UTM parameters are so popular, can I find a way to track and evaluate internal marketing campaigns using URL parameters?

Tracking And Evaluating Internal Marketing Campaigns Using URL Parameters Link

The answer is yes, it is possible!

Not with UTM parameters, because the type of evaluation you can do with these is predetermined in Google Analytics. What we can do, however, is define a new set of URL parameters that we then use to track internal marketing campaigns.

I call these parameters simply ITM parameters. Instead of using utm_source, utm_medium and so on, you would now use the parameters itm_source, itm_medium and so on for tracking your internal advertising. I consciously took the names of the UTM parameters, so that existing URLs can be easily modified with “find and replace.”

OK, let’s get started. I’ll show you how to use Google Tag Manager and Google Analytics to create a set of URL parameters for tracking internal marketing campaigns and for analyses.

ITM Parameters At A Glance Link

Drawing on the UTM parameters, we’ll use the following ITM parameters to track internal marketing campaigns:

  • itm_source
    This identifies the source of the traffic. In the case of internal ads, this is usually your own domain. However, if you employ cross-domain tracking or different ad servers, different traffic sources can appear in this field.
  • itm_medium
    This is the medium of the internal ad; for example, a banner or product recommendation.
  • itm_campaign
    This is the name of the internal marketing campaign.
  • itm_content
    This is a parameter to differentiate between similar content, or links in the same ad; for example, text_link or banner_468x60.
  • itm_term
    This is the search keyword that triggered the internal ad. Alternatively, it could be a keyword to categorize the ad by content.

A URL that uses all five ITM parameters might look something like this:

http://ift.tt/2wwQj1x?itm_source=mydomain.tld&itm_medium=banner_ad&itm_campaign=sale-2017&itm_content=banner_468x60&itm_term=special_offers

Now we have to make sure that the information we are collecting with the ITM parameters is passed to Google Analytics and available for analysis.

Note: To analyze ITM parameters in Google Analytics, we need to create customs reports. In this article, I’ll show you how to create these reports, but I’ve uploaded them to Google Analytics’ Solutions Gallery, too! You’re free to download the reports here10 and here11.

Creating Custom Dimensions In Google Analytics Link

All of the information that we collect with the ITM parameters should be saved in Google Analytics. To this end, we’ll create custom dimensions in Google Analytics. Here, we have to consider their scope: Should we create the new dimensions at the session or hit level?

Let’s go back to the UTM parameters for a moment. The UTM parameters are session-level parameters — that is, the information regarding source, medium, campaign and so on applies to the entire session. What this means for internal campaign tracking is that only the information regarding the last link clicked is captured with UTM parameters.

However, if we want to measure the influence of internal ads on conversion rates, then we would be interested in all of the clicks on internal banners, not just the last one. For this reason, it is advisable to create the new dimensions in Google Analytics at the hit level (for example, pageviews) in order to record the individual clicks on banner ads accurately.

The catch is that session-based and hit-based dimensions can only be combined in custom reports in exceptional cases. That means that we can’t combine conversions (session level) with individual clicks on banners (hit level), and it means that an internal promotion report, like the one we know from enhanced ecommerce, is not available as a custom report.

So, how do we solve this problem?

  • We could capture the ITM parameters at the session level.
    This allows us to use the predefined campaign reports in Google Analytics as templates for reports on internal marketing campaigns. This is similar to tracking with UTM parameters, which also don’t provide detailed information at the hit level. Not ideal, but easy to manage.
  • Or we could capture the information from the ITM parameters at the hit level.
    Then, we can’t establish a direct relationship between the ITM data and conversions. However, we can use data segments to analyze precisely those sessions in which internal banners were clicked and, thus, determine the influence of internal marketing campaigns on conversions.

It’s up to you which method to use. If you capture the information from the ITM parameters on a session basis, it will be easier to manage the reports in Google Analytics, but the data will be less precise. If you go with the second variant, you will capture very accurate click data for internal banners but will have to invest more time to link this data to your conversions.

Alternatively, you could employ both methods in parallel by creating two custom dimensions for each ITM parameter in Google Analytics: one at the session level, the other at the hit level. Then, you will be free to decide how you want to analyze the data. The disadvantage of this solution is that you need 10 custom dimensions in Analytics to implement it, rather than 5. This might be a critical point, since the free version of Analytics allows a maximum of 20 custom dimensions.

Creating Dimensions For The ITM Parameters Link

In the following step, we’ll create two sets of dimension: one at the hit level and one at the session level. If you’ve already decided whether to count clicks on internal ads in sessions only or as hits only, then you’ll only need to create the one set that you require.

Creating New Dimensions at the Hit Level Link

Click on “Admin” in Google Analytics. Select “Custom Dimensions” under “Custom Definitions” in the “Property” area.

Creating custom dimensions at the hit level, step 1.12
Creating custom dimensions at the hit level, step 1.

Now click on “+ New Custom Dimension”:

Creating custom dimensions at the hit level, step 213
Creating custom dimensions at the hit level, step 2.

  1. Name the dimension itm_source_h. The _h at the end of the name indicates the scope “Hit.”
  2. Select “Hit” as the scope of the dimension.
  3. Now just click “Create.”

Repeat these steps for the dimensions itm_medium_h, itm_campaign_h, itm_content_h and itm_term_h.

When you’re done, you should have five new dimensions at the hit level in Google Analytics:

Custom Dimensions at hit level14
Custom dimensions at the hit level

Creating New Dimensions at the Session Level Link

Repeat the steps again for the second set of dimensions (at the session level):

Creating custom dimensions at the session level, step 415
Creating custom dimensions at the session level, step 4.

  1. Name the dimension itm_source_s. The _s at the end of the name indicates the scope “Session.”
  2. Select “Session” as the scope of the dimension.
  3. Again, just click “Create.”

Repeat these steps for the dimensions itm_medium_s, itm_campaign_s, itm_content_s and itm_term_s.

Now, you should have five new dimensions at the session level as well:

Custom dimensions at the session level16
Custom dimensions at the session level

Google Analytics: Excluding Query Parameters Link

When you add the ITM parameters to your URLs, these enhanced URLs will also be displayed in Google Analytics (for example, in the content reports). In this case, you would see the same page twice: once with the additional ITM parameters and once without. We want to prevent this, because the content of the pages with and without ITM parameters would be identical. Thankfully, Google Analytics provides a simple means of excluding query parameters.

First, select your “Property” in the “Admin” view, and then the desired “data view.” Under the “View Settings” menu item, you’ll find the section “Exclude URL Query Parameters”:

Excluding query parameters17
Excluding query parameters

Enter the ITM parameters that are to be used for tracking in the field provided: itm_source, itm_medium, itm_campaign, itm_content, itm_term.

Then, just save the changes, and we’re done here.

Search Console: Adding ITM Parameters Link

Remember that the Google Bot will also find your URLs with the ITM parameters. This could mean that these URLs will also land in Google’s index and cause duplicate content. We can prevent duplicates from arising with a minor tweak in Google Search Console. Select “URL Parameters” from the “Crawl” section in Search Console:

Search Console: adding ITM parameters, step 118
Search Console: adding ITM parameters, step 1.

Now click on the “Add Parameter” button to let Google know that the ITM parameter doesn’t affect the page’s content:

Search Console: adding ITM parameters, step 219
Search Console: adding ITM parameters, step 2.

  1. Enter itm_source in the “Parameter” field.
  2. Select the entry “No: Doesn’t affect page content (ex: tracks usage)” from the dropdown menu.
  3. “Save.”

Repeat these steps for the parameters itm_medium, itm_campaign, itm_content and itm_term as well. When you’re finished, all of the ITM parameters should be configured as follows:

Search Console: ITM parameters added20
Search Console: ITM parameters added (View large version21)

Setting Up Internal Campaign Tracking In Google Tag Manager Link

Now that we have laid the groundwork, we finally come to the point that probably interests you the most: How will the ITM parameters be read with Google Tag Manager and be passed to the custom dimensions in Google Analytics?

This is surprisingly easy. All we need to do is store the content of the ITM parameters in user-defined variables and then send them to Google Analytics with the pageview tag. That’s it!

Let’s begin

Creating User-Defined Variables Link

For each of the five ITM parameters, we’re going to create a variable in Tag Manager. When a URL that contains one of these parameters is called, its value will be saved in the corresponding variable.

Click on “Variables” in the “Workspace” in Tag Manager and then on “New” under “User-Defined Variables”.

Creating user-defined variables, step 122
Creating user-defined variables, step 1. (View large version23)

In the window that opens, we’ll define the first variable, itm_source URL-Parameter, where we’ll be storing the value from the URL parameter itm_source.

Creating user-defined variables, step 224
Creating user-defined variables, step 2. (View large version25)

  1. Name the variable itm_source URL-Parameter.
  2. Select “URL” as the variable type.
  3. Select “Query” as the component type.
  4. The query key is itm_source. This is the name of the URL parameter that is going to be saved in this variable.
  5. Make sure that “Page URL / Default” is selected under “URL Source.”
  6. Click “Save” to finish defining the variable.

Repeat these steps for the four other parameters (itm_medium, itm_campaign, itm_content and itm_term).

Once you’re finished, there will be five variables available in Tag Manager to save the values from the ITM parameters.

User-defined variables in Google Tag Manager26
User-defined variables in Google Tag Manager (View large version27)

In this screenshot, you can see not only the five newly created variables (itm_xxx URL-Parameter), but also five additional variables with the type “Constant” (itm_xxx Index). Let’s take a closer look at the variable itm_source Index:

itm_xxx index variables28
itm_xxx index variables

The variable itm_source Index is a constant and has the value 14.

What is it good for?

In the next step, we’re going to send the contents of the five itm_xxx URL-Parameter variables to the custom dimensions we prepared in Google Analytics. Each of these custom dimensions is addressed using an index number. In the example, the index number for the dimension itm_source is 14 (see the section “Creating Dimensions for the ITM Parameters”). In your case, the number will probably be different. If you use the custom dimensions in different tags, you will probably have to check the number of the relevant dimension every time in Google Analytics. That’s why I like to save the dimension number in its own variable — so that I don’t have to remember it. Saving the index numbers in separate variables is not necessary; you can also just address the dimensions in the tag directly with the respective number.

Enhancing the Pageview Tag for ITM Parameters Link

Even with the most basic configuration of Google Analytics and Tag Manager, you will have at least one tracking tag activated — namely, the tag used to send the pageviews to Google Analytics. This tag is ideally suited to be enhanced for the ITM parameters.

Click on “Tags” in the “Workspace” in Tag Manager, and then select the tag with the trigger “All Pages”:

Enhancing the Pageview tag, step 129
Enhancing the Pageview tag, step 1. (View large version30)

Clicking on the name of the tag will display its details:

Enhancing the Pageview tag, step 231
Enhancing the Pageview tag, step 2.

Make sure that the “Track Type” is set to “Pageview.” This is the tag used to send pageviews to Google Analytics.

Open the configuration page for the tag, and add the ITM parameters:

Enhancing the Pageview tag, step 332
Enhancing the Pageview tag, step 3. (View large version33)

  1. Open the “Custom Dimensions” area.
  2. Click on the button “+ Add Custom Dimension.”
  3. Enter the variable in the “Index” field and the variable in the “Dimension Value” field. In this way, the content of the URL parameter itm_source — which we saved before in the variable itm_source URL-Parameter — will be passed to dimension number 14 in Analytics.
  4. If you don’t use variables to save the index numbers, you can simply enter the respective number here (for example, 15). Add the four remaining dimensions for the variables itm_medium-URL Parameter, itm_campaign URL-Parameter,itm_content URL-Parameter and itm_term-URL Parameter.
  5. “Save.”

Here, at the latest, you’ll have to decide whether the information from the ITM parameters should be saved at the session or hit level in Google Analytics. In the example above, I used the index numbers of the session-based dimensions. You will have to adjust this according to your needs.

Now that we have enhanced the pageviews tag, don’t forget to publish the changes in Tag Manager. Now, when the requested URL contains additional ITM parameters, Tag Manager will read them and pass the information to Google Analytics.

Evaluating Internal Marketing Campaigns In Google Analytics Link

After you have finished setting up ITM parameter tracking in Tag Manager, it’s time to take a closer look at the analysis options in Google Analytics. In doing so, we will have to decide whether to save the ITM parameters at the session or hit level (see the section “Creating Custom Dimensions in Google Analytics”).

Session-Based Evaluation Link

If you save the ITM parameters in custom dimensions with the scope “Session,” then evaluation of the internal marketing campaign will work in much the same way as with UTM parameters. As I’ve already mentioned, the disadvantage of this method is that there is only one set of ITM data for the whole session, which is overwritten each time a URL with ITM parameters is clicked. As a result, you will only see the information relating to the last click on an internal banner. Of course, this was already the case when using UTM parameters.

Session-based tracking of ITM parameters is more convenient in terms of web analytics, but less accurate than capturing the ITM parameters at the hit level.

Creating a Custom Report Link

The simplest way to create a custom report for evaluating internal marketing campaigns is via the standard reports for campaigns. In Google Analytics, select the report via “Acquisition” → “Campaigns” → “All Campaigns.” You can then modify this report to suit your evaluation requirements by clicking on the “Customize” button.

I’ve made the report for session-based evaluation of internal marketing available34 in the Google Analytics Solutions Gallery.

This is what it looks like:

Custom report: internal marketing campaigns (session-based), 135
Custom report: internal marketing campaigns (session-based), 1. (View large version36)

  1. I named the report “Internal Marketing Campaigns (session-based).”
  2. The report has two tabs: an “Explorer,” which you can see here, and a “Flat Table,” which is visible in the next screenshot.
  3. The “Explorer” tab can have various freely customizable metric groups.
  4. The “Explorer” drills down through the dimensions itm_campaign_sitm_content_sitm_term_s. So, the internal marketing campaign level is immediately visible in this report. The two other dimensions, itm_source_s and itm_medium_s, can be added easily as required.
  5. I included a filter, so that the report only displays sessions where the dimension itm_campaign_s is not empty.

Let’s look at the second tab in the report:

Custom report: internal marketing campaigns (session-based), 237
Custom report: internal marketing campaigns (session-based), 2. (View large version38)

  1. This tab features a flat table which is well suited for exporting the data.
  2. The table uses all of the session-based dimensions that store ITM parameters: itm_source_s, itm_medium_s, itm_campaign_s, itm_content_s and itm_term_s.
  3. The specified metrics are freely customizable.

Important tip: The table only displays sessions that have values in all five(!) ITM dimensions. If, for example, you don’t use the parameter itm_term, then the dimension itm_term_s will remain empty, which means that no sessions would be displayed in the table. We can eliminate this restriction in two ways: either by removing the unused dimensions from the report or by using Google Tag Manager to set a default value in the dimension if the corresponding ITM parameter is empty. The second option necessitates more work in Tag Manager, as we would then have to distinguish between mandatory parameters and optional parameters.

Session-Based Evaluation of Internal Marketing Campaigns Link

With the aid of some simple example data, I’d like to illustrate how the data from internal marketing campaigns is presented in this report.

In the “Explorer” tab, it is possible to combine various dimensions with one another; for example, itm_campaign_s (marketing campaign) and itm_content_s (ad content):

Session-based evaluation of internal marketing campaigns, 139
Session-based evaluation of internal marketing campaigns, 1 (View large version40)

Or you can analyze the effectiveness of internal marketing campaigns in relation to the external traffic sources. This, for instance, is not possible with UTM parameters:

Session-based evaluation of internal marketing campaigns, 241
Session-based evaluation of internal marketing campaigns, 2 (View large version42)

The table contained in the report provides you with a session-based view of all the ITM parameters and makes it simple to export data for further processing in Excel, Google Docs, etc.

Session-based evaluation of internal marketing campaigns, 343
Session-based evaluation of internal marketing campaigns, 3 (View large version44)

Hit-Based Evaluation Link

Hit-based evaluation of internal marketing campaigns provides you with an accurate overview of how often specific advertising content has been clicked. However, an additional step is necessary in order to establish the relationship to the conversions achieved. This makes this method of analysis more time-consuming than session-based evaluation. However, it enables more sophisticated analyses.

Creating a Custom Report Link

There is no standard report for hit-based evaluation that we can take and customize for our purposes, so we’ll have to create our own report that will provide us with information on how often the individual pieces of ad content have been clicked. Based on this report, we will then infer the data segments that we need for our analysis. But first, the report.

I’ve also made this report available45 in the Google Analytics Solutions Gallery.

This is what it looks like:

Custom report: internal marketing campaigns (hit-based), 146
Custom report: internal marketing campaigns (hit-based), 1 (View large version47)

  1. I named it “Internal Marketing Campaigns (hit-based).”
  2. The report has two tabs: an “Explorer,” which you can see here, and a “Flat Table,” which is visible in the next screenshot.
  3. I kept the “Explorer” tab simple; it should provide you with an overview of the usage of internal advertising over time. That’s why it only contains a single metric group with the metric “Hits.”
  4. The “Explorer” drills down through the dimensions itm_source_hitem_medium_hitm_campaign_hitm_content_hitm_term_h.
  5. I included a filter so that the report only displays sessions in which the dimension itm_source_h is not empty.

Let’s look at the second tab in the report:

Custom report: internal marketing campaigns (hit-based), 248
Custom report: internal marketing campaigns (hit-based), 2 (View large version49)

  1. This tab features a flat table that provides you with an overview of the clicks on different advertising content and that is well suited for exporting the data.
  2. The table uses all of the hit-based dimensions that store ITM parameters: itm_source_h, itm_medium_h, itm_campaign_h, itm_content_h and itm_term_h.
  3. I only used the metric “Hits.” The session-based metrics such as number of conversions cannot be combined with the selected dimensions.

Important tip: This table also only displays hits that have values in all five(!) ITM dimensions. If, for example, you don’t use the parameter itm_term, then the dimension itm_term_h would remain empty, which means that no hits would be displayed in the table. In this case, you would either have to remove the unused dimensions from the table or use default values for the parameters (see above).

Hit-Based Evaluation of Internal Marketing Campaigns Link

As I already mentioned, combining hit-based data (such as clicks on internal ads) and session-based data (such as conversions) directly is not possible in Google Analytics. That’s why we first use the custom report “Internal Marketing Campaigns (hit-based)” to identify data segments that we can then apply to the acquisition reports, for example.

Let’s look at the report, jumping straight to the table this time:

Hit-based evaluation of internal marketing campaigns, 150
Hit-based evaluation of internal marketing campaigns, 1 (View large version51)

In the table, we see the number of clicks (hits) on internal ads. The data is grouped based on the combinations of ITM parameters that occurred. In a sense, this is our raw data. Now we have to consider which data segments we can identify:

  1. In this example, there is only one internal marketing campaign, but this is already interesting as a data segment. We want to examine more closely how the number of conversions differs between sessions with and without clicks on internal ad content.
  2. Then, we drill down a little deeper and differentiate based on the dimension itm_term_h, which identifies ad content with a relationship to web analytics (webanalyse) or AdWords (adwords).

Even with so little data, we can continue to slice it up into new segments. For example, you could create segments from the dimensions itm_term_h and itm_content_h to examine which type of ad content works better for itm_term_h = "webanalyse" or itm_term_h = "adwords". But that’s just a side note.

I’m not going to go into the details of creating the data segments. You can see the definition of the segments in their names.

We’ll start with the segment itm_campaign_h = marketing2017. To make the difference clearer, I also defined a segment that includes all of the users who didn’t click on internal ads. This segment is named itm_campaign_h != marketing2017. != is the operator for “not equal to.”

Hit-based evaluation of internal marketing campaigns, 252
Hit-based evaluation of internal marketing campaigns, 2 (View large version53)

  1. Even with these two simple segments, we can distinguish between users who clicked on internal advertising and those who didn’t.
  2. We can see significant differences in the number of conversions and the conversion rate.
  3. We can examine the effectiveness of the internal marketing campaign in relation to the traffic sources. This data provides us with valuable starting points for better targeting and optimizing our internal marketing campaigns. Performing this kind of analysis is impossible if you use UTM parameters!

In the second short analysis, we’ll consider the segments itm_term_h = webanalyse and itm_term_h = adwords. These segments enable us to differentiate based on the context of the internal ad with regard to content — in this example, therefore, AdWords (adwords) and Web Analytics (webanalyse).

Hit-based evaluation of internal marketing campaigns, 354
Hit-based evaluation of internal marketing campaigns, 3 (View large version55)

  1. We can now distinguish between internal ads with the subjects AdWords (adwords) and Web Analytics (webanalyse). Once again, I’ve activated the segment itm_campaign_h != marketing2017 (users without clicks on internal ads) as a control group.
  2. If we look at the number of conversions, we’ll notice that the total of the individual segments (3 + 4 + 2 = 9) is greater than the total number of conversions. Here we might have expected that the sum of itm_term_h = webanalyse and itm_term_h = adwords would be six conversions, but it is actually seven conversions. This is a consequence of hit-based evaluation. It is possible that users click on banner ads for AdWords and banner ads for Web Analytics during a single session. These users would, then, be part of both data segments!

Naturally, these examples are just a taste of the many analyses that are possible with data segments. But they clearly show how much more the potential for analysis is when clicks on internal ads are captured at the hit level.

Recommendations Link

  • Don’t use UTM parameters to track internal marketing campaigns. They are intended for external campaigns. If you use these parameters internally, you’ll lose information about the sources of the users’ traffic. Moreover, you’ll only ever have information about a user’s last click on an ad.
  • If you already employ ecommerce tracking on your website, you should check whether you can monitor your internal marketing with enhanced ecommerce tracking.
  • If enhanced ecommerce tracking is not an option for you or you are looking for a solution that’s easy to manage, then consider implementing tracking using ITM parameters.
  • A little technical know-how is required to implement this method of tracking, but later on the parameters will be as easy to use as the UTM parameters.
  • Give some thought to capturing data: Should clicks on internal banners be saved at the hit or session level? Both options have their advantages and disadvantages (data quality versus the time and effort of analysis). If necessary, you could even use both options in parallel. Then, you’d need ten custom dimensions in Analytics instead of five. Using both methods in parallel would allow you to compare them directly.
  • The five ITM parameters that I presented here are not set in stone. I chose them because they make the switch from UTM parameters simple. If you require additional data, adding further tracking parameters is easy.

All done! This article ended up being much longer than I originally planned.

Thanks for sticking with me right to the end and for taking on the challenges of tracking and analyzing internal marketing campaigns with me. As you’ve seen, there are a number of decisions to be made and various settings to be configured.

But they’re worth the effort — internal advertising is one of the most powerful instruments for generating more leads, more conversions and more revenue on your website. In order to employ this instrument profitably, you’ll need data that allows for the evaluation and optimization of your internal marketing campaigns. Particularly if you’re offering personalized product recommendations and the like, detailed data is indispensable for identifying jumping-off points for optimizations.

All the best in the analysis of your internal marketing campaigns!

(da, yk, vf, al, il)

  1. 1 http://ift.tt/2vq5jhG
  2. 2 http://ift.tt/2vq5jhG
  3. 3 http://ift.tt/1lJqzb6
  4. 4 http://ift.tt/2wwYISM
  5. 5 http://ift.tt/2wwVcaI
  6. 6 http://ift.tt/2wwVcaI
  7. 7 http://ift.tt/2wx8DYz
  8. 8 http://ift.tt/2wx8DYz
  9. 9 http://ift.tt/2wx4sMe
  10. 10 http://ift.tt/2vlO1Ug
  11. 11 http://ift.tt/2vmjoyh
  12. 12 http://ift.tt/2wx8Ef5
  13. 13 http://ift.tt/2wwKtNL
  14. 14 http://ift.tt/2wwZ8sg
  15. 15 http://ift.tt/2vmjrdr
  16. 16 http://ift.tt/2vmbyVf
  17. 17 http://ift.tt/2wx3z6a
  18. 18 http://ift.tt/2vm7moq
  19. 19 http://ift.tt/2vm4iJ5
  20. 20 http://ift.tt/2vlO2aM
  21. 21 http://ift.tt/2vlO2aM
  22. 22 http://ift.tt/2vmcdG8
  23. 23 http://ift.tt/2vmcdG8
  24. 24 http://ift.tt/2vmpcI2
  25. 25 http://ift.tt/2vmpcI2
  26. 26 http://ift.tt/2vlGrco
  27. 27 http://ift.tt/2vlGrco
  28. 28 http://ift.tt/2vm7kNk
  29. 29 http://ift.tt/2wwKu4h
  30. 30 http://ift.tt/2wwKu4h
  31. 31 http://ift.tt/2wwzyU1
  32. 32 http://ift.tt/2wwQjyz
  33. 33 http://ift.tt/2wwQjyz
  34. 34 http://ift.tt/2vlO1Ug
  35. 35 http://ift.tt/2wwKtgJ
  36. 36 http://ift.tt/2wwKtgJ
  37. 37 http://ift.tt/2wwFLiV
  38. 38 http://ift.tt/2wwFLiV
  39. 39 http://ift.tt/2vmaXmw
  40. 40 http://ift.tt/2vmaXmw
  41. 41 http://ift.tt/2vHX5DW
  42. 42 http://ift.tt/2vHX5DW
  43. 43 http://ift.tt/2vlA9cG
  44. 44 http://ift.tt/2vlA9cG
  45. 45 http://ift.tt/2vmjoyh
  46. 46 http://ift.tt/2vI32kn
  47. 47 http://ift.tt/2vI32kn
  48. 48 http://ift.tt/2vHLK6W
  49. 49 http://ift.tt/2vHLK6W
  50. 50 http://ift.tt/2vq7rpm
  51. 51 http://ift.tt/2vq7rpm
  52. 52 http://ift.tt/2vpz8Po
  53. 53 http://ift.tt/2vpz8Po
  54. 54 http://ift.tt/2vpJrCO
  55. 55 http://ift.tt/2vpJrCO
  56. 56 http://ift.tt/2vIcJiM
  57. 57 http://ift.tt/1hjqhFD

↑ Back to top

Tweet itShare on Facebook

via Smashing Magazine
Tracking Internal Marketing Campaigns With Google Analytics

You Can Now Download 26,000 Rare Songs Digitized From Obsolete 78 RPM Records

Vinyl-lovers know and play LPs and 45s, but few have the more obscure, earlier-generation 78 r.p.m. records. A lot of that is down to materials choices; while vinyl is reasonably durable, 78s were made from more brittle shellac and tend to shatter over time.

For that reason, The Great 78 Project has been furiously digitizing 78s before the music disappears forever.

The Great 78 Project is a community project for the preservation, research and discovery of 78rpm records. From about 1898 to the 1950s, an estimated 3 million sides (~3 minute recordings) have been made on 78rpm discs. While the commercially viable recordings will have been restored or remastered onto LP’s or CD, there is still research value in the artifacts and usage evidence in the often rare 78rpm discs and recordings.

There are three places where you can download these tracks, but the first two have crappy interfaces where it’s difficult to browse. The best link to click on is this one provided by archiving service George Blood L.P., which has checkbox filters and thumbnails that make it pretty easy to browse and sift through.

To date, the George Blood link provides access to some 26,000 tracks. And there are more on the way: The Great 78 Project currently has around 200,000 78s in total.

Via Kottke


via Core77
You Can Now Download 26,000 Rare Songs Digitized From Obsolete 78 RPM Records

The Vietnam War documentary series by Ken Burns

Together with Lynn Novick, filmmaker Ken Burns, who has previously made long documentary films on The Civil War and World War II, has made a film about perhaps the most controversial and contentious event in American history, The Vietnam War. The film runs for 18 hours across 10 installments and begins on September 17 on PBS.

David Kamp interviewed Novick and Burns for Vanity Fair and proclaims the film a triumph:

I watched the whole series in a marathon viewing session a few days before meeting with the filmmakers — a knock-you-sideways experience that was as enlightening as it was emotionally taxing. For all their unguarded anxiety about doing the war justice, Burns and Novick have pulled off a monumental achievement. Audiovisually, the documentary is like no other Burns-branded undertaking. Instead of folksy sepia and black-and-white, there are vivid jade-green jungles and horrific blooms of napalm that explode into orange and then gradually turn smoky black. The Vietnam War was the first and last American conflict to be filmed by news organizations with minimal governmental interference, and the filmmakers have drawn from more than 130 sources for motion-picture footage, including the U.S. networks, private home-movie collections, and several archives administered by the Socialist Republic of Vietnam. The series’s depiction of the Tet offensive, in which the North Vietnamese launched coordinated attacks on the South’s urban centers, is particularly and brutally immersive, approaching a 360-degree experience in its deft stitching together of footage from various sources.

The sound and music promises to thrill as well. Trent Reznor and Atticus Finch (who did the scores for The Social Network, Gone Girl, and The Girl with the Dragon Tattoo) provided original music to supplement popular music contemporary to the time. They even got The Beatles.

Then there’s all that popular music from the 60s and 70s: more than 120 songs by the artists who actually soundtracked the times, such as Bob Dylan, Joan Baez, the Animals, Janis Joplin, Wilson Pickett, Buffalo Springfield, the Byrds, the Rolling Stones, and even the ordinarily permissions-averse and budget-breaking Beatles. Of the Beatles, Novick noted, “They basically said, We think this is an important part of history, we want to be part of what you’re doing, and we will take the same deal everybody else gets. That’s kind of unprecedented.”

I’m very much looking forward to this.

Tags: David Kamp   Ken Burns   Lynn Novick   The Vietnam War   TV   video   Vietnam War   war
via kottke.org
The Vietnam War documentary series by Ken Burns

Huge trove of digitized 78rpm records

78rpm Discs

Through the Great 78 Project, the Internet Archive has been digitizing the audio from 78rpm records produced from 1898 to the 1950s. Over 25,000 high-quality recordings are currently available from artists like Edith Piaf, Irving Berlin, Lena Horne, and Duke Ellington. This preservation is important because the discs are fragile:

78s were mostly made from shellac, i.e., beetle resin, and were the brittle predecessors to the LP (microgroove) era. The format is obsolete, and just picking them up can cause them to break apart in your hands. There’s no way to predict if the digital versions of these 78s will outlast the physical items, so we are preserving both to ensure the survival of these cultural materials for future generations to study and enjoy.

From 1939, here’s Judy Garland singing Over the Rainbow. And this undated recording of Edith Piaf singing La Vie En Rose. I could listen to these all afternoon.

via kottke.org
Huge trove of digitized 78rpm records