Laravel analytics – how and why I made my own analytics package

I’ve used google analytics for couple of years and it worked quite well for me. The question arises why did I make my own analytics package. There were a couple of reasons to do so:

• Google analytics became quite complex and slow lately. Especially with introduction of the new Google Analytics 4 it became more complex, and I realised that I don’t use even 0.1 percent of its capability. This blog and the other websites I developed as side projects only need simple things like visitor count per day in a specific period, and page views for the top visited pages. That’s it!
• I wanted to get rid of the third party cookies as much as possible
• Third party analytics tools are mostly blocked by ad blockers, so I see lower numbers than the real visitors.

Requirements

• It needs to be a Laravel package, as I want to use it in couple of projects
• Keep it simple, only the basic functionality
  • Track page visits by uri, and also the relevant model ids if applicable (for example blog post id, or product id)
  • Save UserAgents for possible further analysis of visitor devices (desktop vs mobile) and to filter out bot traffic
  • Save IP address for a planned feature: segment users by countries and cities
  • “In house” solution, track the data in the applications own database
  • Only backend functionality for tracking, no frontend tracking
  • Create chart for visitors in the last 28 days, and most visited pages in the same period
• Build the MVP and push back any optional features, like
  • Aggregate the data into separate table instead of querying the page_views table (I’ll build it when the queries become slow)
  • Add geoip databse, and save the user’s country and city based on their IP
  • Add possibility to change the time period shown on the charts

The database

As I mentioned earlier the goal was to keep the whole thing very simple, so the database only consits of one table called laravel_analytics_page_views where the larave_analytics_ prefix is configurable in the config file to prevent potential conflicts with the app’s databses tables.

The schema structure/migration looks like this:

$tableName = config('laravel-analytics.db_prefix') . 'page_views';

Schema::create($tableName, function (Blueprint $table) {
    $table->id();
    $table->string('session_id')->index();
    $table->string('path')->index();
    $table->string('user_agent')->nullable();
    $table->string('ip')->nullable();
    $table->string('referer')->nullable()->index();
    $table->string('county')->nullable()->index();
    $table->string('city')->nullable();
    $table->string('page_model_type')->nullable();
    $table->string('page_model_id')->nullable();
    $table->timestamp('created_at')->nullable()->index();
    $table->timestamp('updated_at')->nullable();

    $table->index(['page_model_type', 'page_model_id']);
});

We track the unique visitors by session_id, which is of course not perfect and not 100% accurate but it does the job.

We create a polymorpthic relation with page_model_type and page_model_id if there is a relevant model to the tracked page we save the type and the id to use in the future if necessary. Also created a combined index for these 2 fiels, as they are mostly queried together when using polymorphic relations.

The middleware

I wanted and universal solution rather than adding the analytics to all the controllers created a middleware which can handle the tracking. The middleware can be added to all routes or to specific group(s) of routes.

The middleware itself is quite simple, it tracks only the get requests and skips the ajax calls. As it doesn’t make sense to track bot traffic, I used the https://github.com/JayBizzle/Crawler-Detect package to detect the crawlers and bots. When a crawler is detected it simply skips the tracking, this way we can avoid having useless data in the table.

It was somewhat tricky how to get the associated model for the url in an universal way. The solution at the end is not totally universal because it assumes that the app uses route model binding and assumes that the first binding is relevant to that page. Again it is not perfect but it fits the minimalistic approach I followed while developing this package.

Here is the code of the middleware:

public function handle(Request $request, Closure $next)
{
    $response = $next($request);

    try {
        if (!$request->isMethod('GET')) {
            return $response;
        }

        if ($request->isJson()) {
            return $response;
        }

        $userAgent = $request->userAgent();

        if (is_null($userAgent)) {
            return $response;
        }

        /** @var CrawlerDetect $crawlerDetect */
        $crawlerDetect = app(CrawlerDetect::class);

        if ($crawlerDetect->isCrawler($userAgent)) {
            return $response;
        }

        /** @var PageView $pageView */
        $pageView = PageView::make([
            'session_id' => session()->getId(),
            'path' => $request->path(),
            'user_agent' => Str::substr($userAgent, 0, 255),
            'ip' => $request->ip(),
            'referer' => $request->headers->get('referer'),
        ]);

        $parameters = $request->route()?->parameters();
        $model = null;

        if (!is_null($parameters)) {
            $model = reset($parameters);
        }

        if (is_a($model, Model::class)) {
            $pageView->pageModel()->associate($model);
        }

        $pageView->save();

        return $response;
    } catch (Throwable $e) {
        report($e);
        return $response;
    }
}

 

The routes

When developing Laravel packages it is possible to set up the package service provider to tell the application to use the routes from the package. I usually don’t use this approach, because this way in the application you don’t have much control over the routes: for example you cannot add prefix, put them in an group or add middleware to them. 

I like to create a class with a static method routes, where I define all the routes.

public static function routes()
{

    Route::get(
        'analytics/page-views-per-days',
        [AnalyticsController::class, 'getPageViewsPerDays']
    );

    Route::get(
        'analytics/page-views-per-path',
        [AnalyticsController::class, 'getPageViewsPerPaths']
    );
}

This way I could easily put the package routes under the /admin part in my application for example.

The frontend components

The frontend part consists of 2 vue components one for the visitor chart and one contains a simple table of the most visited pages. For the chart I used the Vue chartjs library (https://github.com/apertureless/vue-chartjs) 

<template>
    <div>
        <div><strong>Visitors: </strong></div>
        <div>
            <LineChartGenerator
                :chart-options="chartOptions"
                :chart-data="chartData"
                :chart-id="chartId"
                :dataset-id-key="datasetIdKey"
                :plugins="plugins"
                :css-classes="cssClasses"
                :styles="styles"
                :width="width"
                :height="height"
            />
        </div>
    </div>
</template>

<script>


import { Line as LineChartGenerator } from 'vue-chartjs/legacy'
import {
    Chart as ChartJS,
    Title,
    Tooltip,
    Legend,
    LineElement,
    LinearScale,
    CategoryScale,
    PointElement
} from 'chart.js'

ChartJS.register(
    Title,
    Tooltip,
    Legend,
    LineElement,
    LinearScale,
    CategoryScale,
    PointElement
)

export default {
    name: 'VisitorsPerDays',
    components: { LineChartGenerator },
    props: {
        'initialData': Object,
        'baseUrl': String,
        chartId: {
            type: String,
            default: 'line-chart'
        },
        datasetIdKey: {
            type: String,
            default: 'label'
        },
        width: {
            type: Number,
            default: 400
        },
        height: {
            type: Number,
            default: 400
        },
        cssClasses: {
            default: '',
            type: String
        },
        styles: {
            type: Object,
            default: () => {}
        },
        plugins: {
            type: Array,
            default: () => []
        }
    },
    data() {
        return {
            chartData: {
                labels: Object.keys(this.initialData),
                datasets: [
                    {
                        label: 'Visitors',
                        backgroundColor: '#f87979',
                        data: Object.values(this.initialData)
                    }
                ]
            },
            chartOptions: {
                responsive: true,
                maintainAspectRatio: false,
                scales: {
                    y: {
                        ticks: {
                            precision: 0
                        }
                    }
                }
            }
        }
    },
    mounted() {
    },

    methods: {

    },
}
</script>

 

Conclusion

It was quite fun and interesting project and after using it for about an month and analysing the results, it seem to be working fine. If you are interested in the code, or would like to try the package feel free to check it out on GitHub here: https://github.com/wdev-rs/laravel-analytics