Bots & server-side tracking

Most crawlers and monitoring tools never execute JavaScript, so the browser snippet never sees them.

Every event, from any source, is independently re-checked against known bot signatures server-side — a request never has to self-report as a bot for it to be tagged correctly. Bots are recorded with source: "server" so they show up separately and never inflate your human visitor counts. Every example below posts the same event shape to POST /event — see the API reference for the full field list.

Express (server-sdk)

Drop this into an existing Express app to tag every request at the server layer. It fires a fire-and-forget request and calls next() immediately, so it adds effectively no latency:

JavaScript

const express = require('express');
const { shonylabsMiddleware } = require('./server-sdk'); // copied into your project

const app = express();

app.use(shonylabsMiddleware({
  siteId: 'YOUR_SITE_ID',
  apiUrl: 'https://api.shonylabs.com', // optional, this is the default
}));

app.get('/', (req, res) => res.send('hello'));
app.listen(3000);

Laravel

No ready-made package — a small middleware posting the same event shape covers it. Reporting happens in register_shutdown_function, after the response has already been sent to the visitor:

PHP

<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;

class ShonylabsTracking
{
    private const SITE_ID = 'YOUR_SITE_ID';
    private const API_URL = 'https://api.shonylabs.com';

    public function handle(Request $request, Closure $next)
    {
        $userAgent = $request->header('User-Agent', '');
        $clientIp = $request->header('CF-Connecting-IP') ?? $request->header('X-Forwarded-For') ?? $request->ip();

        $event = [
            'site_id' => self::SITE_ID,
            'event_name' => 'pageview',
            'url' => $request->fullUrl(),
            'path' => $request->path(),
            'referrer' => $request->header('referer', ''),
            'visitor_id' => '',
            'session_id' => '',
            'source' => 'server',
            'is_bot' => preg_match('/bot|crawl|spider/i', $userAgent) ? 1 : 0,
            'user_agent' => $userAgent,
            'client_ip' => $clientIp,
        ];

        // Fire-and-forget: reported after the response is sent, so a slow or unreachable
        // ShonyLabs request never adds latency to the actual page load.
        register_shutdown_function(function () use ($event) {
            $ch = curl_init(self::API_URL . '/event');
            curl_setopt_array($ch, [
                CURLOPT_POST => true,
                CURLOPT_POSTFIELDS => json_encode($event),
                CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
                CURLOPT_TIMEOUT_MS => 1500,
                CURLOPT_RETURNTRANSFER => true,
            ]);
            curl_exec($ch);
            curl_close($ch);
        });

        return $next($request);
    }
}

// app/Http/Kernel.php - register it in the 'web' middleware group:
// protected $middlewareGroups = [
//     'web' => [
//         \App\Http\Middleware\ShonylabsTracking::class,
//         // ...
//     ],
// ];

Python (Flask)

A before_request hook that reports in a background thread, so the request the visitor is waiting on is never held up by the ShonyLabs call:

Python

import re
import threading
import requests
from flask import Flask, request

app = Flask(__name__)

SHONYLABS_SITE_ID = "YOUR_SITE_ID"
SHONYLABS_API_URL = "https://api.shonylabs.com"
BOT_RE = re.compile(r"bot|crawl|spider", re.IGNORECASE)


def report_hit(req):
    user_agent = req.headers.get("User-Agent", "")
    client_ip = req.headers.get("CF-Connecting-IP") or req.headers.get("X-Forwarded-For") or req.remote_addr or ""
    event = {
        "site_id": SHONYLABS_SITE_ID,
        "event_name": "pageview",
        "url": req.url,
        "path": req.path,
        "referrer": req.headers.get("Referer", ""),
        "visitor_id": "",
        "session_id": "",
        "source": "server",
        "is_bot": 1 if BOT_RE.search(user_agent) else 0,
        "user_agent": user_agent,
        "client_ip": client_ip,
    }
    try:
        requests.post(f"{SHONYLABS_API_URL}/event", json=event, timeout=1.5)
    except requests.RequestException:
        pass


@app.before_request
def track_request():
    # Fire-and-forget in a background thread so a slow/unreachable request never delays the response.
    threading.Thread(target=report_hit, args=(request._get_current_object(),), daemon=True).start()

Ruby (Rack / Rails)

A Rack middleware works for Rails, Sinatra, or any other Rack-based app. Reporting happens on a background thread, same as the other examples:

Ruby

require "net/http"
require "json"
require "uri"

class ShonylabsTracking
  SITE_ID = "YOUR_SITE_ID"
  API_URL = "https://api.shonylabs.com"
  BOT_RE = /bot|crawl|spider/i

  def initialize(app)
    @app = app
  end

  def call(env)
    Thread.new { report_hit(env) }
    @app.call(env)
  end

  private

  def report_hit(env)
    request = Rack::Request.new(env)
    user_agent = env["HTTP_USER_AGENT"] || ""
    client_ip = env["HTTP_CF_CONNECTING_IP"] || env["HTTP_X_FORWARDED_FOR"] || request.ip

    event = {
      site_id: SITE_ID,
      event_name: "pageview",
      url: request.url,
      path: request.path,
      referrer: env["HTTP_REFERER"] || "",
      visitor_id: "",
      session_id: "",
      source: "server",
      is_bot: user_agent.match?(BOT_RE) ? 1 : 0,
      user_agent: user_agent,
      client_ip: client_ip,
    }

    uri = URI("#{API_URL}/event")
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.open_timeout = 1.5
    http.read_timeout = 1.5
    post = Net::HTTP::Post.new(uri, "Content-Type" => "application/json")
    post.body = event.to_json
    http.request(post)
  rescue StandardError
    nil
  end
end

# config/application.rb (Rails) or config.ru (Sinatra/Rack):
# config.middleware.use ShonylabsTracking

Go

A standard-library net/http middleware, reporting in its own goroutine:

package main

import (
	"bytes"
	"encoding/json"
	"net/http"
	"regexp"
	"time"
)

const shonylabsSiteID = "YOUR_SITE_ID"
const shonylabsAPIURL = "https://api.shonylabs.com"

var botRE = regexp.MustCompile(`(?i)bot|crawl|spider`)

type shonylabsEvent struct {
	SiteID    string `json:"site_id"`
	EventName string `json:"event_name"`
	URL       string `json:"url"`
	Path      string `json:"path"`
	Referrer  string `json:"referrer"`
	VisitorID string `json:"visitor_id"`
	SessionID string `json:"session_id"`
	Source    string `json:"source"`
	IsBot     int    `json:"is_bot"`
	UserAgent string `json:"user_agent"`
	ClientIP  string `json:"client_ip"`
}

func shonylabsTracking(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		go reportHit(r) // fire-and-forget, never blocks the real request
		next.ServeHTTP(w, r)
	})
}

func reportHit(r *http.Request) {
	userAgent := r.UserAgent()
	clientIP := r.Header.Get("CF-Connecting-IP")
	if clientIP == "" {
		clientIP = r.Header.Get("X-Forwarded-For")
	}
	if clientIP == "" {
		clientIP = r.RemoteAddr
	}

	isBot := 0
	if botRE.MatchString(userAgent) {
		isBot = 1
	}

	body, _ := json.Marshal(shonylabsEvent{
		SiteID:    shonylabsSiteID,
		EventName: "pageview",
		URL:       r.URL.String(),
		Path:      r.URL.Path,
		Referrer:  r.Referer(),
		Source:    "server",
		IsBot:     isBot,
		UserAgent: userAgent,
		ClientIP:  clientIP,
	})

	client := http.Client{Timeout: 1500 * time.Millisecond}
	req, err := http.NewRequest(http.MethodPost, shonylabsAPIURL+"/event", bytes.NewReader(body))
	if err != nil {
		return
	}
	req.Header.Set("Content-Type", "application/json")
	if resp, err := client.Do(req); err == nil {
		resp.Body.Close()
	}
}

Nginx access log shipping

If your app isn't covered above, a small standalone process can tail your nginx access log, match crawler user agents, and ship only the bot hits — no application code changes required. It needs a JSON-formatted access log with the fields the shipper expects (hostname, path, referrer, user agent, remote address), and a mapping from each tracked hostname to its site ID.