A/B Testing | Developers Guide

A/B testing is a powerful method to measure the impact of AI-powered shopping assistance on your store's performance. With Nobi's built-in A/B testing capabilities, you can compare key metrics like conversion rates, average order values, and user engagement between customers who have access to Nobi and those who don't.

How We Think About A/B Testing

At Nobi, we believe A/B testing should be:

Simple to implement - No complex configuration or backend changes required

Statistically sound - Consistent user assignment across sessions and pages

Comprehensive - Track both Shopify native events and Nobi-specific interactions

Merchant-friendly - Easy to set up, measure, and analyze results

Our A/B testing approach compares two groups:

Control Group: Customers who don't see Nobi components (your current shopping experience)

Treatment Group: Customers who see and can interact with Nobi components

Setting Up A/B Testing

Basic Configuration

Add the ab-test-control-group attribute to any Nobi component to enable A/B testing:


<!-- 50% control group (won't see Nobi), 50% treatment group (will see Nobi) -->
<nobi-button ab-test-control-group="0.5">Shop With AI</nobi-button>

<!-- 30% control group, 70% treatment group -->
<nobi-search-bar ab-test-control-group="0.3"></nobi-search-bar>

<!-- 80% control group, 20% treatment group -->
<nobi-toggle ab-test-control-group="0.8"></nobi-toggle>

How User Assignment Works

First Visit: Each user gets a persistent random number (0.0-1.0) stored in their browser

Group Assignment: Users are assigned to groups based on their random number:

If user's number < threshold → Control group (components hidden)

If user's number ≥ threshold → Treatment group (components visible)

Consistency: Same user always sees the same experience across all pages and sessions

Event Tracking Overview

User events are emitted to both the window and to Nobi’s analytics system for all key actions, along with whether they took those actions inside Nobi or through your existing tools. It’s then possible to compare conversion rates, AOVs, and other key metrics by group, allowing you to determine the causal improvements attributable to Nobi. You can capture thee events yourself (see our Events page for additional details), or for an extra charge, we can use our existing A/B test setup to provide you with A/B test reporting.

Capturing Transactional Data

For customers on Shopify, our standard implementation will allow us to capture transactional data like add to carts or checkouts. If your store is not built on Shopify, we can provide a way to send us this data via an API so the A/B test can run properly.

Detecting Control Group Assignment

When A/B testing is enabled, you may want to detect when a user is in the control group (Nobi hidden) so you can show your own fallback UX -- for example, your default search bar or product recommendations.

Nobi emits a nobi-component-loaded window event for every user, regardless of whether they are in the treatment or control group. The event includes an isInTreatmentGroup boolean you can use to determine which group the user belongs to.

Showing Fallback UX for Control Group Users

Listen for the event on the window and check the group assignment:


window.addEventListener("nobi-component-loaded", (event) => {
  const { componentType, isInTreatmentGroup } = event.detail.data;

  if (!isInTreatmentGroup) {
    // User is in the control group -- Nobi is hidden for this component.
    // Show your fallback UX here.
    console.log(`Nobi ${componentType} hidden -- showing fallback`);
    document.getElementById("my-fallback-search").style.display = "block";
  }
});

Event Detail Fields

The event detail object has the following structure:


{
  event: "Component Loaded",
  data: {
    componentType: "search-bar",    // Which Nobi component (e.g. "search-bar", "button", "toggle")
    isInTreatmentGroup: false,       // true = Nobi shown, false = Nobi hidden (control group)
    userRandomNumber: 0.3721         // The persistent random number assigned to this user
  },
  timestamp: "2026-04-01T12:00:00.000Z",
  source: "nobi"
}

Using the Generic Event

You can also listen to the generic nobi-analytics event, which fires for all Nobi events (not just component loads). Filter by event.detail.event === "Component Loaded" to handle only the A/B test assignment:


window.addEventListener("nobi-analytics", (event) => {
  if (event.detail.event === "Component Loaded" && !event.detail.data.isInTreatmentGroup) {
    // Control group -- show fallback
  }
});

💡

Per-component detection: If you have multiple Nobi components on the same page with different control group sizes, each fires its own event with the relevant componentType. A user could be in the control group for one component but the treatment group for another.

A/B Testing Nobi How We Think About A/B Testing Setting Up A/B Testing Basic Configuration How User Assignment Works Event Tracking Overview Capturing Transactional Data Detecting Control Group Assignment Showing Fallback UX for Control Group Users Event Detail Fields Using the Generic Event