A/B Testing | Events

A/B testing is a powerful method to measure the impact of AI-powered shopping assistance on your store's performance. With Nobi's built-in A/B testing capabilities, you can compare key metrics like conversion rates, average order values, and user engagement between customers who have access to Nobi and those who don't.

How We Think About A/B Testing

At Nobi, we believe A/B testing should be:

Simple to implement - No complex configuration or backend changes required

Statistically sound - Consistent user assignment across sessions and pages

Comprehensive - Track both Shopify native events and Nobi-specific interactions

Merchant-friendly - Easy to set up, measure, and analyze results

Our A/B testing approach compares two groups:

Control Group: Customers who don't see Nobi components (your current shopping experience)

Treatment Group: Customers who see and can interact with Nobi components

Setting Up A/B Testing

Basic Configuration

Add the ab-test-control-group attribute to any Nobi component to enable A/B testing:


<!-- 50% control group (won't see Nobi), 50% treatment group (will see Nobi) -->
<nobi-button ab-test-control-group="0.5">Shop With AI</nobi-button>

<!-- 30% control group, 70% treatment group -->
<nobi-search-bar ab-test-control-group="0.3"></nobi-search-bar>

<!-- 80% control group, 20% treatment group -->
<nobi-toggle ab-test-control-group="0.8"></nobi-toggle>

How User Assignment Works

First Visit: Each user gets a persistent random number (0.0-1.0) stored in their browser

Group Assignment: Users are assigned to groups based on their random number:

If user's number < threshold → Control group (components hidden)

If user's number ≥ threshold → Treatment group (components visible)

Consistency: Same user always sees the same experience across all pages and sessions

Event Tracking Overview

User events are emitted to both the window and to Nobi’s analytics system for all key actions, along with whether they took those actions inside Nobi or through your existing tools. It’s then possible to compare conversion rates, AOVs, and other key metrics by group, allowing you to determine the causal improvements attributable to Nobi. You can capture thee events yourself (see our Events page for additional details), or for an extra charge, we can use our existing A/B test setup to provide you with A/B test reporting.

A/B Testing Nobi How We Think About A/B Testing Setting Up A/B Testing Basic Configuration How User Assignment Works Event Tracking Overview