Market Basket Analysis: Identifying Products and Content That Go Well Together

Market basket analysis is a powerful technique for uncovering patterns between items in transactions. Whether you're analyzing products in an online store or content on a media platform, identifying these associations can inform marketing strategies, recommendation systems, and website structure. This post covers the basics of market basket analysis, how to run it in Python or R on Snowplow ecommerce data, and how to use the results.

What is Market Basket Analysis?

Market basket analysis, also called basket analysis or shopping basket analysis, is a data mining technique that finds which products or pieces of content people tend to buy or view together. The goal is to identify sets of items that frequently co-occur, helping businesses understand how products or content are associated. Common applications include:

• Site and app layout: Place related products or articles near each other on category pages, product pages, and in the cart.
  
  
• Targeted marketing: Promote related products to customers based on past purchases.
  
  
• Recommendation engines: Suggest content or products based on user behavior patterns.
  
  

Key Terminology

• Items: Individual objects analyzed for associations (e.g., products, articles).
  
  
• Transactions: Groups of items that occur together in a single instance (e.g., a purchase, a content session).
  
  
• Rules: Statements in the form {item(s)} => {item(s)} indicating associations.
  
  
• Support: The fraction of transactions containing a specific item or set of items.
  
  
• Confidence: The probability that a transaction containing item(s) on the left-hand side (LHS) also includes item(s) on the right-hand side (RHS).
  
  
• Lift: The ratio of observed support to expected support, indicating the strength of an association.
  
  

How to calculate them for a rule A → B:

• Support of A: orders containing A ÷ all orders
  
  
• Confidence of A → B: orders containing both A and B ÷ orders containing A
  
  
• Lift of A → B: confidence of A → B ÷ support of B
  
  

For example, of 1,000 orders, 100 include a tent, 200 include a stove, and 60 include both. Support of the pair is 6%, confidence of tent → stove is 60%, and lift is 60% ÷ 20% = 3, so tent buyers are three times as likely as average to buy a stove.

Implementing Market Basket Analysis with Python or R

You can run market basket analysis on Snowplow ecommerce data in your warehouse, using Python or R. The steps below use Python and the mlxtend library, which includes the Apriori algorithm for rule mining. An R version follows the steps.

Step 1: Fetch transaction data

Query the products table built by the Ecommerce Analytics Data Model Pack for one row per product in each completed order:

SELECT
  transaction_id,
  product_id
FROM <your_schema>.snowplow_ecommerce_product_interactions
WHERE is_product_transaction = TRUE

The data model pack is built from Snowplow's ecommerce events, so purchases tracked with the older transaction events won't appear in it.

Step 2: Load the data into Python

Run the query with your warehouse's Python connector and load the result into a pandas DataFrame named rows.

Step 3: Build the baskets

Turn the rows into a basket table with one row per order and one True or False column per product:

import pandas as pd
from mlxtend.frequent_patterns import apriori, association_rules

basket = pd.crosstab(rows["transaction_id"], rows["product_id"]).astype(bool)

Step 4: Run the Apriori algorithm

Find frequent item sets, then turn them into rules and keep those with lift above 1:

itemsets = apriori(basket, min_support=0.005, use_colnames=True)
rules = association_rules(itemsets, metric="lift", min_threshold=1.0)
rules.sort_values("lift", ascending=False).head(20)

On large catalogs, swap apriori for fpgrowth from the same module. It finds the same item sets faster.

To do the same in R with the arules package:

library(DBI)
library(arules)
con <- dbConnect(odbc::odbc(), dsn = "<your warehouse DSN>")
t <- dbGetQuery(con, "SELECT transaction_id, product_id FROM <your_schema>.snowplow_ecommerce_product_interactions WHERE is_product_transaction = TRUE")
txn <- as(lapply(split(as.character(t$product_id), t$transaction_id), unique), "transactions")
basket_rules <- apriori(txn, parameter = list(sup = 0.005, conf = 0.01, target = "rules"))
inspect(head(sort(basket_rules, by = "lift"), 20))

Visualizing the Results

To scan large result sets, plot each rule's support against its confidence, colored by lift:

import matplotlib.pyplot as plt

plt.scatter(rules["support"], rules["confidence"], c=rules["lift"], cmap="viridis")
plt.colorbar(label="lift")
plt.xlabel("support")
plt.ylabel("confidence")
plt.show()

In R, the arulesViz package draws the same view with plot(basket_rules).

Interpreting the Results

Focus on rules with high lift and confidence. High lift (>1) indicates strong associations, while high confidence signifies a reliable rule. Prioritize rules with higher support to maximize impact.

Applications of Market Basket Analysis

• Website Optimization: Group associated content items for improved user experience.
  
  
• Recommendation Engines: Power recommendation systems based on user behavior.
  
  
• Targeted Marketing: Develop marketing campaigns that promote frequently co-purchased items.
  
  

Expanding the Analysis Scope

While the above example focuses on transactions, the analysis can be extended to:

• Add-to-basket data, to spot upsell opportunities. Filter the products table on is_add_to_cart = TRUE and use the session ID (domain_sessionid) as the basket.
  
  
• Session-level content affinity. Use the views table from the Digital Analytics Data Model Pack, with each session as a basket and each page as an item.
  
  
• Multi-session analysis, using the user ID as the basket instead of a session or an order.
  
  

Rules mined from past orders don't change during a visit. To recommend products based on what a shopper is doing in the current session, Snowplow Signals computes user attributes from your behavioral data in real time and serves them to your applications over an API. You can start on the Signals free tier.

Market Basket Analysis on Snowplow Data

With Snowplow ecommerce data modeled in your warehouse, you can find which products and articles go together and test what to do about them. For help implementing market basket analysis on your own data, see the Ecommerce Analytics Data Model Pack or book a demo.

Market basket analysis FAQs

How does the Apriori algorithm work?

The Apriori algorithm works in passes, finding sets of items that appear together in enough transactions and then turning them into association rules. It starts by counting single items and keeping those above a minimum support threshold. Each later pass only combines item sets that survived the previous one, since a set containing an infrequent item can't be frequent. It then keeps the rules whose confidence and lift meet your thresholds.

Can market basket analysis power product recommendations?

Yes, market basket analysis can power frequently-bought-together and customers-also-bought recommendations by suggesting the items on the right-hand side of strong rules. Rules mined from past orders don't change during a visit, so they can't react to what a shopper is doing right now. For in-session recommendations, Snowplow Signals computes user attributes from your behavioral data in real time and serves them to your applications over an API.

How does market basket analysis help retailers increase sales?

Market basket analysis helps retailers increase sales by showing which products customers buy together, so they can raise average order value with cross-sells, bundles, and product placement. Common uses include recommendations on product and cart pages, bundle pricing, and promotions that pair a popular item with a related one. Rules with high lift point to pairings that happen more often than chance, which makes them the strongest candidates to test.

What are the key techniques used in market basket analysis?

The key techniques are association rule algorithms, mainly Apriori, FP-Growth, and Eclat, with results scored by support, confidence, and lift. Apriori builds frequent item sets one size at a time. FP-Growth compresses transactions into a tree so it doesn't rescan the data, which makes it faster on large data sets. Eclat works from lists of the transaction IDs that contain each item. Sequential pattern mining extends the method to the order of purchases.

Is market basket analysis supervised or unsupervised?

Market basket analysis is an unsupervised learning technique. It looks for patterns in transaction data without labeled outcomes or a target variable to predict, so there's no training set of right answers. You set minimum thresholds for support, confidence, and lift, and the algorithm returns every rule that meets them. Deciding which rules are useful is a business judgment, so teams usually review and test rules before they drive recommendations.