The Spurious Correlation
A Socratic Problem-Based Learning Lab for Data Mining focusing on Pattern Mining and Spurious Correlations.
The Spurious Correlation
Act 1: The Hook
Imagine you are a data analyst for a major retail chain. Your latest pattern mining algorithm has discovered a highly confident association rule: customers who buy diapers are incredibly likely to also buy beer on Friday evenings. The marketing team is thrilled and wants to immediately place the beer next to the diapers.
Socratic Prompt: The Initial Discovery
Before making this store layout change, what questions should you ask about this data? Why might two completely unrelated items show such a strong mathematical association in your transaction database?
Act 2: Mental Model
In pattern mining, algorithms like Apriori and FP-Growth are excellent at finding items that frequently co-occur. They rely on metrics such as Support, Confidence, and Lift. However, algorithms do not understand context or causality. A strong mathematical correlation does not necessarily mean one event causes the other. Sometimes, a hidden third variable (a confounding variable) influences both, or the correlation is purely coincidental - a phenomenon known as a "spurious correlation."
Socratic Prompt: Deconstructing the Metrics
If the Confidence of the rule {Diapers} -> {Beer} is 90 percent, does that mean buying diapers causes someone to buy beer? How might calculating the Lift of this rule help you determine if the association is meaningful or just a statistical artifact?
Act 3: The Rabbit Hole
You decide to dig deeper into the data. You find that the correlation disappears when you segment the data by time of day, but reappears strongly only between 5 PM and 8 PM on Fridays. You also discover that the primary demographic driving this pattern is young fathers stopping at the store on their way home from work. But wait, another analyst points out that during the same time frame, the sales of umbrellas and ice cream also spiked together in a different region.
Socratic Prompt: Evaluating Causality vs. Correlation
How do you differentiate between a correlation driven by a shared context (like young fathers preparing for the weekend) and a completely random spurious correlation (like umbrellas and ice cream)? What statistical tests or experimental designs could you propose to test whether acting on these patterns will actually increase revenue?