Data Mining Exercise 2
An overview of Data Mining (DM), its techniques, applications, and challenges.
Warning
This article is a work in progress and may contain incomplete information or inaccuracies. Please verify details from reliable sources.
Data Mining Exercise 2
📥 Simulation Scenario
Scenario: We are measuring the "Heart Rate" of 5 people using a Smartwatch.
💓 Normal Behavior
General people while resting have a heart rate of about 60-100 bpm (beats per minute)
📊 Dataset:
| ID | Observed Value | True Value | Context | Status | | --- | ---
-------------------------- | --------------------- | ------------------------------------- | ----------------------------------- | | P1 | 72 | 72 | Normal resting | ✅ Normal Data | | P2 | 75 | 70 | Resting but moved arm a bit (+5 error) | ⚠️ Normal + Small Noise | | P3 | 185 | 185 | Running away from a dog (actually tired) | 🔵 Valid Outlier (True high value) | | P4 | 350 | 70 | Sensor malfunction (Random Error) | ❌ Noise as Outlier (Error spike) | | P5 | 70 | 180 | Running but watch is loose (-110 error) | 🔴 Valid Data masked by Noise |
1. 🔍 Differentiating between Noise and Outliers
📢 Noise
Definition
Noise is the "Error" or Random variance of data. It is the excess that we do not want. It distorts the true value.
In the table: It is the +5 of P2, the 350 of P4, and the -110 of P5.
🎯 Outliers
Definition
Outliers are "Data Objects" that have characteristics clearly different from the majority of the data group (based on the measured values).
In the table: It is P3 (185) and P4 (350) because they jump far away from the resting group (60-100).
1.1 Is Noise interesting or useful? Answer: Generally "not interesting and not useful" from a Data Analysis perspective.
Explain from data: Look at P2 (75) and P4 (350). The Error that occurred (+5 or +280) does not tell customer behavior or actual health. It is just an obstacle that makes our analysis harder.
Exception: Noise is only interesting if you are a "Sensor design engineer" who wants to study Noise to improve device quality.
1.2 ❓ Are Outliers interesting or useful?
Answer: "Can be both" (depending on context)
🔴 Case where it's not useful:
P4 (350): Is an Outlier caused by Noise → Not useful (must be removed)
✅ Case where it's useful:
P3 (185): Is an Outlier that is a true value (running from a dog) → Very interesting and useful
🎯 This is the heart of Data Mining!
In some types of work, Outliers are what we are looking for:
- 🔐 Fraud Detection: Unusually high credit card swipes (Outlier) are what we look for.
- 🏥 Medical: Unusually high blood values can indicate disease.
1.3 ❓ Can objects that are Noise be Outliers?
Answer: "Yes"
Explain from data:
- Look at P4 (
350) - True value is
70but Noise (sensor error) caused the value to shoot up to350 - This
350is considered an Outlier compared to friends
1.4 ❓ Are objects that are Noise always Outliers?
Answer: "Not always" (Small Noise does not make an Outlier)
Explain from data:
- Look at P2 (
75) - True value is
70but has+5Noise added - The data is
75which is still in normal range (60-100) - P2 has Noise but P2 is not an Outlier
1.5 ❓ Are Outliers always objects that are Noise?
Answer: "Not always" (Valid Outliers exist)
Explain from data:
- Look at P3 (
185) - This value jumps out from the group a lot (is an Outlier)
- But it is not Noise because the heart is actually beating fast (Valid Data)
1.6 ❓ Can Noise make correct data become abnormal data? Or make abnormal data become normal data?
Answer: "Both can happen" This is the scariness of Noise!
🔴 Case A: Makes correct data → Become abnormal data (Normal → Abnormal/Outlier)
Look at P4:
Original: Resting value 70 (Normal)
+
Noise: Sensor malfunction
=
Result: 350 (Abnormal/Outlier)⚠️ Consequence
Causes a False Alarm (Watch warns of heart attack even though just sitting still)
🔵 Case B: Makes abnormal data → Become normal data (Abnormal/Outlier → Normal)
Look at P5:
Original: Running tired 180 (Abnormal/Outlier)
+
Noise: Watch loose, cannot catch (-110)
=
Result: 70 (Normal)⚠️ Consequence
Causes a Missed Detection (Patient has severe arrhythmia but watch says normal, so doctor misdiagnoses)
📝 Key Takeaway
| Concept | Definition | | ------
---- | ------------------------------------------------------------------------------------ | | Noise | "Error component" inserted into data values | | Outlier | "Data point status" that is far away from friends |
🔄 In KDD Process:
-
We try to eliminate Noise
- In the Preprocessing step
- Such as using Smoothing to clean data
-
Then we decide on the remaining Outliers
- E.g. Is P3 garbage to be thrown away?
- Or is it gold (Knowledge) that we need to mine? 💎