Artificial IntelligenceData MiningExercises

Data Mining Exercise 6

Data Preprocessing & Decision Tree - Sampling and Entropy calculation for Node selection

Warning

This article is a work in progress and may contain incomplete information or inaccuracies. Please verify details from reliable sources.

Data Mining Exercise 6

The foundation of Data Preprocessing (Sampling) and Decision Tree Model (Entropy calculation for Node selection).

Question 3: Difference between 2 sampling methods (Stratified vs. Simple Random)

Problem

Want to sample nn from a population with kk subgroups.

  • Method 1: Stratified Sampling (proportional selection)
  • Method 2: Simple Random Sampling (random pool, ignoring groups)

Difference

The core lies in "probability of getting representatives from all groups" and "Precision".

1. Representation Guarantee

Method 1 (Stratified):

  • 100% guarantee to get data from all kk groups according to their real proportions.
  • This method is excellent if our data is Imbalanced (e.g. contains very rare groups).

📌 Example: University Opinion Poll

There are 9,000 students and 100 professors.

If sampling via Stratified, we force a quota that professors must be included.

Method 2 (Simple Random):

  • Relies on luck. If any group kk is very small, there's a high chance we "won't sample them at all".
  • Causing that group's data to be missing (Under-represented).

Example: Simple Random Problem

From the university case, if randomly pooled, we might only get students and no professors, causing a skewed poll result.

2. Standard Error

| Sampling Method | Error Value | Reason | |------

---|----------|--------| | Stratified | Lower | Guarantees population structure. The obtained data is better distributed. | | Simple Random | Higher | Especially if each group is highly diverse. |

Conclusion

✅ Recommendation

If you know the data is already divided into groups and want each group to be represented, you must use Stratified.

(This is also the Standard for splitting Train/Test sets in Classification tasks).

Question 4: Entropy Calculation (Uncertainty value)

Data

Total 8 balls:

  • 🔴 Red: 4 balls
  • 🟢 Green: 2 balls
  • 🔵 Blue: 2 balls

Entropy Formula:

H(X)=−∑i=1np(xi)log⁡2p(xi)H(X) = - \sum_{i=1}^{n} p(x_i) \log_2 p(x_i)

Calculation Steps

1. Find the probability (pp) of each color

P(Red)=48=0.5P(\text{Red}) = \frac{4}{8} = 0.5 P(Green)=28=0.25P(\text{Green}) = \frac{2}{8} = 0.25 P(Blue)=28=0.25P(\text{Blue}) = \frac{2}{8} = 0.25

2. Substitute values into the formula

We must calculate term by term and sum them up (don't forget the minus sign at the front):

H(X)=−[(0.5log⁡20.5)+(0.25log⁡20.25)+(0.25log⁡20.25)]H(X) = - [ (0.5 \log_2 0.5) + (0.25 \log_2 0.25) + (0.25 \log_2 0.25) ]

3. Convert Log values (base 2):

log⁡2(0.5)=log⁡2(2−1)=−1\log_2(0.5) = \log_2(2^{-1}) = -1 log⁡2(0.25)=log⁡2(2−2)=−2\log_2(0.25) = \log_2(2^{-2}) = -2

4. Calculate numbers

Term Red=0.5×(−1)=−0.5Term Green=0.25×(−2)=−0.5Term Blue=0.25×(−2)=−0.5\begin{aligned} \text{Term Red} &= 0.5 \times (-1) = -0.5 \\ \text{Term Green} &= 0.25 \times (-2) = -0.5 \\ \text{Term Blue} &= 0.25 \times (-2) = -0.5 \end{aligned} Sum inside bracket=−0.5+(−0.5)+(−0.5)=−1.5H(X)=−(−1.5)H(X)=1.5 bits\begin{aligned} \text{Sum inside bracket} &= -0.5 + (-0.5) + (-0.5) = -1.5 \\ H(X) &= - (-1.5) \\ H(X) &= \mathbf{1.5} \text{ bits} \end{aligned}

Answer

✅ Result

The Entropy value is 1.5 bits.

Data Mining Connection Trivia

What is the maximum possible value (Max Entropy) for this problem?

If 3 ball colors have exactly equal amounts (equal occurrence probability), the highest Entropy value would be log⁡2(3)≈1.585\log_2(3) \approx 1.585 bits.

💡 Result Interpretation

Our calculated value of 1.5, which is almost the Max, means this dataset "has high uncertainty" (hard to guess what color will be drawn, even though Red is the most abundant).

Connection with Decision Tree

In Decision Tree:

  • We like Nodes with low Entropy (e.g. all Red = Entropy 0).
  • Because it means the data is Pure.
  • The purer it is, the more accurate the prediction!
Propose an edit on GitHub

Data Mining Exercise 5

Data Representation & Data Nature

Data Mining Exercise 7

Min-Max Normalization - Scaling data to a new range

On this page

Data Mining Exercise 6Question 3: Difference between 2 sampling methods (Stratified vs. Simple Random)ProblemDifference1. Representation Guarantee2. Standard ErrorConclusionQuestion 4: Entropy Calculation (Uncertainty value)DataEntropy Formula:Calculation Steps1. Find the probability (pp) of each color2. Substitute values into the formula3. Convert Log values (base 2):4. Calculate numbersAnswerData Mining Connection TriviaWhat is the maximum possible value (Max Entropy) for this problem?Connection with Decision Tree

AI Knowledge Assistant

สวัสดีครับ! ผมคือ AI Assistant ประจำเว็บไซต์

คุณสามารถสอบถามข้อมูลด้าน Computer Science, Business, หรือ Finance ได้เลยครับ