Warning

This article is a work in progress and may contain incomplete information or inaccuracies. Please verify details from reliable sources.

Data Mining Exercise 8

📌 Precaution

In the data table for question 5, some Class Label cells are "blank" (no + sign). I will assume those blank cells are negative class (-) to make it a Binary Class Problem according to the problem.

A	B	Class Label

-----| |T|F|+| |T|T|+| |T|T|+| |T|F|-| |T|T|+| |F|F|-| |F|F|-| |F|F|-| |T|T|-| |T|F|-|

Question 5: Selecting Attribute A or B

Data Summary

Total data count 10 rows

Class (+): 4 people (Rows 1, 2, 3, 5)
Class (-): 6 people (Rows 4, 6, 7, 8, 9, 10)

5a. Calculate Information Gain (Choose A or B?)

Entropy Formula

$H(S) = -p_+ \log_2 p_+ - p_- \log_2 p_-$

Entropy of Parent (P)

$p_+ = \frac{4}{10} = 0.4, \quad p_- = \frac{6}{10} = 0.6$

$H(P) = -(0.4 \log_2 0.4) - (0.6 \log_2 0.6) \approx \mathbf{0.971}$

Information Gain of Attribute A

A = T (7 people): has (+)4, (-)3

$H(A_T) = -(4/7 \log_2 4/7) - (3/7 \log_2 3/7) \approx 0.985$

A = F (3 people): has (+)0, (-)3 (Pure)

$H(A_F) = -(0) - (1 \log_2 1) = 0$

Weighted Entropy:

$\frac{7}{10}(0.985) + \frac{3}{10}(0) = 0.6895$

Information Gain(A):

$0.971 - 0.6895 = \mathbf{0.2815}$

Information Gain of Attribute B

B = T (4 people): has (+)3, (-)1

$H(B_T) = -(0.75 \log_2 0.75) - (0.25 \log_2 0.25) \approx 0.811$

B = F (6 people): has (+)1, (-)5

$H(B_F) = -(1/6 \log_2 1/6) - (5/6 \log_2 5/6) \approx 0.650$

Weighted Entropy:

$\frac{4}{10}(0.811) + \frac{6}{10}(0.650) = 0.3244 + 0.39 = 0.7144$

Information Gain(B):

$0.971 - 0.7144 = \mathbf{0.2566}$

Summary 5a

✅ Choose Attribute A

Because Gain(A) = 0.2815 > Gain(B) = 0.2566

5b. Calculate Gain in Gini Index (Choose A or B?)

Gini Index Formula

$\text{Gini}(S) = 1 - \sum p_i^2$

Gini of Parent (P)

$\text{Gini}(P) = 1 - [(0.4)^2 + (0.6)^2] = 1 - (0.16 + 0.36) = \mathbf{0.48}$

Gini of Attribute A

A = T:

$1 - [(4/7)^2 + (3/7)^2] = 1 - (0.327 + 0.184) \approx 0.489$

A = F:

$1 - [0^2 + 1^2] = 0$

Weighted Gini:

$\frac{7}{10}(0.489) + \frac{3}{10}(0) = \mathbf{0.3423}$

Gain Gini(A):

$0.48 - 0.3423 = \mathbf{0.1377}$

Gini of Attribute B

B = T:

$1 - [(0.75)^2 + (0.25)^2] = 1 - (0.5625 + 0.0625) = 0.375$

B = F:

$1 - [(1/6)^2 + (5/6)^2] = 1 - (0.028 + 0.694) \approx 0.278$

Weighted Gini:

$\frac{4}{10}(0.375) + \frac{6}{10}(0.278) = 0.15 + 0.1668 = \mathbf{0.3168}$

Gain Gini(B):

$0.48 - 0.3168 = \mathbf{0.1632}$

Summary 5b

✅ Choose Attribute B

Because Gain Gini(B) = 0.1632 > Gain Gini(A) = 0.1377

5c. Explanation - Can we choose different Attributes?

Answer

✅ Possible (Yes)

As seen in this example (Info Gain chose A but Gini chose B)

Reason

Information Gain (Entropy):

Tends to be biased towards Attributes that divide into the most "Pure" subgroups.
Even if that group is small (e.g. A=F only has 3 people, but 100% pure, making Entropy drop drastically).

Gini Index:

Tends to prefer divisions that result in "large and moderately pure" groups rather than small, highly pure groups.
Gini will not drop as fast as Entropy when it encounters a small Pure node.