Data Mining Exercise 5

Testing understanding of Data Representation and Data Nature.

Question 1: Test Analysis using Association Analysis

Scenario

An educational psychologist wants to find relationships like "Children who answered question 1 incorrectly with option A often answered question 5 incorrectly with option C." (To analyze which Distractors affect each other).

1) How should the data be transformed to be appropriate?

Raw Data: Usually stored as [Student_ID, Q1, Q2, ..., Q100] where values inside are 'A', 'B', 'C', 'D'.

Problem: Algorithms for Association Analysis (like Apriori) require Binary (0/1) data indicating "whether this item is in the basket or not".

Transformation Example

Suppose there are only 2 questions (Q1, Q2) and options A, B.

| Raw Data | Q1_A | Q1_B | Q2_A | Q2_B | |------

-----------|------|------|------|------| | Stu. 1 answers: A, B | 1 | 0 | 0 | 1 | | Stu. 2 answers: B, B | 0 | 1 | 0 | 1 |

Explanation: We need to create virtual "Items", specifying clearly "Which question + which choice chosen". If a child chooses that choice, put 1 (in basket). If not chosen, put 0.

2) What type and size are the resulting Attributes?

Type: Asymmetric Binary Attributes

• Binary: Because values are only 0 or 1.
• Asymmetric: Because we only care about 1 (the fact the child "chose" to answer that question). We don't care about 0 (what the child didn't choose), just like market basket analysis where we only care about items put in the basket.

Size/Dimensionality: 400 Attributes

Calculation: 100 questions × 4 choices = 400 columns (Items).

Question 2: Correlation between "Rainfall" vs "Temperature" (Daily)

Question

Between adjacent days (Day t and Day t+1), which dataset has a higher Correlation?

Answer

Reason (Why)

1. Nature of Data (Continuity vs. Sporadic)

Temperature:

• Highly Continuous.
• Physically, weather has Thermal Inertia.
• If today is hot at 35°C, tomorrow will likely be 34-36°C.
• Temperature will not jump from 35°C down to 10°C in one day unless there is a really severe storm.
• The graph is a Smooth changing wave.

Rainfall:

• Highly Sporadic/Stochastic.
• Today might have heavy rain of 100mm, tomorrow might be blazing sunny (0mm).
• Rainfall data is full of 0s and has Spiky jumping bars.

2. Autocorrelation

• Temperature: The Correlation value of tomorrow's temperature compared to today will be highly positive (approaching 1).
• Rainfall: The Correlation value will be much lower because rain today does not guarantee rain tomorrow.

Summary from a Data Mining Perspective

| Data Type | Characteristic | Prediction | |------

--------|--------|----------| | Temperature | Easier to predict (Predictable) | Can use Time Series Analysis from past data | | Rainfall | More often an Outlier/Event-based | Must use other factors (Humidity, Air pressure). Past rainfall alone might not be enough |