Decision Trees and Random Forests

1. Simple Example Dataset

We use an extremely small dataset so that every decision can be understood step by step.

TemperatureRainGo Outside
HighNoGo
LowYesStay
HighYesStay

2. Decision Tree: Intuition First

A decision tree is a sequence of yes/no questions, just like human reasoning.

2.1 Human Reasoning

  1. Is it raining?

This single rule correctly classifies all examples.

2.2 Decision Tree as Code

if rain == "Yes":
    decision = "Stay"
else:
    decision = "Go"

This is essentially what a trained decision tree learns.

3. Decision Tree in Python (scikit-learn)

import pandas as pd
from sklearn.tree import DecisionTreeClassifier

X = pd.DataFrame({
    "temperature": ["High", "Low", "High"],
    "rain": ["No", "Yes", "Yes"]
})

y = ["Go", "Stay", "Stay"]

# One-hot encoding
X_encoded = pd.get_dummies(X)

model = DecisionTreeClassifier(random_state=0)
model.fit(X_encoded, y)

What the model learned

4. From Decision Tree to Random Forest

A Random Forest is:

Multiple decision trees trained with randomness, combined by majority voting.

5. Where Does the Randomness Come From?

Each tree is trained using:

  1. Bootstrap sampling: random sampling of rows (with replacement)
  2. Random feature selection: only a subset of features is considered at each split

This makes each tree slightly different.

6. Example Trees in the Forest

Even with the same data, trees may look different.

7. Majority Voting

For a new input:

Temperature = High
Rain = Yes

Predictions:

Final prediction:

Stay

8. Random Forest in Python

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=3,
    random_state=0
)

rf.fit(X_encoded, y)

Prediction Example

test = pd.DataFrame({
    "temperature": ["High"],
    "rain": ["Yes"]
})

test_encoded = pd.get_dummies(test)
test_encoded = test_encoded.reindex(columns=X_encoded.columns, fill_value=0)

rf.predict(test_encoded)

9. Why Random Forest Is Better Than a Single Tree

ModelCharacteristic
Decision TreeEasy to overfit
Random ForestMore stable and robust

Reason:

10. Mental Model

Accuracy comes from diversity + voting, not smarter trees.

11. Step-by-Step Walkthrough (Very Detailed)

11.1 Enumerating All Possible Rules

With two binary features, a decision tree can consider rules such as:

The algorithm evaluates all possible splits and selects the one that maximizes information gain.

In this dataset, splitting by Rain results in perfectly pure nodes.

11.2 Entropy Intuition (No Math)

Entropy becomes zero. This is why Rain is chosen.

12. Decision Tree as Explicit Rules

The trained tree can be written explicitly:

IF Rain = Yes:
    Stay
ELSE:
    Go

This transparency is a key strength of decision trees.

13. Random Forest: Same Data, Different Views

Although the dataset is small, Random Forest still applies randomness.

13.1 Bootstrap Sampling Example

Tree 1 may see:

Tree 2 may see:

Rows can be duplicated or omitted.

13.2 Feature Subsampling Example

At each split, the tree may only consider:

This forces diversity among trees.

14. Random Forest Prediction: Internal Process

For one input:

Temperature = High
Rain = Yes

Each tree:

  1. Applies its own rules
  2. Outputs a prediction

The forest:

15. Bias–Variance Perspective (Intuition)

Lower variance leads to better generalization.

16. Practical Notes

17. Common Misunderstandings

18. Exercises

  1. What happens if you remove the Temperature feature?
  2. Create a dataset where Temperature becomes the first split.
  3. Increase n_estimators and observe prediction stability.

19. Key Takeaways