decision-trees-random-forests

# Decision Tree Implementation import numpy as np from collections import Counter class Node: def __init__(self, feature=None, threshold=None, left=None, right=None, value=None): self.feature = feature self.threshold = threshold self.left = left self.right = right self.value = value class DecisionTree: def __init__(self, max_depth=None): self.max_depth = max_depth def _gini(self, y): classes = np.unique(y) n = len(y) gini = 1 for c in classes: p = len(y[y == c]) / n gini -= p ** 2 return gini def _information_gain(self, X, y, feature, threshold): y_parent = self._gini(y) indices_left = X[:, feature] <= threshold y_left = self._gini(y[indices_left]) y_right = self._gini(y[~indices_left]) n = len(y) n_left = np.sum(indices_left) n_right = n - n_left gain = y_parent - (n_left / n) * y_left - (n_right / n) * y_right return gain def _best_split(self, X, y): best_gain = 0 best_feature = None best_threshold = None n_features = X.shape[1] for feature in range(n_features): thresholds = np.unique(X[:, feature]) for threshold in thresholds: gain = self._information_gain(X, y, feature, threshold) if gain > best_gain: best_gain = gain best_feature = feature best_threshold = threshold return best_feature, best_threshold def _build_tree(self, X, y, depth): if depth == self.max_depth or len(np.unique(y)) == 1: return Node(value=Counter(y).most_common(1)[0][0]) feature, threshold = self._best_split(X, y) indices_left = X[:, feature] <= threshold X_left, y_left = X[indices_left], y[indices_left] X_right, y_right = X[~indices_left], y[~indices_left] left = self._build_tree(X_left, y_left, depth + 1) right = self._build_tree(X_right, y_right, depth + 1) return Node(feature=feature, threshold=threshold, left=left, right=right) def fit(self, X, y): self.root = self._build_tree(X, y, 0) def _predict_instance(self, x, node): if node.value is not None: return node.value if x[node.feature] <= node.threshold: return self._predict_instance(x, node.left) else: return self._predict_instance(x, node.right) def predict(self, X): return [self._predict_instance(x, self.root) for x in X] # Usage: # Instantiate the DecisionTree class dt = DecisionTree(max_depth=5) # Fit the model on the training data dt.fit(X_train, y_train) # Make predictions on the testing data dt_preds = dt.predict(X_test) dt_report = classification_report(y_test, dt_preds) print("Decision Tree Classification Report:") print(dt_report) # Train and evaluate the random forest model using scikit-learn's RandomForestClassifier rf = RandomForestClassifier(n_estimators=10, max_depth=5) rf.fit(X_train, y_train) rf_preds = rf.predict(X_test) rf_report = classification_report(y_test, rf_preds) print("Random Forest Classification Report:") print(rf_report)

The provided code implements a decision tree classifier. 1. The `Node` class represents a node in the decision tree. It has attributes for the feature, threshold, left child, right child, and value (which represents the predicted class label at that node). 2. The `DecisionTree` class is the main class that builds and uses the decision tree. It has an optional parameter `max_depth` to control the maximum depth of the tree. 3. The `_gini` function calculates the Gini impurity of a set of target values. It iterates over the unique classes in the target values and calculates their probabilities using a frequency count. 4. The `_information_gain` function calculates the information gain by splitting the data based on a feature and threshold. It first calculates the Gini impurity of the parent node, then calculates the Gini impurity of the left and right child nodes after the split, and finally calculates the information gain as the difference between the parent impurity and the weighted average of the child impurities. 5. The `_best_split` function finds the best feature and threshold to split the data based on the maximum information gain. It iterates over all features and thresholds, calculates the information gain for each split, and keeps track of the best split so far. 6. The `_build_tree` function recursively builds the decision tree. If the maximum depth is reached or all target values are the same, it creates a leaf node with the most common target value. Otherwise, it finds the best split using `_best_split`, splits the data into left and right subsets, and recursively builds the left and right child nodes. 7. The `fit` function initializes the decision tree by calling `_build_tree` with the training data and target values. 8. The `_predict_instance` function traverses the decision tree to predict the class label for a given instance. It starts from the root node and follows the appropriate child node based on the feature and threshold until it reaches a leaf node, which contains the predicted class label. 9. The `predict` function applies `_predict_instance` to each instance in the input data and returns a list of predicted class labels. The code also includes usage examples at the end, where it instantiates the `DecisionTree` class, fits the model on training data, makes predictions on testing data, and prints the classification report. Additionally, it trains and evaluates a random forest model using scikit-learn's `RandomForestClassifier` and prints the classification report for comparison.