{ "cells": [ { "cell_type": "markdown", "id": "932c7edc-0ff4-4dfd-93f7-84fd86768ca0", "metadata": {}, "source": [ "# 🥦 梯度提升树GBDT\n", "\n", "## 壹丨集成学习与Boosting\n", "\n", "集成学习（Ensemble Learning）使用多个弱评估器，并将它们以某种方式结合起来解决一个问题\n", "\n", "集成学习三个主要研究领域：\n", "\n", "1. 模型融合。包括投票法Voting、堆叠法Stacking、混合法Blending等\n", "2. 弱分类器集成。如装袋法Bagging、提升法Boosting\n", "3. 混合专家模型。将一个复杂任务拆解成几个相对简单且更小的子任务，针对不同子任务训练个体学习器。" ] }, { "cell_type": "code", "execution_count": 1, "id": "3706c4b2-360f-4d0c-a8aa-4cd370e2a419", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sklearn: 1.3.2\n", "matplotlib: 3.7.1\n", "numpy: 1.25.0\n", "pandas: 2.0.2\n" ] } ], "source": [ "# 查看各个库版本\n", "import sklearn\n", "import matplotlib as mlp\n", "import numpy as np\n", "import pandas as pd\n", "\n", "for item in [sklearn, mlp, np, pd]:\n", " print(f'{eval(str(item).split(\" \")[1])}: {item.__version__}')" ] }, { "cell_type": "code", "execution_count": 2, "id": "d181230d-a8fc-4b0e-b743-a353da638fcf", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dict_keys(['data', 'target', 'frame', 'target_names', 'feature_names', 'DESCR'])\n" ] } ], "source": [ "# 导入加利福尼亚房价数据\n", "from sklearn.datasets import fetch_california_housing\n", "from sklearn.model_selection import train_test_split as TTS\n", "\n", "data = fetch_california_housing()\n", "print(data.keys())\n", "\n", "# 划分数据集\n", "x = data['data']\n", "y = data['target']\n", "train_x, test_x, train_y, test_y = TTS(x, y, test_size=0.3, random_state=22)" ] }, { "cell_type": "markdown", "id": "2e3f6a7d-4da6-4ff9-a07f-43768ed979ca", "metadata": {}, "source": [ "## 贰丨GBDT\n", "\n", "GBDT(GBM)是提升法Boosting的代表，逐一建立弱评估器、并且上一个弱评估其的输出结果会影响下一个弱评估器的建立。弱评估器之间高度相关。\n", "\n", ">Bagging法一次性建立多个平行独立的弱评估器，让所有评估器并行计算。如随机森林。\n", "\n", "AdaBoost确立提升法三要素：\n", "\n", "* 损失函数 L(x, y)\n", "* 弱评估器 f(x)\n", "* 综合集成规则\n", "\n", "依赖于三要素，AdaBoost还确立了基本训练流程：依据上一个弱评估器 $f(x)_{t-1}$ 的结果，计算损失函数 $L(x, y)$，并以最小化 $L(x, y)$ 为目标影响下一个弱评估器 $f(x)_t$ 的构建。 \n", "\n", "| 年份 | 算法/理论 | 成就/改进 |\n", "| ---- | ----------------------------------------- | ------------------------------------------------------------ |\n", "| 1988 | General Boosting | 先前的弱评估器上预测错误的样本应该在后续得到更多关注 |\n", "| 1997 | AdaBoost | 从0到1，首次在工程上实现Boosting理论
确立了决策树作为弱评估器，生长规则为C4.5
自适应地**调整数据分布**
自适应地**计算每个弱分类器的权重** |\n", "| 2000 | GBDT
改进ADB | 更新损失函数$L(x,y)$，在数学流程上推广到任意可微函数
放弃调整数据分布、放弃弱分类器权重
自适应地**调整后续弱评估器拟合的目标** |\n", "| 2014 | XGBoost
改进GBDT | 更新损失函数$L(x,y)$，在损失函数中加入正则项
改进弱评估器$f(x)$生长规则，自适应地**调整后续弱评估器的结构**
改进拟合中的数学过程，提升拟合精度
首次在工程上实现了Boosting树的并行，提升运算速度
支持GPU加速 |\n", "| 2016 | LGBM
改进GBDT
受XGBoost启发 | 改进弱评估器$f(x)$的生长规则，自适应地**调整后续弱评估器的结构**
设计了适合于GBDT的数据分布调整方式GOSS，自适应地**调整数据分布**
设计了加速弱分类器分枝的计算方式EFB，在工程上大幅度降低了运算时间与内存消耗
支持GPU加速 |\n", "| 2017 | CatBoost
改进GBDT
受XGBoost启发 | 改进弱评估器$f(x)$的生长规则，自适应额调整后续弱评估器的结构
设计了适用于梯度下降的弱分类器权重调整方式，自适应地**调整弱分类器的权重**
改进离散型变量的分枝方式，提升拟合精度
支持GPU加速 |\n", "\n", "Boosting算法中所有的弱评估器都是回归树，分类算法是在回归输出的基础上加上Sigmoid或Softmax函数\n", "\n", "## 叁丨GBDT相关参数\n", "\n", "| 参数相关流程 | 参数 |\n", "| -------------------- | ------------------------------------------------------------ |\n", "| 损失函数 | **`loss`**，**`alpha`** |\n", "| 集成规则 | `init`，`subsample`，**`learning_rate`** |\n", "| 弱评估器 | **`n_estimators`**，**`criterion`** |\n", "| 弱评估器（抗过拟合） | `min_samples_split`，`min_samples_leaf`，`min_weight_fraction_leaf`，**`max_depth`**，`min_impurity`，`max_features`，`max_leaf_nodes`，`ccp_alpha` |\n", "| 训练流程（结果监控） | `verbose` |\n", "| 训练流程（提前停止） | `validation_fraction`，`n_iter_no_change`，`tol` |\n", "| 训练流程（增量学习） | `warm_start` |\n", "| 随机性控制 | `random_state` |\n", "\n", "`n_estimators`：迭代次数，默认100。随着`n_estimators`增多，训练速度变慢，过拟合可能性增大\n", "\n", "当树的尺寸小的时候，不容易过拟合。计算资源足够时可以增多树的数量，并通过控制过拟合参数调整模型\n", "\n", "`learning_rate`：学习率控制每棵树增长的速度。当学习率大时，算法迭代得快，只需要较少的树就可以快速达到模型的极限附近。当学习率小时，情况完全相反。\n", "\n", "在Boosting中，一个样本在集成算法中的输出值总是等于这个样本在所有树上的输出值的加权平均\n", "\n", "\n", "$$\n", "\\hat{y_i} = \\Sigma wf(x_i)\n", "$$\n", "\n", "\n", "在迭代过程中，持续的求解输出的$\\hat{y_i}$，假设模型中有了`k`棵树，在建立第`k+1`棵新的树时，输出结果表示为：\n", "\n", "\n", "$$\n", "\\hat{y_i}^{(k+1)} = \\hat{y_i}^{(k)} + \\eta f_{k+1}(x_i)\n", "$$\n", "\n", "\n", "其中，$\\eta$就是学习率\n", "\n", "\n", "`loss`：损失函数，可指定: [`squared_error`, `absolute_error`, `huber`, `quantile`]，默认值：`squared_error`\n", "\n", "1. 平方误差：\n", "\n", "\n", "$$\n", "loss = \\Sigma(y-f(x))^2\n", "$$\n", "\n", "\n", "2. 绝对误差：\n", "\n", "\n", "$$\n", "loss =\\Sigma |y-f(x)|\n", "$$\n", "\n", "\n", "3. Huber损失：\n", "\n", "\n", "$$\n", "loss=\\Sigma L(y,f(x))\n", "$$\n", "\n", "\n", "\n", "其中,\n", "\n", "\n", "$$\n", "{\\displaystyle L_{\\delta }(y,f(x))={\\begin{cases}{\\frac {1}{2}}(y-f(x))^{2}&{\\text{for }}|y-f(x)|\\leq \\delta ,\\\\\\delta \\ \\cdot \\left(|y-f(x)|-{\\frac {1}{2}}\\delta \\right),&{\\text{otherwise.}}\\end{cases}}} \\\\ \\\\\n", "\\delta \\in (0,1)\n", "$$\n", "\n", "\n", "> GBDT必须考虑离群值的影响\n", "> Boosting算法更容易被离群值影响，也更擅长学习离群值：\n", "> 当高度关注离群值、并且希望努力将离群值预测正确时，选择MSE\n", "> 努力排除离群值的影响，更关注非离群值时，选择MAE\n", "> 平衡离群值与非离群值时，选择Huber或者Quantileloss\n", "\n", "`random_state`：随机数种子。控制随机规则的参数\n", "\n", "sklearn中的随机性流程包括：\n", "\n", "1. （强制）随机抽取每棵树建立时分枝用的特征，抽取数量由`max_features`决定\n", "2. （强制）随机排序每棵树分枝时所用的特征\n", "3. （可选）随机抽取每棵树建立时训练用的样本，抽取比例由`subsamples`决定\n", "4. （可选）随机抽样部分数据作为验证集来控制提前停止，抽取比例由`validation_fraction`决定\n", "\n", "\n", "\n", "`criterion`：树分枝时使用的不纯度衡量指标。CART树在分枝时会分为两个叶子节点，left和right，需要找到令左右节点的不纯度之和最小的分枝方式。\n", "\n", "父节点的不纯度与左右节点不纯度之和的差值，称为不纯度下降量，下降量越大，该分支对于降低不纯度的贡献越大\n", "\n", "计算不纯度下降量方法包括：弗里德曼均方误差、均方误差\n" ] }, { "cell_type": "markdown", "id": "6f1835bb-50cf-4808-b398-e81598e28314", "metadata": {}, "source": [ "### 1. `n_estimators`调参" ] }, { "cell_type": "code", "execution_count": 6, "id": "7d938bc0-efe4-4a48-b2c1-139ddaa688a3", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Estimators: 50 \t R2: 0.7602 \t Time:3.4205\n", "Estimators: 100 \t R2: 0.7929 \t Time:6.8516\n", "Estimators: 150 \t R2: 0.8056 \t Time:10.2487\n", "Estimators: 200 \t R2: 0.8144 \t Time:13.5612\n", "Estimators: 250 \t R2: 0.8182 \t Time:16.8938\n", "Estimators: 300 \t R2: 0.8229 \t Time:20.2602\n", "Estimators: 350 \t R2: 0.8266 \t Time:23.7398\n", "Estimators: 400 \t R2: 0.8287 \t Time:27.0994\n", "Estimators: 450 \t R2: 0.8305 \t Time:30.2137\n", "Estimators: 500 \t R2: 0.8317 \t Time:33.4626\n" ] } ], "source": [ "from sklearn.ensemble import GradientBoostingRegressor as GBR\n", "from time import time\n", "\n", "for n in range(50, 550, 50):\n", " start = time()\n", " reg = GBR(n_estimators=n, random_state=22)\n", " reg.fit(train_x, train_y)\n", " print(f'Estimators: {n} \\t R2: {reg.score(test_x, test_y):.4f} \\t Time:{time() - start:.4f}')" ] }, { "cell_type": "markdown", "id": "1d1bb685-c9ad-407e-abc1-8c30871326c5", "metadata": {}, "source": [ "### 2. `learning_rate`调参" ] }, { "cell_type": "code", "execution_count": 7, "id": "6487b2c9-b6e9-493b-ad2d-72457490829a", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "learning_rate: 0.01 \t R2: 0.6431 \t Time:13.6895\n", "learning_rate: 0.0575 \t R2: 0.7975 \t Time:13.5025\n", "learning_rate: 0.105 \t R2: 0.8163 \t Time:13.4170\n", "learning_rate: 0.15250000000000002 \t R2: 0.8248 \t Time:13.4009\n", "learning_rate: 0.2 \t R2: 0.8286 \t Time:13.3612\n" ] } ], "source": [ "for lr in np.linspace(0.01, 0.2, 5):\n", " start = time()\n", " reg = GBR(n_estimators=200, learning_rate=lr, random_state=22)\n", " reg.fit(train_x, train_y)\n", " print(f'learning_rate: {lr} \\t R2: {reg.score(test_x, test_y):.4f} \\t Time:{time() - start:.4f}')" ] }, { "cell_type": "markdown", "id": "b35350a7-d6e2-474e-b6ca-d984e0155b6d", "metadata": {}, "source": [ "### 3. 交叉验证" ] }, { "cell_type": "code", "execution_count": null, "id": "cbf61085-5245-43ac-9f0c-9f23c0a07be8", "metadata": {}, "outputs": [], "source": [ "from sklearn.ensemble import GradientBoostingRegressor as GBR\n", "from sklearn.model_selection import cross_val_score as CSV\n", "from sklearn.model_selection import KFold\n", "\n", "gbr = GBR(n_estimators=10, random_state=22)\n", "cv = KFold(n_splits=5, shuffle=True, random_state=22)\n", "result_gbr = CSV(gbr, train_x, train_y, cv=cv, scoring='neg_mean_squared_error')\n", "\n", "# RMSE\n", "rmse = (abs(result_gbr) ** 0.5).mean()\n", "# 方差\n", "var = (abs(result_gbr) ** 0.5).var()\n", "print(f'RMSE = {rmse}\\n'\n", " f'Var = {var}')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } }, "nbformat": 4, "nbformat_minor": 5 }