{ "cells": [ { "cell_type": "markdown", "id": "ad711fea-0c25-403a-8589-dacfe1a119b8", "metadata": {}, "source": [ "# 🤦🏻 用法" ] }, { "cell_type": "code", "execution_count": 1, "id": "10c58809-b2a4-413d-b3b3-427a385c0502", "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import fetch_california_housing\n", "from sklearn.model_selection import train_test_split\n", "\n", "X, Y = fetch_california_housing(return_X_y=True, as_frame=True)\n", "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2)" ] }, { "cell_type": "code", "execution_count": 2, "id": "74828f29-023a-4113-b32b-808604cd91ff", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[iter 0] loss=1.5670 val_loss=0.0000 scale=1.0000 norm=1.1092\n", "[iter 100] loss=1.1400 val_loss=0.0000 scale=2.0000 norm=1.5566\n", "[iter 200] loss=0.9289 val_loss=0.0000 scale=1.0000 norm=0.7054\n", "[iter 300] loss=0.7743 val_loss=0.0000 scale=1.0000 norm=0.6778\n", "[iter 400] loss=0.7006 val_loss=0.0000 scale=1.0000 norm=0.6768\n" ] } ], "source": [ "from ngboost import NGBRegressor\n", "\n", "ngb = NGBRegressor().fit(X_train, Y_train)\n", "Y_preds = ngb.predict(X_test)\n", "Y_dists = ngb.pred_dist(X_test)" ] }, { "cell_type": "code", "execution_count": 3, "id": "eee6b734-6f39-4279-8cd5-306d85f36bb9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test MSE 0.31496311191332554\n", "Test NLL 0.7043259666977987\n" ] } ], "source": [ "from sklearn.metrics import mean_squared_error\n", "\n", "# 计算mse\n", "test_MSE = mean_squared_error(Y_preds, Y_test)\n", "print('Test MSE', test_MSE)\n", "\n", "# 计算负似然对数\n", "test_NLL = -Y_dists.logpdf(Y_test).mean()\n", "print('Test NLL', test_NLL)" ] }, { "cell_type": "code", "execution_count": 4, "id": "500c77a4-6b9e-4907-aa5e-7aaa587bc03c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'loc': array([1.42271054, 2.86045558, 2.14404235, 2.06860307, 2.14030303]),\n", " 'scale': array([0.31714014, 0.51342792, 0.63051949, 0.55790625, 0.37466855])}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 获取一组点的估计分布参数\n", "Y_dists[0:5].params" ] }, { "cell_type": "markdown", "id": "818b7f93-9764-4117-a855-5848b34b28aa", "metadata": {}, "source": [ "NGBoost可以与多种分布一起使用。包括回归分布和分类分布\n", "\n", "回归分布：通过`NGBRegressor()`构造函数传递`Dist`参数，默认是`Normal`\n", "\n", "| 分布 | 参数 | 评分指标 | 参考 |\n", "| ------------- | -------------- | ---------------------- | ------------------------------------------------------------ |\n", "| `Normal` | `loc`，`scale` | `LogScore`，`CRPScore` | [`scipy.stats.norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) |\n", "| `LogNormal` | `s`，`scale` | `LogScore`，`CRPScore` | [`scipy.stats.lognorm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.lognorm.html) |\n", "| `Exponential` | `scale` | `LogScore`，`CRPScore` | [`scipy.stats.expon`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.expon.html) |" ] }, { "cell_type": "code", "execution_count": 5, "id": "95a3f9b8-6aa3-4c70-a526-f00ffb8c4eb3", "metadata": {}, "outputs": [], "source": [ "from ngboost.distns import Exponential,Normal\n", "\n", "X_reg_train, X_reg_test, Y_reg_train, Y_reg_test = train_test_split(X, Y, test_size=0.2)\n", "\n", "ngb_norm = NGBRegressor(Dist=Normal, verbose=False).fit(X_reg_train, Y_reg_train)\n", "ngb_exp = NGBRegressor(Dist=Exponential, verbose=False).fit(X_reg_train, Y_reg_train)" ] }, { "cell_type": "markdown", "id": "1e6038ed-e727-4c1b-b6e1-e0dc3dd01059", "metadata": {}, "source": [ "两种预测方法：\n", "1. `predict()`，返回点预测\n", "2. `pred_dist()`，返回分布对象" ] }, { "cell_type": "code", "execution_count": 6, "id": "2632a7fc-e3cf-4e2d-9b36-9cb9b9638643", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'scale': array([4.10204883, 3.67173243, 1.88355812, 3.62738977, 1.79807525])}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ngb_norm.predict(X_reg_test)[0:5]\n", "ngb_exp.predict(X_reg_test)[0:5]\n", "ngb_exp.pred_dist(X_reg_test)[0:5].params" ] }, { "cell_type": "markdown", "id": "480ec668-e1bd-4868-9409-56eb965f00d7", "metadata": {}, "source": [ "NGBoost使用删失似然作为评分规则来处理右删失数据，使得NGBoost能够估计生存时间分布。\n", "\n", "目前(2024.08)已经实现`LogNormal`、`Exponential`的右删失版本\n", "\n", "> 右删失数据：知道某个时间发生在时间点后，但是不知道确切的时间。如：一项研究跟踪患者5年，一些患者在跟踪结束时依然活着，那么这些患者的生存时间是右删失的" ] }, { "cell_type": "code", "execution_count": 7, "id": "598685c0-8c87-44ee-a0e7-78e3fea2ad35", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[iter 0] loss=0.9572 val_loss=0.0000 scale=4.0000 norm=2.4033\n", "[iter 100] loss=0.3118 val_loss=0.0000 scale=2.0000 norm=0.5934\n", "[iter 200] loss=0.0546 val_loss=0.0000 scale=4.0000 norm=1.0242\n", "[iter 300] loss=-0.1323 val_loss=0.0000 scale=2.0000 norm=0.4097\n", "[iter 400] loss=-0.2071 val_loss=0.0000 scale=1.0000 norm=0.1794\n" ] } ], "source": [ "import numpy as np\n", "from ngboost import NGBSurvival\n", "from ngboost.distns import LogNormal\n", "\n", "X_surv_train, X_surv_test, Y_surv_train, Y_surv_test = train_test_split(X, Y, test_size=0.2)\n", "\n", "# 引入管理审查模拟生存数据\n", "T_surv_train = np.minimum(Y_surv_train, 3) # 事件或审查的时间\n", "E_surv_train = Y_surv_train > 3 # 如果是事件时间则为1，如果是审查时间则为0\n", "\n", "ngb = NGBSurvival(Dist=LogNormal).fit(X_surv_train, T_surv_train, E_surv_train)" ] }, { "cell_type": "code", "execution_count": 8, "id": "b4fb7263-edf7-40ad-83fb-f01605dfa994", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([2.34018201, 3.05263929, 3.14951764, ..., 2.29411069, 2.97176873,\n", " 2.85810047])" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ngb.predict(X_surv_test)" ] }, { "cell_type": "markdown", "id": "5a7d46f3-81fa-470d-946a-015943bee887", "metadata": {}, "source": [ "### 2. 分类分布\n", "\n", "分类分布：通过`NGBClassifier()`构造函数，并通过`Dist`参数传入分类分布，默认是`Bernoulli`，相当于`k_categorical(2)`\n", "\n", "| 分布 | 参数 | 评分指标 | 参考 |\n", "| ------------------ | --------------------- | ---------- | ------------------------------------------------------------ |\n", "| `k_categorical(K)` | `p0`，`p1`...`p{K-1}` | `LogScore` | [Categorical distribution on Wikipedia](https://en.wikipedia.org/wiki/Categorical_distribution) |\n", "| `Bernoulli` | `p` | `LogScore` | [Bernoulli distribution on Wikipedia](https://en.wikipedia.org/wiki/Bernoulli_distribution) |\n", "\n", "`NGBClassifier`对象有三种预测方法：\n", "\n", "* `predict()`：返回最可能的类\n", "* `predict_proba()`：返回类概率\n", "* `pred_dist()`：返回分布对象" ] }, { "cell_type": "code", "execution_count": 9, "id": "c1ce8ffc-2d25-481e-9aba-cdfbd4374b66", "metadata": {}, "outputs": [], "source": [ "from ngboost import NGBClassifier\n", "from ngboost.distns import k_categorical, Bernoulli\n", "from sklearn.datasets import load_breast_cancer\n", "\n", "X, y = load_breast_cancer(return_X_y=True,as_frame=True)\n", "y[0:15] = 2 # 手动将2分类问题修改成3分类问题对应下面的`Dist=k_categorical(3)`\n", "X_cls_train, X_cls_test, Y_cls_train, Y_cls_test = train_test_split(X, y, test_size=0.2)\n", "\n", "ngb_cat = NGBClassifier(Dist=k_categorical(3), verbose=False)\n", "_ = ngb_cat.fit(X_cls_train, Y_cls_train) # Y should have only 3 values: {0,1,2}" ] }, { "cell_type": "code", "execution_count": 10, "id": "d140dabd-599b-436e-8f35-f060dcaa589e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'p0': array([0.0045647 , 0.99269303, 0.00541314, 0.00409154, 0.09781032]),\n", " 'p1': array([0.99517851, 0.00686771, 0.99302614, 0.99553533, 0.90158408]),\n", " 'p2': array([0.00025679, 0.00043926, 0.00156072, 0.00037313, 0.00060561])}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ngb_cat.predict(X_cls_test)[0:5]#%%\n", "ngb_cat.predict_proba(X_cls_test)[0:5]\n", "ngb_cat.pred_dist(X_cls_test)[0:5].params" ] }, { "cell_type": "markdown", "id": "50f279b0-0977-4762-ac4a-a674b59f7e43", "metadata": {}, "source": [ "## 评价分数\n", "\n", "`NGBoost`支持`LogScore`(对数分数，也称负对数似然)和`CRPScore`，由构造函数中的`Score`参数指定" ] }, { "cell_type": "code", "execution_count": 11, "id": "933949fc-6cfd-4a7a-a1d5-b3d12a208db6", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
NGBClassifier(random_state=RandomState(MT19937) at 0x1BAC684D540, verbose=False)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "NGBClassifier(random_state=RandomState(MT19937) at 0x1BAC684D540, verbose=False)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from ngboost.scores import LogScore, CRPScore\n", "\n", "NGBRegressor(Dist=Exponential, Score=CRPScore, verbose=False).fit(X_reg_train, Y_reg_train)\n", "NGBClassifier(Dist=k_categorical(3), Score=LogScore, verbose=False).fit(X_cls_train, Y_cls_train)" ] }, { "cell_type": "markdown", "id": "29313c99-ff38-4c7e-9025-641836454157", "metadata": {}, "source": [ "## 基学习器\n", "\n", "`NGBoost`可以与任何`sklearn`回归器使用作为基础学习器，由构造函数中`Base`参数指定。默认是深度为3的回归树" ] }, { "cell_type": "code", "execution_count": 12, "id": "5fdb39ba-cfa0-47a6-a5bd-f3ccd47dd84e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[iter 0] loss=0.5616 val_loss=0.0000 scale=2.0000 norm=0.9975\n", "[iter 100] loss=0.4483 val_loss=0.0000 scale=2.0000 norm=0.4613\n", "[iter 200] loss=0.4358 val_loss=0.0000 scale=2.0000 norm=0.3779\n", "[iter 300] loss=0.4329 val_loss=0.0000 scale=1.0000 norm=0.1789\n", "[iter 400] loss=0.4308 val_loss=0.0000 scale=1.0000 norm=0.1714\n" ] }, { "data": { "text/html": [ "
NGBSurvival(Base=DecisionTreeRegressor(criterion='friedman_mse', max_depth=5),\n",
       "            Dist=<class 'ngboost.distns.utils.SurvivalDistnClass.<locals>.SurvivalDistn'>,\n",
       "            Score=<class 'ngboost.scores.CRPScore'>,\n",
       "            random_state=RandomState(MT19937) at 0x1BAC684D540)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "NGBSurvival(Base=DecisionTreeRegressor(criterion='friedman_mse', max_depth=5),\n", " Dist=.SurvivalDistn'>,\n", " Score=,\n", " random_state=RandomState(MT19937) at 0x1BAC684D540)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.tree import DecisionTreeRegressor\n", "\n", "learner = DecisionTreeRegressor(criterion='friedman_mse', max_depth=5)\n", "NGBSurvival(Dist=Exponential,Score=CRPScore,Base=learner,verbose=True).fit(X_surv_train,T_surv_train,E_surv_train)" ] }, { "cell_type": "markdown", "id": "9eafef2a-f0d7-4010-8dff-93409b21d49f", "metadata": {}, "source": [ "## 其他参数\n", "\n", "构造函数中配置：\n", "\n", "`learning_rate`：学习率\n", "\n", "`n_estimators`：估计器数量\n", "\n", "`minibatch_frac`：小批量分数\n", "\n", "`col_sample`：列子采样\n", "\n", "`fit`函数中配置：\n", "\n", "`sample_weight`：样本权重" ] }, { "cell_type": "code", "execution_count": 13, "id": "db0f5b78-2961-4dfa-8da9-87eddee73521", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[iter 0] loss=1.5589 val_loss=0.0000 scale=1.0000 norm=1.1022\n" ] }, { "data": { "text/html": [ "
NGBRegressor(col_sample=0.5, minibatch_frac=0.5, n_estimators=100,\n",
       "             random_state=RandomState(MT19937) at 0x1BAC684D540)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "NGBRegressor(col_sample=0.5, minibatch_frac=0.5, n_estimators=100,\n", " random_state=RandomState(MT19937) at 0x1BAC684D540)" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ngb = NGBRegressor(n_estimators=100, learning_rate=0.01,\n", " minibatch_frac=0.5, col_sample=0.5)\n", "ngb.fit(X_reg_train, Y_reg_train)" ] }, { "cell_type": "code", "execution_count": 14, "id": "911d0537-1074-4f74-84f7-c71a01365b37", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[iter 0] loss=1.5539 val_loss=0.0000 scale=1.0000 norm=1.0987\n" ] }, { "data": { "text/html": [ "
NGBRegressor(col_sample=0.5, minibatch_frac=0.5, n_estimators=100,\n",
       "             random_state=RandomState(MT19937) at 0x1BAC684D540)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "NGBRegressor(col_sample=0.5, minibatch_frac=0.5, n_estimators=100,\n", " random_state=RandomState(MT19937) at 0x1BAC684D540)" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ngb = NGBRegressor(n_estimators=100, learning_rate=0.01,\n", " minibatch_frac=0.5, col_sample=0.5)\n", "weights = np.random.random(Y_reg_train.shape)\n", "ngb.fit(X_reg_train, Y_reg_train, sample_weight=weights)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } }, "nbformat": 4, "nbformat_minor": 5 }