####
DART
####
XGBoost mostly combines a huge number of regression trees with a small learning rate.
In this situation, trees added early are significant and trees added late are unimportant.

Vinayak and Gilad-Bachrach proposed a new method to add dropout techniques from the deep neural net community to boosted trees, and reported better results in some situations.

This is a instruction of the dropout mode for tree models. Dropout is controlled by
parameters like ``rate_drop``. The legacy ``dart`` booster name remains available for
compatibility.

**************
Original paper
**************
Rashmi Korlakai Vinayak, Ran Gilad-Bachrach. "DART: Dropouts meet Multiple Additive Regression Trees." [`PMLR <http://proceedings.mlr.press/v38/korlakaivinayak15.pdf>`_, `arXiv <https://arxiv.org/abs/1505.01866>`_].

********
Features
********
- Drop trees in order to solve the over-fitting.

  - Trivial trees (to correct trivial errors) may be prevented.

Because of the randomness introduced in the training, expect the following few differences:

- Training can be slower than ``gbtree`` because the random dropout prevents usage of the prediction buffer.
- The early stop might not be stable, due to the randomness.

************
How it works
************
- In :math:`m`-th training round, suppose :math:`k` trees are selected to be dropped.
- Let :math:`D = \sum_{i \in \mathbf{K}} F_i` be the leaf scores of dropped trees and :math:`F_m = \eta \tilde{F}_m` be the leaf scores of a new tree.
- The objective function is as follows:

.. math::

  \mathrm{Obj}
  = \sum_{j=1}^n L \left( y_j, \hat{y}_j^{m-1} - D_j + \tilde{F}_m \right)
  + \Omega \left( \tilde{F}_m \right).

- :math:`D` and :math:`F_m` are overshooting, so using scale factor

.. math::

  \hat{y}_j^m = \sum_{i \not\in \mathbf{K}} F_i + a \left( \sum_{i \in \mathbf{K}} F_i + b F_m \right) .

**********
Parameters
**********

Dropout uses the same tree parameters as ``gbtree``, such as ``eta``, ``gamma``,
``max_depth``, and others.

Additional parameters are noted below:

* ``sample_type``: type of sampling algorithm.

  - ``uniform``: (default) dropped trees are selected uniformly.
  - ``weighted``: dropped trees are selected in proportion to weight.

* ``normalize_type``: type of normalization algorithm.

  - ``tree``: (default) New trees have the same weight of each of dropped trees.

  .. math::

    a \left( \sum_{i \in \mathbf{K}} F_i + \frac{1}{k} F_m \right)
    &= a \left( \sum_{i \in \mathbf{K}} F_i + \frac{\eta}{k} \tilde{F}_m \right) \\
    &\sim a \left( 1 + \frac{\eta}{k} \right) D \\
    &= a \frac{k + \eta}{k} D = D , \\
    &\quad a = \frac{k}{k + \eta}

  - ``forest``: New trees have the same weight of sum of dropped trees (forest).

  .. math::

    a \left( \sum_{i \in \mathbf{K}} F_i + F_m \right)
    &= a \left( \sum_{i \in \mathbf{K}} F_i + \eta \tilde{F}_m \right) \\
    &\sim a \left( 1 + \eta \right) D \\
    &= a (1 + \eta) D = D , \\
    &\quad a = \frac{1}{1 + \eta} .

* ``rate_drop``: dropout rate.

  - range: [0.0, 1.0]

* ``skip_drop``: probability of skipping dropout.

  - If a dropout is skipped, new trees are added in the same manner as gbtree.
  - range: [0.0, 1.0]

*************
Sample Script
*************

.. code-block:: python

  import xgboost as xgb
  # read in data
  dtrain = xgb.DMatrix('demo/data/agaricus.txt.train?format=libsvm')
  dtest = xgb.DMatrix('demo/data/agaricus.txt.test?format=libsvm')
  # specify parameters via map
  param = {'max_depth': 5, 'learning_rate': 0.1,
           'objective': 'binary:logistic',
           'sample_type': 'uniform',
           'normalize_type': 'tree',
           'rate_drop': 0.1,
           'skip_drop': 0.5}
  num_round = 50
  bst = xgb.train(param, dtrain, num_round)
  preds = bst.predict(dtest)
