Read datasets iteratively with xgboost

 Datawhale dried food  

source ：Coggle Data Science

In the process of reading and training large-scale data sets , Iteratively reading datasets is a good choice , stay Pytorch Supports iterative reading in . Next we will introduce XGBoost The way of iterative reading .

Memory data reading

class IterLoadForDMatrix(xgb.core.DataIter):
    def __init__(self, df=None, features=None, target=None, batch_size=256*1024):
        self.features = features
        self.target = target
        self.df = df
        self.batch_size = batch_size
        self.batches = int( np.ceil( len(df) / self.batch_size ) )
        self.it = 0 # set iterator to 0
        super().__init__()

    def reset(self):
        '''Reset the iterator'''
        self.it = 0

    def next(self, input_data):
        '''Yield next batch of data.'''
        if self.it == self.batches:
            return 0 # Return 0 when there's no more batch.
        
        a = self.it * self.batch_size
        b = min( (self.it + 1) * self.batch_size, len(self.df) )
        dt = pd.DataFrame(self.df.iloc[a:b])
        input_data(data=dt[self.features], label=dt[self.target]) #, weight=dt['weight'])
        self.it += 1
        return 1

Calling method （ This method is more suitable GPU Training ）：

Xy_train = IterLoadForDMatrix(train.loc[train_idx], FEATURES, 'target')
dtrain = xgb.DeviceQuantileDMatrix(Xy_train, max_bin=256)

Reference documents ：

https://xgboost.readthedocs.io/en/latest/python/examples/quantile_data_iterator.html

External data is read iteratively

class Iterator(xgboost.DataIter):
  def __init__(self, svm_file_paths: List[str]):
    self._file_paths = svm_file_paths
    self._it = 0
    super().__init__(cache_prefix=os.path.join(".", "cache"))

  def next(self, input_data: Callable):
    if self._it == len(self._file_paths):
      # return 0 to let XGBoost know this is the end of iteration
      return 0

    X, y = load_svmlight_file(self._file_paths[self._it])
    input_data(X, y)
    self._it += 1
    return 1

  def reset(self):
    """Reset the iterator to its beginning"""
    self._it = 0

Calling method （ This method is more suitable CPU Training ）：

it = Iterator(["file_0.svm", "file_1.svm", "file_2.svm"])
Xy = xgboost.DMatrix(it)

# Other tree methods including ``hist`` and ``gpu_hist`` also work, but has some caveats
# as noted in following sections.
booster = xgboost.train({"tree_method": "approx"}, Xy)

Reference documents ：

https://xgboost.readthedocs.io/en/stable/tutorials/external_memory.html

Sorting is not easy to , spot Fabulous Three even ↓