|
Title
|
|
|
|
Smart initialisation and approximating loss function for robust regression
|
|
|
Author
|
|
|
|
|
|
|
Abstract
|
|
|
|
| Two of the most common methods for robust regression are least trimmed squares (LTS) and least median of squares (LMS) regression. Both of these methods require sorting the squared residuals. Because sorting is not a differentiable operation, end-to-end optimalisation with gradient-based methods is not stable. Furthermore, existing algorithms for estimating LTS and LMS regressors rely on multiple random initial starting points. We propose and investigate two potential improvements to LTS and LMS: (1) the use of soft differentiable sorting in the loss functions and (2) deterministic initialisation of the estimators using the wrapping transformation. The first improvement aims to tackle the drawbacks of using hard sorting by introducing an alternative loss function that can be optimised using gradient based optimisation schemes, while the latter improvement aims to remove the need for multiple random initial starting points, leading to both improved accuracy as well as faster convergence. We perform an extensive experiment on both simulated and real world datasets and compare the performance of our introduced methods with well known baseline methods for robust regression. We show that the deterministic initialisation has significant benefits for LTS and LMS, both for predictive accuracy and for computational speed. The soft loss function mostly benefits LMS, as it makes it possible to apply iterative optimisation schemes to the LMS loss function. We also demonstrate the potential application of the Soft LTS loss function to non-linear regression problems using neural networks. |
|
|
|
Language
|
|
|
|
English
|
|
|
Source (journal)
|
|
|
|
Information sciences. - New York
|
|
|
Publication
|
|
|
|
New York
:
2023
|
|
|
ISSN
|
|
|
|
0020-0255
|
|
|
DOI
|
|
|
|
10.1016/J.INS.2023.119715
|
|
|
Volume/pages
|
|
|
|
651
(2023)
, p. 1-13
|
|
|
Article Reference
|
|
|
|
119715
|
|
|
ISI
|
|
|
|
001087136000001
|
|
|
Medium
|
|
|
|
E-only publicatie
|
|
|
Full text (Publisher's DOI)
|
|
|
|
|
|
|
Full text (open access)
|
|
|
|
|
|
|
Full text (publisher's version - intranet only)
|
|
|
|
|
|