Mdtryhht

I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one.

I want to know in this case which metric should I use it and WHY?

• Accuracy ??




• AUC Test value ??




• RMSE ??




• LogLoss ??

Depends.

The first thing that has to be clear is that you are running an experiment, which means you need to measure both with the same metric.

Which one? Depends on which underlying problem you are solving, if what you are doing is to determine which algorithm is better, your conclusion will only be applicable to your specific dataset

• Accuracy: Is possible to measure the accuracy as the comparing metric, but it becomes trivial if your dataset is unbalanced, which means you have much more positives than negatives or viceversa. The accuracy is used when the dataset is balanced and if is equally bad to have a mistake on positives and negatives. Also, it has the problem of being too dependant on the threshlod for defining positives/negatives.




• Area Under the Curve: The AUC is one of the most robust metrics for measuring the ability of a model to separate positives and negatives, is unsensitive to threshold and immune to unbalanceness. I would use this.




• RMSE: I only know RMSE for continuous regression, not for classification.




• LogLoss: Its use is in multinomial classification

I would say AUC is the best overall metric for classification but does not have to be the only metric, accuracy is useful too. For reference you can check this Quora regarding accuracy vs. AUC:

They both measure different things, so they are complementary.

Accuracy: Measures, for a given threshold, the percentage of points
correctly classified, regardless of which class they belong to.

AUC: Measures the likelihood that given two random points — one from
the positive and one from the negative class — the classifier will
rank the point from the positive class higher than the one from the
negative one (it measures the performance of the ranking really).

Log loss can also be a good candidate as an overall metric, why can be read here from FastAI:

Log Loss vs Accuracy

Accuracy is the count of predictions where your predicted value equals
the actual value. Accuracy is not always a good indicator because of
its yes or no nature.

Log Loss takes into account the uncertainty of
your prediction based on how much it varies from the actual label.
This gives us a more nuanced view into the performance of our model.

RMSE on the other hand is a regression metric and should not be used for classification.