Mdtryhht

I made some search to learn precision and recall and I saw some graphs represents inverse relationship between precision and recall and I started to think about it to clarify subject. I wonder the inverse relationship always hold? Suppose I have a binary classification problem and there are positive and negative labeled classes. After training some of the actual positive examples are predicted as true positives and some of them false negatives and some of the actual negative examples are predicted as true negatives and some of them false positives. To calculate precision and recall I use these formulas:
$$Precision = fracTPTP + FP$$ and $$Recall = fracTPTP + FN$$ If I decrease false negatives then true positives increases and in that case don't precision and recall both increase?

Thanks for clear statement of the problem. The point is that if you want to decrease false negatives, you should sufficiently lower the threshold of your decision function. If the false negatives are decreased, as you mentioned, true positives increase but false positives can also increase. As a result, recall will increase and precision will decrease.

You are correct @Tolga, both can increase at the same time. Consider the following data:

Prediction | True Class
 1.0 | 0
 0.5 | 1
 0.0 | 0

If you set your cut off point as 0.75, then you have

$$ Precision = fracTPTP + FP = frac00 + 1 = 0 $$
$$ Recall = fracTPTP + FN = frac00 + 1 = 0$$

then if you decrease your cut off point to 0.25, you have

$$ Precision = fracTPTP + FP = frac11 + 1 = 0.5 $$
$$ Recall = fracTPTP + FN = frac11 + 0 = 1$$

and so you can see, both precision and recall increased when we decreased the number of False Negatives.

If we decrease the false negative, recall always increases, but precision may increase or decrease which I will illustrate.

Recall

If we denote $FN$ as $x$, then
$$TP = P - x$$
therefore, recall would be
$$r = fracP-xP-x+x = 1- fracxP$$
which always increases by decrease in $x$.

Precision

For precision, the relation is not as straightforward. Lets start with two examples.

First case: decrease in precision, by decrease in false negative:

label model prediction
1 0.8
0 0.2
0 0.2
1 0.2

For threshold $0.5$ (false negative = $(1, 0.2)$),

$$p = frac11+0=1$$

For threshold $0.0$ (false negative = $$),

$$p = frac22+2=0.5$$

Second case: increase in precision, by decrease in false negative (the same as @kbrose example):

label model prediction
0 1.0
1 0.4
0 0.1

For threshold $0.5$ (false negative = $(1, 0.4)$),

$$p = frac00+1=0$$

For threshold $0.0$ (false negative = $$),

$$p = frac11+2=0.33$$

Analysis of precision

We first assume that

If we decrease $x:=FN$ by lowering the threshold (labeling more instances as positive),
$FP$ will increase toward $N$ as well, meaning more negatives will be labeled positive,

which is a realistic assumption, and then proceed to approximate the $FN$-$FP$ relation with rate $a > 0$ as
$$FP = N - ax,$$
which works for a short interval like $[x_0, x_0+Delta x)$, meaning rate $a$ can change by $x$. Then, precision would be
$$p = fracP-xP-x+N-ax = fracP-xP+N-(1+a)x$$
and its derivative w.r.t. $x$ would be
$$fracdpdx = fracaP-N(P+N-(1+a)x)^2$$
which is negative when $aP-N < 0$. From here, we are working with a smooth, continuous $x$.

Therefore, by decreasing $x$, recall always increases, but
precision increases only if $a < N/P$.

Relation to ROC curve

This section gives a good intuition about the relation. ROC curve in terms of previous variables can be written as:
$$beginalign*
&tpr=fracP-xP, fpr=fracN-axN\
&Rightarrow x=fracN(1-fpr)a\
&Rightarrow tpr =1-fracN(1-fpr)aP (*)\
endalign*$$
Thus, the slope of ROC curve $(*)$ would be:
$$slope=fracNaP$$
Here, we are working with a smooth, continuous ROC curve.

As a result,

1. In the left part of ROC curve we have $slope > 1$, thus precision increases, and




2. In the right part we have $slope < 1$, thus precision decreases.

This is intuitive,

By going right (decrease in false negative), recall and precision are increasing, but at a certain point (red bar) precision starts to decrease, although recall continues increasing.

Note that in a ragged, non-smooth ROC curve, by going right, precision may increase and decrease periodically.