In k-fold-cross-validation, why do we compute the mean of the metric of each foldBest practice with cross validationWhy k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set?What to do after selecting a model with cross-validation?Some confusions on Model selection using cross-validation approachCross validation for highly imbalanced data with undersamplingCross-validation for model comparison: use the same folds?Validity of cross-validation for model performance estimation
Is it okay to have an email address called "SS"?
Is there a name for this equivalence relation?
How can I use Charisma instead of Strength or Dexterity for weapon attacks without being a Hexblade warlock?
Trading stock more quickly vs. holding it
Can only rich people become president?
Has the 75 move draw rule ever been invoked in a FIDE rated standard game involving a GM?
Left a meeting without apparent reason. What to do?
Can Shadow Pokémon be Shiny in Pokémon Go?
Got an email saying my password is weak, reason for concern?
Why are VW tail light bulbs listed as 12W / 5W?
The spectrum of the sum of two periodic signals
Function defined everywhere but continuous nowhere
How to initiate a conversation with a person who recently had transition but you were not in touch with them?
Multithreading program stuck in optimized mode but runs normally in -O0
Best way to get my money back from a friend having family problems
What to do with developers who don't follow requirements?
Is it safe to run disc brakes up front on a mountain bike that lacks a disc caliper mounting point in the rear?
Is there an uncertainty associated with the value 0 K for absolute zero?
Where are the NPC mages' spellbooks in the Lost Mine of Phandelver adventure?
What are those two silvery objects resting on Ben Kenobi's table when R2-D2 plays Princess Leia's message?
Is exploit-free software possible?
Short story where algae runs a kind of software simulation of another world where life is evolving
Differences between vehicles used on the Moon and the ones used on Mars
How to align multiple elements across align and its nested aligned?
In k-fold-cross-validation, why do we compute the mean of the metric of each fold
Best practice with cross validationWhy k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set?What to do after selecting a model with cross-validation?Some confusions on Model selection using cross-validation approachCross validation for highly imbalanced data with undersamplingCross-validation for model comparison: use the same folds?Validity of cross-validation for model performance estimation
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;
$begingroup$
In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric.
Source : https://scikit-learn.org/stable/modules/cross_validation.html
However, why can't we just compute the metric directly on all the predictions, as we have a estimation for all the data.
Thanks
cross-validation evaluation
asked Jun 14 at 8:42
Alexis PisterAlexis Pister
4399 bronze badges
$endgroup$
add a comment
|
$begingroup$
In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric.
Source : https://scikit-learn.org/stable/modules/cross_validation.html
However, why can't we just compute the metric directly on all the predictions, as we have a estimation for all the data.
Thanks
cross-validation evaluation
asked Jun 14 at 8:42
Alexis PisterAlexis Pister
4399 bronze badges
$endgroup$
add a comment
|
$begingroup$
In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric.
Source : https://scikit-learn.org/stable/modules/cross_validation.html
However, why can't we just compute the metric directly on all the predictions, as we have a estimation for all the data.
Thanks
cross-validation evaluation
asked Jun 14 at 8:42
Alexis PisterAlexis Pister
4399 bronze badges
$endgroup$
In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric.
Source : https://scikit-learn.org/stable/modules/cross_validation.html
However, why can't we just compute the metric directly on all the predictions, as we have a estimation for all the data.
Thanks
cross-validation evaluation
cross-validation evaluation
asked Jun 14 at 8:42
Alexis PisterAlexis Pister
4399 bronze badges
asked Jun 14 at 8:42
Alexis PisterAlexis Pister
4399 bronze badges
asked Jun 14 at 8:42
Alexis PisterAlexis Pister
4399 bronze badges
asked Jun 14 at 8:42
Alexis PisterAlexis Pister
4399 bronze badges
asked Jun 14 at 8:42
Alexis PisterAlexis Pister
4399 bronze badges
4399 bronze badges
add a comment
|
add a comment
|
3 Answers
3
active
oldest
votes
$begingroup$
It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).
But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.
(this is my interpretation, there might be other reasons)
answered Jun 14 at 11:46
ErwanErwan
2,2951 gold badge3 silver badges11 bronze badges
$endgroup$
add a comment
|
$begingroup$
I'm not 100% sure what you mean. In k-fold CV, you partition the training set into $k$ subsets of equal size. Holding out one of these folds at a time, you train the model on the remaining $k-1$ folds to make a prediction for the held-out fold. Thus, in the end, you have one prediction for each observation in your training data.
Now, you can compute average accuracy in two equivalent ways: for each fold, compute the average accuracy, then average the k averages. Or, you average accuracy for every single observation. The two values are the same (up to rounding errors), since all you do is take the intermediate step of calculating averages for each fold. The advantage is that you save on memory because you only have to retain $k$ values (average accuracy per fold) instead of $N$ values (one value for each observation in the training set).
answered Jun 14 at 11:46
Frederic SchneiderFrederic Schneider
211 bronze badge
$endgroup$
add a comment
|
$begingroup$
Purpose of K-Fold cross validation is to evaluate new model performance on unseen data only.
So how K-fold works?
- Model gets trained with training data.
- Split the cross validation data set in to K-Folds(or K equal parts).
- Get the hyper-parameter from the step-1 and train a new model with K-1 folds.
- Evaluate step-3 model using left over Kth fold.
- Repeat the process K times.
- Take mean value of accuracy.
So we need a score of all K folds for model evaluation.
answered Jun 14 at 10:23
vipin bansalvipin bansal
6491 silver badge8 bronze badges
$endgroup$
add a comment
|
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f53773%2fin-k-fold-cross-validation-why-do-we-compute-the-mean-of-the-metric-of-each-fol%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).
But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.
(this is my interpretation, there might be other reasons)
answered Jun 14 at 11:46
ErwanErwan
2,2951 gold badge3 silver badges11 bronze badges
$endgroup$
add a comment
|
$begingroup$
It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).
But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.
(this is my interpretation, there might be other reasons)
answered Jun 14 at 11:46
ErwanErwan
2,2951 gold badge3 silver badges11 bronze badges
$endgroup$
add a comment
|
$begingroup$
It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).
But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.
(this is my interpretation, there might be other reasons)
answered Jun 14 at 11:46
ErwanErwan
2,2951 gold badge3 silver badges11 bronze badges
$endgroup$
It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).
But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.
(this is my interpretation, there might be other reasons)
answered Jun 14 at 11:46
ErwanErwan
2,2951 gold badge3 silver badges11 bronze badges
answered Jun 14 at 11:46
ErwanErwan
2,2951 gold badge3 silver badges11 bronze badges
answered Jun 14 at 11:46
ErwanErwan
2,2951 gold badge3 silver badges11 bronze badges
answered Jun 14 at 11:46
ErwanErwan
2,2951 gold badge3 silver badges11 bronze badges
2,2951 gold badge3 silver badges11 bronze badges
add a comment
|
add a comment
|
$begingroup$
I'm not 100% sure what you mean. In k-fold CV, you partition the training set into $k$ subsets of equal size. Holding out one of these folds at a time, you train the model on the remaining $k-1$ folds to make a prediction for the held-out fold. Thus, in the end, you have one prediction for each observation in your training data.
Now, you can compute average accuracy in two equivalent ways: for each fold, compute the average accuracy, then average the k averages. Or, you average accuracy for every single observation. The two values are the same (up to rounding errors), since all you do is take the intermediate step of calculating averages for each fold. The advantage is that you save on memory because you only have to retain $k$ values (average accuracy per fold) instead of $N$ values (one value for each observation in the training set).
answered Jun 14 at 11:46
Frederic SchneiderFrederic Schneider
211 bronze badge
$endgroup$
add a comment
|
$begingroup$
I'm not 100% sure what you mean. In k-fold CV, you partition the training set into $k$ subsets of equal size. Holding out one of these folds at a time, you train the model on the remaining $k-1$ folds to make a prediction for the held-out fold. Thus, in the end, you have one prediction for each observation in your training data.
Now, you can compute average accuracy in two equivalent ways: for each fold, compute the average accuracy, then average the k averages. Or, you average accuracy for every single observation. The two values are the same (up to rounding errors), since all you do is take the intermediate step of calculating averages for each fold. The advantage is that you save on memory because you only have to retain $k$ values (average accuracy per fold) instead of $N$ values (one value for each observation in the training set).
answered Jun 14 at 11:46
Frederic SchneiderFrederic Schneider
211 bronze badge
$endgroup$
add a comment
|
$begingroup$
I'm not 100% sure what you mean. In k-fold CV, you partition the training set into $k$ subsets of equal size. Holding out one of these folds at a time, you train the model on the remaining $k-1$ folds to make a prediction for the held-out fold. Thus, in the end, you have one prediction for each observation in your training data.
Now, you can compute average accuracy in two equivalent ways: for each fold, compute the average accuracy, then average the k averages. Or, you average accuracy for every single observation. The two values are the same (up to rounding errors), since all you do is take the intermediate step of calculating averages for each fold. The advantage is that you save on memory because you only have to retain $k$ values (average accuracy per fold) instead of $N$ values (one value for each observation in the training set).
answered Jun 14 at 11:46
Frederic SchneiderFrederic Schneider
211 bronze badge
$endgroup$
I'm not 100% sure what you mean. In k-fold CV, you partition the training set into $k$ subsets of equal size. Holding out one of these folds at a time, you train the model on the remaining $k-1$ folds to make a prediction for the held-out fold. Thus, in the end, you have one prediction for each observation in your training data.
Now, you can compute average accuracy in two equivalent ways: for each fold, compute the average accuracy, then average the k averages. Or, you average accuracy for every single observation. The two values are the same (up to rounding errors), since all you do is take the intermediate step of calculating averages for each fold. The advantage is that you save on memory because you only have to retain $k$ values (average accuracy per fold) instead of $N$ values (one value for each observation in the training set).
answered Jun 14 at 11:46
Frederic SchneiderFrederic Schneider
211 bronze badge
answered Jun 14 at 11:46
Frederic SchneiderFrederic Schneider
211 bronze badge
answered Jun 14 at 11:46
Frederic SchneiderFrederic Schneider
211 bronze badge
answered Jun 14 at 11:46
Frederic SchneiderFrederic Schneider
211 bronze badge
211 bronze badge
add a comment
|
add a comment
|
$begingroup$
Purpose of K-Fold cross validation is to evaluate new model performance on unseen data only.
So how K-fold works?
- Model gets trained with training data.
- Split the cross validation data set in to K-Folds(or K equal parts).
- Get the hyper-parameter from the step-1 and train a new model with K-1 folds.
- Evaluate step-3 model using left over Kth fold.
- Repeat the process K times.
- Take mean value of accuracy.
So we need a score of all K folds for model evaluation.
answered Jun 14 at 10:23
vipin bansalvipin bansal
6491 silver badge8 bronze badges
$endgroup$
add a comment
|
$begingroup$
Purpose of K-Fold cross validation is to evaluate new model performance on unseen data only.
So how K-fold works?
- Model gets trained with training data.
- Split the cross validation data set in to K-Folds(or K equal parts).
- Get the hyper-parameter from the step-1 and train a new model with K-1 folds.
- Evaluate step-3 model using left over Kth fold.
- Repeat the process K times.
- Take mean value of accuracy.
So we need a score of all K folds for model evaluation.
answered Jun 14 at 10:23
vipin bansalvipin bansal
6491 silver badge8 bronze badges
$endgroup$
add a comment
|
$begingroup$
Purpose of K-Fold cross validation is to evaluate new model performance on unseen data only.
So how K-fold works?
- Model gets trained with training data.
- Split the cross validation data set in to K-Folds(or K equal parts).
- Get the hyper-parameter from the step-1 and train a new model with K-1 folds.
- Evaluate step-3 model using left over Kth fold.
- Repeat the process K times.
- Take mean value of accuracy.
So we need a score of all K folds for model evaluation.
answered Jun 14 at 10:23
vipin bansalvipin bansal
6491 silver badge8 bronze badges
$endgroup$
Purpose of K-Fold cross validation is to evaluate new model performance on unseen data only.
So how K-fold works?
- Model gets trained with training data.
- Split the cross validation data set in to K-Folds(or K equal parts).
- Get the hyper-parameter from the step-1 and train a new model with K-1 folds.
- Evaluate step-3 model using left over Kth fold.
- Repeat the process K times.
- Take mean value of accuracy.
So we need a score of all K folds for model evaluation.
answered Jun 14 at 10:23
vipin bansalvipin bansal
6491 silver badge8 bronze badges
answered Jun 14 at 10:23
vipin bansalvipin bansal
6491 silver badge8 bronze badges
answered Jun 14 at 10:23
vipin bansalvipin bansal
6491 silver badge8 bronze badges
answered Jun 14 at 10:23
vipin bansalvipin bansal
6491 silver badge8 bronze badges
6491 silver badge8 bronze badges
add a comment
|
add a comment
|
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f53773%2fin-k-fold-cross-validation-why-do-we-compute-the-mean-of-the-metric-of-each-fol%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
