How to handle columns with categorical data and many unique values The 2019 Stack Overflow Developer Survey Results Are Indecision trees on mix of categorical and real value parametersPandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)Imputation of missing values and dealing with categorical valuesHow to deal with categorical variablesOne hot encoding error “sort.list(y)…”One hot encoding vs Word embeddingHow to implement feature selection for categorical variables (especially with many categories)?ML Models: How to handle categorical feature with over 1000 unique values“Binary Encoding” in “Decision Tree” / “Random Forest” AlgorithmsDealing with multiple distinct-value categorical variables
How can I autofill dates in Excel excluding Sunday?
When should I buy a clipper card after flying to OAK?
Deal with toxic manager when you can't quit
What is the accessibility of a package's `Private` context variables?
Why do UK politicians seemingly ignore opinion polls on Brexit?
Am I thawing this London Broil safely?
Return to UK after being refused entry years previously
How to notate time signature switching consistently every measure
Output the Arecibo Message
Can we generate random numbers using irrational numbers like π and e?
Button changing it's text & action. Good or terrible?
Did Section 31 appear in Star Trek: The Next Generation?
What tool would a Roman-age civilization have for the breaking of silver and other metals into dust?
Right tool to dig six foot holes?
Can someone be penalized for an "unlawful" act if no penalty is specified?
What does Linus Torvalds mean when he says that Git "never ever" tracks a file?
Is flight data recorder erased after every flight?
Is three citations per paragraph excessive for undergraduate research paper?
Protecting Dualbooting Windows from dangerous code (like rm -rf)
What is the meaning of the verb "bear" in this context?
How come people say “Would of”?
slides for 30min~1hr skype tenure track application interview
Why do some words that are not inflected have an umlaut?
"as much details as you can remember"
How to handle columns with categorical data and many unique values
The 2019 Stack Overflow Developer Survey Results Are Indecision trees on mix of categorical and real value parametersPandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)Imputation of missing values and dealing with categorical valuesHow to deal with categorical variablesOne hot encoding error “sort.list(y)…”One hot encoding vs Word embeddingHow to implement feature selection for categorical variables (especially with many categories)?ML Models: How to handle categorical feature with over 1000 unique values“Binary Encoding” in “Decision Tree” / “Random Forest” AlgorithmsDealing with multiple distinct-value categorical variables
$begingroup$
I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.
I also have another column with 145 nunique values that I could also use in my model that represents product category.
Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?
Can you point me to the right direction if I should use another encoding also?
machine-learning data categorical-data encoding
asked 2 days ago
dungeondungeon
344
$endgroup$
add a comment |
$begingroup$
I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.
I also have another column with 145 nunique values that I could also use in my model that represents product category.
Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?
Can you point me to the right direction if I should use another encoding also?
machine-learning data categorical-data encoding
asked 2 days ago
dungeondungeon
344
$endgroup$
add a comment |
$begingroup$
I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.
I also have another column with 145 nunique values that I could also use in my model that represents product category.
Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?
Can you point me to the right direction if I should use another encoding also?
machine-learning data categorical-data encoding
asked 2 days ago
dungeondungeon
344
$endgroup$
I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.
I also have another column with 145 nunique values that I could also use in my model that represents product category.
Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?
Can you point me to the right direction if I should use another encoding also?
machine-learning data categorical-data encoding
machine-learning data categorical-data encoding
asked 2 days ago
dungeondungeon
344
asked 2 days ago
dungeondungeon
344
asked 2 days ago
dungeondungeon
344
asked 2 days ago
dungeondungeon
344
asked 2 days ago
dungeondungeon
344
344
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
For categorical columns, you have two options :
- Entity Embeddings
- One Hot Vector
For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.
Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.
Articles that explain Embeddings :
An Overview of Categorical Input Handling for Neural Networks
On learning embeddings for categorical data using Keras
Google Developers > Machine Learning > Embeddings: Categorical Input Data
Exploring Embeddings for Categorical Variables with Keras by Florian Teschner
answered 2 days ago
Shamit VermaShamit Verma
1,5191314
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48875%2fhow-to-handle-columns-with-categorical-data-and-many-unique-values%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
For categorical columns, you have two options :
- Entity Embeddings
- One Hot Vector
For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.
Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.
Articles that explain Embeddings :
An Overview of Categorical Input Handling for Neural Networks
On learning embeddings for categorical data using Keras
Google Developers > Machine Learning > Embeddings: Categorical Input Data
Exploring Embeddings for Categorical Variables with Keras by Florian Teschner
answered 2 days ago
Shamit VermaShamit Verma
1,5191314
$endgroup$
add a comment |
$begingroup$
For categorical columns, you have two options :
- Entity Embeddings
- One Hot Vector
For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.
Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.
Articles that explain Embeddings :
An Overview of Categorical Input Handling for Neural Networks
On learning embeddings for categorical data using Keras
Google Developers > Machine Learning > Embeddings: Categorical Input Data
Exploring Embeddings for Categorical Variables with Keras by Florian Teschner
answered 2 days ago
Shamit VermaShamit Verma
1,5191314
$endgroup$
add a comment |
$begingroup$
For categorical columns, you have two options :
- Entity Embeddings
- One Hot Vector
For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.
Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.
Articles that explain Embeddings :
An Overview of Categorical Input Handling for Neural Networks
On learning embeddings for categorical data using Keras
Google Developers > Machine Learning > Embeddings: Categorical Input Data
Exploring Embeddings for Categorical Variables with Keras by Florian Teschner
answered 2 days ago
Shamit VermaShamit Verma
1,5191314
$endgroup$
For categorical columns, you have two options :
- Entity Embeddings
- One Hot Vector
For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.
Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.
Articles that explain Embeddings :
An Overview of Categorical Input Handling for Neural Networks
On learning embeddings for categorical data using Keras
Google Developers > Machine Learning > Embeddings: Categorical Input Data
Exploring Embeddings for Categorical Variables with Keras by Florian Teschner
answered 2 days ago
Shamit VermaShamit Verma
1,5191314
edited 2 days ago
answered 2 days ago
Shamit VermaShamit Verma
1,5191314
answered 2 days ago
Shamit VermaShamit Verma
1,5191314
answered 2 days ago
Shamit VermaShamit Verma
1,5191314
1,5191314
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48875%2fhow-to-handle-columns-with-categorical-data-and-many-unique-values%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
