Data prepration for logistic regression : Value either “not available” or a “year”2019 Community Moderator ElectionLogistic Regression implementation does not convergeLogistic regression on biased dataIntuition for Logistic Regression PerformanceUpdate statement for Logistic RegressionSimple logistic regression wrong predictionslogistic regressionThe test of randomness was a logistic regression predicting missingness from all other variablesDealing with NaN (missing) values for Logistic Regression- Best practices?Logistic regression in pythonWhy is logistic regression not sigmoidal?
Is it possible for a PC to dismember a humanoid?
Why do bosons tend to occupy the same state?
Will a Schottky diode save my LEDs against reversed voltage?
Short story with a alien planet, government officials must wear exploding medallions
Should I tell management that I intend to leave due to bad software development practices?
Where would I need my direct neural interface to be implanted?
One verb to replace 'be a member of' a club
What's the in-universe reasoning behind sorcerers needing material components?
Is there a hemisphere-neutral way of specifying a season?
GFCI outlets - can they be repaired? Are they really needed at the end of a circuit?
What does “the session was packed” mean in this context?
What does the expression "A Mann!" means
Alternative to sending password over mail?
Different meanings of こわい
Can we compute the area of a quadrilateral with one right angle when we only know the lengths of any three sides?
How do conventional missiles fly?
How to compactly explain secondary and tertiary characters without resorting to stereotypes?
Why is the sentence "Das ist eine Nase" correct?
Am I breaking OOP practice with this architecture?
In 'Revenger,' what does 'cove' come from?
Can compressed videos be decoded back to their uncompresed original format?
Why is it a bad idea to hire a hitman to eliminate most corrupt politicians?
files created then deleted at every second in tmp directory
Office and personal life politics
Data prepration for logistic regression : Value either “not available” or a “year”
2019 Community Moderator ElectionLogistic Regression implementation does not convergeLogistic regression on biased dataIntuition for Logistic Regression PerformanceUpdate statement for Logistic RegressionSimple logistic regression wrong predictionslogistic regressionThe test of randomness was a logistic regression predicting missingness from all other variablesDealing with NaN (missing) values for Logistic Regression- Best practices?Logistic regression in pythonWhy is logistic regression not sigmoidal?
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked Mar 19 at 6:43
toomtoom
1112
$endgroup$
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked Mar 19 at 6:43
toomtoom
1112
$endgroup$
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked Mar 19 at 6:43
toomtoom
1112
$endgroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
logistic-regression missing-data
asked Mar 19 at 6:43
toomtoom
1112
asked Mar 19 at 6:43
toomtoom
1112
asked Mar 19 at 6:43
toomtoom
1112
asked Mar 19 at 6:43
toomtoom
1112
asked Mar 19 at 6:43
toomtoom
1112
1112
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
add a comment |
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
2,536318
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
2,536318
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
2,536318
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
2,536318
$endgroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
2,536318
edited Mar 19 at 12:32
answered Mar 19 at 7:52
EsmailianEsmailian
2,536318
answered Mar 19 at 7:52
EsmailianEsmailian
2,536318
answered Mar 19 at 7:52
EsmailianEsmailian
2,536318
2,536318
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58