Data Source Cleanup

Variable Selection

Due to the constraints of time, bandwidth and knowledge, we opted to choose a limited number of variables from the complete set for the purposes of modeling. The list was reduced to 54 variables, based on research performed by the team that suggested certain elements that might be of most interest to the goal of predicting suicide attempts.

Variable selected for prediction: V6309

V6309 is a Yes/No response question, "Have you ever attempted suicide?"

Attempted Suicide - V1144 Made Suicide Plan - V6305 Thought About Suicide - V6301 Education Categories - EDCAT Four Race Categories - RACE Employment Status - EMP Major Depression Recency - DEPREC Alc abuse w/ or w/o Dependence - ALCALT1 Drug Abuse w/ or w/o Dependence - ALCALT2 Participant's Age - V12 Participant's Sex - V13 Marriage Status - V4515 Temperamental - V5114 Nervousness - V5118 Discontentedness - V5113 Irritability - V5115 Tenseness - V5112 Self-Identity Question - V4428 Self-Respect Question - V4433 Physiological Pain - V5225 Father attempted suicide - V6649 Mother attempted suicide - V6749 Hopelessness - V5918 Physically Abused as a Child - V6143 Victim of Molestation - V6126 Victim of Rape - V6114 Perception of Physical Health - V101 Perception of Mental Health - V102 Number of Life Traumas Experienced - V6215 Mania - MANLT1 Generalized Anxiety Disorder w/ Hierarchy - GADLT2 Agoraphobia w/o Panic Disorder - AGOLT2 Major Depression w/o Hierarchy - DEPLT1 Conduct Disorder - CDLT Bipolar I w/o Hierarchy - BP1LT1 Drug Dependence - DRGDLT Dysthymia w/o Hierarchy - DYSLT2 Major Depression w/ Hierarchy - DEPLT2 Religion Categories - REL Panic Attack - PTLT Alchohol Abuse w/o Dependence - ALCALT2 Adult Antisocial Behavior - AABLT Antisocial Personality w/ Hierarchy - ASPLT2 Alchohol Dependence - ALCDLT Social Phobia - SOCLT Post-Traumatic Stress Disorder = PTSDLT Generalized Anxiety Disorder w/o Hierarchy - GADLT1 Bipolar I w/ Hierarchy - BP1LT2
Value Scraping & Translation

To make use of the integer values and for purposes of readability, we needed to translate them to category lables. This information was available in the study code book, as well as via the ICPSR website.

Example Web Page of Variable 6301

Using Python, a script was created using BeautifulSoup to scrape all the table values for each variable and convert them into a dictionary format. The dictionary was then used to map the study data values to their respective category labels.

Jupyter Notebook - Scraping
Jupyter Notebook - Convert HTML to Dictionary
Jupyter Notebook - Apply Category Labels
Data Preparation for Model Input

Final steps were performed to prepare the data for entry into the machine learning model:

  • Remove missing cases from the response (missing by survey design) leaving 5872 cases.
  • Convert categorical predictors to one-hot encoded using pandas.get_dummies(). This increases the number of predictors to 161 – all but one predictor is 0-1-valued.
  • Scale the predictors