noable.me | WILLIAM WOODHEAD

A programmer's tech quests

ML with Kaggle Ames House Prices

Here is the Github repo for the project code.

Summary

In this project, I built a full end-to-end machine learning pipeline on the Kaggle Ames House Prices dataset. Using careful feature engineering and Gradient Boosting, I achieved a leaderboard score of 0.13339 (top ~2k). The main focus was learning classical ML workflows on tabular data rather than leaderboard chasing.

This post is aimed at beginners and intermediates who want to see another's process for learning classical machine learning on tabular data.

Evaluation

Goal: Predict the sales price for houses. For each Id in the test set, predict the value of SalePrice.

Metric: Submissions are evaluated on Root-Mean-Squared-Error (RMSE) between the logarithm of the predicted value and the logarithm of the observed sales price.

For cross-validation, RMSE is reported on the original SalePrice scale for interpretability. Kaggle leaderboard scores are computed on log-transformed values.

Result

1 entry. Scored 0.13339. 1964 on Leaderboard. Here is the Github repo.

Analysis

EDA

I started with an Exploratory Data Analysis (EDA) checklist. EDA focused on missingness, sparsity, feature correlations, and target distribution — with the goal of informing imputation and scaling decisions.

Here are some interesting findings.

Hygiene checks

The datasets are in good condition. Their columns match (apart from the target SalePrice), and there is a good description in the documentation for almost all of the features.

Missing values and feature categorisation

Quick missing and sparse checks on the combined training and test data. We can see that there are missing values for several features. (yellow indicates missing/sparse values)

This directed me to an imputation strategy for all features, categorising all features into:

1. numeric or categorical
2. structural (missing=non-existent. eg. PoolArea) or substantive (missing=unknown. eg. LotArea)
3. ordinal or nominal for categorical features.

Here is one example of the ordinal categories for one feature:

• Ex Excellent
• Gd Good
• TA Typical - slight dampness allowed
• Fa Fair - dampness or some cracking or settling
• Po Poor - Severe cracking, settling, or wetness
• NA No Basement

I was comfortable assuming that the implied ordering reflects true value differences, while acknowledging this may not always hold perfectly.

Imputation strategy

At this point, I created a feature descriptor database in src/ames/data/features.py to capture the different classes of feature outlined above. Here is a view of it in a table:

Numeric feature correlation

The next step was to explore numeric feature<->feature correlation and feature<->target correlation.

The following image shows top correlating numeric features across both training and test datasets. If we look at the red squares above 0.8 on the heatmap, we can health check it: For example, the top scoring feature<->feature correlation is 0.89 between GarageArea and GarageCars. It is self-explanatory why these two features might correlate highly.

Seeing this heatmap sparked ideas on feature ablation tests for the Feature Engineering phase.

The following image shows the features that correlate most with the target SalePrice. Looking at the first column, we can see that OverallQual and GrLivArea correlate most highly. These are likely to be strong predictors of SalePrice in the modelling phase.

This following image shows a visual-check overview of how each feature scatterplots with SalePrice. I found this very useful to see which features have high sparsity, which are discrete, and which show strong linear correlation. Looking at this chart, it became clear to me that:

• MSSubClass is a categorical in disguise.
• MoSold and YrSold should be dropped since they exhibit no variation on SalePrice.

The following image shows the top Mutual Information scores for numerics features against SalePrice. Mutual Information captures both linear and non-linear dependency by estimating how much knowing one variable reduces uncertainty about the other. Again we can see some familiar features topping the list.

Categorical feature correlation

The following image shows the top Mutual Information scores for categorical features against SalePrice. Here we can see for the first time how strong Neighborhood is as a predictor of SalePrice.

This checks out intuitively. Anyone who has thought about house prices for more than a few seconds knows that location really matters when it comes to house prices.

Target encoding

Target encoding was straight-forward since Kaggle specified it in the competition brief. They want competitors to predict the logarithm of the target. I did some analysis to see why this makes sense.

When we look at the distribution for SalePrice we observe that it is never negative, it is right skewed, and it has some high outliers. We don't want to ignore the high outliers since big house prices are important data points. But we want to find a way to make the distribution more balanced and suitable for linear modelling.

The left hand side is the distribution of SalePrice. The right hand side of this image is the log distribution. We can see that the log distribution is more balanced, and the effect of the outliers has been reduced. This makes it more suitable for linear modelling.

Categorical encoding

Categorical features can be split into ordinal and nominal features. For the ordinal features, the decision was straightforward. I would use OrdinalEncoder. For the nominal features, the options are either to:

• one hot encode
• drop
• make binary

I investigated the top cardinality of the categorical nominal features across training and test datasets:

You can see that there isn't high cardinality in these datasets. Neighborhood has the highest, but we know this is a strong predictor of SalePrice so we don't want to lose any information here. And even then it's only 25.

My thinking at this point was to one-hot encode all categorical nominal features.

One experiment might be to ignore feature categories with low frequency using the min_frequency parameter of OneHotEncoder.

Numeric scaling

In this project I wanted to explore both linear and tree-based models. Linear models require appropriate scaling of all numeric features. So I went through all the numeric features and analysed what scaling might be appropriate. This is what I came up with:

The process of building this table allowed me to group the features into classes. When grouped, these are the overarching heuristics:

• Sparse & low correlation -> binary
• Sparse & high correlation -> zero flag + distribution
• Right-skewed & continuous -> log
• Discrete small counts -> standard
• Poisson-like, mild outliers -> robust
• Irrelevant -> drop
• Categorical in disguise -> one-hot

Modelling

With the EDA complete, my next step was to choose some models and establish baselines.

Choose models

The goal of this project was to explore linear and tree-based models. The models I wanted to evaluate were the following:

Setup decisions

• Use sklearn Pipeline to avoid gotchas with preprocessing.
• Use Cross Validation to avoid overfitting. Use n_splits=10 as a reasonable default.
• Use manual Grid Search to find hyperparameter ranges. This avoid introducing complex dependencies when I'm trying to learn.

Establishing a baseline

The next step was to establish a baseline pipeline, that does imputing, encoding, and modelling. My decision for the baseline was to:

• Drop Id, MoSold
• Impute all features' missing values
• StandardScaler for all numeric features
• OneHotEncoder for all categorical nominal features
• OrdinalEncoder for all categorical ordinal features

My baseline imputation strategy was as follows:

# numeric structural
SimpleImputer(strategy="constant", fill_value=0)
# numeric substantive
SimpleImputer(strategy="median", add_indicator=True)
# categorical structural
SimpleImputer(strategy="constant", fill_value="NA")
# categorical substantive
SimpleImputer(strategy="constant", fill_value="Unknown")
# categorical ordinal structural
SimpleImputer(strategy="constant", fill_value="NA")
# categorical ordinal substantive
SimpleImputer(strategy="most_frequent", add_indicator=True)

Baseline results

With some rough grid search for hyperparameters, these were the initial baseline results with cv_params = {"n_splits": 10} and commit short hash 5a67de1:

We can see that GradientBoost outperforms all other models, and the linear models perform worse than the tree-based models on the baseline pipeline.

At this stage, I decided to move forward with Ridge, RandomForest & GradientBoost.

I was curious to see if I could get the Ridge model working better with feature scaling.

Feature Engineering

Custom scaling

This experiment was to implement custom scaling as described above, for all numeric features to improve linear models. As expected, we observe strong improvement in the Ridge model, but no improvement in the GradientBoost.

commit short hash 5a67de1:

At this point, I decided to move forward just with GradientBoost, due to its performance.

Ablation

I attempted a number of different ablation strategies with GradientBoost and couldn't improve on the score. In practice, Gradient Boosting proved robust to feature ablation; removing correlated or weak features did not improve validation RMSE.

Computed features

In this experiment, I introduce 2 new features:

• Total square footage: TotalSF = 1stFlrSF + 2ndFlrSF + TotalBsmtSF Square footage per room above ground: SqftPerRmAbvGr = TotalSF / (TotRmsAbvGrd + FullBath)

commit short hash 5a67de1:

We observe that these computed features do improve GradientBoost. GradientBoost benefits from computed feature interaction more effectively than any ablation or scaling.

Final tuning

Now with GradientBoost running on the Computed Values Preprocessor, I went forward with some final tuning. Sticking with cross validation n_splits=10, I was able to grid search improved parameters for GradientBoost:

commit short hash 5a67de1:

This was the final tuned model, achieving an RMSE of $23,364.31 under 10-fold cross-validation.

Submission

The final model is stored in src/ames/models/best.py The predictions can be run with:

# Create submission.csv in folder
conda activate ames
python ./src/ames/models/best.py

Learnings

I learned a huge amount on this project. Beforehand I had never run any ML models, and I had never entered into a Kaggle competition. I learned from scratch all about:

• Linear regression with OLS, Lasso, Ridge
• Tree-based regression with Random Forest and Gradient Boost
• EDA, sparsity, distributions
• Data cleaning and imputation
• Feature engineering and baseline testing
• Correlation, both with regression and Mutual Information
• Encoding strategies
• python packages sklearn ,pandas, scipy
• Preprocessor pipelines
• Feature dicts
• Hyperparameter tuning
• Cross Validation

The biggest surprise to me was how much time I had to spend on preprocessing compared to running models. Perhaps 70% of the time spent was in EDA and preprocessing. I also had to spend a lot of time understanding the architecture of the models to gain an intuitive sense of how to improve their performance, for example why scaling improves linear models but not tree-based models.

What I didn’t focus on

• Extensive leaderboard tuning
• Advanced boosting libraries (XGBoost / LightGBM)
• Ensembling or stacking

These were deliberately avoided to keep the focus on fundamentals.

If I were to take this further I might look into:

• XGBoost / LightGBM
• Target encoding for Neighborhood
• Bayesian hyperparameter search
• Out-of-fold stacking
• SHAP for interpretability

Key Takeaways

• Preprocessing is where all your time goes
• Tree models love feature interactions
• Linear models love well-considered scaling