---
title: "Unit 03 Lab Agenda"
date: "02/13/2024"
format: 
  html: 
    embed-resources: true
    toc: true 
    toc_depth: 4
editor_options: 
  chunk_output_type: console
---

## Main agenda

### 1. Best practices reminder!

- paths, projects, here (may not get errors when running locally, but will likely get some at some point when knitting.)
- reminder that to get full credit you need to turn in a knitted html file!
- warnings - determine why you are getting them, comment, and suppress
- Do you need to update your R version?

### 2. KNN errors regarding missing levels 

- Creates issues when predicting in validation and/or test and with plots

- My apologies about the comments on you guys' for this warning. It's not caused by you not using the data dictionary (you probably well did!) but by the update from R and `recipe`.

- John just updated the latest version of `step_` series using `step_novel` to handle the missing levels in factors following the best practice of `recipe` (1.1.0): https://jjcurtin.github.io/book_iaml/app_dummy_coding.html

### 3. class_ames function -> See the key for a more complete version

- Apply to all datasets (including test!)
- Customize it!

```{r}
class_ames <- function(df){
  df |>
    mutate(across(where(is.character), factor)) |> 
    mutate(overall_qual = factor(overall_qual, levels = 1:10), 
           garage_qual = suppressWarnings(fct_relevel(garage_qual,
                                                      c("no_garage", "po", "fa", 
                                                      "ta", "gd", "ex"))))
}
```


### 4. Purpose of error tibble

We noticed a lot of you didn't intend to use a tracking tibble. It might look like a little chunky adding them all way down but will help compare across the models (both for you, the others skimming the results - like us, and the future YOU!)

### 5. Transformation -> Transformation demo 


### 6. Winners fit great models! Walk us through your process?

Yeah, we heard a lot keen words calling our lunch winner to share some special taste of data. 

Let's welcome Chris!



## Question list

I do wanna answer all the questions for you guys. Check out the above code if they have already answered for yours. If not, see the extra answers here and let's discuss.


### Step/recipe

- I would like to go more in depth with the step function. There are a lot of different ways to play with your data using it, and it would be great to go over what each one is and when to use it in discussion together.
- I really enjoyed learning about the step_() functions. I would like to see more of them as they seem to be a very intuitive tool for our projects.

### EDA/Predictor

- In the homework we were trying to design models based on the results of modeling EDA. I found that the most potentially interesting bivariate relationships involved many many levels. Trying to determine how they might interact with one another and to the outcome was very difficult. Tips on making sense of such relationships would be useful.
- Going over how to do **categorical variable interactions** with knn would be helpful.
- How do we code **interactions** between **categorical and numerical** predictors, or between **categorical and categorical** predictors?
    - Always, in the recipe workflow - `step_interact`
    
- How do we interpret different values of **correlation** coefficients (e.g., strong vs. weak correlations, positive vs. negative)?
    - Recall a little back on 610 (or if you're not there... https://en.wikipedia.org/wiki/Correlation_coefficient); basically it's describing how (linearly) 2 variables are related to each other, and thus how one of them could 'predict'(not in a certainly causal way) another
    - Pay attention to the multicollinearity

### Transformation -> Transformation demo & Chris' recipe

- Transformations of the outcome variable in the prep() and bake() workflow. Most recommendations I saw said that this specific workflow was really difficult to implement those sorts of transformations.
- Other functions of transforming predictors, since I only used step_YeoJohnson. :D
- If step_log() transformation is not effective enough, what are some other transformations that be applied?
- How do we transform the outcome variable within the prep/bake workflow?
- How do we de-transform the box-cox transformation for the outcome variable sales price?
- I'm curious on how to transform an outcome variable and then how to transform it back to a raw score in the end.
- I want to have an example using power transformation in LM
- Please go into more detail on transformations.
- I really just want to know which set of predictors made the best model if a log transformation was really necessary to get a good RMSE. I think it would be fun to code along to.

### Factor/Nominal -> Did those have been answered in the key?

- I was having a lot of issues the multiple-level categorical predictors in my model. It was giving me an error that it was generating more 'truth' observations than 'estimate' observations and I could not find how to fix this.
    - Check out the `step_novel` as mentioned above
    
- Can we review EDA for modeling that would help us know if there is a strong interaction among variables in our dataset?
- going over the differences between factor/fct, fct_collpase, and reorder and when to use each one
    - factor(): Converts a vector to a factor	When defining categorical variables, manually setting order
    - fct_collapse(): Merges multiple factor levels into fewer categories	When you need to simplify categories (e.g., grouping rare labels)
    - fct_reorder(): Reorders factor levels based on another variable	When ordering categories by a numeric statistic for analysis or visualization
    
- collapse levels and dummy walk through
- In recipe(), I want to explore more on how to feature engineer the features we have, how can we change give levels to ordinal data?
- Examples showing how to collapse levels of categorical variables and dummy coding

### KNN

- I think I'm still a little bit confused about what k does as a hyperparameter and what the different values mean
    - Yeah, it controls how many of the neighbors the model is picking from the vicinity of the data point to be predicted, and the larger the more global it is (which could be more biased because it's losing possibly informative local properties) and the smaller the more local (and thus possibly adapted to the local noise, which could lead to overfitting)
    
- Strategies for selecting the optimal k.
- I want to know how to get the best k in the knn model in R.
    - Best practice is to create tracking data series of the hyperparameters and plot them with the performance metric to see the optimal point

### General

- You mentioned before that after training, validation, and selecting the best model, that one optional step is before taking that best model into test, training it on both the training and validation data. At what exact point(s) in the workflow/code do you do this? Is it only when prepping the recipe to bake the test data? Or do you go one step before that and re-train the model using a feature matrix also baked using both training and validation data?
    - When we finish the model selection using the validation data, we want to combine the trn+val with the best model recipe to train a model, and test it in the test data, to have a best estimate on the performance of the final model.
    - When we finish the estimating of the performance of the model using the test data, we would move to fit a final model using the full data (trn+val+test) and it may have even better performance.

- It would be great if we deal with other available engines for the broad classes of algorithms, for example KNN(classification), logistic, glm and so on.
    - I guess you guys are doing great with the new ones in unit 4? More in the future.
    
- Is John's dataset different from ours? The RMSE is huge compared to his even when doing very similar things.
    - Yes, it's the same.
    
- Some further skills of feature engineering and suggestions about appropriate time to engineer based on characteristics of features and research conditions would be helpful.
    - Uh, this depends. But generally speaking, it does take a long time to clean data and handcraft it for new features generated based on the generic ones. And if your data is dynamic like in industrial settings, it would be even more flexible.
    
## Questions/Supplements from the lab discussion

### Three-way interaction (See lm key rep_3)

- Why we don't use 3-way interaction in recipe steps?
    - Actually we could add them in, but the `step_interact` will only include the 3-way term but not all the relevant 2-way terms
    - If we would like to consider a 3-way interaction from a canonical LM view, we may also consider the 2-way terms
    
### How do we choose between YeoJohnson/Power and Log for transformation

- First, we usually don't apply YeoJohnson to the outcome variable, because it wouldn't return us a specific value, so we cannot convert the predict outcome back for comparison

- Second, (for positive values) log is compressing a variable, and power (with an exponent/power number>1) is enlarging a variable or (with an exponent/power number<1) is compressing a variable (also pay attention to the range of the variable, whether it is crossing 1)
    - Depends on the value of the exponent the power transformation plays different role. Only when the power number is less than 1 it is doing more similar job as log transformation. How much we want to compress the value? The Box-Cox method could help determine the optimal value for power transformation. Will we prefer log than power? It will be relevant with the actual shape of the relationship curve. Sometimes (maybe in biology or other field), the relationship has some theoretical explanation/hypothesis, so researcher could be more intended to use a transformation accordingly. But for social sciences, the relationship is more possibly complex and hard to determine. So, the application of transformation will more depends on the real data you have.
    -  For the sake of explanation, log transformation will be easier most of the time, be cause we can interpret in terms of relative magnitude. For example, if income increases from \$10,000 to \$20,000, the log transformation shows a change of log(20,000) - log(10,000) = 0.69, which can be interpreted as approximately a 69% relative increase, whereas a power transformation like x^0.5 (square root) would change from √10,000 = 100 to √20,000 ≈ 141, which lacks a straightforward percentage interpretation.

- For some data ranging from a very small number to a very large number, across sacles (especially ^10 scales), log transformation could help boil down the range of the values to a shared similar scale (e.g. 10, 10^3, 10^6 to 1, 3, 6).

    
```{r}
# Load required library
library(ggplot2)
library(tidyverse)

# Generate a range of values
x_vals <- seq(0, 3, length.out = 20)  

# Apply transformations
df <- data.frame(
  x = x_vals,
  x_ori = x_vals,
  log_x = ifelse(x_vals > 0, log(x_vals), sign(x_vals) * log(abs(x_vals) + 1)),  # Handle negatives
  sqrt_x = ifelse(x_vals >= 0, sqrt(x_vals), NaN),  # x^(1/2)
  square_x = x_vals^2, # x^2
  reciprocal_x = ifelse(x_vals != 0, 1 / x_vals, NaN)  # x^(-1)
)

# Convert to long format for plotting
df_long <- df |> 
  pivot_longer(-x, names_to = "transformation", values_to = "value")

# Plot transformations
ggplot(df_long, aes(x = x, y = value, color = transformation)) +
  geom_line(size = 0.5) +
  theme_minimal() +
  labs(title = "Log and Power Transformations",
       x = "Original Values (x)", 
       y = "Transformed Values",
       color = "Transformation") +
  theme(legend.position = "bottom") 

```

### When to transform the predictor and when to transform the outcome

When the skewed relationship is only seen in the EDA for a single feature, it will be preferred to transform the feature. But if the relationship seems to be skewed for all the independent-dependent plot or even after the transformation of the feature there is still skew, a transformation toward the outcome variable might worth consideration.

### Some `step_`s

The usage of the recipe varies across people, so I'm sorry for the unclear walking through of the steps. In case it's needed, here added some explanation of the steps appear in the key but not used too much in the scripts from you guys.

- `step_other`: automatically collapse some levels of a predictor by the algorithm, and we could set a threshold to define the collapsing.

- `step_nzv`: remove variables that are highly sparse and unbalanced (having little variance across rows or little diversity - which means it's 'almost the same' for all the items)

- `step_mutate_at`: specifying columns/features to apply a function

### Interaction (mis-usage in the original key!) in KNN

Because KNN naturally captures non-linear relationships, it's not contributive to add interaction terms in a KNN based recipe. So, we can still add them in the recipe, but it will barely benefit the results.

### Regularization/Penalty

It's a technique control overfitting. And we will step deeper in future units.


    
