---
title: "Unit 12 Lab Agenda"
author: "Coco Yu"
date: "`r lubridate::today()`"
format: 
  html: 
    embed-resources: true
    toc: true 
    toc_depth: 4
editor_options: 
  chunk_output_type: console
---

```{r}
#| echo: FALSE
#| message: FALSE
options(conflicts.policy = "depends.ok")
devtools::source_url("https://github.com/jjcurtin/lab_support/blob/main/fun_ml.R?raw=true")
tidymodels_conflictRules()
library(tidyverse)
library(tidymodels)
library(tokenizers) # you wont need this one loaded in the future as it is wrapped by other functions
library(tidytext)
library(textrecipes)  # step_* functions for NLP
library(data.table, include.only = "fread")
library(kableExtra, exclude = "group_rows")

theme_set(theme_classic())
options(tibble.width = Inf, dplyr.print_max=Inf)

devtools::source_url("https://github.com/jjcurtin/lab_support/blob/main/fun_eda.R?raw=true",
                     sha1 = "c045eee2655a18dc85e715b78182f176327358a7")
devtools::source_url("https://github.com/jjcurtin/lab_support/blob/main/fun_plots.R?raw=true")

cl <- parallel::makePSOCKcluster(parallel::detectCores(logical = FALSE))
doParallel::registerDoParallel(cl)
```

```{r}
path_data <- "data"
data <- read_csv(here::here(path_data, "alcohol_tweets.csv"), col_types = cols()) |> 
  mutate(user_drinking = factor(user_drinking)) |> 
  glimpse()
```


### Cheat sheet

- [Base R function cheat sheet](https://iqss.github.io/dss-workshops/R/Rintro/base-r-cheat-sheet.pdf)

- [A compilation of cheat sheets](https://posit.co/resources/cheatsheets/)

### Package conflicts

> When I load the package "data.table", there is a error indicating the conflict:

```

data.table 1.17.0 using 8 threads (see ?getDTthreads).  Latest news: r-datatable.com
Error: Conflicts attaching package ‘data.table’:

The following objects are masked from ‘package:lubridate’:

    hour, isoweek, mday, minute, month, quarter, second, wday, week, yday, year

The following objects are masked from ‘package:dplyr’:

    between, first, last

The following object is masked from ‘package:purrr’:

    transpose

```

> I have tried to fix it with `exclude = ` and `conflictRules()` but both do not work, I wonder how can I deal with this warning message?

**Solution 1**: When you know only one function (or very few) functions will be used in your codes, you can use the `::` operator to specify which package the function comes from. For example, if you want to use the `fread()` function from the `data.table` package, you can write `data.table::fread()` instead of just `fread()`. This way, you can avoid conflicts with other packages that may have functions with the same name.

**Solution 2**: Likewise, you can use `include.only = ` to specify which functions you want to include from the package. For example, if you only want to use the `fread()` function from the `data.table` package, you can write `library(data.table, include.only = "fread")`. This way, you can avoid loading all the functions from the package and reduce the chances of conflicts with other packages.

**Solution 3**: When you know for sure which functions are conflicting with the ones in other packages, and you prefer the ones in another package, then use `exc`lude = ` to exclude the conflicting functions from the package. 


### Data Cleaning/Preparation

#### N/A values

You might have N/A values in your dataset, especially when you have a large and messy text dataset. You can use the `data |> filter(is.na(col))` function to check for N/A values in your dataset and get a sense of how many N/A values you have, and what might be the possible reasons for them.

Depending on your EDA, you can decide on how to handle N/A values. Here are some common strategies:

- Remove rows with N/A values: You can use the `data |> filter(!is.na(col))` function to remove rows with N/A values. This is a simple and effective way to handle N/A values, but it can also lead to loss of information if you have a lot of N/A values in your dataset.

- Impute missing values: You can use the `data |> mutate(col = ifelse(is.na(col), value, col))` function to impute missing values with a specific value. This can be useful when you want to keep all the rows in your dataset, but it can also introduce bias if the imputed value is not representative of the true value. It can be hard to impute missing values in text data.

**Homework Example**

#### Regular Expressions

website: https://regex101.com/

Regex is useful when you want to do data cleaning. For example, you can use regex to match a specific pattern in a string, such as email addresses, phone numbers, or URLs. It can also be used to extract or replace specific parts of a string, such as removing punctuation or whitespace.

Below are some examples that you might need to use regex for data cleaning:

- **Extract dates**: `str_extract("The date is 2025-04-21", "\\d{4}-\\d{2}-\\d{2}")` to extract the date in the format YYYY-MM-DD.

- **Standardize formats**: For example, run `str_replace_all("(123) 456-7890", "\\D", "")` to remove all non-digit characters from the phone number. This can be useful when you have varying formatting of phone addresses in your dataset and want to standardize them or remove any formatting characters. `str_replace_all("☎️1234567890", "\\D","")` will return the same result as the previous example.

- **Make all text lowercase**: run `str_to_lower("Hello World")` to convert all text to lowercase. This can be useful when you want to standardize the text in your dataset and remove any case sensitivity.

- **Detecting specific patterns**: For example, when you're doing spam filtering for emails, you might want to detect specific patterns in the email addresses or subject lines. You can use regex to match specific keywords or phrases that are commonly found in spam emails. For example, you can use `str_detect("Free money!!!", "free|money")` to check if the string contains either "free" or "money".

- **Removing unwanted characters**: For example, when you're doing text analysis, you might want to remove unwanted characters such as punctuation marks or special characters. You can use regex to match and remove these characters. For example, you can use `str_remove_all("Hello, World!", "[[:punct:]]")` to remove all punctuation marks from the string.

- **Data Scraping**: For example, when you're scraping data from websites, you might want to extract specific information from the HTML code. You can use regex to match and extract the relevant parts of the HTML code. For example, you can use `str_extract("<a href='https://example.com'>Click here</a>", "href='(.*?)'")` to extract the URL from the anchor tag.

#### Stop Words

**Things to consider when you create your own stop word lists**

- *Start with a standard stop word list*: You can start with a standard stop word list. This list includes common stop words like "the", "is", "and", etc. You can then add or remove words from this list based on your specific needs.

- *Domain-specific stop words*: You might want to create a custom stop word list that includes words that are specific to your domain or industry. For example, if you're analyzing tweets about sports, you might want to include words like "game", "team", or "player" in your stop word list.

- *Most common words*: You might want to create a custom stop word list that includes the most common words (but might not useful for your predictions) in your dataset. This can help you remove noise from your data and improve the performance of your model. You can look through words that have the highest frequency (or term frequency, tf-idf) in your dataset, and think about whether they will add value to your analysis. 

**Use different stop words for different goals**

- *sentiment analysis*: you might want to keep certain words that are commonly used in positive or negative sentiments, such as "good" or "bad". In this case, you might want to create a custom stop word list that excludes these words.

- *topic modeling*: you might want to keep certain words that are commonly used in specific topics, such as "alcohol" or "drink" if you're analyzing alcohol-related content on Twitter. In this case, you might want to create a custom stop word list that excludes these words.

*Tips: We recommend always keeping a low bar for stop words. You can always remove them later if you find that they are not useful for your analysis. This way, you can avoid losing potentially valuable information in your dataset.*

*In machine learning, you can always train models with and without stop words and compare the results. This way, you can determine whether removing stop words improves or degrades the model's performance.*

#### Abbreviations

For common abbreviations, you can use regex to match and replace them with their full forms. For example, you can use `str_replace_all("LOL", "laughing out loud")` to replace "LOL" with "laughing out loud". This can be useful when you want to standardize the text in your dataset and remove any abbreviations.

For other uncommon abbreviations, it is okay to leave them as they are. They likely occur very infrequently and will not have a significant impact on your analysis. Remember that the goal of data cleaning is to improve the quality of your data, but it's never possible to achieve perfect data quality. You should always balance the effort you put into data cleaning with the potential benefits it brings to your analysis.

#### Emojis

Emojis can be useful for sentiment analysis, as they can convey emotions and sentiments that are not easily expressed in words. For example, a smiley face emoji can indicate a positive sentiment, while a frowning face emoji can indicate a negative sentiment.

To handle emojis in your text data, you can use regex to match and replace them with their corresponding meanings. For example, you can use `str_replace_all("😊", "happy")` to replace the smiley face emoji with the word "happy". This can help you standardize the text in your dataset and make it easier to analyze.

However, this approach can be time-consuming and may not always be accurate, as some emojis can have multiple meanings depending on the context. For example, the "😂" emoji can indicate laughter or sarcasm, depending on the context in which it is used. To handle this, you can use a more sophisticated approach, such as using pre-trained word embeddings (e.g., BERT) that can capture the meaning of emojis in context.

**Coding Tips**: To find out all the emojis in your text data, you can use the `stringi` package. For example, you can use `stringi::stri_extract_all_regex(text, "\\p{Emoji}")` to extract all emojis from the text. This can help you identify which emojis are present in your dataset and how frequently they occur.

```{r}
stringi::stri_extract_all_regex("sample emojis 🤭😭😂✈️", "\\p{Emoji}")
```


### Tokenization

We will now tokenize the text data using the `unnest_tokens()` function from the `tidytext` package. This function allows us to extract unigrams (single words), bigrams (two-word combinations), and other n-grams from the text data.

```{r}
tokens <- data |> 
  unnest_tokens(word, text, drop = FALSE)

tokens_bigram <- data |> 
  unnest_tokens(bigram, text, token = "ngrams", n = 2, drop = FALSE)
```

By visualizing the tokens a little more, we can see the most common unigrams and bigrams in the text data. This can help us understand the distribution of words and phrases in the text data, and identify any patterns or trends that may be relevant to our analysis. We might use information from this analysis to inform our feature engineering process, such as creating stop words or selecting specific n-grams to include in our model.

Let's take a look at our unigrams first! The distribution is pretty skewed -- but don't worry, this is normal for text data.

```{r}
tokens |> 
  count(word, sort = TRUE) |> 
  slice(1:1000) |>
  mutate(word = reorder(word, -n)) |> 
  ggplot(aes(word, n)) +
  geom_col() +
  xlab('Words') + 
  ylab('Raw Count') +
  theme(axis.text.x=element_blank(),
        axis.ticks.x=element_blank())
```

Next, we can skim the top 50 most common unigrams. This is a little more interesting, as we can see some common phrases that are used in the text data. We see that some most frequent words are probably not relevant to our analysis, such as "the", "a", and "an", indicating that more cleaning is needed. However, we can already see that some of the most common words are related to alcohol, such as "party", "beer", and "drinking". This suggests that the text data is likely related to alcohol consumption, which is relevant to our analysis.

```{r}
tokens |> 
  count(word, sort = TRUE) |> 
  slice(1:50) |> 
  mutate(word = reorder(word, -n)) |> 
  ggplot(aes(word, n)) +
  geom_col(color = "black") +
  xlab('Words') + 
  ylab('Raw Count') +
  theme(axis.text.x=element_text(angle = 60, hjust = 1))
```

We can also create a scrollable table to look through some top unigrams.

```{r}
tokens |> 
  count(word, sort = TRUE) |> 
  slice(1:200) |> 
  kbl() |> kable_styling(bootstrap_options = c('striped', 'condensed')) |> 
  scroll_box(height = "500px", width = "100%")
```

Now let's take similar steps to look at the bigrams. 

```{r}
tokens_bigram |> 
  count(bigram, sort = TRUE) |> 
  slice(1:50) |> 
  mutate(bigram = reorder(bigram, -n)) |> 
  ggplot(aes(bigram, n)) +
  geom_col(color = "black") +
  xlab('Words') + 
  ylab('Raw Count') +
  theme(axis.text.x=element_text(angle = 60, hjust = 1))
```

Through looking at the top bigrams, we can see that more cleaning is needed -- clearly for N/A values, and url links. Nonetheless, we can see that some of the most common bigrams might be related to alcohol, such as "fucked up", "get drunk", and "a party". This suggests that bigrams might be useful for our analysis, as they can capture more context and meaning than unigrams alone.

```{r}
tokens_bigram |>
  filter(str_detect(bigram, " ")) |> 
  count(bigram, sort = TRUE) |> 
  kbl() |> kable_styling(bootstrap_options = c('striped', 'condensed')) |> 
  scroll_box(height = "500px", width = "100%")
```

### Word Embeddings

#### slide_windows() and skipgrams

For more information, please consult the [book](https://smltar.com/embeddings).

Skipgram is a word embedding technique that is used to learn word representations from a large corpus of text. It is based on the idea that words that appear in similar contexts tend to have similar meanings. The skipgram model works by predicting the context words given a target word, which allows it to learn the relationships between words in the text.

Sliding window defines the context of the target word. It is a fixed-size window that moves over the text, capturing the surrounding words as context. The size of the sliding window determines how many context words are considered for each target word. For example, if the window size is 2, then for each target word, the two words to the left and two words to the right will be considered as context words.

Below is an example of how to create a customized function to create skipgrams using a sliding window approach. This function takes a tibble and a window size as input, and returns a tibble with the skipgrams and their corresponding window IDs.

We calculate the pmi -- pointwise mutual information -- for each word pair in the skipgrams. PMI is a measure of association between two words, and it is calculated as the logarithm of the ratio of the joint probability of the two words occurring together to the product of their individual probabilities. A higher PMI value indicates a stronger association between the two words.

```{r}
slide_windows <- function(tbl, window_size) {
  skipgrams <- slider::slide(
    tbl, 
    ~.x, 
    .after = window_size - 1, 
    .step = 1, 
    .complete = TRUE
  )
  
  safe_mutate <- safely(mutate)
  
  out <- map2(skipgrams,
              1:length(skipgrams),
              ~ safe_mutate(.x, window_id = .y))
  
  out |> 
    transpose() |> 
    pluck("result") |> 
    compact() |> 
    bind_rows()
}

tidy_pmi <- tokens |> 
  slide_windows(4L) |>
  unnest(word) |> 
  unite(window_id, id, window_id) |> 
  widyr::pairwise_pmi(word, window_id)

tidy_pmi |> 
  arrange(desc(pmi)) |> 
  head(200) |> 
  print_kbl()
```

#### Out of Vocabulary (OOV) words

OOV words are words that are not present in the vocabulary of the word embedding model. This can happen when the model is trained on a specific corpus and does not include all possible words. OOV words can be problematic for text analysis, as they can lead to loss of information and reduced performance of the model.

To handle OOV words, you can use different strategies depending on the context and the specific requirements of your analysis. Here are some common strategies:

- **Ignore OOV words**: You can simply ignore OOV words and not include them in your analysis. This is a simple approach, but it can lead to loss of information and reduced performance of the model. You can ignore OOV words if you think they are not relevant to your analysis or if they occur very infrequently in your dataset.

- **Use subword embeddings**: You can use subword embeddings, which break down words into smaller units (subwords) to handle OOV words. This allows you to represent OOV words as a combination of subwords, which can improve the performance of the model. For example, the word "unhappiness" can be represented as "un", "happi", and "ness".

- **Use a larger vocabulary**: You can use a larger vocabulary that includes more words to reduce the number of OOV words. This can be done by training the model on a larger corpus or using a pre-trained model with a larger vocabulary. However, this approach can also increase the complexity and size of the model.


### Hooray! 🎉

<font style = "color:red; background-color:yellow">**Congratulations on finishing on all your application assignments for this semester😊!!**</font>

Now is exercise time! 

Quickly go over your previous assignments -- did you run into any errors at all? Most likely yes! The last exercise we want to do is help you learn ways how to seek help if you need it as we transfer out of this course. 

And that means, we're now going back to creating reproducible examples using the `reprex` package!

Find your favorite error you had in the past (or even warning signs), and try to create a reproducible example from it. Slack your results to a thread that includes the TAs.

Good luck!