--- title: "String Parsing and Regular Expressions" output: html_document: toc: yes code_folding: show code_download: true --- ```{r setup, include = FALSE} source(here::here("setup.R")) knitr::opts_chunk$set(collapse = TRUE, message = FALSE, fig.height = 5, fig.width = 6, fig.align = "center") options(htmltools.dir.version = FALSE) set.seed(12345) library(ggplot2) library(lattice) library(tidyverse) library(gridExtra) theme_set(theme_minimal() + theme(text = element_text(size = 16), panel.border = element_rect(color = "grey30", fill = NA))) ``` ## Data Cleaning After reading data from text files or web pages it is common to have to * clean up string variables; * extract numbers. This can be done using * high-level functions that usually work but not always; * using string subsetting; * using _regular expressions_. Some examples of strings that need to be processed: ```{r, eval = FALSE} "12%" "New York *" "2,100" "Temp: 12 \u00b0F" ``` Some of the most common cases are covered here. Much more is available in [_R for Data Science_](https://r4ds.had.co.nz/), in particular in the chapters * [_Data Import_](https://r4ds.had.co.nz/data-import.html); * [_Strings_](https://r4ds.had.co.nz/strings.html). ## Removing a Percent Sign Reading the GDP growth rate data from a [web page](https://www.multpl.com/us-real-gdp-growth-rate/table/by-quarter) produced a data frame with a column like ```{r} s <- c("12%", "2%") ``` This can be converted to a numeric variable by * extracting the sub-string without the `%` * and then using `as.numeric`. ```{r} nchar(s) substr(s, 1, nchar(s) - 1) as.numeric(substr(s, 1, nchar(s) - 1)) ``` An alternative is to use `sub()` function to replace `"%"` by the empty string `""`. ```{r} as.numeric(sub("%", "", s)) ``` The function `parse_number` in the `readr` package ignores the percent sign and extracts the numbers correctly: ```{r} library(readr) parse_number(s) ``` ## Removing Grouping Characters Numbers are sometimes written using _grouping characters_: ```{r} s1 <- c("800", "2,100") s2 <- c("800", "2,100", "3,123,500") ``` * The comma is often used as a grouping character in the US. * Other countries use different characters. * Other countries also use different characters for the decimal separator. `sub` and `gsub` can be used to remove grouping characters: ```{r} sub(",", "", s1) sub(",", "", s2) ``` ```{r} gsub(",", "", s2) as.numeric(gsub(",", "", s2)) ``` * `sub` replaces the first match to a pattern; * `gsub` replaces all matches. `parse_number` can again be used: ```{r} parse_number(s2) ``` `parse_number` is convenient but may be less robust: * In Switzerland the grouping character is `'`. * If `parse_number` has its defaults set to Swiss conventions: ```{r} parse_number(s2, locale = locale(grouping_mark = "'")) ``` ## Separating City and State Data often has city and state specified in a variable like ```{r} s <- c("Boston, MA", "Iowa City, IA") ``` If all state specifications are in two-letter form then city and state can be extracted as sub-strings: ```{r} substr(s, 1, nchar(s) - 4) substr(s, nchar(s) - 1, nchar(s)) ``` This would not work if full state names are used. An alternative is to use a _regular expression_. ## Regular Expressions Regular expressions are a language for expressing patterns in strings. Regular expressions should be developed carefully, like any program, starting with simple steps and building up. The simplest regular expressions are literal strings, like `%`. More complex expressions are built up using _meta-characters_ that have special meanings in regular expressions. Many punctuation characters are regular expression meta-characters. Paul Murrell's [_Introduction to Data Technologies_](http://www.stat.auckland.ac.nz/~paul/ItDT/) provides a good introduction in Section 9.9.2 and an extensive reference in Chapter 11. The [Strings chapter](https://r4ds.had.co.nz/strings.html) in [R for Data Science](https://r4ds.had.co.nz/) also provides an introduction to regular expressions, but uses its own set of functions from the `tidyverse`. The web site [Regular-Expressions.info](https://www.regular-expressions.info/) is a useful on-line resource. ## Regular Expression Meta-Charcters Some important meta-characters are `.`, `*`, `+`, and `?`: * The period `.` stands for any character. * The asterisk `*` means zero, one, or more of the preceding character specification. * The plus sign `+` means one, or more of the preceding character specification. * The question mark `?` means zero or one of the preceding character specification. The pattern `",.*"` matches a comma `,` followed by zero or more characters: ```{r} sub(",.*", "", s) ``` The pattern `".*, "` matches zero or more characters followed by a comma and a space: ```{r} sub(".*, ", "", s) ``` ## Trimming White Space If the data file is not consistent on the use of spaces in the separator another possibility is to * remove the characters through the comma; * trim the white space from the result. ```{r} sub(".*,", "", s) ``` ```{r} trimws(sub(".*,", "", s)) ``` ## Using `separate` If the city-state variable is already in a data frame or tibble then the `separate` function from the `tidyr` package can be used: ```{r} library(tibble) library(tidyr) d <- tibble(citystate = s) d ``` ```{r} separate(d, citystate, c("city", "state"), sep = ", ") ``` ## Escaping Meta-Characters Reading data from [city temperatures](https://www.timeanddate.com/weather/) produces a variable that looks like ```{r} s <- c("London *", "Sydney") ``` The `*` indicates daylight saving or summer time. We would like to * extract the city name; * extract whether there is a `*`. The `*` is a meta-character. To include a literal meta-character in a pattern the meta-characters needs to be _escaped_. A meta-character is escaped by preceding it by a backslash `\`. But the backslash is a meta-character for R strings! To put a backslash into an R string it needs to be written as `\\`. The pattern we want to match a space followed by a `*` is `␣\*`, with `␣` denoting a space character. An R string containing these three characters is written as `" \\*"`. It is often useful to write a pattern once and save it in a variable.\: ```{r} (pat <- " \\*") ``` This string contains three characters: ```{r} nchar(pat) ``` Standard printing includes the backslash escape, and other escape characters, so the printed string can be read back into R: ```{r} "a, b and c" ``` The `writeLines` function is useful to see the characters in a string. ```{r} writeLines(pat) ``` To help make the space more visible we can add a delimiter: ```{r} writeLines(paste0("'", pat, "'")) ``` Another option is to use `sprintf` with `writeLines`: ```{r} writeLines(sprintf("'%s'", pat)) ``` This pattern removes the space and asterisk if present: ```{r} s sub(pat, "", s) ``` The `grep` and `grepl` functions check whether a pattern matches in elements of a character vector. * `grep` is short for Get REgular exPression. * `grep` is a standard Linux command-line utility for searching text files. * In R, `grep` returns the indices of the elements that match the pattern. * `grepl` returns a logical vector indicating whether there is a match: ```{r} grep(pat, s) grepl(pat, s) ``` ## Matching Numbers Reading temperature data might produce a string like ```{r} s <- c("32F", "-11F") ``` This can be processed as ```{r} substr(s, 1, nchar(s) - 1) ``` or as ```{r} sub("F", "", s) ``` An alternative uses some more regular expression features: * match the number within the string; * extract a sub-match. A pattern to match an integer, possibly preceded by a sign is ```{r} intpat <- "[-+]?[[:digit:]]+" s sub(intpat, "X", s) ``` The `[` and `]` meta-characters define character sets; any character between these will match. `[:digit:]` specifies a _character class_ of digits. There are a number of _character classes_, including * `[:alpha:]` alphabetic letters; * `[:digit:]` digits; * `[:space:]` white space (spaces, tabs). A _sub_pattern_ can be extracted using _back references_: ```{r} sub("([-+]?[[:digit:]]+).*", "\\1", s) ``` * Sub-patterns can be specified with `(` and `)`. * _Back references_ can be used to refer to previous sub-patterns by number. * The digit needs to be escaped with a `\`; * In an R string the `\` needs to be escaped with a second `\`. A sub-string approach for a temperature embedded in a string: ```{r} s <- c("Temp: 32F", "Temp: -11F") (s1 <- substr(s, 6, nchar(s))) (s2 <- substr(s1, 1, nchar(s1) - 1)) as.numeric(s2) ``` Using regular expressions, sub-patterns, and back references: ```{r} sub(".*[[:space:]]+([-+]?[[:digit:]]+).*", "\\1", s) ``` `parse_number` is again an alternative: ```{r} parse_number(s) ``` ## City Temperatures The [city temperatures](https://www.timeanddate.com/weather/) data used previously can be read using ```{r, message = FALSE} library(rvest) library(dplyr) weather <- read_html("https://www.timeanddate.com/weather/") w <- html_table(html_nodes(weather, "table"))[[1]] w1 <- w[c(1, 4)]; names(w1) <- c("city", "temp") w2 <- w[c(5, 8)]; names(w2) <- c("city", "temp") w3 <- w[c(9, 12)]; names(w3) <- c("city", "temp") ww <- rbind(w1, w2, w3) ww <- filter(ww, city != "") head(ww) ``` Cleaning up and extracting `dst`: ```{r} www <- mutate(ww, dst = grepl(" \\*", city), city = sub(" \\*", "", city), temp.txt = temp, ## for checking on conversion failures` temp = as.numeric(sub("([-+]?[[:digit:]]+).*", "\\1", temp))) ``` Check on `NA` values from conversion: ```{r} filter(www, is.na(temp)) www <- select(www, -temp.txt) ``` Five highest and lowest temperatures: ```{r} slice_max(www, temp, n = 5) ``` ```{r} slice_min(www, temp, n = 5) ``` Temperatures for northern and southern hemisphere (approximately): ```{r temp-densities, eval = FALSE} ggplot(www, aes(x = temp, fill = dst)) + geom_density(alpha = 0.5) ``` ```{r temp-densities, echo = FALSE} ``` ## Tricky Characters Some examples: ```{r} (s <- head(ww$temp)) nchar(s) substr(s, 1, nchar(s) - 3) substr(s, 1, nchar(s) - 2) as.numeric(substr(s, 1, nchar(s) - 2)) as.numeric("82 ") sub(" .*", "", s) ``` The problem is _two_ non-ascii characters. The `stri_escape_unicode` function from the `stringi` can make these characters more visible: ```{r} stringi::stri_escape_unicode(s) ``` The troublesome characters are: * [No-break space U00A0](https://www.fileformat.info/info/unicode/char/00a0/index.htm). * [Degree symbol U00B0](https://www.fileformat.info/info/unicode/char/00b0/index.htm). Using the unicode specification for the no-break space does work: ```{r} sub("\u00a0.*", "", s) ``` ## Variations in Regular Expression Engines Many tools and languages support working with regular expressions. R supports: * The POSIX standard for _extended regular expressions_. This is the default engine. * Perl-compatible regular expressions ([PCRE](https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions). This engine is selected by adding `perl = TRUE` in calls to functions using regular expressions. Different engines can differ in how certain expressions are interpreted, especially when non-ASCII characters are involved. Different engines also sometimes offer shorthand notations, in particular for character classes. Some examples: | POSIX class | similar to | shorthand | | | ----------- | ----------- | --------- | ------------------------------ | |`[:digit:]` | `[0-9]` | `\d` | digits | |`[:upper:]` | `[A-Z]` | `\u` | upper case letters | |`[:lower:]` | `[a-z]` | `\l` | lower-case letters | |`[:alpha:]` | `[A-Za-z]`| | upper- and lower-case letters | |`[:space:]` | `[ \t\n]` | `\s` | whitespace characters | The shorthand versions need to have their `\` escaped when used in an R string: ```{r} intpat <- ".*\\s([-+]?\\d+).*" sub(intpat, "\\1", c("Temp: 32F", "Temp: -11F")) ``` ## Raw Strings _Raw strings_ can make writing regular expressions a little easier. In R a raw string is specifies as `r"(...)"`. The `...` characters can be any characters and are taken literally, without special interpretation that might require escaping. For the integer pattern: ```{r} intpat <- ".*\\s([-+]?\\d+).*" intpat_raw <- r"(.*\s([-+]?\d+).*)" intpat == intpat_raw ``` Python, C++, and other languages provide similar facilites. R's raw string syntax it modeled after the one used in C++. ## A Note on Sorting Non-ASCII characters can create issues for sorting strings, but even ASCII character sort order is not the same in all [locales](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes). The `LETTERS` data set contains the upper-case letters in alphabetical order for the English sorting convention and most other locales: ```{r} LETTERS ``` But in Estonian, Latvian, and Lithuanian: ```{r} stringr::str_sort(LETTERS, locale = "est") ``` Ordering of lower case and upper case letters in English and most other locales: ```{R} stringr::str_sort(c("A", "a"), locale = "eng") ``` But in Danish, and also Maltese: ```{R} stringr::str_sort(c("A", "a"), locale = "dan") ``` ```{r, echo = FALSE, eval = FALSE} ## read a table from Wikipedia with the locale abbreviations url <- "https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes" library(rvest) library(dplyr) w <- read_html(url) ## different letter sort order tbl <- html_table(html_nodes(w, "table"), fill = TRUE) ltbl <- tbl[[2]] lcodes <- ltbl[[6]] v <- sapply(lcodes, function(loc) identical(stringr::str_sort(LETTERS, locale = loc), LETTERS)) ltbl[!v, c("Language family", "ISO language name", "639-2/T")] ## different upper/lower case sort order ss <- c("a", "A") vv <- sapply(lcodes, function(loc) identical(stringr::str_sort(ss, locale = loc), ss)) ltbl[!vv, c("Language family", "ISO language name", "639-2/T")] ``` ## Encoding Issues Files contain a sequence of 8-bit integers, or _bytes_. These are the integers from 0 through 255. For text files, these bytes are interpreted as representing characters. The mapping from bytes to characters is called an _encoding_. The encoding for the characters used in American English is ASCII: the _American Standard Code for Information Interchange_. The [ASCII encoding](https://ascii-tables.com/) uses only the integers 0 through 127. The ASCII encoding is adequate for American uses; even UK text files need more: the pound sign £. Encodings that use integers 128 through 255: * Latin1 (aka ISO-8859-1) for western European languages (includes £); * Latin2 (aka ISO-8859-2) for eastern European languages. Many encodings are available to support other alphabets and character sets. Fortunately most systems now use [Unicode](https://home.unicode.org/) with the [UTF-8 encoding](https://en.wikipedia.org/wiki/UTF-8) for representing non-ASCII characters. If you need to read a file with non-ASCII characters using `read.csv()` or similar base functions a good place to start is to specify `encoding = "UTF-8"`. The functions in the `readr` package default to assuming the encoding is UTF-8. Getting the encoding wrong can result in a few messed up characters or in an entire string being messed up: ```{r include = FALSE} x1 <- "El Ni\xf1o was particularly bad this year" x2 <- "\x82\xb1\x82\xf1\x82\xc9\x82\xbf\x82\xcd" ``` Reading files without specifying the correct encoding might produce a strings like ```{r} x1 x2 ``` If the correct encoding is known, then these can be fixed after the fact with `iconv()`: ```{r} iconv(x1, "Latin1", "UTF-8") iconv(x2, "Shift-JIS", "UTF-8") ``` Re-reading the file with the proper encoding specified may be a better option. The `readr` function `guess_encoding()` may help identify the correct encoding if it is not specified in the data documentation. Handling encoding issues in R can be more complicated on Windows, but this will improve with the next release of R. ## Getting the Current Temperature The tools described here come in handy when scraping data from the web. This code gets the current temperature in Iowa City from the National Weather Service: ```{r, eval = FALSE} library(xml2) url <- "http://forecast.weather.gov/zipcity.php?inputstring=Iowa+City,IA" page <- read_html(url) xpath <- "//p[@class=\"myforecast-current-lrg\"]" tempNode <- xml_find_first(page, xpath) nodeText <- xml_text(tempNode) as.numeric(sub("([-+]?[[:digit:]]+).*", "\\1", nodeText)) ``` An example of creating a current temperature map is described [here](`r here::here("weather.html")`). ## Reading Chapters [_Data Import_](https://r4ds.had.co.nz/data-import.html) and [_Strings_](https://r4ds.had.co.nz/strings.html) in [_R for Data Science_](https://r4ds.had.co.nz/). Chapter [_String processing_](https://rafalab.dfci.harvard.edu/dsbook/) in [_Introduction to Data Science Data Analysis and Prediction Algorithms with R_](https://rafalab.dfci.harvard.edu/dsbook/). ## Exercises 1. Complete all lessons and exercises in the https://regexone.com/ online interactive tutorial. 2. Consider the code ```{r, eval = FALSE} library(tidyverse) filter(mpg, grepl(---, model)) ``` For which of the following regular expressions in place of `---` will this code return the subset of rows for all models that contain either `4wd` or `awd` in their model names? a. "[a4]wd " b. "[a4]wd" c. "4awd" d. "[[4a]]wd"