---
title: "Data and Data Frames"
output:
  html_document:
    toc: yes
    code_download: true
---

<link rel="stylesheet" href="stat4580.css" type="text/css" />

```{r setup, include=FALSE}
source(here::here("setup.R"))
knitr::opts_chunk$set(collapse = TRUE, message = FALSE,
                      fig.height = 5, fig.width = 6, fig.align = "center")
options(width = 80)
```


## Data Structures and Data Attribute Types

Data comes in many different forms.

Some of the most common data structures are

* rectangular tables
* networks and trees
* geometries, regions

Other forms include

* collections of text
* video and audio recordings
* ...

We will work mostly with tables.

Many other forms can be reduced to tables.

[Stevens](https://en.wikipedia.org/wiki/Stanley_Smith_Stevens) (1945)
 classifies [scales of
 measurement](https://en.wikipedia.org/wiki/Level_of_measurement) for
 attributes, or variables, as
<!-- permanent link: https://en.wikipedia.org/w/index.php?title=Level_of_measurement&oldid=1060772847 --> 

* nominal (e.g. hair color)
* ordinal (e.g. dislike, neutral, like)
* interval (e.g. temperature) --- compare by difference
* ratio (e.g. counts) --- compare by ratio

These are sometimes grouped as

* qualitative: nominal
* quantitative: ordinal, interval, ratio

These can be viewed as _semantic classifications_

_Computational considerations_ often classify variables as

* categorical
* integer, discrete
* real, continuous

Another consideration is that some scales may be cyclic:

* hours of the day
* angles, longitude

These distinctions can be important in choosing visual representations.

Other typologies include one proposed by [Mosteller and Tukey
(1977)](https://en.wikipedia.org/wiki/Level_of_measurement#Mosteller_and_Tukey's_typology_(1977)):

1. Names
2. Grades (ordered labels like beginner, intermediate, advanced)
3. Ranks (orders with 1 being the smallest or largest, 2 the next
   smallest or largest, and so on)
4. Counted fractions (bound by 0 and 1)
5. Counts (non-negative integers)
6. Amounts (non-negative real numbers)
7. Balances (any real number)


## Data Types in R

R variables can be of different types.

The most common types are

* `numeric` for real numbers
* `integer`
* `character` for text data or nominal data
* `factor` for nominal or ordinal data

Factors can be

* unordered, for nominal data
* ordered, for ordinal data

<div class = "alert">
`factors` are more efficient and powerful for representing nominal or
ordinal data than `character` data but can take a bit more getting
used to.
</div>

Membership predicates and coercion functions are

| Predicate       | Coersion|
|:----------------|:-----------------|
| `is.numeric`   | `as.numeric`    |
| `is.integer`   | `as.integer`    |
| `is.character` | `as.character`  |
| `is.factor`    | `as.factor`     |
| `is.ordered`   | `as.ordered`    |

<div class = "alert">
Conversion of factors with numeric-looking labels to numeric data
should always go through `as.character` first.
</div>


## Data Frames: Organizing Cases and Variables

Tabular data in R is usually stored as a _data frame_.

A data frame is a collection of _variables_, each with a value for
every _case_ or _observation_.

The `faithful` data set is a data frame:

```{r}
class(faithful)
names(faithful)
```

Most tools we work with in R use data organized in data frames.

Our `plot()` and `lm()` expressions from the
[introductory section](intro.html)
can also we written as

```{r, eval = FALSE}
plot(waiting ~ eruptions, data = faithful,
     xlab = "Eruption time (min)",
     ylab = "Waiting time to next eruption (min)")
fit <- lm(waiting ~ eruptions, data = faithful)
```

<div class = "alert">
`plot()` only uses the `data` argument when the plot is specified as a
_formula_, like

```r
waiting ~ eruptions
```
</div>


## Examining the Data in a Data Frame

`head()` provides an idea of what the raw data looks like:
```{r}
head(faithful)
```

`str()` is also useful for an overview of an object's structure:
```{r}
str(faithful)
```

Another useful function available from the `dplyr` or `tibble`
packages is `glimpse()`:

```{r}
library(dplyr)
glimpse(faithful)
```

`summary()` shows basic statistical properties of the variables:

```{r}
summary(faithful)
```

`summary()` with a character variable and a factor variable:

```{r}
ffl <- mutate(faithful,
              type = ifelse(eruptions < 3, "short", "long"),
              ftype = factor(type))
summary(ffl)
```


## Variables in a Data Frame

A Data frame is a list, or vector, of variables:
```{r}
length(faithful)
```

The dollar sign `$` can be used to examine individual variables:
```{r}
class(faithful$eruptions)
class(faithful$waiting)
```

The variables can also be extracted by numerical or character index using the
  element extraction operation `[[`:
```{r}
class(faithful[[1]])
class(faithful[["waiting"]])
```


## Dimensions

The numbers of rows and columns can be obtained using `nrow()` and `ncol()`:

```{r}
ncol(faithful)
nrow(faithful)
```

`dim()` returns a vector of the dimensions:

```{r}
dim(faithful)
```


## Simple Visualizations

`plot` has a _method_ for data frames that tries to provide a
reasonable default visualization for numeric data frames:
```{r}
plot(faithful)
```


## Sample Data Sets

The
[`datasets`](http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/00Index.html)
package in the base R distribution contains a number of data sets you
can explore.

Another package with a useful collection of data sets is [`dslabs`](https://cran.r-project.org/package=dslabs).

Many other data sets are contained in contributed packages as examples.

There are also many contributed packages designed specifically for
making particular data sets available.


## Tidy Data

The useful concept of a _tidy_ data frame was introduced fairly
[recently](https://www.jstatsoft.org/article/view/v059i10) and is
described in a [chapter in _R for Data
Science_](https://r4ds.hadley.nz/data-tidy.html).

The idea is that

* every observation should correspond to a single row;
* every variable should correspond to a single column.

Tidy data is computationally convenient, and many of the tools we will
use are designed around tidy data frames.

A large range of these tools can be accessed by loading the
`tidyverse` package:

```{r, eval = FALSE}
library(tidyverse)
```

But for now I will load the needed packages individually.

<div class = "alert">
The term _tidy_ is a little unfortunate.

* Data that is not _tidy_ isn't necessarily _bad_.
* For human reading, and  for some computations, data in a wider format can
  be better.
* For other computations data in a longer, or narrower, format can be better.
</div>


## Tibbles

Many tools in the `tidyverse` produce slightly enhanced data frames
called _tibbles_:

```{r}
library(tibble)
faithful_tbl <- as_tibble(faithful)
class(faithful_tbl)
```
Tibbles print differently from standard data frames:

```{r, highlight.output=c(1, 3, 14)}
faithful_tbl
```

For the most part data frames and tibbles can be used interchangeably.


## Tidying Data

Many data sets are in tidy form already.

If they are not, they can be put into tidy form.

The tools for this are part of _data technologies_.

The tasks involved are part of what is sometimes called _data
wrangling_.


## An Example: Global Average Surface Temperatures

Among many data sets available at
<https://data.giss.nasa.gov/gistemp/> is data on monthly global
average surface temperatures over a number of years.

<!-- paywalled now, so use www/archive.org -->

These data from 2017 were used for the widely cited
[Bloomberg hottest year visualization](https://www.bloomberg.com/graphics/hottest-year-on-record/2016/).

[A version from 2015](https://www.bloomberg.com/graphics/hottest-year-on-record/2015/embed/) may work a little better.

The current data are available in a formatted text file at

<https://data.giss.nasa.gov/gistemp/tabledata_v4/GLB.Ts+dSST.txt>

or as a
[_CSV_ (comma-separated values)](https://en.wikipedia.org/wiki/Comma-separated_values) file at

<https://data.giss.nasa.gov/gistemp/tabledata_v4/GLB.Ts+dSST.csv>

```{r download-GLB, echo = FALSE}
```

The first few lines of the CSV file:

```{r}
#| echo: false
#| comment: "   "
readLines("GLB.Ts+dSST.csv", 6) |> writeLines()
```

The CSV file is a little easier (for a computer program) to read in,
so we will work with that.

The numbers in the CSV file represent deviations in degrees Celcius
from the average temperature for the base period 1951-1980.

The file available on January 16, 2025, is now available
[locally](https://stat.uiowa.edu/~luke/data/GLB.Ts+dSST.csv).

We can make sure it has been downloaded to our working directory with

```{r download-GLB}
if (! file.exists("GLB.Ts+dSST.csv"))
    download.file("https://stat.uiowa.edu/~luke/data/GLB.Ts+dSST.csv",
                  "GLB.Ts+dSST.csv")
```

Assuming this locally available file has been downloaded, we can read
in the data and drop some columns we don't need with

```{r}
library(readr)
gast <- read_csv("GLB.Ts+dSST.csv", skip = 1)[1 : 13]
```

<div class = "alert">
* The function `read_csv` is from the `readr` package, which is part of the
  `tidyverse`.
* An alternative is the base R function `read.csv`.
</div>

A look at the first few lines:

```{r}
head(gast, 5)
```

And the last few lines:

```{r}
tail(gast, 5)
```

The `print()` method for tibbles abbreviates the output.

It is neater and provides some useful additional information on
variable data types.

But it shows only the first few rows and may not explicitly show some
columns.

If some columns are skipped, you can ask to see more by calling
`print()` explicitly:

```{r}
print(tail(gast), width = 100)
```

<!--
* The last values in the `Aug` - `Dec` columns are missing and coded as `***`.
* This causes these columns to be read as character vectors indicated
  by `<chr>`.
* The others have been read as numeric, coded `<dbl>` (for 
  _double precision_).
* We can fix the these columns now or deal with them later.

One way to fix them now is to use `mutate_if`:

```{r, warning = FALSE}
gast <- mutate(gast, across(where(is.character), as.numeric))
head(gast)
```

Each observation consists of a year, a month, and a temperature.
-->

The format with one column per month is compact and useful for viewing
and data entry.

But it is not in _tidy format_ since

* the monthly temperature  variable is spread over 12 columns;
* the month variable is encoded in the column headings.

For obvious reasons this data format is often referred to as _wide
format_.

The tidy, or _long_, format would have three variables: `Year`,
`Month`, and `Temp`.

One way to put this data frame in tidy, or long, format uses
`pivot_longer()` from the `tidyr` package:

```{r}
library(tidyr)
lgast <- pivot_longer(gast,
                      -Year,  ## specifies the columns to use -- all but Year
                      names_to = "Month",
                      values_to = "Temp")
```

The first few rows of the result:

```{r}
head(lgast)
```

During plotting it is likely that the `Month` variable will be
converted to a _factor_.

By default, this will order levels alphabetically, which is not what we want:

```{r}
levels(as.factor(lgast$Month))
```

We can guard against this by converting `Month` to a factor with the
right levels now:

```{r}
lgast <- mutate(lgast, Month = factor(Month, levels = month.abb))
levels(lgast$Month)
```

<!-- paywalled now, so use www/archive.org -->

We can now use this tidy version of the data to create a static
version of the [Bloomberg hottest year
visualization](https://web.archive.org/web/20190202194432/https://www.bloomberg.com/graphics/hottest-year-on-record/).

The basic framework is set up with

```{r}
library(ggplot2)
p <- ggplot(lgast) +
    ggtitle("Global Average Surface Temperatures") +
    theme(plot.title = element_text(hjust = 0.5))
```

<div class = "alert">
`ggplot` objects only produce output when they are printed.

To see the plot in `p` we need to print it, for example by using a
line with only `p`.
</div>

```{r gast-base, eval = FALSE}
 p
```
```{r gast-base, eval = TRUE, echo = FALSE}
```

Then add a layer with lines for each year (specified by the `group`
argument to `geom_line`).

```{r gast-lines, eval = FALSE}
p + geom_line(aes(x = Month,
                  y = Temp,
                  group = Year))
```
```{r gast-lines, eval = TRUE, echo = FALSE, warning = FALSE}
```

We can use color to distingish the years.

```{r gast-color1, eval = FALSE}
p1 <- p +
    geom_line(aes(x = Month,
                  y = Temp,
                  color = Year,
                  group = Year),
              na.rm = TRUE)
p1
```
Saving the plot specification in the variable `p1` will make it easier
to experiment with color variations:

```{r gast-color1, eval = TRUE, echo = FALSE}
```

```{r, include = FALSE}
## Compute the past year and make sure it is in the file
library(lubridate)
past_year <- year(today()) - 1
past_year
stopifnot(past_year %in% lgast$Year)
```

One way to highlight the `past_year` `r past_year`:

```{r gast-color2, eval = FALSE}
lgast_last <- filter(lgast, Year == past_year)
p1 + geom_line(aes(x = Month,
                   y = Temp,
                   group = Year),
               linewidth = 1,
               color = "red",
               data = lgast_last,
               na.rm = TRUE)
```
```{r gast-color2, echo = FALSE}
```

A useful way to show more recent data in the context of the full data
is to show the full data in grey and the more recent years in black:

```{r gast-full-grey, eval = FALSE}
lgast2k <- filter(lgast, Year >= 2000)
ggplot(lgast, aes(x = Month,
                  y = Temp,
                  group = Year)) +
    geom_line(color = "grey80") +
    theme_minimal() +
    geom_line(data = lgast2k)
```
```{r gast-full-grey, echo = FALSE, warning = FALSE}
```

If you want to update your plot later in the year then the current
year's entry may contain missing value indicators that you will have
to deal with.
<!--
The version of the data [on the
web](https://data.giss.nasa.gov/gistemp/tabledata_v3/GLB.Ts+dSST.txt)
may have been updated to include the missing values for `r past_year`.
If you want to update your plot later in the year you will see
similar missing value markers for the remaining months of
`r (past_year + 1)`.
-->

The New York Times on January 18, 2018, published
[another visualization](https://www.nytimes.com/interactive/2018/01/18/climate/hottest-year-2017.html)
of these data showing average yearly temperatures ([via Google may work better](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwj_4PmhnM31AhXFjYkEHfQxATEQFnoECAYQAQ&url=https%3A%2F%2Fwww.nytimes.com%2Finteractive%2F2018%2F01%2F18%2Fclimate%2Fhottest-year-2017.html&usg=AOvVaw3qouxHQAJCqrt_F3fbLEGE)).

To recreate this plot we first need to compute yearly average temperatures.

This is easy to do with the `summarize()` and `group_by()` functions
from `dpyr`:

```{r}
library(dplyr)
atemp <- lgast |>
    group_by(Year) |>
    summarize(AveTemp = mean(Temp, na.rm = TRUE))
head(atemp)
```

Using `na.rm = TRUE` ensures that the mean is based on the available
months if data for some months is missing.

A simple version of the plot is then produced by

```{r gast-nyt, eval = FALSE}
ggplot(atemp) +
    geom_point(aes(x = Year, y = AveTemp))
```
```{r gast-nyt, echo = FALSE}
```

A variation showing record years:

```{r  gast-nyt-rec, eval = FALSE}
library(ggrepel)
atemp_rec <- filter(atemp, cummax(AveTemp) == AveTemp)
ggplot(atemp, aes(x = Year, y = AveTemp)) +
    geom_point() +
    geom_point(data = atemp_rec, color = "red") +
    geom_text_repel(aes(label = Year),
                    data = atemp_rec,
                    color = "blue")
```
```{r gast-nyt-rec, echo = FALSE}
```

Matching the representation of record years in the Bloomberg plot:

```{r gast-bb-rec, eval = FALSE}
lgast_rec <- filter(lgast, Year %in% atemp_rec$Year)
ggplot(lgast, aes(x = Month, y = Temp, group = Year)) +
    geom_line(color = "grey80") +
    theme_minimal() +
    geom_line(data = lgast_rec, aes(color = Temp)) +
    scale_color_gradient(low = "blue", high = "red")
```
```{r gast-bb-rec, echo = FALSE}
```

Another variation on the Bloomberg plot showing just a few years 20
years apart:

```{r gast-skip, eval = FALSE}
lg_by_20 <-
    filter(lgast,
           Year %in% seq(2020, by = -20, len = 5)) |>
    mutate(Year = factor(Year))
ggplot(lg_by_20, aes(x = Month,
                     y = Temp,
                     group = Year,
                     color = Year)) +
    geom_line()
```
Converting `Year` to a factor results in a discrete color scale and legend.
```{r gast-skip, echo = FALSE}
```


## Handling Missing Values

The data for 2019 available in early 2020 is also available
[locally](https://stat.uiowa.edu/~luke/data/GLB.Ts+dSST-2019.csv).

```{r, echo = FALSE}
if (! file.exists("GLB.Ts+dSST-2019.csv"))
    download.file("https://stat.uiowa.edu/~luke/data/GLB.Ts+dSST-2019.csv",
                  "GLB.Ts+dSST-2019.csv")
```

Assuming this locally available file has been downloaded, we can read
in the data and drop some columns we don't need with

```{r}
gast2019 <- read_csv("GLB.Ts+dSST-2019.csv", skip = 1)[1 : 13]
```

The last three columns are read as character variables:

```{r}
head(gast2019, 5)
```

The reason is that data for October through December were not
available:

```{r}
tail(gast2019, 2)
```

We want to convert these columns to numeric, with missing values
represented as `NA`.

One option is to handle them individually:

```{r}
gast2019$Oct <- as.numeric(gast2019$Oct)
tail(gast2019, 2)
```

Another option is to convert all character columns to numeric with

```{r}
gast2019 <- mutate(gast2019, across(where(is.character), as.numeric))
tail(gast2019, 2)
```

The warnings are benign and can be suppressed with the `warning =
FALSE` chunk option.

Since we know the missing value pattern `***` we can also avoid the
need to fix the data after the fact by specifying this at read time:

```{r}
gast2019 <- read_csv("GLB.Ts+dSST-2019.csv", na = "***", skip = 1)[1 : 13]
tail(gast2019, 2)
```

A plot highlighting the year 2019 shows only the months with available
data:

```{r gast-2019, eval = FALSE}
lgast2019 <- gast2019 |>
    pivot_longer(-Year,
                 names_to = "Month",
                 values_to = "Temp") |>
    mutate(Month = factor(Month, levels = month.abb))
ggplot(lgast2019, aes(x = Month,
                      y = Temp,
                      group = Year)) +
    geom_line(color = "grey80",
              na.rm = TRUE) +
    geom_line(data = filter(lgast2019, Year == 2019),
              na.rm = TRUE)
```

Adding `na.rm = TRUE` in the `geom_line` calls suppresses warnings;
the plot would be the same without these.

```{r gast-2019, echo = FALSE}
```

<div class = "alert">
Outline of the tools used:

* Data processing:
    * Reading: `read_csv`, `read.csv`;
    * Reshaping: `pivot_longer`;
    * Cleaning: `is.character`, `as.numeric`, `mutate`, `across`, `factor`;
    * Summarizing: `group_by`, `summarize`.
* Visualization geometries:
    * `geom_line` for a line plot;
    * `geom_point` for a scatter plot.
</div>


## Reading

Stevens' classification of scales of measurement is described in a
[Wikipedia
article](https://en.wikipedia.org/wiki/Level_of_measurement).

A good introduction to the concept of _tidy data_ is provided in a
[chapter in _R for Data
Science_](https://r4ds.hadley.nz/data-tidy.html).


## Interactive Tutorial

An interactive [`learnr`](https://rstudio.github.io/learnr/) tutorial
for these notes is [available](`r WLNK("tutorials/datafrm.Rmd")`).

You can run the tutorial with

```{r, eval = FALSE}
STAT4580::runTutorial("datafrm")
```


## Exercises

1. Which of the Stevens classifications (nominal, ordinal, interval, ratio)
   best characterizes these variables:

    a. Daily maximal temperatures in Iowa City.
    b. Population counts for Iowa counties.
    c. Education level of job applicants using the [Bureau of Labor
       Statistics
       classification](https://www.bls.gov/careeroutlook/2014/article/education-level-and-jobs.htm).
    d. Major of UI students.

<!--
Which answers are correct for exercise 1:
* a: interval; b: ratio; c: ordinal; d: nominal
  a: nominal; b: interval; c: ordinal; d: ratio
  a: nominal; b: interval; c: ordinal; d: ratio
  a: interval; b: ratio; c: nominal; d: ordinal
fmt <- function(x)
    paste(letters[seq_along(x)], x, sep = ": ", collapse = "; ")
stev <- c("nominal", "ordinal", "interval", "ratio")
fmt(sample(stev, 4))
-->

2. Which of these data sets are in tidy form?

    a. The builtin data set `co2`
    b. The builtin data set `BOD`
    c. The `who` data set in package `tidyr` (`tidyr::who`)
    d. The `mpg` data set in package `ggplot2` (`ggplot2::mpg`)

<!--
Which answers are correct for exercise 2:
* a: not tidy; b: tidy; c: not tidy; d: tidy
* a: tidy; b: tidy; c: not tidy; d: tidy
* a: not tidy; b: not tidy; c: not tidy; d: tidy
* a: not tidy; b: tidy; c: tidy; d: not tidy
-->

The next exercises use the data in the variable `gapminder` in the package
`gapminder`. You can make it available with

```{r, eval = FALSE}
data(gapminder, package = "gapminder")
```
3. Use the function `str` to examine the value of the gapminder
   variable.  How many cases are there in the data set? How many of
   the variables are factors?

4. Use the functions `class` and `names` to find the class and
   variable names in the `gapminder` data.

5. Use `summary` to compute summary information for the variables.

6. Fill in the values for `---` needed to produce plots of life
   expectancy against year for the countries in continent Oceania.
<!-- ## nolint start -->
```{r, eval = FALSE}
library(dplyr)
library(ggplot2)
data(gapminder, package = "gapminder")
ggplot(filter(gapminder, continent == "Oceania"),
       aes(x = ---, y = ---, color = country)) +
    geom_line()
```
<!-- ## nolint end -->
