Data Cleaning

After reading data from text files or web pages it is common to have to

This can be done using

Some examples of strings that need to be processed:

"12%"
"New York *"
"2,100"
"Temp: 12 \u00b0F"

Some of the most common cases are covered here.

Much more is available in R for Data Science, in particular in the chapters

Removing a Percent Sign

Reading the GDP growth rate data from a web page produced a data frame with a column like

s <- c("12%", "2%")

This can be converted to a numeric variable by

nchar(s)
## [1] 3 2
substr(s, 1, nchar(s) - 1)
## [1] "12" "2"
as.numeric(substr(s, 1, nchar(s) - 1))
## [1] 12  2

An alternative is to use sub() function to replace "%" by the empty string "".

as.numeric(sub("%", "", s))
## [1] 12  2

The function parse_number in the readr package ignores the percent sign and extracts the numbers correctly:

library(readr)
parse_number(s)
## [1] 12  2

Removing Grouping Characters

Numbers are sometimes written using grouping characters:

s1 <- c("800", "2,100")
s2 <- c("800", "2,100", "3,123,500")

sub and gsub can be used to remove grouping characters:

sub(",", "", s1)
## [1] "800"  "2100"
sub(",", "", s2)
## [1] "800"      "2100"     "3123,500"
gsub(",", "", s2)
## [1] "800"     "2100"    "3123500"
as.numeric(gsub(",", "", s2))
## [1]     800    2100 3123500

parse_number can again be used:

parse_number(s2)
## [1]     800    2100 3123500

parse_number is convenient but may be less robust:

parse_number(s2, locale = locale(grouping_mark = "'"))
## [1] 800   2   3

Separating City and State

Data often has city and state specified in a variable like

s <- c("Boston, MA", "Iowa City, IA")

If all state specifications are in two-letter form then city and state can be extracted as sub-strings:

substr(s, 1, nchar(s) - 4)
## [1] "Boston"    "Iowa City"
substr(s, nchar(s) - 1, nchar(s))
## [1] "MA" "IA"

This would not work if full state names are used.

An alternative is to use a regular expression.

Regular Expressions

Regular expressions are a language for expressing patterns in strings.

Regular expressions should be developed carefully, like any program, starting with simple steps and building up.

The simplest regular expressions are literal strings, like %.

More complex expressions are built up using meta-characters that have special meanings in regular expressions.

Many punctuation characters are regular expression meta-characters.

Paul Murrell’s Introduction to Data Technologies provides a good introduction in Section 9.9.2 and an extensive reference in Chapter 11.

The Strings chapter in R for Data Science also provides an introduction to regular expressions, but uses its own set of functions from the tidyverse.

The web site Regular-Expressions.info is a useful on-line resource.

Regular Expression Meta-Charcters

Some important meta-characters are ., *, +, and ?:

The pattern ",.*" matches a comma , followed by zero or more characters:

sub(",.*", "", s)
## [1] "Boston"    "Iowa City"

The pattern ".*, " matches zero or more characters followed by a comma and a space:

sub(".*, ", "", s)
## [1] "MA" "IA"

Trimming White Space

If the data file is not consistent on the use of spaces in the separator another possibility is to

sub(".*,", "", s)
## [1] " MA" " IA"
trimws(sub(".*,", "", s))
## [1] "MA" "IA"

Using separate

If the city-state variable is already in a data frame or tibble then the separate function from the tidyr package can be used:

library(tibble)
library(tidyr)
d <- tibble(citystate = s)
d
## # A tibble: 2 × 1
##   citystate    
##   <chr>        
## 1 Boston, MA   
## 2 Iowa City, IA
separate(d, citystate, c("city", "state"), sep = ", ")
## # A tibble: 2 × 2
##   city      state
##   <chr>     <chr>
## 1 Boston    MA   
## 2 Iowa City IA

Escaping Meta-Characters

Reading data from city temperatures produces a variable that looks like

s <- c("London *", "Sydney")

The * indicates daylight saving or summer time.

We would like to

The * is a meta-character.

To include a literal meta-character in a pattern the meta-characters needs to be escaped.

A meta-character is escaped by preceding it by a backslash \.

But the backslash is a meta-character for R strings!

To put a backslash into an R string it needs to be written as \\.

The pattern we want to match a space followed by a * is ␣\*, with denoting a space character.

An R string containing these three characters is written as " \\*".

It is often useful to write a pattern once and save it in a variable.:

(pat <- " \\*")
## [1] " \\*"

This string contains three characters:

nchar(pat)
## [1] 3

Standard printing includes the backslash escape, and other escape characters, so the printed string can be read back into R:

"a, b
 and c"
## [1] "a, b\n and c"

The writeLines function is useful to see the characters in a string.

writeLines(pat)
##  \*

To help make the space more visible we can add a delimiter:

writeLines(paste0("'", pat, "'"))
## ' \*'

Another option is to use sprintf with writeLines:

writeLines(sprintf("'%s'", pat))
## ' \*'

This pattern removes the space and asterisk if present:

s
## [1] "London *" "Sydney"
sub(pat, "", s)
## [1] "London" "Sydney"

The grep and grepl functions check whether a pattern matches in elements of a character vector.

grep(pat, s)
## [1] 1
grepl(pat, s)
## [1]  TRUE FALSE

Matching Numbers

Reading temperature data might produce a string like

s <- c("32F", "-11F")

This can be processed as

substr(s, 1, nchar(s) - 1)
## [1] "32"  "-11"

or as

sub("F", "", s)
## [1] "32"  "-11"

An alternative uses some more regular expression features:

A pattern to match an integer, possibly preceded by a sign is

intpat <- "[-+]?[[:digit:]]+"
s
## [1] "32F"  "-11F"
sub(intpat, "X", s)
## [1] "XF" "XF"

The [ and ] meta-characters define character sets; any character between these will match.

[:digit:] specifies a character class of digits.

There are a number of character classes, including

A sub_pattern can be extracted using back references:

sub("([-+]?[[:digit:]]+).*", "\\1", s)
## [1] "32"  "-11"

A sub-string approach for a temperature embedded in a string:

s <- c("Temp:  32F", "Temp: -11F")
(s1 <- substr(s, 6, nchar(s)))
## [1] "  32F" " -11F"
(s2 <- substr(s1, 1, nchar(s1) - 1))
## [1] "  32" " -11"
as.numeric(s2)
## [1]  32 -11

Using regular expressions, sub-patterns, and back references:

sub(".*[[:space:]]+([-+]?[[:digit:]]+).*", "\\1", s)
## [1] "32"  "-11"

parse_number is again an alternative:

parse_number(s)
## [1]  32 -11

City Temperatures

The city temperatures data used previously can be read using

library(rvest)
library(dplyr)
weather <- read_html("https://www.timeanddate.com/weather/")
w <- html_table(html_nodes(weather, "table"))[[1]]

w1 <- w[c(1, 4)]; names(w1) <- c("city", "temp")
w2 <- w[c(5, 8)]; names(w2) <- c("city", "temp")
w3 <- w[c(9, 12)]; names(w3) <- c("city", "temp")
ww <- rbind(w1, w2, w3)
ww <- filter(ww, city != "")
head(ww)
## # A tibble: 6 × 2
##   city        temp 
##   <chr>       <chr>
## 1 Accra       77 °F
## 2 Addis Ababa 73 °F
## 3 Adelaide    42 °F
## 4 Algiers     73 °F
## 5 Almaty      46 °F
## 6 Amman       64 °F

Cleaning up and extracting dst:

www <- mutate(ww,
              dst = grepl(" \\*", city),
              city = sub(" \\*", "", city),
              temp.txt = temp,   ## for checking on conversion failures`
              temp = as.numeric(sub("([-+]?[[:digit:]]+).*", "\\1", temp)))

Check on NA values from conversion:

filter(www, is.na(temp))
## # A tibble: 0 × 4
## # ℹ 4 variables: city <chr>, temp <dbl>, dst <lgl>, temp.txt <chr>
www <- select(www, -temp.txt)

Five highest and lowest temperatures:

slice_max(www, temp, n = 5)
## # A tibble: 7 × 3
##   city         temp dst  
##   <chr>       <dbl> <lgl>
## 1 Baghdad        95 FALSE
## 2 Managua        91 FALSE
## 3 Riyadh         91 FALSE
## 4 Bangkok        90 FALSE
## 5 Havana         90 TRUE 
## 6 Kuwait City    90 FALSE
## 7 San Juan       90 FALSE
slice_min(www, temp, n = 5)
## # A tibble: 5 × 3
##   city        temp dst  
##   <chr>      <dbl> <lgl>
## 1 Anadyr        22 FALSE
## 2 St. John's    37 TRUE 
## 3 Anchorage     40 TRUE 
## 4 Melbourne     41 FALSE
## 5 Moscow        41 FALSE

Temperatures for northern and southern hemisphere (approximately):

ggplot(www, aes(x = temp, fill = dst)) +
    geom_density(alpha = 0.5)

Tricky Characters

Some examples:

(s <- head(ww$temp))
## [1] "77 °F" "73 °F" "42 °F" "73 °F" "46 °F" "64 °F"
nchar(s)
## [1] 5 5 5 5 5 5
substr(s, 1, nchar(s) - 3)
## [1] "77" "73" "42" "73" "46" "64"
substr(s, 1, nchar(s) - 2)
## [1] "77 " "73 " "42 " "73 " "46 " "64 "
as.numeric(substr(s, 1, nchar(s) - 2))
## Warning: NAs introduced by coercion
## [1] NA NA NA NA NA NA
as.numeric("82 ")
## [1] 82
sub(" .*", "", s)
## [1] "77 °F" "73 °F" "42 °F" "73 °F" "46 °F" "64 °F"

The problem is two non-ascii characters.

The stri_escape_unicode function from the stringi can make these characters more visible:

stringi::stri_escape_unicode(s)
## [1] "77\\u00a0\\u00b0F" "73\\u00a0\\u00b0F" "42\\u00a0\\u00b0F"
## [4] "73\\u00a0\\u00b0F" "46\\u00a0\\u00b0F" "64\\u00a0\\u00b0F"

The troublesome characters are:

Using the unicode specification for the no-break space does work:

sub("\u00a0.*", "", s)
## [1] "77" "73" "42" "73" "46" "64"

Variations in Regular Expression Engines

Many tools and languages support working with regular expressions.

R supports:

Different engines can differ in how certain expressions are interpreted, especially when non-ASCII characters are involved.

Different engines also sometimes offer shorthand notations, in particular for character classes.

Some examples:

POSIX class similar to shorthand
[:digit:] [0-9] \d digits
[:upper:] [A-Z] \u upper case letters
[:lower:] [a-z] \l lower-case letters
[:alpha:] [A-Za-z] upper- and lower-case letters
[:space:] [ \t\n] \s whitespace characters

The shorthand versions need to have their \ escaped when used in an R string:

intpat <- ".*\\s([-+]?\\d+).*"
sub(intpat, "\\1", c("Temp:  32F", "Temp: -11F"))
## [1] "32"  "-11"

Raw Strings

Raw strings can make writing regular expressions a little easier.

In R a raw string is specifies as r"(...)".

The ... characters can be any characters and are taken literally, without special interpretation that might require escaping.

For the integer pattern:

intpat <- ".*\\s([-+]?\\d+).*"
intpat_raw <- r"(.*\s([-+]?\d+).*)"
intpat == intpat_raw
## [1] TRUE

Python, C++, and other languages provide similar facilites.

R’s raw string syntax it modeled after the one used in C++.

A Note on Sorting

Non-ASCII characters can create issues for sorting strings, but even ASCII character sort order is not the same in all locales.

The LETTERS data set contains the upper-case letters in alphabetical order for the English sorting convention and most other locales:

LETTERS
##  [1] "A" "B" "C" "D" "E" "F" "G" "H" "I" "J" "K" "L" "M" "N" "O" "P" "Q" "R" "S"
## [20] "T" "U" "V" "W" "X" "Y" "Z"

But in Estonian, Latvian, and Lithuanian:

stringr::str_sort(LETTERS, locale = "est")
##  [1] "A" "B" "C" "D" "E" "F" "G" "H" "I" "J" "K" "L" "M" "N" "O" "P" "Q" "R" "S"
## [20] "Z" "T" "U" "V" "W" "X" "Y"

Ordering of lower case and upper case letters in English and most other locales:

stringr::str_sort(c("A", "a"), locale = "eng")
## [1] "a" "A"

But in Danish, and also Maltese:

stringr::str_sort(c("A", "a"), locale = "dan")
## [1] "A" "a"

Encoding Issues

Files contain a sequence of 8-bit integers, or bytes.

These are the integers from 0 through 255.

For text files, these bytes are interpreted as representing characters.

The mapping from bytes to characters is called an encoding.

The encoding for the characters used in American English is ASCII: the American Standard Code for Information Interchange.

The ASCII encoding uses only the integers 0 through 127.

The ASCII encoding is adequate for American uses; even UK text files need more: the pound sign £.

Encodings that use integers 128 through 255:

Many encodings are available to support other alphabets and character sets.

Fortunately most systems now use Unicode with the UTF-8 encoding for representing non-ASCII characters.

If you need to read a file with non-ASCII characters using read.csv() or similar base functions a good place to start is to specify encoding = "UTF-8".

The functions in the readr package default to assuming the encoding is UTF-8.

Getting the encoding wrong can result in a few messed up characters or in an entire string being messed up:

Reading files without specifying the correct encoding might produce a strings like

x1
## [1] "El Ni\xf1o was particularly bad this year"
x2
## [1] "\x82\xb1\x82\xf1\x82ɂ\xbf\x82\xcd"

If the correct encoding is known, then these can be fixed after the fact with iconv():

iconv(x1, "Latin1", "UTF-8")
## [1] "El Niño was particularly bad this year"
iconv(x2, "Shift-JIS", "UTF-8")
## [1] "こんにちは"

Re-reading the file with the proper encoding specified may be a better option.

The readr function guess_encoding() may help identify the correct encoding if it is not specified in the data documentation.

Handling encoding issues in R can be more complicated on Windows, but this will improve with the next release of R.

Getting the Current Temperature

The tools described here come in handy when scraping data from the web.

This code gets the current temperature in Iowa City from the National Weather Service:

library(xml2)
url <- "http://forecast.weather.gov/zipcity.php?inputstring=Iowa+City,IA"
page <- read_html(url)
xpath <- "//p[@class=\"myforecast-current-lrg\"]"
tempNode <- xml_find_first(page, xpath)
nodeText <- xml_text(tempNode)
as.numeric(sub("([-+]?[[:digit:]]+).*", "\\1", nodeText))

An example of creating a current temperature map is described here.

Reading

Chapters Data Import and Strings in R for Data Science.

Chapter String processing in Introduction to Data Science Data Analysis and Prediction Algorithms with R.

Exercises

  1. Complete all lessons and exercises in the https://regexone.com/ online interactive tutorial.

  2. Consider the code

    library(tidyverse)
    filter(mpg, grepl(---, model))

    For which of the following regular expressions in place of --- will this code return the subset of rows for all models that contain either 4wd or awd in their model names?

    1. “[a4]wd”
    2. “[a4]wd”
    3. “4awd”
    4. “[[4a]]wd”
LS0tCnRpdGxlOiAiU3RyaW5nIFBhcnNpbmcgYW5kIFJlZ3VsYXIgRXhwcmVzc2lvbnMiCm91dHB1dDoKICBodG1sX2RvY3VtZW50OgogICAgdG9jOiB5ZXMKICAgIGNvZGVfZm9sZGluZzogc2hvdwogICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0ic3RhdDQ1ODAuY3NzIiB0eXBlPSJ0ZXh0L2NzcyIgLz4KPHN0eWxlIHR5cGU9InRleHQvY3NzIj4gLnJlbWFyay1jb2RlIHsgZm9udC1zaXplOiA4NSU7IH0gPC9zdHlsZT4KYGBge3Igc2V0dXAsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKGhlcmU6OmhlcmUoInNldHVwLlIiKSkKa25pdHI6Om9wdHNfY2h1bmskc2V0KGNvbGxhcHNlID0gVFJVRSwgbWVzc2FnZSA9IEZBTFNFLAogICAgICAgICAgICAgICAgICAgICAgZmlnLmhlaWdodCA9IDUsIGZpZy53aWR0aCA9IDYsIGZpZy5hbGlnbiA9ICJjZW50ZXIiKQoKb3B0aW9ucyhodG1sdG9vbHMuZGlyLnZlcnNpb24gPSBGQUxTRSkKCnNldC5zZWVkKDEyMzQ1KQpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkobGF0dGljZSkKbGlicmFyeSh0aWR5dmVyc2UpCmxpYnJhcnkoZ3JpZEV4dHJhKQp0aGVtZV9zZXQodGhlbWVfbWluaW1hbCgpICsKICAgICAgICAgIHRoZW1lKHRleHQgPSBlbGVtZW50X3RleHQoc2l6ZSA9IDE2KSwKICAgICAgICAgICAgICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfcmVjdChjb2xvciA9ICJncmV5MzAiLCBmaWxsID0gTkEpKSkKYGBgCgoKIyMgRGF0YSBDbGVhbmluZwoKQWZ0ZXIgcmVhZGluZyBkYXRhIGZyb20gdGV4dCBmaWxlcyBvciB3ZWIgcGFnZXMgaXQgaXMgY29tbW9uIHRvIGhhdmUgdG8KCiogY2xlYW4gdXAgc3RyaW5nIHZhcmlhYmxlczsKCiogZXh0cmFjdCBudW1iZXJzLgoKVGhpcyBjYW4gYmUgZG9uZSB1c2luZwoKKiBoaWdoLWxldmVsIGZ1bmN0aW9ucyB0aGF0IHVzdWFsbHkgd29yayBidXQgbm90IGFsd2F5czsKCiogdXNpbmcgc3RyaW5nIHN1YnNldHRpbmc7CgoqIHVzaW5nIF9yZWd1bGFyIGV4cHJlc3Npb25zXy4KClNvbWUgZXhhbXBsZXMgb2Ygc3RyaW5ncyB0aGF0IG5lZWQgdG8gYmUgcHJvY2Vzc2VkOgoKYGBge3IsIGV2YWwgPSBGQUxTRX0KIjEyJSIKIk5ldyBZb3JrICoiCiIyLDEwMCIKIlRlbXA6IDEyIFx1MDBiMEYiCmBgYAoKU29tZSBvZiB0aGUgbW9zdCBjb21tb24gY2FzZXMgYXJlIGNvdmVyZWQgaGVyZS4KCk11Y2ggbW9yZSBpcyBhdmFpbGFibGUgaW4gW19SIGZvciBEYXRhClNjaWVuY2VfXShodHRwczovL3I0ZHMuaGFkLmNvLm56LyksIGluIHBhcnRpY3VsYXIgaW4gdGhlIGNoYXB0ZXJzCgoqIFtfRGF0YSBJbXBvcnRfXShodHRwczovL3I0ZHMuaGFkLmNvLm56L2RhdGEtaW1wb3J0Lmh0bWwpOwoqIFtfU3RyaW5nc19dKGh0dHBzOi8vcjRkcy5oYWQuY28ubnovc3RyaW5ncy5odG1sKS4KCgojIyBSZW1vdmluZyBhIFBlcmNlbnQgU2lnbgoKUmVhZGluZyB0aGUgR0RQIGdyb3d0aCByYXRlIGRhdGEgZnJvbSBhIApbd2ViIHBhZ2VdKGh0dHBzOi8vd3d3Lm11bHRwbC5jb20vdXMtcmVhbC1nZHAtZ3Jvd3RoLXJhdGUvdGFibGUvYnktcXVhcnRlcikKcHJvZHVjZWQgYSBkYXRhIGZyYW1lIHdpdGggYSBjb2x1bW4gbGlrZQoKYGBge3J9CnMgPC0gYygiMTIlIiwgIjIlIikKYGBgCgpUaGlzIGNhbiBiZSBjb252ZXJ0ZWQgdG8gYSBudW1lcmljIHZhcmlhYmxlIGJ5CgoqIGV4dHJhY3RpbmcgdGhlIHN1Yi1zdHJpbmcgd2l0aG91dCB0aGUgYCVgCgoqIGFuZCB0aGVuIHVzaW5nIGBhcy5udW1lcmljYC4KCmBgYHtyfQpuY2hhcihzKQpzdWJzdHIocywgMSwgbmNoYXIocykgLSAxKQphcy5udW1lcmljKHN1YnN0cihzLCAxLCBuY2hhcihzKSAtIDEpKQpgYGAKCkFuIGFsdGVybmF0aXZlIGlzIHRvIHVzZSBgc3ViKClgIGZ1bmN0aW9uIHRvIHJlcGxhY2UgYCIlImAgYnkgdGhlCmVtcHR5IHN0cmluZyBgIiJgLgoKYGBge3J9CmFzLm51bWVyaWMoc3ViKCIlIiwgIiIsIHMpKQpgYGAKClRoZSBmdW5jdGlvbiBgcGFyc2VfbnVtYmVyYCBpbiB0aGUgYHJlYWRyYCBwYWNrYWdlIGlnbm9yZXMgdGhlIHBlcmNlbnQKc2lnbiBhbmQgZXh0cmFjdHMgdGhlIG51bWJlcnMgY29ycmVjdGx5OgoKYGBge3J9CmxpYnJhcnkocmVhZHIpCnBhcnNlX251bWJlcihzKQpgYGAKCgojIyBSZW1vdmluZyBHcm91cGluZyBDaGFyYWN0ZXJzCgpOdW1iZXJzIGFyZSBzb21ldGltZXMgd3JpdHRlbiB1c2luZyBfZ3JvdXBpbmcgY2hhcmFjdGVyc186CgpgYGB7cn0KczEgPC0gYygiODAwIiwgIjIsMTAwIikKczIgPC0gYygiODAwIiwgIjIsMTAwIiwgIjMsMTIzLDUwMCIpCmBgYAoKKiBUaGUgY29tbWEgaXMgb2Z0ZW4gdXNlZCBhcyBhIGdyb3VwaW5nIGNoYXJhY3RlciBpbiB0aGUgVVMuCgoqIE90aGVyIGNvdW50cmllcyB1c2UgZGlmZmVyZW50IGNoYXJhY3RlcnMuCgoqIE90aGVyIGNvdW50cmllcyBhbHNvIHVzZSBkaWZmZXJlbnQgY2hhcmFjdGVycyBmb3IgdGhlIGRlY2ltYWwgc2VwYXJhdG9yLgoKYHN1YmAgYW5kIGBnc3ViYCBjYW4gYmUgdXNlZCB0byByZW1vdmUgZ3JvdXBpbmcgY2hhcmFjdGVyczoKCmBgYHtyfQpzdWIoIiwiLCAiIiwgczEpCnN1YigiLCIsICIiLCBzMikKYGBgCgpgYGB7cn0KZ3N1YigiLCIsICIiLCBzMikKYXMubnVtZXJpYyhnc3ViKCIsIiwgIiIsIHMyKSkKYGBgCgoqIGBzdWJgIHJlcGxhY2VzIHRoZSBmaXJzdCBtYXRjaCB0byBhIHBhdHRlcm47CgoqIGBnc3ViYCByZXBsYWNlcyBhbGwgbWF0Y2hlcy4KCmBwYXJzZV9udW1iZXJgIGNhbiBhZ2FpbiBiZSB1c2VkOgoKYGBge3J9CnBhcnNlX251bWJlcihzMikKYGBgCgpgcGFyc2VfbnVtYmVyYCBpcyBjb252ZW5pZW50IGJ1dCBtYXkgYmUgbGVzcyByb2J1c3Q6CgoqIEluIFN3aXR6ZXJsYW5kIHRoZSBncm91cGluZyBjaGFyYWN0ZXIgaXMgYCdgLgoKKiBJZiBgcGFyc2VfbnVtYmVyYCBoYXMgaXRzIGRlZmF1bHRzIHNldCB0byBTd2lzcyBjb252ZW50aW9uczoKCmBgYHtyfQpwYXJzZV9udW1iZXIoczIsIGxvY2FsZSA9IGxvY2FsZShncm91cGluZ19tYXJrID0gIiciKSkKYGBgCgojIyBTZXBhcmF0aW5nIENpdHkgYW5kIFN0YXRlCgpEYXRhIG9mdGVuIGhhcyBjaXR5IGFuZCBzdGF0ZSBzcGVjaWZpZWQgaW4gYSB2YXJpYWJsZSBsaWtlCgpgYGB7cn0KcyA8LSBjKCJCb3N0b24sIE1BIiwgIklvd2EgQ2l0eSwgSUEiKQpgYGAKCklmIGFsbCBzdGF0ZSBzcGVjaWZpY2F0aW9ucyBhcmUgaW4gdHdvLWxldHRlciBmb3JtIHRoZW4gY2l0eSBhbmQgc3RhdGUKY2FuIGJlIGV4dHJhY3RlZCBhcyBzdWItc3RyaW5nczoKCmBgYHtyfQpzdWJzdHIocywgMSwgbmNoYXIocykgLSA0KQpzdWJzdHIocywgbmNoYXIocykgLSAxLCBuY2hhcihzKSkKYGBgCgpUaGlzIHdvdWxkIG5vdCB3b3JrIGlmIGZ1bGwgc3RhdGUgbmFtZXMgYXJlIHVzZWQuCgpBbiBhbHRlcm5hdGl2ZSBpcyB0byB1c2UgYSBfcmVndWxhciBleHByZXNzaW9uXy4KCgojIyBSZWd1bGFyIEV4cHJlc3Npb25zCgpSZWd1bGFyIGV4cHJlc3Npb25zIGFyZSBhIGxhbmd1YWdlIGZvciBleHByZXNzaW5nIHBhdHRlcm5zIGluIHN0cmluZ3MuCgpSZWd1bGFyIGV4cHJlc3Npb25zIHNob3VsZCBiZSBkZXZlbG9wZWQgY2FyZWZ1bGx5LCBsaWtlIGFueSBwcm9ncmFtLApzdGFydGluZyB3aXRoIHNpbXBsZSBzdGVwcyBhbmQgYnVpbGRpbmcgdXAuCgpUaGUgc2ltcGxlc3QgcmVndWxhciBleHByZXNzaW9ucyBhcmUgbGl0ZXJhbCBzdHJpbmdzLCBsaWtlIGAlYC4KCk1vcmUgY29tcGxleCBleHByZXNzaW9ucyBhcmUgYnVpbHQgdXAgdXNpbmcgX21ldGEtY2hhcmFjdGVyc18gdGhhdApoYXZlIHNwZWNpYWwgbWVhbmluZ3MgaW4gcmVndWxhciBleHByZXNzaW9ucy4KCk1hbnkgcHVuY3R1YXRpb24gY2hhcmFjdGVycyBhcmUgcmVndWxhciBleHByZXNzaW9uIG1ldGEtY2hhcmFjdGVycy4KClBhdWwgTXVycmVsbCdzIFtfSW50cm9kdWN0aW9uIHRvIERhdGEKICBUZWNobm9sb2dpZXNfXShodHRwOi8vd3d3LnN0YXQuYXVja2xhbmQuYWMubnovfnBhdWwvSXREVC8pIHByb3ZpZGVzCiAgYSBnb29kIGludHJvZHVjdGlvbiBpbiBTZWN0aW9uIDkuOS4yIGFuZCBhbiBleHRlbnNpdmUgcmVmZXJlbmNlIGluCiAgQ2hhcHRlciAxMS4KClRoZSBbU3RyaW5ncyBjaGFwdGVyXShodHRwczovL3I0ZHMuaGFkLmNvLm56L3N0cmluZ3MuaHRtbCkgaW4gW1IgZm9yCkRhdGEgU2NpZW5jZV0oaHR0cHM6Ly9yNGRzLmhhZC5jby5uei8pIGFsc28gcHJvdmlkZXMgYW4gaW50cm9kdWN0aW9uIHRvCnJlZ3VsYXIgZXhwcmVzc2lvbnMsIGJ1dCB1c2VzIGl0cyBvd24gc2V0IG9mIGZ1bmN0aW9ucyBmcm9tIHRoZQpgdGlkeXZlcnNlYC4KClRoZSB3ZWIgc2l0ZQpbUmVndWxhci1FeHByZXNzaW9ucy5pbmZvXShodHRwczovL3d3dy5yZWd1bGFyLWV4cHJlc3Npb25zLmluZm8vKSBpcyBhCnVzZWZ1bCBvbi1saW5lIHJlc291cmNlLgoKCiMjIFJlZ3VsYXIgRXhwcmVzc2lvbiBNZXRhLUNoYXJjdGVycwoKU29tZSBpbXBvcnRhbnQgbWV0YS1jaGFyYWN0ZXJzIGFyZSBgLmAsIGAqYCwgYCtgLCBhbmQgYD9gOgoKKiBUaGUgcGVyaW9kIGAuYCBzdGFuZHMgZm9yIGFueSBjaGFyYWN0ZXIuCgoqIFRoZSBhc3RlcmlzayBgKmAgbWVhbnMgemVybywgb25lLCBvciBtb3JlIG9mIHRoZSBwcmVjZWRpbmcgY2hhcmFjdGVyCiAgc3BlY2lmaWNhdGlvbi4KCiogVGhlIHBsdXMgc2lnbiBgK2AgbWVhbnMgb25lLCBvciBtb3JlIG9mIHRoZSBwcmVjZWRpbmcgY2hhcmFjdGVyCiAgc3BlY2lmaWNhdGlvbi4KCiogVGhlIHF1ZXN0aW9uIG1hcmsgYD9gIG1lYW5zIHplcm8gb3Igb25lIG9mIHRoZSBwcmVjZWRpbmcgY2hhcmFjdGVyCiAgc3BlY2lmaWNhdGlvbi4KClRoZSBwYXR0ZXJuIGAiLC4qImAgbWF0Y2hlcyBhIGNvbW1hIGAsYCBmb2xsb3dlZCBieSB6ZXJvIG9yIG1vcmUKY2hhcmFjdGVyczoKCmBgYHtyfQpzdWIoIiwuKiIsICIiLCBzKQpgYGAKClRoZSBwYXR0ZXJuIGAiLiosICJgIG1hdGNoZXMgemVybyBvciBtb3JlIGNoYXJhY3RlcnMgZm9sbG93ZWQgYnkgYQpjb21tYSBhbmQgYSBzcGFjZToKCmBgYHtyfQpzdWIoIi4qLCAiLCAiIiwgcykKYGBgCgoKIyMgVHJpbW1pbmcgV2hpdGUgU3BhY2UKCklmIHRoZSBkYXRhIGZpbGUgaXMgbm90IGNvbnNpc3RlbnQgb24gdGhlIHVzZSBvZiBzcGFjZXMgaW4gdGhlCnNlcGFyYXRvciBhbm90aGVyIHBvc3NpYmlsaXR5IGlzIHRvCgoqIHJlbW92ZSB0aGUgY2hhcmFjdGVycyB0aHJvdWdoIHRoZSBjb21tYTsKCiogdHJpbSB0aGUgd2hpdGUgc3BhY2UgZnJvbSB0aGUgcmVzdWx0LgoKYGBge3J9CnN1YigiLiosIiwgIiIsIHMpCmBgYAoKYGBge3J9CnRyaW13cyhzdWIoIi4qLCIsICIiLCBzKSkKYGBgCgoKIyMgVXNpbmcgYHNlcGFyYXRlYAoKSWYgdGhlIGNpdHktc3RhdGUgdmFyaWFibGUgaXMgYWxyZWFkeSBpbiBhIGRhdGEgZnJhbWUgb3IgdGliYmxlIHRoZW4KdGhlIGBzZXBhcmF0ZWAgZnVuY3Rpb24gZnJvbSB0aGUgYHRpZHlyYCBwYWNrYWdlIGNhbiBiZSB1c2VkOgoKYGBge3J9CmxpYnJhcnkodGliYmxlKQpsaWJyYXJ5KHRpZHlyKQpkIDwtIHRpYmJsZShjaXR5c3RhdGUgPSBzKQpkCmBgYAoKYGBge3J9CnNlcGFyYXRlKGQsIGNpdHlzdGF0ZSwgYygiY2l0eSIsICJzdGF0ZSIpLCBzZXAgPSAiLCAiKQpgYGAKCgojIyBFc2NhcGluZyBNZXRhLUNoYXJhY3RlcnMKClJlYWRpbmcgZGF0YSBmcm9tCltjaXR5IHRlbXBlcmF0dXJlc10oaHR0cHM6Ly93d3cudGltZWFuZGRhdGUuY29tL3dlYXRoZXIvKQpwcm9kdWNlcyBhIHZhcmlhYmxlIHRoYXQgbG9va3MgbGlrZQoKYGBge3J9CnMgPC0gYygiTG9uZG9uICoiLCAiU3lkbmV5IikKYGBgCgpUaGUgYCpgIGluZGljYXRlcyBkYXlsaWdodCBzYXZpbmcgb3Igc3VtbWVyIHRpbWUuCgpXZSB3b3VsZCBsaWtlIHRvCgoqIGV4dHJhY3QgdGhlIGNpdHkgbmFtZTsKCiogZXh0cmFjdCB3aGV0aGVyIHRoZXJlIGlzIGEgYCpgLgoKVGhlIGAqYCBpcyBhIG1ldGEtY2hhcmFjdGVyLgoKVG8gaW5jbHVkZSBhIGxpdGVyYWwgbWV0YS1jaGFyYWN0ZXIgaW4gYSBwYXR0ZXJuIHRoZSBtZXRhLWNoYXJhY3RlcnMKbmVlZHMgdG8gYmUgX2VzY2FwZWRfLgoKQSBtZXRhLWNoYXJhY3RlciBpcyBlc2NhcGVkIGJ5IHByZWNlZGluZyBpdCBieSBhIGJhY2tzbGFzaCBgXGAuCgpCdXQgdGhlIGJhY2tzbGFzaCBpcyBhIG1ldGEtY2hhcmFjdGVyIGZvciBSIHN0cmluZ3MhCgpUbyBwdXQgYSBiYWNrc2xhc2ggaW50byBhbiBSIHN0cmluZyBpdCBuZWVkcyB0byBiZSB3cml0dGVuIGFzIGBcXGAuCgpUaGUgcGF0dGVybiB3ZSB3YW50IHRvIG1hdGNoIGEgc3BhY2UgZm9sbG93ZWQgYnkgYSBgKmAgaXMgCmDikKNcKmAsIHdpdGggYOKQo2AgZGVub3RpbmcgYSBzcGFjZSBjaGFyYWN0ZXIuCgpBbiBSIHN0cmluZyBjb250YWluaW5nIHRoZXNlIHRocmVlIGNoYXJhY3RlcnMgaXMgd3JpdHRlbiBhcyBgIiBcXCoiYC4KCkl0IGlzIG9mdGVuIHVzZWZ1bCB0byB3cml0ZSBhIHBhdHRlcm4gb25jZSBhbmQgc2F2ZSBpdCBpbiBhIHZhcmlhYmxlLlw6CgpgYGB7cn0KKHBhdCA8LSAiIFxcKiIpCmBgYAoKVGhpcyBzdHJpbmcgY29udGFpbnMgdGhyZWUgY2hhcmFjdGVyczoKCmBgYHtyfQpuY2hhcihwYXQpCmBgYAoKU3RhbmRhcmQgcHJpbnRpbmcgaW5jbHVkZXMgdGhlIGJhY2tzbGFzaCBlc2NhcGUsIGFuZCBvdGhlciBlc2NhcGUKY2hhcmFjdGVycywgc28gdGhlIHByaW50ZWQgc3RyaW5nIGNhbiBiZSByZWFkIGJhY2sgaW50byBSOgoKYGBge3J9CiJhLCBiCiBhbmQgYyIKYGBgCgpUaGUgYHdyaXRlTGluZXNgIGZ1bmN0aW9uIGlzIHVzZWZ1bCB0byBzZWUgdGhlIGNoYXJhY3RlcnMgaW4gYSBzdHJpbmcuCgpgYGB7cn0Kd3JpdGVMaW5lcyhwYXQpCmBgYAoKVG8gaGVscCBtYWtlIHRoZSBzcGFjZSBtb3JlIHZpc2libGUgd2UgY2FuIGFkZCBhIGRlbGltaXRlcjoKCmBgYHtyfQp3cml0ZUxpbmVzKHBhc3RlMCgiJyIsIHBhdCwgIiciKSkKYGBgCgpBbm90aGVyIG9wdGlvbiBpcyB0byB1c2UgYHNwcmludGZgIHdpdGggYHdyaXRlTGluZXNgOgoKYGBge3J9CndyaXRlTGluZXMoc3ByaW50ZigiJyVzJyIsIHBhdCkpCmBgYAoKVGhpcyBwYXR0ZXJuIHJlbW92ZXMgdGhlIHNwYWNlIGFuZCBhc3RlcmlzayBpZiBwcmVzZW50OgoKYGBge3J9CnMKc3ViKHBhdCwgIiIsIHMpCmBgYAoKVGhlIGBncmVwYCBhbmQgYGdyZXBsYCBmdW5jdGlvbnMgY2hlY2sgd2hldGhlciBhIHBhdHRlcm4gbWF0Y2hlcyBpbgplbGVtZW50cyBvZiBhIGNoYXJhY3RlciB2ZWN0b3IuCgoqIGBncmVwYCBpcyBzaG9ydCBmb3IgR2V0IFJFZ3VsYXIgZXhQcmVzc2lvbi4KCiogYGdyZXBgIGlzIGEgc3RhbmRhcmQgTGludXggY29tbWFuZC1saW5lIHV0aWxpdHkgZm9yIHNlYXJjaGluZyB0ZXh0IGZpbGVzLgoKKiBJbiBSLCBgZ3JlcGAgcmV0dXJucyB0aGUgaW5kaWNlcyBvZiB0aGUgZWxlbWVudHMgdGhhdCBtYXRjaCB0aGUgcGF0dGVybi4KCiogYGdyZXBsYCByZXR1cm5zIGEgbG9naWNhbCB2ZWN0b3IgaW5kaWNhdGluZyB3aGV0aGVyIHRoZXJlIGlzIGEgbWF0Y2g6CgpgYGB7cn0KZ3JlcChwYXQsIHMpCmdyZXBsKHBhdCwgcykKYGBgCgoKIyMgTWF0Y2hpbmcgTnVtYmVycwoKUmVhZGluZyB0ZW1wZXJhdHVyZSBkYXRhIG1pZ2h0IHByb2R1Y2UgYSBzdHJpbmcgbGlrZQoKYGBge3J9CnMgPC0gYygiMzJGIiwgIi0xMUYiKQpgYGAKClRoaXMgY2FuIGJlIHByb2Nlc3NlZCBhcwoKYGBge3J9CnN1YnN0cihzLCAxLCBuY2hhcihzKSAtIDEpCmBgYAoKb3IgYXMKCmBgYHtyfQpzdWIoIkYiLCAiIiwgcykKYGBgCgpBbiBhbHRlcm5hdGl2ZSB1c2VzIHNvbWUgbW9yZSByZWd1bGFyIGV4cHJlc3Npb24gZmVhdHVyZXM6CgoqIG1hdGNoIHRoZSBudW1iZXIgd2l0aGluIHRoZSBzdHJpbmc7CgoqIGV4dHJhY3QgYSBzdWItbWF0Y2guCgpBIHBhdHRlcm4gdG8gbWF0Y2ggYW4gaW50ZWdlciwgcG9zc2libHkgcHJlY2VkZWQgYnkgYSBzaWduIGlzCgpgYGB7cn0KaW50cGF0IDwtICJbLStdP1tbOmRpZ2l0Ol1dKyIKcwpzdWIoaW50cGF0LCAiWCIsIHMpCmBgYAoKVGhlIGBbYCBhbmQgYF1gIG1ldGEtY2hhcmFjdGVycyBkZWZpbmUgY2hhcmFjdGVyIHNldHM7IGFueSBjaGFyYWN0ZXIKYmV0d2VlbiB0aGVzZSB3aWxsIG1hdGNoLgoKYFs6ZGlnaXQ6XWAgc3BlY2lmaWVzIGEgX2NoYXJhY3RlciBjbGFzc18gb2YgZGlnaXRzLgoKVGhlcmUgYXJlIGEgbnVtYmVyIG9mIF9jaGFyYWN0ZXIgY2xhc3Nlc18sIGluY2x1ZGluZwoKKiBgWzphbHBoYTpdYCBhbHBoYWJldGljIGxldHRlcnM7CgoqIGBbOmRpZ2l0Ol1gIGRpZ2l0czsKCiogYFs6c3BhY2U6XWAgd2hpdGUgc3BhY2UgKHNwYWNlcywgdGFicykuCgpBIF9zdWJfcGF0dGVybl8gY2FuIGJlIGV4dHJhY3RlZCB1c2luZyBfYmFjayByZWZlcmVuY2VzXzoKCmBgYHtyfQpzdWIoIihbLStdP1tbOmRpZ2l0Ol1dKykuKiIsICJcXDEiLCBzKQpgYGAKCiogU3ViLXBhdHRlcm5zIGNhbiBiZSBzcGVjaWZpZWQgd2l0aCBgKGAgYW5kIGApYC4KCiogX0JhY2sgcmVmZXJlbmNlc18gY2FuIGJlIHVzZWQgdG8gcmVmZXIgdG8gcHJldmlvdXMgc3ViLXBhdHRlcm5zIGJ5IG51bWJlci4KCiogVGhlIGRpZ2l0IG5lZWRzIHRvIGJlIGVzY2FwZWQgd2l0aCBhIGBcYDsKCiogSW4gYW4gUiBzdHJpbmcgdGhlIGBcYCBuZWVkcyB0byBiZSBlc2NhcGVkIHdpdGggYSBzZWNvbmQgYFxgLgoKQSBzdWItc3RyaW5nIGFwcHJvYWNoIGZvciBhIHRlbXBlcmF0dXJlIGVtYmVkZGVkIGluIGEgc3RyaW5nOgoKYGBge3J9CnMgPC0gYygiVGVtcDogIDMyRiIsICJUZW1wOiAtMTFGIikKKHMxIDwtIHN1YnN0cihzLCA2LCBuY2hhcihzKSkpCihzMiA8LSBzdWJzdHIoczEsIDEsIG5jaGFyKHMxKSAtIDEpKQphcy5udW1lcmljKHMyKQpgYGAKClVzaW5nIHJlZ3VsYXIgZXhwcmVzc2lvbnMsIHN1Yi1wYXR0ZXJucywgYW5kIGJhY2sgcmVmZXJlbmNlczoKCmBgYHtyfQpzdWIoIi4qW1s6c3BhY2U6XV0rKFstK10/W1s6ZGlnaXQ6XV0rKS4qIiwgIlxcMSIsIHMpCmBgYAoKYHBhcnNlX251bWJlcmAgaXMgYWdhaW4gYW4gYWx0ZXJuYXRpdmU6CgpgYGB7cn0KcGFyc2VfbnVtYmVyKHMpCmBgYAoKCiMjIENpdHkgVGVtcGVyYXR1cmVzCgpUaGUgW2NpdHkgdGVtcGVyYXR1cmVzXShodHRwczovL3d3dy50aW1lYW5kZGF0ZS5jb20vd2VhdGhlci8pCmRhdGEgdXNlZCBwcmV2aW91c2x5IGNhbiBiZSByZWFkIHVzaW5nCgpgYGB7ciwgbWVzc2FnZSA9IEZBTFNFfQpsaWJyYXJ5KHJ2ZXN0KQpsaWJyYXJ5KGRwbHlyKQp3ZWF0aGVyIDwtIHJlYWRfaHRtbCgiaHR0cHM6Ly93d3cudGltZWFuZGRhdGUuY29tL3dlYXRoZXIvIikKdyA8LSBodG1sX3RhYmxlKGh0bWxfbm9kZXMod2VhdGhlciwgInRhYmxlIikpW1sxXV0KCncxIDwtIHdbYygxLCA0KV07IG5hbWVzKHcxKSA8LSBjKCJjaXR5IiwgInRlbXAiKQp3MiA8LSB3W2MoNSwgOCldOyBuYW1lcyh3MikgPC0gYygiY2l0eSIsICJ0ZW1wIikKdzMgPC0gd1tjKDksIDEyKV07IG5hbWVzKHczKSA8LSBjKCJjaXR5IiwgInRlbXAiKQp3dyA8LSByYmluZCh3MSwgdzIsIHczKQp3dyA8LSBmaWx0ZXIod3csIGNpdHkgIT0gIiIpCmhlYWQod3cpCmBgYAoKQ2xlYW5pbmcgdXAgYW5kIGV4dHJhY3RpbmcgYGRzdGA6CmBgYHtyfQp3d3cgPC0gbXV0YXRlKHd3LAogICAgICAgICAgICAgIGRzdCA9IGdyZXBsKCIgXFwqIiwgY2l0eSksCiAgICAgICAgICAgICAgY2l0eSA9IHN1YigiIFxcKiIsICIiLCBjaXR5KSwKICAgICAgICAgICAgICB0ZW1wLnR4dCA9IHRlbXAsICAgIyMgZm9yIGNoZWNraW5nIG9uIGNvbnZlcnNpb24gZmFpbHVyZXNgCiAgICAgICAgICAgICAgdGVtcCA9IGFzLm51bWVyaWMoc3ViKCIoWy0rXT9bWzpkaWdpdDpdXSspLioiLCAiXFwxIiwgdGVtcCkpKQpgYGAKCkNoZWNrIG9uIGBOQWAgdmFsdWVzIGZyb20gY29udmVyc2lvbjoKCmBgYHtyfQpmaWx0ZXIod3d3LCBpcy5uYSh0ZW1wKSkKd3d3IDwtIHNlbGVjdCh3d3csIC10ZW1wLnR4dCkKYGBgCgpGaXZlIGhpZ2hlc3QgYW5kIGxvd2VzdCB0ZW1wZXJhdHVyZXM6IApgYGB7cn0Kc2xpY2VfbWF4KHd3dywgdGVtcCwgbiA9IDUpCmBgYApgYGB7cn0Kc2xpY2VfbWluKHd3dywgdGVtcCwgbiA9IDUpCmBgYAoKVGVtcGVyYXR1cmVzIGZvciBub3J0aGVybiBhbmQgc291dGhlcm4gaGVtaXNwaGVyZSAoYXBwcm94aW1hdGVseSk6CgpgYGB7ciB0ZW1wLWRlbnNpdGllcywgZXZhbCA9IEZBTFNFfQpnZ3Bsb3Qod3d3LCBhZXMoeCA9IHRlbXAsIGZpbGwgPSBkc3QpKSArCiAgICBnZW9tX2RlbnNpdHkoYWxwaGEgPSAwLjUpCmBgYApgYGB7ciB0ZW1wLWRlbnNpdGllcywgZWNobyA9IEZBTFNFfQpgYGAKCgojIyBUcmlja3kgQ2hhcmFjdGVycwoKU29tZSBleGFtcGxlczoKCmBgYHtyfQoocyA8LSBoZWFkKHd3JHRlbXApKQpuY2hhcihzKQpzdWJzdHIocywgMSwgbmNoYXIocykgLSAzKQpzdWJzdHIocywgMSwgbmNoYXIocykgLSAyKQphcy5udW1lcmljKHN1YnN0cihzLCAxLCBuY2hhcihzKSAtIDIpKQphcy5udW1lcmljKCI4MiAiKQpzdWIoIiAuKiIsICIiLCBzKQpgYGAKClRoZSBwcm9ibGVtIGlzIF90d29fIG5vbi1hc2NpaSBjaGFyYWN0ZXJzLgoKVGhlIGBzdHJpX2VzY2FwZV91bmljb2RlYCBmdW5jdGlvbiBmcm9tIHRoZSBgc3RyaW5naWAgY2FuIG1ha2UgdGhlc2UKY2hhcmFjdGVycyBtb3JlIHZpc2libGU6CgpgYGB7cn0Kc3RyaW5naTo6c3RyaV9lc2NhcGVfdW5pY29kZShzKQpgYGAKClRoZSB0cm91Ymxlc29tZSBjaGFyYWN0ZXJzIGFyZToKCiogW05vLWJyZWFrIHNwYWNlIFUwMEEwXShodHRwczovL3d3dy5maWxlZm9ybWF0LmluZm8vaW5mby91bmljb2RlL2NoYXIvMDBhMC9pbmRleC5odG0pLgoKKiBbRGVncmVlIHN5bWJvbAogIFUwMEIwXShodHRwczovL3d3dy5maWxlZm9ybWF0LmluZm8vaW5mby91bmljb2RlL2NoYXIvMDBiMC9pbmRleC5odG0pLgoKVXNpbmcgdGhlIHVuaWNvZGUgc3BlY2lmaWNhdGlvbiBmb3IgdGhlIG5vLWJyZWFrIHNwYWNlIGRvZXMgd29yazoKCmBgYHtyfQpzdWIoIlx1MDBhMC4qIiwgIiIsIHMpCmBgYAoKCiMjIFZhcmlhdGlvbnMgaW4gUmVndWxhciBFeHByZXNzaW9uIEVuZ2luZXMKCjwhLS0gTm90ZSBvbiB1bmljb2RlIGRpZ2l0czoKIGh0dHBzOi8vdW5peC5zdGFja2V4Y2hhbmdlLmNvbS9xdWVzdGlvbnMvNDE0MjI2L2RpZmZlcmVuY2UtYmV0d2Vlbi0wLTktZGlnaXQtYW5kLWQKLS0+CgpNYW55IHRvb2xzIGFuZCBsYW5ndWFnZXMgc3VwcG9ydCB3b3JraW5nIHdpdGggcmVndWxhciBleHByZXNzaW9ucy4KClIgc3VwcG9ydHM6CgoqIFRoZSBQT1NJWCBzdGFuZGFyZCBmb3IgX2V4dGVuZGVkIHJlZ3VsYXIgZXhwcmVzc2lvbnNfLiBUaGlzIGlzIHRoZQogIGRlZmF1bHQgZW5naW5lLgoKKiBQZXJsLWNvbXBhdGlibGUgcmVndWxhciBleHByZXNzaW9ucwogIChbUENSRV0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvUGVybF9Db21wYXRpYmxlX1JlZ3VsYXJfRXhwcmVzc2lvbnMpLgogIFRoaXMgZW5naW5lIGlzIHNlbGVjdGVkIGJ5IGFkZGluZyBgcGVybCA9IFRSVUVgIGluIGNhbGxzIHRvCiAgZnVuY3Rpb25zIHVzaW5nIHJlZ3VsYXIgZXhwcmVzc2lvbnMuCgpEaWZmZXJlbnQgZW5naW5lcyBjYW4gZGlmZmVyIGluIGhvdyBjZXJ0YWluIGV4cHJlc3Npb25zIGFyZQppbnRlcnByZXRlZCwgZXNwZWNpYWxseSB3aGVuIG5vbi1BU0NJSSBjaGFyYWN0ZXJzIGFyZSBpbnZvbHZlZC4KCkRpZmZlcmVudCBlbmdpbmVzIGFsc28gc29tZXRpbWVzIG9mZmVyIHNob3J0aGFuZCBub3RhdGlvbnMsIGluCnBhcnRpY3VsYXIgZm9yIGNoYXJhY3RlciBjbGFzc2VzLgoKU29tZSBleGFtcGxlczoKCnwgUE9TSVggY2xhc3MgfCAgc2ltaWxhciB0byB8IHNob3J0aGFuZCB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8CnwgLS0tLS0tLS0tLS0gfCAtLS0tLS0tLS0tLSB8IC0tLS0tLS0tLSB8IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSB8CnxgWzpkaWdpdDpdYCB8ICBgWzAtOV1gICAgfCAgYFxkYCAgICAgfCAgIGRpZ2l0cyAgICAgICAgICAgICAgICAgICAgICAgfAp8YFs6dXBwZXI6XWAgfCAgIGBbQS1aXWAgIHwgIGBcdWAgICAgIHwgICB1cHBlciBjYXNlIGxldHRlcnMgICAgICAgICAgIHwKfGBbOmxvd2VyOl1gIHwgIGBbYS16XWAgICB8ICBgXGxgICAgICB8ICBsb3dlci1jYXNlIGxldHRlcnMgICAgICAgICAgICB8CnxgWzphbHBoYTpdYCB8ICBgW0EtWmEtel1gfCAgICAgICAgICAgfCAgdXBwZXItIGFuZCBsb3dlci1jYXNlIGxldHRlcnMgfAp8YFs6c3BhY2U6XWAgfCAgYFsgXHRcbl1gIHwgIGBcc2AgICAgIHwgIHdoaXRlc3BhY2UgY2hhcmFjdGVycyAgICAgICAgIHwKClRoZSBzaG9ydGhhbmQgdmVyc2lvbnMgbmVlZCB0byBoYXZlIHRoZWlyIGBcYCBlc2NhcGVkIHdoZW4gdXNlZCBpbiBhbiBSCnN0cmluZzoKCmBgYHtyfQppbnRwYXQgPC0gIi4qXFxzKFstK10/XFxkKykuKiIKc3ViKGludHBhdCwgIlxcMSIsIGMoIlRlbXA6ICAzMkYiLCAiVGVtcDogLTExRiIpKQpgYGAKCgojIyBSYXcgU3RyaW5ncwoKX1JhdyBzdHJpbmdzXyBjYW4gbWFrZSB3cml0aW5nIHJlZ3VsYXIgZXhwcmVzc2lvbnMgYSBsaXR0bGUgZWFzaWVyLgoKSW4gUiBhIHJhdyBzdHJpbmcgaXMgc3BlY2lmaWVzIGFzIGByIiguLi4pImAuCgpUaGUgYC4uLmAgY2hhcmFjdGVycyBjYW4gYmUgYW55IGNoYXJhY3RlcnMgYW5kIGFyZSB0YWtlbiBsaXRlcmFsbHksCndpdGhvdXQgc3BlY2lhbCBpbnRlcnByZXRhdGlvbiB0aGF0IG1pZ2h0IHJlcXVpcmUgZXNjYXBpbmcuCgpGb3IgdGhlIGludGVnZXIgcGF0dGVybjoKCmBgYHtyfQppbnRwYXQgPC0gIi4qXFxzKFstK10/XFxkKykuKiIKaW50cGF0X3JhdyA8LSByIiguKlxzKFstK10/XGQrKS4qKSIKaW50cGF0ID09IGludHBhdF9yYXcKYGBgCgpQeXRob24sIEMrKywgYW5kIG90aGVyIGxhbmd1YWdlcyBwcm92aWRlIHNpbWlsYXIgZmFjaWxpdGVzLgoKUidzIHJhdyBzdHJpbmcgc3ludGF4IGl0IG1vZGVsZWQgYWZ0ZXIgdGhlIG9uZSB1c2VkIGluIEMrKy4KCgojIyBBIE5vdGUgb24gU29ydGluZwoKTm9uLUFTQ0lJIGNoYXJhY3RlcnMgY2FuIGNyZWF0ZSBpc3N1ZXMgZm9yIHNvcnRpbmcgc3RyaW5ncywgYnV0IGV2ZW4KQVNDSUkgY2hhcmFjdGVyIHNvcnQgb3JkZXIgaXMgbm90IHRoZSBzYW1lIGluIGFsbApbbG9jYWxlc10oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvTGlzdF9vZl9JU09fNjM5LTFfY29kZXMpLgoKVGhlIGBMRVRURVJTYCBkYXRhIHNldCBjb250YWlucyB0aGUgdXBwZXItY2FzZSBsZXR0ZXJzIGluIGFscGhhYmV0aWNhbApvcmRlciBmb3IgdGhlIEVuZ2xpc2ggc29ydGluZyBjb252ZW50aW9uIGFuZCBtb3N0IG90aGVyIGxvY2FsZXM6CgpgYGB7cn0KTEVUVEVSUwpgYGAKCkJ1dCBpbiBFc3RvbmlhbiwgTGF0dmlhbiwgYW5kIExpdGh1YW5pYW46CgpgYGB7cn0Kc3RyaW5ncjo6c3RyX3NvcnQoTEVUVEVSUywgbG9jYWxlID0gImVzdCIpCmBgYAoKT3JkZXJpbmcgb2YgbG93ZXIgY2FzZSBhbmQgdXBwZXIgY2FzZSBsZXR0ZXJzIGluIEVuZ2xpc2ggYW5kIG1vc3Qgb3RoZXIKbG9jYWxlczoKCmBgYHtSfQpzdHJpbmdyOjpzdHJfc29ydChjKCJBIiwgImEiKSwgbG9jYWxlID0gImVuZyIpCmBgYAoKQnV0IGluIERhbmlzaCwgYW5kIGFsc28gTWFsdGVzZToKCmBgYHtSfQpzdHJpbmdyOjpzdHJfc29ydChjKCJBIiwgImEiKSwgbG9jYWxlID0gImRhbiIpCmBgYAoKYGBge3IsIGVjaG8gPSBGQUxTRSwgZXZhbCA9IEZBTFNFfQojIyByZWFkIGEgdGFibGUgZnJvbSBXaWtpcGVkaWEgd2l0aCB0aGUgbG9jYWxlIGFiYnJldmlhdGlvbnMKdXJsIDwtICJodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9MaXN0X29mX0lTT182MzktMV9jb2RlcyIKbGlicmFyeShydmVzdCkKbGlicmFyeShkcGx5cikKdyA8LSByZWFkX2h0bWwodXJsKQoKIyMgZGlmZmVyZW50IGxldHRlciBzb3J0IG9yZGVyCnRibCA8LSBodG1sX3RhYmxlKGh0bWxfbm9kZXModywgInRhYmxlIiksIGZpbGwgPSBUUlVFKQpsdGJsIDwtIHRibFtbMl1dCmxjb2RlcyA8LSBsdGJsW1s2XV0KdiA8LSBzYXBwbHkobGNvZGVzLAogICAgICAgICAgICBmdW5jdGlvbihsb2MpCiAgICAgICAgICAgICAgICBpZGVudGljYWwoc3RyaW5ncjo6c3RyX3NvcnQoTEVUVEVSUywgbG9jYWxlID0gbG9jKSwKICAgICAgICAgICAgICAgICAgICAgICAgICBMRVRURVJTKSkKbHRibFshdiwgYygiTGFuZ3VhZ2UgZmFtaWx5IiwgIklTTyBsYW5ndWFnZSBuYW1lIiwgIjYzOS0yL1QiKV0KCiMjIGRpZmZlcmVudCB1cHBlci9sb3dlciBjYXNlIHNvcnQgb3JkZXIKc3MgPC0gYygiYSIsICJBIikKdnYgPC0gc2FwcGx5KGxjb2RlcywKICAgICAgICAgICAgIGZ1bmN0aW9uKGxvYykKICAgICAgICAgICAgICAgIGlkZW50aWNhbChzdHJpbmdyOjpzdHJfc29ydChzcywgbG9jYWxlID0gbG9jKSwKICAgICAgICAgICAgICAgICAgICAgICAgICBzcykpCmx0YmxbIXZ2LCBjKCJMYW5ndWFnZSBmYW1pbHkiLCAiSVNPIGxhbmd1YWdlIG5hbWUiLCAiNjM5LTIvVCIpXQoKYGBgCgoKIyMgRW5jb2RpbmcgSXNzdWVzCgpGaWxlcyBjb250YWluIGEgc2VxdWVuY2Ugb2YgOC1iaXQgaW50ZWdlcnMsIG9yIF9ieXRlc18uCgpUaGVzZSBhcmUgdGhlIGludGVnZXJzIGZyb20gMCB0aHJvdWdoIDI1NS4KCkZvciB0ZXh0IGZpbGVzLCB0aGVzZSBieXRlcyBhcmUgaW50ZXJwcmV0ZWQgYXMgcmVwcmVzZW50aW5nIGNoYXJhY3RlcnMuCgpUaGUgbWFwcGluZyBmcm9tIGJ5dGVzIHRvIGNoYXJhY3RlcnMgaXMgY2FsbGVkIGFuIF9lbmNvZGluZ18uCgpUaGUgZW5jb2RpbmcgZm9yIHRoZSBjaGFyYWN0ZXJzIHVzZWQgaW4gQW1lcmljYW4gRW5nbGlzaCBpcyBBU0NJSToKdGhlIF9BbWVyaWNhbiBTdGFuZGFyZCBDb2RlIGZvciBJbmZvcm1hdGlvbiBJbnRlcmNoYW5nZV8uCgpUaGUgW0FTQ0lJIGVuY29kaW5nXShodHRwczovL2FzY2lpLXRhYmxlcy5jb20vKSB1c2VzIG9ubHkgdGhlCmludGVnZXJzIDAgdGhyb3VnaCAxMjcuCgpUaGUgQVNDSUkgZW5jb2RpbmcgaXMgYWRlcXVhdGUgZm9yIEFtZXJpY2FuIHVzZXM7IGV2ZW4gVUsgdGV4dCBmaWxlcwpuZWVkIG1vcmU6IHRoZSBwb3VuZCBzaWduIMKjLgoKRW5jb2RpbmdzIHRoYXQgdXNlIGludGVnZXJzIDEyOCB0aHJvdWdoIDI1NToKCiogTGF0aW4xIChha2EgSVNPLTg4NTktMSkgZm9yIHdlc3Rlcm4gRXVyb3BlYW4gbGFuZ3VhZ2VzIChpbmNsdWRlcyDCoyk7CgoqIExhdGluMiAoYWthIElTTy04ODU5LTIpIGZvciBlYXN0ZXJuIEV1cm9wZWFuIGxhbmd1YWdlcy4KCk1hbnkgZW5jb2RpbmdzIGFyZSBhdmFpbGFibGUgdG8gc3VwcG9ydCBvdGhlciBhbHBoYWJldHMgYW5kIGNoYXJhY3RlciBzZXRzLgoKRm9ydHVuYXRlbHkgbW9zdCBzeXN0ZW1zIG5vdyB1c2UgW1VuaWNvZGVdKGh0dHBzOi8vaG9tZS51bmljb2RlLm9yZy8pCndpdGggdGhlIFtVVEYtOCBlbmNvZGluZ10oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvVVRGLTgpIGZvcgpyZXByZXNlbnRpbmcgbm9uLUFTQ0lJIGNoYXJhY3RlcnMuCgpJZiB5b3UgbmVlZCB0byByZWFkIGEgZmlsZSB3aXRoIG5vbi1BU0NJSSBjaGFyYWN0ZXJzIHVzaW5nIGByZWFkLmNzdigpYApvciBzaW1pbGFyIGJhc2UgZnVuY3Rpb25zIGEgZ29vZCBwbGFjZSB0byBzdGFydCBpcyB0byBzcGVjaWZ5CmBlbmNvZGluZyA9ICJVVEYtOCJgLgoKVGhlIGZ1bmN0aW9ucyBpbiB0aGUgYHJlYWRyYCBwYWNrYWdlIGRlZmF1bHQgdG8gYXNzdW1pbmcgdGhlIGVuY29kaW5nCmlzIFVURi04LgoKR2V0dGluZyB0aGUgZW5jb2Rpbmcgd3JvbmcgY2FuIHJlc3VsdCBpbiBhIGZldyBtZXNzZWQgdXAgY2hhcmFjdGVycwpvciBpbiBhbiBlbnRpcmUgc3RyaW5nIGJlaW5nIG1lc3NlZCB1cDoKCmBgYHtyIGluY2x1ZGUgPSBGQUxTRX0KeDEgPC0gIkVsIE5pXHhmMW8gd2FzIHBhcnRpY3VsYXJseSBiYWQgdGhpcyB5ZWFyIgp4MiA8LSAiXHg4Mlx4YjFceDgyXHhmMVx4ODJceGM5XHg4Mlx4YmZceDgyXHhjZCIKYGBgCgpSZWFkaW5nIGZpbGVzIHdpdGhvdXQgc3BlY2lmeWluZyB0aGUgY29ycmVjdCBlbmNvZGluZwptaWdodCBwcm9kdWNlIGEgc3RyaW5ncyBsaWtlCgpgYGB7cn0KeDEKeDIKYGBgCgpJZiB0aGUgY29ycmVjdCBlbmNvZGluZyBpcyBrbm93biwgdGhlbiB0aGVzZSBjYW4gYmUgZml4ZWQgYWZ0ZXIgdGhlCmZhY3Qgd2l0aCBgaWNvbnYoKWA6CgpgYGB7cn0KaWNvbnYoeDEsICJMYXRpbjEiLCAiVVRGLTgiKQppY29udih4MiwgIlNoaWZ0LUpJUyIsICJVVEYtOCIpCmBgYAoKUmUtcmVhZGluZyB0aGUgZmlsZSB3aXRoIHRoZSBwcm9wZXIgZW5jb2Rpbmcgc3BlY2lmaWVkIG1heSBiZSBhIGJldHRlcgpvcHRpb24uCgpUaGUgYHJlYWRyYCBmdW5jdGlvbiBgZ3Vlc3NfZW5jb2RpbmcoKWAgbWF5IGhlbHAgaWRlbnRpZnkgdGhlIGNvcnJlY3QKZW5jb2RpbmcgaWYgaXQgaXMgbm90IHNwZWNpZmllZCBpbiB0aGUgZGF0YSBkb2N1bWVudGF0aW9uLgoKSGFuZGxpbmcgZW5jb2RpbmcgaXNzdWVzIGluIFIgY2FuIGJlIG1vcmUgY29tcGxpY2F0ZWQgb24gV2luZG93cywgYnV0CnRoaXMgd2lsbCBpbXByb3ZlIHdpdGggdGhlIG5leHQgcmVsZWFzZSBvZiBSLgoKCiMjIEdldHRpbmcgdGhlIEN1cnJlbnQgVGVtcGVyYXR1cmUgCgpUaGUgdG9vbHMgZGVzY3JpYmVkIGhlcmUgY29tZSBpbiBoYW5keSB3aGVuIHNjcmFwaW5nIGRhdGEgZnJvbSB0aGUgd2ViLgoKVGhpcyBjb2RlIGdldHMgdGhlIGN1cnJlbnQgdGVtcGVyYXR1cmUgaW4gSW93YSBDaXR5IGZyb20gdGhlIE5hdGlvbmFsCldlYXRoZXIgU2VydmljZToKCmBgYHtyLCBldmFsID0gRkFMU0V9CmxpYnJhcnkoeG1sMikKdXJsIDwtICJodHRwOi8vZm9yZWNhc3Qud2VhdGhlci5nb3YvemlwY2l0eS5waHA/aW5wdXRzdHJpbmc9SW93YStDaXR5LElBIgpwYWdlIDwtIHJlYWRfaHRtbCh1cmwpCnhwYXRoIDwtICIvL3BbQGNsYXNzPVwibXlmb3JlY2FzdC1jdXJyZW50LWxyZ1wiXSIKdGVtcE5vZGUgPC0geG1sX2ZpbmRfZmlyc3QocGFnZSwgeHBhdGgpCm5vZGVUZXh0IDwtIHhtbF90ZXh0KHRlbXBOb2RlKQphcy5udW1lcmljKHN1YigiKFstK10/W1s6ZGlnaXQ6XV0rKS4qIiwgIlxcMSIsIG5vZGVUZXh0KSkKYGBgCgpBbiBleGFtcGxlIG9mIGNyZWF0aW5nIGEgY3VycmVudCB0ZW1wZXJhdHVyZSBtYXAgaXMgZGVzY3JpYmVkCltoZXJlXShgciBoZXJlOjpoZXJlKCJ3ZWF0aGVyLmh0bWwiKWApLgoKCiMjIFJlYWRpbmcKCkNoYXB0ZXJzIFtfRGF0YSBJbXBvcnRfXShodHRwczovL3I0ZHMuaGFkLmNvLm56L2RhdGEtaW1wb3J0Lmh0bWwpIGFuZApbX1N0cmluZ3NfXShodHRwczovL3I0ZHMuaGFkLmNvLm56L3N0cmluZ3MuaHRtbCkgaW4gW19SIGZvciBEYXRhClNjaWVuY2VfXShodHRwczovL3I0ZHMuaGFkLmNvLm56LykuCgpDaGFwdGVyIFtfU3RyaW5nCnByb2Nlc3NpbmdfXShodHRwczovL3JhZmFsYWIuZGZjaS5oYXJ2YXJkLmVkdS9kc2Jvb2svKQppbiBbX0ludHJvZHVjdGlvbiB0byBEYXRhIFNjaWVuY2UgRGF0YSBBbmFseXNpcyBhbmQgUHJlZGljdGlvbgpBbGdvcml0aG1zIHdpdGggUl9dKGh0dHBzOi8vcmFmYWxhYi5kZmNpLmhhcnZhcmQuZWR1L2RzYm9vay8pLgoKCiMjIEV4ZXJjaXNlcwoKPCEtLQpmcm9tIGh0dHBzOi8vcmFmYWxhYi5naXRodWIuaW8vZHNib29rL3N0cmluZy1wcm9jZXNzaW5nLmh0bWwjZXhlcmNpc2VzLTQxCi0tPgoxLiBDb21wbGV0ZSBhbGwgbGVzc29ucyBhbmQgZXhlcmNpc2VzIGluIHRoZSBodHRwczovL3JlZ2V4b25lLmNvbS8KICAgb25saW5lIGludGVyYWN0aXZlIHR1dG9yaWFsLgoKMi4gQ29uc2lkZXIgdGhlIGNvZGUKCiAgICBgYGB7ciwgZXZhbCA9IEZBTFNFfQogICAgbGlicmFyeSh0aWR5dmVyc2UpCiAgICBmaWx0ZXIobXBnLCBncmVwbCgtLS0sIG1vZGVsKSkKICAgIGBgYAoKICAgIEZvciB3aGljaCBvZiB0aGUgZm9sbG93aW5nIHJlZ3VsYXIgZXhwcmVzc2lvbnMgaW4gcGxhY2Ugb2YgYC0tLWAgd2lsbAogICAgdGhpcyBjb2RlIHJldHVybiB0aGUgc3Vic2V0IG9mIHJvd3MgZm9yIGFsbCBtb2RlbHMgdGhhdCBjb250YWluIGVpdGhlcgogICAgYDR3ZGAgb3IgYGF3ZGAgaW4gdGhlaXIgbW9kZWwgbmFtZXM/CgogICAgYS4gIlthNF13ZCAiCiAgICBiLiAiW2E0XXdkIgogICAgYy4gIjRhd2QiCiAgICBkLiAiW1s0YV1dd2QiCg==