bb9128fac91692ad4f46a785d772dd39?s=200 avatar

bb9128fac91692ad4f46a785d772dd39?s=200

Data Scientist and Engineer, I love tinkering with data!

Head of Data

Pablo Seibelt

Learn about the R package we use internally at Auth0 for ETL / Data analysis.



Today I'll be presenting [R for Auth0](https://github.com/auth0/rauth0), our internal R package to smoothen our workflow when working with the R language. At Auth0, we use R not only for statistical analysis but also for most of our ETL processing jobs.

## Auth0's Data Technology Stack

First I'll present some insight into our technology stack, we use:

* [R](https://www.r-project.org/) for ETL, Analysis and Machine Learning
* [RStudio](https://www.rstudio.com/) as an interface for working with R
* [Python](https://www.python.org/) as our importer/exporter for external APIs
* [Amazon Redshift](https://aws.amazon.com/redshift/) as our data warehouse
* [Apache Airflow](https://airflow.apache.org/) as our job scheduling system

Except for Amazon Redshift, our stats backend is all open source software.

## Enter `rauth0`

Working with ETL processes every day we noticed some recurring patterns, table loading, upserting, slowly changing dimensions, `ggplot` theming and others, that we could simplify by centralizing in one place. So we have this package which centralizes functionality that we reuse in many processes.

We will walk through some of the most interesting pieces of rauth0 in hopes it'll be useful to build out your own internal R package for your organization, or just personal use. The package is MIT licensed so you can use to adapt and use the code without significant restrictions.

## Plotting Beautifully

We teamed up with Auth0's design team member [Julian Leiss](https://twitter.com/julianleiss) to work out some better defaults for `ggplot2`, in hopes of making our R generated charts better for presentation, and more according to the company's style. Working on this, we defined these palettes for discrete variables:


```r
library(ggplot2)
library(rauth0)

base_plot =
ggplot(diamonds, aes(clarity, fill = cut)) +
geom_bar() +
theme_auth0()

base_plot + scale_fill_auth0_discrete(palette='default') +
  ggtitle('Default palette')
```

![Example graph of Auth0 default palette](https://images.ctfassets.net/23aumh6u8s0i/6j1Qi33oNsiCKj90gb5Al2/1f1041eb6bcd05935747f469bc9dfde6/default-palette)

```r
base_plot + scale_fill_auth0_discrete(palette='sequential') +
  ggtitle('Sequential palette')
```

![Example graph of Auth0 sequential palette](https://images.ctfassets.net/23aumh6u8s0i/1ykVJ3GIXWP6r3qzMlGCjQ/f9aa1301dde98c679df176f404b224f7/sequential-palette)

```r
base_plot + scale_fill_auth0_discrete(palette='colorful') +
  ggtitle('Colorful palette')
```

![Example graph of Auth0 colorful palette](https://images.ctfassets.net/23aumh6u8s0i/6wtPB7Qp3Sr2FskojiWhYh/10dd5b15aa955fbd4d1f3f1c958d1eef/colorful-palette)
And this one for continous variables:


```r
dsamp <- diamonds[sample(nrow(diamonds), 1000), ]
ggplot(dsamp, aes(carat, price)) +
  geom_point(aes(colour = price)) +
  theme_auth0() +
  scale_color_auth0_gradient(midpoint = 10000) +
  labs(title="Continous scale", subtitle='Carat vs price')
```

![Example chart of continous scale](https://images.ctfassets.net/23aumh6u8s0i/6FTD3h3ZGtRNDT6CjThg4m/806963dac484e473f56e65ed67799b5d/continous-scale)

## Accessing the Data Warehouse

As mentioned earlier, we will use Amazon Redshift for accessing the data warehouse (DWH), as such, we use the open source R package `redshiftTools` to upload data into the DWH. Although that is just a piece of the picture, we also need to read data, and there are some repetitive manipulations we do.

For example, many times we may want to connect with `dbplyr` and figure out which images are the most common as a Twitter thumbnail:

```r
library(tidyverse)
library(dbplyr)

con = dwh_connect()

page_metadata = tbl(con, in_schema('prod', 'dim_page_metadata'))
twitter_image_count = filter(page_metadata, page_meta_twitter_image != '<Blank>', scd_end_date == '9999-12-31') %>%
  group_by(page_meta_twitter_image) %>%
  count() %>%
  arrange(desc(n))

knitr::kable(collect(twitter_image_count, n=10), format='markdown') # Top 10
```

|page_meta_twitter_image                                                        |     n|
|-------------------------------------------------------------------------------|------|
|https://cdn.auth0.com/website/authenticate/meta-twitter.png                    | 10186|
|https://cdn.auth0.com/blog/blog/home-twitter-card.png                          |  2465|
|/modules/twittercard/1x-new.png                                                |   687|
|https://cdn.auth0.com/blog/illustrations/auth0.png                             |   339|
|https://cdn.auth0.com/website/assets/modules/twittercard/1x-new-383f8d95d1.png |    67|
|https://cdn.auth0.com/website/assets/modules/opengraph/1x-new-69e09d5054.png   |    61|
|https://auth0.com/learn/wp-content/uploads/2016/01/1x-new.png                  |    59|
|https://cdn.auth0.com/blog/series-c/auth0-logo.png                             |    36|
|https://cdn.auth0.com/blog/angular/logo3.png                                   |    23|
|https://cdn.auth0.com/blog/jwtalgos/logo.png                                   |    12|

The beauty of this is that with `dbplyr` we can also inspect which query is the client running to make this work:

```r
show_query(twitter_image_count)
```

```
## <SQL>
## SELECT "page_meta_twitter_image", COUNT(*) AS "n"
## FROM prod.dim_page_metadata
## WHERE (("page_meta_twitter_image" != '<Blank>') AND ("scd_end_date" = '9999-12-31'))
## GROUP BY "page_meta_twitter_image"
## ORDER BY "n" DESC
```

In the same spirit of data manipulation, we have Redshift compatible functions to run arbitrary queries and statements, create views, replace tables with data or with an SQL query, etc. I can't stress enough how painlessly it is to work with R and Amazon Redshift with these tools available, especially "dbplyr".

### Kimball methods

We also have in this library, functions to update SCD type 1 and type 2 dimensions, I invite you to look at functions `dwh_update_scd_type1`, `dwh_update_scd_type1_in_database` and `dwh_update_scd_type2`, they are quite complex but worth the effort because of the problems they solve.

They might have some edge cases not considered in our current use cases and will evolve over time. This can be considered an alternative to buying a full-fledged ETL tool without having to use a graphical interface for designing the data flows, with its advantages and disadvantages.

### A real clickstream data example

For example, this is the real production code for building the sessions dimension for clickstream data, there is another process that generates the staging tables (`stg_session_events` with a row every pageview/event and `stg_sessions` has one only row per session).

```
library(rauth0)
library(tidyverse)
library(dbplyr)

con = dwh_connect()
all_events = tbl(con, 'events_metrics') %>% filter(received > '2012-01-01')

# Only use pageviews, tracked events and identify calls
filteredEvents = filter(all_events, type %in% c('track', 'page', 'identify'), sql("url_protocol(url)") %in% c('http', 'https'))


dim_ref = tbl(con, 'stg_sessions')
map_ref = tbl(con, 'stg_session_events')
dim_page = tbl(con, in_schema('prod', 'dim_page'))
dim_sessions = tbl(con, in_schema('prod', 'dim_session'))

dwh_set_execution_slots(con, 3)

onlyNew = select(map_ref, id=event_id, session_key) %>%
  anti_join(dim_sessions, by='session_key') %>%
  compute()

basePageviews = onlyNew %>% # Only sessions not in dimension already, recalculating is costly
  inner_join(filteredEvents, by=c('id')) %>%
  inner_join(dim_page, by=c('url'='page_full_url')) %>%
  # Remove consecutive visits/tracks to the same url
  mutate(
    prev_page=sql('lag(page_clean_url) over (partition by session_key order by received, id)')
  ) %>%
  filter(prev_page!=page_clean_url)

actionNum = basePageviews %>%
  group_by(session_key) %>%
  tally()

actionSequence = basePageviews %>%
  # Get sequence numbers
  mutate(
    url=sql('left(coalesce(page_clean_url, url), 500)::varchar(500)'),
    seq=sql('row_number() over(partition by session_key order by received, id)'),
    revseq=sql('row_number() over(partition by session_key order by received desc, id)')
  ) %>%
  # Build string for first five and last 5 pageviews
  filter(seq<=5 | revseq <=5) %>%
  mutate(
    path_firstfive = ifelse(seq<=5, url, NA),
    path_lastfive = ifelse(revseq<=5 & seq > 5, url, NA)
  ) %>%
  group_by(session_key) %>%
  summarize(
    firstfive=sql("listagg(path_firstfive, ' -> ') within group (order by received)"),
    lastfive=sql("listagg(path_lastfive, ' -> ') within group (order by received)")
  ) %>%
  ungroup() %>%
  mutate(
    firstfive=sql('firstfive::varchar(2520)'),
    lastfive=sql('lastfive::varchar(2520)')
  ) %>%
  inner_join(actionNum, by='session_key') %>%
  mutate(
    action_sequence=case_when(
      n > 10 ~ concat(concat(firstfive, ' -> [...] -> '), lastfive),
      !is.na(lastfive) ~ concat(concat(firstfive, ' -> '), lastfive),
      TRUE ~ firstfive
    )
  ) %>%
  inner_join(dim_ref, by='session_key') %>%
  select(session_key, action_sequence, session_id) %>%
  compute()

dwh_upsert_table_from_temp_table(con, "prod.dim_session", remote_name(actionSequence), keys=c('session_key'))
```

## Final Notes

We use R and the other technologies mentioned for our ETL processing and it has been very fruitful and smooth for us, we hope some of the ideas on this article can be useful for your Data Warehousing efforts.

You can access and download our internal library here: [https://github.com/auth0/rauth0](https://github.com/auth0/rauth0)


RAuth0 - Our internal R package

A guide through the available drivers and tools to make your life easier when using Amazon Redshift from R and/or RStudio




[Amazon Redshift](https://aws.amazon.com/redshift/) is one of the hottest databases for Data Warehousing right now, it's one of the most cost-effective solutions available, and allows for integration with many popular BI tools. Unfortunately, the status of the drivers compatibility is a little more shaky, but there is a way to make it work very nicely with [R](https://www.r-project.org/)!

<include src="TweetQuote" quoteText="The status of Redshift drivers compatibility with R is a little shaky, but there is a way to make it work fine."/>

First of all, let's go through the 3 options we have for connecting to Amazon Redshift. For all of the connections, we'll define these variables for connecting:


```r
dbname="dbname"
host='my-redshift-url.amazon.com'
port='5439'
user='myuser'
password='mypassword'
```

## RJDBC

This is the ["official" way to use Amazon Redshift with R](https://aws.amazon.com/blogs/big-data/connecting-r-with-amazon-redshift/), using the JDBC driver on [SQL Workbench/J](http://www.sql-workbench.net/) is the official way to connect to it according to the documentation, and this driver can be loaded like this:



```r
# Save the driver into a directory
dir.create('~/.redshiftTools')
download.file('http://s3.amazonaws.com/redshift-downloads/drivers/RedshiftJDBC41-1.1.9.1009.jar','~/.redshiftTools/redshift-driver.jar')
install.packages('RJDBC')
```


```r
suppressPackageStartupMessages(library(RJDBC))

# Use Redshift driver
driver <- JDBC("com.amazon.redshift.jdbc41.Driver", "~/.redshiftTools/redshift-driver.jar", identifier.quote="`")

# Create connection    
url <- sprintf("jdbc:redshift://%s:%s/%s?tcpKeepAlive=true&ssl=true&sslfactory=com.amazon.redshift.ssl.NonValidatingFactory", host, port, dbname)
jconn <- dbConnect(driver, url, user, password)
```

Of course you can change the url with the options you need for your particular setup. The first 3 lines you only need to run them once, they download Redshift's official JDBC driver and install the RJDBC package.

This package was the only one I found which supported transactions on Redshift (`BEGIN`, `COMMIT`, `ROLLBACK`) until recently, otherwise on other packages DDL operations will autocommit.

Until a few days ago since the writing of this blogpost, this was the recommended package for uploading data and inserting/deleting data, since you usually want be able to unite your operations which modify data into a single transaction.

This package has some big problems though, which I'll explain later on this post.

## RPostgreSQL

Amazon Redshift is mostly PostgreSQL compatible, so most PostgreSQL drivers work well. [RPostgreSQL](https://code.google.com/archive/p/rpostgresql/) is configured like this:



```r
install.packages('RPostgreSQL')
```

```r
library(RPostgreSQL)
drv <- dbDriver("PostgreSQL")
pconn_rsql <- dbConnect(drv,
                 host = host,
                 port = port,
                 user = user,
                 password = password,
                 dbname = dbname)
```

There are two major problems with this driver though: It [lacks SSL support](https://stackoverflow.com/questions/38942118/can-not-connect-postgresqlover-ssl-with-rpostgresql-on-windows), and you don't have transactions like you do with the official driver. In my humble opinion this is not a good option, even if you don't use SSL now, if your company requires it in the future you may need to change all your code to switch drivers. The good thing is this driver works fine with [dbplyr](http://dbplyr.tidyverse.org/).

<include src="TweetQuote" quoteText="The RPostgreSQL driver doesn't support SSL-secured Redshift clusters."/>

## RPostgres

This is another PostgreSQL library, that has much better support using the [libpq library](https://www.postgresql.org/docs/9.6/static/libpq.html), you'll probably need to install the postgresql development header libraries locally for this one to be installed succesfully. [Check out the official docs for installing instructions if you run into issues.](https://github.com/r-dbi/RPostgres)



```r
install.packages('RPostgres')
```

```r
library(RPostgres)
pconn_r <- dbConnect(RPostgres::Postgres(),
               host = host,
               port = port,
               user = user,
               password = password,
               dbname = dbname,
               sslmode='require')
```

RPostgres supports transactions, SSL and works fine with dbplyr.

## Gotchas With The Libraries

So, what are some gotchas with these libraries?. We'll run this in SQL Workbench/J to have some data to test:


```sql
create table sicatest (
  a varchar(100),
  b bigint,
  c date,
  d timestamp,
  e boolean
);
insert into sicatest values
('Hello', 1, '2017-10-01', '2017-10-01 20:00:00', TRUE);
insert into sicatest values
('Hello ñandu', 9223372036854775807, '2017-12-30', '2017-12-11 23:59:00', FALSE);
insert into sicatest values
('こんにちは', 9223372036854775807, '2017-12-30', '2017-12-11 23:59:00', NULL);
commit;
```

We deliberately added the Ñ letter from spanish, and こんにちは (Kon'nichiwa) which is hello in japanese to have some non-ascii stuff, and see if it's handled correctly.

So, let's start with RJDBC:


```r
suppressPackageStartupMessages(library(dplyr))
sicatest = dbGetQuery(jconn, 'select * from sicatest')
dplyr::glimpse(sicatest)
```

```
## Observations: 3
## Variables: 5
## $ a <chr> "こんにちは", "Hello ñandu", "Hello"
## $ b <dbl> 9223372036854775808, 9223372036854775808, 1
## $ c <chr> "2017-12-30", "2017-12-30", "2017-10-01"
## $ d <chr> "2017-12-11 23:59:00.000000", "2017-12-11 23:59:00.000000", ...
## $ e <chr> NA, "false", "true"
```

Well, that didn't go as expected, right? If you look closely, the table has the number `9223372036854775807`, but the query has returned `9223372036854775808` 😱. This happens because the numeric integer is automatically converted to a floating point numeric, which loses precision with big numbers.

Also unfortunately, it has returned dates and booleans as strings, which is incorrect, but we can work around that. Let's compare with `RPostgres`.



```r
sicatest2 = dbGetQuery(pconn_r, 'select * from sicatest')
glimpse(sicatest2)
```

```
## Observations: 3
## Variables: 5
## $ a <chr> "Hello ñandu", "Hello", "こんにちは"
## $ b <S3: integer64> 9223372036854775807, 1, 9223372036854775807
## $ c <date> 2017-12-30, 2017-10-01, 2017-12-30
## $ d <dttm> 2017-12-11 23:59:00, 2017-10-01 20:00:00, 2017-12-11 23:59:00
## $ e <lgl> FALSE, TRUE, NA
```
Well well, that's much better isn't it? the numbers aren't modified, they are of the correct type (int64), and it correctly guessed types `date`, `datetime` and `logical/boolean`! If integer64 is problematic in your case, you can also choose to convert bigint fields into other types, by using the `bigint` parameter when creating the connection.

Great stuff! I think this is the kind of library we want to work in a day-to-day basis. Also, you can use this same connection to explore data with dbplyr:


```r
sica_ref = tbl(pconn_r, 'sicatest')

r = select(sica_ref, d, a) %>%
  mutate(rank=min_rank(desc(d))) %>%
  filter(rank==1) %>%
  collect()
knitr::kable(r, format='markdown')
```



|d                   |a           | rank|
|:-------------------|:-----------|----:|
|2017-12-11 23:59:00 |Hello ñandu |    1|
|2017-12-11 23:59:00 |こんにちは  |    1|

Before, this library failed with transactions, so something like this failed, but now it works fine!


```r
dbExecute(pconn_r, "BEGIN")
dbExecute(pconn_r, "COMMENT on table sicatest is 'best table ever'")
dbExecute(pconn_r, "ROLLBACK")
```

That's why, for maximum efficiency with R + Redshift, I recommend to use `RPostgres`, as it's the best library available today.

<include src="TweetQuote" quoteText="For maximum efficiency with R + Redshift, we can use RPostgres."/>

Having said that, there's an additional tool which I'd like to introduce to you, which is the `redshiftTools` R Package, it'll add nicely to your toolkit and supports either RJDBC or RPostgres connections. This package is MIT licensed and it's source is available at [https://github.com/sicarul/redshiftTools](https://github.com/sicarul/redshiftTools).


```r
  install.packages(c('devtools', 'httr', 'aws.s3'))
  devtools::install_github("RcppCore/Rcpp")
  devtools::install_github("r-dbi/DBI")
  devtools::install_github("sicarul/redshiftTools")
```

After installing, you'll have 4 helpful functions to use, which are explained in full detail in [the package's README](https://github.com/sicarul/redshiftTools).

`rs_create_statement`: Generates the SQL statement to create a table based on the structure of a data.frame. It allows you to specify sort key, dist key and if you want to allow compression to be added or not.

`rs_replace_table`: Deletes all records in a table, then uploads the provided data frame into it. It runs as a transaction so the table is never empty to the other users.

`rs_upsert_table`: Deletes all records matching the provided keys from the uploaded dataset, and then inserts the rows from the dataset. If no keys are provided, it acts as a regular insert.

`rs_create_table`: This just runs *rs_create_statement* and then *rs_replace_table*, creating your table and uploading it.

This package is helpful because uploading data with inserts in Redshift is super slow, this is the recommended way of doing replaces and upserts [per the Redshift documentation](http://docs.aws.amazon.com/redshift/latest/dg/t_Loading-data-from-S3.html), which consists of generating various CSV files, uploading them to an S3 bucket and then calling a copy command on the Redshift server, all of that is handled by the package.

I hope this guide let's you unlock the full potential of R + Amazon Redshift, two great tools that work very well together when well configured. We use this methodology inside Auth0 and we think it's very useful for other organizations with similar infrastructure.

If you are interested in reading other posts about our work with Data @ Auth0, you may enjoy [Machine Learning for everyone](https://auth0.com/blog/machine-learning-for-everyone/) and [Adding Authentication to Shiny Server in 4 Simple Steps](https://auth0.com/blog/adding-authentication-to-shiny-server/).

<include src="asides/MarketBasketLinksAbout" />


"amazon-redshift"