About me

• Author of Scala for Data Science (Packt Publishing)
• Main author of jupyter-gmaps, a library for visualizing geographical data in Jupyter notebooks
• Main contributor to Scala Plotly client, which we are developing at ASI

Scala in data science

One of the most common languages for building large data science applications

• Great native libraries (Akka, Spark)
• Access to everything written for the JVM

Scala in data science

But there is no good data visualization library

Data visualization libraries are hard to build

Data visualization libraries are hard to build

Data visualization libraries are hard to build

At a conservative two hours per commit, it would take a single person 23 years to build

Data visualization libraries are hard to build

Data visualization libraries are hard to build

• Very good knowledge of the problem domain
• Very good knowledge of the underlying platform

            
    POST /v1/plot
    
{
      "data" : {
        "x": ["2016-02-22 18:40", "2016-02-22 19:14", ... ],
        "y": [732.83, 729.65, ... ]
      },
      "figure": {
        "title": "Stock price ($)",
        ...
      }
    }
              
          

Plotly clients are easy to build

• We don't need knowledge of the problem domain: we can just rely on Plotly to provide sane defaults
• We don't need to know how to render graphs, just how to generate the right JSON

            
import co.theasi.plotly
import util.Random

// Generate uniformly distributed x
val xs = (0 until 100)

// Generate random y
val ys = xs.map { x => x + 5.0 * Random.nextDouble }

val p = Plot().withScatter(xs, ys)

draw(p, "basic-scatter", writer.FileOptions(overwrite=true))
// returns  PlotFile(pbugnion:173,basic-scatter)
            
          

            
import co.theasi.plotly
import util.Random

// Generate uniformly distributed x
val xs = (0 until 100)

// Generate random y
val ys = xs.map { x => x + 5.0 * Random.nextDouble }

val p = Plot().withScatter(xs, ys)

draw(p, "basic-scatter", writer.FileOptions(overwrite=true))
// returns  PlotFile(pbugnion:173,basic-scatter)
            
          

Come and contribute!

Github is huge.

Github is huge.

• More than 35,000,000 repositories
• More than 1,750,000,000 commits

Github is huge.

• Linux
• jquery
• docker
• atom
• rails
• ...

But for every popular repository, there are hundreds of unsucessful ones

• 82% of repositories have zero stars
• 98% of repositories have ten stars or fewer

The distribution of stars has a Gini coefficient of 0.6

So why do some repositories become popular?

Rule #1: Make programs, not websites

Make programs, not websites

The 98^th percentile repositories written in a:

• programming language have 19 stars
• markup language have 4 stars

Rule #2: Choose a trending language

If I'm in the top 2% of repos for language X, I have Y stars:

Rule #3: hard work pays off

Rule #3: hard work pays off

For repos of at least 1000 commits written in...

But some languages provide better ROI

Rule #4: Make it approachable

Make it approachable

• Websites don't get stars
• Choose a trending language
• Work really hard
• Make it approachable

Making this presentation

• Data engineering parts done in Scala with Akka streams
• Initial data exploration in Python
• Plots and automated pipelines in Scala with scala-plotly-client
• Slides in reveal.js