Making ggplot2 Accessible

Paul Murrell, Debra Warren
The University of Auckland
A. Jonathon Godfrey
Massey University
August 2018

BrailleR

A histogram 

library(BrailleR)
h <- hist(faithful$eruptions)
plot of chunk unnamed-chunk-2

The hist() function draws a histogram

BrailleR

'BrailleR' can generate text from plots 

VI(h)

This is a histogram, titled: Histogram of faithful$eruptions
"faithful$eruptions" is marked on the x-axis.
Tick marks for the x-axis are at: 2, 3, 4, and 5 
There are a total of 272 elements for this variable.
Tick marks for the y-axis are at: 0, 20, 40, and 60 
It has 8 equal-width bins, starting at 1.5 and ending at 5.5 .
The mids and counts for the bins are:
mid = 1.75  count = 55 
mid = 2.25  count = 37 
mid = 2.75  count = 5 
mid = 3.25  count = 9 
mid = 3.75  count = 34 
mid = 4.25  count = 75 
mid = 4.75  count = 54 
mid = 5.25  count = 3

The VI() function from the 'BrailleR' package can generate a text description of the histogram. In combination with a screen reader, this provides some information about the histogram for blind or visually impaired R users.

BrailleR

'BrailleR' uses information returned by hist() 

h[1:5]

$breaks
[1] 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5

$counts
[1] 55 37  5  9 34 75 54  3

$density
[1] 0.40441176 0.27205882 0.03676471 0.06617647 0.25000000
[6] 0.55147059 0.39705882 0.02205882

$mids
[1] 1.75 2.25 2.75 3.25 3.75 4.25 4.75 5.25

$xname
[1] "faithful$eruptions"

The hist() function returns an object containing information about the histogram as well as drawing the histogram. VI() uses that information to create a text description of the histogram.

BrailleR

Not all plotting functions return information 

p <- plot(eruptions ~ waiting, faithful)
plot of chunk unnamed-chunk-6

The plot() functions does not return ANY information about the plot that it draws.

BrailleR

Not all plotting functions return information 

p

NULL

This makes it hard for VI() to do anything.

BrailleR

'BrailleR' augments some plotting functions 

> library(BrailleR)

Attaching package: ‘BrailleR’

The following objects are masked from ‘package:graphics’:

    boxplot, hist

The 'BrailleR' package has its own versions of some major plotting functions that augment the information that is returned ...

BrailleR

'BrailleR' has its own plotting functions 

s <- with(faithful, ScatterPlot(waiting, eruptions))
plot of chunk unnamed-chunk-8

... and the 'BrailleR' package has some new plotting functions ...

BrailleR

'BrailleR' has its own plotting functions 

str(s[1:2])

List of 2
 $ data     :'data.frame':	272 obs. of  2 variables:
  ..$ x: num [1:272] 43 45 45 45 46 46 46 46 46 47 ...
  ..$ y: num [1:272] 1.98 1.87 1.92 2.2 1.78 ...
 $ ExtraArgs:List of 4
  ..$ xlab: chr "waiting"
  ..$ ylab: chr "eruptions"
  ..$ main: chr ""
  ..$ sub : chr ""

... that generate information about the plot.

BrailleR

'BrailleR' has a lot of work to do 

methods(VI)

 [1] VI.aov        VI.boxplot    VI.data.frame VI.default   
 [5] VI.dotplot    VI.ggplot     VI.hist       VI.histogram 
 [9] VI.list       VI.lm         VI.matrix     VI.summary.lm
[13] VI.tsplot     VI.TukeyHSD  
see '?methods' for accessing help and source code

The 'BrailleR' package has to do a lot of work for each individual plot that it wants to support

ggplot2

A 'ggplot2' plot 

library(ggplot2)
g <- ggplot(faithful) +
         geom_histogram(aes(x=eruptions), breaks=h$breaks)
plot of chunk unnamed-chunk-12

'ggplot2' is a quite popular package for drawing plots in R

ggplot2 in BrailleR

(OLD) 'BrailleR' text from a 'ggplot2' plot 

VI(g)

This untitled chart;
has no subtitle;
and no caption.
with x-axis "eruptions" labeled from  to  and 
     y-axis "count" labeled from  to ;

But 'BrailleR' could not generate text descriptions for 'ggplot2' plots

ggplot2 in BrailleR

'ggplot2' always generates a plot object 

class(g)

[1] "gg"     "ggplot"

names(g)

[1] "data"        "layers"      "scales"      "mapping"    
[5] "theme"       "coordinates" "facet"       "plot_env"   
[9] "labels"

'ggplot2' always generates a "ggplot" object

ggplot2 in BrailleR

'ggplot2' exports ggplot_build() 

gb <- ggplot_build(g)
names(gb)

[1] "data"   "layout" "plot"  

names(gb$plot)

[1] "data"        "layers"      "scales"      "mapping"    
[5] "theme"       "coordinates" "facet"       "plot_env"   
[9] "labels"

And 'ggplot2' exports the ggplot_build() function, which generates more information about what is drawn

ggplot2 in BrailleR

(NEW) 'BrailleR' text from a 'ggplot2' plot 

VI(g)

This is an untitled chart with no subtitle or caption.
It has x-axis 'eruptions' with labels 2, 3, 4 and 5.
It has y-axis 'count' with labels 0, 20, 40 and 60.
The chart is a bar chart containing 8 vertical bars.
Bar 1 is centered at 1.75, and spans from 0 to 55.
Bar 2 is centered at 2.25, and spans from 0 to 37.
Bar 3 is centered at 2.75, and spans from 0 to 5.
Bar 4 is centered at 3.25, and spans from 0 to 9.
Bar 5 is centered at 3.75, and spans from 0 to 34.
Bar 6 is centered at 4.25, and spans from 0 to 75.
Bar 7 is centered at 4.75, and spans from 0 to 54.
Bar 8 is centered at 5.25, and spans from 0 to 3.

So now 'BrailleR' can not generate text descriptions for 'ggplot2' plots!

ggplot2 in BrailleR

(NEW) 'BrailleR' text from a 'ggplot2' plot 

g2 <- ggplot(faithful[1:5,]) +
      geom_point(aes(x=waiting, y=eruptions))
print(g2)
plot of chunk unnamed-chunk-18

And this just works for a range of plots straight away

ggplot2 in BrailleR

(NEW) 'BrailleR' text from a 'ggplot2' plot 

VI(g2)

This is an untitled chart with no subtitle or caption.
It has x-axis 'waiting' with labels 60, 70 and 80.
It has y-axis 'eruptions' with labels 2, 3 and 4.
The chart is a set of 5 points.
The points are at:
(79, 3.6), 
(54, 1.8), 
(74, 3.33), 
(62, 2.28) and 
(85, 4.53)

And this just works for a range of plots straight away

ggplot2 in BrailleR

Issues:

  • How to decide what to "say" about a plot?

Deciding what to say and how to say it is hard; there is no "best" solution

ggplot2 in BrailleR

Solutions:

  • Base text off a "template" that can be modified 
{{#title}}This chart has title '{{title}}'.{{/title}}
{{^title}}This chart is untitled.{{/title}}
{{#subtitle}}It has the subtitle '{{subtitle}}'.{{/subtitle}}
{{#caption}}It has caption '{{caption}}'.{{/caption}}

So we allow for flexibility; people can use their own template (based on the R package 'whisker' and the Mustache templating language; ...

ggplot2 in BrailleR

Solutions:

  • Rather than cat() text onto the screen, generate an object that contains the text 
grep("axis", VI(g))

It has x-axis 'eruptions' with labels 2, 3, 4 and 5.
It has y-axis 'count' with labels 0, 20, 40 and 60.

... and people can do things with the object, like searching

ggplot2 in BrailleR

Solutions:

  • Rather than cat() text onto the screen, generate an object that contains the text 
sort(VI(g2))

This is an untitled chart with no subtitle or caption.
It has x-axis 'waiting' with labels 60, 70 and 80.
It has y-axis 'eruptions' with labels 2, 3 and 4.
The chart is a set of 5 points.
The points are at:
(54, 1.8), 
(62, 2.28), 
(74, 3.33), 
(79, 3.6) and 
(85, 4.53)

... and people can do things with the object, like sorting

ggplot2 in BrailleR

The overall design involves gathering information about a plot, generating text from the information (using a template), and then writing methods that work with the combination of information and text.

Example

qp <- qplot(displ, hwy, data=mpg, facets=.~year,
            geom=c("point", "smooth"))
print(qp)
plot of chunk unnamed-chunk-22

With 'ggplot2', we describe a plot in terms of a small number of building blocks - we can create an infinite array of plots by combining the building blocks in different ways.

Example

VI(qp)

This is an untitled chart w/ no subtitle or caption.
The chart is comprised of 2 panels arranged horizontally.
The panels represent different values of year.
Each panel has x-axis 'displ' w/ labels 2, 3, 4, 5, 6 and 7.
Each panel has y-axis 'hwy' w/ labels 20, 30 and 40.
Each panel has 2 layers.
Panel 1 represents data for year = 1999.
Layer 1 of panel 1 is a set of 117 points.
Layer 2 of panel 1 is a smooth curve w/ confidence intervals.
Panel 2 represents data for year = 2008.
Layer 1 of panel 2 is a set of 117 points.
Layer 2 of panel 2 is a smooth curve w/ confidence intervals.

This means that the support for 'ggplot2' in 'BrailleR' can just add support for a small set of building blocks and then we can generate accessible text for an infinite array of plots.

Summary

'BrailleR' now has 'ggplot2' support

This vastly extends the range of R plots for which we can generate accessible text

We have also extended the flexibility of the generation of accessible text

We have also made it easy to search and sort accessible text

Slide says it all

Acknowledgements

Debra Warren (Master's project)

Jonathon Godfrey ('BrailleR')

The 'whisker' R package

The Mustache templating language