This document describes a plan to expand the R graphics engine to support graphical definitions: graphical features like gradients, patterns, masks, and clipping paths.

The Features Section outlines the features that I would like to target and the Demonstrations Section provides a glimpse of how some new features are currently implemented at the user level (in the R-defs branch).

The Internal details Section has more on the internal design and internal code changes that have been made.

The Issues Section contains some of the major design issues that I have identified so far and the Discussion Section contains some questions and (some) answers that I have already thought a bit about, including the changes that would be imposed on graphics device maintainers.

1. Introduction

The R graphics engine only supports a limited set of graphical features: basic shapes like lines, rectangles, and circles; paths (including shapes with holes); text; raster images; and rectangular clipping regions. These features are sufficient to produce a wide range of statistical plots, but there are many things that R graphics cannot do.

For example, the image below contains a set of filled contours within a map of New Zealand. The natural way to achieve this result would be to draw the filled contours and apply a clipping path to the filled contours based on the boundary of New Zealand. That is not something that can be achieved in R graphics (because R graphics only supports rectangular clipping regions).¹

The plan is to add new graphical features to the R graphics engine, such as clipping paths, in order to expand the range of graphical images that can be produced with R.

2. Issues

There are a number of issues that need to be resolved when adding a new graphics feature to the R graphics engine.

3. Features

This section briefly describes the set of graphical features that are currently being considered.

4. Demonstrations

A new R-defs branch has been created for experimenting. So far, that includes the addition of support for linear and radial gradient fills, patterns, clipping paths, and masks on both Cairo graphics devices and the pdf device (except for patterns), with a 'grid' interface. The following set of examples show how the 'grid' interface has been implemented so far.

NOTE: support for graphics definitions has only been implemented for Cairo-based graphics devices and the pdf device so far. Other graphics devices should run, but will not produce output from graphics definitions. Third-party graphics devices will hopefully error out (if they make use of R_GE_checkVersionOrDie), but otherwise they are likely to segfault until they are updated.

On the positive side, Cairo devices and the pdf device can be used on all major platforms, so there is already an option for accessing these new features on all major platforms.

Gradients

A linear gradient is defined by a start point (default: bottom-left) and an end point (default: top-right) and a series of "stops". Each stop is defined by a position on the line from start point to end point, plus a colour. If the first stop is not at the start point, or the last stop is not at the end point, we can specify how the gradient "extends" beyond the first and last stops.

The following code creates a linear gradient that starts one inch off the bottom of the image, proceeds vertically to the top of the image, and transitions from red to yellow then back to red.

grid.rect(gp=gpar(fill=linearGradient(c("red", "yellow", "red"),
                                      c(0, .5, 1),
                                      x1=.5, y1=unit(1, "in"),
                                      x2=.5, y2=1,
                                      extend="none")))

When the gradient fill is specified on a grob, the start and end points are relative to the grob bounding box. The following code fills a rectangle in the central quarter of the image with the default linear gradient; the gradient starts at the bottom-left of the rectangle and ends at the top-right of the rectangle.

grid.rect(width=.5, height=.5,
          gp=gpar(fill=linearGradient()))

A gradient fill can also be specified on a viewport, in which case the gradient is relative to the viewport and all subsequent drawing makes use of that gradient (unless a new explicit fill is specified).

In the code below, we push a viewport that fills the whole image and specify the default linear gradient fill, so the gradient is relative to the whole image. We then draw a rectangle, with no explicit fill, so it "inherits" the gradient from the viewport.

pushViewport(viewport(gp=gpar(fill=linearGradient())))
grid.rect()

In the next example, we have the same viewport, but we draw the rectangle only in the central quarter of the image. This shows that the gradient that the rectangle inherits is relative to the viewport rather than being relative to the rectangle.

pushViewport(viewport(gp=gpar(fill=linearGradient())))
grid.rect(width=.5, height=.5)

The next example shows another variation on inheritance of gpar settings. This time a viewport is pushed with the default linear gradient and we draw a rectangle in the left third of the image that inherits that gradient (relative to the viewport). Then we push another viewport with a green fill and draw a rectangle within that viewport (in the centre of the image) and that rectangle inherits the green fill. Finally, we pop the second viewport and return to the first viewport and restore its linear gradient so that when we draw a rectangle in the right third of the image it again inherits the linear gradient (relative to the viewport).

pushViewport(viewport(gp=gpar(fill=linearGradient())))
grid.rect(x=.2, width=.2, height=.5)
pushViewport(viewport(gp=gpar(fill="green")))
grid.rect(x=.5, width=.2, height=.5)
popViewport()
grid.rect(x=.8, width=.2, height=.5)

The next example also demonstrates inheritance of gpar settings, but this time it is inheritance between viewports. We push a viewport with the default gradient, then push another viewport in the central quarter of the image, then draw a rectangle. The second viewport inherits the gradient from the first viewport and then the rectangle inherits the gradient from the second viewport. The important point is that the gradient is relative to the first viewport, which is where it was defined.

pushViewport(viewport(gp=gpar(fill=linearGradient())))
pushViewport(viewport(width=.5, height=.5))
grid.rect()

The next example just shows that gradients work with translucent colours. We draw the text "Reveal" then over the top draw a rect with a gradient fill that transitions from (opaque) white to transparent.

grid.text("Reveal", gp=gpar(fontface="bold", cex=3))
grid.rect(gp=gpar(fill=linearGradient(c("white", "transparent"),
                                      x1=.2, x2=.8, y1=.5, y2=.5)))

The next example shows that radial gradients work via the same interface. It is expected that fill patterns would work the same way.

As with linear gradients, we have control over the start and end of the radial gradient, though in the radial case it is a start circle and an end circle. In the code below, we make a gradient that starts near the top right of the image (with white) and ends with a circle that fills the image (black).

grid.rect(gp=gpar(fill=radialGradient(c("white", "black"),
                                      cx1=.8, cy1=.8)))

The next example shows that we can add a gradient to an existing image, in this case a 'ggplot2' plot. Here we edit the background of the plot panel to add a grey-to-white gradient.

library(ggplot2)
ggplot(mtcars) +
    geom_point(aes(x=disp, y=mpg)) +
    theme_bw()
grid.force()
grid.edit("panel.background", grep=TRUE,
          gp=gpar(fill=linearGradient(c("grey", "white"), x1=.5, x2=.5)))

The next example is similar, but demonstrates that, even with gradients only implemented in 'grid', we can still add gradients to 'graphics' plots if we first convert them using the 'gridGraphics' package.

plot(mpg ~ disp, mtcars, pch=16)
library(gridGraphics)
grid.echo()
g <- grid.grab()
box <- grid.grep("box", g, grep=TRUE)
g <- editGrob(g, box$name,
              gp=gpar(fill=linearGradient(c("grey", "white"), x1=.5, x2=.5)))
g <- reorderGrob(g, box$name)
grid.newpage()
grid.draw(g)

Patterns

The fill gpar can also be a general pattern that is based on a 'grid' grob. The pattern function takes a grob and draws it as the fill for another grob.

pat <- pattern(circleGrob(1:3/4, r=unit(5, "mm"),
                          gp=gpar(fill=c("red", "green", "blue"))))
grid.rect(gp=gpar(fill=pat))

pat <- pattern(circleGrob(1:3/4, r=unit(5, "mm"),
                          gp=gpar(fill=c("red", "green", "blue"))))
grid.rect(width=.5, gp=gpar(fill=pat))

pat <- pattern(circleGrob(1:3/4, r=unit(5, "mm"),
                          gp=gpar(fill=c("red", "green", "blue"))))
pushViewport(viewport(gp=gpar(fill=pat)))
grid.rect(width=.5)

Patterns are full drawings so can have gradients and clipping (see "Clipping paths" below).

pat <- circleGrob(1:3/4, r=unit(5, "mm"),
                  vp=viewport(clip=rectGrob(height=unit(5, "mm")),
                              gp=gpar(fill=linearGradient(y1=.5, y2=.5))))
grid.rect(gp=gpar(fill=pattern(pat)))

Patterns can "extend" like gradient fills; probably the most typical use of a pattern will draw a simple shape and then specify extend="repeat" to fill a region. To do this, we must specify a size for the pattern (this specifies a "size" for the "tile" that gets repeated). For example, the code below fills a rectangle with a polka-dot pattern by specifying a circle as the pattern and specifying that the pattern size is just a little bit larger than the circle and specifying that the pattern repeats.

pat <- pattern(circleGrob(r=unit(3, "mm"),
                          gp=gpar(col=NA, fill="grey")),
               width=unit(8, "mm"),
               height=unit(8, "mm"),
               extend="repeat")
pushViewport(viewport(gp=gpar(fill=pat)))
grid.rect(width=.5)

Clipping paths

The clip argument for the viewport function can be a 'grid' grob (in addition to the existing "on", which only enforces a rectangular clipping region). The grob defines a path that is used to set the clipping region.

A simple example is clipping output (in this case text) to a circle. The text and the circle are both drawn in grey and then the text is drawn black with the circle used as a clipping path.

grid.text("testing", gp=gpar(cex=3, col="grey"))
cg <- circleGrob(r=.25, gp=gpar(col="grey"))
grid.draw(cg)
pushViewport(viewport(clip=cg))
grid.text("testing", gp=gpar(cex=3))

A slightly more dramatic example involves clipping text to several circles at once. Note that this still only requires a single grob to specify the clipping path.

grid.text("testing", gp=gpar(cex=3, col="grey"))
cg <- circleGrob(x=1:3/4, r=.1, gp=gpar(col="grey"))
grid.draw(cg)
pushViewport(viewport(clip=cg))
grid.text("testing", gp=gpar(cex=3))

The clipping path is relative to the viewport and clipping paths are inherited like the existing clip settings.

In the following code (after drawing everything in grey without any clipping), we push a viewport with three circles as the clipping path, then we draw the text "test one", which gets clipped to the circles. Next, we push another viewport in the bottom third of the image, which inherits the clipping path, and we draw the text "test two", which also gets clipped to the circles. Finally, we pop those two viewports and push a third viewport in the top third of the image; now there is no clipping path in effect (because we popped the viewport with a clipping path), so when we draw the text "test three" the text is not clipped at all.

cg <- circleGrob(y=1:3/4, r=.1, gp=gpar(col="grey"))
grid.text("test one", gp=gpar(cex=3, col="grey"))
grid.draw(cg)
pushViewport(viewport(y=0, height=1/3, just="bottom"))
grid.text("test two", gp=gpar(cex=3, col="grey"))
popViewport()
pushViewport(viewport(clip=cg))
grid.text("test one", gp=gpar(cex=3))
pushViewport(viewport(y=0, height=1/3, just="bottom"))
grid.text("test two", gp=gpar(cex=3))
popViewport(2)
pushViewport(viewport(y=1, height=1/3, just="top"))
grid.text("test three", gp=gpar(cex=3))

Clipping paths are just paths (outlines of shapes), they do not reflect, for example, stroke width or fill colours.

path <- circleGrob(r=.3, gp=gpar(lwd=30, col="grey", fill=NA))
grid.text("testing", gp=gpar(col="grey", cex=3))
pushViewport(viewport(clip=path))
grid.draw(path)
grid.circle(r=.3)
grid.text("testing", gp=gpar(cex=3))

Although a clipping path can be an arbitrary 'grid' grob, clipping and masks (see below) are disallowed within a clipping path. Any viewport that sets clip=TRUE or clip=FALSE or sets clip to be a grob, or that sets mask to be a grob, will generate a warning and the clipping setting (or the mask) will be ignored for determining the clipping path.

This constraint has been applied for several reasons: to make life easier for graphics devices; because it is very difficult or impossible to achieve some nested clipping effects on some devices; formats like PDF and SVG enforce limits on what can go in a clipping path; and fancy clipping shapes can be achieved by using a mask instead.

Masks

A 'grid' viewport can now have a mask (in addition to clip). This value should be either "inherit" (the default), in which case the viewport inherits the mask from its parent, "none", in which case the viewport does not mask its content, or a 'grid' grob that defines a new mask for the viewport.

The following example defines a circle grob with a radial gradient fill that starts at black in the centre and gradually transitions to transparent at the circumference. A viewport is pushed with this circle grob as the mask, then a rectangle is drawn half the width of the viewport and filled with solid black. The result is that only the parts of the rectangle where the mask is non-transparent are drawn; the retangle is filtered by the mask.

mask <- circleGrob(gp=gpar(col=NA,
                           fill=radialGradient(c("black", "transparent"))))
pushViewport(viewport(mask=mask))
grid.rect(width=.5, gp=gpar(fill="black"))

The following code blends two 'ggplot2' plots using a mask. This time the mask is a linear gradient from black 30% of the way up the page to transparent 70% of the way up the page. First, we draw a plot with the default ggplot2 them, then we draw a black-and-white themed version of the same plot on top, using the mask, so from 30% up the page the black-and-white plot transitions smoothly into the default themed plot.

gg1 <- ggplot(mtcars) +
    geom_point(aes(x=disp, y=mpg))
gg2 <- gg1 +
    theme_bw()
grad <- linearGradient(c("black", "transparent"), x1=.5, x2=.5, y1=.3, y2=.7)
mask <- rectGrob(gp=gpar(col=NA, fill=grad))
print(gg1)
pushViewport(viewport(mask=mask))
print(gg2, newpage=FALSE)

Here is an example of a relatively complex mask that cannot be achieved with clipping paths (in the current proposal) because it would require nesting clipping paths. The mask consists of two narrow rectangles, but the left rectangle is drawn within a viewport with a clipping path. The same effect could be achieved by replacing the clipping path with a mask (nested masks are ok).

mask <- gTree(children=gList(rectGrob(x=.25, width=.3, height=.8,
                                      gp=gpar(fill="black"),
                                      vp=viewport(clip=circleGrob(r=.4))),
                             rectGrob(x=.75, width=.3, height=.8,
                                      gp=gpar(fill="black"))))
pushViewport(viewport(mask=mask))
grid.rect(gp=gpar(fill="grey"))
popViewport()

5. Internal details

Gradients

A patternFill property has been added to the R_GE_gcontext (graphical context) in the graphics engine. This property can be either NULL (indicating no pattern fill) or an R object, which represents a reference to one of the current pattern fills on the graphics device.

This property is in addition to the existing fill property. If patternFill is non-NULL then it overrides fill. This makes it easy for a graphics device to ignore patternFill if it wants to.

A grDevices::linearGradient function has been added. This is not meant to be called directly by users; it is there for 'graphics' and 'grid' to use. The idea is the 'graphics' and 'grid' have user-level interfaces to define gradients and those are converted to graphics engine versions of the gradients for drawing. There is also a grDevices::radialGradient.

A setPattern device function has been added to allow the graphics engine to ask a graphics device to create a pattern. The function takes a "Pattern" object (either a linear or radial gradient so far) and returns an R object (a reference to be used in a patternFill call).

(A graphics device can just always return NULL if it generates a new pattern every time setPattern is called ?)

There is also a releasePattern device function so that, e.g., 'grid' can release a pattern that is just set on a grob (and release all patterns on new page). Specify NULL to release all patterns.

NOTE that the "Pattern" sent to setPattern is an SEXP, which means it is sort of an opaque pointer in C code, so the graphics engine provides a C API for extracting information from a "Pattern" object. For example, R_GE_isPattern can be called to check for a pattern fill, R_GE_patternType can be called to get the pattern type, and R_GE_linearGradientNumStops can be called to get the number of stops from a LinearGradient SEXP.

The Cairo graphics device (x11(type="cairo"), png(type="cairo"), cairo_pdf(), and svg()) has implemented setPattern and releasePattern and maintains an array of patterns. It observes the new R_GE_gcontext.gradientFill setting and uses the value as an index into its array of patterns.

The 'grid' package provides its own linearGradient function for defining a linear gradient in 'grid'. One notable difference is that the start and end points for the gradient can be defined using 'grid' units. The gpar function now also allows a GridPattern as the value for the fill property.

When gpar settings are enforced during drawing a grob, a pattern fill is resolved to a graphics engine Pattern (for example, the start and end points for a linear gradient are converted to locations on the graphics device). If the pattern fill is specified on a viewport, the pattern definition is relative to that viewport; if the pattern fill is specified on a grob, then the pattern definition is relative to the bounding box of the grob.

'grid' calls setPattern to "resolve" a pattern and generates a GridResolvedPattern, which contains the reference from setPattern. This allows 'grid' to reuse the pattern reference, e.g., when drawing multiple shapes from a single grob, or when revisiting a viewport (so that it does not have to ask the graphics device for a new pattern for every shape or for every visit to a viewport).

When drawing a grob, 'grid' resolves the pattern relative to the grob, gets the pattern reference and uses that for every shape that is drawn from this grob, then releases the pattern.

When pushing a viewport, 'grid' resolves the pattern relative to the viewport and stores the resolved pattern in the pushed viewport, so that revisits to the viewport can reuse the resolved pattern. The resolved pattern is only stored in the pushed viewport so that replays of the grid display list (e.g., grid.edit) or the graphics engine display list (e.g., device resize or device copy) will always resolve the pattern again.

Patterns

Patterns use the same R_GE_context slot as gradients; a drawing can have a gradient OR a pattern fill OR a normal fill. Patterns are passed to graphics devices via the same setPattern device function.

There is a grDevices::tilingPattern function for 'graphics' and 'grid' to use to create a pattern.

'grid' creates a GridTilingPattern object, which is a list with a component, f, that contains a function to draw the grob that defines the pattern, plus the location, size, and extend rule for the pattern.

The Cairo graphics device creates a pattern by running the pattern function within a cairo_push_group and cairo_pop_group. This means that the pattern is drawn on a temporary surface and that drawing is turned into a Cairo pattern to use as the source for filling other drawing.

The same pattern resolution in 'grid' is used as for gradients; the same pattern array on the Cairo device is used as for gradients.

Clipping paths

A setClipPath device function has been added to allow the graphics engine (or graphics systems) to ask a device to define a clipping path. The function takes a path and a ref and should return a reference to the clipping path on the device. All three values are SEXPs, but all the device needs to know about the path is that it is an R function (with no arguments) that the device should call to create the clipping path. More on how that works in a second. All the graphics engine (or graphics systems) need to know about the reference to a clipping path is ... nothing. These references are generated by the device, passed back to the graphics engine, then sent in again to the device. The graphics engine (or graphics systems) only store the references and pass them back to the device; the references only have meaning within the graphics device.

There is also a releaseClipPath device function, which takes a reference to a clipping path so that the graphics engine (or graphics systems) can indicate to the device that a clipping path is no longer required.

When a 'grid' viewport is created with a grob as the clip argument, a "GridClipPath" object is created, which is a function that draws the grob. When a 'grid' viewport is pushed, the clipping path is "resolved" by calling setClipPath on the graphics device (via R and C interface functions in the 'grDevices' package).

The Cairo device maintains an array of clipping paths. When a new clipping path is requested (with a call to setClipPath) the device starts a new path and enters an "append mode" before it runs the R function that defines the clipping path. Cairo device functions that perform drawing (e.g., Cairo_Rect) only append shapes to the current path when the device is in "append mode" (when not in append mode, these device functions start a new path, add the shape to the path, set parameters like colour, then stroke and/or fill the path). After the R function has run, the path is closed and the clipping region is set from the path. The path is also saved in the array of clipping paths and an integer index to array is returned as the reference (as an R integer vector).

The 'grid' package stores the clipping path reference that is returned, along with the original "GridClipPath", creating a "GridResolvedClipPath". This enhanced clipping path object is stored in the pushed viewport so that, when a viewport is revisited, 'grid' can send the reference to the clipping path as well as the clipping path itself in the call to setClipPath, which allows the graphics device to reuse cached clipping paths (if it is set up that way).

The Cairo device checks whether the ref argument to setClipPath is non-NULL, in which case it looks up the relevant clipping path in its array of clipping paths and reuses that, rather than generating a new clipping path.

Masks

A setMask device function has been added to allow the graphics engine (or graphics systems) to ask a device to define a mask. The function takes a mask and a ref and should return a reference to the mask on the device. All three values are SEXPs, but all the device needs to know about the mask is that it is an R function (with no arguments) that the device should call to create the mask. As with clipping paths, the references are generated by the device, passed back to the graphics engine, then sent in again to the device. The graphics engine (or graphics systems) only store the references and pass them back to the device; the references only have meaning within the graphics device.

There is also a releaseMask device function, which takes a reference to a mask so that the graphics engine (or graphics systems) can indicate to the device that a mask is no longer required.

The Cairo device maintains an array of masks. When a new mask is requested (with a call to setMask) the device starts a new group (cairo_push_group) so that all drawing on the device is on a temporary surface, calls the mask function, then saves the group as a pattern (cairo_pop_group) on the array of masks and returns the index of the mask in that array.

The 'grid' package stores the mask reference that is returned, along with the original "GridMask", creating a "GridResolvedMask". This enhanced mask object is stored in the pushed viewport so that, when a viewport is revisited, 'grid' can send the reference to the mask as well as the mask itself in the call to setMask; this allows the graphics device to reuse cached masks (if it is set up that way).

The Cairo device also keeps track of the current Mask, as an index into its mask array. This starts as -1, which means there is no mask in effect. setMask updates this index, including resetting to -1 if sent a NULL mask.

The 'grid' package calls setMask whenever a viewport is pushed or revisited to update the current mask on the device.

The shape-drawing functions on the Cairo device check whether the mask is non-negative. If so, rather than drawing directly, they draw to a temporary surface (cairo_push_group, cairo_pop_group), collect the result as a pattern, then mask the pattern, using the current mask, to draw on the main device surface (cairo_mask).

The pdf device

Gradients and clipping paths and masks have also been implemented on the pdf graphics device.

The pdf device maintains an array of "definitions", which are basically strings of PDF code.

Resolving a gradient (via setPattern) creates a definition that contains the PDF code defining the gradient and the array index to that definition is returned as the reference. When a shape is drawn with a gradient fill, code something like /Def3 scn is generated to indicate that definition 3 (a gradient) is to be used as the fill colour. On dev.off, definition strings are written out at the end of the PDF file and the link between /Def3 and the appripriate PDF object number is resolved.

Semi-transparent gradients are a bit more complicated because they require creating an additional "soft mask" definition. In this case, the reference returned is two array indices, one for the gradient and one for the soft mask. When a shape is drawn with a semi-transparent gradient fill an additional piece of code, something like /Def4 gs is generated to enforce the soft mask for the gradient fill. Again, the actual definition string for the soft mask is written out at the end of the PDF file, when links between definition names and object numbers are resolved.

Resolving a pattern creates a definition that contains PDF code to define the pattern, with the array index of that definition returned as a reference. The definition of a pattern contains a "stream" of drawing code, which is obtained by evaluating the R function from the pattern and capturing its output within a temporary definition (that is copied to the pattern definition and not written out itself). When a shape is drawn with a pattern fill, the pattern is set as the fill colour as for gradient fills. Pattern definitions are stored and written out at the end of the PDF file, when links between definition names and object numbers are resolved. An added complication with patterns is that the definition is actually only finalised at this point because the pattern must contain its own Resources Dictionary (it cannot just refer to the document's Resource Dictionary because that contains a reference to the pattern itself!).

Resolving a clipping path (via setClipPath) creates a definition that contains the PDF code that draws the clipping path. Again, the array index of that definition is returned as the reference. The PDF code describing the clipping path is obtained by evaluating the R function for the clipping path and capturing the output within the clipping path definition. The setClipPath call also writes the definition string out (plus W n to enforce the clipping path), but it can simply reuse the definition string (rather than regenerating it) if the reference is not NULL. A clipping path definition is NOT written out at the end of the file (because it is written inline whenever a clipping path is set).

A limitation: the fill rule used by the clipping operator follows the device fill rule. So it is possible to add a path with one fill rule, but have the clipping path use a different fill rule.

Resolving a mask (via setMask) creates a definition that contains the PDF code that defines the mask content. Masks are implemented as alpha soft masks. The mask is described by a "stream" of PDF code that is obtained by evaluating the R function for the mask and capturing its output within a temporary definition (as for patterns). The array index of that definition is returned as the reference. Mask definitions are stored and written out at the end of the PDF file, when links between definition names and object numbers are resolved. The PDF device monitors whether a mask is currently defined and, when drawing normally, if a mask is defined it is enforced with code something like /Def5 gs.

Unlike clipping paths, it is valid to nest masks within masks (and gradients and clipping paths within masks); masks are essentially self-contained drawings. This can be accommodated by the PDF device because all of these features are just stored in definitions as the mask content is captured, and then the definitions are written to file at the end, and only references to other definitions are recored in the mask content.

This implementation on the pdf device provides some evidence that the overall design of graphics definitions support has not been too tightly coupled with the Cairo graphics devices, which gives some hope that other graphics devices will be able to implement at least some of these features.

6. Discussion

The argument to augment the R graphics engine with new graphics features is a simple one: R cannot do some things now, so we should make it so it can. But how important is it to allow R to produce more sophisticated graphical output ?

One motivation for these changes is to allow people to produce more of their figures and diagrams via code. One workflow that people employ is to generate a plot using R and then use Adobe Illustrator to "touch up" the plot. This sort of workflow is not automatable, or reproducible, or shareable, etc. If we can help people to do everything via code and remove manual steps from workflows, that is a big win.

A simple example of the usefulness of clipping paths is the ability to clip to (rectangular) rotated viewports (!).

From my own perspective, having better support for sophisticated graphical features in R would allow me to expand on work already done on importing graphics from other systems into R ('grImport', 'grImport2', 'metapost', 'dvir', 'displayEngine'). Without these features, it is not possible to represent images from other systems properly in R.

Anecdotally, I believe that others would like to have the ability to make use of additional graphics features too. As one piece of hard evidence, this work is being partially funded by a donation from R Studio to the University of Auckland Foundation.

Design

The design of the API for these new features has been focused on flexibility - making things possible. This is reflected particularly in the 'grid' API where clipping paths, patterns, and masks are all defined in terms of a grob.

This flexibility is good in the sense that it makes reasonably complex things possible (e.g., masks containing patterns and clipping paths), but it does place a reasonable burden on the graphics devices. For example, graphics devices must be written to allow for recursion; drawing of a shape may trigger drawing of further shapes (e.g., in order to resolve the fill pattern for the original shape).

This flexibility may also impact on efficiency. For example, it is only currently possible to specify a single pattern fill for a grob (which may draw multiple shapes); pattern fills are not vectorised.

As use-cases appear, there may be a need for adding a simpler interface for patterns, masks, and clipping paths, both to allow graphics devices to support a less flexible API (with less effort) and to allow greater efficiency when drawing a large number of shapes.

Fortunately, the current API passes an SEXP as the specification for a pattern or clipping path or mask to resolve, so it should be possible in future to pass other obects via this interface with, say, classes used to distinguish between different sorts of specifications.

External graphics devices

A device must implement dev->setPattern, but it can just return NULL if unsupported. A device can also just create a new pattern every time. If it returns a meaningful pattern reference, it must do something sensible with that reference when provided via R_GE_gcontent.patternFill.

There is new R_GE_gcontent.patternFill, but a device can just ignore it (because R_GE_gcontext.fill is always set; it will just be "transparent" when a pattern fill has been specified).

A device must implement dev->setClipPath, but can just return NULL. There are three levels of support: none, in which case the device ignores the clipping path request (and returns NULL); non-caching, where the device sets clipping paths, but returns NULL, so it will always generate a new clipping path on every request; caching, where the device sets clipping paths, caches clipping paths, and returns a reference to the cache, and the reference only has to make sense to the device.

A device must implement dev->setMask, but can just return NULL. As with clipping paths, there are three levels of support: none, non-caching, and caching.

As an example of the (minimal) changes necessary to update a device (WITHOUT support for any of the new features), the following diff shows the changes to get the quartz device running on macOS (at r78533).

The plan is to implement at least one caching device (Cairo, plus maybe PDF), a non-caching device (always generates new definition on every use), and a non-implementing device (does not support features). These then provide templates for external device maintainers.

Efficiency

Definitions are resolved every time a graphics system calls the graphics engine because the graphics engine is a "flat" coordinate system whereas the graphics systems (particularly 'grid' units) are dynamic and depend on the current drawing context.

This may present some challenges for implementing graphical definitions efficiently. However, the worst case scenario is something like filling thousands of data symbols with the same gradient fill.

Another issue is revisiting viewports (up/down). This could also generate a lot of redundant recalculations (or very large file sizes). Though not at the same scale as the thousands-of-shapes problem, it could be more common (?). This also can happen during calculation of things like "grobWidth" units.

The inefficiency could bite both in terms of speed and, for file-based devices, memory.

In both cases, allowing the graphics device to cache its definitions should be beneficial.

The plan is to actually do some measurements to demonstrate that there is a problem and that caching is helpful (or not).

TODOs

Neither Cairo nor PDF devices have currently attempted to include text output in clipping paths or masks.

The R graphics engine does not accommodate the idea of a gradient fill within text.

The Cairo device currently has a fixed limit on the number of definitions allowed (per page).

Issues

What to do about pattern fill on multiple shapes from single grob: grid.rect(width=1:2/3, gp=gpar(gradientFill=linearGradient())) Currently just resolving pattern based on bounding box of all shapes.

The current implementation of pattern fills in 'grid' "nicely" incorporates them into the gpar(fill=) interface. This may have implications for packages that build on 'grid' if they try to manipulate 'grid' gpars in any way, other than just setting them (e.g., 'gridSVG' converts gpar values to SVG display attributes).

These changes introduce some "modal" behaviour (like whether the device is "appending" or "drawing"); how to make sure we elegantly unwind when an error occurs (e.g., when hit an error while playing/recording a clipping path) ? One defence being employed for Cairo and PDF devices is to only produce warnings rather than errors when things go wrong (in the case of PDF, ask the use politely to shut down the PDF device). Also, modes are reset on new page, so it may be possible to recover a corrupted device by starting a new page.

What will the impact be on saved/restored plots ? (the graphics engine display list and snapshots thereof) I have some test cases working within-session. It would certainly be a bad idea to save from R 4 and replay in R 3 (though the reverse should be ok?).

The fact that clipping paths (mostly) and masks can be arbitrary 'grid' grobs places a reasonable burden on the graphics device. The device has to evaluate an R function and capture the resulting output. A possible future path for devices that cannot meet these demands may be to allow alternate objects as clipping paths and masks. Currently, the SEXP sent to the device is an R function, but that could conceivably be limited to simpler R objects, like a simpler description of one or more shapes. That might make the device's job a LOT easier.

PDF viewers: can get some variation between viewers (e.g., evince, xpdf, gs/gv, and firefox). Can also get (bigger) variation between versions of viewers. This bites harder for more complex drawings. For example, nested tiling patterns are NOT good in gv 3.7.4, xpdf 3.04, and evince 3.18.2 (Ubuntu 16.04), but are fine in gv ??, xpdf ??, and evince ?? (Ubuntu 18.04).

'gridSVG'

Reasons for keeping 'gridSVG': still the only way to support some stuff (e.g., Cairo does not support filters); still the only way to retain structure and labelling when export.

The impact on 'gridSVG' may be interesting. There may be differences between the way that features are implemented in 'grid' vs 'gridSVG'. For example, gradient fills might be added to gpar in 'grid', rather than creating a separate class of grob (as in 'gridSVG'). This means that 'gridSVG' may need updating to work with new 'grid' gradients. (Yes!)

'grImport2' is another interesting case - it may still end up with ext=gridSVG as well as being able to draw directly to normal R graphics devices.

Fonts

Proper font support is another issue entirely. Currently addressed by packages like 'showtext' and 'extrafont'. Maybe a future project.

Extensions

It would be nice if the graphics device API allowed for people to write R code that talked directly to the graphics device without having to be controlled by (and filtered through) the graphics engine. This would allow, for example, people to access features on graphics devices that are still not supported by the graphics engine (sort of like what 'showtext' sneakily does now to hijack text rendering; 'tikzDevice' also does something like this for creating "tags" in tikZ output ?). This would be a little like DVI specials (?) TODO: look at what 'showtext' and 'tikzDevice' need to do and how they currently work around the graphics engine. Another example that has been mentioned is hyperlinks.

Footnotes

¹ It is actually possible to get this particular result currently in R graphics. We can generate a path consisting of the border of the entire image with the border of New Zealand inside it (i.e., a rectangular path with a hole the shape of New Zealand) and fill that with opaque white. However, that sort of workaround is more awkward and often runs into difficulties, for example, if we want to add other output to the image, but not obscure it with the opaque white fill. In general, we can generate may different graphical results just with the existing R graphics features, but we may have to work very hard to get what we want. It could be said that all this proposal is aiming for is greater convenience, but in some cases it is MUCH greater convenience.

Graphical Definitions in R Graphics

Table of Contents:

1. Introduction

2. Issues

3. Features

4. Demonstrations

Gradients

Patterns

Clipping paths

Masks

5. Internal details

Gradients

Patterns

Clipping paths

Masks

The `pdf` device

6. Discussion

Design

External graphics devices

Efficiency

TODOs

Issues

'gridSVG'

Fonts

Extensions

Footnotes

Graphical Definitions in R Graphics

Table of Contents:

1. Introduction

2. Issues

3. Features

4. Demonstrations

Gradients

Patterns

Clipping paths

Masks

5. Internal details

Gradients

Patterns

Clipping paths

Masks

The pdf device

6. Discussion

Design

External graphics devices

Efficiency

TODOs

Issues

'gridSVG'

Fonts

Extensions

Footnotes

The `pdf` device