http://orcid.org/0000-0002-3224-8858
by Paul Murrell
http://orcid.org/0000-0002-3224-8858
Version 1: original publication
This document
by Paul
Murrell is licensed under a Creative
Commons Attribution 4.0 International License.
This document describes a plan to expand the R graphics engine to support graphical definitions: graphical features like gradients, patterns, masks, and clipping paths.
The code changes are being
tracked in a
subversion branch called R-defs.
This is a living document that will evolve as the code is developed.
The Features Section outlines the features that I
would like to target and the
Demonstrations Section provides a glimpse of how some
new features are currently implemented at the user
level (in the R-defs branch).
The Internal details Section has more on the internal design and internal code changes that have been made.
The Issues Section contains some of the major design issues that I have identified so far and the Discussion Section contains some questions and (some) answers that I have already thought a bit about, including the changes that would be imposed on graphics device maintainers.
The R graphics engine only supports a limited set of graphical features: basic shapes like lines, rectangles, and circles; paths (including shapes with holes); text; raster images; and rectangular clipping regions. These features are sufficient to produce a wide range of statistical plots, but there are many things that R graphics cannot do.
For example, the image below contains a set of filled contours within a map of New Zealand. The natural way to achieve this result would be to draw the filled contours and apply a clipping path to the filled contours based on the boundary of New Zealand. That is not something that can be achieved in R graphics (because R graphics only supports rectangular clipping regions).1
The plan is to add new graphical features to the R graphics engine, such as clipping paths, in order to expand the range of graphical images that can be produced with R.
There are a number of issues that need to be resolved when adding a new graphics feature to the R graphics engine.
Can we generate the same output on all R graphics devices?
This horse has already bolted. The output on different R graphics devices has never been identical and there are already graphics features that are not supported on all R graphics devices (e.g., translucent colours are not supported on the PostScript device). The plan is to double-down on this approach and just allow graphics devices to opt out of any or all of the new features.
This is partly because not all devices will be capable of supporting the new features and partly because it makes it easy to roll out the new features; graphics devices can initially opt out of all new features and add support as and when they are able.
SEXPs in graphics devices:
Can we avoid sending R's SEXP structures to
graphics devices ?
In the early days, there was an effort to avoid using R's internal C structures within graphics devices (so that they only had to include R's graphics-related header files and they did not have to worry about complications like R's memory management).
This horse has also bolted. From R 2.13.2 (at least), when the
cap graphics device procedure was added (to allow
raster "screen shots" of current device state), devices have
been able to at least return R structures.
The plan is again to double-down. Some of the new features will require sending relatively complex information to the graphics device (e.g., gradient information and pattern information), so rather than define complex new C structures, it makes sense to make use of R's existing flexible structures.
One problem is that an SEXP in C is basically an opaque pointer. How does the device know about changes to the SEXP structure? The proposal is to write a C API for interacting with the SEXP structures and encourage devices to use that. An example has been developed for gradient fills (see the Demonstrations Section).
How to decide which graphics features to support?
This is almost the inverse of the device-independence problem. Do we shoot for the union of all features supported by all possible graphics devices ? Or do we target the feature set of one particular device or format (e.g., SVG) ?
The plan is to take ideas from everywhere. There are features that will only be supported by some formats (e.g., SVG has filters, but Cairo has only very limited filter support, whereas Cairo has mesh patterns, but SVG does not). There are also subtleties between formats, like the 'gradientUnits' in SVG that are not supported in Cairo, but we can hopefully make an interface for these (at least in 'grid') even if the graphics engine only goes as far as Cairo support.
The PDF feature set is probably the most sophisticated (?). For example, most graphics formats have linear and radial gradients, Cairo graphics has "mesh patterns" that SVG does not, but PDF has seven types of "shaders" (type 2=Linear gradients, 3=Radial gradients, 4&5=Gouraud-shaded triangle meshes, 6=Coons patch meshes, and 7=Tensor-product patch meshes) including Function-based shaders (type 1) that define the colour of every point in the image using a mathematical function. Gradient meshes have been proposed for SVG, but it does not look like they will make it into SVG 2.0.
Will the new features be implemented for 'graphics' or 'grid' or both ?
The new features will be implemented within the graphics engine. This means that, in theory, interfaces can be made for both 'graphics' and 'grid'.
The plan is to focus, initially at least, on 'grid'. This is because 'grid' allows more flexible and interesting definitions, for example, the use of units in locating gradient stops. 'grid' also allows for things like adding features by post-editing because we are able to address graphical objects by name.
It is also possible in the meantime
to apply the new features to 'graphics'
output via gridGraphics::grid.echo
and functions like grid::grid.edit.
Which graphics devices should implement which features ?
Obviously, only some features can be added to some graphics devices because of limitations in graphics formats and/or graphics libraries. This issue relates more to when support gets added to different graphics devices. The problem is that implementing all possible features on all possible devices is a big job (made more difficult by the development environment on some platforms, like Windows).
The proposal is to be feature-greedy and just focus initially on the Cairo graphics devices. This gives a reasonable coverage (raster on Linux at least, PDF, and SVG) from a single set of code changes. The idea would be to introduce as many features as possible, on the Cairo devices, for 4.0.0 and then fill in the gaps. Once the Cairo changes are implemented, they will hopefully provide a template so that others could contribute with patches for other devices.
A danger is that only implementing for Cairo may "bias" the graphics engine implementation towards that device and cause problems for implementations on other graphics devices. Perhaps implementing for two devices (e.g., also the PDF device) rather than one would be more prudent.
Another issue is that the default, "graphapp", Windows device is NOT going to be able to support these features very easily (if at all). GraphApp does not support any of these features (?) (even in the latest version and not even the underlying Windows GDI supports gradient fills (though it looks like clipping paths are possible). OTOH, implementing features on the Cairo device makes them available on all platforms (?). I am unsure about whether the RStudio plot pane (RStudioGD) will be able to support these features (across platforms).
Third-party graphics devices can of course update as much as they
want when they want, though they will have to immediately
provide some minimal stubs for the changes to the graphics
device API (GraphicsDevice.h).
This section briefly describes the set of graphical features that are currently being considered.
This includes linear gradients and radial gradients. Gradients can be applied to both fills and strokes (at least on some devices).
Gradient meshes are also supported on several devices (Cairo and PDF).
Pattern fills are similar to gradient fills in that they define a "paint" for filling a region. The difference is that a pattern fill is based on "tiling" a smaller image.
There are other sorts of pattern fills in some formats (e.g., function-based shaders in PDF).
Allow clipping to an arbitrary path rather than just a rectangle. This relies on being able to specify a "path" separate from drawing (stroking or filling) it.
Similar to clipping except that affect translucency of rendered output (rather than just visible or not). Again, relies on being able to describe paths without drawing them.
These are essentially raster operations, but a format like SVG allows them to be defined as part of a vector image (they are applied as part of the rasterization of the vector image, for example, when drawing to screen).
Implementing these in the graphics engine is probably too hard. I don't think any of the standard graphics devices will support them (including the Cairo svg device).
These may have to remain the domain of packages like 'gridSVG' (which provides full access to SVG features) or 'magick' (which allows us to apply raster operations to raster images), possibly in combination with the 'rasterize' package.
It looks like Thomas Lin Pedersen has been thinking along similar lines (to the latter) with a 'ggfx' package.
A new R-defs
branch has been created for experimenting.
So far, that includes the addition of support for
linear and radial gradient fills, patterns, clipping paths, and masks
on both Cairo graphics devices and the pdf device
(except for patterns),
with a 'grid' interface. The following set of examples
show how the 'grid' interface has been implemented so far.
NOTE: support for graphics definitions has only been implemented
for Cairo-based graphics devices and the pdf device
so far. Other graphics devices
should run, but will not produce output from graphics definitions.
Third-party graphics devices will hopefully error out (if they make
use of R_GE_checkVersionOrDie), but otherwise they
are likely to segfault until they are updated.
On the positive side, Cairo devices and the pdf device
can be used on all major platforms,
so there is already an option for accessing these new features
on all major platforms.
The fill gpar can now be a linear gradient
(as well as a normal R colour).
A linear gradient is defined by a start point (default: bottom-left) and an end point (default: top-right) and a series of "stops". Each stop is defined by a position on the line from start point to end point, plus a colour. If the first stop is not at the start point, or the last stop is not at the end point, we can specify how the gradient "extends" beyond the first and last stops.
The following code creates a linear gradient that starts one inch off the bottom of the image, proceeds vertically to the top of the image, and transitions from red to yellow then back to red.
When the gradient fill is specified on a grob, the start and end points are relative to the grob bounding box. The following code fills a rectangle in the central quarter of the image with the default linear gradient; the gradient starts at the bottom-left of the rectangle and ends at the top-right of the rectangle.
A gradient fill can also be specified on a viewport, in which case the gradient is relative to the viewport and all subsequent drawing makes use of that gradient (unless a new explicit fill is specified).
In the code below, we push a viewport that fills the whole image and specify the default linear gradient fill, so the gradient is relative to the whole image. We then draw a rectangle, with no explicit fill, so it "inherits" the gradient from the viewport.
In the next example, we have the same viewport, but we draw the rectangle only in the central quarter of the image. This shows that the gradient that the rectangle inherits is relative to the viewport rather than being relative to the rectangle.
The next example shows another variation on inheritance of gpar settings. This time a viewport is pushed with the default linear gradient and we draw a rectangle in the left third of the image that inherits that gradient (relative to the viewport). Then we push another viewport with a green fill and draw a rectangle within that viewport (in the centre of the image) and that rectangle inherits the green fill. Finally, we pop the second viewport and return to the first viewport and restore its linear gradient so that when we draw a rectangle in the right third of the image it again inherits the linear gradient (relative to the viewport).
The next example also demonstrates inheritance of gpar settings, but this time it is inheritance between viewports. We push a viewport with the default gradient, then push another viewport in the central quarter of the image, then draw a rectangle. The second viewport inherits the gradient from the first viewport and then the rectangle inherits the gradient from the second viewport. The important point is that the gradient is relative to the first viewport, which is where it was defined.
The next example just shows that gradients work with translucent colours. We draw the text "Reveal" then over the top draw a rect with a gradient fill that transitions from (opaque) white to transparent.
The next example shows that radial gradients work via the same interface. It is expected that fill patterns would work the same way.
As with linear gradients, we have control over the start and end of the radial gradient, though in the radial case it is a start circle and an end circle. In the code below, we make a gradient that starts near the top right of the image (with white) and ends with a circle that fills the image (black).
The next example shows that we can add a gradient to an existing image, in this case a 'ggplot2' plot. Here we edit the background of the plot panel to add a grey-to-white gradient.
The next example is similar, but demonstrates that, even with gradients only implemented in 'grid', we can still add gradients to 'graphics' plots if we first convert them using the 'gridGraphics' package.
The fill gpar can also be a general pattern that is based
on a 'grid' grob. The pattern function takes a grob
and draws it as the fill for another grob.
Patterns on grobs are relative to the grob.
Patterns on viewports are relative to the viewport.
Patterns are full drawings so can have gradients and clipping (see "Clipping paths" below).
Patterns can "extend" like gradient fills; probably the most
typical use of a pattern will draw a simple shape and then
specify extend="repeat"
to fill a region.
To do this, we must specify a size for the pattern (this specifies
a "size" for the "tile" that gets repeated).
For example, the code below fills a rectangle with a polka-dot
pattern by specifying a circle as the pattern and specifying that
the pattern size is just a little bit larger than the circle
and specifying that the pattern repeats.
The clip argument for the viewport function
can be a 'grid' grob (in addition to the existing
"on", which only enforces a rectangular clipping region).
The grob defines a path that is used
to set the clipping region.
A simple example is clipping output (in this case text) to a circle. The text and the circle are both drawn in grey and then the text is drawn black with the circle used as a clipping path.
A slightly more dramatic example involves clipping text to several circles at once. Note that this still only requires a single grob to specify the clipping path.
The clipping path is relative to the viewport and
clipping paths are inherited like the existing clip
settings.
In the following code (after drawing everything in grey without any clipping), we push a viewport with three circles as the clipping path, then we draw the text "test one", which gets clipped to the circles. Next, we push another viewport in the bottom third of the image, which inherits the clipping path, and we draw the text "test two", which also gets clipped to the circles. Finally, we pop those two viewports and push a third viewport in the top third of the image; now there is no clipping path in effect (because we popped the viewport with a clipping path), so when we draw the text "test three" the text is not clipped at all.
Clipping paths are just paths (outlines of shapes), they do not reflect, for example, stroke width or fill colours.
Although a clipping path can be an arbitrary 'grid' grob,
clipping and masks (see below) are disallowed within a clipping path.
Any viewport that sets clip=TRUE or clip=FALSE
or sets clip to be a grob, or that sets mask
to be a grob, will generate a warning and the clipping setting
(or the mask) will be ignored for determining the clipping path.
This constraint has been applied for several reasons: to make life easier for graphics devices; because it is very difficult or impossible to achieve some nested clipping effects on some devices; formats like PDF and SVG enforce limits on what can go in a clipping path; and fancy clipping shapes can be achieved by using a mask instead.
A 'grid' viewport can now have a mask (in addition to
clip). This value should be either "inherit"
(the default), in which case the viewport inherits the mask from
its parent, "none", in which case the viewport does not
mask its content, or a 'grid' grob that defines a new mask for the
viewport.
The following example defines a circle grob with a radial gradient fill that starts at black in the centre and gradually transitions to transparent at the circumference. A viewport is pushed with this circle grob as the mask, then a rectangle is drawn half the width of the viewport and filled with solid black. The result is that only the parts of the rectangle where the mask is non-transparent are drawn; the retangle is filtered by the mask.
The following code blends two 'ggplot2' plots using a mask. This time the mask is a linear gradient from black 30% of the way up the page to transparent 70% of the way up the page. First, we draw a plot with the default ggplot2 them, then we draw a black-and-white themed version of the same plot on top, using the mask, so from 30% up the page the black-and-white plot transitions smoothly into the default themed plot.
Here is an example of a relatively complex mask that cannot be achieved with clipping paths (in the current proposal) because it would require nesting clipping paths. The mask consists of two narrow rectangles, but the left rectangle is drawn within a viewport with a clipping path. The same effect could be achieved by replacing the clipping path with a mask (nested masks are ok).
A patternFill
property has been added to the R_GE_gcontext
(graphical context)
in the graphics engine.
This property can be either NULL
(indicating no pattern fill) or an R object, which
represents a reference to one of the current pattern fills on
the graphics device.
This property is in addition to the existing fill
property. If patternFill is non-NULL
then it overrides fill. This makes it easy
for a graphics device to ignore patternFill
if it wants to.
A grDevices::linearGradient function has been added.
This is not meant to be called
directly by users; it is there for 'graphics' and 'grid' to use.
The idea is the 'graphics' and 'grid' have user-level
interfaces to define gradients and those are
converted to graphics engine versions of the gradients
for drawing. There is also a grDevices::radialGradient.
A setPattern device function has been added to
allow the graphics engine to ask a graphics device to create
a pattern. The function takes a "Pattern" object
(either a linear or radial gradient so far) and returns an R object
(a reference to be used in a patternFill call).
(A graphics device can just always return NULL
if it generates
a new pattern every time setPattern is called ?)
There is also a releasePattern device function
so that, e.g., 'grid' can release a pattern that is just set on
a grob (and release all patterns on new page). Specify
NULL
to release all patterns.
NOTE that the "Pattern" sent to
setPattern is an SEXP,
which means it is sort of an opaque pointer in C code,
so the graphics engine provides a C API for extracting
information from a "Pattern" object.
For example,
R_GE_isPattern can be called to check for a pattern fill,
R_GE_patternType can be called to get the pattern type, and
R_GE_linearGradientNumStops can be called
to get the number of stops from a LinearGradient SEXP.
The Cairo graphics device (x11(type="cairo"),
png(type="cairo"),
cairo_pdf(), and svg())
has implemented setPattern
and releasePattern and maintains an array
of patterns. It observes the new
R_GE_gcontext.gradientFill setting
and uses the value
as an index into its array of patterns.
The 'grid' package provides its own linearGradient
function for defining a linear gradient in 'grid'.
One notable difference is that the start and end points for
the gradient can be defined using 'grid' units.
The gpar function now also allows
a GridPattern as the value for the
fill property.
When gpar settings are enforced during drawing a grob, a
pattern fill is resolved to a graphics engine
Pattern (for example, the start and end
points for a linear gradient are converted to locations on the graphics
device).
If the pattern fill is specified on a viewport, the pattern definition
is relative to that viewport; if the pattern fill is specified
on a grob, then the pattern definition is
relative to the bounding box of the grob.
'grid' calls setPattern to "resolve" a pattern
and generates a GridResolvedPattern, which contains
the reference from setPattern. This allows
'grid' to reuse the pattern reference, e.g., when drawing
multiple shapes from a single grob, or when revisiting a
viewport (so that it does not have to ask the graphics device
for a new pattern for every shape or for every visit to a viewport).
When drawing a grob, 'grid' resolves the pattern relative to the grob, gets the pattern reference and uses that for every shape that is drawn from this grob, then releases the pattern.
When pushing a viewport, 'grid' resolves
the pattern relative to the viewport
and stores the resolved pattern in the pushed viewport, so that
revisits to the viewport can reuse the resolved pattern.
The resolved pattern is only stored in the pushed viewport so
that replays of the grid display list (e.g., grid.edit)
or the graphics engine display list (e.g., device resize or
device copy) will always resolve the pattern again.
When starting a new page, 'grid' releases all patterns.
Patterns use the same R_GE_context slot as gradients;
a drawing can have a gradient OR a pattern fill OR a normal fill.
Patterns are passed to graphics devices via the same
setPattern device function.
There is a grDevices::tilingPattern function for
'graphics' and 'grid' to use to create a pattern.
'grid' creates a GridTilingPattern object,
which is a list with a component, f,
that contains a function to draw the grob that defines the pattern,
plus the location, size, and extend rule for the pattern.
The Cairo graphics device creates a pattern by running the pattern
function within a cairo_push_group and
cairo_pop_group. This means that the pattern
is drawn on a temporary surface and that drawing is turned into
a Cairo pattern to use as the source for filling other drawing.
The same pattern resolution in 'grid' is used as for gradients; the same pattern array on the Cairo device is used as for gradients.
A setClipPath device function has been added to allow
the graphics engine (or graphics systems) to ask a device to
define a clipping path. The function takes a path
and a ref and should return a reference to
the clipping path on the device. All three values are SEXPs,
but all the device needs to know about the path is that
it is an R function (with no arguments) that the device should
call to create the clipping path. More on how that works in a second.
All the graphics engine (or graphics systems) need to know about the
reference to a clipping path is ... nothing. These references are
generated by the device, passed back to the graphics engine, then
sent in again to the device. The graphics engine (or graphics
systems) only store the references and pass them back to the device;
the references only have
meaning within the graphics device.
There is also a releaseClipPath device function,
which takes a reference to a clipping path so that the graphics engine
(or graphics systems) can indicate to the device that a clipping path
is no longer required.
When a 'grid' viewport is created with a grob as the clip
argument, a "GridClipPath" object is created, which
is a function that draws the grob. When a 'grid' viewport is pushed,
the clipping path is "resolved" by calling setClipPath
on the graphics device
(via R and C interface functions in the 'grDevices' package).
The Cairo device maintains an array of clipping paths.
When a new clipping path is requested (with a call to
setClipPath) the device starts a new path
and enters an "append mode"
before it runs the R function
that defines the clipping path. Cairo device functions that perform
drawing (e.g., Cairo_Rect) only append shapes to
the current path when the device is in "append mode" (when not in
append mode, these device functions start a new path, add the shape
to the path, set
parameters like colour, then stroke and/or fill the path).
After the R function has run, the path is closed and the clipping
region is set from the path.
The path is also saved in the array of clipping paths and
an integer index to array is returned as the reference (as an
R integer vector).
The 'grid' package stores the clipping path reference that is returned,
along with the original "GridClipPath", creating a
"GridResolvedClipPath". This enhanced clipping path
object is stored in the pushed viewport so that, when a viewport
is revisited, 'grid' can send the reference to the clipping path
as well as the clipping path itself in the call to setClipPath,
which allows the graphics device to reuse cached clipping paths
(if it is set up that way).
The Cairo device checks whether the ref argument
to setClipPath is non-NULL, in which case it looks
up the relevant clipping path in its array of clipping paths
and reuses that, rather than generating a new clipping path.
'grid' releases all clipping paths on a grid.newpage.
A setMask device function has been added to allow
the graphics engine (or graphics systems) to ask a device to
define a mask. The function takes a mask
and a ref and should return a reference to
the mask on the device. All three values are SEXPs,
but all the device needs to know about the mask is that
it is an R function (with no arguments) that the device should
call to create the mask. As with clipping paths, the references are
generated by the device, passed back to the graphics engine, then
sent in again to the device. The graphics engine (or graphics
systems) only store the references and pass them back to the device;
the references only have
meaning within the graphics device.
There is also a releaseMask device function,
which takes a reference to a mask so that the graphics engine
(or graphics systems) can indicate to the device that a mask
is no longer required.
The Cairo device maintains an array of masks.
When a new mask is requested (with a call to
setMask) the device starts a new group
(cairo_push_group) so that all drawing
on the device is on a temporary surface, calls the mask function,
then saves the group as a pattern (cairo_pop_group)
on the array of masks and returns the index of the mask in that
array.
The 'grid' package stores the mask reference that is returned,
along with the original "GridMask", creating a
"GridResolvedMask". This enhanced mask
object is stored in the pushed viewport so that, when a viewport
is revisited, 'grid' can send the reference to the mask
as well as the mask itself in the call to setMask;
this allows the graphics device to reuse cached masks
(if it is set up that way).
The Cairo device also keeps track of the current Mask, as an index
into its mask array.
This starts as -1, which means there is no mask in effect.
setMask updates this index, including resetting to
-1 if sent a NULL mask.
The 'grid' package calls setMask whenever a viewport
is pushed or revisited to update the current mask on the device.
The shape-drawing functions on the Cairo device check whether
the mask is non-negative. If so, rather than drawing directly,
they draw to a temporary surface (cairo_push_group,
cairo_pop_group), collect the result as a pattern,
then mask the pattern, using the current mask, to draw on the
main device surface (cairo_mask).
pdf device
Gradients and clipping paths and masks have also been implemented on
the pdf graphics device.
The pdf device maintains an array of "definitions",
which are basically strings of PDF code.
Resolving a gradient (via setPattern) creates
a definition that contains the PDF code defining the gradient
and the array index to that definition is returned as the
reference.
When a shape is drawn with a gradient fill, code something like
/Def3 scn is generated to indicate that definition 3
(a gradient) is to be used as the fill colour.
On dev.off, definition strings are written out at the end of
the PDF file and the link between /Def3 and the
appripriate PDF object number is resolved.
Semi-transparent gradients are a bit more complicated because
they require creating an additional "soft mask" definition.
In this case, the reference returned is two array indices, one for
the gradient and one for the soft mask.
When a shape is drawn with a semi-transparent gradient fill an
additional piece of code, something like /Def4 gs
is generated to enforce the soft mask for the gradient fill.
Again, the actual definition string for the soft mask is written
out at the end of the PDF file, when links between definition names
and object numbers are resolved.
Resolving a pattern creates a definition that contains PDF code to define the pattern, with the array index of that definition returned as a reference. The definition of a pattern contains a "stream" of drawing code, which is obtained by evaluating the R function from the pattern and capturing its output within a temporary definition (that is copied to the pattern definition and not written out itself). When a shape is drawn with a pattern fill, the pattern is set as the fill colour as for gradient fills. Pattern definitions are stored and written out at the end of the PDF file, when links between definition names and object numbers are resolved. An added complication with patterns is that the definition is actually only finalised at this point because the pattern must contain its own Resources Dictionary (it cannot just refer to the document's Resource Dictionary because that contains a reference to the pattern itself!).
Resolving a clipping path (via setClipPath) creates
a definition that contains the PDF code that draws the clipping path.
Again, the array index of that definition is returned as the reference.
The PDF code describing the clipping path is obtained by
evaluating the R function for the clipping path and capturing
the output within the clipping path definition.
The setClipPath call also writes the definition string
out (plus W n to enforce the clipping path), but
it can simply reuse the definition string (rather than regenerating it)
if the reference is not NULL.
A clipping path definition is NOT written out at the end of the file
(because it is written inline whenever a clipping path is set).
A limitation: the fill rule used by the clipping operator follows the device fill rule. So it is possible to add a path with one fill rule, but have the clipping path use a different fill rule.
Resolving a mask (via setMask) creates a definition
that contains the PDF code that defines the mask content.
Masks are implemented as alpha soft masks.
The mask is described by a "stream" of PDF code that is obtained
by evaluating the R function for the mask and capturing its output
within a temporary definition (as for patterns).
The array index of that definition is returned as the reference.
Mask definitions are stored and written out at the end of the PDF file,
when links between definition names and object numbers are resolved.
The PDF device monitors whether a mask is currently defined and,
when drawing normally, if a mask is defined it is enforced with
code something like /Def5 gs.
Unlike clipping paths, it is valid to nest masks within masks (and gradients and clipping paths within masks); masks are essentially self-contained drawings. This can be accommodated by the PDF device because all of these features are just stored in definitions as the mask content is captured, and then the definitions are written to file at the end, and only references to other definitions are recored in the mask content.
This implementation on the pdf device
provides some evidence that the overall design of graphics
definitions support has not been too
tightly coupled with the Cairo graphics devices, which gives
some hope that other graphics devices will be able to implement
at least some of these features.
The argument to augment the R graphics engine with new graphics features is a simple one: R cannot do some things now, so we should make it so it can. But how important is it to allow R to produce more sophisticated graphical output ?
One motivation for these changes is to allow people to produce more of their figures and diagrams via code. One workflow that people employ is to generate a plot using R and then use Adobe Illustrator to "touch up" the plot. This sort of workflow is not automatable, or reproducible, or shareable, etc. If we can help people to do everything via code and remove manual steps from workflows, that is a big win.
A simple example of the usefulness of clipping paths is the ability to clip to (rectangular) rotated viewports (!).
From my own perspective, having better support for sophisticated graphical features in R would allow me to expand on work already done on importing graphics from other systems into R ('grImport', 'grImport2', 'metapost', 'dvir', 'displayEngine'). Without these features, it is not possible to represent images from other systems properly in R.
Anecdotally, I believe that others would like to have the ability to make use of additional graphics features too. As one piece of hard evidence, this work is being partially funded by a donation from R Studio to the University of Auckland Foundation.
The design of the API for these new features has been focused on flexibility - making things possible. This is reflected particularly in the 'grid' API where clipping paths, patterns, and masks are all defined in terms of a grob.
This flexibility is good in the sense that it makes reasonably complex things possible (e.g., masks containing patterns and clipping paths), but it does place a reasonable burden on the graphics devices. For example, graphics devices must be written to allow for recursion; drawing of a shape may trigger drawing of further shapes (e.g., in order to resolve the fill pattern for the original shape).
This flexibility may also impact on efficiency. For example, it is only currently possible to specify a single pattern fill for a grob (which may draw multiple shapes); pattern fills are not vectorised.
As use-cases appear, there may be a need for adding a simpler interface for patterns, masks, and clipping paths, both to allow graphics devices to support a less flexible API (with less effort) and to allow greater efficiency when drawing a large number of shapes.
Fortunately, the current API passes an SEXP as the specification for a pattern or clipping path or mask to resolve, so it should be possible in future to pass other obects via this interface with, say, classes used to distinguish between different sorts of specifications.
How much work needs to be done to upgrade a graphics device for these changes ?
A device must implement dev->setPattern, but it can just
return NULL if unsupported.
A device can also just create a new pattern every time.
If it returns a meaningful pattern reference, it must do something sensible
with that reference
when provided via R_GE_gcontent.patternFill.
A device
must implement dev->releasePattern, but it can just do nothing.
There is new R_GE_gcontent.patternFill, but a device
can just ignore it
(because R_GE_gcontext.fill is always set; it will just
be "transparent" when a pattern fill has been specified).
A device
must implement dev->setClipPath, but can just return
NULL.
There are three levels of support: none, in which case the device
ignores the clipping path request (and returns NULL); non-caching,
where the device sets clipping paths, but returns NULL, so it will
always generate a new clipping path on every request; caching,
where the device sets clipping paths, caches clipping paths,
and returns a reference to the cache, and the reference only has
to make sense to the device.
A device
must implement dev->releaseClipPath, but can just do nothing.
A device must implement
dev->setMask, but can just return NULL.
As with clipping paths, there are three levels of support:
none, non-caching, and caching.
A device
must implement dev->releaseMask, but can just do nothing.
As an example of the (minimal) changes necessary to update a device
(WITHOUT support for any of the new features), the following
diff shows the changes to get the quartz device
running on macOS (at r78533).
@@ -371,6 +371,12 @@
+static int RQuartz_setPattern(SEXP pattern, pDevDesc dd);
+static void RQuartz_releasePattern(int index, pDevDesc dd);
+static SEXP RQuartz_setClipPath(SEXP path, SEXP ref, pDevDesc dd);
+static void RQuartz_releaseClipPath(SEXP ref, pDevDesc dd);
+static SEXP RQuartz_setMask(SEXP path, SEXP ref, pDevDesc dd);
+static void RQuartz_releaseMask(SEXP ref, pDevDesc dd);
@@ -430,6 +436,13 @@
+ dev->setPattern = RQuartz_setPattern;
+ dev->releasePattern = RQuartz_releasePattern;
+ dev->setClipPath = RQuartz_setClipPath;
+ dev->releaseClipPath = RQuartz_releaseClipPath;
+ dev->setMask = RQuartz_setMask;
+ dev->releaseMask = RQuartz_releaseMask;
@@ -1282,6 +1295,24 @@
+static int RQuartz_setPattern(SEXP pattern, pDevDesc dd) {
+ return -1;
+}
+
+static void RQuartz_releasePattern(int index, pDevDesc dd) {}
+
+static SEXP RQuartz_setClipPath(SEXP path, SEXP ref, pDevDesc dd) {
+ return R_NilValue;
+}
+
+static void RQuartz_releaseClipPath(SEXP ref, pDevDesc dd) {}
+
+static SEXP RQuartz_setMask(SEXP path, SEXP ref, pDevDesc dd) {
+ return R_NilValue;
+}
+
+static void RQuartz_releaseMask(SEXP ref, pDevDesc dd) {}
The plan is to implement at least one caching device (Cairo, plus maybe PDF), a non-caching device (always generates new definition on every use), and a non-implementing device (does not support features). These then provide templates for external device maintainers.
In an unusual move (for me), I have not completely ignored efficiency concerns.
Definitions are resolved every time a graphics system calls the graphics engine because the graphics engine is a "flat" coordinate system whereas the graphics systems (particularly 'grid' units) are dynamic and depend on the current drawing context.
This may present some challenges for implementing graphical definitions efficiently. However, the worst case scenario is something like filling thousands of data symbols with the same gradient fill.
Another issue is revisiting viewports (up/down). This could also generate a lot of redundant recalculations (or very large file sizes). Though not at the same scale as the thousands-of-shapes problem, it could be more common (?). This also can happen during calculation of things like "grobWidth" units.
The inefficiency could bite both in terms of speed and, for file-based devices, memory.
In both cases, allowing the graphics device to cache its definitions should be beneficial.
The plan is to actually do some measurements to demonstrate that there is a problem and that caching is helpful (or not).
Neither Cairo nor PDF devices have currently attempted to include text output in clipping paths or masks.
The R graphics engine does not accommodate the idea of a gradient fill within text.
The PDF device does not (yet) allow rasters within patterns or masks.
The Cairo device currently has a fixed limit on the number of definitions allowed (per page).
What to do about pattern fill on multiple shapes from single
grob:
grid.rect(width=1:2/3, gp=gpar(gradientFill=linearGradient()))
Currently just resolving pattern based on bounding box
of all shapes.
The current implementation of pattern fills in 'grid' "nicely" incorporates them into the gpar(fill=) interface. This may have implications for packages that build on 'grid' if they try to manipulate 'grid' gpars in any way, other than just setting them (e.g., 'gridSVG' converts gpar values to SVG display attributes).
These changes introduce some "modal" behaviour (like whether the device is "appending" or "drawing"); how to make sure we elegantly unwind when an error occurs (e.g., when hit an error while playing/recording a clipping path) ? One defence being employed for Cairo and PDF devices is to only produce warnings rather than errors when things go wrong (in the case of PDF, ask the use politely to shut down the PDF device). Also, modes are reset on new page, so it may be possible to recover a corrupted device by starting a new page.
What will the impact be on saved/restored plots ? (the graphics engine display list and snapshots thereof) I have some test cases working within-session. It would certainly be a bad idea to save from R 4 and replay in R 3 (though the reverse should be ok?).
The fact that clipping paths (mostly) and masks can be arbitrary 'grid' grobs places a reasonable burden on the graphics device. The device has to evaluate an R function and capture the resulting output. A possible future path for devices that cannot meet these demands may be to allow alternate objects as clipping paths and masks. Currently, the SEXP sent to the device is an R function, but that could conceivably be limited to simpler R objects, like a simpler description of one or more shapes. That might make the device's job a LOT easier.
PDF viewers: can get some variation between viewers (e.g., evince, xpdf, gs/gv, and firefox). Can also get (bigger) variation between versions of viewers. This bites harder for more complex drawings. For example, nested tiling patterns are NOT good in gv 3.7.4, xpdf 3.04, and evince 3.18.2 (Ubuntu 16.04), but are fine in gv ??, xpdf ??, and evince ?? (Ubuntu 18.04).
Reasons for keeping 'gridSVG': still the only way to support some stuff (e.g., Cairo does not support filters); still the only way to retain structure and labelling when export.
The impact on 'gridSVG' may be interesting. There may be differences
between the way that features are implemented in 'grid' vs 'gridSVG'.
For example, gradient fills might be added to gpar
in 'grid', rather than creating a separate class of grob (as in 'gridSVG').
This means that 'gridSVG' may need updating to work with
new 'grid' gradients. (Yes!)
'grImport2' is another interesting case - it may still end up with
ext=gridSVG as well as being able to draw directly
to normal R graphics devices.
Proper font support is another issue entirely. Currently addressed by packages like 'showtext' and 'extrafont'. Maybe a future project.
It would be nice if the graphics device API allowed for people to write R code that talked directly to the graphics device without having to be controlled by (and filtered through) the graphics engine. This would allow, for example, people to access features on graphics devices that are still not supported by the graphics engine (sort of like what 'showtext' sneakily does now to hijack text rendering; 'tikzDevice' also does something like this for creating "tags" in tikZ output ?). This would be a little like DVI specials (?) TODO: look at what 'showtext' and 'tikzDevice' need to do and how they currently work around the graphics engine. Another example that has been mentioned is hyperlinks.
Probably also best as a separate, future project.
1 It is actually possible to get this particular result currently in R graphics. We can generate a path consisting of the border of the entire image with the border of New Zealand inside it (i.e., a rectangular path with a hole the shape of New Zealand) and fill that with opaque white. However, that sort of workaround is more awkward and often runs into difficulties, for example, if we want to add other output to the image, but not obscure it with the opaque white fill. In general, we can generate may different graphical results just with the existing R graphics features, but we may have to work very hard to get what we want. It could be said that all this proposal is aiming for is greater convenience, but in some cases it is MUCH greater convenience.
This document
by Paul
Murrell is licensed under a Creative
Commons Attribution 4.0 International License.