Changes to R Base Graphics

Paul Murrell
Department of Statistics
The University of Auckland
paul@stat.auckland.ac.nz

This is a high-level description of the changes made to R base graphics between R version 1.3.1 and R version 1.4.0

The basic idea of these changes is to separate the graphics into three components: graphics devices, graphics engine, and graphics systems. R's main graphics are an example of a graphics system -- I refer to these as "base graphics".

This separation means that R's base graphics do not have any particularly special status (apart from the fact that they get loaded by default). Other graphics systems can be written and can get equal access to R's graphics engine and graphics devices (prior to these changes, other graphics systems had to go via R's base graphics, which was not convenient for some things. For example, controlling the clipping region on devices was very awkward.)

The interface to the graphics device component is given in GraphicsDevice.h. This describes the data and functions that every device must store -- the graphics engine assumes this information about every device. Graphics calls to the device must specify all locations and dimensions in device units.

The interface to the graphics engine component is given in GraphicsEngine.h. This describes the functions that the graphics engine can perform and is used by graphics systems to produce graphical output. Calls to the graphics engine must use device coordinates -- the graphics engine provides some helper functions for converting between device coordinates, normalised device coordinates, and inches.

The interface to R's base graphics is still given by Rgraphics.h until further changes are made (see below).

Both the graphics engine and the graphics devices must contain no information about a particular graphics system. Graphics systems must register with the graphics engine using GEregisterSystem. The system must provide a callback function when it registers -- the graphics engine will use this callback to signal the graphics system when important events occur (e.g., when a new device has been created). GEregisterSystem returns an integer that the graphics system must record -- the graphics engine records system-specific information with every device amd this integer is used within the callback function to get system-specific information back out of a device.

Calls to the graphics engine (and devices) must specify any relevant graphical parameters. For example, a call to GELine must specify a line colour, a line type, and a line width. (It used to be the case that the graphics engine and even graphics devices would look back up to the graphics system to see what the current settings were for graphcal parameters. This reflected the assumption that there would only ever be one graphics system to look back up to.) This reduces the allowable set of graphics parameters to those understood by the graphics engine; these are: col (for lines and borders), fill (for filling rects, polygons, ...), lwd (line width), lty (line type), font (font face), ps (point size of text), cex (character scaling factor). Other par() parameters are restricted to base graphics (as they should be).

The creation, destruction, copying, ... of devices is handled by the graphics engine.

There are three types of information stored about each R device: device-level information (including device-specific information) in the NewDevDesc structure; graphics engine information in the GEDevDesc structure; and graphics system-specific information in the GESystemDesc structure. The graphics engine is the master both in relation to the devices and in relation to graphics systems. The GEDevDesc structure (per device) contains information about the device in a NewDevDesc and information about every registered graphics system (e.g., current graphics system state) in an array of GESystemDesc's.

Registering a new graphics system

Special care needs to be taken in the code for the GE_InitState event. This code can rely on the system knowing its register index, but the dd->gesd[systemRegisterIndex] structure is empty. It is the callback code's responsibility to place a system state object within the dd->gesd[systemRegisterIndex] structure before trying to call code that will access that structure.

Graphics Structures

The R graphics engine is always loaded. (The R base graphics system is currently also always loaded, but this may not be the case forever.) During an R session, graphics devices may be created and deleted, and graphics systems may be loaded and unloaded. This section documents what happens during the creation/deletion of devices and the loading/unloading of systems, paying particular attention to the allocation and freeing of structures that hold the descriptions of the graphics devices and the graphics systems. First of all, a brief description of the structures involved:

For each graphics device, there is a GEDevDesc. (stored in the global R_Devices array). This structure represent the graphics engine's record of the device. There is information about the device itself, in a NewDevDesc structure, and graphics-state information for the device for each registered graphics system in an array of GESystemDesc structures.
```
    GEDevDesc
        NewDevDesc *dev
	GESystemDesc gesd[]
```
1. The NewDevDesc structure contains basic generic information about the device, PLUS device-specific information in a device-specific structure, PLUS pointers to device-specific implementations of the required set of graphical functions.
```
        NewDevDesc
	    < generic info >
	    < graphics function pointers >
	    void *deviceSpecific
  
```
2. The GESystemDesc structure contains system-specific information in a system-specific structure, PLUS a callback function pointer.
```
        GESystemDesc
	    void *systemSpecific
	    GEcallback callback
  
```
For each graphics system, there is a GESystemDesc structure (stored in the global registeredDevices array). This is just used to retain a record of which systems are registered and the respective callback hooks.

Now lets track the creation and free'ing of these structures, and which of the graphics engine, the graphics device, and the graphics system is responsible for these, for the cases of creating/deleting a device and loading/unloading a graphics system. The text is colour-coded by what sort of structure is being dealt with; there should be corresponding lines in the creation/deletion sections and in the loading/unloading sections (to ensure that all structures that are created are also freed). Also, ideally, the code responsible for the actions (engine, device, or system) should be the same in corresponding lines of the same colour. Finally, there should be a good reason if the order of creation is not reversed when structures are free'ed.

Creating a device

    device creates NewDevDesc
    device sets basic information and graphics function pointers
    device creates deviceSpecific
    device puts deviceSpecific in NewDevDesc
    engine creates GEDevDesc
    engine puts NewDevDesc in GEDevDesc 
    engine creates GESystemDesc for each registered system
    each system creates systemSpecific
    system puts systemSpecific in GESystemDesc
    engine puts GESystemDescs in GEDevDesc

Deleting a device

    device frees deviceSpecific
    each system frees systemSpecific
    engine frees GESystemDesc for each registered system    
    engine frees NewDevDesc
    engine frees GEDevDesc

Loading a graphics system

    engine creates GESystemDesc for each existing device
    system creates systemSpecific for each GESystemDesc
    system puts systemSpecific in GESystemDesc
    engine puts GESystemDescs in GEDevDesc
    engine creates GESystemDesc for global array

Unloading a graphics system

    system frees systemSpecific for each existing device
    engine frees GESystemDesc for each existing device    
    engine frees GESystemDesc in global array

Writing a New Graphics System: Tricky bits/possible areas for redesign

These are some issues that have arisen in the implementation of grid, that would need to be considered when writing another graphics system.

The system callback function which the graphics engine uses to ping registered graphics systems whenever something "important" happens, can be a bit tricky to use -- especially when initialising the graphics system state for a new device. The graphics system has to make sure not to try to access this information in the device during initialisation of the information (because it is not stored in the device until the initialisation is complete).
It is possible for the graphics system to be detached when the graphics engine display list contains calls to the graphics system. The minimum protection against this is to use the R_registerRoutines function so that R will report that the graphics system calls are missing (rather than just seg faulting!).

Things that still need doing

The changes made so far have been "minimalist" in that there has not been a true separation of the base graphics from the graphics engine (because there is not enough time to do this at the moment). Some things that need getting to are:

recordPlot() and replayPlot() (R-level functions) are only base graphics functions at the moment -- they could be made graphics engine functions.
recordPlot() and replayPlot() are compatible with "snapshots" (used in Windows plot history), but only by good fortune at this stage; they need thinking about to ensure compatibility for any future graphics system, in particular saving such objects and loading them into a new session. An important consideration when making the change will be how to handle incompatibility with old snapshots (pre device changes), which is also only fortuitously achieved at this point.
As well as worrying about recording and replaying the display list across sessions, we need to worry about recording the display list with calls from a non-base package, then detaching the non-base package, and replaying the display list. One option is to get graphics systems to "sign" the calls in the display list -- when the graphics engine goes to play calls on the display list it would only play calls which are signed by graphics systems that are currently registered. This would mean that if you detached a graphics system then any output from that system on the display list would not be redrawn (actually, at the moment when you detach a graphics system it turns off all graphics devices to be safe), and if you saved the display list and attempted to replay it after detaching the package or attempted to replay it in another session then any output from that system would not be drawn.
graphics.c still consists of a mixture of base graphics code, code to support the Graphics.h interface, and graphics engine code (e.g., display list code). This needs separating so that graphics engine code ends up in engine.c and base graphics code ends up in base.c and only Graphics.h interface support is left in graphics.c. Here's a list of the various code blocks in graphics.c and where they should end up:

Conversion code

(GConvert, etc, ...) This stuff is all base-related; all of the coordinate systems are base coordinate systems. The engine provides a separate set of simple device-inches-NDC transforms. This stuff should all end up in base.c

Plot layout code

This code sets up all of the base regions (includes layout code). Should all end up in base.c

Colour and Line Type code

This is engine-related stuff; should end up in engine.c

Device code

This still has some base code tangled up in what should be engine-only stuff. Needs detangling and shifting to engine.c

Display List code

This still has some base code tangled up in what should be engine-only stuff. Needs detangling and shifting to engine.c
The base graphics code has been modified at the very lowest level to work with the new device driver API. It should be further modified so that it calls the graphics engine instead.
A particular example of this is the overlap between the base GPar structure and the NewDevDesc structure. The NewDevDesc structure respresents information about a device and this can be removed from the base GPar structure, which should contain only information about base graphics graphical parameters. This change will require modifications of the base graphics code (e.g., in graphics.c and plot.c) to access device information from a NewDevDesc rather than a GPar.
Another important issue here is that there currently exists two copies of some code, where there is a G* function and a GE* equivalent. Need to at least modify the G* functions so that they call the GE* equivalent. This has now been done (in development version post 1.8.0). Most G* functions in graphics.c (basically the ones that are to do with drawing things) are now simply wrappers that call the corresponding GE* function in engine.c. This allows allows code in places like plot.c to remain unchanged. Modifying plot.c to go directly to the GE* functions will require a bit of care. (NOTE: the graphicsQC package has proved invaluable for checking that changes to the C code do NOT produce changes in output and would be an important tool in further changes.) The current state of the code is represented in the following diagram (most of the high-level base C code goes through Graphics.h to graphics.c which goes through GraphicsEngine.h to engine.c):
All devices (that I know about) have been converted to the new device API so code that caters for the old device API can be removed. For example, the insides of Rf_gpptr et al; lots of stuff in graphics.c; all pointer coercions from/to DevDesc, NewDevDesc, and GEDevDesc!!