Invertible Reproducible Documents

A major obstacle to the goal of inverting the final document is the difficulty of converting a final document format back into the source document format. For example, if the source document is a LaTeX-based reproducible document (as in the figure A source document), powerful tools exist to convert that document to a PDF document, but the inverse, from a PDF document to a LaTeX document, is very much harder to achieve (especially if we want to get back to a LaTeX document that is identical to the original source document, apart from explicit changes made by the client to the PDF document).

The solution chosen in this experiment is to focus on HTML-based reproducible documents. The reason for this is because an HTML-based source document gets processed to an HTML final document; the final document format is the same as the source document format. This improves our chances of being able to convert from final document to source document because at least the format of the two documents is the same. Because HTML is an XML-like markup language, we will also be able to use the powerful XML tool set (e.g., XPath) to manipulate the documents.

The figure An HTML-based source document shows a concrete, though extremely simplified, example of the sort of source documents we will work with and the figure An HTML final document shows the final document that is produced by processing that source document. The important point is that much of the content of the final document is very similar to the content of the source document, which is what we need to have any hope of getting from the final document back to the source document.

In a standard workflow, the final document is often treated as a static document that the client can view, but cannot modify. In other words, the final document is usually a dead end (see the figure 'knitr' workflow).

The choice of HTML, rather than, say, PDF, as the final document format does not initially improve this situation because an HTML-only web page is designed just for viewing in a web browser, not for editing. Although, technically, it would be possible for the client to modify the raw HTML, that is not something that clients are typically asked or expected to do.

The solution chosen in this experiment is to add the ability to modify the HTML final document by adding javascript code to the final HTML document, specifically, the CKEditor javascript library , which provides a relatively familiar word-processing GUI style of editing for the client.

The figure An editable HTML final document provides a simple demonstration; the paragraph of normal text on that page can be edited by clicking on it.

The processing of a source document to a final document can lose information from the source document. For example, if we compare the figure An HTML-based source document with the figure An HTML final document we see that the code chunk in the source document ...

... does not appear at all in the final document.

The solution chosen in this experiment is to introduce a pre-processing step that makes a copy of all code chunks in such a way that those code chunk copies are retained in the final document. The figure A pre-processed HTML-based source document shows how this works for the simple example in the figure An HTML-based source document.

The figure A knitted pre-processed HTML-based source document shows the result of processing that pre-processed source document with 'knitr', to show that the code chunk copies are retained in the processed document.

The processing of a source document to a final document can also add information. For example, if we compare the figure An HTML-based source document with the figure An HTML final document we see that the output from processing the code chunk in the source document ...

... is new content in the final document; that content was not part of the source document.

This gain of information during processing from source document to final document creates two problems, one simple and one hard. The simple problem is that conversion from final document back to source document will have to remove all of this gained information. That can be handled as just a text-processing task (as long as we can easily identify which content has been added).

The hard problem arises because the client can see some of the gained information. And if the client can see it, the client may want to modify it. And if the client modifies content that is not contained in the source document, how do we convert that change into a change to the source document?

The solution chosen in this experiment is to disallow editing of content that has been added during processing from source document to final document. Instead, the client is only allowed to annotate that content.

Annotation is enabled by adding javascript code to the final HTML document, specifically, the Annotator javascript library , which allows annotations to be added to selected regions of an HTML document.

The figure An annotatable HTML final document provides a simple demonstration; the date text (the output from the code chunk in the source document) can be annotated by selecting any part of the date text with the mouse.

Most of the steps involved in the inverted workflow (other than the 'knitr' step) could be characterised as "text processing" - searching for patterns within a document and then modifying, replacing, or removing the matching sections. For example, the sew() step involves searching for code chunks using the patterns <!--begin.rcode and end.rcode-->, making a copy and, in the copy, changing rcode to keepcode.

Because the documents that we are working with are HTML-based, another way to view the steps in the inverted workflow is as manipulations of XML content. For example, when we want to find all top-level elements within the body of the document, it is simpler to use an XPath expression like //body/* rather than attempt to search the document for opening and closing tags using text processing.

Both of these approaches are employed within the solution, with XML tools used to help locate important places within the document (using functions from the 'XML' package for R, ; a combination of getNodeSet() to identify fragments of the XML using XPath expressions and getLineNumber() to map a location within the XML to a location in the text document), but with text processing employed to actually make most changes.

The reason for this mixture of approaches is because text processing has a lower impact on the document. The round trip from text document, to XML objects within R, and back to text document, introduces side-effects (both white space and special entities) which interfere with our desire to be able to convert between source and final document and back again.

The R functions in the inverted workflow work quite hard to make sure that the only differences between the original source document and an inverted final document are differences that have been deliberately introduced by editing or annotating the final document. In particular, the functions work hard to avoid any spurious changes in white space, including changes in indenting and changes in where line breaks occur.

The reason for this emphasis is that it makes it possible to use simple, standard, text processing tools, e.g., 'diff', to easily identify changes. This has major flow-on benefits for version control because tools like 'git' and 'svn' can then reliably track the changes. Complex situations, for example when the consultant makes changes to the source document at the same time as the client is making changes to the final document, which require a merge of different versions of the document, can also be handled with existing version control tools.

The figure A 'diff' of the round trip shows output from the 'diff' program comparing the original source document, from the figure An HTML-based source document, with the inverted final document, from the figure An inverted final document, to show that the only changes that have been made to the original source document are either direct modifications using CKEditor or new annotations.

A summary of the steps involved in the inverted workflow, along with example code, are shown below:

• Create a source document, e.g., source.Rhtml.
• Edit the source document until it is ready to send to the client.
• Pre-process the source document ...

            sew("source.Rhtml")
          

• Process the source document with 'knitr' to create the final document ...

            knit("source.post.Rhtml")
          

• Post-process the final document to enable editing ...

            snap("source.post.html")
          

• Upload the editable final document for the client to edit on the web server via http://stat220.stat.auckland.ac.nz/cke/upload.html
• Client edits and annotates the final document via the web browser and clicks the "save" button to create files test-annotations.txt and test-changes.txt on the web server.
• Merge changes and annotations into the final document ...

            annotations("source.edit.html")
            changes("source.anns.html")
          

• Invert the final document to produce the new source document ...

            rip("source.save.html")
          

The main limitations arise from the fact that the R functions described in this experiment are tightly coupled with the software tools chosen to solve the various issues: 'knitr', CKEditor, and Annotator. For example, the sew() step looks for code chunks based on the markup used by the 'knitr' package for HTML-based reproducible documents.

One consequence of this tight coupling is that the functions are only of use for this particular combination of tools; the functions are not very general. Another consequence is that the functions are vulnerable to changes in the underlying tools. For example, if the 'knitr' package changed its markup for code chunks, the functions would fail. So the functions are also fragile.

The choice of HTML as the source document format also imposes some limitations. For example, some components of a report, such as tables and equations, may be harder to produce in HTML compared to using LaTeX. On the other hand, HTML provides some possibilities, such as interactivity, beyond what LaTeX can offer.

This article has focused on the 'knitr' package as a tool for reproducible documents, but many other tools exist for this purpose. This section considers the advantages and disadvantages that other tools might offer as a basis for an inverted workflow.

Beyond 'knitr'

Within the R ecosystem, the 'Sweave' package , which predates 'knitr', can also be used, with the help of the 'R2HTML' package , as the basis for an HTML-based workflow. Assuming that the HTML produced by 'R2HTML' is as regular and predictable as the HTML produced by 'knitr', it should not be difficult to modify the functions to work with 'Sweave' instead of 'knitr'.

Some other packages provide alternative document formats, e.g., the 'odfWeave' package allows a workflow based on Open Office documents. The idea of generalising the inverted workflow to other document formats will be discussed in more depth later, but one package deserves attention here: the 'SWord' package. The interesting aspect of this package is that the source document and final document are not only both Word documents, they are the same document. The figure The SWord workflow shows a conceptual diagram of the workflow when source and final document are the same thing.

This approach completely avoids the issue that the inverted workflow is trying to solve because there is no document transformation to invert, plus the document is not only editable, but it is a format that clients will be comfortable editing. However, one downside to this solution is that it forces the consultant to work in Word and that will not be ideal if the consultant prefers to work with simpler, text-based document formats in order to benefit from being able to work programmatically with the source document and to obtain full benefit from version control tools such as svn and git.

Another related tool that needs mentioning is the 'patchDVI' package . This is relevant to a LaTeX-to-PDF reproducible workflow and allows a client who is viewing a PDF final document to link a location in the PDF document back to the corresponding location in a LaTeX-based source document. This tool is more aimed at enabling the source document author (the consultant rather than the client) to map PDF locations back to the LaTeX source, so it has a different aim compared to the inverted workflow.

Beyond R

The discussion of an inverted workflow has been centred around reproducible tools in the R environment, but reproducible documents and shared documents exist in many other forms as well. Two notable examples are IPython notebooks and the R Extension for MediaWiki. Both of these are similar to 'SWord' in that they work with a single document that both consultant and client can edit, so there is no transformation to invert, but the consultant and client both have to work in the same environment, which is likely to make one or the other unhappy.

The inverted workflow described in this article is limited to an HTML-based source document that is processed by 'knitr' to an HTML final document, and that is made editable via CKEditor and annotatable via Annotator. This section considers how many of these constraints could be relaxed in future work.

Beyond CKEditor and Annotator

While it would be a simple matter to include a different javascript library than CKEditor, one important feature of CKEditor that might be difficult to replicate is the ability to enable editing via CKEditor in only a subset of the elements of an HTML document (and to save changes per element).

The feature of the Annotator library that was most useful was the fact that it provides quite detailed information, in the form of both XPath expressions and character offsets, about exactly which part of the document was annotated. It was also useful that the Annotator library also allowed annotation on a subset of the elements of an HTML document.

Another avenue for future experimentation would be to explore the lifecycle of annotations in a more sophisticated fashion. Having received information from the client in the form of an annotation, it might be desirable for the consultant to round-trip the annotation back to the client with information about whether the annotation has been resolved. In other words, the inversion from final document back to source document in the inverted workflow is currently too one-directional and does not consider the possibility of recycling information from the client, in the form of a modified source document, back to the client in a new final document.

Beyond HTML

The choice of HTML as the document format for the inverted workflow proved successful, but it is an open question whether that success could be replicated with other document formats. It appears unlikely that an inversion of the common LaTeX source document to PDF final document transformation would be possible, but there may be some hope with an Rmarkdown source document to HTML final document workflow. Future work is planned to explore this further.

Beyond consultant/client mode

This experiment was motivated by the scenario where a consultant produces a report for a client, but reproducible documents are also being used in several other important contexts, including scientific reproducibility and literate code development.

It is less likely that an inverted workflow would be of use in literate code development because both the producer and consumer of source and final documents are developers with technical skills to work on the source document (the final document can safely be read-only because everyone is capable of working directly on the source document).

However, An inverted workflow may have some relevance in the context of scientific reproducibility. In this case, the technical skills of the producer and consumer of source and final documents can vary widely and there is potential value in a consumer being able to transmit comments to the producer via changes to the final document. This applies not only to the relationship between one researcher as author and another researcher as reader, but also to the peer-review process where the reader is a reviewer.

Reusable documents versus reproducible documents

The standard reproducible document workflow involves the consultant working on a source document and then processing that document to produce a read-only final document for the client.

The inverted workflow described in this report allows the client to modify, as well as view, the final document and allows the final document to be processed back into a source document.

A further generalisation of this idea of allowing invertible documents, is the idea of allowing reusable documents. The point of making a document reusable is that we might want to perform other processing tasks with the final document. For example, we might want to extract all source code from the final document (a process sometimes described as "tangling"). More generally, some third party might want to process the final (or source) document in some way that the original author has not predicted.

There are several reasons why the workflow described in this report allows the final document to be inverted: information from the source document is never thrown away (e.g., all source code is kept even when it is not visible in the final document); processing of the source document to the final document happens in a predictable way, with structure and labelling (we know or we can identify what additional content gets added during processing); and the format of the final document is a markup format (HTML).

These features - retention of content, predictable processing output, and use of markup - are desirable features for any workflow where a document author wishes to produce a final document that can not only be used for a specific, predetermined purpose, but can also be reused in ways that the author might not have anticipated.