Known limitations of Markdown support

The Markdown support relies on a relatively recent addition to Pandoc, the sourcepos extension. At present (Pandoc 2.19.2 as included in RStudio), this extension only works with the Commonmark reader, whereas by default, Pandoc uses its own dialect of Markdown. To enable concordances, patchDVI switches the Pandoc input setting to commonmark_x+sourcepos.

Unfortunately, the commonmark_x reader doesn’t support most of the extensions that the standard markdown reader supports. This page: https://github.com/jgm/pandoc/wiki/Roadmap should list the current status. For the definitions of the extensions, see the Pandoc documentation at https://pandoc.org/MANUAL.html.

Of particular note are the lack of citations and raw_tex. Citations in Markdown source like [@Author] will not currently be processed in Commonmark, though support for this is planned at some future time. Raw LaTeX will need to be “fenced”, e.g. entered as

```{=latex}
LaTeX
```

(The \pagebreak and \newpage macros receive special treatment, so they don’t need fencing as long as they are separated by blank lines from text above and below.) The “roadmap” indicates that raw LaTeX will never be supported in Commonmark, as it is incompatible.

Technical description of concordance records

The concordance records are text records in the following format. There are four parts, separated by colons:

1. The label concordance to indicate the type of record.
2. The output .tex filename.
3. The input .Rnw filename.
4. The input line numbers corresponding to each output line.

The third component is compressed using a simple encoding: The first number is the first line number; the remainder of line numbers are a run-length encoding of the differences. Thus if the input file is as shown here:

the output file would be as shown below, with the concordance as shown there in the second column.

This concordance would be recorded in the file sample-concordance.tex as

\Sconcordance{concordance:sample.tex:sample.Rnw:%
1 1 1 1 2 7 0 1 2}

The numeric part of this file may be interpreted as shown below:

The same format for concordance records is used in R Markdown, but when the output format is HTML, the target file is skipped and it is wrapped in an HTML comment instead, e.g.

<!-- concordance::sample.Rnw:1 1 1 1 2 7 0 1 2 -->

Patching .dvi Files

The \Sconcordance macro expands to a \special macro when producing a .dvi file. This is included verbatim in the .dvi file. The concordance: prefix identifies it as a patchDVI concordance. The patchDVI() function scans the whole file until it finds this sort of record. (There may be more than one, if multiple files make up the document.) Source references are also recorded by LaTeX in \special records; their prefix is src:. The patchDVI() function reads each src: special and if it refers to a file in a concordance: special, makes the substitution. At the end, it rewrites the whole .dvi file.

Patching .synctex Files

When using pdflatex, the \Sconcordance macro expands to a \pdfobj macro containing the concordance, which eventually is embedded in the .pdf file. However, the Synctex scheme of source references does not write the references to the .pdf file directly. Instead, they are written to a separate file with extension .synctex, or a compressed version of that file, with extension .synctex.gz. The patchSynctex() function reads the concordances from either the .pdf file (when pdflatex was used) or the .dvi file, and the source references from the Synctex file. It rewrites only the Synctex file when it makes its changes.

Patching .html Files

If R Markdown has an HTML target, the concordances are embedded by Pandoc in span and div markup, e.g.

<span data-pos="sample.knit.md@13:1-13:5">This</span>

The driver reads the data-pos attributes, works out the corresponding line from the true source file, and rewrites the concordances using the concordance record in a comment as described at the end of Section Technical description of concordance records.

Currently I am only aware of one use for these values: the HTML Tidy checks done in R 4.3.x will report the original source lines in the .Rmd file. If I become aware of other uses for these records, it seems likely this format will need to change.

The Complete Process

The patchDVI package contains five functions for each of knitr, R Markdown and Sweave designed to run all four of the steps listed at the start of this section.

The functions are all named in the same style: Stage1Stage2, where Stage1 is one of knit, render, or Sweave, and Stage2 is one of DVI, Miktex, PDF, PDFMiktex, dvipdfm. The full list was given near the end of the introduction to Section Project Management Functions. The Stage1 parts specify the initial processor as being knitr::knit, Sweave, or rmarkdown::render. The Stage2 part specifies how the resulting .tex files are processed: knitDVI and knitMiktex and related functions produce .dvi files in the general case and for MikTeX respectively; knitPDF and knitPDFMiktex and friends do the same for direct .pdf output from pdflatex. Finally, knitDVIPDFM and related functions run the two-step conversion using first latex and then dvipdfm.

In each case, the LaTeX processing functions are customizable.

For example, a few years ago I had a text editor that allowed me to call external functions with arguments depending on the name of the current file and the line number within it. I had it call a Windows batch file with the line set as argument %1 and the filename set as argument %2; the batch file invoked R using the command line

echo patchDVI::SweaveMiktex('%2',
    preview='yap -1 -s"%1%2" "\x25s"')
    | Rterm --slave

(all on one long line). This passed the current file to SweaveMiktex(), and set the preview command to use the yap options -1 to update the current view (rather than opening a new window), and to jump to the line corresponding to the editor line. The code "\x25s" is simply "%s" encoded without an explicit percent sign, which would be misinterpreted by the Windows command processor. When patchDVI calls the previewer, the main .dvi filename will be substituted for %s.

Installing or Loading Packages

In a complex project, there are often a number of different packages required. When updating R, you may end up with a tedious exercise to make sure these are all installed and updated.

The needsPackages() function helps with this. It takes a character vector naming packages that will be used in the current document. By default, it installs any that are not already installed. Optionally, it can update them using update.packages(), load them, or attach them to the search list. For example, this document uses no packages other than patchDVI itself, so it could have

patchDVI::needsPackages("patchDVI")

near the start to ensure it is available, if this wasn’t a nonsensical statement. (Why would you be able to run it if patchDVI wasn’t already installed?)

Using patchDVI with TeXShop

TeXShop is a nice LaTeX editor on MacOS. Dave Gabrielson of the University of Manitoba helped me to work out these instructions. I have updated them in December, 2013 for TeXShop 2.47.

1. In Preferences -- Typesetting -- Sync Method, choose SyncTeX.
2. Instructions depend on which language you are using.
   1. To use with Sweave, create a file called Library/TeXShop/Engines/Sweave.engine in your home directory containing the lines below. Give it executable permissions. Replace <locale> with your locale string, e.g. en_CA.UTF-8 for Canadian English using UTF-8 encoding. The locale line can be omitted if you only use plain ASCII characters, but is probably necessary for other cases.

   #!/bin/bash
   export LC_ALL=<locale>
   Rscript -e "patchDVI::SweavePDF('$1')"

   2. To use with knitr, create a file called Library/TeXShop/Engines/knitr.engine in your home directory containing the lines below, and give it executable permissions. Replace <locale> with your locale string, e.g. en_CA.UTF-8 for Canadian English using UTF-8 encoding. The locale line can be omitted if you only use plain ASCII characters, but is probably necessary for other cases.

   #!/bin/bash
   export LC_ALL=<locale>
   Rscript -e "patchDVI::knitPDF('$1',\
       envir = globalenv())"

   3. To use with R Markdown, create a file called Library/TeXShop/Engines/markdown.engine in your home directory, and give it executable permissions. Replace <locale> with your locale string, e.g. en_CA.UTF-8 for Canadian English using UTF-8 encoding. The locale line can be omitted if you only use plain ASCII characters, but is probably necessary for other cases.

   #!/bin/bash
   export LC_ALL=<locale>
   Rscript -e "patchDVI::renderPDF('$1',\
     envir = globalenv())"

   4. For other vignette engines, use a weave argument in the above, as appropriate.
3. Install the patchDVI package into R.
4. When editing a .Rnw file in TeXShop, choose the knitr or Sweave engine from the menu; for .Rmd files, use the markdown engine.
5. If you have multiple files in your project, your main file must be a .Rnw file (e.g. Main.Rnw) which lists all .Rnw files in a .SweaveFiles variable, and you need to add the line

%!TEX root = Main.Rnw

to each subordinate file. 6. For Sweave, add the \SweaveOpts{concordance=TRUE} line to your document. For knitr, add a code chunk similar to this:

 <<results='asis'>>=
 patchDVI::useknitr()
 @

somewhere near the start of your document.

The TeXShop previewer supports SyncTeX; you right click in the preview, and choose Sync from the menu to jump to your source location.

Using patchDVI with TeXworks

TeXworks is an editor for multiple platforms, somewhat similar to TeXShop. These instructions have been tested in version 0.4.5, with MikTeX 2.9 on Windows, and version 0.6.8 on MacOS.

NB: Some versions of TeXworks had a bug in setting the HOME directory of the user. With those versions, R will not find a locally installed copy of patchDVI. To work around the bug, set the R_USER environment variable to your Windows home directory, e.g. R_USER=C:/Users/Murdoch.

TeXworks can work with the patchDVI project management features using a script to tell it to process the current file through knitr/Sweave, but preview the main file. See the instructions below for my current best attempt at such a script. It can also use the TeXShop approach of specifying the TEX root file to be a .Rnw file.

The instructions are given first for Sweave, then below for knitr.

1. Add a new SweavePDF command: In Edit | Preferences | Typesetting click on the + sign near the bottom. Set the name of the tool to be SweavePDF. Set the program to Rscript, and add two arguments, one per line:
   1. -e
   2. patchDVI::SweavePDF('$fullname')
2. Install the patchDVI package into R.
3. Tell TeXworks to open Sweave files by editing the file pattern configuration file texworks-config.txt. This file is found via the Help | Settings and Resources... | Resources menu entry. I have this line there:

file-open-filter:   TeX documents (*.Rnw *.Rmd *.tex)

4. When editing a .Rnw file in TeXworks, choose the SweavePDF engine from the menu.
5. Add the \SweaveOpts{concordance=TRUE} line to your document.
6. If you are using the project management features of patchDVI and are editing a subordinate file, TeXworks will not open or update the PDF preview after it processes changes. There are four workarounds for this.

The simplest is to manually open the .pdf file the first time. After that it will be updated automatically. Unfortunately, if you happen to be editing the main file, the .pdf will be opened automatically, and then updates won’t happen if you later edit a subordinate file.

The next simplest is the TeXShop approach: include a line

%!TEX root = Main.Rnw

near the top of the file, and make sure that Main.Rnw refers to all subordinate Sweave files.

To use TeXworks with knitr, the instructions are very similar to those above, but with two changes.

In step 1, replace the second line of the command (the SweavePDF() call) with the following longer command, all on one line:

patchDVI::knitPDF('$fullname', envir = globalenv())

In step 5, insert the following code chunk into your file:

 <<results='asis'>>=
 patchDVI::useknitr()
 @

To use TeXworks with R Markdown, make these changes:

In step 1, replace the second line of the command (the SweavePDF() call) with the following:

patchDVI::renderPDF('$fullname', envir = globalenv(), output_format = "patchDVI::pdf_documentC")

Using patchDVI with WinEdt

WinEdt is a Windows editor with LaTeX support. The configuration options have changed a number of times; I do not know how to implement these instructions in the latest version. These instructions apply to version 5.5, and assume you are using it with MikTeX.

1. In Options -- Execution Modes choose Texify, and click on Browse for Executable. Find the Rscript executable in your R installation, directory bin/i386 or bin/x64, and choose it. In the Switches line, put

-e

and in the Parameters line, put

"patchDVI::SweaveMiktex('%n%t', '%N.tex')"

The quotes are necessary! 2. Do the same for the PDF Texify command, replacing SweaveMiktex with SweavePDFMiktex. 3. In Options -- Execution modes, make sure Start Viewer and Forward Search are selected for LaTeX and PDF LaTeX.

When you preview a file in yap, double clicking should jump back to the editor. If it doesn’t (or it opens the wrong editor), while you’re in yap choose View -- Options -- Inverse DVI search. You should see WinEdt (auto-detected) as an option; if so, select it. If not, create a new entry for WinEdt, and for the command line, put in

"path\to\winedt.exe" "[Open(|%f|);SelPar(%l,8)]"

after editing the path as necessary.

Using patchDVI with RStudio

RStudio is a very nice front end for working in R and with individual .Rnw or Markdown files. If you are using it, I’m going to assume you’re using knitr as well, and these instructions have been worked out for that combination.

RStudio is less flexible than the other editors for specifying customized processing of a file, so these instructions were worked out assuming that you already have it configured for knitr. It is probably possible to do something similar for Sweave; I just haven’t tried.

You need to set up your individual chapter files as for TeXShop/TeXworks, i.e. with a

%!TEX root = Main.Rnw

comment at the top of each. This tells RStudio to run knitr on the main file when you click Compile PDF. (It will also work if you use the knitr style

%!RNW root = Main.Rnw

but then your files won’t work in TeXShop/TeXworks.)

In the main file, you need a code chunk containing a line to set the .SweaveFiles variable naming all chapter files (but not the main file), and then running knitInRStudio():

.SweaveFiles <- c("a.Rnw", "b.Rnw")
patchDVI::knitInRStudio()

It is safe to put these lines in your file even if you sometimes process it in a different way: if you are not in RStudio, knitInRStudio() does nothing.

One remaining issue with this approach is that you won’t see the knitr progress messages from knitting the chapter files. If you want to see those messages, add the chunk option childOutput = TRUE to the code chunk holding this code.