Use bookdown 2 23

README.md

@@ -1,41 +1,92 @@
 # Repository `docs`
 
-Documentation for Stan language and platform
+Repository for the sources and published documentation set, versioned for each Stan minor release.
 
-## GitHub Pages
+* The Stan User's Guide - example models and techniques for coding statistical models in Stan and using them to do inference and prediction.
+* The Stan Reference Manual - specification for Stan language and core inference algorithms.
+* The Stan Functions Reference - functions and distributions built into the Stan language.
+* The CmdStan Guide - guide to the reference command-line interface.
 
-This repository uses
-[GitHub Pages](https://help.github.com/categories/github-pages-basics)
-to serve the
-[project pages](https://help.github.com/articles/user-organization-and-project-pages/#project-pages-sites) site
-with URL `https://mc-stan.org/docs`.
-The publishing strategy is to serve the contents of the directory `docs` on branch `master`.
-The `docs` directory contains an empty file named `.nojekyll` so that GitHub will treat the contents
-as pre-generated HTML instead of trying to run [jekyll](https://jekyllrb.com).
 
-## Directory Structure
+## Repository directory structure
 
-* `src` : directory of source files for Stan and CmdStan guides and reference manuals, each in its own named subdirectory.
+* `src` : directory of source files for Stan and CmdStan guides and reference manuals, each in its own named subdirectory:
+    + `src/cmdstan-guide` - CmdStan Guide
+    + `src/functions-reference` - Stan Functions Reference
+	+ `src/reference-manual` - Stan Reference Manual
+	+ `src/stan-users-guide` - Stan Users Guide
 
 * `docs`: the directory `docs` on branch `master` is the [publishing source](https://help.github.com/articles/configuring-a-publishing-source-for-github-pages/) for the project pages site.  Whenever a verified member of the Stan organization pushes to `docs` on branch `master`,
 GitHub (re)builds and (re)deploys the website.
-
-* `build.py`: python script which compiles markdown files under `src` to html and pdf and populates the `docs` dir with the generated documentation.
 
-  + arg 1: MAJOR Stan version, required (expecting number, should be positive int)
-  + arg 2: MINOR Stan version, required (expecting number, should be positive int)
-  + arg 3: name of document (optional - will build entire docset)
-  + arg 4: output format, either "html" or "pdf" (optional - default builds both html and pdfs)
-
-* `LICENSE`: licensing terms.
+## Documentation toolset
+
+The documentation source files are written in [Rmarkdown](https://rmarkdown.rstudio.com)
+and the RStudio's [bookdown package](https://github.com/rstudio/bookdown) converts these to HTML and pdf.
+**Required bookdown version:** 2.23 or higher, cf [the bookdown changelog](https://github.com/rstudio/bookdown/blob/main/NEWS.md#changes-in-bookdown-version-023).
+Reported via [issue #380](https://github.com/stan-dev/docs/issues/380).
+
+
+The conversion engine is [Pandoc](https://pandoc.org).  It is bundled with RStudio.
+To use the build scripts to build the docset,
+you might need to [install Pandoc](https://pandoc.org/installing.html) separately.
+
+To build the pdf version of the docs, you will need to [install LaTeX](https://www.latex-project.org/get/) as well.
+The Stan documentation uses the [Lucida fonts](https://www.pctex.com/Lucida_Fonts.html),
+which must be [installed manually](https://tex.stackexchange.com/questions/88423/manual-font-installation).
+
+
+## Scripts to build and maintain the docset
+
+**`build.py`**
+
+The program `build.py` convert the markdown files under `src` to html and pdf and populates the `docs` dir with the generated documentation.
+Requires Python 3.7 or higher, due to call to `subprocess.run`, kwarg `capture_output`.
+  + 2 required argments:  <Major> <minor> Stan version, expecting 2 positive integer arguments, e.g. `2 28`
+  + 2 optional arguments:  <document> <format>.  The document name corresponds to the name of the `src` subdirectory or `all`.  The output format is either `html` or `pdf`.
+
 
+**Build script examples**
 
-## Generating the static site
+* `python build.py 2 28` - creates directory `docs/2_28` as needed; populates it will all generated documentation.
+* `python build.py 2 28 functions-reference` - builds both HTML and pdf versions of the Stan functions reference, resulting documents are under `docs/2_28`
+* `python build.py 2 28 functions-reference pdf` - builds only the pdf version of the Stan functions reference,  resulting document is `docs/2_28/functions-reference_2_28.pdf`
+* `python build.py 2 28 all pdf` - builds all pdfs from the Stan documentation set, resulting pdfs are in `docs/2_28`.
+
 
-1. Install R packages:  bookdown, arm, kableExtra, rstan, bayesplot
+**Additional scripts**
 
-2. Install on OS and ensure on PATH: pandoc (version at 2.3 or later), pandoc-citeproc, pdflatex
+The release process generates a new documentation set and adds links and redirects across the docset.
 
-3. Run `python build.py <MAJOR> <MINOR> (<document> (<format>))`
+* `add_redirects.py` manages the redirects from unversioned links to the latest version.
+* `link_to_latest.py` adds the "latest version" link into a docset.
+
+The Stan Functions Reference contains HTML comments which describe the function signature for all functions.  The script `extract_function_sigs.py` is used to scrape these signatures into a plain text file.
+
+## Build a single docset in R:  `bookdown::render_book`
+
+To build a single documet, you must have R or RStudio installed.
+To build a document from the command line, first `cd` to the correct `src` dir,
+then use the `Rscript` utility.
+
+```
+# build html
+> Rscript -e "bookdown::render_book('index.Rmd', output_format='bookdown::gitbook')"
+
+# build pdf
+> Rscript -e "bookdown::render_book('index.Rmd', output_format='bookdown::pdf_book')"
+```
+
+The output will be written to subdirectory `_build`.
+
+## GitHub Pages
+
+This repository uses
+[GitHub Pages](https://help.github.com/categories/github-pages-basics)
+to serve the
+[project pages](https://help.github.com/articles/user-organization-and-project-pages/#project-pages-sites) site
+with URL `https://mc-stan.org/docs`.
+The publishing strategy is to serve the contents of the directory `docs` on branch `master`.
+The `docs` directory contains an empty file named `.nojekyll` so that GitHub will treat the contents
+as pre-generated HTML instead of trying to run [jekyll](https://jekyllrb.com).
 
-4. The generated html and/or pdfs will be in the versioned subdirectory of `docs`.

build.py

@@ -29,11 +29,18 @@ def pushd(new_dir):
     os.chdir(previous_dir)
 
 def shexec(command):
-    returncode = subprocess.call(command, shell=True)
+    returncode = subprocess.call(command, shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
     if returncode != 0:
         print('Command {} failed'.format(command))
         raise Exception(returncode)
 
+def check_bookdown_2_23():
+    command = ["Rscript", "-e", "utils::packageVersion(\"bookdown\") > 0.22"]
+    check = subprocess.run(command, capture_output=True, text=True)
+    if not 'TRUE' in check.stdout:
+        print("R Package bookdown must be version 2.23, see README for build tools requirements")
+        sys.exit(1)
+
 def safe_rm(fname):
     if os.path.exists(fname):
         os.remove(fname)
@@ -43,13 +50,16 @@ def make_docs(docspath, version, document, formats):
     srcpath = os.path.join(path, "src", document)
     with pushd(srcpath):
         if ("pdf" in formats):
+            print('render {} as pdf'.format(document))
             command = "Rscript -e \"bookdown::render_book(\'index.Rmd\', output_format=\'bookdown::pdf_book\')\""
             shexec(command)
             srcpdf = os.path.join(srcpath, "_book", "_main.pdf")
             pdfname = ''.join([document,'-',version,".pdf"])
             pdfpath = os.path.join(docspath, pdfname)
             shutil.move(srcpdf, pdfpath)
+            print('output file: {}'.format(pdfpath))
         if ("html" in formats):
+            print('render {} as html'.format(document))
             command = "Rscript -e \"bookdown::render_book(\'index.Rmd\', output_format=\'bookdown::gitbook\')\""
             shexec(command)
             [safe_rm(f) for f in ("stan-manual.css", "_main.rds")]
@@ -58,11 +68,16 @@ def make_docs(docspath, version, document, formats):
             shutil.rmtree(htmlpath, ignore_errors=True)
             command = ' '.join(["mv _book", htmlpath])
             shexec(command)
+            print('output dir: {}'.format(htmlpath))
         else:
             [safe_rm(f) for f in ("stan-manual.css", "_main.rds")]
             shutil.rmtree("_book", ignore_errors=True)
 
 def main():
+    if sys.version_info < (3, 7):
+        print('requires Python 3.7 or higher, found {}'.format(sys.version))
+        sys.exit(1)
+    check_bookdown_2_23()
     global all_docs
     global all_formats
     if (len(sys.argv) > 2):

src/cmdstan-guide/_output.yml

@@ -3,7 +3,8 @@ bookdown::gitbook:
   css: stan-manual.css
   split_by: "chapter"
   config:
-    search: TRUE
+    search:
+        engine: lunr
     sharing: null
     download: null
     edit: null

src/functions-reference/_output.yml

@@ -2,7 +2,8 @@ bookdown::gitbook:
   css: ../stan-manual.css
   split_by: "section"
   config:
-    search: TRUE
+    search:
+        engine: lunr
     sharing: null
     download: null
     edit: null
@@ -18,10 +19,10 @@ bookdown::gitbook:
       before: |
         <li style="font-size:110%; font-weight:400; font-family: Verdana, Helvetica, sans; line-height:1.4; margin: 0.5em 0 0 1em"><a href="https://mc-stan.org/docs/functions-reference/index.html" style="color:#4183C4">Stan Functions Reference</a></li>
       after: null
-
 subparagraph: yes
 bookdown::pdf_book:
   latex_engine: "pdflatex"
   includes:
     in_header: header.tex
     after_body: postamble.tex
+

src/reference-manual/_output.yml

@@ -2,7 +2,8 @@ bookdown::gitbook:
   css: ../stan-manual.css
   split_by: "section"
   config:
-    search: TRUE
+    search:
+        engine: lunr
     sharing: null
     download: null
     edit: null

src/stan-users-guide/_output.yml

@@ -3,7 +3,8 @@ bookdown::gitbook:
   css: stan-manual.css
   split_by: "section"
   config:
-    search: TRUE
+    search:
+        engine: lunr
     sharing: null
     download: null
     edit: null

src/stan-users-guide/before-chapter.R

@@ -1,11 +1,9 @@
 # LIBRARY CONFIGURATION
 
-library(arm)
-library(bayesplot)
-  # theme_set(bayesplot::theme_default())
 library(dplyr)
 library(ggplot2)
 library(kableExtra)
+
 library(knitr)
   knitr::opts_chunk$set(
     cache = TRUE,
@@ -24,31 +22,9 @@ library(knitr)
     tidy = FALSE,
     warning = FALSE
   )
-library(lubridate)
-library(rstan)
-  rstan_options(auto_write = FALSE)
-  options(mc.cores = 1)
-
-
-# UTILITY FUNCTIONS
-
-printf <- function(pattern, ...) {
-  cat(sprintf(pattern, ...))
-}
-
-print_file <- function(file) {
-  cat(paste(readLines(file), "\n", sep=""), sep="")
-}
-
-extract_one_draw <- function(stanfit, chain = 1, iter = 1) {
-  x <- get_inits(stanfit, iter = iter)
-  x[[chain]]
-}
 
 # GENERAL R CONFIGURATION
 
 options(digits = 2)
 options(htmltools.dir.version = FALSE)
 
-
-set.seed(1123)

src/stan-users-guide/latent-discrete.Rmd

@@ -60,27 +60,27 @@ c \; ? \; x_1 \: : \: x_2
 \ x_2 & \quad\text{if } c \text{ is false (i.e., zero).}
 \end{cases}
 $$
-<!--As of version 2.10, Stan supports the conditional operator.-->
 
 
 ### Marginalizing out the discrete parameter {-}
 
 To code this model in Stan, the discrete parameter $s$ must be
 marginalized out to produce a model defining the log of the
-probability function $p(e,l,D_t)$.  The full joint probability factors
-as
+probability function $p(e,l,D_t)$.
+The full joint probability factors as
 \begin{align*}
 p(e,l,s,D) &=  p(e) \, p(l) \, p(s) \, p(D \mid s, e, l) \\
- &= \textsf{exponential}(e \mid r_e) \ \textsf{exponential}(l \mid r_l) \
+ &= \textsf{exponential}(e \mid r_e) \ \textsf{exponential}(l \mid r_l) \,
     \textsf{uniform}(s \mid 1, T) \\
- &\qquad \prod_{t=1}^T \textsf{Poisson}(D_t \mid t < s \; ? \; e \: : \: l),
+ & \qquad \prod_{t=1}^T \textsf{Poisson}(D_t \mid t < s \; ? \; e \: : \: l).
 \end{align*}
 
-To marginalize, an alternative factorization into prior and likelihood
-is used,
+
+To marginalize, an alternative factorization into prior and likelihood is used,
 $$
 p(e,l,D) = p(e,l) \, p(D \mid e,l),
 $$
+
 where the likelihood is defined by marginalizing $s$ as
 \begin{align*}
 p(D \mid e,l) &= \sum_{s=1}^T p(s, D \mid e,l) \\