manual testing framework

We aren't testing code that goes into the manual thoroughly and automatically enough. We need a framework where code citations in the manual are automatically extracted from running code and then pasted into the manual.

Extend this to example I/O. The introduction is riddled with references to stan_major=1, etc.

I'll try to create a branch that uses the listings latex package.

@jrnold

It looks like there are two main packages that are being used now. There's the listings and minted. I tried with minted because it does some useful things.

With minted, I was able to change the existing code blocks to be processed by minted.
Pros:

full syntax highlighting; I think @jrnold was behind it!
will be easy to include from files and specify start lines and end lines
will be able to remove blank spaces up front easily
there are options for syntax highlighting. Colors for keywords, etc.

Cons:

relies on python and the Pygments package, which is straightforward to install
it's slow. The manual currently builds in ~20 secs. With the minted package, it builds in ~2:20. That's without including files, etc. If we use a simpler lexer (just text instead of Stan), it still builds in ~2:00.

Thoughts?

I'm going to try listings as a comparison.

If anyone wants to try minted, use this branch: feature/issue-319-manual-testing.

To change the style of how the code is rendered, change line 211 of stan-manuals.sty:
\usemintedstyle{friendly}

I tried with \usemintedstyle{bw} and it'll generate the Stan code like it is in the manual now.

What's the output look like?

You mean it builds end to end from clean-all in 2 mins, or that each run of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?

I'm not keen on dependencies, but then I added the fonts,
so I can hardly object if someone else thinks we need code
highlighting.

I really don't want too much highlighting---otherwise the code looks like an Xmas tree and I find it hard to read. Also, it might not print well if the colors aren't dark and high-contrast (not that I ever print anything). Also, the highlighting has to be correct. Inconsistent or incomplete highlighting is worse than none at all.

With the bw style, it looks like the existing manual.

I checked in a version that'll make it look like a Christmas tree. (I just
did that to demonstrate that we could.)

It builds end to end from clean-all in 2 minutes and change.

On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter notifications@github.com
wrote:

What's the output look like?

You mean it builds end to end from clean-all in 2 mins, or that each run
of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?

I'm not keen on dependencies, but then I added the fonts,
so I can hardly object if someone else thinks we need code
highlighting.

I really don't want too much highlighting---otherwise the code looks like
an Xmas tree and I find it hard to read. Also, it might not print well if
the colors aren't dark and high-contrast (not that I ever print anything).
Also, the highlighting has to be correct. Inconsistent or incomplete
highlighting is worse than none at all.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F
.

You committed a verison with \usemintedstyle{friendly}.

This is way too slow. It takes over 1s/page on my machine,
and it needs multiple pass, so I estimated over half an hour
and got bored waiting and gave up.

Maybe we could do this for release version, but for
work-in-progress this is going to be a huge drag on my
working on the manual.

I wrote this entire email between pages 110 and 297.

Bob

On Sep 2, 2016, at 6:54 PM, Daniel Lee notifications@github.com wrote:

If anyone wants to try minted, use this branch: feature/issue-319-manual-testing.

To change the style of how the code is rendered, change line 211 of stan-manuals.sty:
\usemintedstyle{friendly}

I tried with \usemintedstyle{bw} and it'll generate the Stan code like it is in the manual now.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

What are you running on? Did you have to do something
to juice Python?

Bob

On Sep 2, 2016, at 7:00 PM, Daniel Lee notifications@github.com wrote:

With the bw style, it looks like the existing manual.

I checked in a version that'll make it look like a Christmas tree. (I just
did that to demonstrate that we could.)

It builds end to end from clean-all in 2 minutes and change.

On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter notifications@github.com
wrote:

What's the output look like?

You mean it builds end to end from clean-all in 2 mins, or that each run
of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?

I'm not keen on dependencies, but then I added the fonts,
so I can hardly object if someone else thinks we need code
highlighting.

I really don't want too much highlighting---otherwise the code looks like
an Xmas tree and I find it hard to read. Also, it might not print well if
the colors aren't dark and high-contrast (not that I ever print anything).
Also, the highlighting has to be correct. Inconsistent or incomplete
highlighting is worse than none at all.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

No idea. I'm using python 2.7.10 and I just installed Pygments today.

On Sep 2, 2016, at 1:03 PM, Bob Carpenter notifications@github.com wrote:

What are you running on? Did you have to do something
to juice Python?

Bob

On Sep 2, 2016, at 7:00 PM, Daniel Lee notifications@github.com wrote:

With the bw style, it looks like the existing manual.

I checked in a version that'll make it look like a Christmas tree. (I just
did that to demonstrate that we could.)

It builds end to end from clean-all in 2 minutes and change.

On Fri, Sep 2, 2016 at 12:57 PM, Bob Carpenter notifications@github.com
wrote:

What's the output look like?

You mean it builds end to end from clean-all in 2 mins, or that each run
of pdflatex takes 2 mins, so the whole thing takes like 10 mins+?

I'm not keen on dependencies, but then I added the fonts,
so I can hardly object if someone else thinks we need code
highlighting.

I really don't want too much highlighting---otherwise the code looks like
an Xmas tree and I find it hard to read. Also, it might not print well if
the colors aren't dark and high-contrast (not that I ever print anything).
Also, the highlighting has to be correct. Inconsistent or incomplete
highlighting is worse than none at all.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/issues/319#issuecomment-244430457, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F7csHaPLa4JIQdUXC3meBEV7WZnlks5qmFV0gaJpZM4BIN0F
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

I really hate that output. Maybe if all the colors
were darker and less opposite each other and there was
no bold to mess up weight I might be able to learn to
accept it if everyone else really really wants highlighting.

Seems to have failed somehow at the end, but the real problem
is that it takes a long long time.

> make clean-all
> time make manual

...

make[1]: *** No rule to make target `stan-reference.w18'.  Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
  building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf;  cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt;

real    17m16.612s
user    15m28.673s
sys 1m39.578s

Running with make -j4 doesn't seem to be any faster.

Bob

I saw that error too.

Changing the style to bw helps, but yes, the time is a killer. I'll try the listings package out. Less dependencies is better.

On Sep 2, 2016, at 1:23 PM, Bob Carpenter notifications@github.com wrote:

I really hate that output. Maybe if all the colors
were darker and less opposite each other and there was
no bold to mess up weight I might be able to learn to
accept it if everyone else really really wants highlighting.

Seems to have failed somehow at the end, but the real problem
is that it takes a long long time.
> make clean-all
> time make manual

...

make[1]: *** No rule to make target `stan-reference.w18'. Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt;

real  17m16.612s
user  15m28.673s
sys   1m39.578s
Running with make -j4 doesn't seem to be any faster.

Bob
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

Why do you want to do this? Is it that people want syntax
highlighting? If it's not subtle (by which I mean you can
barely see it at arm's length), I'm going to object.

Bob

On Sep 2, 2016, at 7:30 PM, Daniel Lee notifications@github.com wrote:

I saw that error too.

Changing the style to bw helps, but yes, the time is a killer. I'll try the listings package out. Less dependencies is better.
On Sep 2, 2016, at 1:23 PM, Bob Carpenter notifications@github.com wrote:

I really hate that output. Maybe if all the colors
were darker and less opposite each other and there was
no bold to mess up weight I might be able to learn to
accept it if everyone else really really wants highlighting.

Seems to have failed somehow at the end, but the real problem
is that it takes a long long time.
> make clean-all
> time make manual

...

make[1]: *** No rule to make target `stan-reference.w18'. Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt doc/stan-functions-2.11.0.txt;

real    17m16.612s
user    15m28.673s
sys 1m39.578s
Running with make -j4 doesn't seem to be any faster.

Bob
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

I want to split out the Stan code chunks into valid Stan programs. We can
include bits of it in the manual. Then, we'd be able to test that the
chunks in the manual compile and we can generate output for the manual so
we can keep up to date with output.

On Fri, Sep 2, 2016 at 2:00 PM, Bob Carpenter notifications@github.com
wrote:

Why do you want to do this? Is it that people want syntax
highlighting? If it's not subtle (by which I mean you can
barely see it at arm's length), I'm going to object.

Bob
On Sep 2, 2016, at 7:30 PM, Daniel Lee notifications@github.com wrote:

I saw that error too.

Changing the style to bw helps, but yes, the time is a killer. I'll try
the listings package out. Less dependencies is better.
On Sep 2, 2016, at 1:23 PM, Bob Carpenter notifications@github.com
wrote:

I really hate that output. Maybe if all the colors
were darker and less opposite each other and there was
no bold to mess up weight I might be able to learn to
accept it if everyone else really really wants highlighting.

Seems to have failed somehow at the end, but the real problem
is that it takes a long long time.
> make clean-all
> time make manual

...

make[1]: *** No rule to make target `stan-reference.w18'. Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf
doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt
doc/stan-functions-2.11.0.txt;

real 17m16.612s
user 15m28.673s
sys 1m39.578s
Running with make -j4 doesn't seem to be any faster.

Bob
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/issues/319#issuecomment-244445990, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F7Y2oN8Su82oyWZeUKdq3ZUK2sViks5qmGQzgaJpZM4BIN0F
.

OK. I could live with runtime for that functionality.
But the style has to be beter.

How would we markup the files that get included?

Also, this seems like a very low priority compared to all
the bugfixes and critical refactoring issues we have to do.

Bob

On Sep 2, 2016, at 8:14 PM, Daniel Lee notifications@github.com wrote:

I want to split out the Stan code chunks into valid Stan programs. We can
include bits of it in the manual. Then, we'd be able to test that the
chunks in the manual compile and we can generate output for the manual so
we can keep up to date with output.

On Fri, Sep 2, 2016 at 2:00 PM, Bob Carpenter notifications@github.com
wrote:
Why do you want to do this? Is it that people want syntax
highlighting? If it's not subtle (by which I mean you can
barely see it at arm's length), I'm going to object.

Bob
On Sep 2, 2016, at 7:30 PM, Daniel Lee notifications@github.com wrote:

I saw that error too.

Changing the style to bw helps, but yes, the time is a killer. I'll try
the listings package out. Less dependencies is better.
On Sep 2, 2016, at 1:23 PM, Bob Carpenter notifications@github.com
wrote:

I really hate that output. Maybe if all the colors
were darker and less opposite each other and there was
no bold to mess up weight I might be able to learn to
accept it if everyone else really really wants highlighting.

Seems to have failed somehow at the end, but the real problem
is that it takes a long long time.
> make clean-all
> time make manual

...

make[1]: *** No rule to make target `stan-reference.w18'. Stop.
Latexmk: All targets (stan-reference.pdf) are up-to-date
building src/docs/stan-reference/stan-functions.txt
cp src/docs/stan-reference/stan-reference.pdf
doc/stan-reference-2.11.0.pdf; cp src/docs/stan-reference/stan-functions.txt
doc/stan-functions-2.11.0.txt;

real 17m16.612s
user 15m28.673s
sys 1m39.578s
Running with make -j4 doesn't seem to be any faster.

Bob
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/stan/issues/319#issuecomment-244445990, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F7Y2oN8Su82oyWZeUKdq3ZUK2sViks5qmGQzgaJpZM4BIN0F
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

Navigation Menu

Repository navigation

manual testing framework #20

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

manual testing framework #20

Description

Activity

betanalpha commented on Nov 21, 2013

syclik commented on Apr 4, 2015

syclik commented on Sep 2, 2016

syclik commented on Sep 2, 2016

bob-carpenter commented on Sep 2, 2016

syclik commented on Sep 2, 2016

bob-carpenter commented on Sep 2, 2016

bob-carpenter commented on Sep 2, 2016

syclik commented on Sep 2, 2016

bob-carpenter commented on Sep 2, 2016

syclik commented on Sep 2, 2016

bob-carpenter commented on Sep 2, 2016

syclik commented on Sep 2, 2016

bob-carpenter commented on Sep 2, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions