Convert to use jupyter_sphinx?

The recent update to the msprime documentation uses jupyter-sphinx to render live code chunks using jupyter. It seems to work pretty well --- see example here. We use the same framework to build and deploy the docs, which would ultimately live at https://tskit.dev/tutorials.

I propose we repurpose this repo to use jupyter-sphinx, such that we create a single sphinx site in which each tutorial consists of one rst file (much like the msprime docs is split into files). That is, we move away from using notebooks which have proved to be really tedious as a format for defining tutorials.

Advantages:

All the tutorials will be built during CI
We're using rst (or perhaps even markdown - should be possible) with chunks of jupyter interspersed and so reviewing and updating the tutorial content through git will be much simpler. Also we're using the same frameworks as the msprime docs, so everything should be immediately familiar to devs.
We can assign tags and labels to the tutorial content so that we link to the tutorial content downstream using intersphinx.

Perfect! sounds like a plan

This will be repo for all tskit-dev tutorials right? (i.e. tskit and msprime)?

I think so, yeah. I just doesn't make sense to split them up, IMO

This sounds great. Only caveat is that the forward simulation tutorial took 30 minutes or so to process on my laptop, iirc. It runs a bunch of replicates to compare to msprime.

Eek, that's a long time. I was hoping to make this something that gets built each time, but maybe that's not so practical.

I guess if we're going to demonstrate real stuff then it's going to take more than a few seconds to run the notebook sections. Which makes the intial plan of building every tutorial every time (e.g., during development) impractical.

I guess we could have a separate sphinx site for each tutorial then? There is a bit of boilerplate involved, but perhaps it's not too bad. That still doesn't allow for a 30 minute run time per tutorial, though.

I guess we just bin the current tutorial. The reader will just have to trust that it is correct, or copy/paste the code out and test for themselves.

Even 10min would prob be manageable. Would it be possible to get it down to that?

Maybe, but maybe that isn't the point? The tutorial should show how, but maybe not follow through all the way?

One thing we could do for this tutorial is to provide code as source files. The online tutorial could display, import, and run one example. But anyone could clone and run more for themselves?

Brainstorming here: we could wrap expensive calculations in a method that caches the result and recomputes only if the code has changed? (knitr has an option like this for r+markdown) I don't know how to do this in a way that's invisible in the html, though.

Or, maybe just doing this for expensive bits:

outfile = 'results.trees'
if os.path.isfile(outfile):
   ts = tskit.load(outfile)
else:
   ts = msprime.simulate(...)
   ts.dump(outfile)

is good practice, even. Then we either (a) check the output into the repository, or (b) it takes a while to run the first time someone builds the docs, but after that it's quick? We'd have to remember to rebuild the file if stuff changes, of course.

I think keeping the tutorials to being runnable in a few seconds (say, 10 max) is a reasonable constraint to work within. Do you really learn more as a reader by looking at big examples than you do from small ones?

Re the caching, we'll sort of get this for free if we keep each tutorial in a separate RST file. On the first build when you start a new tutorial, sphinx will build all the pages. But, each subsequent build will only make your tutorial, as there won't be any changes to the others.

It's probably good discipline to build all the tutorials each time, as then we'll find out when stuff breaks.

Do you really learn more as a reader by looking at big examples than you do from small ones?

I think that's a good question... I think both big examples and small ones have their place. It's nice to see how all the concepts come together.

build all the tutorials each time

each time what?

I'm a little confused. For the docs, when do the examples run? Is it every time a user loads the page or? If it's not every time a user loads the page, and just say when there is an update to the code base, then it doesn't seem like a big deal to me if it take up to 30 min to run.

Also, in this setup is it just demos or can users actually interact and run the code?

I agree on keeping them short. The first ones were longer because it allowed demonstration that they are correct, and not because the examples were especially big. That is a plus of the notebook format--I run it locally before committing. But the notebooks have too many other drawbacks.

I think I'll probably split the current tutorial into some number of different ones. A mix of external and embedded code will facilitate sharing among them as necessary. But first, I need to remind myself what's in there to begin with, as it has been a while.

each time what?
I'm a little confused. For the docs, when do the examples run? Is it every time a user loads the page or? If it's not every time a user loads the page, and just say when there is an update to the code base, then it doesn't seem like a big deal to me if it take up to 30 min to run.

The examples run at build-time. Just like in msprime or tskit when you run "make docs", the same thing will happen. So, if we go with the outline I'm proposing above, every tutorial will be built every time we update the repo AND every time a tutorial writer checks out the repo and starts writing a new tutorial. (But see the point above about caching)

Also, in this setup is it just demos or can users actually interact and run the code?

It's just demos the way it's one at the moment in msprime (see e.g. here), but I think you can also make these executable by turning on thebe lab support. I haven't tried it out, though.

The examples run at build-time

In that case, it seems like the run time is mostly relevant for developers and not for the end users? Yeah, so caching seems like a good idea so it doesn't take a long time for tutorial writers.

Obsoleted by jupyter-book.

Navigation Menu

Repository navigation

Convert to use jupyter_sphinx? #10

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Convert to use jupyter_sphinx? #10

Description

Activity

agladstein commented on Nov 3, 2020

agladstein commented on Nov 3, 2020

jeromekelleher commented on Nov 3, 2020

molpopgen commented on Nov 3, 2020

jeromekelleher commented on Nov 3, 2020

molpopgen commented on Nov 3, 2020

benjeffery commented on Nov 3, 2020

molpopgen commented on Nov 3, 2020

petrelharp commented on Nov 4, 2020

jeromekelleher commented on Nov 4, 2020

agladstein commented on Nov 4, 2020

molpopgen commented on Nov 4, 2020

jeromekelleher commented on Nov 4, 2020

agladstein commented on Nov 4, 2020

jeromekelleher commented on Jan 19, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions