[documentation] Embbed source code in documentation and link them

[codec-abc]

Hi,

When browsing the generated documentation (the std lib in my case) there is no way (as far as I know) to find the matching source code that the documentation is describing. I think it would be a nice addition if it can be made possible. As an example the Rust doc has links (the [src] links) that redirect to a page containing the source code with line numbers, syntax highlighting and the relevant parts highlighted.

This have the following advantage:

• Users can go easily back and forth from documentation to the matching code which is neat to understand or debug it.
• The user is sure to see the matching code for which he/she has the documentation. He/She does not have to find the version of the source code that was used to generate the documentation in case new commits were pushed since the last time the documentation was generated. Yet, that does not mean that the source code is using when compiling the code is the one that would be in the documentation.

The only minor drawback that I see is that it would make the documentation slightly bigger because the source code need to be included in it.

(Sorry if a similar issue has been opened before)

[SeanTAllen]

@codec-abc would you be interested in adding this to generated documentation?

[codec-abc]

Sure. A couple of notes:

1. The docs is generated using MkDocs that doesn't have built-in support for embedding source code.
2. I don't know how to achieve syntax highlighting, anchors, line numbers and folding.
3. If we manage to achieve it, it can be great to reuse the solution for any pony libraries. I don't if Pony come with a standard way to generate documentation for a library (maybe Pony Stable?).

I can focus on the first point to start, but since I don't know neither the Pony compiler nor MkDocs, could you share your ideas on how to achieve it?

[SeanTAllen]

@codec-abc pony's standard library documentation is generated using command to the compiler that can already be used by user standard libraries. There's no need to add to pony stable. See docgen.c in the compiler as an entry point. Docgen creates markdown files in a format that mkdocs can turn into .html. So, syntax highlighting, anchors are all things can you achieve via markdown.

For "embedding source code", you'd need to make a decision, it can point to code at a remote URL, it can point to a file/files that are hosted locally, you can embed directly in the same file. I'd start with getting the mechanics in place in a way that requires no fancy javascript so embedding directly in the same doc and hiding and showing is probably not the best idea. However, you could include the source at the end of the current file and have an anchor that points to it (and an anchor back) or something of that sort.

First step: get familiar with the documentation generation code in docgen.c.

There's a make command to generate documentation for ponyc but its currently tied to Linux:

See #2316

Note these two ponyc commands from the help:

  --docs, -g      Generate code documentation.
  --docs-public   Generate code documentation for public types only.

that's what is used to generate standard library documentation and can be used to generate library docs as well. --docs and --docs-public create the markdown that mkdocs can then use to generate html. There's a few open issues here for making it easier to create different output formats but those are on hold until the pony in pony compiler work by @jemc is done. IE, its going to be a while.

[codec-abc]

Thanks, it does help a lot.
I wil (try to) embed the source code locally to avoid broken URL and mismatch between the source code and the documentation. Moreover, I think it will be better if the code it not in the same markdown file as the documentation. Otherwise, it clutters the documentation with the source code. Most of the time the text in the docstring is enough and having the source code would just make page bigger and noisier. Ideally, I could just put the source files untouched along the markdown files and just create a link from each section of the markdown to relevant section of the source. I will experiment by doing a small Pony project and adding the source code manually just to gain experience with MkDocs. If I manage to get a nice result I will try to do it automatically in the docgen.c file.

[SeanTAllen]

Sounds like a plan @codec-abc !

[codec-abc]

I have made a bit of progress and found a solution that I think can work quite well. But it not easy as it seems since Highlight.js -the JS library used by MkDocs to do the syntax highlighting- does not support some feature such as line numbers and highlighting of a block of source code.

My current (manual) prototype involve doing the following:

1. For each pony source file do the following:

• Add <div class="pony-full-source" hidden> as the first line of the file
• Escape the characters of the source so it is valid HTML (ie, > become &gt and so on).
• Add </div> at the end
• Add the processed file to the generated documentation with a markdown extension by adding a line in the mkdocs.yml at the end of the pages entry (Because MkDocs does not support build of hidden files yet so it is best if the source code page is at the end to avoid cluttering the documentation). eg:

pages:
# The pages already generated today
- source :
  - main.pony: src/main.md
  - other.pony: src/other.md
  # And so on...

2. For each relevant section of the documentation add a link to the processed source file generated. Anchors can be used to specify the relevant line numbers. Example: http://127.0.0.1:8000/src/main/#17 to highlight line 17 of the main.pony source file. I plan to support a lines ranges too. This remain to do as I have not started to explore the C code that generated the documentation.
3. Add a bit of JS and CSS -that have to be rewritten more cleanly- in the current generated documentation to do the "magic" by adding these lines to mkdocs.yml:

extra_css: [extra.css]
extra_javascript: [extra.js]

This is what it does: It tells MkDocs to run the bit of the JS file. That JS file search the page for an hidden div containing the HTML escaped source code of the pony file. It set the content of the file into a <pre><code></code></pre> element. Then, it "read" the source code to know how many lines are in the file and add a HTML element doing the numbering (since Hightlight.js does not do it). Then, it calls Highlight.js to do the highlighting on the code block. The class attribute on the numbering element and the extra CSS are needed to be sure the line numbers height and padding match the one of the source code (I am no expert and I am quite sure the CSS can be improved too). Finally, the JS file try to extract the anchor from the URL and if it finds something it set the background color of the matching line in the numbering element to a bright yellow.

Here is a screen capture of my prototype:

In the near future, if you are OK with the current direction this is going I will clean up the existing code that I have and try to automate my current solution. Then, I will try to add links from the generated documentation to the corresponding section in the source code.

[SeanTAllen]

This looks interesting @codec-abc. Could you upload the prototype somewhere that folks could play with it?

[codec-abc]

Sure, here is the MkDocs handcrafted prototype. It should work out of the box by typing mkdocs serve. Since, it the a manual prototype the only page that has full support for the "source code feature" is the last one (src/main) available at http://127.0.0.1:8000/src/main/. If you want to test line highlighting you can try http://127.0.0.1:8000/src/main/#17(be sure to refresh the page since I don't monitor window.location changes yet)

[codec-abc]

Also I just realized that the whole HTML escaping thing is not really necessary. The line numbering can be added after the code block creation. So basically, the source code can be wrapped in just 2 lines ```ponyc at the beginning and ``` at the end. But I think it would be wise to leave a sort of a tag to know on which pages we want to run the script, because on most pages it is not necessary and lines highlighting using anchor become problematic when there is more than one code block in a page, because we cannot know which one to highlight.

[SeanTAllen]

Sorry, I meant, can you upload the generated HTML somewhere so people can play around with that?

[SeanTAllen]

@codec-abc how does one navigate to the source code?

[codec-abc]

That is what is left to do. Like I said (but not very clearly) I have to add links from the already generated documentation to the source code. I have not explored docgen.c but I hope it is possible.

[SeanTAllen]

Perhaps I misunderstood what you meant by a prototype

[codec-abc]

Well yes it is far from done. I can clean up and add a few links manually for now. But from a user perspective it demonstrate what it should look like since what is left to do is mostly automation and should be not visible to the user.

[SeanTAllen]

how do i see what it would look like? what i downloaded appears to be the same as what i would get currently.

[codec-abc]

I am cleaning up a bit. I will upload a minimalist version with manual links added. It would be far less confusing.

[SeanTAllen]

sounds good.

[codec-abc]

Here it is. If I have done things properly you should see what looks like a standard generated pony documentation. Expect it has a source file embedded that you can see in clicking on main.pony in the left ribbon. Moreover, when looking at the page of the SDLEvent struct there is 2 more links than usual. There is a link on the struct name redirecting to the source where the struct definition is. And there is another link on the create constructors redirecting to its definition.

EDIT: Do you want that I try to upload it to Github pages so other can view it more easily?

[SeanTAllen]

I don't get the src when clicking on the main.pony.

A GH pages that people can easily go navigate would be perfect @codec-abc. Would make feedback/acceptance much easier.

[codec-abc]

I have uploaded the "prototype generated documentation" to Github pages. It seems to work like I intended. I have only tested on FF nightly and chrome. Please let me know if you have issues on your side (Like a missing source file).

[codec-abc]

By the way, I should note that the prototype "design" come from this comment on the MkDocs issues pages.

[codec-abc]

@SeanTAllen Did you discuss the "prototype" with the others Pony members? If yes, do you have any feedback?

[SeanTAllen]

There was no sync this week. I'm not sure if I'll be able to attend the next couple but this has been marked as needing discussion.

[codec-abc]

I made some progress and I am pretty happy with the result. Normally, all the sources filed end up in the documentation and links to each relevant part are added automatically. I have updated my example. This time there is no manual fix involved. The documentation is put straight out to GitHub pages once I ran mkdocs build on the generated mkdocs project. Just be sure to have JavaScript enabled.

[mfelsche]

I really like this very much. Just one small comment, the [Src] link should go after the type parameters, this is apparent in the Set docs.

Thanks for pushing this.

[jemc]

A few thoughts from me:

• I think maybe using [source] instead of [Src] - it may be a little more intuitive to avoid the abbreviation and use the full word.

• If it's easy, I think it might be nice to only show the source that makes up the body of the type or method, instead of showing the source for the entire file. If this isn't easy, or would make the solution hackier, it's probably fine to leave it the way it is.

Overall, I like it! Thanks for working on it.

[codec-abc]

Thank you for the positive feedback, I will do a bit of cleanup and try to have something ready for review in the next few days/weeks.

@mfelsche I agree with you. Because of its current location it shows up on the left ribbon too. I would probably place it after and make it smaller in the same occasion.

@jemc I will change the name. About the second point, I don't think it is easy and I like having the whole file. If not, we might run into situation like this: An user want to see code that make method a() so he/she clicks on the link but, let's say most of method is in a private helper method _a_impl(). So now he/she has to navigate back to the previous page, find method _a_impl() click on the source button. So instead, I would like to have the whole file with links to other relevant parts and even other files. So if a() calls _a_impl() I would like to have a link directly in the source code that would lead to _a_impl(). Anyway, as both options are hard to do, I think I am just gonna stick to what I have for now.

About my current work on that, there is a few hacks that I made that I don't like:

1- I added a zero space invisible character for after each triple backtick in the source code. Otherwise, the markdown parser thinks the code section is terminated and the rendering is messed up. The result is okay but if the user copy paste the source from the browser and try to compile it, he/she probably gonna have some weird error because of an invisible character.
2- I introduced platform specific functions to retrieve the name of the source file based on its path. I am not even sure my code compile on Linux/MacOS.
3- I am not happy with the directory structure of the source files. I have put them in the same directory which forces me to add numbers at end of filename to avoid name clashing. I would like to create a folder by package instead.
4- The current Pony parser in highlight.js is slow and big files give it troubles. It usually freeze the browser in those cases. I can "fix" it by putting a new highlight.js version on the dependencies but that seems really hacky.

If you have ideas on how to solve some of these points, please share your ideas.

[jemc]

1- I added a zero space invisible character for after each triple backtick in the source code. Otherwise, the markdown parser thinks the code section is terminated and the rendering is messed up. The result is okay but if the user copy paste the source from the browser and try to compile it, he/she probably gonna have some weird error because of an invisible character.

On GitHub, at least, you can keep adding more backticks to the outer codefence - for example, typing this:

``````pony
actor Main
  """
  This actor is the main one in the program.
  ```
  This is an inline code fence!
  ```
  Isn't that neat?
  """
  new create(env: Env) =>
    None
``````

renders as this:

actor Main
  """
  This actor is the main one in the program.
  ```
  This is an inline code fence!
  ```
  Isn't that neat?
  """
  new create(env: Env) =>
    None

Incidentally, I was able to escape that first one by adding yet more backticks on the starting block - nine in total.

In truth, this only kicks the problem down the road, but at least it eliminates the common case of using triple-backticks as code examples in docstrings.

2- I introduced platform specific functions to retrieve the name of the source file based on its path. I am not even sure my code compile on Linux/MacOS.
3- I am not happy with the directory structure of the source files. I have put them in the same directory which forces me to add numbers at end of filename to avoid name clashing. I would like to create a folder by package instead.

If you share your branch, we can give better feedback on these two points.

4- [...] I can "fix" it by putting a new highlight.js version on the dependencies but that seems really hacky.

I'm not very well-versed in how the docgen works, but what about this is hacky? It sounds like you're just upgrading the dependency?

[codec-abc]

Thanks.

I will definitively try the trick of the repeated backticks.

For 2 and 3, you can look here but I would suggest to wait a bit because the code I wrote is far from perfect and I can certainly clean easily a big part myself.

About 4, it is hacky because MkDocs provide already highlight.js by itself and I would provide another -more up to date- version. When browsing a page, both scripts would be loaded and if the newly script is included after the older it should override it.

[jemc]

MkDocs provide already highlight.js by itself

Can we get them to update to a more recent version?

[SeanTAllen]

Are we worried about highlight.js for pony standard library docs or for documentation generation in general? if its the former, we can swap out the supplied one for one of our own at the time we create the site. that's not a particularly great solution given that its problematic for other folks using documentation generation. however, our "library" and "application" documentation generation isn't the greatest right now.

[codec-abc]

I found some time to do a bit of cleanup and tried the suggestions. The trick of the multiples backticks work nicely which allowed me to get rid of a big chunk of ugly code. Thanks @jemc !

I gave another look at the source to find a way to get rid of platform specific code but I didn't managed to do it. I also reworked the structure of the embedded source. Now each source is in a directory whose name is the one of the package of the source file. This should avoid unnecessary filename clashes but I fear that it can create other problems. I also tried to override highlight.js by putting an escaped version that I could dump into a file when generating the source documentation, but the C compiler complains that my string literal is too big (~200ko of Javascript).

At this point, I don't think I can make any real sensible progress by myself, so what should I do next?

[jemc]

Mkdocs ticket for updating highlight.js is here: mkdocs/mkdocs#1284.

[mfelsche]

What do you think about creating our own theme (http://www.mkdocs.org/user-guide/custom-themes/) - for now just by copying the currently used theme from the mkdocs sources (https://github.com/mkdocs/mkdocs/tree/master/mkdocs/themes), include a more recent highlight.js manually (maybe via one of those fancy javascript buildtools), putting it into the .docs/ dir in the repo (where already some extra javascript code is living) and reference it in the mkdocs config file generated?

While it might be more effort for us, maintaining this copied code, it will surely give us more control for changes like this and avoids hacks.

[codec-abc]

I Like the idea. However, is the .docs/ directory part of a "compiler installation"? For example, on Windows one can download a Pony compiler version from Bintray, as explained in the Readme. And the zip files does not seem to contain the .docs/ directory. Then, the compiler will not be able to copy the necessary files along the generated documentation.

[SeanTAllen]

@codec-abc we could add docs-support directory that is installed alongside the usual installation directories that contains all the contents of .docs/.

That would be work for @kulibali and I. @kulibali could handle windows. I'd take the other platforms.

[SeanTAllen]

@codec-abc there are PRs to open and/or merged to address the work @kulibali and I were going to do. You can move forward.

[SeanTAllen]

@codec add_exec_dir in package.c has the code for how Ponyc figures out its exec directory.

The logic could be extracted to a more reusable function.