Generate CPP source file from code blocks in markdown

[bryevdv]

Description

We are looking for an authoring tool to help us present tutorials for a c++ library. Each tutorial needs to walk through an entire source file, interspersing the source code with rich commentary that links to other documentation, research papers, other tutorials, etc. Quarto seems very promising for this!

However, we definitely want a "single source" of truth for the code itself, to make sure that the actual c++ source file that a user compiles locally matches up with the code in the tutorial HTML page. So our question: is there a way to synthesize a new file from all the code blocks in a markdown file? E.g. a kind of lightweight "literate programming" that simply concatenates the code blocks together in order?

We could certainly write a post-processor to do this extraction but would prefer to use existing standard tools if for this if they are available. I had a look at quarto convert but it only seems to deal with notebooks.

Thanks for any pointers or insights, and if there is some different better approach we would be glad to hear about that, too.

---

Toy example with two cpp blocks

---
title: "C++ Tutorial"
code-block-bg: true
code-block-border-left: "#31BAE9"
---

# Some example

## Setup

Here is how the example code starts with includes and logger init, etc.

```cpp
#include "realm.h"
#include "realm/cmdline.h"
#include "realm/network.h"

#ifdef REALM_USE_OPENMP
#include <omp.h>
#endif

using namespace Realm;

Logger log_app("app");
```

## Tasks

We need to define tasks and task IDs for tasks. Below is a hello task for
CPU and optionally (if CUDA is enabled) a GPU variant.

```cpp
enum {
  MAIN_TASK = Processor::TASK_ID_FIRST_AVAILABLE + 0,
  HELLO_TASK,
};

void hello_cpu_task(const void *args, size_t arglen, const void *userdata,
                    size_t userlen, Processor p)
{
  log_app.print() << "Hello world from CPU!";
}

#if defined(REALM_USE_CUDA) || defined(REALM_USE_HIP)
void hello_gpu_task(const void *args, size_t arglen, const void *userdata,
                    size_t userlen, Processor p)
{
  log_app.print() << "Hello world from GPU!";
}
#endif
```

[mcanouil]

See:

• Command line tool to extract code #389 (comment)

(or more directly: #9112 (reply in thread))

[bryevdv]

Thanks for the reference @mcanouil when I run this code on the file above (saved as foo.md) I seem to get empty results

> quarto inspect foo.md | jq '
  .fileInformation[].codeCells |
  map(
    .metadata = (.metadata | to_entries | map("#| \(.key): \(.value)") | join("\n"))
  ) |
  group_by(.language) |
  map({
    language: .[0].language,
    contents: map(.metadata + "\n" + .source) | join("\n")
  })
'
[]

[bryevdv]

I suppose things sort of work if I rename foo.md -> foo.qmd and also replace ```cpp -> ```{cpp} but that makes the output generation take much longer, and also all the C++ code formatting disappears.

[bryevdv]

Adding --no-execute improves the execution time and lets the quarto inspect execution work successfullty. I just need a way to make the syntax highlighting come back then

"qmd" version output (BAD)

plain "md" version output (GOOD)

[cderv]

Let me clarify some context about syntax here before trying to answer your initial question.

I suppose things sort of work if I rename foo.md -> foo.qmd

.md doc can work with quarto, but this means they are not expecting to contains computation. No execution engine will be used (or more precisely, the markdown engine will be used which leads to no execution of computation of cell.

.qmd needs to be used to work with computations.

and also replace cpp -> {cpp}

The former is the Source code block syntax, and the latter is the execution code block syntax. Those are totally different usage. You shouldn't need to use computation if you only want to demo code and not execute the cell.

I just did a full explanation recently:

• Execution options: Why are there differences between syntax options? #12265 (comment)

I suppose things sort of work if I rename foo.md -> foo.qmd and also replace cpp -> {cpp} but that makes the output generation take much longer, and also all the C++ code formatting disappears.

By doing this, you transform your cells into computation cells. This is why the #389 trick is not working: the quarto inspect logic is working for computation cells. This is why you get "empty results" on .md or with .qmd and ```cpp cell.

which means that an engine will be bonded.
See https://quarto.org/docs/computations/execution-options.html#engine-binding

If you only have {cpp} cell in your document, then engine: jupyter will be used, and a kernel for CPP cell will try to be used. THis is why it takes longer - because some computations workflow happens.

Adding --no-execute improves the execution time

--no-execute flag will avoid reexecuting a .ipynb used as input. It should not do anything on a .qmd file.

Note that first quarto render will be longer when using Jupyter as a daemon needs to start. Once open, later render are faster as we do keep the daemon open by default.

Anyhow, It seems you don't need that as you only want to document cell and not run them.

I'll comment about the highlighting in the comment below.

Let me try to answer you initial question.

So our question: is there a way to synthesize a new file from all the code blocks in a markdown file? E.g. a kind of lightweight "literate programming" that simply concatenates the code blocks together in order

There is no built-in way to do that for now, especially with non computation cells.

With computation cells, you could leverage the engine feature and its language to try to do that. Or use the quarto inspect trick. Maybe in the future, a new feature:

• Quarto convert for script files (R, py) #588

For non-computation cell, you can leverage Lua and build an Extension (https://quarto.org/docs/extensions/) to catch code and do this.

With Lua you can iterate the CodeBlock and match the language, then write the content to a file, possibly according to attributes you set on the codeblock.

Here is an example from your document (written by Claude Sonnet 3.7 with some prompted help)

--[[
extract-code-blocks.lua
A Pandoc Lua filter that extracts code blocks with a specific language
(by default C++) from a Quarto document and combines them into a single file.
]]

-- Variable to collect all C++ code
local cpp_code = ""
local block_count = 0

-- Function to process code blocks
function CodeBlock(block)
  -- Check if the code block is a C++ code block
  if block.classes[1] == "cpp" then
    block_count = block_count + 1
    
    -- Add a comment indicating where this code block came from
    local comment = "\n// Code block #" .. block_count .. "\n"
    
    -- Append this code block to our collected code
    cpp_code = cpp_code .. comment .. block.text .. "\n"
    
    -- Return the block unchanged in the document
    return block
  end
  
  -- Return other code blocks unchanged
  return block
end

-- After all blocks are processed, write the collected code to a single file
function Pandoc(doc)
  if block_count > 0 then
    -- Create a file containing all C++ code
    local filename = "extracted_code.cpp"
    local file = io.open(filename, "w")
    
    if file then
      file:write(cpp_code)
      file:close()
      io.stderr:write("Extracted " .. block_count .. " C++ code blocks to: " .. filename .. "\n")
    else
      io.stderr:write("Error: Could not write to " .. filename .. "\n")
    end
  else
    io.stderr:write("No C++ code blocks found in the document.\n")
  end
  
  -- Return the document unchanged
  return doc
end

You can try on your example

---
title: "C++ Tutorial"
code-block-bg: true
code-block-border-left: "#31BAE9"
filters:
  - extract-code-blocks.lua
---

# Some example

## Setup

Here is how the example code starts with includes and logger init, etc.

```cpp
#include "realm.h"
#include "realm/cmdline.h"
#include "realm/network.h"

#ifdef REALM_USE_OPENMP
#include <omp.h>
#endif

using namespace Realm;

Logger log_app("app");
```

## Tasks

We need to define tasks and task IDs for tasks. Below is a hello task for
CPU and optionally (if CUDA is enabled) a GPU variant.

```cpp
enum {
  MAIN_TASK = Processor::TASK_ID_FIRST_AVAILABLE + 0,
  HELLO_TASK,
};

void hello_cpu_task(const void *args, size_t arglen, const void *userdata,
                    size_t userlen, Processor p)
{
  log_app.print() << "Hello world from CPU!";
}

#if defined(REALM_USE_CUDA) || defined(REALM_USE_HIP)
void hello_gpu_task(const void *args, size_t arglen, const void *userdata,
                    size_t userlen, Processor p)
{
  log_app.print() << "Hello world from GPU!";
}
#endif
```

And this will create a the following cpp file

// Code block #1
#include "realm.h"
#include "realm/cmdline.h"
#include "realm/network.h"

#ifdef REALM_USE_OPENMP
#include <omp.h>
#endif

using namespace Realm;

Logger log_app("app");

// Code block #2
enum {
  MAIN_TASK = Processor::TASK_ID_FIRST_AVAILABLE + 0,
  HELLO_TASK,
};

void hello_cpu_task(const void *args, size_t arglen, const void *userdata,
                    size_t userlen, Processor p)
{
  log_app.print() << "Hello world from CPU!";
}

#if defined(REALM_USE_CUDA) || defined(REALM_USE_HIP)
void hello_gpu_task(const void *args, size_t arglen, const void *userdata,
                    size_t userlen, Processor p)
{
  log_app.print() << "Hello world from GPU!";
}
#endif

You could improve by setting custom attributes on cell

```{.cpp extract.filename='code1.cpp'}
// content to a first file
```

```{.cpp extract=false}
// this won't be written to a file
```

```{.cpp extract.filename='code2.cpp'}
// content to new file
```

Note the dot in {.cpp} => this is still a source code block and not a computation cell.

See more about Lua in https://quarto.org/docs/extensions/lua.html

I hope it helps you understand what you can do with Quarto leveraging Pandoc Lua filters.

[bryevdv]

Thanks for the very detailed explanation @cderv A single integrated workflow is definitely the ideal goal, so an extension seems like it might work well. But there also no Lua experience at all on our team that I know of, so maintenance cost is a consideration as well. We'll have to consider that against cost a plain "Pandoc + sed script" solution it seems. Thanks!

[cderv]

You could also pre-render or post-render scripts using bash script and sed if you more at ease with that. Nice thing about Lua is the AST so that all your CodeBlock node are identified. Otherwise with sed, I guess you need to use regex.

[bryevdv]

The Lua seems self-explanatory, even if it is unfamiliar, so we'll probably try to stick with the Quarto solution (especially since there may be use to include actually executable python code at a later point)

[bryevdv]

@mcanouil Is this just a bug related to highlighting cpp code blocks? (the quarto kind with curly braces, e.g. {cpp}) If I put a python block and a cpp block side by side:

```{python}
print("foo")
```

```{cpp}
#include "realm.h"
#include "realm/cmdline.h"
#include "realm/network.h"

#ifdef REALM_USE_OPENMP
#include <omp.h>
#endif

using namespace Realm;

Logger log_app("app");
```

you can see how syntax highlighting is applied to python but not cpp (at least, not more than "default" code block formatting):

It's just weird since ```cpp (no curly braces) highlights just fine. This is so close to working for our needs.

[bryevdv]

Digging further into intermediate output, it seems {python} gets turned into something like {.python .cell-code} which Pandoc understands. But {cpp} is just left as {cpp} which presumably Pandoc does not understand. I'm just not sure looking through all the Quarto docs whether this is expected behavior or not.

[cderv]

What happens here is the following

This is computation cell

```{cpp}
#include "realm.h"
#include "realm/cmdline.h"
#include "realm/network.h"

#ifdef REALM_USE_OPENMP
#include <omp.h>
#endif

using namespace Realm;

Logger log_app("app");
```

Engine will be used, and here it will be engine: jupyter. If you don't have a kernel that handle cpp cells, the cell will not be matched to any execution.

And it seems we are passing the markdown as is. You can set keep-md to see the content after exection

---
title: "C++ Tutorial"
code-block-bg: true
code-block-border-left: "#31BAE9"
keep-md: true
---

::: {#e6bebe09 .cell execution_count=1}
``` {.python .cell-code}
print("foo")
```

::: {.cell-output .cell-output-stdout}
```
foo
```
:::
:::


```{cpp}
#include "realm.h"
#include "realm/cmdline.h"
#include "realm/network.h"

#ifdef REALM_USE_OPENMP
#include <omp.h>
#endif

using namespace Realm;

Logger log_app("app");
```

the cpp cell has not be executed, and it is passed asis.

This is because some filters could decide to do something about them. This is how shinylive or quartolive extension can work.

If you don't want to execute cell content, then you should use the Source code block syntax

---
title: "C++ Tutorial"
code-block-bg: true
code-block-border-left: "#31BAE9"
keep-md: true
---

```{python}
print("foo")
```

```{.cpp}
#include "realm.h"
#include "realm/cmdline.h"
#include "realm/network.h"

#ifdef REALM_USE_OPENMP
#include <omp.h>
#endif

using namespace Realm;

Logger log_app("app");
```