As I have been exploring Hugo I’m finding a lot of things I like but it seems that it doesn’t allow for custom pre-processors. That’s okay. I’ll just do it myself. After trying multiple ideas, there seems to be one big concession I needed to make. Since Hugo has no ability to call outside executables, there needs to be two pipelines that are happening at the same time. While not optimal, the final product actually works pretty well.
A first I thought that I could use the shortcodes to directly call the Lilypond binary, but Hugo does not allow calling outside programs. The other option I came across was using the abcjs library however it doesn’t support some of the more complex things I want to do. (e.g., quasi-Schenker graphs where stems and note lengths don’t matter) So for the solution I decided on I am leveraging the extensive features of Hugo, specifically short codes and data templates. The Gulp script will build the Lilypond files and write images to the static directory and the results to the data directory. Hugo will then pull the information from the data directory and include the files in the markdown pages.
Gulp
This is the main “magic” that is going to happen here. There will be two primary Gulp Tasks. The first task is a non-watching task that will just build all of the files. The second task will watch the lilypond files and rebuild the one that is changed. Both of them will then write the image information to a data file in the Hugo data directory.
gulpfile.ts
1/**
2 * Imports
3 */
4import * as cp from "child_process";
5import * as crypto from "crypto";
6import * as fs from "fs";
7import * as path from "path";
8import * as gulp from "gulp";
9// The type defs for gulp-tap are wrong.
10const tap = require("gulp-tap");
11import * as rimraf from "rimraf";
12import { BufferFile } from "vinyl";
13
14/**
15 * Script Consts
16 */
17const LILYPOND_BASE_OPTIONS = [
18 "-dbackend=eps",
19 "-dno-gs-load-fonts",
20 "-dinclude-eps-fonts",
21 "-dpixmap-format=png16m",
22 "-dresolution=96",
23 "--png",
24 "-daux-files=#f",
25 "-drelative-includes=#t"
26];
27const LILYPOND_IN_DIR = "assets/lilypond";
28const LILYPOND_MAP = new Map();
29const LILYPOND_OUT_DIR = "static/images/lilypond";
30
31/**
32 * Main Gulp Task Functions
33 */
34
35/**
36 * This function will clean out directories and build all lilypond excerpts
37 * @param cb
38 */
39// This will clean out dirs and build all files.
40function buildAllLily(cb: () => void): void {
41 // make sure the out directory exists...
42 fs.mkdirSync(LILYPOND_OUT_DIR, { recursive: true });
43
44 // clean old files
45 cleanOutDirs();
46
47 // read all of the files in the lilypond directory and hash/compile
48 return gulp
49 .src([`${LILYPOND_IN_DIR}/**/*.ly`, `!${LILYPOND_IN_DIR}/headers/*`])
50 .pipe(
51 tap((file: BufferFile) => {
52 // calculate new hash
53 file.hash = calcHash(file.contents);
54
55 // build file
56 buildImg(`${LILYPOND_IN_DIR}/${file.relative}`, file.hash);
57
58 LILYPOND_MAP.set(file.relative, file.hash);
59 })
60 );
61}
62
63/**
64 * Dumps the lilypond file map to data dir
65 * @param cb
66 */
67function writeLilyMap(cb: () => void) {
68 return fs.writeFile(
69 "data/lilypond.json",
70 JSON.stringify(mapToObject(LILYPOND_MAP)),
71 () => {
72 cb();
73 }
74 );
75}
76
77/**
78 * Watches lilypond files and processes changes
79 */
80function watchFiles() {
81 // load current files in hash so we don't reprocess existing files.
82 try {
83 let jsonFile = fs.readFileSync("data/lilypond.json", "utf8");
84 let currentRecords = JSON.parse(jsonFile);
85 for (let [key, val] of Object.entries(currentRecords)) {
86 LILYPOND_MAP.set(key, val);
87 }
88 } catch (e) {
89 console.log("No data file found. Using blank hash map.");
90 }
91
92 // setup gulp watch task
93 const watchTask = gulp.watch([
94 `${LILYPOND_IN_DIR}/**/*.ly`,
95 `!${LILYPOND_IN_DIR}/headers/*`
96 ]);
97
98 // since we are only watching lilypond files, we want to be notified
99 // about everything
100 watchTask.on("all", (_, fPath) => {
101 let contents = fs.readFileSync(fPath);
102 let hash = calcHash(contents);
103 buildImg(`${fPath}`, hash);
104 LILYPOND_MAP.set(path.relative(LILYPOND_IN_DIR, fPath), hash);
105 writeLilyMap(() => {});
106 });
107
108 return watchTask;
109}
110
111/**
112 * Helper Functions
113 */
114
115/**
116 * Creates the lilypond img file and saves to OUT_DIR
117 * @param fileName string
118 * @param hash string
119 */
120function buildImg(fileName: string, hash: string) {
121 // We need to use Array.from to actually clone the array or headaches happen.
122 let options = Array.from(LILYPOND_BASE_OPTIONS);
123 options.push(`--include=/app/assets/lilypond/headers`);
124 // Add output file
125 options.push(`--output=/app/${LILYPOND_OUT_DIR}/${hash}`);
126 // Add input file
127 options.push(`/app/${fileName}`);
128
129 let dockerOptions = [
130 "run",
131 "--rm",
132 "-v", `${process.cwd()}:/app`,
133 "-w", "/app",
134 "gpit2286/lilypond:dev",
135 "lilypond"
136 ];
137
138 // Build the file
139 let res = cp.spawnSync("docker", dockerOptions.concat(options), {});
140 // log lilypond output... which is written to error... 🤷
141 console.log(res.stderr.toString("utf8"));
142}
143
144/**
145 * MD5 Hashes input buffer
146 * @param fileContents Buffer
147 */
148function calcHash(fileContents: Buffer): string {
149 return crypto
150 .createHash("md5")
151 .update(fileContents)
152 .digest("hex");
153}
154
155/**
156 * Cleans LILYPOND_OUT_DIR and data file.
157 */
158function cleanOutDirs() {
159 // remove all previous files
160 rimraf.sync(`${LILYPOND_OUT_DIR}/*`);
161 rimraf.sync("data/lilypond.json");
162}
163
164/**
165 * Turns a map into an object so that JSON.stringify works properly
166 * @param map Map<any, any>
167 */
168function mapToObject<K, V>(map: Map<K, V>) {
169 const out = Object.create(null);
170 map.forEach((value: V, key: K) => {
171 if (value instanceof Map) {
172 out[key] = mapToObject(value);
173 } else {
174 out[key] = value;
175 }
176 });
177 return out;
178}
179
180exports.buildAll = gulp.series(buildAllLily, writeLilyMap);
181exports.watch = gulp.series(watchFiles);
Hugo
For the most part, Hugo is going to do what Hugo does. We will still use the hugo server to watch and recompile, but we need to connect the Gulp output to the website itself. For this, I decided to create a shortcode in Hugo.
layouts/shortcodes/lilypond.html
1{{/*
2 Lilypond shortcode
3 lilypond code_path title
4 codepath - path to lilypond include beyond the assets/lilypond without the ly
5 title - excerpt title
6*/}}
7
8{{ $fullPath := printf "%s.ly" (.Get 0) }}
9{{ $hash := index (.Site.Data.lilypond) $fullPath }}
10
11<figure>
12 <img
13 data-src="/images/lilypond/{{ print $hash }}.png"
14 data-sizes="auto"
15 alt="{{ print (.Get 1) }}"
16 title="{{ print (.Get 1) }}"
17 class="lazyload">
18 <figcaption
19 class="image-caption">
20 {{ print (.Get 1) }}
21 </figcaption>
22</figure>The last part of this is adding a couple of config options to the config.toml. We ignore the files in the assets/lilypond so that Hugo doesn’t recompile when the Lilypond files are edited. The second exclude, .eps files, is because when Lilypond generates the png files, it briefly creates an eps file. This eps file is then noticed by Hugo and Hugo tries to copy it from the static folder to the public folder. When Hugo goes to copy the file, it has already been deleted and Hugo throws all kinds of errors. So, we just tell Hugo to ignore the file when it’s created.
ignoreFiles = [ "assets/lilypond", "eps$" ]The good, the bad, and the ugly
For the most part this works great. Run gulp watch and hugo server and they both do what they are supposed to do – almost. There is currently some discussion about short codes and when they should re-generate. Although a little bit of a headache, until Hugo gets a more robust shortcode regeneration scheme, you will just need to re-save the file that the lilypond shortcode exists on.