1 files changed, 439 insertions, 0 deletions
diff --git a/_posts/2020-05-20-makefile-escaping.md b/_posts/2020-05-20-makefile-escaping.md
new file mode 100644
index 0000000..d468cc3
--- /dev/null
+++ b/_posts/2020-05-20-makefile-escaping.md
@@ -0,0 +1,439 @@
+---
+title: Escaping characters in Makefile
+---
+TL;DR: visit [this page] for a short and concise version of this article.
+{: .alert .alert-success }
+
+[this page]: {% link _notes/makefile.md %}
+
+I'm a big sucker for irrelevant nitpicks like properly quoting arguments in
+shell scripts.
+I've also recently started using GNU make as a substitute for one-line shell
+scripts (so instead of a bunch of scripts like build.sh, deploy.sh, test.sh I
+get to have a single Makefile and can just run `make build`, `make deploy`,
+`make test`).
+
+As a side note, there's an excellent [Makefile style guide] available on the
+web.
+I'm going to be using a slightly modified prologue suggested in the guide in
+all Makefiles in this post:
+
+[Makefile style guide]: https://clarkgrubb.com/makefile-style-guide
+
+```
+MAKEFLAGS += --no-builtin-rules --no-builtin-variables --warn-undefined-variables
+unexport MAKEFLAGS
+.DEFAULT_GOAL := all
+.DELETE_ON_ERROR:
+.SUFFIXES:
+SHELL := bash
+.SHELLFLAGS := -eu -o pipefail -c
+```
+
+`make` invokes a shell program to execute recipes.
+As issues of properly escaping "special" characters are going to be discussed,
+the choice of shell is very relevant.
+The Makefiles in this post specify `bash` explicitly using the `SHELL`
+variable, but the same rules should apply for all similar `sh`-like shells.
+
+Quoting arguments
+-----------------
+
+You should quote command arguments in `make` rule recipes, just like in shell
+scripts.
+This is to prevent a single argument from being expanded into multiple
+arguments by the shell.
+
+{% capture out1 %}
+# Prologue goes here...
+
+test_var := Same line?
+export test_var
+
+test:
+	@printf '%s\n' $(test_var)
+	@printf '%s\n' '$(test_var)'
+	@printf '%s\n' $$test_var
+	@printf '%s\n' "$$test_var"
+{% endcapture %}
+
+{% capture out2 %}
+Same
+line?
+Same line?
+Same
+line?
+Same line?
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd='cat Makefile' out=out1 %}
+{% include jekyll-theme/shell.html cmd='make test' out=out2 %}
+
+This is quite often sufficient to write valid recipes.
+
+One thing to note is that you shouldn't use double quotes `"` for quoting
+arguments, as they might contain literal dollar signs `$`, interpreted by the
+shell as variable references, which is not something you always want.
+
+Escaping quotes
+---------------
+
+What if `test_var` included a single quote `'`?
+In that case, even the quoted `printf` invocation would break because of the
+mismatch.
+
+{% capture out1 %}
+# Prologue goes here...
+
+test_var := Includes ' quote
+
+test:
+	printf '%s\n' '$(test_var)'
+{% endcapture %}
+
+{% capture out2 %}
+printf '%s\n' 'Includes ' quote'
+bash: -c: line 0: unexpected EOF while looking for matching `''
+make: *** [Makefile:11: test] Error 2
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd='cat Makefile' out=out1 %}
+{% include jekyll-theme/shell.html cmd='make test' out=out2 %}
+
+One solution is to take advantage of how `bash` parses command arguments, and
+replace every quote `'` by `'\''`.
+This works because `bash` merges a string like `'Includes '\'' quote'` into
+`Includes ' quote`.
+
+{% capture out1 %}
+# Prologue goes here...
+
+escape = $(subst ','\'',$(1))
+
+test_var := Includes ' quote
+
+test:
+	printf '%s\n' '$(call escape,$(test_var))'
+{% endcapture %}
+
+{% capture out2 %}
+printf '%s\n' 'Includes '\'' quote'
+Includes ' quote
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd='cat Makefile' out=out1 %}
+{% include jekyll-theme/shell.html cmd='make test' out=out2 %}
+
+Surprisingly, this works even in much more complicated cases.
+You can have a recipe that executes a command that takes a whole other command
+(with its own separate arguments) as an argument.
+I guess the most common use case is doing something like `ssh 'rm -rf
+$(junk_dir)'`, but I'll use nested `bash` calls instead for simplicity.
+
+{% capture out1 %}
+# Prologue goes here...
+
+escape = $(subst ','\'',$(1))
+
+test_var := Includes ' quote
+
+echo_test_var := printf '%s\n' '$(call escape,$(test_var))'
+bash_test_var := bash -c '$(call escape,$(echo_test_var))'
+
+test:
+	printf '%s\n' '$(call escape,$(test_var))'
+	bash -c '$(call escape,$(echo_test_var))'
+	bash -c '$(call escape,$(bash_test_var))'
+{% endcapture %}
+
+{% capture out2 %}
+printf '%s\n' 'Includes '\'' quote'
+Includes ' quote
+bash -c 'printf '\''%s\n'\'' '\''Includes '\''\'\'''\'' quote'\'''
+Includes ' quote
+bash -c 'bash -c '\''printf '\''\'\'''\''%s\n'\''\'\'''\'' '\''\'\'''\''Includes '\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'' quote'\''\'\'''\'''\'''
+Includes ' quote
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd='cat Makefile' out=out1 %}
+{% include jekyll-theme/shell.html cmd='make test' out=out2 %}
+
+That's somewhat insane, but it works.
+
+Shell output
+------------
+
+The `shell` function is one of the two most common ways to communicate with the
+outside world in a Makefile (the other being environment variables).
+This little `escape` function we've defined is actually sufficient to deal with
+the output of the `shell` function safely.
+
+{% capture out1 %}
+# Prologue goes here...
+
+escape = $(subst ','\'',$(1))
+
+cwd := $(shell basename -- "$$( pwd )")
+
+simple_var := Simple value
+composite_var := Composite value - $(simple_var) - $(cwd)
+
+.PHONY: test
+test:
+	@printf '%s\n' '$(call escape,$(cwd))'
+	@printf '%s\n' '$(call escape,$(composite_var))'
+{% endcapture %}
+
+{% capture cmd2 %}
+mkdir "Includes ' quote" && \
+    cd "Includes ' quote" && \
+    make -f ../Makefile test
+{% endcapture %}
+{% capture out2 %}
+Includes ' quote
+Composite value - Simple value - Includes ' quote
+{% endcapture %}
+
+{% capture cmd3 %}
+mkdir 'Maybe a comment #' && \
+    cd 'Maybe a comment #' && \
+    make -f ../Makefile test
+{% endcapture %}
+{% capture out3 %}
+Maybe a comment #
+Composite value - Simple value - Maybe a comment #
+{% endcapture %}
+
+{% capture cmd4 %}
+mkdir 'Variable ${reference}' && \
+    cd 'Variable ${reference}' && \
+    make -f ../Makefile test
+{% endcapture %}
+{% capture out4 %}
+Variable ${reference}
+Composite value - Simple value - Variable ${reference}
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd='cat Makefile' out=out1 %}
+{% include jekyll-theme/shell.html cmd=cmd2 out=out2 %}
+{% include jekyll-theme/shell.html cmd=cmd3 out=out3 %}
+{% include jekyll-theme/shell.html cmd=cmd4 out=out4 %}
+
+Environment variables
+---------------------
+
+Makefiles often have parameters that modify their behaviour.
+The most common example is doing something like `make install
+PREFIX=/somewhere/else`, where the `PREFIX` argument overrides the default
+value "/usr/local".
+These parameters are often defined in a Makefile like this:
+
+```
+param_name ?= Default value
+```
+
+They should be `escape`d and quoted when passed to external commands, of
+course.
+However, things get complicated when they contain dollar signs `$`.
+`make` variables may contain references to other variables, and they're
+expanded recursively either when defined (for `:=` assignments) or when used
+(in all other cases, including `?=`).
+
+{% capture out1 %}
+# Prologue goes here...
+
+escape = $(subst ','\'',$(1))
+
+test_var ?= This is safe.
+export test_var
+
+.PHONY: test
+test:
+	@printf '%s\n' '$(call escape,$(test_var))'
+	@printf '%s\n' "$$test_var"
+{% endcapture %}
+
+{% capture cmd2 %}
+test_var='Variable ${reference}' make test
+{% endcapture %}
+{% capture out2 %}
+Makefile:15: warning: undefined variable 'reference'
+Variable
+Variable ${reference}
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd='cat Makefile' out=out1 %}
+{% include jekyll-theme/shell.html cmd=cmd2 out=out2 %}
+
+Here, `$(test_var)` is expanded recursively, substituting an empty string for
+the `${reference}` part.
+One attempt to solve this is to escape the dollar sign in the variable value,
+but that breaks the `"$$test_var"` case:
+
+{% capture cmd1 %}
+test_var='Variable $${reference}' make test
+{% endcapture %}
+{% capture out1 %}
+Variable ${reference}
+Variable $${reference}
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd=cmd1 out=out1 %}
+
+A working solution would be to use the `escape` function on the unexpanded
+variable value.
+Turns out, you can do just that using the `value` function in `make`.
+
+{% capture out1 %}
+# Prologue goes here...
+
+escape = $(subst ','\'',$(1))
+
+test_var ?= This is safe.
+test_var := $(value test_var)
+export test_var
+
+.PHONY: test
+test:
+	@printf '%s\n' '$(call escape,$(test_var))'
+	@printf '%s\n' "$$test_var"
+{% endcapture %}
+
+{% capture cmd2 %}
+test_var="Quote '"' and variable ${reference}' make test
+{% endcapture %}
+{% capture out2 %}
+Quote ' and variable ${reference}
+Quote ' and variable ${reference}
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd='cat Makefile' out=out1 %}
+{% include jekyll-theme/shell.html cmd=cmd2 out=out2 %}
+
+This doesn't quite work though when [overriding variables] on the command line.
+For example, this doesn't work:
+
+[overriding variables]: https://www.gnu.org/software/make/manual/html_node/Overriding.html#Overriding
+
+{% capture cmd1 %}
+make test test_var='Variable ${reference}'
+{% endcapture %}
+{% capture out1 %}
+Makefile:16: warning: undefined variable 'reference'
+make: warning: undefined variable 'reference'
+Variable
+Variable
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd=cmd1 out=out1 %}
+
+This is because `make` ignores all assignments to `test_var` if it's overridden
+on the command line (including `test_var := $(value test_var)`).
+
+This can be fixed using the `override` directive for these cases only.
+A complete solution that works for seemingly all cases looks like something
+along these lines:
+
+```
+ifeq ($(origin test_var),environment)
+    test_var := $(value test_var)
+endif
+ifeq ($(origin test_var),environment override)
+    test_var := $(value test_var)
+endif
+ifeq ($(origin test_var),command line)
+    override test_var := $(value test_var)
+endif
+```
+
+Here, we check where the value of `test_var` comes from using the `origin`
+function.
+If it was defined in the environment (the `environment` and `environment
+override` cases), its value is prevented from being expanded using the `value`
+function.
+If it was overridden on the command line (the `command line` case), the
+`override` directive is used so that the unexpanded value actually gets
+assigned.
+
+The snippet above can be generalized by defining a custom function that
+produces the required `make` code, and then calling `eval`.
+
+```
+define noexpand
+ifeq ($$(origin $(1)),environment)
+    $(1) := $$(value $(1))
+endif
+ifeq ($$(origin $(1)),environment override)
+    $(1) := $$(value $(1))
+endif
+ifeq ($$(origin $(1)),command line)
+    override $(1) := $$(value $(1))
+endif
+endef
+
+test_var ?= This is safe.
+
+$(eval $(call noexpand,test_var))
+```
+
+I couldn't find a case where the combination of `escape` and `noexpand`
+wouldn't work.
+You can even safely use other variable as the default value of `test_var`, and
+it works:
+
+{% capture out1 %}
+# Prologue goes here...
+
+escape = $(subst ','\'',$(1))
+
+define noexpand
+ifeq ($$(origin $(1)),environment)
+    $(1) := $$(value $(1))
+endif
+ifeq ($$(origin $(1)),environment override)
+    $(1) := $$(value $(1))
+endif
+ifeq ($$(origin $(1)),command line)
+    override $(1) := $$(value $(1))
+endif
+endef
+
+simple_var := Simple value
+
+test_var ?= $(simple_var) in test_var
+$(eval $(call noexpand,test_var))
+
+simple_var := New simple value
+composite_var := Composite value - $(simple_var) - $(test_var)
+
+.PHONY: test
+test:
+	@printf '%s\n' '$(call escape,$(test_var))'
+	@printf '%s\n' '$(call escape,$(composite_var))'
+{% endcapture %}
+
+{% capture out2 %}
+New simple value in test_var
+Composite value - New simple value - New simple value in test_var
+{% endcapture %}
+
+{% capture cmd3 %}
+make test test_var='Variable ${reference}'
+{% endcapture %}
+{% capture out3 %}
+Variable ${reference}
+Composite value - New simple value - Variable ${reference}
+{% endcapture %}
+
+{% capture cmd4 %}
+test_var='Variable ${reference}' make test
+{% endcapture %}
+{% capture out4 %}
+Variable ${reference}
+Composite value - New simple value - Variable ${reference}
+{% endcapture %}
+
+{% include jekyll-theme/shell.html cmd='cat Makefile' out=out1 %}
+{% include jekyll-theme/shell.html cmd='make test' out=out2 %}
+{% include jekyll-theme/shell.html cmd=cmd3 out=out3 %}
+{% include jekyll-theme/shell.html cmd=cmd4 out=out4 %}