That *** missing separator bug in GNU Make

Recently at work I’ve been dealing with a code base that had to support two build setups — a local build on the developer machine and a remote one executed on a build farm. The local setup was used to build just the module in question and the remote build integrated this module together with all the code required to build a full Android image.

So far so good. I got the thing into a working state on my local machine and then went on to set it up for the build farm where I then triggered the (remote) build. After a while I got a build error like this:

foobar.mk:1234 *** missing separator. Stop.

Fair enough, it was probably some TAB being converted to SPACEs somewhere, which is a pretty common cause of this error when you work with Makefiles. But this time it wasn’t — I couldn’t find any tab related issue at all. The strangest thing, of course, was that the build worked successfully on my local machine.

The error was happening with a code that, in short, resembled this:

define some-function
  $(info Something)
  $(eval ANOTHER := $(call other-function,$(1),...))
  $(if $(SOME),normal,thing)
  etcetera...
endef

$(foreach value,$(list),$(if something,$(call some-function,...)))

In the end, as it usually is once the root problem is found, the fix was fairly simple — I just had to add a \ (backslash) character to the end of all but the last line of the some-function definition.

That was enough to solve the problem but I wanted to understand more deeply why the error happened only on the remote build and not on my local machine. So, this writeup is my attempt at explaining the why.

#The first clue

The first thing I noticed that actually led to an understanding was that the versions of Make on my local and the remote machines were different — the remote machine had a quite old one, GNU Make 3.81, whereas my development machine had a fairly current version, GNU Make 4.2.1.

GNU Make 3.81 was released in 2006, 16 years ago. GNU Make 4.2.1 is from 2016 which is quite recent given the release cadence of the project.

So, the first thing to do was to get the same Make version 3.81 running on my local machine so I could do more testing. The way I got this old Make was compiling it myself from source.

You can find a note here explaining how to build the versions of GNU Make used in this post.

#The simplest Makefile that reproduced the issue

Then, the next thing to do was to come up with the simplest Make script that reproduced the issue, which I did with a slightly simplified version of the snippet shown previously:

define some-function
  $(info Hello)
  $(info World)
endef

$(call some-function)

# 👇 Just a trick to fix warnings when executing...
all:;@:

And that was it — when executed with Make 3.81 the error got triggered:

$ ./make-3.81 -f Simplest.mk
Hello
World
Simplest.mk:6: *** missing separator.  Stop.
$
$ make --version
GNU Make 4.2.1
...
$
$ make -f Siplest.mk
Hello
World

#What does the `*** missing separator` error mean?

This infamous error is the way Make signals a syntax error. In a nutshell, Makefiles can contain these 4 types of constructs:

Rules
Comments
Variable definitions
Directives

Directive is an umbrella construct that can actually mean a lot of different things like conditionals and function calls. The important thing here is that Make throws the *** missing separator error when it fails at trying to interpret a construct it has identified as a rule.

“How?” you would say — “My makefile is correct, it doesn’t have any syntax error, I’m looking at it and it’s all valid syntax”. Yes, a rule has a relatively simple structure, it looks like this:

# Items between [] are optional
target : [prerequisites] [; recipe]
[\t   recipe]
#👆 mind the TAB character

You can see above that the : (colon) character is mandatory if the line is intended to be a rule. The problem is that if you forget it, like having a line with a command like this:

CFLAGS += -Wall
# ...
bash do_something.sh

Or really any line with non empty strings (or lists of them) like this:

SRC_FILES += winter.c summer.c
# ...
some-string
# ...
some values here aaa bbb ccc

You would get the *** missing separator. Stop error.

Now, another thing Make has is that its rule construct can be declared dynamically, meaning that, for example, the target item can be the result of a variable evaluation like this:

objects := foo.o bar.o. foobar.o

$(objects): thirdy-party/some-lib/include/thing.h
\t     ...

So, like in the 2 examples before, if you happen to have a variable or expression that ends up expanding to a non-empty string, depending on the position the thing is on your script, Make will interpret it as a rule which will be invalid since the colon will be missing.

The following example starts from a simple variable expansion and composes it in different ways — and all of them would result in the same error.

my-var := some-value

## All of the following lines end up resolving to just
## a line containing `some-value` and hence will trigger
## the `*** missing separator` error.

$(my-var)

$(if something,$(my-var))

$(foreach var, aaa, $(my-var))

# The error is triggered for any composition, like this:
$(foreach var, aaa, $(if something, $(my-var)))

# or this:
$(foreach i, $(items-list), $(if $(filter bbb, $(things)), $(my-var)))

Now, what if my-var had no value at all? If you assign an empty value for it, like this:

my-value :=

Then the missing separator error goes away — which makes sense because there’s no target for Make to try to interpret as a rule. So, the error is only triggered if there is a non-empty value being returned.

#Back to the bug…

We can change that early simplest Makefile that reproduced the bug to be more aligned with the examples we’ve just seen:

define my-var
  $(info Hello)
  $(info World)
endef

# The only meaningful change is this line:
$(my-var)

The only change here was the $(call my-var) to $(my-var). The use of Makefile functions isn’t important here.

Now, given what we’ve seen, the only way the $(my-var) line could trigger the error is if the content of the my-var variable is somehow non-empty. If we look up the GNU Make documentation for the $(info ...) function we’ll see that it returns an empty string, so the value of my-var variable would expand to this:

define my-var
<space><space>
<space><space>
endef

$(my-var)

In fact, we can go a little further and set the content of the variable to any number of empty lines and the bug will still be reproduced:

$ cat -n Simplest-v3.mk
     1  # Notice the empty lines here:
     2  define my-var
     3
     4
     5  endef
     6
     7  $(my-var)
     8
     9  all:;@:

$ ./make-3.81 -f Simplest-v3.mk
Simplest-v3.mk:7: *** missing separator.  Stop.

$ # As before, recent versions of Make don't reproduce the bug:
$ make -f Simplest-v3.mk

So Make 3.81 is somehow considering the content of the my-var non-empty…

If we inspect the content of my-var we can see that it’s indeed a newline character:

$ cat -n Simplest-v4.mk
     1  # Notice the empty lines here:
     2  define my-var
     3
     4
     5  endef
     6
     7  $(info Value: '$(my-var)')
     8
     9  all:;@:

$ ./make-3.81 -f Simplest-v4.mk
Value : '
'
$ make -f Simplest-v4.mk
Value : '
'

$ ./make-3.81 -f Simplest-v4.mk | xxd -g 1
00000000: 56 61 6c 75 65 3a 20 27 0a 27 0a      Value: '.'.
$ make -f Simplest-v4.mk | xxd -g 1
00000000: 56 61 6c 75 65 3a 20 27 0a 27 0a      Value: '.'.

So I went on searching for bugs like that and ended up stumbling across this page:

And that was it, the bug was found.

You see, when assigning/creating to a variable with the define directive, GNU Make versions prior to the fix would consider a value containing newlines (with or without whitespaces) as non-empty which would in turn cause the value to be recognized as a valid rule’s target resulting then in the *** missing separator error.

#Bonus: testing the fix

The bug page doesn’t show, but if you search in the GNU Make git repository, you’ll find the fix commit:

commit e97159745d3359285cef535af780cd8e2b6b0791
Author: Paul Smith <[email protected]>
Date:   Mon Mar 21 00:36:55 2016 -0400

  [SV 46995] Strip leading/trailing space from variable names
  
  * makeint.h: Change MAP_SPACE to MAP_NEWLINE, and add MAP_PATHSEP
  and MAP_SPACE which is now MAP_BLANK|MAP_NEWLINE.  Create
  NEW_TOKEN(), END_OF_TOKEN(), ISBLANK(), ISSPACE() macros.
  * main.c (initialize_stopchar_map): Set MAP_NEWLINE only for
  newline characters.
  * Convert all uses of isblank() and isspace() to macros.
  * Examine all uses of isblank() (doesn't accept newlines) and
  change them wherever possible to ISSPACE() (does accept newlines).
  * function.c (func_foreach): Strip leading/trailing space.
  * variable.c (parse_variable_definition): Clean up.
  * tests/scripts/functions/foreach: Test settings and errors.
  * tests/scripts/functions/call: Rewrite to new-style.
  * tests/scripts/misc/bs-nl: Add many more tests for newlines.

You can also view this commit here.

Now, how could we be sure this is the commit that actually fixes the bug?

Well, we could download the git repository and compile two versions of Make: one with the git HEAD exactly at commit before this and another on exactly this commit. If the bug reproduces on the first version and doesn’t on the second we’re finished. Indeed, this is exactly what happens and this note has more details.

#Why did the backslash `\` solve the issue on Make 3.81?

The question of why it worked is equivalent to asking How does line-splitting or line escaping work on Make?

As you’ve probably noticed already, traditional Unix tools usually have this “line-oriented” syntax. Make is no different and, for convenience, we usually use the \ to break a long line into a more readable form.

The answer to this question, however, has more to do with the define-endef construct because it’s an exception to this line-based syntax — the define directive exists precisely for the purpose of enabling multiline value assignment.

For reference:
※ Stack overflow: Is it possible to create a multi-line string variable in Makefile
※ GNU Make manual: Defining Multi-Line Varibles

Besides its <start>-<stop> syntax, technically, the only different thing this construct does is that it keeps the newline characters inside of it (except for the last one). And… it has one small additional feature — it allows for the usage of line-escaping! So, when using it, you can choose which newlines you want to keep and which ones are just for reading convenience.

Consider this example:

1 define greating
2   Hello
3   World
4 endef
5 
6 $(info Greating: '$(greating)')
7 all:;@:

When executing it, we got this:

$ make -f foo1.mk
Greating: '  Hello
  World'

Now let’s see how the usage of escaping alters the values:

$ cat -n foo2.mk
     1  define greating-v2
     2    Hello \
     3    World
     4  endef
     5
     6  $(info Greating (v2): '$(greating-v2)')
     7  all:;@:

$ make -f foo2.mk
Greating (v2): '  Hello World'

Noticed the 2 SPACEs before the World in greating-v2 were removed? You can learn more about how GNU Make handles line-spliting here. The answer from this Stack Overflow is very good too.

So, to answer the question: using the backslash inside the define-endef pair made the newlines characters to not be included in the value being assigned to the variable. Since what triggered the bug was the presence of the newline characters in the resolved value of the variable, the bug didn’t get triggered.

1	define greating
2	Hello
3	World
4	endef
5
6	$(info Greating: '$(greating)')
7	all:;@: