Why doesn't "complement" work as a function?

Question

I'm trying to implement the complement function from Common Lisp (CL) in Emacs Lisp, but I'm confused by why defining it as a function (as opposed to a macro) does not work.

The macro definition is straightforward:

(defmacro complement (function)
  `(lambda (&rest args)
     (not (apply ,function args))))

The function definition, however, does not work.

(defun complement (f)
  (lambda (&rest args)
    (not (apply f args))))

Something like (complement #'zerop) gives

(lambda (&rest args) (not (apply f args)))

Where the f is not replaced with the zerop.

Why is this? The equivalent function definition works fine in CL. Does it have something to do with scope?

Answer 1

Yes, this behavior is explained by differences in Variable Scoping between Emacs Lisp and Common Lisp.

In Common Lisp (a lexically scoped lisp) a lambda returned by complement you defined is turned into a closure, i.e. it captures the lexical environment that existed when it was created.

Here's what I get from SBCL:

*
(defun my-complement (f)
  (lambda (&rest args)
    (not (apply f args))))

MY-COMPLEMENT
* (my-complement #'zerop)

#<CLOSURE (LAMBDA (&REST ARGS) :IN MY-COMPLEMENT) {1002CD5BEB}>

In Emacs Lisp (which uses dynamic binding) your complement returns the form "(lambda (&rest args) (not (apply f args)))", which does not "see" that f was bound to #'zerop because it cannot "capture" the environment it is created in. (Closures can exist only in the context of lexical binding.)

You can enable lexical binding in Emacs Lisp by setting lexical-binding to t. This is a file-local variable, so you would usually put

;; -*- lexical-binding: t -*-

near the top of the file to achieve this.

With this change the functional version of complement works in Emacs Lisp.

As for why this works as a macro in Emacs Lisp, a macro generates the form with the value of f captured (you took care of it by using a backquote and a comma) and then the resulting form is evaluated, turning the form (lambda (&rest args) ...) into an anonymous function you can call.

Both steps are important: without the backquote you can't capture f, and without evaluation all you get is a funny-looking list (lambda (&rest args) ...) that you'd have to eval "by hand".

PS: I'm not a Lisp expert. Comments, corrections, and edits are always welcome.

Answer 2

What @Constantine said about lexical and dynamic scoping is true, and it explains the difference from Common Lisp behavior.

However, there is something misleading in your question. This really has nothing to do with macros. Here is a definition using defun instead of defmacro. Note that you need a quote mark (') before the use of ,function: ',function, because you want 'zerop and not zerop (which would be evaluated as a variable).

(defun complement (function)
  "Return a function that complements the effect of FUNCTION.
The resulting function applies FUNCTION to any number of args
and returns the Boolean complement of the result."
  `(lambda (&rest args)
     (not (apply ',function args))))

(funcall (complement #'zerop) 0)   ; Returns nil
(funcall (complement #'zerop) 3)   ; Returns t

(complement #'zerop) ; Returns (lambda (&rest args) (not (apply (quote zerop) args)))

When you use ',function you are not leaving function as a free variable in the resulting function (lambda form). Instead, because of the backquote surrounding the comma, the value of FUNCTION that is passed to function complement is substituted for ,function, and then quoted.

So when zerop is passed as argument to complement it is substituted for ,function and then quoted, producing 'zerop within the resulting lambda form. The backquote expression constructs a list:

(lambda (&rest args) (not (apply 'zerop args)))

When you use complement that list is evaluated to a function and used as such.

So the difference between this approach and the use of free variable function in the lexical binding case (that is, the use of a closure that couples (a) the lambda form that has free variable function with (b) an environment in which that variable is bound to zerop) is that:

1. In this approach there is no variable in the resulting form, and

2. The resulting form is a list, not a function.

Number 1 means that you cannot use the variable for anything when the lambda form is used as a function, because, Hey! - it isn't there.

And it means that unless you explicitly go to the trouble of byte-compiling the resulting form (which is a list, whose car is lambda etc.), evaluation is slower, because you are interpreting a list as a lambda form and that as a function - you do not have a (possibly byte-compiled) function directly.

When might you care whether you have a variable at the time the function is used? When you need to do something with that variable as a variable! For example, you assign it another value than function at some point. In most cases you do not need the variable as a variable - it is enough to have its value (in this case, zerop) when the function is used.