Macro Macros

A developer detailed the challenges of building domain-specific languages for graph database queries in Clojure, focusing on handling variables in Datomic and SPARQL. The post explores using symbols for variables and the need for quoting to avoid evaluation, with references to libraries like HoneySQL and Flint.

This is a follow up to my recent posts on Domain Specific Languages DSLs for database queries in Clojure https://dev.to/quoll/clojure-query-dsls-137m , and different quoting forms in Clojure https://dev.to/quoll/quoting-difficulties-4mad to enable this. I had originally thought that I would write this post in a "Tutorial" style. But then it occurred to me that I used to get the best responses from people when I just documented what I learned as I learned it. I think it's a more "open" and personable style of writing, which may be what people liked. This approach might have extra appeal in the current era of AI slop, since people ought to see that I've written it myself. Unless I usually sound like an AI 🙃 I'm also hoping that this approach will be easier to write, since I don't need to restructure my exploration as an instructional post. On the other hand, it may be a terrible idea. But I won't know unless I try… SQL is very structured around column selection, so it does not usually need anything like variables inside a query. This allows HoneySQL https://github.com/seancorfield/honeysql to build most of its structures using functions that take keywords and values as arguments. On the other hand, graph query languages like Datomic https://docs.datomic.com/ , SPARQL https://www.w3.org/TR/sparql12-query/ , and GQL https://en.wikipedia.org/wiki/Graph Query Language typically use pattern matching on the graph, where parts of the pattern contain a variable that will be "bound" to associated values when a pattern matches. For instance: ?person :hasFriend ?friend … is a pattern for matching edges in a graph where something is connected by a property labelled :hasFriend to another node. Every edge in the graph that matches this pattern, leads to a pair of nodes that the variables ?person and ?friend get bound to. GQL would do something similar with a MATCH clause of: php person:Person - :HAS FRIEND - friend:Person Like the SPARQL form, this binds the first node which must be of type Person to the variable person , and the second node which is also a Person to the variable friend . I'm focused on Datomic and SPARQL, which are more closely related to each other, so I won't address GQL here. The issue with variables is that keywords are already being used in queries such as for the :hasFriend property , so we need something else. The obvious candidate is the Symbol , and this is the route that Datomic took. However, Clojure uses symbols to refer to values, meaning that they need to be inserted in queries without being evaluated. That was why I did that post on quoting symbols: quoting is a mechanism to create a symbol as a part of a structure without asking Clojure to instead insert the value the symbol references. i.e. if I say: js let ?person "bad data" ?person :hasName "Fred" ;; returns= "bad data" :hasName "Fred" This isn't what I wanted. It is even worse if I don't have ?person predefined as the let expression does , since the code can't even run. Instead, I want the symbol in the first position: js let ?person "bad data" '?person :hasName "Fred" ;; returns= ?person :hasName "Fred" This doesn't care if the symbol is defined or not. We get back the structure that we wanted with a symbol in it. A colleague has been using Flint https://github.com/yetanalytics/flint to build SPARQL queries programmatically. I like Flint, because it uses Datomic-style queries to express SPARQL. Datomic has been designed to use from Clojure, with a query language based in Clojure data structures. So Flint should make it just as easy to query SPARQL. However, because the queries are being built programmatically, we run the risk of reusing a variable name each time a new expression is added into the query. Reusing a variable for a different purpose will generally break a query, so he was generating new variable names whenever he was generating a new query clause. There are a couple of ways to generate new variables for a query. One way is to generate a new name, build a symbol with that name, and then insert it without quoting the name being used to carry that symbol: let ?container gensym "?container" :my-object :contains ?container ?container :value "hello" ;; result= :my-object :contains ?container180 ;; ?container180 :value "hello" Another approach is to use an auto gensym: :my-object :contains ?container ?container :value "hello" ;; result= :my-object :contains ?container 2 auto ;; ?container 2 auto :value "hello" Both of these seem fine, but it gets harder to read and write when new constraints are added. For instance, I may be looking for a :value that is a string containing the substring "lo": let ?container gensym "?container" ?value gensym "?value" :my-object :contains ?container ?container :value ?value :filter ' contains ~?value "lo" ;; result= :my-object :contains ?container140 ;; ?container140 :value ?value141 ;; :filter contains clojure.core/unquote ?value "lo" Which doesn't work. We need syntax quoting instead: let ?container gensym "?container" ?value gensym "?value" :my-object :contains ?container ?container :value ?value :filter contains ~?value "lo" ;; result= :my-object :contains ?container144 ;; ?container144 :value ?value145 ;; :filter user/contains ?value145 "lo" Again… the quoting didn't work: let ?container gensym "?container" ?value gensym "?value" :my-object :contains ?container ?container :value ?value :filter ~'contains ~?value "lo" ;; result= :my-object :contains ?container148 ;; ?container148 :value ?value149 ;; :filter contains ?value149 "lo" To be fair, I did know how to quote that all along, but I wanted to demonstrate that it can catch people out. This is all very contrived. But what about if I want to construct something more complex. Say, I want a general way to select an entity and its contained value, and then I decide that I want to filter that by containing "lo" . The first part would be in a function, but I will need to provide the binding ?value variable to filter on it: defn select-entity-value ?entity ?value let ?container gensym "?container" ?entity :contains ?container ?container :value ?value let ?e gensym "?e" ?v gensym "?v" ~@ select-entity-value ?e ?v :filter ~'contains ~?v "lo" ;; result= ?e178 :contains ?container180 ;; ?container180 :value ?v179 ;; :filter contains ?v179 "lo" Splice-unquoting like this the ~@ syntax isn't really necessary, though quoting everything means that the expression inside the :filter doesn't need its own quoting since it is a list, and we have to quote lists, or else they get executed . We could even avoid almost quotes with: let ?e gensym "?e" ?v gensym "?v" conj select-entity-value ?e ?v :filter list 'contains ?v "lo" But now we have conj and list in the expression, we're still quoting contains , and we have the messy gensym declarations at the top. This DSL doesn't look all that easy to use. Until now, I've been rehashing what I've talked about already, and some of the things my colleague was building. It all worked, but it just looked… clunky. I found myself thinking that surely we could do better, right? Patterns are not too complex to work with, since they are just short vectors, containing keywords, symbols, or simple values like strings and numbers. The mess really showed up when we tried to introduce a filter. The filter we used here was the contains function. There are a lot of other functions in SPARQL, so there isn't anything particularly special about that function. Hopefully, whatever we do to address one function could be applied to all of them. It would be nice to just insert the contains expression directly into the query. Something like: :filter contains ?v "lo" Bindings work similarly, except instead of evaluating an expression and passing through everything that returns a true result, they evaluate an expression and save it in a variable. For instance, to save a the lower-case form of the string ?v , you can bind it: :bind lcase ?v ?lowv The problems here are that contains and + looks like a function. Even if we created a function for it, the ?v is not bound to anything, so the function would fail. However, Clojure macros can accept any symbol as an argument. Can we use them somehow? Macros are Clojure code that generates Clojure data. The trick is that the macro is called during compilation, and the resulting data is inserted into the source code before it gets compiled. This lets you generate code using code. Clojure itself uses macros everywhere. Most of the "built in" syntax is actually just macros that write out other code. At the end of the day, Clojure only contains a few " special forms https://clojure.org/reference/special forms " that get presented to the compiler. But don't let that list of special forms fool you: many of those e.g. defn , fn , and let are actually macros as well, with simpler special forms underlying them. Many of these macros accept symbols that are not bound, safely generating code that uses those symbols. For instance, consider an expression for an identity function: fn x x We can see what the fn macro expands to by using macroexpand : js = macroexpand ' fn x x fn x x This is just a rewrite to use fn instead, which is much simpler: fn does not do argument destructuring, it does not handle pre nor post conditions, it does not attach metadata, and it always expects parentheses around a function argument list and body even when there is only a single arity. But importantly, notice how we can provide fn with an expression that includes x and it returns a new expression that also includes x ? We don't need x to exist before doing this. That may be what we need. Let's try it out. Can we create a contains macro that accepts a symbol argument and returns a list with appropriate symbols in it? My first attempt was laughable: defmacro contains a b ~'contains ~a ~b So when I call contains "foot" "foo" it should return a list containing the symbol contains , the string "foot" and the string "foo". That was what was returned, but then that goes to the compiler, and the symbol contains doesn't exist, so it fails. Doh. I needed an actual symbol object in the position of contains : defmacro contains a b let c symbol "contains" ~c ~a ~b But I didn't need to evaluate that to know what it would do… calling contains "foot" "foo" would generate the list contains "foot" "foo" and that would be evaluated, which then repeats the process until the macro evaluation overflows the stack. I had forgotten that I'm not returning the data structure anymore. Now I'm returning code that evaluates to the required data structure. Aside from quoting a list which gets tricky, because macro expansion essentially unquotes what you've quoted, so you need to double-quote , you can create a list with the list function: defmacro contains a b let c symbol "contains" list ~c ~a ~b How does this look? js = macroexpand ' contains "foot" "foo" clojure.core/list contains "foot" "foo" Oh. That was sort of obvious in hindsight. Let's forget inserting the symbol and just put it in place: js defmacro contains a b list symbol "contains" ~a ~b = macroexpand ' contains "foot" "foo" clojure.core/list clojure.core/symbol "contains" "foot" "foo" = contains "foot" "foo" contains "foot" "foo" See why I don't usually like writing this way? Showing how long it takes to get to something that should be easy is embarrassing. But I'm really not happy about embedding the symbol by generating one based on a string. Is there another way? What if I quote it? js defmacro contains a b list 'contains ~a ~b = macroexpand ' contains "foot" "foo" clojure.core/list quote user/contains "foot" "foo" That got me part of the way there, but I forgot that I'm in a syntax quote, so contains is resolved with its full namespace. I can write out quote ~'contains , but I'm curious… can I just chain the quote/unquote syntax here? js defmacro contains a b list '~'contains ~a ~b = macroexpand ' contains "foot" "foo" clojure.core/list quote contains "foot" "foo" = contains "foot" "foo" contains "foot" "foo" I might stick to the quote … form though, since cascading quote/unquote could get hard to read. It turns out that this mattered later on. That did work, but what about the arguments? Right now they are being passed through, and they just appear verbatim in the final form. That doesn't work for variables like ?value : js = contains ?value "foo" Syntax error compiling at REPL:1:1 . Unable to resolve symbol: ?value in this context We need to quote the arguments too. js defmacro contains a b list quote ~'contains quote ~a quote ~b = macroexpand ' contains ?v "foo" clojure.core/list quote contains quote ?v quote "foo" = contains ?v "foo" contains ?v "foo" This looked great Until my colleague pointed out that he couldn't pass non-literal values into the macro. Of course he couldn't 🤦♀️ In this case, I need to check if the argument is a symbol, and if it is, then look for the form ?name or $name SPARQL allows either ? or $ to indicate a variable . I'll create a helper function that tests if an argument is a symbol that starts with one of these characters, and only quote the ones that match. However, a function like that can't just return quote a , since that just evaluates to a and won't get inserted into the output correctly. Instead, I need to return a list of the form quote a : defmacro contains a b let vquote fn s if and symbol? s {\? \$} first name s list 'quote s s a vquote a b vquote b list quote ~'contains ~a ~b user= macroexpand ' contains ?v "foo" clojure.core/list quote contains quote ?v "foo" user= macroexpand ' contains v "foo" clojure.core/list quote contains v "foo" OK, I'll admit it… it took me a few iterations to get that right. Quoting quote was not something I was expecting. This works for contains , but what about all the other SPARQL functions? Do I have to write this out for all of them as well? Then there are functions like regex that can take 2 or 3 arguments. It would be nice if I could take a function name, and generate the appropriate macro. Except, macros are just code, and macros generate code. So I could always try generating a macro with a macro, right? Well, sort of. It turned out to be harder than I thought. Let's try creating a macro that generates a macro for a single-argument function, like lcase . I decided to skip the argument handling to start with, just to get the basic structure down. So my first attempt was: defmacro sparql-fn1 s defmacro ~s ~'a list quote unquote ~s ~'a This made it clear that I wasn't really sure of how to perform the quote ~'contains when I didn't have a literal value to put into the quoted position. I thought I'd try it anyway, and it complained about "No such var: user/s", which didn't really surprise me. So let's see, what did it look like? I'll reformat the output so it's not all on a single line: js = macroexpand ' sparql-fn1 lcase do clojure.core/defn lcase &form &env a clojure.core/seq clojure.core/concat clojure.core/list quote clojure.core/list clojure.core/list clojure.core/seq clojure.core/concat clojure.core/list quote quote clojure.core/list clojure.core/seq clojure.core/concat clojure.core/list quote clojure.core/unquote clojure.core/list user/s clojure.core/list quote user/a . var lcase setMacro var lcase Let's remove the clojure.core namespaces, the redundant seq calls, and use some quoting syntax: do defn lcase &form &env a concat list 'list list concat list 'quote list concat list 'unquote list user/s list 'user/a . var lcase setMacro var lcase This helped me figure out where I'm passing things in poorly, but there was something much more important going on here. The returned data structure was not the code to call defmacro but instead was a do block with 3 steps: lcase . This takes 3 arguments, prepending &form and &env before the argument I had declared. var for it. The second step calls setMacro on that var. lcase var.The arguments &form and &env are explained in the Clojure docs https://clojure.org/reference/macros special variables as special variables that are available inside macros without having to declare them. We are seeing this being set up. Also, rather than creating a "macro", a function is being created that is then converted to a macro. I'd seen elements of this before, but I'd never really dug into it. This prompted me to look at the defmacro source by typing source defmacro a the repl , and… yes, this is what it's doing. Most surprisingly, the code of the macro is not in the function, but rather the list structure of the macro is being constructed and returned. Reading the body, it looks like the final structure evaluates to: list 'list list 'quote list 'unquote user/s 'user/a The user/s part came about due to my unfortunate quote/unquote structure, and the user/a is because of how I'm calling macroexpand , but the rest of it actually evaluates to what I was expecting, to wit: list quote unquote user/s user/a And that structure would evaluate to what gets inserted into source code. So the function here generates a structure that evals to what the macro returns, which then evals a second time to what gets inserted into the source. So I can manually create a macro by creating a function that needs to evaluate twice to create the final code structure. That's going to be confusing. But then I realized that I don't have to worry about the quote/unquote mess anymore. If I want to quote something, then using quote does not hide the quoted value, because now I am creating a list where quote is the first element, and the thing to be quoted is exactly what I want. So it's more complex, but more flexible. Let's try then. To start with, I decided on a form that just takes all of its arguments, and wraps each one in the helper function that quotes it if the argument is a symbol starting with $ or ? . defmacro sparql-fn s let argq fn a list 'if list 'and list 'symbol? 'a list {\? \$} list 'first list 'name 'a list 'list list 'quote 'quote a a list 'do list 'defn s ' &form &env & args list 'list list 'quote 'cons list 'list list 'quote 'quote list 'quote s list 'concat list 'list list 'quote 'list list 'map list 'fn ' a argq 'a 'args list '. list 'var s ' setMacro list 'var s This took me a few attempts, but it worked out pretty well: js = sparql-fn contains 'user/contains = sparql-fn lcase 'user/lcase = contains ?x "lo" contains ?x "lo" = lcase "Hello" lcase "Hello" = lcase a Syntax error compiling at REPL:1:1 . Unable to resolve symbol: a in this context Let's see what the generated code looks like: js = macroexpand ' sparql-fn lcase do defn lcase &form &env & args list quote cons list quote quote quote lcase concat list quote list map fn a if and symbol? a {\? \$} first name a list quote quote a a args . var lcase setMacro var lcase How does the returned list structure eval then? Working it manually, I came up with: list 'cons list 'quote 'lcase concat ' list map fn a … args The function here was a bit long, so I elided it. It's looking good, but evaluating it one more time, gets us to: cons quote lcase list map fn a … args Which, when called as lcase "Hello" should return a list containing 'lcase followed by the function mapped over all arguments. The function returns its input for a string, so the final list would be lcase "Hello" . Similarly, for lcase ?value it's the same kind of thing, but now the function will return a list of quote ?value , so the final output will be lcase quote ?value . There is no checking on the arguments at all, but I know that Fink is already doing that kind of work, so I thought this was where I could stop. At this point I decided to try it with all of the SPARQL functions, and immediately hit a wall with concat . This is a function that appears in both SPARQL and in the above macro. As soon as concat is declared as a SPARQL macro, then the next thing I try to declare as a SPARQL macro will use the new concat macro instead of clojure.core/concat . Fink also tries to map and to SPARQL's && , so that should be addressed as well: defmacro sparql-fn s let argq fn a list 'if list 'clojure.core/and list 'symbol? 'a list {\? \$} list 'first list 'name 'a list 'list list 'quote 'quote a a list 'do list 'defn s ' &form &env & args list 'list list 'quote 'cons list 'list list 'quote 'quote list 'quote s list 'clojure.core/concat list 'list list 'quote 'list list 'map list 'fn ' a argq 'a 'args list '. list 'var s ' setMacro list 'var s Finally, I can write SPARQL queries using Fink with clean filters: ' ?entity :contains ?c ' ?c :name ?name :filter contains ?name "lo" Well, this wasn't a tutorial. It was more like a litany of my mistakes as I stumbled towards understanding. I'm sure an LLM could have told me how to do this immediately, but then I wouldn't have learned anything. I have no idea if anyone else will ever read this, but if you did, then thanks. If not, then that's OK. Writing these things down helps my clarify and consolidate what I learned. This is something I used to do regularly a long time ago, and I should do more of it. As per my AI policy, I'm going to leave the typos alone so you know I really did type this. But let me know if you see any sentences that make no sense, and I'll clean them up.