i did not know this (preference for syntax-parse) which would have come in handy a couple of weekends ago when I was banging away with syntax-case

TIL: you can use define-sequence-syntax to make efficient generators for for.

So here’s the question: does CL have any cooperating macros like define-sequence-syntax?

Btw, cooperating macros remind me of https://www.cs.utah.edu/plt/publications/jfp12-draft-fcdf.pdf, which explains the importance of syntax object in Racket.

I don’t think it’s straightforward. I was actually pondering this myself: I am not even close to being a macro expert but generally macros operate on the sub-expressions that are their arguments. It would be nice for macros to have access (read and/or write — dare we dream) to the context within which they are called.

for example it sounds like for knows about define-sequence-syntax . Could we design a macro system where it doesn’t and define-sequence-syntax does the same transformations from within for?

I was thinking that maybe if instead of syntax subtrees macros accepted syntax zippers but I haven’t fleshed the idea out very well

> define-sequence-syntax does the same transformations from within for Can you clarify what you mean by this? Perhaps a concrete example would be nice.

I’m not sure if define-match-expander would be an example of a cooperating macro?

And, it’s Emacs Lisp not Common Lisp (but I think the macro system is similarly not-Scheme)?

But pcase (ELisp’s match) has a pcase-defmacro to let you define your own patterns.

https://github.com/emacs-mirror/emacs/blob/master/lisp/emacs-lisp/pcase.el

Yes, I would definitely count pcase + pcase-def as cooperating macros.

The example is precisely define-sequence-syntax. IIUC define-sequence-syntax defines a macro that does different things depending on whether it is in a for clause or if it is in any other context. If I interpret the code correctly this is done by creating a sequence-transformer clause that for knows how to handle.

Now what if we want to create define-for-pattern-syntax that allows us to destructure the id:

(struct foo (a b)) ;; A pattern to destructure foo in for caluses (define-for-pattern-syntax (foo-for-pat a b) ...) ;; Here a and b are bound first to 1 and 2 and then to 2 and 3 (for ([(foo-for-pat a b) (list (foo 1 2) (foo 2 3))]) (do-stuff a b)) But for doesn’t know about it so my understanding is that we need a new kind of for that knows about our hypothetical define-for-pattern-syntax for this to be fasible.

Ah yes, you are right that for only knows how to handle these sequence syntax. If you have other extensions in other positions, for wouldn’t work with them.

https://lexi-lambda.github.io/blog/2018/10/06/macroexpand-anywhere-with-local-apply-transformer/

Yeah, but that requires a wrapper

All the Racket docs (while great at what they do) are about showing someone who already knows what they’re doing what the functions are. And the initial intro isn’t great.

For example, if I just open up the Racket Reference (https://docs.racket-lang.org/reference) and go down to Macros (https://docs.racket-lang.org/reference/Macros.html) and then click the opening section, which I’d expect to be a basic tutorial, I get this: https://docs.racket-lang.org/reference/stx-patterns.html - the syntax-case definition with no real explanation of how (or why) it’s used. For people who don’t already know Lisp this would be crazy confusing. And for people who don’t know Scheme, it’s also equally frustrating to dive in and start using; constantly met with errors about syntax objects or pattern variables being used outside of templates, and what is this () after the stx parameter that looks like a parameter list, but never appears to be used for anything in any example?

Ideally, there’d be an “Introduction to Macros” in the Racket Reference (inserted before 12.1), that gives a nice, simple overview of how Racket does macros. It’d discuss how the reader parses syntax objects, and calls functions at runtime, which pattern match against rules to transform one syntax object into another. And there’d be a couple trivial examples that show creating a syntax object with syntax, then #', and then parsing from a string using the reader. Finally, it’d show using syntax-case (outside of define-syntax) to transform one syntax object into another and evaluate it.

Next, you’d take the user into syntax-rules briefly, explaining how it’s just wrapping a syntax-case in a function, removing a littler boiler-plate. Then show how define-syntax does the same thing, but binds the resulting function and adds it to the reader (I know the proper terms for this in CL, but not in Racket).

Finally, at this point, the programmer should have a pretty solid basis from which to start building more knowledge:

• Discuss the difficulties of syntax-case and how syntax-parse can be better. • Discuss for syntax macros briefly. • Discuss syntax quasi quoting, splicing, and give an example or two of where this is helpful (this is a big one for CL’ers who are used to backquoting and splicing everything in macros, but that’s not usually needed in Scheme). • Discuss how there are a myriad of other types of syntax macros like define-provide-syntax and splicing-let-syntax and make-set!-transformer. None of these extra “discuss…” need to be more than a sentence or two basically informing the programmer that there is a lot more than the basics of what has been shown and linking to the appropriate place in the docs.

My 2p.

It’s certainly possible to write macros like for in such a way that you don’t need a separate thing like define-sequence-syntax, using @lexi.lambda’s approach or just writing your macros in a certain way (a sort of cps).

You could also imagine a macro system in which certain macros were expanded early, so that their expansion happened before for.

If you need a macro to expand with its own definition context, is there a better/cleaner way than this? (let () (define <stuff>) (define <morestuff>) exprs ...)

I think a lot of this is covered in the Macro chapter in the Guide: https://docs.racket-lang.org/guide/macros.html\|https://docs.racket-lang.org/guide/macros.html Perhaps you could submit an issue to start discussion on how it could be improved.

Not really I think, but checkout block

Also useful https://docs.racket-lang.org/syntax-parse-example/index.html

sounds like setf and (defmethod (setf foo) ...) might be another example along these lines

(I think that one’s Common Lisp. I don’t know much about either Common Lisp or Emacs Lisp.)

I ended up with a kind of syntax zipper in Fexpress, and CPS macros remind me of it.

Anyhow, these syntactic continuations have layers like call stack continuations do. If you only want to sequence-macroexpand something when you know it’s supposed to be a sequence term, and if you use the presence of the surrounding for to conclude that something is supposed to be a sequence term, then you’ve potentially learned other things from the for as well. So the layer(s) of syntactic continuation you built up by traversing into the for aren’t entirely unsemantic, and their type may depend on for. Whatever you call that type, that’s the name that will line up with the name define-sequence-syntax when you compare the approaches. I don’t think there’s a reduction in the number of entities in the system.

But there is an inversion of control, and I think both directions are valuable. The Racket for macro has a certain kind of optimization it’s trying to do, and it gives sequence term operations the ability to help it out with that by implementing a certain interface. On the other hand, Fexpress’s clambda has a certain kind of optimization it’s trying to do, and it needs sufficient help from its context to do that. So whether to introduce a new DSL macroexpander or to introduce a new type of syntactic continuation frame is probably a question of which part of the syntax has most of a plan put together and which part of the syntax has the missing information needed to complete that plan.

The inversion could flip around with another inversion if the other side has its own elaborate plan for how to supply the information. (It could flip around back and forth.) I think what samth said about a “system in which certain macros […] expansion happened before for” is one of these cases. The first round of that would likely be rather light on semantics, just crawling deeply over an s-expression looking for escape sequences the same way syntax-parse looks for ~ or quasiquote looks for unquote. In that case, the syntactic continuation assembled by that traversal would indeed practically be an s-expression zipper. I think that accurately represents the information the traversal paid attention to, so the more semantic and less semantic options are on more of a spectrum of choices than a dichotomy.

Oh, now that I read the conversation again…

If you want to extend for in another way it’s not prepared for, then you’re stuck with it not cooperating and thus not letting you learn much about its semantics. You might need a DSL that lets users insert that information somewhere themselves, likely by putting escape sequences in the relevant parts of the program, putting a signature declaration somewhere to supplement for’s forthcomingness with semantics, or just migrating to a new definition of for.