How do I expand a module until certain identifiers show up from an external tool? In a macro I can use local-expand with a stop-id list, but my tool isn’t locally expanding: it’s reading code from a file and turning it into a syntax object with read-syntax. Do I just have to reimplement the stop-id list feature of local-expand in terms of expand-syntax-once?

Can’t you call local-expand directly?

No, that would only work from inside the code. My tool is outside it.

It raises the error local-expand: not currently expanding

Tricky.

Digging through the macro-debugger source I found it uses an undocumented magic parameter called current-expand-observe defined in the primitive '#%expobs module.

I wonder what happens if I just… poke at that

I think, I have seen that somewhere. Perhaps in some of Ryan’s writings?

I’m guessing it’s some sort of hook he added to the expander to report information to the macro stepper

Wow, cool

So the reason I asked about the above is: I’m making a refactoring tool. And I just got it working.

How deep is your partial evaluator going to get?

It fully expands the code

Oh wait

you mean constant folding? heh :p

probably not very deep, I’m more interested in things like rewriting let forms to define forms

Ah, I see

It’s using syntax-parse too, so rules can use syntax classes and pattern directives to match things

That’s awesome!! Did you consider nanopass also? https://pkgs.racket-lang.org/package/nanopass

It technically supports any function from syntax? to (option/c syntax?)

so if nanopass can be bridged into that shape it should work

Thank you so much @ryanc for creating current-expand-observe, hopefully I’m not doing something too awful with it: (define (expansion-events stx) (define current-expand-observe (dynamic-require ''#%expobs 'current-expand-observe)) (define events '()) (define (add-event! sig val) (set! events (cons (expansion-event sig val) events))) (parameterize ([current-expand-observe add-event!]) (expand stx)) (reverse events))

I meant: It seems to be doing something similar to nanopass (or I misunderstand the objective), but maybe nanopass does not do what you want?

Or if I misunderstand, mind to give a quick comparison?

I think the problem with nanopass is that you need a well-defined grammar to work with. But for unexpanded code, there’s no grammar (or rather, the grammar is not useful). You can’t give semantics to any piece of code at all

It’s not doing any of the compiler stuff itself. It’s just calling racket’s read-syntax and expand functions, and listening to a bunch of messages that the macro stepper integration spits out.

@sorawee so you mean that jack’s tool is much more flexible, because it’s case-by-case in a sense?

@notjack but your example, and ‘refactoring’, suggest similar goals, no?

although it could be much more general I guess

@notjack Cool idea. Now we need to collect refactoring ideas for you.

My understanding is that the problem is that nanopass doesn’t integrate with the racket macro expander. It’s not for creating embedded DSLs that use local-expand.

@soegaard2 Throw ’em at me. Currently I’m working on a bunch of ones that rewrite let to define.

ok, that make sense, thanks

Can’t it do reindenting too? (let ([a 42] [foobarbax 32]) a-body) ==> (let ([a 42] [foobarbax 32]) a-body)

So now I’m seeing that your tool could be used to entirely replace a racket->racket compiler, is that correct? Although it applies only before expansion, but it could well be applied as well after expansion

@laurent.orseau Kind of.

The way to make sense of any Racket code is to fully expand it first, so that you have a fully expanded code. But fully expanded code is too radically different from the surface syntax, so IIUC @notjack tries to find a middle ground where the code is expanded enough to reveal its meaning, but not too much to deform the program.

Actually my program fully expands it. However, the macro expander also logs each expansion step as it expands things using that current-expand-observe function. So as the expander transforms the syntax, I’m able to add each intermediate syntax object to a list. I then filter that list for syntax-original? elements.

Oh my, maybe I’m getting a little too excited, but it seems to be the next meta-level for Racket :star-struck:

I think this doesn’t quite work, because syntax-original? can return true for a syntax object whose outer shape is original, but whose inner contents were inserted by a macro. I could maybe do better by checking that the syntax object is deeply original (original and every subsyntax object is original) and that the syntax-source corresponds to the file I’m refactoring.

Why do you need to filter syntax-original elements?

oh, because that means they are what’s written in the text file?

right, exactly

so if you observe a form that contains a non-original element, that means it’s too late.

I want to refactor source code in the file, not generated code cobbled together from eighteen different macros

I guess you should mark your own refactors as syntax-original too though?

If you look at that screenshot of the sig = … val = … I shared earlier, look at the cases where it says sig = visit

e-graph is the solution to everything :slightly_smiling_face:

what’s e-graph?

@notjack they’re all the same. Should I pay attention to something in particular?

@notjack I don’t know if this is useful or not for your purpose, but maybe? https://docs.racket-lang.org/progedit/index.html

@laurent.orseau The current-expand-observe emits at least one visit event for every single syntax object in your program, including syntax objects created during macro expansion. I limited it to the first few there but basically the expander visited that (if …) expression three times. It also emits visit events for all of the subexpressions, as well as for a bunch of expressions that look like (#%plain-module-begin …) like you would see in the macro stepper in drracket if you turned off all macro hiding.

@soegaard2 yup I’m gonna do something similar-ish to that

@soegaard2 is that what raco uses or —fix-pkg-deps?

@soegaard2 also I didn’t see your question earlier about indentation and formatting: this tool doesn’t handle that at all, it pretty much just relies on syntax->string from the syntax/to-string module.

@laurent.orseau Don’t know.

@notjack Ok.

@notjack So if you’re writing something like progedit that can work with syntax-classes and syntax-parse, that’s indeed awesome :slightly_smiling_face:

yes :grin:

Some refactoring rules I just got (mostly) working:

(define-refactoring-rule let-to-block #:literals (let) [(let ([x:id rhs:expr] ...) body:expr ...) (block (define x rhs) ... (block body ...))]) (define-refactoring-rule single-block-elimination #:literals (block) [(block expr) expr]) (define-refactoring-rule immediate-define-block-elimination #:literals (define block) [(define header:function-header (block body:expr ...)) (define header body ...)])

BEHOLD!

Nifty!

time for bed!

Nice! Maybe the source-replacement could also keep track of the rule that was applied?

That would help with debugging, one could also keep usage stats for getting rid of rules that are almost never used (imagine you have thousands of rules), or it could be used also for writing correctness proofs

probably a good idea to track the file the replacement is applied to also.

Yeah there’s lots of stuff like that I’m going to improve

It should make it fairly easy to help the racket compiler with many ad-hoc rules. One could even add rules at the top of their own files if they know special cases that the compiler can’t see.

I thought it’s time for bed :stuck_out_tongue:

@laurent.orseau see https://www.cs.cornell.edu/~ross/publications/eqsat/

and more recently https://egraphs-good.github.io/

Have you seen sexp-rewrite by @ryanc?

https://github.com/rmculpepper/sexp-rewrite (GPL3, and in elisp)

Sometimes I get irritated by apples/oranges name pairs.

So I go read some some point-free style code.

And I feel fortunate to have any names at all. :)

@spdegabrielle (or anyone who edits the wiki): is there a wiki page that lists Racket support in non English languages? There’s an open issue (https://github.com/racket/racket/issues/1902) that announces https://github.com/OnRoadZy/RackGuideInChinese. However, wiki is more suitable for this kind of information. We should add it there, and close the issue.

Yup. I like it. Main difference between that and my thing is that my thing’s rewrite rules operate on syntax objects that have their binding structure available, so it knows not to rewrite stuff if you shadow let with your own let form. It also means rewrite rules have access to syntax-local-value, so you could make type-aware rewrites for a DSL implemented in the Type Systems as Macros style.

@sorawee added to https://github.com/racket/racket/wiki and announced on the Racket Taiwan Discord at https://discord.gg/xpwzAcx - it accepts several languages: English, Chinese, and many languages in Taiwan!

I just pushed a complicated change to the readline binding (really editline) to work with CS’s requirement that callbacks are atomic.

The v8.0 release will have a much simpler change, which is the same as what was in place for a while: make the callbacks non-atomic after all.

The drawback of the simpler change is that it’s non-composable trick — it won’t cooperate with anything else that tries to use the same trick concurrently. It’s possible that using the simple trick is the right idea, though, and readline should just get special treatment. The complicated approach puts each readline call in a separate OS thread, which might have other problems, especially related to Ctl-C. So, watch out for trouble if you build from Git or use snapshots, and if there are too many problems, then the non-composable trick will probably be the way to go.

Thanks! :slightly_smiling_face: