Hm, so, I think this is probably a very basic question, but I’m not sure.

I want my tokenizer to use “lexer-src-pos” normally, for debugging purposes, but to just use “lexer” when I tell it to, because that way, test cases are simpler to write. Using an optional "#:simpler" argument with a lambda form seems like the way to tell the program to distinguish the cases, but I’m fuzzy about how to proceed next. I’ve gotten the impression that lisp languages are more flexible about modifying code, so there’s probably a way that I can just replace “lexer-src-pos” with an expression that chooses dynamically what function to use in that position? How should I in the simplest way make the function switch between using “lexer” and “lexer-src-pos”?

(define debug? #t) (define lex (if debug? lexer-src-pos lexer)) (lex "an interesting string")

But … why not use lexer-src-pos always?

Well, lexer-src-pos gives more data, unnecessary for checking test cases. I suppose I could make something to cut away the redundant data, but it seemed easier to fix it at the source - and also it’s an opportunity to get more familiar with coding Racket in that way.

Hm, this doesn’t seem to work. Says “bad syntax in lexer”. I made a test case work just fine with this syntax, switching between “print” and “println” on command, so I think it’s something about the lexer that does it. #lang br/quicklang (require brag/support) (define test (lambda (text #:break? [break? #f]) (define customprint (if break? println print)) (define lex (if break? lexer lexer-src-pos)) (customprint text) (customprint text) )) (test "hi") (test "hi" #:break? #t)

Ah. I misrembered, I thought lexer and lexer-src-pos were functions, but they are not.

How about: (define lex/src (lexer-src-pos ...)) (define (lex in) (position-token-token (lex/src in)) Then lex/src and lex are functions, and now (define my-lex (if break? lex lex/src)) will work.

I’ll try it out. Thanks!

There’s a regexp replace quickscript in the quickscript-extra package, and one predefined template does just that

Need to say that it replaces within the selected text only

Hm, looks like it objects to the ellipsis. #lang br/quicklang (require brag/support) (define test (lambda (text #:break? [break? #f]) (define customprint (if break? println print)) (define lex/src (lexer-src-pos ...)) (define (lex in) (position-token-token (lex/src in))) (define my-lex (if break? lex lex/src)) (customprint text) (customprint text) )) unsaved editor:6:33: lexer-src-pos: not a regular expression / action pair at: … in: (lexer-src-pos …) #(175 3)

The ellipsis is a placeholder for your grammar rules.

ah, OK

The idea is to make one lexer lex/src that produces tokens with source locations. And then make a new lexer lex that calls lex/src and just throws away the source locations.

In this way you only need your grammar rules in one place.

Otherwise for this particular task it’s also very simple to write your own quickscript, since the default input of the script is the selected text and the output is the replacement string, so it’s just a matter of calling string-trim. Edit: see code below

This one almost worked! …maybe I should just edit the output. Seems easier at this point. If this new wrinkle is fixable, though, I will have learned something new and interesting.

OK, so the new problem is that the relevant code piece goes: (apply-tokenizer-maker make-tokenizer script1) where make-tokenizer is the function I just edited. So I’m just feeding the function to the program that calls it, and there’s nowhere I can put the optional parameter I spent all this time putting in…

So, is there a thing I can do where I tell apply-tokenizer “oh, and when you call make-tokenizer, also give it this parameter”? (In the program language I’m writing, I’ll definitely make a point of adding that functionality…)

You can use parameters

See make-parameter and parameterize

thanks, I’ll check it out!

https://docs.racket-lang.org/guide/parameterize.html

@elyandarin alternatively, you can curry your make-tokenizer .

(define (make-tokenizer option) (lambda (... original-arguments ...) ... do-something-with-option ...)) ;; equivalently, you can use a short-hand form: (define ((make-tokenizer option) ... original-arguments ...) ... do-something-with-option ...) (apply-tokenizer-maker (make-tokenizer your-option-here) script1)

I’ve normally just use the lexer with all of the source information, but just don’t test that part.

I use check-match (https://docs.racket-lang.org/rackunit/api.html?q=check-match#%28form._%28%28lib._rackunit%2Fmain..rkt%29._check-match%29%29)

and just use _ for the parts I don’t care about

Thanks

Is there a reason ~? should be restricted to one or two expressions? Regardless of that answer, is there a good way to currently express the higher-arity version (without unrolling it) : I’m exposing a restricted macro system and the higher-arity version would cover a lot of the intended case-based behaviour (without, for example, exposing and thus parenthesis grouping multiple pattern-template clauses).

@gfb That’s a neat idea. Could you describe your restricted macro system more?

@notjack It’s probably not particularly exciting out-of-context. I’ve made a teaching language for an introductory course, lean heavily on algebraic stepping with some custom algebraic rules, and on the last day would like talk about syntactic sugar (mainly to get them halfway to other languages, e.g. loops, python list comprehensions, R tidyverse piping, etc). So I’ve got a little macro called shorthand wrapping define-simple-macro, export a wrapper around expand-once to see that as us adding an algebraic rule that happens at compile-time, export a pattern-expander that’s a hybrid of ~or* / ~optional / ~seq that covers enough interesting examples without having all three independently, and export ~? but would like something close to the dual of that hybrid pattern expander. If that’s still of interest, I’ll come back and link to the details after the lecture.

In DrRacket, if I choose “Enable automatic parentheses, square brackets, and quotes”, I can select a text and type ( to put parentheses around the text. However, if I type [ , [] will replace the text. Is there a way to make the behavior similar to ( ? Also, is this considered a bug?