@spdegabrielle they also had assoc lists ;)

@robby What about enriching the lexers with a function get-sub-type that by default just returns the type, but could be overloaded in racket to check characters and return 'line-comment for example?

I’m thinking: Each lexer module requires "lexer-base.rkt" which contains a dict, and each lexer registers in this dict a function get-sub-type (or something) that does what’s described above. The module-lexer is a little special and would need to update this function depending on the raw-lexer.

The bonus is that this dict would then be extensible, so would allow for future augments of the interface.

I’m open to criticisms and suggestions of course. I’m just trying to find a way to solve this issue.

@laurent.orseau I was thinking of hash tables

@laurent.orseau I still don’t understand the problem. Why do you need the lexer to tell you the difference between line comments and bracketed comments? In reply to earlier comments, yes to your bullet 1. That’s what I In reply to your bullet 2. I don’t yet understand why you need this information. As for 3. I think I need more help to understand why you are making these changes and why the current situation isn’t working.

Ok, let me reboot then

If you are still fixing the automatic comments thing, I don’ tunderstand why you don’t just ask the colorer whether or not the character is a comment.

(you may have to insert and then remove it)

Consider this case, where @ indicates the position of the cursor (define (get-dict) '((a . b) (c . d) (e . f) ; blah blah (blah @ )) The colorer classifies this position as 'white-space because the next character is #\newline. Thus if you press ), smart-skip skips to the next paren.

You could instead check the previous position, which would be inside the comment, but then that breaks this case: (define (get-dict) '((a . b) (c . d) (e . f) #\|blah blah\|#@ )) where here skipping is the right behaviour, but the colorer returns the same information as in the previous case, so we can’t differentiate the two cases.

Yes, the colorer only colors what’s there

So if you want to know what color something would be, you have to insert it

At least, I think that’s how things are supposed to work!

But smart-skip precisely prevents this insertion

So you say: insert ( when typed, look at the color, erase it, then skip it if it is indeed a comment? That sounds rather suboptimal, no?

That would trigger a whole lexing/coloring phase just for checking whether we are inside a comment… when the lexer should already know that :confused:

It sounds a lot less suboptimal than changing the lexer interface

changing the lexer interface sounds extremely horrible.

I dont’ think inserting it and checking on it is that bad.

(that bad overall, I mean)

What concretely is the problem? I can see how you wouldn’t like the way the code looks, but are there other problems?

Efficiency. DrRacket already has issues in this space, I don’t want to make it worse just for a paren

I have to ask: have you measured anything?

Plus, really, if it’s racket code, I just need to check whether the character at the beginning of the comment is ; . This is all I need, but I just need to know whether it’s racket code or something else

That’s not how drracket works.

From DrRacket’s perspective “it is just racket code” isn’t the right way to think about it

DrRacket asks the language “what kind of thing is this?” and the language says

I understand that now, hence my proposal for a more general approach

It might be #lang scribble or racket or whatever

You have cited efficiency as a concern

but you haven’t shown anytthing that’s slow

Please do that first.

You mean for the case at hand or in general?

I want to be very clear here:

changing the lexer interface is a huge undertaking

We do not even know the number of people that have code that depend on it.

And the ones that we do know are non trivial to change

Further more, there are serious concerns with the proposed API

We need to keep the interface simple and work with the information it gives us

I was planning on making it completely backward compat., just augmenting it. What exists should work just as fine.

But I understand the need for simplicity too

or else we are pushing too much work into the language implementation when it can be done once and for all inside drracket

Your suggestion above is not backwards compatible unless you count “does not work with the new feature” as backwards compatible. While I agree that one could think that way, I would prefer to make things work without having to change every language and I don’t see that extending the interface has that property.

re: efficiency:

I would prefer that too, but the fix you propose is also DrRacket specific, no?

Hm.

I thought we were talking about DrRacket?

At least I give the mean to do it right for other people using the colorer and lexers elsewhere

Okay, lets talk more about this later on. I’ll come back and read your comments more but I can’t do this online talking thing for so long anymore.

sure

sorry to be a downer

In any case, I’m still looking for whatever is best, and that you’re happy with. I’ll give insert/delete a shot (but it badly hurts my intuition :stuck_out_tongue: )

(at least it’s a solution nonetheless)

1. The culprit code is in framework/private/racket.rkt here: https://github.com/racket/gui/blob/master/gui-lib/framework/private/racket.rkt#L517 so any change to this code would be racket-specific anyway

2. I just discovered that the code in racket.rkt already checks for the type of comment in an ad-hoc way, for example here: https://github.com/racket/gui/blob/master/gui-lib/framework/private/racket.rkt#L1656

In light of this, I think my first proposal was fine: https://racket.slack.com/archives/C8FS3SS22/p1583743470019900

Thank you, @laurent.orseau! I do agree that there are possible performance issues here (especially when you need to wait for the colorer) but I think that we need to fix those instead of making the interface more complex. It is less work overall is my guess. As for the file named “racket.rkt” that is a flaw in the name of the file. It isn’t (supposed to be) racket-specific. I agree that insert-brace-pair is problematic.

But I do agree that this is not making the status quo substantially worse than it already is.

Thanks for pointing that out!

If you were to make a pull request for this, perhaps putting the results of this discussion into a comment near the undesireable code would be great

Thanks Robby, okay I’ll do that then.