I’ve found virtual connections often help in scenarios I don’t even realize I have yet. :slightly_smiling_face:

Seriously: I like any approach that helps with the thankless !@#$-ery of determining exactly in what order/when/how to start/stop various stateful things.

I know, 99% perspiration 1% inspiration, but dang, I want to get to the part where my app does something.

> determining exactly in what order/when/how to start/stop various stateful things. My approach (which I’ve shamelessly stolen from the Clojure community) is to be extremely explicit about this:

https://docs.racket-lang.org/component/index.html#%28part._.Guide%29

Stateful things have explicit start/stop behaviour. You specify what the dependencies are between things and the system figures out in which order they should be started/stopped.

I mean, I like both extremes. If there is a reliable way (e.g. virtual connections) to handle init/lifetimes/term from dependencies already expressed in Racket code, that’s wonderful. But if there’s not, yes, I like getting super explicit about it.

I think the worst case is in the middle. Things like (define worker (thread ___)) as a module level expression? Maybe OK in a tiny app. Often trouble as things grow. Better put in a start-worker function.

At least that’s my experience.

@mflatt When compiled, are linklets annotated to be pure functional or have side effects, or something of the sort?

Yup. Same here

No. There is often information about individual exports of the linklet, but not the linklet as a whole.

That’s probably even better. Does that mean that if a function foo(proc) takes as argument a procedure, and on one particular call foo(add1) of the function it happens that the input procedure is a pure function (add1), then the call is marked as ‘pure’?

No, I don’t think that can happen. If foo(add1) is inlined, then maybe the result is detected as pure. But the annotations are definitely not as rich as “this function foo is pure if it receives a pure argument”. More to the point, I don’t think that purity is something that is tracked by any of the compilers right now. The current properties are simpler things, like “is a function that returns a single value” or “is a struct field accessor”.

I see. In principle, then, does purity propagate well, or is it a much hairier problem than that?

It seems that check syntax knows something about it too.

Purity propagates well in the sense that a function that calls a known-pure function can be detected as pure. It’s the higher-order part of foo(add1) that makes things much harder — that is, parameterizing things over purity. I don’t think Check Syntax knows anything about purity, but maybe I misunderstand what you mean.

The schemify layer infers purrity to some degree, but that’s linklet-local. Ditto for the linklet-flattener, where purity is helpful for pruning unused code.

Check-syntax knows if an identifier is ever set!’ed, hence my stretch to purity.

Ah, ok. That’s much simpler than tracking and propagating information about functions.

Ok, so there is still some knowledge about purity then.

and it is used to some extent. That’s cool.

Is there a mechanism for combining together multiple profiler outputs into one?

Right now we run Herbie on a few hundred benchmarks, and have some custom-built aggregation, but it doesn’t extend to the level of individual function calls

It would be good to know how much time Herbie spends, say, evaluating candidate programs

There would also be the possibility of making the profiles finder-grained if aggregation were possible (run a profile per-phase)

@pavpanchekha there’s not a mechanism that exists currently, but the raw data should be relatively easy to simply combine

Is there a way to save profile data to disk and read it back in a structured format?

I can definitely just dump the runtimes of each function but the caller/callee pointers are sometimes handy

I posted a question about a database connection management issue I was having earlier, and as is typical with Racket: 1) A fantastic solution already existed (virtual connections), and 2) A member of the Racket community (thanks @philip.mcgrath) pointed me in the right direction w/in an hour.

I’ve been programming professionally for over thirty years in over eight languages, and I’ve never been as pleased with a language, and it’s community, as I have been w/ Racket.

Just to report back, it only took a few minutes to switch to virtual connections, and everything worked perfectly the first time. Very happy!

@pavpanchekha you can either combine profile? results, or you can combine the (undocumented) sampler output. I don’t know that either one has a trivially-serializable form, but at least profile? values should be easy to serialize.

Also, adding things to the library itself to serialize/combine these values would be really nice.

Ok, I’ll look into doing that

Combining the values is a first step, but we also would need to trasmit them across Places

I have a style question. Is it a fine thing to do to export internal functions through submodules for the purpose of writing tests?

The internal functions have some mutual dependency and it is not easy to pull them into a separate module

@shu—hung I think parts of the contract system do this, at least I remember doing it when I was implementing collapsible contracts and writing the test cases

Gotcha. Good to know that this is not a weird thing to do

A strategy that I have used is to put all implementations into a private directory and use (provide (all-defined-out)) in all of the private modules, and then have the public contracted API reprovided from a different module.

Right! I forgot about this approach

@samth looks like profile structs can’t be sent from place to place; is there any ugly hack to change that (it’s a graph structure, of serializable things) or should I just write my own?

I would just change the library to make them either transparent or serializable

@mflatt Should the following code error:

#lang racket (foo) (define-syntax foo (syntax-rules () [(_) 42]))

Which seems odd given that this code works:

(module foo '#%kernel (#%require (for-syntax racket/base)) (foo) (define-syntaxes (foo) (syntax-rules () [(_) 42])))

Not ideal, but it’s not clear how to do better. The difference is in the #%module-begins. You can make the former work with an explicit #%expression wrapper, which avoids the partial expansion in #%module-begin that creates the difference.

A define-expression-syntax form that automatically added the #%expression wrapper might help. Inserting those wrappers manually is kind of a pain because you have to delay parsing, so you have to make two macros and have one expand to the other inside the wrapper.

any idea why a free-id-table lookup would fail when the debug-scopes package shows matching scopes? I’m looking for this: "n⁰˙˙³ˢˡⁱ⁼⁴⁺ᵘˢᵉ⁼" in a free-id-table with these keys: ("n⁰˙˙³ˢˡⁱ⁼⁴⁺ᵘˢᵉ⁼")

oh wow, free-identifier=? returns #true now I’m really confused

okay I reduced the problem; the lookup fails when there is an outer -let in the code below, but succeeds if there isn’t #lang racket/base (require (for-syntax racket/base syntax/parse syntax/id-table)) (define-for-syntax tbl (make-free-id-table)) (define-for-syntax (lookup n) (free-id-table-ref tbl n #f)) (define-syntax (-define stx) (syntax-parse stx [(_ name:id e) (free-id-table-set! tbl #'name 42) #'(define name e)])) (define-syntax (-let stx) (syntax-parse stx [(_ () e ...) #'(let () e ...)] [(_ ((name:id e0)) e1) (unless (lookup #'e0) (raise-user-error 'not-found)) #'(void)])) ;; OK ;(-define n 3) ;(-let ((m n)) ; (void)) ;; expands to `error: not found` (-let () (-define n 3) (-let ((m n)) (void)))

The identifier really shouldn’t be found, because the binding of the identifer inserted into the table is different (at the time that it is inserted) than the binding at of the identifier that is looked up. The inserted identifier (well, the syntax-local-introduce of that identifier) doesn’t acquire a binding until after the define is expanded. One possible solution to reorder binding and insertion into the table: (define-syntax (-define stx) (syntax-parse stx [(_ name:id e) #'(begin (define name e) (define-syntaxes () (begin (free-id-table-set! tbl #'name 42) (values))))])) That the original -define works at the module level is essentially a hashing accident; it’s like mutating a key used in a hash table, where the mutated key happens to be found because the original and mutated variant have the same hash code. (There are implementation details for free-id tables that make this particular hashing accident consistent, though.)