That’s pretty neat, I started to fiddle around with using replace-evt and handle-evt to see if I could figure out a threadless version but wasn’t able to make it work. I see I was missing choice-evt. This is quite fun exercise.

@abmclin I love events — faster than threads for IO-bound concurrency. There’s a thread-friendly “cooperative” multitasking DSL or two in them somewhere, I’m sure.

With a more convenient arrangement, they’re easier to work with. I’ve started collecting event-based primitives in another project, documented here (http://docs.racket-lang.org/neuron/Control_Your_Resources.html#%28def._%28%28lib._neuron%2Fconcurrency..rkt%29._evt-set%29%29)

The reset of the procedures in that section are implemented as events, threads, or both.

Neuron is (at the moment) a dataflow networking library, where thread-like “processes” can exchange messages through synchronous channels.

@dedbox what is the end goal of the Neuron?

@githree roughly, to turn the Internet into a giant artificial brain.

Right now, it’s just a dataflow networking library on top of a custom concurrency model

I have apps working as PoCs driving a multiplayer game server back end and small-scale web app stuff.

@dedbox interesting what problem does Neuron address?

That’s a great question

I think the Web has become too dependent on massive centralized services and want to level the playing field, so to speak, without having to fight big companies head on

I think society has problems related to these circumstances and want to provide healthier alternatives to toxic echo chambers and stuff like that

I like it even more now :wink:

Can you give an ultimate use case for Neuron

That’s a tough one

It’s a toolkit for growing software ecosystems, like Racket and its package manager and all the communications tools we use to coordinate and share.

sorry :wink:

It’s all about building communities online by leveraging real world relationships

Right now, it’s just concurrency and some distributed computing

The next big step is to add capability-based security

thanks for your time - I’ve found neuron on the pkgs website but wasn’t sure what exactly is the purpose of it - it is much clearer now

Cool, thanks for asking. It’s good practice :smiley:

love the idea, will keep an eye on the project!

@dedbox thanks for the reference to Neuron, it looks interesting. I’m encouraged to study concurrency more extensively. I don’t have a good grasp of the implications of “faster than threads for IO-bound concurrency.” To me concurrency has always had that notorious reputation of being difficult to reason about so it’ll be a good exercise for me to use Racket’s events to solve classical concurrency problems.

looks like Neuron’s been inspired by the pi-calculus, is that a fair assessment?

@abmclin yes, pi calculus also uses synchronous channels

and the dual give/take emit/recv are modeled on pi calculus channel operations

You know, in Neuron’s control flow code, there’s no obvious correlation between events and IO performance. For example, a bridge, which forwards values between two processes, isfaster as a composite event than a collection of threads. A proxy, which does roughly the same thing for a single process, is faster as threads.

my question is how is bridge faster than using a collection of threads? Is it due to overhead of threads or because using events basically is as fast as copy between two memory locations?

I wish I knew how to answer questions like that easily

I had been under the impression that threads are very cheap in Racket

Yeah, and processes are thread + kitchen sink, but sometimes using processes is faster than using threads.

interesting, I find that surprising, though I don’t really have a good intuition yet when Racket threads and concurrency are involved

My totally un-educated guess is some sort of compile-time optimizations

ok

were you using timing measurements to compare the performance of say bridge versus threads

am just curious about how one would go around measuring this kind of stuff

I’d implement using threads first, then raco setup -D neuron && (time raco test control.rkt <three times>) and eyeball the average

Then, repeat with events only

Sometimes, try a combination.

makes sense, ok cool

Threaded code is much nicer to look at

as opposed to using only or mostly events for concurrency?

Yes, events have more syntactic noise because you’re using a continuation-passing style.

ok understood

is Neuron still experimental or a stable alpha package?

Been struggling with that. I think the package is too big.

Part 1 is stable alpha, Part 2 design is stablizing now.

Parts 3 and 4 are vaporware

I noticed that based on how much documentation had been filled out for each of the parts

What you see so far is about one Masters thesis worth ofwork

while I’ve only given it a cursory glance, the package doesn’t seem that big to me, but I’ll defer to your judgement

seems like a nice piece of work, I won’t try to worry too much about trying to perfect it or you’ll never be able to be at peace :sob:

haha learning that the hard way

might be better to let it gain more real-world experience to help guide future improvements

The project stalled until I watched @notjack start the net2 package.

Any more advice on how to think about sharing this would be a huge help

if you feel all the relevant procedures and operations are there and ready to be used in real-world applications then I’d say the first version is ready

part 3 seems rather important since I can see this being useful for types of applications that need to spawn asynchronous processes such as email notifications, background processing, updates, etc

so would be nice to see how Part 3 could help with those kind of scenarios

Ok, good to know.

I’m also thinking of gitlab which basically is a large distributed system of Ruby processes handling various operations such as building, testing, listening and receiving git client connections, monitoring programs, and admin functionality

most of it is implemented in propriety licensed Ruby libraries so not really available for open-source without paying up

I’m thinking of how the Neuron abstractions could serve similar purposes while I’m studying the package docs

the only reason I’m even thinking about gitlab is because I spent the last month setting up and configuring a local gitlab instance for my use so was poking around the innards

Cool! I’ve never peeked inside Gitlab.

It seemed complicated.

it is! but it does its job very well

Neuron is about ready for custom back-end data pipelines. I’ve used similar designs for artificial intelligence and machine learning applications. That could be a v1.

hello, neuron is cool :wave:

why not? Seems you already have use cases to showcase Neuron, why not create small-scale examples of AI and ML apps using Neuron. That would be a good way to eat your own dog food.

@notjack hello!

Then maybe v1 should include HTTP/1.1 endpoints

are you thinking of incorporating Racket’s networking support or something from scratch?

From scratch. I actually had them working on an older prototype.

and someday there will be http2 in racket :D

Yes, net2 will be a drop-in performance upgrade.

that’s neat

@abmclin by networking support, you mean the net library?

yes that’s the library I’m thinking of, it has http functions and etc, seems a shame to reinvent the wheel if you could avoid that

It didn’t work out so well with the Neuron paradigm.

HTTP has changed drastically in the time since the net implementation was written, so it could use some reinventing

yeah such as http2

more than that though, the way http1 is used has changed a lot

JSON web APIs (and even XML ones) weren’t nearly so common then

right that is true, the use cases have changed a lot

it’s all about user-side interactivity with JSON/XML snippets being passed back and forth

plus the http/1.1 spec has been updated and some behaviors in the net implementation are now highly discouraged

ahh I hadn’t been aware of that, would you mind giving an example or two of the discouraged behaviour?

also I really don’t like fiddling with headers as : joined bytestrings directly

yeah I won’t like to do that either

This used to be legal:

Foo: value1
value2
value3
Bar: whatever

Header lines that don’t contain colons were considered continuations of the previous header line

that behavior was deprecated in the updated HTTP/1.1 RFCs (7230 through 7235)

I’m following you so far

I probably should review the RFCs, it’s been a long since I’ve done that

the reason that behavior is now deprecated is because the previously legal format is too error prone?

I think so, I’m not positive on the specifics

I do know that all headers except Set-Cookie are defined to allow “splicing” values together with commas

ok, I rarely ever need to work with headers directly in my work so I’m not sure either

Foo: value1\nFoo:value2 is equivalent to Foo: value1,value2

ahh that does look awkward

I can see how allowing multiple legal variants can lead to ambiguity, especially when different browser implementations are available in the wild

there’s also stuff the current racket http implementations encourage around connections that can lead to issues

for instance, opening multiple TCP connections to the same origin is very rarely a good idea

simultaneously, I mean

yeah that’s definitely a no-no most of the time

I’ll be looking forward to seeing your new net2 library

it’s slow going since I won’t have much free time for a while, but I’m looking forward to it too :)

always appreciate your hard work one way or another :slightly_smiling_face:

happy to hear it’s been useful :thumbsup:

@dedbox from the conversation above - are you saying Neuron could already be used for let’s say some ETL process?

If you’re writing your own interop layer, then sure.

Out of the box, Neuron instances can exchange messages over the network using s-exp or JSON.

cool, looking forward to see some sample application as I feel I may have found a use case to play with it in my project

Cool, thanks! I’ll announce any new developments here.