Hey, everyone. It’s good to see chatter in here.

The net2 design docs are easy to read. After the concepts are flushed out and the API starts to form, I should be more helpful.

@peschkaj hi! :wave:

@dedbox which reminds me: Axon is very interesting. I want net2 to allow transports between named agents in different Racket threads, and it might be easier to design that by integrating it with Axon to prove the concept

hello, @notjack

@notjack named agents, as in actors or maybe pi calculus channels?

@dedbox named agents as in “things that can answer authoritatively for a specific URI authority”

@notjack ok, so universally named agents.

yes, so non-axon networks can communicate with axon networks and vice versa

Cool! I see a lot of overlapping concepts between net2 and axon.

agreed

I think I want net2 to be responsible for providing what’s needed for Racket concurrency / distribution abstractions to talk to the wider internet

so, URIs

I’m not planning on implementing any new sort of concurrency paradigm like axon / marketplace / syndicate

but I do want a racket program to be able to run servers implemented in axon / marketplace / syndicate all at once in the same Racket process, and I want non-racket programs to be able to talk to those servers

It’s tempting to keep plugging on code,. Maybe it’s better to document concepts first, to discuss?

I wrote some of this in the ROADMAP.md doc, described as “Racket Transport Addresses” (or something like that, not remembering off the top of my head)

more discussion here gladly welcomed

Ok, so axon gives TCP connectors and listeners, and sexp (and json and HTTP1) messengers.

A core design principle is to hoist ordinary functions to network-bound actor-like filters.

gotcha, looking over the tests I was able to see how it let you take functions and (in my basic, fuzzy understanding) turn them into unix processes with lifetimes that can send and receive their results and arguments

Close! It uses racket threads l.

right, each “process” wraps a thread that can be both gracefully and forcibly killed by other “processes”

and a process can have shutdown / termination logic

and they’re racket green threads, so super cheap and you can have a million of ’em

(I’m assuming you’re typing stuff? no rush but slack keeps changing its mind on whether you are)

Yeah, that’s pretty much the whole of it. Re-reading ROADMAP.md now.

On phone. Slack gets extra confused

gotcha

how familiar are you with http virtual hosting?

I built out a few large HTTP virtual hosting services, before “the cloud.”

perfect

so an explicit feature I want net2 to enable is for a single Racket process with a single Racket thread to offer multiple http APIs to the internet by way of virtual hosting and using racket transport addresses to locate distinct in-memory servers and communicate their locations to arbitrary internet clients

basically I want a single racket program to be able to run multiple http services / proxies / gateways etc, and I want client code in that racket program to interact with those services without having to know that they’re local so that deployment topology is transparent to the client

which means URI trickery

So, you want all agents to appear “remote,” even though some might be local?

yes - or at least, I want the interface to remote and local agents to allow changing an agent from local to remote without requiring that clients of that agent be restarted or redeployed

otherwise, distributed programming will never be simple

oops - in the part above where I said “for a single Racket process with a single Racket thread” I meant “OS thread” in the second part

That’s starting to sound like agent mobility.

but lower level

agent interface mobility?

that sounds reasonable

the broader problem I’m trying to solve here is making it easier to do things in heterogeneous distributed systems, rather than homogenous ones

You wrote that chains of ports would imact performance. So, routing occurs at the bytes level?

yes, because in a heterogenous distributed system messages are a lie

there are only bytes

go tit

*got it

in net2, communication between authorities will always occur over transports and transports only send and receive bytes

to send and receive messages, you have to build messaging protocols on top of byte transports and that is a userland responsibility, not something defined by some common runtime that all parties in the network share

because requiring that all agents in the system share a common runtime or a common definition of what bytes are what means you can’t horizontally scale, and it means you can’t communicate at all with agents outside the system that don’t share that runtime - so you end up with two notions of networking, one for “actors inside my collection of distributed erlang nodes” and one for “ugh this is how I do tcp connections to everybody else”

That’s what turned me away from ZeroMQ

The “lower” half of axon tries to solve that problem.

which bits are the lower half?

the encoder, for example

it ties a printer to an output-port

then it prints the messages it receives onto the port

I guess by “lower” I meant closer to actual byte IO

gotcha

yes, that part I imagine net2 being responsible for

(in an ideal world, Hackett would be in a position where there’d be ways to easily derive strongly typed, performant byte serializations and deserializations for message types and net2 users would just use that)

That’s good for axon.

It’s supposed to be for concurrent messaging over the network.

All of its major bugs have involved closing ports

oh! that’s good to know, because it means choosing to integrate the disposable package directly with net2 is a good idea

net2 connectors will (hopefully) manage all connection pooling and connection lifecycles

That’s the only feature from python asyncio I miss in Racket

I’ve heard of python asyncio but haven’t used it, how does it handle that?

It just makes the sockets you open play nicely with the cooperative event loop.

But it gives you a way to group sockets together and time events precisely

Racket is more of a thread soup.

End up using more semaphores.

interesting

found this part of docs: https://docs.python.org/3/library/asyncio-eventloop.html#creating-connections

heh, they even call abstract bidirectional streams “transports”

I thought so too

I mean, I thought of that doc while reading the roadmap

yeah there’s a lot of similar ideas, which is good to see

I think the main differentiator net2 will focus on is making host/port/protocol parameters suitably abstract, so we can do weird things like hook up Axon networks to the internet

also connection management responsibilities will be divided between disposables and racket custodians and explicitly controllable by those means, instead of just being something magic that the framework does for you

That would be great. I’m writing a multiplayer game server right now. The first version was in python using asyncio, as an exercise. It was not so pleasant.

ah, games will be tricky

since they often use udp and datagram-based network logic

net2 is explicitly not touching that kind of I/O, at least for the near future

That was conspicuously missing from the roadmap. Can you elaborate?

sure

the gist of it is: that level of I/O is too low for me to get anything useful done if I try to integrate it into racket

net2 will assume reliable bidirectional bytestreams exist, and build a pile of stuff on top of that

however!

if a different library implemented ways to define reliable bidi bytestream protocols on top of datagram based sockets, possibly in ways that let programs assume things you can’t assume in TCP so you can get better perf, then that library could use net2 to make transports that the rest of net2 could work on top of

and if that doesn’t work, it ought to be possible to adapt net2 codecs and messengers to work with datagram communication without serious effort

so there’s parts of net2 that could in theory work with that sort of communication, but making it a design constraint means I’d have to do a lot more work before I could make interesting things with http on top of net2

many multiplayer games that don’t have very strict latency requirements can often be made to work over TCP though, so it’s not a total loss

also, the Tokio framework in Rust is breaking a lot of new ground in this area already so I’d like to give them more time to figure these design problems out so I don’t have to :p

tonyg has done a bunch of stuff in that space too (have you seen his library for talking to ethernet directly in racket? so cool)

I thought it might fit nicely. Generalize port+address to URI. Provide encoders to chunk or truncate messages over ~64K. Do not provide UDP transport. That takes care of at least 80% of UDP use cases and keeps me in the library.

it’s definitely possible, and likely possible to do quite nicely

just not gonna spend time thinking about it yet

axon has code for that right now

the only missing Racket piece to this game server is a user-friendly linear algebra engine

the udp.rkt module I presume?

oh!

there’s a package for that I think

http://docs.racket-lang.org/glpk/index.html

raco pkg install glpk

hmm, I’m not sure if that’s for general linear algebra though

seems to only be for limited kinds of optimization problems involving linear algebra?

….it’s been a very long time since I took classes on this stuff

Cool, thanks.

@dedbox btw I peeked at your game server, specifically the sqlite querying parts. I think @royall is considering how to make sql nicer in Racket so you two might have stuff to talk about there.

Here’s the work in progress: https://github.com/dedbox/axon-game-server/blob/snapshot/server.rkt

It shows TCP and UDP in action together.

At top, it listens for UDP broadcast pings for local server discovery.

Then, it uses TCP for asset data exchange.

Finally, it uses UDP again to blast world state updates l.

This hybrid approach is convenient, reliable, and performant in the right places, and the API for both protocols is similar.

what performance are you targeting? in terms of latency between clients seeing state changes

The SQL is a necessary evil at this point. I wouldn’t mind a friendlier persistence API.

Low latency for world updates, low complexity for asset exchange and session management.

low latency as in less than a second, less than a few frames in 60fps, or less than a millisecond? ballpark guesses are fine

Single digits milliseconds.

okay, that’s entirely doable with TCP and nontrivial messages

depending on ping latency

Sure. This hybrid pattern represents a more general use case, for things like VoIP and RTSP.

gotcha

Cool, so now I can imagine how axon would use net2. Maybe I can sketch some example uses.

I’ll need to figure out and document how racket transport addresses work first, which I’ll do after I get connectors, listeners, and TCP/TLS/Unix transports implemented

I’m thinking specifically of some database conveniences along the lines of Clojure’s HugSQL. Probably a good excuse to build myself a #lang and include some conveniences like optionally using transactions for bulk writes or returning large result sets via generator.

partial markdown be damned

tbqh I don’t understand half of today’s conversation, so this will all be a nice learning experience for me

I want to finish my current project of building a knockoff of the Symfony CLI component before starting something else in earnest

It’s not a good language, but PHP’s widespread adoption means there’s lots of handy libraries for it. I’d like to see Racket have the same level of off-the-shelf convenience for common tasks.

@royall the practical upshot of the stuff talked about is that if you’ve got some racket code that needs to talk to some http API, and that API does something weird, your racket code can easily start a proxy server between you and that API and that proxy can translate the weird API into a much cleaner API. So it’s easy to make reusable standards-based HTTP client libraries that do really powerful and neat things, but you still use them with actual real-life non-standards-based HTTP services.

When you put it that way, it sounds really superb

like, HTTP auth schemes are really incredible and can be dynamically discovered and reconfigured on the fly. But fucking nobody uses them (pardon my french). So a client library that does those fancy things is useless unless it’s easy to translate between existing services like AWS or whatever the startup kids are using these days.

@royall hello!

@notjack will net2 support half-open TCP connections?

@dedbox I would like it to, though at the moment I’m relying on the kernel tcp / ip stack which doesn’t grant as much control over that

I had trouble making the client side see when the serving side closes its out-port.

when the client was already waiting to receive

Do you know if that’s expected TCP or Racket behavior?

It depends on whether the connection was closed gracefully, forcibly, or just abandoned

close-output-port

That’s a graceful close. The kernel will send a TCP FIN packet which should cause the kernel on the other end to close the connection, then in racket land you’ll get an eof? value when you try to read from the port

Assuming both sides of connection are implemented in racket?

yes

Forceful close is sending TCP RST packet, abandoned is one side just stops sending anything at all (like if network connectivity dropped) and the other side closes when it can’t receive ACKs on either sent data or heartbeat packets

With the racket/tcp module, there is no way for racket code to send anything other than a FIN packet

Only kernel is capable of RST or abandoning

Ok, good to know.