@notjack got some clarity on Codec API. flushing out details now.

So I’ve been wrapping my head around codec composition and inversion.

The framing codec concept is important for messengers, which have to convert messages to byte arrays for transport.

(and from)

I think framing codecs should be invertible.

because you generally want to be able to parse the things you can print, and vice versa

There are a lot of non-framing codecs, though.

Mainly because the API signature for codecs says, “a codec is any function that maps an argument onto a return value.”

But we only care about the ones that will be used with messengers. (or rather, transports, but messengers are the glue that binds them.)

So useful codecs are the ones that we can compose with framing codecs.

The composition of an invertible framing codec and a non-invertible, non-framing codec is a non-invertible framing codec, which would contradict the model.

So it’s clear that, if framing codecs are invertible, all useful codecs are invertible.

I’m still following the logic, but it’s possible that we don’t want or need messengers to compose or invert.

Or maybe something at a higher layer in the API stack.

Because an HTTP client prints requests and parses responses.

Maybe that’s not a great example because HTTP requests and responses are both HTTP messages.

But what about odd protocols like, I don’t know, an XML-to-JSON microservice or a WebSockets-enabled web server?

We could still use multiple invertible codecs and just ignore the inverses.

We could do that with a composite messenger type that can use a different codec for each direction.

so I’ve been thinking about this a bunch too and I think we can make it work by expressing codecs as invertible converters between one “high level message” type and a “segment” (lazy stream of bounded length) of low level messages, where a “message” is any value with a size (natural number with abstract meaning, not necessarily number of bytes) where that size is accessible by the codec in constant time

the most primitive message type would be a bytestring, with codecs that use them as a low-level message type being codecs that emit chunks of bytes

the “segment” idea is to allow codecs to do things in a lazily streaming parser-combinator manner but with more guarantees on when you can and can’t commit to a parse

I think the haskell trifecta library does this a bit with it’s notion of parsers producing and consuming “ropes” of values but honestly I can’t figure out much of anything from it’s docs (blasted haskellers, thinking types are good enough docs for anybody)

so codec might end up looking like this maybe:

actually I have no idea how it would look right now

hm

(also I gotta run to meeting)

K

That trifecta doc is pretty dense :sweat_smile:

But I think I get what’s going on with the sizes

another thought: the “high-level vs low-level message” thing is basically what lexing before parsing does

so composing codecs would be like nesting multiple kinds of lexing

ok, that’s an angle I can understand

e.g. bytes -> simple tokens -> more complex tokens -> parsed values

so maybe the codec interface should work in a way where you could stick a parser combinator library inside the parsing logic of a single message, but with the codec having extra logic around the parser that knows how to bound the amount of low level messages to read so the parser is applied to finite input of known size

oh! maybe this!

-- start a message read attempt with the following logic:
--   1. take a max number of elements to read and a stream
--   2. reads small number of elements in order to decide on a "segment termination predicate"
--   3. caller can now skip the elements consumed to produce the predicate
--   4. then caller consumes stream elements and hands off to parser combinator until termination predicate is true or until caller decides too many elements
startRead :: Stream s => Nat -> s a -> (Nat, Predicate a)

ok

To decode an HTTP message start line, I need to read from the current input position to the next line terminator.

right

So I might have a line decoder that I can instruct to carve itself up with parser combinators and other codecs?

hmmm

I think it might be more like a tokenization pass

bytes -> method-token + whitespace-token + target-url-token + whitespace-token + newline-token

because for each of those token types implementations are allowed to attach finite length limits, and the spec mandates minimum length support

This makes me want decoders to be able to use other decoders like an input port.

I think that makes sense

So we can build messages as a series of progressively more abstract tokens.

yes, that’s definitely what I want

also

I think it might help to think of the underlying bytes input and output as a stream of byte-chunk tokens

that would let codecs figure out segmentation / buffering / MTU logic

so what level of detail is appropriate for tokens? A TCP packet orEthernet frames? A utf8-encoded character? a 2GB binary blob? All of the above?

all of the above - a token would be the “message type” I described earlier

it’s just a value with a size of some sort

ok, so “messages” are like tokens, and codecs/messengers are like lexers. Then protocols are like parsers.

in that they produce and consume streams of tokens

hmm

not sure

alright here’s a simple and pretty practical example

json-rpc is a protocol for RPC via sending and receiving json values

it’s not too low level while having mostly simple parsing/lexing

Client sends either requests, notifications, or batches containing a mix of requests and notifications. Server sends response for each request, not necessarily in a batch (multiple responses for a single batch request), and sends nothing for notifications.

so there’s the following types involved:

• JsonRpc = Request \| Response \| Notification \| Batch \| Error
• Json = ... structured representation of json type ...
• JsonToken = LeftBracket \| RightBracket \| String \| Number \| Null \| Undefined \| Boolean \| Whitespace \| Newline \| Colon

you’d start with a Codec JsonToken ByteString - it implements the basic json token serialization and deserialization logic between one JsonToken and possibly many chunks of bytes (as would happen with a very large string value)

even though this sounds like lexing, it has parsing elements to it because correctly reading a json number can involve some pretty wild stuff

but with that codec you’d have to tell it which kind of token to read, it wouldn’t try to figure it out from the bytes it’s given

so it’s not like a parser which just figures out what it’s given

(or maybe that would be better expressed with separate codecs for each of the token types, with all codecs having the same Codec JsonToken ByteString type)

(I’m shooting from the hip at the moment)

anyway

then you do the same thing at one level higher: make a bunch of codecs of type Codec Json JsonToken

then do again for Codec JsonRpc Json

since at each of those levels, sending and receiving one value of the left type involves sending and receiving multiple values of the right type - and you can give a size to the left type value that tells you a bound on how many values of the right type you’ll send or receive

So, one codec “extracts” JsonTokens from ByteStrings without necessarily examining every byte. Another “extracts” Json objects from JsonTokens. Another “extracts” JsonRpc requests from Json objects.

Yes

And the protocol spec says which bytes to examine.

also yes

in that there’s a spec saying how to serialize json values to Unicode, then there’s UTF–8 for turning Unicode to bytes

(maybe there should be a UnicodeStr type or something in between the JsonToken and ByteString types)

Then a protocol is a set of codecs along with a set of rules on how and when to use them.

I think the parser combinator way to do this would be to use monadic bind to implement the chaining from simple types into complex types

but not doing it monadically might mean the codecs can have more control over how to combine a high level codec with a low level one

and yes, the JsonRpc protocol (version 2.0) says stuff like “one of you is the client and one is the server. only the client initiates communication. only the server sends Response or Error objects. only the client sends Request or Notification objects. either party may send Batch objects”

that sort of logic absolutely shouldn’t go inside codecs or a parser, because it’s so incredibly ad-hoc from protocol to protocol that any common framework for it will get really, really complicated

codec encapsulates the pure logic of serialization and deserialization

messenger encapsulates the logic of actually using a codec over a transport (that can be very nontrivial and involve things like stream multiplexing)

net2 abdicates responsibility of actually sending and receiving the correct messages at the correct times, since that’s protocol specific and can’t be part of net2’s generic framework

but it can provide building blocks for common patterns, like a client that only lets you send messages and wait for response messages - it wouldn’t let you read responses at any random time

I just stumbled onto a blog about HTTP/2 using words like frame and stream multiplexing. I should probably read the HTTP2 RFC.

there’s a very good overview blog post by one of the spec authors

lemme link you

hmmm maybe I’m misremembering an amalgamation of articles

here’s some of them: - https://www.mnot.net/blog/2016/04/22/ideal-http - https://www.mnot.net/blog/2014/01/30/http2_expectations

also this one on header compression: https://www.mnot.net/blog/2013/01/04/http2_header_compression

cool, thanks