I’d been thinking about the current package policy, I was wondering if it would be feasible to layer a protocol on top of the current package system. Create a package of packages, a “meta-package”, give it a canonical name, and each package in the meta-package would represent a major version with minor versions being kept within each package.

I’m not sure how feasible that idea would be and whether the current system can be hacked in that fashion.

I mean, as users of packages I think everyone agrees we want them to act like persistent data structures — only add new capabilities, never take away old ones?

As developers of packages: There’s no shame in saying, well, after making Thing, I realize it should have had a whole other design — and going on to make Other Thing.

I think there’s some shame in saying, Thing is now a shadow of its former self, and will suddenly break things. And transitively break things.

I’d love to see us stick with this idea(l) as long as we can?

But I don’t know what to do about situations like @leif trying to wrap or use something that is rampantly mutating its interface with contract-busting major versions.

Have thing1, thing2, …, thingN and then have a bonus package named thing that just reexports thingN ?

@greg, I’m sympathetic to that idea, it is frustrating to manage compatibility breaking issues and I’m interested in seeing how far we can go with one package per compatibility-respecting code , but on the other hand, it would be nice to have some notion of namespace management to handle various versions

I strongly believe that Racket would benefit from a more ordinary language package management solution, with version bounds and a constraint solver.

@soegaard2 yeah that pretty much sums up what I am talking about.

I’ve discussed a little bit about what that would take to do right with… @mflatt and @samth, IIRC?

But it’s been a little while.

Would that be possible to reexport packages through one package?

@abmclin If I recall correctly, there were a discussion on the mailing list (about the time the new package system arrived)

yes, I do recall that as well, I read all of correspondence that got generated when the new system was released, I’d have to go back and review to see if any viable ideas were proposed

FWIW, I thought about adding a metaprotocol of sorts on top of the existing package system, but since then I’ve realized it would probably be easier to just extend the package system to add more metadata to packages and features to raco pkg.

yeah, I’m in agreement, it seems it’d be easier to just extend the system, most of the ideas to date can be categorized as “adding more metadata” and have better metadata aware tools.

So, my current understanding is that there would need to be four changes to get the package system to where I want it to be. (1) Packages need to become a mapping from package name + version to package sources, instead of just package name. This can be added by adding more metadata to the existing package protocol. (2) Packages need to be able to specify version ranges in their info.rkt files. (3) raco pkg needs to be extended to do constraint solving when installing packages. (4) Racket needs support for package “sandboxes” a la Cabal sandboxes, Bundler environments, or Python virtualenv, so that multiple versions of a package can be installed on the same machine without breaking everything.

None of those things are anywhere near new research, but they are still a lot of work.

Should there also be Scribble enhancements — so that as well as “added in version x” you can say “taken away in version x” or “gratuitously changed in version x”? I am mostly but not entirely joking. :slightly_smiling_face:

I think so! Having better tooling support for deprecations would be good, too.

or available in #lang version4.5

though that probably would be an abuse of the lang system!

Well, to be clear, my vision does not involve installing multiple versions of a package at the same time, or specifying versions in imports a la PLaneT. There are various reasons why you want to decouple version specification from module imports.

Are these versions a new dimension by which we multiply the {-lib -doc -test} packages, too?

@lexi.lambda your understanding seems spot-on and is what I was thinking too. The package sources are already uniquely identified by the package checksum so it would be a simple mapping from name to the checksum.

@greg No, not in the sense that they would take up multiple spots in the package index. Think closer to a generalization of version exceptions.

@greg I would consider those packages to be “satellite packages” and grouped together

There are a lot of other changes I would ideally make to Racket’s package system if we could start from scratch, including a way to subsume the lib/doc/test split, but those things are far harder than anything I mentioned so far, since they would involve more fundamental changes to how Racket internally treats modules and module loading.

The package system is currently designed in such a way that the core of Racket knows effectively nothing about packages. Packages just extend collections, and Racket only knows about collections.

There are downsides to this, though, such as the fact that it’s impossible to resolve conflicts in the case where multiple packages provide modules at the same path.

(And that includes multiple versions of the same package, in the proposed system.)

If the pkg mgr could handle modules (for the lib/doc/test purpose), it could handle modules like api1, api2 … and that feels better to me than arbitrary version numbers and disappearing interfaces.

I don’t think you want multiple versions to be distinct packages because you really want to be able to depend on version ranges.

But I don’t care about version ranges. Those are a proxy for capabilities, which I do care about. :slightly_smiling_face:

I’d rather say, “I need api-that-gives-me-foo, and the docs, but not the tests”, for example..

I’m not sure if changing module treatment and loading would be good idea, at the end of the day, the packages are being combined to create a set of collections that are available for Racket to load. lib/tests/docs don’t necessarily need to be handled specially at the module level.

Indeed, I feel would rather have a capability that points to specified set of interfaces and dependencies, instead of harder to understand version numbers.

hmmm, that does seem like a potentially fruitful direction, but I fear it likely need a lot more research before it’d be viable for general community use.

@greg You could certainly try and design something much more elaborate than version numbers, but generally some notion of semantic versioning tends to be a good set of compromises to solve that problem.

Especially since often you want to break compatibility in a small way, and you want some dependent packages to use conditional compilation to pave over the differences.

I guess I’ve logged some time with things like QueryInterface in COM, fboundp in Emacs Lisp, and dynamic-require in Racket. And I’ve found it preferable (to me) to ask directly for the thing I actually care about. Than to have a level of indirection looking at hopefully correct docs and change logs.

If the desired function turns out not to exist, and the pkg versioning theory is “only add”, then I can give the user an error message like “Please update to the latest ___”, and that’s it. [Edit: Or, maybe gracefully downgrade something.]

Maybe that’s impractical but I stubbornly want to live in a world like that :slightly_smiling_face:

For what it’s worth, I abandoned the idea of using Racket at work because of the lack of versioning. And I’ve sort of drifted away from Racket development in general because of it. The burden on package authors to divine a perfect architecture in the first version is far, far too much in a field that strongly benefits from rapid iteration on incremental changes.

The strategy of adding an entirely new package for breaking changes isn’t good enough, since often the change in question is actually quite small, and you don’t want to completely fragment your own ecosystem just to change a small thing that likely won’t impact 90% of users.

Those are great points. And to be clear, I’m definitely in favor of a boolean “I’m ready/willing for other packages to depend on mine” flag, and having the option not to set that until you’re ready!

The current package policy works okay with very small teams, but I now work in a large organization that needs the ability to move asynchronously and update at their own pace.

JS’s and Haskell’s package systems have supported our needs there quite nicely.

for open source racket the problem is somewhat mitigated by the package-wide continuous build system

but if you’re not using that and you write a lot of packages the pain will magnify

We have a number of internal packages that break compatibility on a monthly basis, and that’s okay, because we publish detailed changelogs and let people upgrade whenever they get the chance.

CI can’t change the fundamental problem of backwards incompatible changes retroactively breaking code that worked before without changing anything, which is unacceptable.

I didn’t say it changed the fundamental problem, I said it somewhat mitigated it

I didn’t claim you did, but I’m mostly just expressing that these problems are not imaginary, since I have basically ditched Racket for anything “real” because the tooling doesn’t support my needs.

@lexi.lambda it’s concerning that you weren’t able to accommodate the current package system to your organization’s needs, that says to me the system needs at least a review. It’s been in use for how long as it been? 3–4 years? That’s long enough to reassess how successful the system has been and what improvements would be needed.

I don’t think the changes I mentioned would be terribly controversial; they were formed in a discussion between me and a few other people. Someone just needs to implement them.

But that takes time and effort, and currently nobody has enough of that to spare.

I’m in a position where I’m using Racket at work for small-scale projects, I’ve yet had the opportunity to seriously use the package system to manage my needs, so it’ll be interesting to see how far I’m able to go.

One example of something we’ve done at my organization that has been extremely successful is a Haskell DSL we use for writing deployment scripts. We use Haskell’s package versioning system to write self-executing scripts in Haskell (with a relevant shebang), which is pretty cool. Since the scripts express the version they depend on, we can judiciously break backwards compatibility whenever it makes sense, which has let us rapidly iterate without feeling a need for BDUF.

I couldn’t really do that in Racket, even if it would be a good tool for the job.

How well would a two stage solution work that first added everything except the ability to install multiple versions at the same time? The other three pieces are much easier to implement and might work as a temporary solution.

They’re mostly independent, but the sandboxing is pretty necessary to avoid inventing Cabal Hell in the Racket ecosystem. Sandboxing is actually mostly implemented, too, so it’s actually probably not that much effort to add.

If it were a huge amount of work, I would agree with you, but I don’t think it makes a ton of sense to skip if you’re also going to do the comparatively monumental amount of work to implement the other parts.

may you point me to where the sandbox environment has been implemented? Is it a raco feature?

There isn’t really user-facing tooling for it, so the work would basically just be to integrate it with raco, but the infrastructure exists. I can’t remember exactly where the relevant features are documented, but the feature is called “tethering”.

ooh I do remember coming across that term

It’s alluded to here: http://docs.racket-lang.org/raco/dirs.html#%28def._%28%28lib._setup%2Fdirs..rkt%29._find-addon-tethered-console-bin-dir%29%29

An important question is whether implementing it would require help from mflatt, since he likely won’t have time to focus on stuff other than racket7

I think it could be done with minimal knowledge of Racket internals. A few questions here and there maybe, but I don’t see why much of it would involve anything complicated.

I would bet the majority of the work would be extending raco pkg install to do constraint solving and handle the notion of version conflicts.

Suppose the four main items are designed and made available in a side-branch of the racket mainline. Would it be able to accommodate the current style of additive changes. Suppose one package favors the additive style and other one takes the version numbering approach. How do we manage users experience so they don’t get confused by two different styles?

A package could easily just only make additive changes by only ever bumping the minor version.

But there would certainly be some tricky migration/compat issues to work out.

I don’t think any of them are super hard, though.

and by setting max version to #f indefinitely really

IIRC, the proposed compatibility solution was to basically (for now) treat packages specified without bounds as >=1 && <2.

The version constraint solving doesn’t sound like the hard part, especially if it’s implemented with the aid of a logic programming dsl

I don’t really mean the constraint solving algorithm itself, but I mean plumbing the inputs and outputs of that algorithm through the rest of the system.

You need to set up the infrastructure to make the version information available to the solver and configurable by users. You need to handle all the corner cases of version conflicts and solver failures. You need to present meaningful error messages when the solver doesn’t come up with a solution. And you need to implement all of this while maintaining backwards compatibility with the old system.

indeed oof that sums up the problem.

Doing package management right is really hard! I don’t know of any programming language that got it right from the beginning, most added it on only after it became obvious how necessary it was. :p

(And many langs still have awful package management. Go comes to mind…)

we just need to find that magic wand everybody keeps talking about.

I agree with @lexi.lambda Saying ‘make a new package’ whenever you make a breaking change seems fine, until you realize that what is going to happen is that your going to get a package catalog with hundreds of packages per actual package.

@abmclin Oh, its in the cellar, let me go grab it.

Plus different users have different ideas of what a breaking change is

and the current package index is not easy to parse so your own study case is only going to exacerbate it.

@notjack That too. So….this is why I make a new package for every git commit.

(Or at least every commit I push.)

Plus you can’t even tell ahead of time whether a change is going to break any dependents with perfect accuracy!

in a large enough ecosystem someone’s always using your thing in a way you never could have anticipated

Tbh that’s one of the reasons why I think Hackage’s ability to edit constraints after publishing a package is a good feature.

A lot of tools get unhappy with the notion that releases are not immutable. But otherwise, if you don’t have that option, those bad constraints stick around and mess up the solver forever.

a release’s content being immutable while its declared relationships to other releases being mutable doesn’t sound so bad to me either

But yeah. All of Maven, Bundler, NPM/Yarn, Cabal, etc. are tools added long after the lang was invented to solve this problem.

I guess maybe Cargo gets this right from the beginning? But Rust is unique in a lot of different ways. :)

so we have several notions at work here, some set of release source files, versions, canonical names, bundling variants with or without docs, constraints. Need to figure out how to stir all of them into a workable backwards compatible solution.

the idea of dependency relationships being modified after the release doesn’t sound too bad to me either.

Racket’s package model is more mutable than many, anyway, since the package system doesn’t even host the package sources.

The obvious next step for me is to study other package solutions, see what ideas they overlap in and what work and doesn’t work.

So a bad actor could just nuke a git repo or push a change to a release tag and cause a lot of problems. But I would not recommend addressing that problem with all of this.

there’s a really good blog post about package systems that’s helpful for this

https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

That reminds me of that time when someone blew away their npm contributions and crashed the npm network which had been depending on one small packages released by that author departing with all of his toys.

wasn’t that the “leftpad” package?

yeah that sounds right

IMO, the Bundler/Cabal models are a really good set of tools to study. Yarn is good, too, but its model is different from Racket’s, so it’s not really relevant.

thanks @notjack and @lexi.lambda it’s probably foolish of me to contemplate coming up with something that makes an impact but this is definitely a problem area that deserves a closer look.

at the very least a tracking issue in the racket repo with some sort of plan and a breakdown of the work to do would be a big step forward

ha, #general anybody wants to join in?

maybe after I finish working on stuff for racketcon :p

and a few other projects, or next year…

I’d be interested in working on it if I had about 40 more hours per week. ;)

more seriously, is the general management interested in entertaining a significant update to the package system? Or would we need to do some lobbying?

I do realize this likely may need changes to be made to DrRacket’s package manager, and the package index site, so that’ll be nontrivial to coordinate.

the package site probably wouldn’t have to change to support this

The package server would. But not necessarily the frontend, I suppose.

I found the email thread from a little over a year ago between me, @mflatt, and @samth. Here’s the bulk of what we concluded, as outlined by Matthew: * Change the information for a package in a catalog to include a version -&gt; source mapping. Concretely, this mapping could go under a `pkg-versions` key added to the current table. So, for backward compatibility, we still have the current information (for the latest version) as it's reported now. Also, package clients could continue to deal with catalogs that don;t include a `pkg-versions` entry. The version listed for a given source and checksum would be checked against the `version` declaration in a package's "info.rkt" file. The package manager would still only allow the installation of one checksum (as selected by version number, typically) per package. * Change the catalog at <http://pkgs.racket-lang.org\|pkgs.racket-lang.org> to accumulate a version-keyed mapping by noticing when, in the case of a Git-based source, a new checksum has a new version. When the version changes, the server can keep the previous version mapped to the previous checksum. For sources not based on Git, package authors would have to use some extra interface or tool to set a mapping. We'd want a `raco` tool for tasks like that. (It's strange to me that such a tool doesn't exist already.) The tool could also help authors using a Git repo update the package version number appropriately. If <http://pkgs.racket-lang.org\|pkgs.racket-lang.org> missed a version due to a rapid sequence of version updates, if broken versions need to be removed, if a new revision for an old version needs to be introduced on a branch, etc. --- all of those would be possible manually. The automatic construction of the version table by <http://pkgs.racket-lang.org\|pkgs.racket-lang.org> would just be a convenience for the common case and make it behave similar to the current system in those simple cases. * Change the package-manager client to support the usual constraints on dependency versions. The package installation and update tools would have to solve those dependencies. The current `#:version` field in a dependency would be interpreted as allow &gt;= version and &lt;= the next major version (non-inclusive). A new keyword for a dependency would support more general constraints. [The "graphs" package declares a dependency on "base" version "5.3.2". That looks like the only package whose version dependency would need to be adjusted to fix this rule.] I imagine that `raco pkg update` should by default only update to a checksum for a version whose major number matches the current installation. Also, it should downgrade a package to solve constraints only if the user says that downgrades are ok. Right now, most dependency specifications are unversioned. I don't think it will work to assume "1.x", unless we add a special case for "base" to default to "6.x" and figure out something for "0.x". Unless we find something better, we'll end up allowing dependency specifications without a compatibility level, which is too bad. The dependency information currently provided by <http://pkgs.racket-lang.org\|pkgs.racket-lang.org> is not precise enough to resolve dependencies --- it merges `deps` and `build-deps` --- but that should be easy to fix. * A stack-like sequence of catalogs could provide a suitable set of default versions/checksums of packages (but, as always, that catalog can be overridden for a given package by specifying a non-catalog source) and serve as an archive of available packages. * Tools like `raco pkg catalog-archive` could take a version constraint, which would typically be specified on "base", and solve constraints in the same way as installation.

There was a little more discussion after that, but that’s close to what we generally ended up agreeing on. I don’t think any of those changes would be especially controversial if someone went and implemented them, but that’s no small feat.

that was very helpful, good to know how the others stand on it.

one relevant question here, are the current package tools designed to help end-users or developers? Should the tools attempt to help both audiences?

As long as someone follows that kind of plan – one that doesn’t involve changing the way that module-name resolution works – then I think anyone could implement the changes. It’s mostly a matter of updating the package client in “collects/pkg” (and tests and docs), the package-catalog server’s implementation at https://github.com/racket/pkg-index , possibly the pkg-build service’s implementation at https://github.com/racket/pkg-build , and the GUI package manager at https://github.com/racket/gui-pkg-manager .

Anything that changes the way module-name resolution works, in contrast, is almost certainly more trouble than you expect, even after I tell you that it’s trouble.

@mflatt thank you for the references to the repositories. I believe at this time the module-name resolution should be left alone. Modifying it in anyway would open a too big can of worms. I think the sensible approach at this time is to extend the current system in consecutive ways.

@lexi.lambda I had to step away, and I’m catching up, and I think this thing you said is interesting for me:

> We have a number of internal packages that break compatibility on a monthly basis, and that’s okay, because we publish detailed changelogs and let people upgrade whenever they get the chance.

This is using the package manager in a way that hadn’t really occurred to me. Here, “breaking changes” not only isn’t bad, it’s good — because it means “people are making progress”.

Whereas I’ve been thinking about the public package manager as a way for people/orgs to provide code for others to use. In which case, breaking changes is something I don’t want to do to other people. If I do it accidentally, I’ll try to fix it quickly. And I’d like to depend on packages with a similar attitude.

So that’s why I’m feeling, geez, this seems like a lot of work to support breaking changes, and instead why not try to encourage and help people not do that in the first place. However, you’re looking at it also from the point of view of using the package manager as part of in-house configuration management?

This gets at the distinction between regular end-users and developer users. The link referred to by @notjack goes into the distinction in depth and is insightful about that difference.

Ah OK … will catch up more! :slightly_smiling_face:

@lexi.lambda. @abmclin, @leif, @greg: I’m a bit late to the discussion, but this looks like it may lead to a good Racketeer Office Hours project in Seattle! :)

FWIW, @lexi.lambda I don’t think any ‘lobbying’ to management, so to speak, is required to improve the package manager. Its more about not having the resources…..

That’s what I was saying, I think you want to direct that comment at @abmclin.

Like, if someone where to do it, I don’t think anyone would reject the idea, just….its a lot of work.

Ah, okay

Thanks

In that case… @abmclin ^

@greg It’s not really configuration management. It’s that we have an internal tool that starts as the most minimal thing that could possibly work, since we want to move quickly and do agile, incremental software design. The thing is, the tool is naturally going to grow in scope as our needs grow and we develop it further, and that means we could either (a) slot all the new features into the existing model, or (b) continuously refactor the architecture to meet new needs.

We need the ability to break changes as our software evolves in order to be able to add new features without things turning into a big ball of mud. That’s true whether or not it’s an internal tool or not, really. Moving fast sometimes means continuously rethinking the interface to keep it clean even in the face of new scope.

yay! I’m gonna meet even more of you in Seattle

I would prefer to use a library that kept a clean, consistent interface and kept migration to a new version well-documented over one that infinitely preserved backwards compatibility but ended up with an awful API.

Other features on the wish list: A daily / weekly email with new packages with links to documentation (and/or info on updated packages)

I agree… this is also why I spawn off new projects from old so that they can age/version at their own natural rates

@soegaard2 that would be lovely… “This week in Racket” type of thing would be awesome to keep up (for the time being)

@lexi.lambda Or one like…cough…ffmpeg…cough….where the API just keeps getting more and more things tacked onto it (sometimes in api breaking ways. :disappointed: ), and now there’s 50 billion ways to do the same thing, and only one of them is right. But because all of the tutorial (intentionally singular) was written 10 years ago so it uses an old api…..) sigh. :disappointed:

yeah, dynamic libs tend to suffer from API rot because C has no ability to ever break compatibility.

@leif luckily C is known to give friendly error messages, when things go wrong :wink:

I won’t pretend it’d be easy, I understand it’s a lot of work. This is something I’d only be able to work on the nights and other periods of free time so I have to be realistic, but I won’t mind making an attempt. I think there’s enough information to be able to construct a viable roadmap draft at the minimum as suggested by @notjack

@soegaard2 lol……… Although I do have to say that is actually one area i was pleasantly surprised. Despite the shitty language, FFmpeg’s error messages are actually quite nice.

And the code itself was quite nice. The only problem is the documentation, also known as the code. :disappointed:

Is it just me, or are the latest nightlies of DrRacket kind of slow?

nightlies?

http://pre.racket-lang.org/

@zenspider ya

Well…technically I’m using ones I’ve compiled from source, but they’re pretty similar.

I was (poorly) trying to imply that DrRacket is just slow.

oh, I see…lol.

Well yes it is. But even more slow than usual….

Like, usually I can type without a second of lag…. :confused:

That doesn’t sound right?

Okay, so its not just me then.

I’m going to blow everything away then and see if that helps.

@mflatt Are there any examples of using the FFI for C functions that have callbacks?

(And I want to pass in a racket function)

I’ve never gotten it to work anyway.

@leif There are examples in the draw and GUI libraries; I see png_create_read_struct in racket/draw/unsafe/png is passed error-esc as a callback

okay thanks.

Would this work if the last argument to the callback was a va_list?

I ask because the function I am trying to call is: http://ffmpeg.org/doxygen/3.2/group__lavu__log.html#ga14034761faf581a8b9ed6ef19b313708

is there a way to lookup reverse dependencies in the package catalog? e.g. find all packages that use my packages?

@leif Taking a va_list argument is tricky, because the representation depends on the system. If you can ignore the va_list argument, then just have your callback take the first two

@mflatt I see. I thought a va_list arg was just a pointer to the actual list structure. (Where the list structure is platform dependent.)

Compared to ..., which just puts it inlined.

Although I have to admit that my knowledge in this particular area of C is a bit…er…limited. (I can’t find many good resources on it anyway.)

@leif https://github.com/dyoo/ffi-tutorial/tree/master/ffi/tutorial/examples/struct-with-callbacks

@soegaard2 Thanks. while a good callback example, it doesn’t seem to handle a va_list.

Ok. I look in that repo for examples. A shame he never got to write the tutorial.

A strange thing about the FFI: The paper on the FFI is easier to grasp than the documentation.

Ha…ya.

IMO, that’s because the paper actually has a rational for why they designed it the way they did.

Which makes it a lot easier to predict.

Anyway, I sadly don’t see anything about va_list in that repo. Oh well. :disappointed:

@mflatt okay, here is an alternative that might work. Is there any way to intercept what an FFI call prints to stdout?

@leif no

rats. :disappointed:

So basically I would have to have my whole program run in an environment that captures the stdout….that’s unfortunate.

Thanks anyway though.

@notjack I’ve done that this way: https://gist.github.com/greghendershott/b20effb9d9c48211e1c11d9486257918