Core team: How did you forward http://github.com/plt/racket\|github.com/plt/racket to http://github.com/racket/racket\|github.com/racket/racket, while keeping the plt account? Did you just use the “change username” tool on github, and then (re)created the plt username too? (I’ve been thinking of getting a more descriptive username on github, but I’m worried about the consequences)

After building from source, what command do I use to rebuild it without messing up earlier raco pkg --clone?

Seems to me that running make will not mess with that. I guess that kind of info is stored in your pref file/dir, and make doesn’t override this.

Right. It’s recorded in the package-installation information.

Probably @samth knows the answer…

Interesting. There must be something else that made me getting package conflicts

I believe we created the racket organization account, and then transferred the repository from the plt user to the racket organization.

I think in your case just renaming should preserve links

I see, thanks

Ah, my fault indeed

I manually modified racket/share/links.rktd without modifying racket/share/pkgs/pkgs.rktd

That seems a little risky indeed :slightly_smiling_face:

Hello, What would be the right type definition for the following function? #lang typed/racket/base (: test (All (A ...) (A ... -> Boolean))) (define (test . rst)(if (null? rst) #t (and (car rst) #f))) (define #:forall (A ...)(test2 . [rst : A ... A]) : Boolean (if (null? rst) #t (and (car rst) #f)))

both typecheck, but take the none empty branch for (test) and (test2)

(using racket 7.7.0.7)

@bedeke: (: test (All (A) (A * -> Boolean)))

Thanks, but I really want the rest arguments to be different types

@bedeke that looks like a TR optimizer bug

that is, when I add #:no-optimize to the #lang line, i get #t

Or, more fundamentally, this typechecks: (: test (All (A ...) (A ... -> False))) (define (test . rst) (null? rst))

I’m looking for a procedure which I’d call any. It’s a type of fold that’d say true if any of the elements of the list satisfy a predicate. (Similarly, I’m look for a procedure that I’d call all.) I can’t seem to find any such procedures in the standard libraries by this name.

@anything they’re called ormap and andmap

Aha! Thank you!

I might as well ask: why isn’t there a procedure called true? if there is one called false? in racket/bool?

in general you don’t need to look for true values because everything that’s not false is true

Hey folks, very quick question: is there a way to install a specific version of a package with raco ?

If you need to be sure that you get the version you need, you can use the --checksum option to raco pkg install. Or you can specify a version in your info.rkt and raco pkg install will ensure you get at least that version.

There’s no way to install an older version in general, but if it points to a git repository you can use the git://... package specification to pick out a particular branch or commit.

Good point. I mean, while false? is an alternative to not, false? exists while true? does not.

wonderful, thanks! I’ll give the git thing a try

@samth the second step in what I was trying to do was: #lang typed/racket/base (struct (A) foo1 ([a : A])) (struct (A) foo2 ([a : (-> A A)])) (: test1 (All (A B ...) ((foo1 A) (List (foo2 (∩ A B)) ...) -> A))) (define (test1 a bs) (foo1-a a)) (: test2 (All (A B ...) ((foo1 A) (foo2 (∩ A B) )... -> A))) (define (test2 a . bs) (test3 a bs)) this fails the typecheck with: Type Checker: Polymorphic function `test1' could not be applied to arguments: Argument 1: Expected: (foo1 A) Given: (foo1 A) Argument 2: Expected: (List (foo2 (∩ A B)) ... B) Given: (List (foo2 (∩ A B)) ... B) Result type: A Expected result: A in: (test1 a bs) given the error message, I’m hoping it can be achieved…

what is test3 here?

~I think that was supposed to be test1, and I think the issue here is that the type (List T ...) requires at least 1 element.~

… after using (inst test1 A)…

Er, after re-reading the docs, I don’t know about that…

But the error then is: Type Checker: type mismatch expected: Null given: (List (foo2 (∩ A B)) ... B) in: bs

Oh, that was dumb of me. I only instantiated the first type variable.

Ai, bad copy of code, it should be test1, ie just calling test1 with the rest arguments gathered in a list

So, (inst test1 A B ... B) works fine.

oh thanks… due to the error text I forgot to try inst…

I find it hard to predict when you need to instantiate type variables explicitly and when they can be inferred.

And, of course, when you use the same names for your type variables and you get error messages that say: “I expected A but found A” it’s obviously pretty confusing.

@samth you said you can install a specific version using a checksum option, but then you said there’s no way to install an older version in general. Can you clarify that?

The reason I’m asking is envisioning a scenario where the latest version of a package has a bug, and I need to stick with the previous version that works.

I mean, you can supply the checksum command line argument, and that will verify that you get that checksum, but it won’t let you get an old checksum

That seems quite odd.

So you can specify a checksum of the latest version to ensure you get the latest version, but you can’t specify any previous checksum? Wow.

If you need to depend specifically on an old version, then you can depend on a package named by a specific git revision using the the syntax for that, such as <git://github.com/samth/foo#branch>

How do folks deal with the scenario I mentioned? In the Ruby world, it’s extremely common to specify the specific version of all gems (“package”) used in a project.

I just did raco pkg show and only two packages on my lapto end in ".git"

Is there a way for non-github packages?

There’s been a bunch of discussion of this recently in various places, one place to look is https://alex-hhh.github.io/2020/05/dependency-management-in-racket-applications.html

Thanks. I remember seeing some of the discussions, but I wasn’t at the point where it was a pressing issue yet. That’s changing. It appears the current situations requires some hacks, but is the long term goal to allow versioning, as with most other language ecosystems?

I just looked through a thread on the mailing list about package versioning. If the accepted way to make backwards incompatible changes to a package is to change the package name, that’s fine. I’m used to a major.minor.patch versioning of libraries/gems/packages, but whatever. However, I’m more concerned about the scenario where a package is updated and introduces a bug. I realize a few ways to handle this have been adopted by various people, but thus far, they’ve been unsatisfactory to me. @jeapostrophe mentioned an option of <https://groups.google.com/d/msg/racket-users/4iI-SanIbzk/sGHYijLPAAAJ|creating your own package catalog> which seems interesting. If you are using this method, would you mind commenting in a thread, and if you can point me to documentation, blog posts, etc. to help me get setup, that would be great.

For this method to work for me, it would have to allow me to pin a particular version of a package after the package has been updated w/ a bug. In other words, if I have to have already pinned a package before it’s updated with a bug, then this is a fail, but I think that’s obvious.

By the way, I really hope the long term goal is simply to be able to specify a list of packages and their versions, and then be able to install all of those versioned packages with a raco pkg command. Ideally, this would be on a project/directory basis as different applications may have different needs. Anything else seems ridiculous frankly.

Here’s an example from the Ruby world: https://bundler.io/

@badkins One data point: https://alex-hhh.github.io/2020/05/dependency-management-in-racket-applications.html

Yes, while I appreciate that, it doesn’t seem like the right approach to me. Bundler is far superior, and we need to get there.

We don’t want a bunch of users creating their own ad-hoc solutions to this very common problem.

Just out of curiosity, do you manage any production apps/servers?

Then you should be aware of the importance of configuration management, right? It’s extremely important to be able to depend on a specific set of libraries at specific versions.

I expect Python has a similar system to Ruby’s Bundler.

It’s exactly as prevalent as there are people managing production apps/servers, right?

I think you’re saying you haven’t been bitten by it yet, but that’s an extremely risky way to proceed.

Sure. Envision this scenario. You install a set of packages, develop an application, and everything is fine. Then you setup a new server, install the same set of packages (some of which have been updated), and your tests fail. How do you get the set of packages on your development machine installed on the server?

Or, you updated packages on your development machine, and tests fail - how do you roll back?

Right. I’m not interested in Docker.

@badkins you might be interested in https://racksnaps.defn.io/

How do you handle needing packages from different days?

You’re forced to upgrade at that point.

I don’t think that will work then.

I really don’t get the resistance to a versioning system.

Not at all - sorry for the confusion! I was referring to pushback against versioning I’ve seen in various threads.

I was pointing out racksnaps as a potential solution to:

> However, I’m more concerned about the scenario where a package is updated and introduces a bug. But it’s for sure not a general solution.

I also didn’t get the sense that there was much resistance, just nuance and folks pitching in about their experience. I’m sure if someone builds versioning atop raco pkg and it ends up being a good system, then people will use it.

This is a related, but different question. For one scenario, it would be sufficient for me to duplicate the package environment I have on my laptop to new production servers. I see that there is a raco pkg archive command - could that be used to accomplish this? I develop on MacOS, and I deploy on Linux.

Maybe it was just defensiveness for not having a versioning system. I suppose the main pushback is for not allowing backward incompatible changes i.e. forcing a new package name. I’m less opinionated about that.

@popa.bogdanp to build something on top of raco pkg it seems that the current system would need to allow installing a specific version; otherwise, it seems like it might be an entirely new package system.

You could have a system that generates a catalog on the fly that points to specific source versions of each package. The catalogs themselves are just mappings from package names to package sources (URLs).

In this scenario the catalog is kind of the analog of a lockfile in other package managers.

I guess I don’t understand how to retrieve all the source versions for a package using the current package system.

I suppose that’s easy enough if they’re using git; otherwise, I’m not sure.

For example, csv-reading

The package server would have to be extended to support that kind of metadata and to be able to host all the versions of those packages (IMO, relying on Git is not a great idea). Or someone could create an alternative package server that supports those things.

BTW @popa.bogdanp thank you for racksnaps - I didn’t mean to be so dismissive. I think it could be extremely useful even if I can’t use it directly as intended. For example, if Neil updates csv-reading, I would have no way of getting the previous version (if I didn’t have it installed locally already), but I could grab it from racksnaps and put it into my custom catalog.

No worries! Also, another way to address your original question could be to layer different snapshots.

I hadn’t thought about it much and I’ve never tried it, but conceptually it should work just fine.

So say you want to rely on packages from 2016/06/14, but then a new package comes around on the 18th, you could just add the snap from the 18th as a secondary catalog.

Of course, that doesn’t work in many scenarios (eg. when you just want to upgrade a single package).

Generally, though, I’ve found that folks just don’t make many breaking changes. When they do it’s easy to upgrade and relying on daily snapshots prevents bugs from suddenly appearing.

not sure where the right place to ask this question is but I was wondering if expand supports reading in a module that has unbound identifiers. I was hoping I could use drracket/check-syntax to analyse such files.

> We don’t want a bunch of users creating their own ad-hoc solutions to this very common problem.

@badkins I have a feeling that, while several people agree that there is a problem, there is no agreement at all as to what the problem is and how to solve it. I came up with my solution, which works for me, @popa.bogdanp came up with his, which works for him, but other people don’t find either solution satisfactory. Perhaps further concrete proposals and pilot implementations will need to appear, and maybe one will emerge as a community favourite and will become the “official” one.

Well, I feel this is a very solved problem in other language communities. I suppose there are some subtle differences, but the problem seems clear enough to me. We have a history of operating system packages & numerous programming language package systems. I’ve used Bundler for years in Ruby, and it’s done everything I’ve ever wanted.

I understand the strong desire for backward compatibility in the Racket community, and I think that’s a strength, so I’m personally fine with changing the name of a package when it breaks backward compatibility. I realize others feel differently, but that’s a separate problem from the issue of unintentional breakage due to the combination of bugs and package author’s delay in fixing them.

Having said that, once you solve this more important problem (which I think requires versioning), there’s really no need to rename a package - just consider the version part of the name where a major version change indicates a backward incompatible change.

@badkins you seem familiar wit Bundler and think it will be a good fit for Racket. Do you volunteer to provide a pilot implementation, so the rest of the community can evaluate it to see if it fits their needs?

This question comes up often enough from people that we could just add true? and write down the explanation of why you usually don’t need it in its documentation

Since Bundler exists already, it’s easy to evaluate - it seems to be well documented. There may be better managers, but I’m not familiar with them.

I feel like there was a recent discussion about adding it, but I agree generally

I think they all basically do the same thing - allow specifying a set of packages with a versioning scheme that’s flexible enough to specify a variety of versions, and allow locking a set of packages at specific versions.

Seems like the first step is to simply allow something like raco pkg install foo@2.3.7 (I stole the @ from Julia)

not sure how much has changed since it was published, but this post is a good summary of the state of the art https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527\|https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

@notjack No one is suggesting that they want to write a package manager. People only want to give advice for someone else to do it.

yup

C’mon guys…

@badkins it is a full time job and then some to make one

I’d be content for folks to simply admit it’s important to be able to install a specific version of a package.

I think that’s important, yes

Great, ’cause I got the sense that feeling is far from universal, so it’s a good start.

I also think getting to that point will take a lot of work, way more than I’d be able to do as someone who writes Racket in my spare time for fun

and this is coming from someone who works on CI systems for a living

So @mflatt before I invest more time in this, is it safe to say the core team would be interested in enhancements to raco pkg that allow for 1) installing a specific version of a package, 2) specifying a set of versions for each dependency package, and 3) having a versioning scheme such as <major>.<minor>.<patch> where a bump in major number may break backward compatibility?

Rather than reinvent the wheel, I’d probably just go with RubyGems versions specifiers for dependencies: Most of the version specifiers, like >= 1.0, are self-explanatory. The specifier ~> has a special meaning, best shown by example. ~> 2.0.3 is identical to >= 2.0.3 and < 2.1. ~> 2.1 is identical to >= 2.1 and < 3.0. ~> 2.2.beta will match prerelease versions like 2.2.beta.12. ~> 0 is identical to >= 0.0 and < 1.0. https://guides.rubygems.org/patterns/#pessimistic-version-constraint

@badkins I think that while people would like something like bundler, there are some tricky issues that make it not obvious what the best design for Racket is

Or, I should say, people would like the properties that bundler provides, as you say

I will try to write down some more of my thoughts here later this evening

Bundler-like functionality wouldn’t be the first step. I’d rather build the bare minimum that makes sense to allow the fancier functionality to be built upon it i.e. the primitives.

@badkins I would say that everyone (core team included) would be interested in a worked-out implementation that does those things.

Awesome. I’m basically married to Racket for the long term now, so I’m going to need this eventually. For right now, I just need to solve the “I need these specific packages at these specific versions” problem regardless of how ugly/manual, but I’m happy to get involved in a longer term project that ends up with something like Bundler. I think I misread comments as pushback, so glad to hear I was mistaken.

I also have an idea that I think would accomplish the key thing you want without much change at all, but the kids have to go to bed first

I think @jeapostrophe’s “personal catalog” idea may get me there once I figure it out…

The non-git packages are a bit of a pain, but maybe racksnap will provide what I need there re: getting a previous version of a non-git package after the author has updated it an overwritten the package source with the latest.

I’m also wondering if an existing package manager couldn’t be leveraged. I don’t really see why something like Bundler couldn’t be used for another language, but I may be missing something.

If you are looking for a personal package catalog, you can just place packages you need in the same directory (use one directory for each package) and use the pkg/dirs-catalog command to create a catalog for these packages. The blog post that I wrote uses essentially this approach, with the added step that the packages are managed as git submodules. However, you can ignore the top half of the blog post, and the bottom half will show you how to create a personal catalog.

Thanks, I noticed that part of your blog post after I asked the question above. What’s the best way to install packages on a server using that catalog?

I think the personal catalog approach is the 80/20 solution I need for now, and the web framework is both a higher priority, and something I’m better at, than a package manager. I’d probably be more likely to provide funding for the latter if push comes to shove.

The remaining 20% is to use git and clone the directory with packages onto the server — note that putting them somewhere else would not save you anything as the source code for the packages still needs to be downloaded.

There are also ways to create binary packages (which are tied to a particular Racket version and dependent packages) and put those in a catalog — this would make installation and deployment faster, but managing them would be more complex — I have not looked into this, so you’ll need to read the Racket documentation for how to do this.

I’d rather not manually load packages into git. If I have a well tested set of packages on my laptop, I think I can raco pkg archive - the docs state this will create a catalog from installed packages. Then I just need to provide access to that on the server. I suppose a worst case scenario might be scp’ing the directory to the server ?

Oh, it looks like you have to manually list all the packages with that approach :disappointed:

you can create one “meta package” which depends all the packages you need, and archive that one, including all its dependencies

Gotcha. I should probably make my app a package, then I could just archive it w/ dependencies. I’ve been putting that off…

@badkins About a year and a half ago, I wrote this comment about what concretely would need to be done to move towards a more Bundler-like world for Racket package management. I think it’s all still relevant, if you are interested. https://www.reddit.com/r/Racket/comments/9rn5ms/raco_package_versioning/e8jx6ws/

I don’t think the required effort to get something useful is that huge, since as I said, a lot of the technical foundations are already in place. But it’s big enough that nobody has done it.

Awesome - thanks. I really need to finish my web framework first, but package versioning may be the next thing I get involved in after that.

I’m not sure I can attract non-lisp folks with the web framework (doesn’t matter to me), but if I did, and they came from Ruby, Elixir, Python, etc., they’re going to want versioning very soon :slightly_smiling_face:

@lexi.lambda long-term, I would certainly want what you describe in point 1, but I’d be fine with that coming later. #’s 2 & 3 would be a higher priority for me

FWIW, I think point 1 is actually likely to be the easiest of those points to implement, since sandbox tethering already provides most of the required functionality. But I could be wrong.

I worked for years in a mode where I had multiple active projects with wildly different versions of libraries, so both Bundler and rbenv/rvm were very handy for having tons of different versions of ruby and various gems, but that’s unpleasant :slightly_smiling_face:

Right now I just want to lock down packages so they’re the same on my dev machine and server.

Yeah, that makes sense. But just FYI, I think the Bundler model of generating a per-project bin/ directory with launcher executables that configure everything properly would work really naturally with the way the existing Racket infrastructure is set up.

Good to hear.

I’d be most competent on #2 at the moment.

Yeah, I think most of the challenge of #2 is getting the UI right and handling backwards-compatibility gracefully.

The actual protocol changes should be very straightforward.

First, here’s some overall thoughts. I think people with experience in other language package managers mostly miss the following things: 1. Being able to record the packages they depend on, and then check in something so they get the same packages when they rebuild in a different environment. 2. Having a per-project scope for packages, and being able to install different versions of a package for different projects. 3. Being able to specify specific versions of packages that you depend on, and have the package installation mechanism figure out appropriate versions of everything to install 4. Being able to have two different versions of a dependency appear in your dependency tree while everything keeps working.

There’s also providing a smoother way of handling backwards incompatible changes, but I think that’s fundamentally separable

From my personal point of view, I care about (1), (2) and (3). I have solutions for (1) and (3) (some might call them workarounds, but I am happy with them), but (2) would be nice to have.

Also, (2) might be achievable, as suggested here: https://groups.google.com/d/msg/racket-dev/dHAFwzlFwNA/xfVGxIYaAgAJ, but I didn’t have time to explore that option

I think the solutions for each of these ends up being pretty different, and it’s important to think about the different goals here. For 1, which is the problem that is most likely to cause things to go badly wrong, people have built a number of tools, such as racksnaps and the catalog mechanism in general, and @alexharsanyi has demonstrated how to put things together. But I think there are some other simple things that would help. For example, a tool could rewrite your info.rkt file to change the dependency on “foo” to one on "<git://github.com/bar/foo#hash|git://github.com/bar/foo#hash>". Then, assuming that repository stays in place, new installs would get exactly the same package content. Of course, that’s not perfect, but it could be further improved by adding a lock file that was separately consulted by raco pkg install. Another simple thing would be for each dependency to allow specifying which catalog to look in — then you could use different racksnaps for different packages. None of these changes alter anything fundamental, but they would make the deployment safety use case easier.

For 2, the question is how much you want to share. If you don’t care about any sharing, you can just have multiple racket installs and never use user scope (this is basically what I do). The tethering approach @mflatt described lets you share more, but all the installations have to share the same version of everything shared, and everything those packages depends on. One question in this situation is the role of DrRacket. Basically, if different projects need different versions of one of DrRacket’s dependencies, then you’d need multiple copies of DrRacket.

For 3, some bigger changes to the protocol are required — the server needs to remember how to get past version, and has to know when a new version has been released. http://pkgs.racket-lang.org\|pkgs.racket-lang.org doesn’t store package content, and I don’t think we want to change that, so that’s one constraint. However, I think something in this area is doable, but it would be a lot of work in a number of places.

For 4, I think it’s much harder. There are two big challenges that I see. One is that the collections system assumes that collection paths are unique, and installing two versions of the same package would almost certainly break that. The other, more subtle issue is you’d probably end up with bugs related to the two copies trying to interact and failing, typically because the different struct definitions would produce distinct types. This is a fundamental problem with this idea in any language, but most other languages don’t have it as bad as Racket. In Haskell or Rust, either the incompatible type is exposed in the interface or it isn’t. If it isn’t, there’s no problem, and if it is, there’s a type error. In Python or JS or Ruby, typically values from external libraries are treated structurally, ie using duck typing. So the instances play well together in most cases. Racket is somewhat unique in being untyped but relying heavily on nominal tags. Of course, even in other languages two versions of a package could do something weird and conflict using the filesystem or other hidden state in a way that caused subtle bugs, but it’s much rarer. Notably, this is a problem that came up regularly with PLaneT, the previous package system. There ended up being some metadata that said whether this meant the two installs should be prohibited, but there’s only so much that this can really be solved.

Finally, on the issue of backwards incomparable changes, I think this could be addressed by having packages grow an additional field (say edition) which would basically be part of the name, except that you’d write it separately in the info.rkt file and all the different editions of a package would show up together on the package web site. This would be a pretty small change to the system but would make this use case much easier to handle than currently, where you basically add a digit to your package name. Note that widespread use of this would exacerbate problem 4 above (one reason that I prefer the “maintain compatibility” approach).

Well, that was a lot of text. Hopefully that helps, @badkins. And I’m interested in other people’s thoughts who are interested in the package system @lexi.lambda @popa.bogdanp @notjack And I should say that I’m potentially interested in implementing some of the simpler ideas for 1 & 2, but it is not at the top of my stack right now.

I think the comment I linked above (which you responded to, so I know you read it at one point) is essentially consistent with what you’ve just written, albeit it doesn’t include a direct analog to point 1.

IIRC, bundler (and some other package systems) don’t attempt to solve (4); they require that the dependencies of a given application agree on a single version of any one package.

Though not solving that problem adds a different one: figuring out which, if any, version can be used given all the dependencies.

Yes, I agree; my comment proposes a Cabal-/Bundler-esque model, not an npm-esque one. The latter seems theoretically useful but very hard to get to from where we are right now, and I don’t think it’s essential to the usefulness of the system.

I do think your explicit discussion of DrRacket is a very good point that I didn’t mention explicitly. In some ways it is rather at odds with the DrRacket philosophy: it would mean having a DrRacket installation per project, but DrRacket is supposed to be an operating system! Historically Racket has gone to great lengths to allow DrRacket to run everything in-process, without the need for some extra-linguistic mechanism, but this would be an explicit compromise on that point.

That said, I think the current package system is already an explicit compromise on that idea, relative to PLaneT. So that isn’t really new, it’s just a deeper ramification than we’ve seen so far.