> Does the collection system give us anything meaningfully useful? JS doesn’t have this concept and it seems to do OK
The closest equivalent I know of is in @mflatt’s message here: https://groups.google.com/d/msg/racket-dev/dHAFwzlFwNA/xfVGxIYaAgAJ
My only concern in that thread was that it wasn’t a one-step operation, and it leverages a search relative to a Racket installation, not each dependent module. I’m wondering if it would be worth having the module resolver considers collects-like directories and link files relative to an arbitrary Racket module. That way a collects-like directory acts a bit more like node_modules.
I think @sorawee means something different — that a package (like “w3s”) and a library path (like “css”) don’t have any necessary connection, and that multiple packages can put things in the same collection, or a single package can put things in multiple collections, or both.
Matthew’s email is about something different, which is how to have a more node-like package management experience (which, even in that system, would allow multi-collection packages and multi-package collections).
I didn’t even know that multi-package collections were possible.
As for “why collections vs packages” there are a few different answers. One is that these features were necessary to support breaking up the pre-packages Racket distribution into multiple packages, so we had to have them. A second is that if you have a package that provides a collection and you want to break it up into two packages, you need collections to work the way they do, and for similar reasons as when it was needed to break up Racket originally. A third is what I mentioned above — typed/2htdp/universe isn’t in the typed-racket package, but still is in the typed collection, and this is a nice namespacing feature.
I understand the motivation. It’s just that I can write a package that breaks other people’s code in that way.
Also, I think you did know that collections could be defined in multiple packages; that’s what the discussion about css was about.
And how would you break someone else’s code?
Yeah, sorry. I went somewhere.
The w3c package is a good example. It has a css directory with a syntax.rkt module. I cannot make a css (collection) directory with a syntax.rkt module in my own package without causing a conflict on the default catalog. As a client, I have to address this by ensuring both packages are in a different scope, correct?
I also had problems with my online doc links 404ing whenever there was a build error.
I reproduced the conflict by publishing tweedledee and tweedledum packages on the catalog. They both define a hill/king module.
It looks like you basically compete over filesystem structure with everyone else.
Certainly you can define two packages that conflict with each other and thus can’t both be installed (they don’t actually live in the same place in the file system).
Yeah… That’s a limitation that I have to find a way to get past somehow.
But I understand if that’s not what @sorawee meant
That limitation doesn’t seem like solvable problem, fundamentally. You could add more namespacing but that wouldn’t mean they would work while installed at the same time.
Note that npm doesn’t solve this problem either; they just don’t let you publish a new package to the global registry that conflicts with any existing one.
Right, but you can organize your code however you want with JS packages.
Maybe I’ve been doing it wrong, but Racket packages feel like they have a refactoring burden to make them fit like a jigsaw piece.
What do they have to fit with?
most of the time, you just define a single collection, with the same name as the package, and don’t worry any further
The files/directories from all other packages.
I can start a project, write my code in a way that makes sense, have it pass tests, publish it, and then find out it can’t exist in the same place as another package because someone else used a file of the same name in the same place.
As a user and a library author, this is tricky to deal with.
I can either add numbers to my directory names (e.g. polyglot2), or I can choose another for all affected directories in my project. In other ecosystems I only have to worry about the package name.
That’s why I’m over here trying to learn how the module resolver works to see if there’s something I can shim. But if this problem isn’t solvable, then I’ll just have to think a bit more.
That you can’t have two libraries with the same name isn’t solvable. Honestly, my recommendation is that if you just do a search for the name you’ve picked in the docs, it will tell you if it’s already taken.
Ok.
I agree that there’s an additional burden that there are two possible conflicting names, even if most of the time they are the same name, but that seems like a one-time worry. Maybe the reasons I outlined above aren’t worth it, but it’s definitely something that’s not possible to change at this point.
FWIW it hasn’t been a problem until now. If Racket ever gets as popular as JavaScript, then it is time to rethink :slightly_smiling_face:
I understand. I’m not going to complain about it or anything. My chief concern is how I translate this to my users when I ship something for them. A package conflict seems inevitable. It certainly complicated release planning for me on at least two occassions.
Maybe I can give them an installer that only ever copies my code from a package source directly into their project and avoid the collection namespace altogether.
Personally, I think “a package conflict seems inevitable” is the wrong assumption; my feeling is more like “package conflicts are not worth worrying about”.
I only think this way because I do not agree with the Racket ethos on backwards-compatibility and package management. My working habits with breaking changes alone creates a conflict because I want to reuse the same collection name in a new package edition, not create a new collection name.
I don’t see how that makes a difference — if you used the same collection name, then your two packages would conflict, but that’s what you want — people should be either using the old version or the new one
For now, absolutely. If a user installs the new version, and then installs a package that depends on my old version, I’d like to be able to give them a solution. Right now I wouldn’t know what to tell them.
Well, besides shrugging and telling them to do more work if they want both.
Package conflicts come from people naming their collections badly because they don’t understand the package<->collection relationship
There are two possible issues there. One is the user has two projects, and they want to use the old version with the one project and the new version with the other project. That’s inconvient, and that’s what the ideas in Matthew’s email might help with, and which would be good to implement. The other is that they have one project, and through transitive dependencies need both versions of your project. That doesn’t work, and the sense in which it doesn’t work is much more built-in and would be very hard to fix.
(And from people naming their docs “manual” because scribble should namespace those based on collection)
I run into conflicts with my own packages on, say, main.rkt if I want to do a new edition.
I’m confused, isn’t the problem you’re hitting a conflict between the collection names you want (which were picked well) and the collection names in the w3c package? (which were named badly)
and also, what do you mean a “new edition”?
It can. I was stating an example there, and for my own project.
New edition = backwards-incompatible version
More reading on: https://sagegerard.com/polyglot3-package-nightmare.html
^ Optional. That just shows how I was thinking about one of my own projects in this topic, but it might bring in more variables than this discussion needs.
Oh. You’re not making a new version of your package. You’re making a new package. So its collections need new names, because otherwise there’s no way to tell which of the two versions some other code is referring to when it uses your code. That has nothing at all to do with the collection systems and everything to do with the fact that racket’s package management tools don’t have good support for versioning.
Yep. I think I got the context. Thanks for fleshing this out more for me.
Yeah. I wish I didn’t learn that the hard way.
my 100% sincere advice is to get a list of your known users and change your philosophy from “never make a backwards-incompatible change” to “avoid breaking known users”
until you have thousands of users, “never make a backwards-incompatible change” is all pain for no gain
and 99% of packages do not have more than ten users
I already follow that philosophy. IMO, Racket makes it harder to do that.
That’s kind of what that article touched on.
(In that I have to do a LOT of unexpected work to avoid breaking users)
Not hypothetical users. Avoid breaking specific users you actually know about and whose code you can read.
For instance, by searching for other packages that depend on your package
I know I have users with applications that depend on my packages in private repos. I don’t know how to query the catalog for package dependents. That seems handy
Nice. Thanks!
also try designing backwards-incompatible changes such that they can be gradually performed. For instance, if you want to add a new required argument to a function, make it a three step process: first make it optional, then send pull requests to your open source users and emails to your closed source users, then after everyone’s updated and a suitable grace period has elapsed make the argument required.
One other thing I would suggest is that extending raco pkg install so that --checksum abc123 with a package name that mapped to a git repository checked out that commit (instead of failing when that wasn’t the checksum found in the catalog) might make things easier for you. A similar specification could be allowed in info.rkt, with similar behavior.
Sort of like a lockfile? (but different)
you could use it as a lockfile, or you could use it to allow specifying old versions (making it easier for people to break backwards compatibility)
I’ll take it. Any improvement is worth implementing with what I’m trying to do.