Where the reader is the user, and in some cases the text -> racket object reader.

@mflatt For the purposes of maintaining two separate namespaces, would there be anything dangerous about making each namespace’s scope a global, well-known scope, rather than a separate scope per module? I’m trying to think about if there are corner cases that would break hygiene, but I don’t think so, since the existing per-module scope will prevent bindings across modules from interacting in undesirable ways.

@lexi.lambda would that not entail needing a scope that purports to be distinct from other scopes and the same across builds?

It would, but that can be done with the make-syntax-delta-introducer trick Matthew pointed out to me the other day: (define-simple-macro (define-value/type-introducers value-introducer:id type-introducer:id) #:with scopeless-id (datum->syntax #f 'introducer-id) #:with value-id ((make-syntax-introducer #t) #'scopeless-id) #:with type-id ((make-syntax-introducer #t) #'scopeless-id) (begin (define value-introducer (make-syntax-delta-introducer #'value-id #'scopeless-id)) (define type-introducer (make-syntax-delta-introducer #'type-id #'scopeless-id)))) (define-value/type-introducers value-introducer type-introducer)

By embedding the syntax objects in the resulting program and using make-syntax-delta-introducer, the scope is preserved across module instantiations.

I just tried this approach, and it seems to work fine. It simplifies the implementation a lot, so I am leaning towards it—it eliminates the need for a patched version of racket/splicing and a reimplementation of module+ to cooperate with submodules.

@samth saw racket/racket#1870, I’ve looked into racket on docker stuff a bunch and I’m curious what your plan is there

@notjack just using it in the linked repo for automated release testing purposes

at a first pass I’m guessing the github issue is dockerfile related rather than racket related

@samth Try changing the # mflatt section of the dockerfile to this:

CMD ["/racket-in-place/bin/racket", "-l", "tests/racket/test", "-l", "tests/htdp-lang/test-htdp", "-l", "tests/compiler/embed/test", "-f", "racket-test-core/tests/racket/pack.rktl"]

then to run tests, run the command sudo docker build . -f Dockerfile-tests -t racket-test && sudo docker run racket-test

The GitHub issue fails if I connect to the image and run the command manually

images and containers are different things - image is like binary you compile, container is like process you run

that dockerfile is running the test command when you build the image, so you’re in “build mode” and have an AUFS filesystem that does weird things

I think you want to run the tests inside a container spawned from the image instead

containers are the only things you can connect / shell into, you can’t “connect” to an image

(as an example of AUFS weirdness, putting RUN rm -rf /some/dir in a dockerfile does something that’s almost, but not quite, entirely unlike what you meant)

Sorry, what I mean is that if I run that docker file, and then connect to the resulting container, and then run the failing test, it continues to fail

Also, what should I do instead of rm?

Also, is there a way to roll back to an earlier point in the build inside a dockerfile?

hmmm

how familiar are you with docker?

@notjack I’ve written some docker stuff before, but only for basically scripting a container creation reproducibly

@samth okay, so the most important thing to know is that you should assume containers have read-only filesystems and try and understand dockerfiles and the docker command line in terms of that

more accurately, assume containers have immutable filesystems

Ok, that’s confusing since I can download and install things

exactly

also, your suggested command also fails

could you paste all output of the sudo <docker build+run> command?

basically just some cache hits and then the usual racket test output and then the error message

@notjack https://gist.github.com/83e2062f81ce02d55d9f97db41d5c30b

@samth can you nuke your cache first? the output of each build step would be useful

remind me how to do that?

docker build --no-cache

@notjack, same error: https://gist.github.com/e6f518b171f893cc89a95fc2543e95b3

@samth I don’t think I see error output in that log

yeah that’s because I failed to redirect stderr

it’s just the same last line

ah okay

I don’t know racket’s tests well enough to debug the specific failure, but I can give general docker guidance that will probably help

My suspicion is that it’s something to do with running as root or in /

@samth Ah, then I probably know how to fix that. But first…

@samth Think of containers as extremely well-isolated UNIX processes (possibly with subprocesses), and images as executables with metadata describing that isolation. Running a container involves choosing an image, setting up isolation described by that image, then running the image’s executable in that isolated environment. Network configuration, OS capabilities, user and group config, and what the filesystem looks like are all included in an image’s isolation metadata. This means that by default:

• Attempting to write or create files in a container is only visible to processes inside that container and writes won’t persist after the container dies
• The container has an isolated network with a different IP address than the IP address of the host machine the container is running on
• The container runs as a root user, but this does not affect the container isolation. Images can further restrict this if they want by creating users and user groups and defining what user/group the container runs as when it first starts up.
• Containers running on the same host share the operating system kernel (unlike VMs where each gets its own hypervisor) but syscalls and the like can be intercepted and restricted
• All processes inside a container run in a single process group and the host may kill them all and restrict their resources arbitrarily
• Images do clever things to make having lots of them cheap in terms of storage space, and containers are very cheap to start and run (unlike VMs)

Right, I understand a bunch about how containers work, but have not used them a huge amount (this is a common academic affliction)

@samth Gotcha. So I think the part that’s confusing here is how a dockerfile gets turned into an image, and how an image describes the filesystem of containers run from it. An image has a list of layers, starting with the “base layer” which is empty(-ish), and each layer describes a filesystem diff from the previous layer. So images are like git repos, and if you take a base image (like ubuntu) and make ten different images that add 10MB of different stuff on top of it, you can cache all the layers in the ubuntu image when downloading those ten images.

Dockerfiles describe how to build images. Each RUN, ADD, and COPY command makes a new layer. So if you add a file in one RUN command, then remove it in the next, you don’t actually save any space in the image. In fact, you double the space taken up by the thing you wanted to remove, because now the image has to store it twice: once in a diff that adds it, and once in a diff that removes it.

right, that all makes sense

the reason I’m removing things in the other dockerfile in that repo is to have a clean state, not to save space

what I’d prefer to do is basically have the equivalent of slideshow 'alts for dockerfiles

IOW, download some stuff, and then say, first do this, and then go back to the layer after downloading and do this other stuff, etc

Yeah docker does not encourage doing that within an image because saving space and structuring layers so you can cache them across different builds of different images is crucial. Instead, you can use “multi-stage builds” which let you define a pipeline of multiple different images in a single dockerfile and you can use files from one image as an input to building the next image.

You do this by having multiple FROM commands

https://docs.docker.com/engine/userguide/eng-image/multistage-build/#before-multi-stage-builds

ok, that looks helpful

other random tips:

• Always use absolute paths to refer to stuff in the image’s filesystem, because otherwise you have to keep track of the current directory inside the image and the current directory outside the image while you’re building it. Gets messy fast.
• Pay attention to how ENTRYPOINT and CMD are different, and always use “exec form”
• Make sure to use the most recent stable versions of Docker and anything related to Docker, because the space is progressing so fast that a “best practice” from six months ago may easily be terrible advice today
• Read this: https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/

thanks

you can have a look at https://github.com/samth/docker-racket-build for where I currently am

I’ve been peeking at it. FYI, I’ve got a repo (https://github.com/jackfirth/racket-docker) that builds docker images for already-released racket versions using the installers.

but I do think racket should have an official docker image that’s built from source as part of its release process.

the goal here isn’t to produce anything for general consumption

but to have a nice contained way of doing testing

producing a docker image would be nice to do too

baby steps :)

Is there some magic that prevents DrRacket from recompiling racket/base code with errortrace annotations? If I call a function in racket/base and it raises an error—perhaps I’ve called (first '())—the stacktrace doesn’t include frames inside the implementation of first. But if I define first myself in some file I require, it does include frames inside my implementation.

@notjack is there a reason that xvfb keeps randomly failing to start?

@samth probably docker’s default security setup assumes you won’t be doing gui-like things in containers

answer, it doesn’t like being re-run in close succession with itself

this thread right here: https://twitter.com/lambda_calculus/status/923989942354759682

@samth also, docker does not give containers access to the gpu

I don’t think that’ll be a problem

@royall…okay, I laughed at that one a bit too much

the whole thread is actually great

and not just for the joke

how am I not already following this person

oh were you not already following david?

you should do that

done

Racket users are the best thing on Twitter

besides dril, ofc

added a :dril: emoji, you’re all welcome

@samth did a little googlin’ and I think the Right Way for testing graphics stuff inside docker containers is to put the xvfb headless server inside its own separate container

since X is a client server model anyways

@mflatt I just noticed that distro-build-test is not on http://pkgs.racket-lang.org\|pkgs.racket-lang.org, and that it’s info.rkt specifies a collection that’s probably not what’s intended (my guess is 'multi is intended)

@notjack I made you a collaborator in the hopes that you’ll fix all my bugs :slightly_smiling_face:

@samth there’s been a change in Google’s FOSS policies so I might not be able to help much, unless you’re fine with patches being copyright Google instead of copyright me (they’d be under the same license either way)

I don’t particularly care who owns the copyright

probably because docker uses the Open Containers Initiative standardized image format which many different container runtimes understand now