So I’m using the software Tiled, a game map editor and it stores a tile as an array of ints: "data":[2], it has an option to compress it to base64 gzip, that is it first it compresses with gzip and then applies base64 to the result, turning into this: "data":"H4sIAAAAAAAACmNiYGAAAJcXTYsEAAAA", so how would i convert the string above back to 2 ? here is what i tried so far:

(require file/gzip) (require file/gunzip) (require net/base64) (define unb64 (base64-decode (string->bytes/utf-8 "H4sIAAAAAAAACmNiYGAAAJcXTYsEAAAA"))) (inflate (open-input-bytes unb64) (current-output-port)) //inflate: unknown inflate type and i get: inflate: unknown inflate type

What’s the first, say 20, characters of (base64-decode (string->bytes/utf-8 "H4sIAAAAAAAACmNiYGAAAJcXTYsEAAAA"))) ?

#"\37\213\b\0\0\0\0\0\0\ncb``\0\0\227\27M\213\4\0\0\0"

There are variations on base64 encoding, so you might be running into a problem with that

According to Wikipedia a gzip header begins with 1f, 8b or 31, 139. But your strings contains 37, 213.

To check that the data is as you expect: 1. Save the string in a file. 2. Use command line tools to base64 decode and unzip it.

37 octal is 31 decimal

Oooh!

And 139 is 213 octal.

213 octal is 139 decimal… so it looks like it’s the right header

{ "compressionlevel":-1, "height":1, "infinite":false, "layers":[ { "compression":"gzip", "data":"H4sIAAAAAAAACmNiYGAAAJcXTYsEAAAA", //this was [2] w/ no encoding , regular base64(uncompressed): AgAAAA== "encoding":"base64", "height":1, "id":1, "name":"a", "opacity":1, "type":"tilelayer", "visible":true, "width":1, "x":0, "y":0 }], "nextlayerid":2, "nextobjectid":1, "orientation":"orthogonal", "renderorder":"right-down", "tiledversion":"1.6.0", "tileheight":32, "tilesets":[ { "firstgid":1, "source":"tsd/breaking.json" }], "tilewidth":32, "type":"map", "version":"1.6", "width":1 } the entire file if it helps lol…

While a fun little exercise, just wanted to note for @jestarray that - unless you have to - the benefits of using the base64/gzipped “compression” for your maps will be mostly wasted unless:

• You have very few unique tiles (read: huffman encoding will benefit you a lot); and • You have fairly large maps Otherwise the 2D nature of the maps will have a difficult time benefitting a lot. Most 2D game tilemaps have good compression is when it’s done in blocks (e.g. a 4x4 tile house repeated in multiple places), but since Tiled (last I looked) is row-major in storing the data, all the benefits of those groupings is lost and can’t be compressed well.

Then you add the base64 on there and you lose compression there.

Just noting that you may want to save yourself grief and have it be straight-forward, uncompressed data. :slightly_smiling_face:

When you say that data was "[2]", do you mean that it was a string containing 2 in square brackets, just as you wrote it, or do you mean it was just the number 2?

@jaz no quotes: "layers":[ { "data":[2], } ] so not a string afaik

Okay, but interpreted literally, that’s an array, right?

Which would need to be serialized somehow.

the square brackets are in there yeah

Okay, the docs say that after you decode: Now you have an array of bytes, which should be interpreted as an array of unsigned 32-bit integers using little-endian byte ordering.

That makes sense…. Okay:

So, you’re using inflate in your example, but I think you need to use a gunzip function. (My limited understanding is that the deflate format is used in the data section of gzip data, but it’s not the whole thing?) At any rate, if I do the following: #lang racket/base (require net/base64 file/gunzip) (define text #"H4sIAAAAAAAACmNiYGAAAJcXTYsEAAAA") (define raw (base64-decode text)) (define raw-in (open-input-bytes raw)) (define out (open-output-bytes)) (gunzip-through-ports raw-in out) (get-output-bytes out) I get: #"\2\0\0\0" Which, according to the line I quoted above, would be interpreted as an array containing a single 32-bit, little-endian integer, 2.

:smile: thanks a lot, thats just what i wanted!

@jaz would you happen to know if there’s a zlib infliate version as well in the standard library? cant find it anywhere

I think that’s the inflate you were using.

IIUC, gzip is an archive format that uses the same kind of compression as zlib/pkzip .

i tried that but I get: inflate: error in compressed data on the same data [2] compressed with zlib instead: "data":"eJxjYmBgAAAADAAD",

https://groups.google.com/g/racket-users/c/3CvjHLAmwSQ reading this thread and it seems i have to add ADLER–32 check to the end of the byte string

seems like theres a zlib-inflate in net/gitcheckout module

alright got it workin now, thanks again :9

Ooh just missed this, yes there is a zlib inflate in git-checkout, unfortunately it is not provided. I copied just the zlib related functions (and their deps) into a separate module and it works fine.

I literally was looking at this Wednesday

probably should expose this or duplicate this in the gunzip module lol, excluding all of the git specific exception stuff

https://gist.github.com/samdphillips/d00c63ce04aa76da38b72d424736e138

Pretty much copied and maybe renamed a few functions

:kissing_heart:

Also line 87 to the end are some utilities you can probably ignore (or use)

are you going to PR this :0 ?

I didn’t really try to understand what it was doing and most of the deps were obvious

Bleh I probably should.

:bow: please :laughing:

It looks like it’s really gzip format, so you want gunzip-through-ports instead of inflate. The inflate expansion will be used inside gunzip-through-ports , but there’s gzip metadata.

Oh, I should scroll down…