Squeezing 2.33 GiB of Phigros charts into 46 MB, losslessly

phigros.tools started as a save analyzer. You log in with TapTap (or plug your phone in over WebUSB, or drop a file), and it shows you your ranking score, your Best 27 + 3 Phi, what to grind next, and so on. Everything runs in your browser.

Then I wanted a chart browser. Then I wanted to watch the charts. So now there’s a canvas player at /charts/<id>/play/<level> that renders any chart in the game, with the game’s own note sprites and hit sounds, close enough to the real thing that I checked it frame by frame against screen recordings.

The player needs the charts. That’s where this post starts.

What a Phigros chart is

If you’ve never played it: Phigros is a rhythm game where notes fall onto judgment lines, and you tap them when they reach the line. The twist is that the lines aren’t fixed. They slide across the screen, spin, fade in and out, and change scroll speed, all synced to the music. A lot of the game’s identity is in that choreography. The easiest way to get it is to watch one: here’s Horizon Blue (IN), at 1:30, running in the player this post is about.

A chart file describes all of it. It’s JSON, roughly:

chart
├── formatVersion, offset
└── judgeLineList[]
    ├── bpm
    ├── notesAbove[], notesBelow[]
    ├── speedEvents[]              how fast notes scroll
    ├── judgeLineMoveEvents[]      where the line is (x, y)
    ├── judgeLineRotateEvents[]    its angle
    └── judgeLineDisappearEvents[] its opacity

Notes belong to a line, above or below it. Each has:

• a type: tap, drag, hold or flick,
• a time,
• a positionX along the line,
• a holdTime,
• a speed,
• a floorPosition: how far along the line’s scroll the note sits, which decides when it appears on screen.

A hold note looks like this:

{
  "type": 3,
  "time": 2048,
  "positionX": -1.5,
  "holdTime": 64.0,
  "speed": 1.0,
  "floorPosition": 11.428572
}

Events all have the same shape: a startTime, an endTime, and a start and end value, linearly interpolated in between. Move events carry a second pair (start2/end2) for the y coordinate. Speed events have a single value instead:

{
  "startTime": 2048.0,
  "endTime": 2049.0,
  "start": 0.5,
  "end": 0.50390625,
  "start2": 0.25,
  "end2": 0.2519531
}

{
  "startTime": 0.0,
  "endTime": 4096.0,
  "value": 2.2
}

The first is a move event: the line glides from (0.5, 0.25) to (0.5039, 0.2520) in one tick. The second says notes on this line scroll at speed 2.2 for 4096 ticks.

Times are in ticks, 32 per beat, so a tick lasts 1.875 / bpm seconds. Everything else is a float.

The problem

Phigros charts live inside the game’s APK, in Unity asset bundles managed by Addressables, as JSON. Pulling them out is mostly bookkeeping: parse the Addressables catalog, notice that each chart’s internal_id has been mangled while dependency_key still points at the real <hash>.bundle, open the bundle, grab the TextAsset.

That gives 1009 charts. Together they weigh 2,499,346,022 bytes. The median chart is 1.4 MiB, the biggest (Retribution and Exoplanetary Mirage, IN/AT) are 14 MiB each.

You can’t send a 14 MiB JSON file to someone’s phone because they clicked “play”. You also can’t commit 2.33 GiB of JSON to git and push it to GitHub, not if you want to stay friends with either of them. The obvious fixes:

What I tried	Size
Raw JSON	2.33 GiB
gzip	~309 MB
Reshape into compact arrays, raw	1.16 GB
Compact arrays + gzip	255 MB
Plain LZMA	~120 MB

Reshaping the JSON barely helps once a compressor is involved. The redundancy isn’t in the syntax.

I had one hard rule: the output must decode to exactly the original charts. Not “visually identical” or “within a pixel”. If the raw file says 0.23036835, the decoder produces 0.23036835. My first ideas were lossy anyway: round positions to 1/100,000 of the screen, simplify curves as long as they stay close enough. They’d have been smaller, but I don’t want a chart viewer that quietly changes the charts. So I needed a real codec, and a lossless one: binary is fine, loss isn’t.

I gave myself a budget of 50 MB or less.

What’s actually in a chart

Before compressing anything, it’s worth counting. Across the whole set:

• ~650k notes
• ~160k speed events
• ~28.5M judge-line events: 13.9M move, 6.9M rotate, 7.7M alpha

Notes are what you think of when you think “chart”, and they’re a rounding error. Almost all the bytes are judge lines being moved, spun and faded.

A few probes over the data told me most of what I needed:

87% of events last exactly one tick. 24.7M of 28.4M. Chart editors don’t store “ease from A to B over two beats”; they bake the curve into a long run of tiny linear segments.

Events are glued together. Of 28,446,823 consecutive pairs, all but 8 are contiguous in time (each one starts when the previous one ends). And 97.2% are contiguous in value: the next segment’s start equals the previous segment’s end.

The numbers are f32s printed by Ryu. The JSON came out of Rust’s serde_json, which formats floats with Ryu: an algorithm that prints the shortest decimal string that parses back to the exact same bits. The f32 nearest to 0.1 comes out as 0.1, not 0.100000001490116. That matters because it means I don’t have to store text. If I store the exact f32 bits, I can regenerate the exact same characters. A small Python writer reproduced 40/40 sample files byte for byte.

Floats are the hard part

Integers are easy to compress losslessly. Floats aren’t, and “lossless” makes them harder in a few ways:

• Subtraction isn’t reversible. The usual trick is to store value − prediction. With floats, prediction + (value − prediction) doesn’t always give back value, because each operation rounds. One lost bit in the last place and the chart is no longer the chart.
• Floats aren’t associative. (a + b) + c and a + (b + c) can differ. So any computation the encoder does, the decoder has to repeat with the same operations in the same order, even in a different language.
• Text isn’t values. The file stores decimal strings. 0.1, 0.10000000149011612 and 1e-1 can all be the same f32. To promise identical files, I either store the text or find a rule that regenerates it exactly.
• Edge cases are real. The raw files contain both 0.0 and -0.0. They compare equal as numbers, but they’re different bytes, and I had to keep both (just in case).

The answer to the first two was to stop doing arithmetic on the stored values. Every float is kept as its exact 32-bit pattern. Predictions can be computed however I like, but the difference is taken between bit patterns, as integers, which is always reversible. More on that below.

So a typical chart is: long chains of one-tick segments, each starting where the last one stopped, tracing a smooth curve. The only real information per event is “where does the curve go next”. That’s a prediction problem.

Predicting the next value

Every event’s end value gets a guess, and only the difference between the guess and the real value is stored. The guess can only use what the decoder will also have when it gets there: the previous points on the same curve. Here are the candidates:

• Order 0, previous value: the curve stays where it is. Guess = the last point.
• Order 1, linear: draw the line through the last 2 points and follow it one more tick.
• Order 2, quadratic: same, with the parabola through the last 3 points.
• Order 3, cubic: the cubic through the last 4.

“Take the polynomial through the last k + 1 points and evaluate it a bit further” is Lagrange extrapolation. For points (t_j, v_j):

P(t) = Σ_j  v_j · Π_{m≠j} (t − t_m) / (t_j − t_m)

Each point’s value is weighted by a product that equals 1 at its own time and 0 at every other point’s time, so the sum passes through all the points. With one point it’s flat, with two it’s a straight line, and so on.

No single order wins everywhere. Smooth easings want high orders; corners, where one easing ends and the next begins, want low ones. So the codec computes all four and uses whichever was closest on the previous value. Curves change character slowly, so the last winner is usually a good bet. Nothing is stored about which order was used, since the decoder can make the same choice from data it already has.

If you’re wondering how the four get computed: bluntly. Every event, each order is evaluated from scratch with the textbook formula above, over a history of at most 4 points. That’s 20 multiply-divide pairs in total, which is nothing next to the range coder. The history is cleared whenever a value jumps instead of continuing, so a new curve never extrapolates from the old one. Newton’s divided differences would be the smarter way: the Newton form builds order k + 1 on top of order k, so adding a point costs O(1) instead of a full recompute. It’s the same polynomial on paper, but not in floating point: it rounds differently, and the decoder has to reproduce the encoder’s arithmetic bit for bit, in another language. The naive loop is trivial to port operation for operation, and at 4 points speed doesn’t matter, so it stayed.

I measured how many bits the end values cost under each option (order-0 entropy of the residuals, i.e. the best size you could hope for coding each one independently):

Predictor	Size
Previous value	60.8 MiB
Linear extrapolation	56.2 MiB
Quadratic	46.7 MiB
Adaptive order	42.8 MiB
Adaptive order + context	31.1 MiB

Per value, going from quadratic to adaptive-plus-context took move x from 8.9 to 5.9 bits, move y 6.6 → 4.2, rotation 12.7 → 8.7, alpha 11.5 → 7.9.

Taking the difference without rounding

Now the problem from earlier: the prediction and the real value are both floats, and subtracting floats rounds. Here’s a real f32 case where storing value − prediction doesn’t survive the round trip:

encode:  0.13436425 − 1847.4337 = −1847.2993
decode:  1847.4337 + −1847.2993 = 0.13439941

The subtraction had to round to fit an f32 near 1847, and the lost digits don’t come back.

The fix is to stop treating them as numbers. An f32 is 32 bits: a sign, an 8-bit exponent and a 23-bit mantissa, in that order.

So for positive floats, reading those bits as an unsigned integer gives the same order as the float values: the next representable float up (one ULP away) is exactly the next integer. So the prediction (computed in float64, rounded to f32) and the real value become two integers, and the residual is their integer difference.

encode:  1056964612 − 1056964609 = 3
decode:  1056964609 + 3          = 1056964612

Integer subtraction never rounds, so this always reverses. It also makes the residual size mean the same thing everywhere: “3” is three floats away from the guess whether the value is 0.001 or 40,000.

Negative floats need one more step, because their raw bits count the wrong way (a bigger magnitude gives a bigger integer). The codec flips them, so the whole float line maps to a single sorted integer line, with -0 sitting just below 0 as its own distinct value.

Coding the residuals

Prediction turns the chart into a stream of mostly small integers. Something still has to turn those into as few bits as possible, and that’s the job of an entropy coder.

I used a binary range coder (a practical form of arithmetic coding), from the same family as LZMA’s. The idea: the whole file is one number in the interval [0, 1). For each bit you encode, split the current interval in two, sized by how likely you think a 0 is, and keep the half matching the actual bit. A bit you predicted with 99% confidence shrinks the interval by only 1%, which costs about 0.015 bits of output. A surprise costs a lot. The decoder makes the same predictions, so it can follow the same splits back.

“How likely is a 0” is a probability, one per kind of decision, and each one learns as it goes: after every bit, it moves a little toward what just happened. Mine starts at 50/50 and moves fast at first, by 1/(n+2) of the gap after n updates. Once n reaches 30 the step stops shrinking, so the probability can still follow changes later in a chart. The coder uses only integer arithmetic, which matters later.

That’s how a single bit gets coded. A residual is a whole integer though, so it has to be turned into a series of yes/no questions, and the choice of questions decides how much the probabilities can learn.

Residuals are mostly tiny but sometimes huge: the same stream holds 0, 3 and 40,000. The informative part is their scale. “Was the last one small? Then this one probably is too” predicts well. The exact low digits of a big residual, on the other hand, are basically noise. So the questions go from most to least predictable. First the scale (how many bits long is it?), then the sign (close to a coin flip, but it leans a little on the previous sign: that context alone saved about 1 MiB), then the first two bits below the leading 1 (still a bit skewed toward small values). The rest is noise, so it’s written at a flat 1 bit each, with no model to maintain. It’s the idea behind Elias gamma and Exp-Golomb codes, except every question here has its own learned probability.

A residual is written as:

1. its magnitude bucket (bit length, 0–63) through a 6-bit binary tree (one adaptive bit per level),
2. a sign bit,
3. the top 2 mantissa bits with adaptive probabilities,
4. the remaining low bits nearly raw.

The probabilities are picked by context: the size of the previous residual in the same stream (capped at 24), plus the predictor order and the previous sign for motion values. If the last residual was tiny, this one probably is too, and the coder learns that per stream.

Why 24? I never justified it at the time, so I measured it afterwards, encoding the same 120 random charts with different caps:

Cap	Size
8	4,825,817 B
12	4,701,791 B
16	4,625,432 B
20	4,603,536 B
24	4,594,893 B
32	4,592,669 B
64	4,592,911 B

Too few contexts, and small and large residuals share the same probabilities. Too many, and each probability sees so few bits that it barely learns (and the tables grow: the motion-value channel alone has 4 × cap × 2 contexts). The curve flattens around 20. Going from 24 to 32 saves 0.05%, and 48 or 64 change nothing. So 24 sits just past the knee: not magic, but not a random number either (I totally picked it randomly).

The structure around the residuals uses the facts from earlier:

• Time: one flag bit for “starts where the last one ended” (it almost always does). Durations are coded as a delta from the previous duration, so a run of one-tick events costs next to nothing.
• Start value: one flag bit for “equals the previous end”.
• End value: predicted as above, residual coded.

Notes

Notes are small, but a few tricks still paid off:

• Positions repeat. A 32-entry move-to-front cache of recent X positions catches 78.4% of notes. A hit costs a flag and a 5-bit index; a miss falls back to a delta.
• Hold durations are usually whole ticks, so they’re coded as integers relative to the previous hold.
• floorPosition is the one I like most. Every note stores how far along the line’s scroll it sits, which is a big, ugly, non-repeating float. But it isn’t independent data: it’s the integral of the line’s speed events up to the note’s time. So the codec integrates the speed events itself (t = tick × 1.875 / bpm), predicts the floorPosition, and stores only the leftover. Same for the rare speed events that carry their own floorPosition.

How it went

Step	Size
First model, full set	75.8 MiB, 351 s
Fixed a bug in the curve-point history	45.4 MiB, 32 s on 32 cores
Position cache, handling for odd files	44.0 MiB (46.2 MB, 54x)

Things that didn’t make sense, or didn’t work:

• Bias correction on the predictor: 71.1 MiB. An outright disaster.
• Restarting the curve history on every jump instead of keeping both sides: 47.7 vs 39.8 MiB in that experiment.
• Resetting probabilities per chart with a fixed adaptation speed cost about 2 MiB. Switching to count-based adaptation (fast at first, settling after ~30 updates) won it back and then some. That’s the RATE_LIMIT = 30 in the decoder.
• Caching gotcha: at one point a stale Numba cache made two different experiments produce “identical numbers to three decimals”. Always suspect the cache.

Five files were weird: they print every float in float64 style with odd trailing digits, so exact-f32 regeneration doesn’t reproduce the text. These get a small text patch appended after the model payload. The browser skips the patch, since it only restores formatting, not values. One file (QZKagoRequiem HD) has long exact repeats and briefly compressed better with plain LZMA; I dropped the LZMA path anyway so the browser wouldn’t need an LZMA decoder, at a cost of about 4 KB.

The easing curves are the one thing left on the table. The baked samples match start + (end − start) · ease(p) exactly at quarter points, but not in between, so I never found the exporter’s exact arithmetic. If someone does, most of the motion data becomes “this easing, from here to there”, and the file shrinks to a fraction of its current size. That’s the remaining big lossless gain.

Verifying it

“Lossless” only counts if it’s checked, so it’s checked on every build:

• The Python side decoded all 1009 charts back to byte-identical JSON.
• The TypeScript decoder (phc.ts, the one the site ships) decodes every chart and compares every key and every number, as f32, against the raw file. It also checks that each note’s stored floorPosition agrees with the scroll the player integrates from the speed events.

Getting the TypeScript decoder to agree bit for bit was mostly discipline. The range coder is integer-only, so it ports directly. The predictors aren’t, so the port does every float64 operation in the same order as the Python encoder. Floating point isn’t associative, and a reordered sum is a different chart.

In the browser

The biggest charts went from 14 MB of JSON to 276 KB downloads. Fetch, decode and parse takes about 85 ms for those, around 1 ms for small ones. The whole library is 47 MB on disk, in 1009 .phc files.

The player on phigros.tools uses them directly. Pick any song, any difficulty, press play. You can also share a link to a specific moment with ?t=.