Atomic, Not Binary: Opaque Blobs in VCSs

2025-08-18T00:00:00+00:00

a bit of background</h2>
One of the most commonly discussed issues with git</a>1</a> is the storage of binary files.
There are, broadly, two halves to this problem.
The fist half is that binary files tend to be large, and git stores a full snapshot of a file every time it changes, so frequently changing files can cause a repository to balloon in size. This is especially inconvenient if most people cloning the respository do not care about those files.
git-lfs</a>, a git plugin written by GitHub, attempts to resolve this, but is a bit clunky, and ends up being somewhat difficult when multiple hosts are involved. New work</a> is being done in git to alleviate the need for plugins and help make git more usable with large files natively.
But, crucially, this is not only a problem for binary files -- large text-based files also suffer from this problem (consider, for example, large generated json files). Nor is in inherent to binary files -- small binary files work exacty the same as small text files here.
So, we come to the second half: diffing. Despite git's internals working with snapshots, git's user interface largely deals with diffs. Diffs are how we review changes to files when sharing them, how we check what's in a commit, and how we remind ourselves what we were working on when we get bored with a project and abandon it for 2 years, only to pick it back up as a way to cope with anxiety about the clusterfuck going on outside 2</a>.
A common talking point is that binary files are not diffable. This presents a problem to a dVCS: if you can't diff a file, you can't merge the changes to the file if two or more people make changes independently. While traditional cVCS</abbr></dfn>s3</a> use file locking to get around this problem 4</a> 5</a>, file locking isn't feasible in a dVCS; one of the main points of a dVCS is that you don't need or have a centralized authority to mediate changes and file copying 6</a>.
And so, if you have binary files in your repository, you're relegated to choosing to either abandon your own changes or overwrite someone else's when applying conflicting patches. Or, manually coordinating with the people who might also be working on those files. Which... sucks. It's not great.
But! this is also not a problem inherent to binary files. It just happens that our main tool for diffing is... outdated.

let's talk terminology</h2>
Let's take a step back. When we say a file is diffable, what do we mean? A naive answer would be to say:

Given a file with 2 revisions (A and B), we call that file "diffable" if and only if we can use our diff tool to produce a patch</dfn> that describes the changes to get from A to B. </blockquote>
This is a clumsy definition, but it's a start. Let's see if we can improve it a bit. First of all, that last clause "changes to get from A to B" has a bit of a hole in it: technically: a diff that says "replace the entire contents of the file" meets this criterion. So instead, we'll talk about the minimal set of changes</dfn> required. For our purposes, we'll define that as "the smallest possible patch" 7</a>.

Given a file with 2 revisions (A and B), we call that file "diffable" if and only if we can use our diff tool to produce a minimal patch that describes the changes to get from A to B. </blockquote>
Getting better, but there's another hole here: we have an ambient dependendency in our definition: "our diff tool". Let's close that up: instead of saying that a file is "diffable", we'll say that it's "diffable under a tool":

Given a diff tool `diff</code> and a file with 2 revisions (A and B), we call that file "diffable under diff</code>" if and only if we can use diff</code> to produce a minimal patch that describes the changes to get from A to B. </blockquote>`
`Alright, now our definition is shaping up. Notice that we no longer talk about whether a file is diffable in abstract -- instead, we talk about how it relates to a specific tool. Let's do one final revision, though: let's define how we measure minimal. We'll say that our diff has a "unit" (for example, a line), and each piece of the diff cannot be smaller than the unit. Then:`
`Given a diff tool diff</code>, a "smallest possible change" δ, and a file with 2 revisions (A and B), we say that a minimal patch is a patch consisting of the fewest multiples of δ to describe the changes to get from A to B, and we call that file "diffable under diff</code> measuring with δ" if and only if we can use diff</code> to produce such a minimal patch. </blockquote>``Now, you might be saying "why is that important?". Well, a) it's ✨foreshadowing✨, but b) assume we're still talking about only text. Suppose we have a diff tool that produces line-by-line diffs (like we're used to with the standard git-diff</code> output) but only understands windows line endings8</a>. Per our definition above, files written on linux would not be considered diffable under that tool; diff</code> would consider the whole file to be one giant line, and produce a non-minimal diff consisting of the entire file 9</a>.`
`There's another problem with our definition that's lurking just outside the bounds of our terminology: we need a way to compose diffs, an that involves merging, and merging involves both humans and computers.`
let's talk about people</h2> Consider a giant json lockfile file that's not well-formatted for humans. We could produce a diff of this using our standard git-diff</code>, and it might very well be minimal while measuring lines, or even close-to-minimal while measuring with json nodes. but! such a patch would not be particularly readable, and given the types of automated changes made to such a file, it's probably not particularly mergable by line-wise by computers with our typical merge algorithms. Doubly so for humans. Nor does it particularly make sense to merge it as such -- it's generated by a tool, so it likely just makes sense to regenerate it 10</a>. Now, this is a bit fuzzy, but all of computing is a bit fuzzy, because at the end of the day it's humans using the computers. Consider an SVG file or a .ma file</a>. It might be formatted nicely, we might be able to get a minimal diff measuring both line wise and element-wise. But, merging it doesn't really make sense: changes to the text of a graphics file format are not particularly reviewable as text, and merging two sets of changes often produces nonsensical results -- the relationship between parts of the file is visuospatial, not textual. These files are diffable under git-diff</code> under our previous definitions, but doing so doesn't really make a ton of sense. We need some new terms. let's talk terminology, again</h2> Let's consider a rust source file or blog post in markdown, and compare it to our json lockfile and svg file. Both types of file are diffable, but the source file and markdown file and be broken up into chunks that we can safely re-combine as humans, while the lockfile and svg cannot. Let's call the source file and markdown file compound file</dfn>s, and the svg an atomic file</dfn> (we'll get to the lockfile in a moment). We'll say: Given a file with two revisions A and B, we call the file "compound" if it can be split into chunks such that we can combine chunks from A with chunks from B and still produce a coherent file. An atomic file any file that is not compound. </blockquote> Let's consider another example: a pallette of tiles for a 2d video game. In theory, it is safe to work on one tile while someone else works on another, even though all tiles are stored in a single file. Thus, we can say that a tileset file is compound, with its chunks being the individual tiles. Doing so, however, would need a specialized diff tool that worked with tileset images. Recall our definition of "diffable" -- a tileset is diffable (measured in tiles) only under a diff</code> tool that understands tileset images. Let's revisit our lockfile: it's not particularly easy to combine changes text-wise, so usually we'd just regenerate it when a merge conflict happened. However, it'd be nice if our lockfile generator could read the old lockfile and leave any relevant locked packages in place, only deviating when a package specifier was changed, added, or deleted from the main dependency file. That'd get us a mostly reasonable solution, but what if instead, it could take both old files from each revision, and only complain if the revisions specified mutually incompatible package revisions? Then we'd get a perfect new lockfile, without any risk of accidentally introducing package revisions we did not intend to. Such a program is a merge tool</dfn> -- instead of considering changes textually, it interprets them semantically. Similarly how to we said that a compound file is diffable under a given diff tool, diffs for a compound file can also mergable</dfn> under a given merge tool. a case for why this matters</h2> Our tileset example above might be a bit academic at this point, given the existing processes and issues around gamedev SCM</abbr>, but changing how we think about diffing and merging is also applicable to every-day programming. Merging lockfiles is a task causes issues today, and while line-wise diff/merge tooling largely works for many programming languages, AST-based tools like difftastic</a> can provide a much nicer experience. Furthermore, abandoning the preconception of "binary means non-diffable and non-mergable" paves the way for us to think about programming languages which store binary ASTs instead of text files, annotated with additional information (consider a language that pre-computes and stores typechecking like Unison does</a>, but does not need a whole specialized VCS/SCM to manage it, just some standard plugins). Just like how it's become common for programming languages to ship language servers as a standard piece of dev tooling, we could have a future in which syntax-aware diff and merge tools ship standard as well, and where our VCSs handled compound binary files (using standard plugins shipped with the tooling to edit those files) just as easily as they handle compound text today. Our tools could be a lot better, we just need to think in the right frame of reference. ^{1 and often other dVCS</abbr></dfn>s </div>}^{2 what, don't look at me like that, you know it's true </div>}^3e.g. Perforce</a>, the favorite VCS of gamedevs. </div> ^4For example, this is how Perforce works</a> (permalink</a>) </div> ^6there is an argument that in this day and age GitHub effectively functions as such a centralized authority. Whether or not that's true, the mentality with which users use Git remains distributed: anybody can go off and start working on a patch on their own, then create a PR or send a patch in -- they don't need to somehow be blessed with write access to a repository to do so 11</a>. </div> ^{11 ...execpt in some corporate uses of GitHub, where forking is turned off. Then you need write access. That's a bad model that discourages transparency and collaboration between teams at a company. But that's a blog post for another day. </div>}^{7 yes, yes, i know i'm still handwaving a bit. One definition here might be: if we say that we have a patch language consisting of an operation change(range-from, range-to, contents)</code> that change chunks of the file at the given ranges, we can say that the total size of of the patch is the sum of the size of the contents</code> of each operation in the patch. There's still a hole here, but we'll get to it in a minute. </div>}^8that is, \r\n</code> instead of just \n</code> on linux, or \r</code> on some very old systems. the distinction these days is mostly 12</a> a historical artifact. </div> ^12...mostly. brief aside: \r</code> stands for "carriage return" (sometimes written CR</code>), and \n</code> stands for "line feed" (sometimes written LF</code>). Back in typewriter days, carriage return moved the typewriter carriage back to its starting position, while line feed scrolled the paper down a line. This was carried over into the teletype era 13</a>, and then into the modern era. These days, very few things treat \r</code> and \n</code> separately. However, the sixel</a> graphics API that originally appeared on DEC terminals retains a separate concept of carriage returns and line feeds -- you write all the pixel positions of a single color in one go, and then CR</code> back to the beginning of the line and write pixel positions for a new color, and so on, only LF</code>-ing when you're done with a line. </div> ^13per the vim book</a>, page 120, apparently part of this reason this was carried was to prevent smudging -- the extra time it took to transmit the LF</code> after the CR</code> gave the printer time to finish returning the print head. CR</code> was also apparently involved in overwriting to make bold text. </div> ^{9 "wait", i hear you asking, "wouldn't that patch still be minimal if our 'lower bound' is still lines?". Sure, but crucially, the lower bound of our measurement might not be the lower bound our diff tool uses. Here, we consider a line terminated by any newline to be our lower bound for measuring the minimality of a patch, while our imagined windiff</code> only considers \r\n</code>, so windiff</code> does not perform favorably. </div> ^{10 with the caveat of "regenerating it might produce slightly different results than some theoretical merging", but often the results are identical or close enough since the regeneration can be done by feeding in one of the two original files as a base. Ideally, we'd be able to feed in both files. We're getting there. </div> ^{5 i have been informed by one of my awesome gamedev friends that file locking is also used in gamedev to prevent changes to approved assets. this is an interesting workflow to me, because it is somewhat antithetical to the way i am used to thinking about code and releases -- in my usual realm, approval comes from reviews, and approval management is done with owners/permissions lists (and it's just assumed that if you're an owner you're going to do a reasonable job). but! my understanding is that gamedevs often work under a very different set of constraints, and it's always interesting to see the ways that affects tooling and processes. </div>}}}

Other Poetry I Like

2024-11-21T00:00:00+00:00

Just doing a quick archive, cause a lot of this stuff is from my cohost, which will shut down soon. Stuff with semi-durable links will be linked, stuff with transient links (e.g. the bird site) is transcribed so that it doesn't disappear into the ether.

If you're on this list and you have a more canonical location, i'm happy to link there instead.

Stories of My Friend M

2022-03-29T00:00:00+00:00

CW: Grief, Death / Background</summary>
One year ago today, one of my best and longest friends passed away suddenly. This is a eulogy I never got to give. </details> </section>
We told stories, my friend M and I. All sorts of stories, over and over, for almost 10 years: stories of fantastical places, stories of our lives, stories about us telling stories, stories with friends.
There’s a saying we have in Judaism when someone has passed away: Zichrona Livracha (זכרונה לברכה)1</a>. It means "may her memory be a blessing” – one of the myriad forms of "nobody is truly ever dead, as long as we keep telling their stories”. In that spirit, I’d like to tell you a bit about M, tell you pieces of her story, with some meandering miscellaneous lore along the way 2</a>.
I don’t remember the exact moment I first met M, but I can tell you for certain it was at university, in one of two places: a Science Fiction Society (SFS) meeting, or in the Octawedge. The Octawedge was not, contrary to its name, wedge-shaped – rather, it was an octagonal room in the campus center, a replacement "hanging out area” for SFS folks, the old "wedge"3</a> having been deprecated long since. SFS folks would hang out there, clustered around Rounds 4</a> in large, comfy chairs by the fireplace, hacking on personal projects, working on homework, debating sci-fi, telling stories, or generally just being around each other.
M was very active in the SFS – she was what one might call a classic High Nerd: she could debate Tolkien at length, tell you all about her favorite classic X-men comics. She loved Inu-Yasha & The Last Unicorn, Elf Quest and fanfics, Star Trek & the video game Pharaoh. She could regale you with stories about different types of LARPs, or modern, slightly tongue-in-cheek retellings of Jewish myths. She was also a rocket scientist 5</a>, roboticist, and software developer, as well as multi-time SFS officer 6</a>. In addition to all that, she was a wonderful friend, always ready with advice if I wanted it or ways to cheer me up if I was down.
While I met M via the SFS, I suspect the point at which we became fast friends was via a tabletop RPG. M’s boyfriend (now husband), A, and I had already been in a different RPG together & knew each other from the SFS, and so when A decided to run a new Eberron-based DnD game, I signed up, along with our mutual friend, D. From then on, I spent many a weeked at A & M’s apartment, eating takeout 7</a>, playing DnD, telling stories, and basking in the presence of M’s glorious cat, nicknamed The Floof 8</a>.
Life progressed after uni, but storytelling stayed constant – A & M got married at an amazing Steampunk-themed wedding, and A, M, D, & I played DnD afterwards (A & M actually met in another game run by D, and started dating in-game before doing so IRL). A & M lived pretty close to me, so I’d often spend weekends at their apartment. M would give A & me little grocery "quests”, with silly doodles & puns as "quest rewards”. We’d go out to eat or cook 9</a> in 10</a>, play games, watch movies, and debate all sorts of things 11</a>.
And, of course, we continued to tell stories: A’s & D’s games both continued, and we added a Shadowrun game run by A & M to the mix to fill spaces between the others. Even when I moved to CA, we continued to tell our stories online, and I’d occasionally fly back and crash at their apartment for a week, working during the week and then having a marathon game session all weekend.
I’d like to say we got to see how the grand, sprawling epic fantasy story that A wrote ended, to say that our characters retired happily in Shadowrun, having worked out all their mysterious pasts and life goals, I’d like to say that A & M finished saving the world from the consequences of saving the world the last time in D’s game. But those of you who play know: runners rarely get to ever retire, not every hero makes it through the final confrontation with the BBEG alive, and, unfortunately, sometimes players suddenly have their stories cut short. One year ago today, M passed away, suddenly, in the middle of the night.
There’s so many more stories I could tell you about M: about how she hated spoilers with the fiery passion of a thousand suns; about how she earned her SFS officer nickname of God Shampoo; about how, after getting her US citizenship, she wanted to celebrate the most ridiculously stereotypically American way she could think of: eating burgers & going shooting. I could tell about the times she & A dropped everything and drove 40 minutes to take me out to dinner when I had a rough day. Alas, there’s only so many places to stick anecdotes in a blog post before it becomes oversaturated.
All those stories, however, all the stories we shared, live on, whether they be epic game moments, goofy IRL stories, or any other myriad of wonderful times. When I think back on those stories, I smile; when our friend group regale each other with the stories, we laugh. And in this I think, I can say, for certain: to me, M’s memory is a blessing.
^{1
often just z”l or ז״ל
</div>
^{2
one of the things M could discuss at length was Tolkien, so I’m sure the
meandering lore would be fine 😉
</div>
^{3
which was, in fact, wedge-shaped
</div>
^{4
varying-sized odd cylindrical tables
</div>
^{5
I mean this somewhat literally: she studied aeronautics, and eventually
got a PhD in it
</div>
^{6
As you might ascertain by that list, M had an aura of Getting Shit Done.
</div>
^{7
this was how I discovered that Indian food was, in fact, delicious
</div>
^{8
The Floof was also known as Chicha (formal), The Cheech (semi formal),
and a myriad of other appellations. She was a beautiful long haired ragdoll
with zero sense of depth. At one apartment, she would run in circles around
us while we played & then shoot off down a hallway like a particle
accelerator, only to be stopped by an unfortunately placed door.
</div>
^{9
M loved my beer Mac and Cheese recipe, so it was not uncommon for it to
Be Declared that we were having mac & cheese when A came to pick me up
</div>
^{10
Once, A & I covered their kitchen with drying pasta. M got home,
sighed, shook her head, and then rounded us up to go out for dinner
</div>
^{11
M was also Jewish, so we did a good deal of Jewish arguing, which drove
A nuts. He eventually learned to appreciate Jewish arguing… like 2 years
ago.
</div>}}}}}}}}}}}

Metamagical

Atomic, Not Binary: Opaque Blobs in VCSs

Other Poetry I Like

I Want to Say Your Name (TDoR 2024)

TDoR 2024

Walk (A Reminder to Myself)

Summoning Some Nix Daemons While Wearing a Hat

Stories of My Friend M

So long, Kubernetes