Emi's Blog

<- Back to blog

Inching Over the Finish Line 60 Times: Gradle, Mappings, and Maven

I spent a weekend and more in what I can only affectionately refer to as the Gradle and Maven trenches. But first, some context. At time of writing, I'm finishing up the process of porting my mod, EMI, from being Fabric exclusive to Forge and, more broadly, to an xplat build system. This has involved a lot of technical work in a lot of areas, but most of it is not relevant to my recent goal: publishing artifacts and sources for EMI on Maven. As an xplat mod and API, I need to publish different jars for Fabric/Quilt consumers, Forge consumers, Intermediary xplat consumers, and Mojmap xplat consumers.

Four different distributions?

Yep! Though, to understand why, we're going to have to talk about mappings. If you know everything there is to know about mappings, you can skip past it to Publishing.

Mappings

Minecraft is released as a jar file, and for basically its entire life has had its class, method, and field names obfuscated using ProGuard. This process minimizes the names of all symbols to an extreme degree. Everything is a short string of letters, most methods are named simply a. Worse yet, there's no consistency, a class called faa in one snapshot can be ddq in the next.

Early modding interfaced with these names directly, manually replacing class files and delicately calling all the impossible to understand names. Naturally though, this is infeasible, and a solution needed to be developed. The idea is simple; some modding collective like Forge gets together when Minecraft releases a version and creates a file that maps every Official (obfuscated) name to something that makes more sense. Then, when modders mod the game, they transform the Minecraft jar file to this mapping so they can understand what they're doing, then when they build their mod, the build system unmaps it back to the official mappings, and everything is great!

Haha, it's not that simple. Mappings solve the issue of understanding what you're doing, but this still presents two problems:

• Everything breaks on minor versions, because the Official names have changed.
• If something has a bad name, you have to wait to change it.

To solve this, at the start over every version an intermediary layer is created that contains names that are still mostly nonsense, but can be used as a source of truth. There are also efforts to keep what these names point to the same, so the intermediaries are incrementally generated every release to try and track changes. Modloaders, like Forge, Fabric, or Quilt, will generate their intermediary layer, and use this to remap the game at runtime from the official names. Mods are written with whatever updated version of the mappings with sensible names exists, and will be compiled to the intermediary, and loaded alongside the base game. Voilà! We've overcomplicated everything, but now we have a working system, let's put a name to all the pieces in the modern ecosystem.

Wait a second, Mojang's official mappings?

Yeah. While you're at it you might also be concerned about Forge seemingly having two intermediaries? Well it's the result of mixing Mojmap class names with SRG methods and fields. Before Mojmap existed, it mixed Official obfuscated class names with SRG methods and fields instead. And doesn't seem to have a name? I've seen people call it "FART" in reference to ForgeAutoRenamingTool, but I'm going to go with Mixed SRG.

Things probably still aren't super clear. Let's see where things are being used.

Wait why's Quilt using Intermediary, doesn't it have Hashed? Also what's VanillaGradle and why does it use a dev mapping as an intermediary format? I appreciate the observations, but we should take a step back.

The textile loaders (Fabric and Quilt) use Fabric's incremental Intermediary, which is what the loaders map Official names to at runtime. Fabric Loom allows you to use arbitrary mappings, primarily Yarn or Quilt Mappings respectively, at dev time. Yarn and Quilt Mappings are updated several times each version, and developers can use whatever version they want. Mods are built from named mappings into Intermediary, the consistent names for the version.

Forge generates an incremental SRG intermediary, and mixes that with Mojmap class names to create Mixed SRG, which is what the loader maps Official names to at runtime. ForgeGradle only allows you to use Mojmap. Mods are built from Mojmap into Mixed SRG.

No seriously what's Mojmap?

Oh right, so for a while the loaders were doing their thing, back when Forge used MCP as dev mappings and Mixed SRG used Official class names mixed with SRG. The date that would go down in history was September 4th, 2019, 8 years after the game's official release, and 9 after the game's alpha release, Mojang released snapshot 19w36a. Inside of this snapshot was a brand new change, Mojang still uses ProGuard to obfuscate the game, but now releases a file that maps all of the Official names to the original names used by the developers.

This set of mappings is often called Mojmap, though it's worth noting that it has no official name. Some people also call it the "Official Mappings", but that can be confused with the Official obfuscated mappings, and Mojmap sounds cooler. Why didn't they just stop obfuscating the game you ask? Well that's not clear, but considering Mojmap is under a restrictive, custom license, it's probably safe to assume the reason is legal.

Then why doesn't everybody use that?

That's a complicated question. The answer is a combination of historical, legal, and social factors. The most notable legal portion of Mojmap's license, last updated in 21w03a in early 2021 is it does not allow distribution of the mappings "complete and unmodified".

Forge is reluctant to use Mojmap at runtime, and compromises with Mixed SRG using only the class names, and using SRG for the method and field names. Modernly, SRG utilizes Mojmap to match names. At dev time, however, ForgeGradle uses Mojmap.

Fabric's Intermediary is completely independent of Mojmap, incrementally generated each snapshot using Matcher to try and keep the names on the same symbols. This approach is reasonably effective at tracking methods even when their signatures change or Mojang renames them. Fabric uses Yarn for the development of its libraries. Yarn is semi-cleanroom set of mappings, backed by Intermediary. It does not allow references or "inspiration" from other sets of more restrictive mappings.

Quilt doesn't use Mojmap directly, and instead hashes it to create an intermediary. This intermediary is intended to eventually replace their usage of Fabric's Intermediary, but currently is only used for internal development purposes and not used at runtime. Unlike Intermediary, its generation is not incremental, but instead relies on Mojang not changing their names too often. Quilt Mappings is a set of mappings forked from Yarn with a less restrictive stance, allowing contributions derived from other sets of mappings, including Mojmap.

Both Fabric and Quilt use a version of Loom, which allows usage of arbitrary mappings, including Mojmap, though as they each have their own set of mappings, Yarn and Quilt Mappings respectively, Mojmap is not the default.

My thoughts on mappings

SRG is quite vestigial. It provides no clear benefit to Forge other than legal posturing and adherence to legacy tools.

As a format, Intermediary has a pretty reasonable leg to stand on for existing, being both non-derivative of Mojmap and having unique benefits for usage. While not without its flaws, Yarn's semi-cleanroom stance and contribution process sets it apart as a very high quality set of mappings. It's hard to imagine that community reverse engineered names could be better than the real ones used by the game's developers. However, when your goal is strictly making good names rather than the rest of the responsibility for the code, great things can happen.

Much of the things said about Yarn apply to Quilt Mappings. It retains many of the names, and can have a quicker and more informed iteration cycle with references to Mojmap. On the other hand, Hashed is a pretty disappointing format. It's most comparable to modern SRG, but without the legacy to back it up. Its primary benefit over Intermediary is not being Intermediary, as Quilt has an interest in reducing reliance on Fabric, and as Intermediary requires manual intervention for generation, requires a single authority. Quilt currently has one way compatibility with Fabric, allowing Fabric mods to load, and allowing Quilt devs to use APIs developed on Fabric. Switching to Hashed would break both of these without serious work, and would significantly complicate the publishing landscape of both the textiles and the general xplat ecosystems.

Publishing

Okay enough about mappings, you get the idea. What's the deal with publishing and xplat?

APIs in mods are published on Maven in the loader's respective intermediary format, Mixed SRG on Forge, and Intermediary on the textile loaders. ForgeGradle and Loom will remap the mods from the intermediary to the dev mappings when depended on. Xplat mods use build systems that lets them reuse common code that only depends on Minecraft, and have much smaller loader specific implementations. The two most common xplat build systems are Architectury and VanillaGradle.

Architectury uses a fork of Fabric's Loom to allow it to work with Forge. Because it uses Loom, it produces Intermediary binaries, and has specific tooling to remap the Forge output to Mixed SRG.

VanillaGradle on its own isn't a full build system, but rather a minimalist build system that only knows about vanilla Minecraft. However, it is the core of the MultiLoader Template. The MultiLoader Template uses VanillaGradle for the common xplat sourceset, and then uses Fabric Loom and ForgeGradle on the respective modloaders. ForgeGradle natively produces Mixed SRG binaries, and Fabric Loom natively produces intermediary binaries. So everything works!

Xplat artifacts

Xplat artifacts? Why would you want that? You can't run a mod without a modloader.

But you can depend on its API.

So it's time to talk about APIs and dependencies from the perspective of an xplat mod. Both are equipped to readily handle Forge and Fabric dependencies on either end, but there is another use case. For the more familiar, consider a mod like JEI, Patchouli, or Cloth Config. These mods release on both Fabric and Forge, and use an xplat build system. So, logically, if you're making an xplat mod, surely you don't need to have a dedicated Fabric and Forge implementation of these mods, you could just depend on them in your xplat module!

Well, I appreciate the enthusiasm. It is, of course, not that simple. The main question is, as an artifact producer, what do you export your artifact as so that xplat consumers can use it? Forge uses Mixed SRG, and Fabric uses Intermediary, which one do you pick for the common ground? Architectury and VanillaGradle raise their hands confidently at the same time and shout out completely different answers. Architectury uses Intermediary, it's a derivative of Loom after all, this is its natural format. VanillaGradle uses Mojmap. Hold on a second, Mojmap? That wasn't one of the options.

Oops! VanillaGradle is, by construction, supposed to be simple. It doesn't know what an intermediary is, it only knows what Minecraft is and how to get it to be Mojmap. Mojmap is all it knows, and all it wants to know.

Well that's a shame, each one has a completely incompatible intermediary, and both build systems are widely used. How are xplat APIs like Patchouli, JEI, and Cloth Config solving this issue then? All of these mods produce consumable xplat artifacts after all. Here's a list of the different strategies employed by these mods to produce widely consumable xplat artifacts:

• // TODO pls fix

Uh oh the list is empty

Wait are you saying that no xplat libraries properly expose artifacts for both types of consumer? Of course not! And by that I mean yes, no xplat libraries properly expose artifacts for both types of consumer.

Small caveat time, while ForgeGradle and Loom remap dependencies to be used, this isn't an absolutely necessary step. If your dependency and you use the same version of the same mappings, the remapping step will just map no names (nothing found, nothing to do!). And then you'll be left with a jar that just works on accident. Neat!

This isn't really a solution, though. You can't just say "use my mappings or perish" because inevitably a mod will need two incompatible mappings for its dependencies. Either two different mappings, or more insidiously two different versions of the same mappings. Also no one is going to change their mappings for a dependency. Either way, you're eventually going to get issues, which is why intermediaries exist, and why ForgeGradle and Loom handle this.

Everyone should just use Mojmap!

This is often repeated when people start talking about problems with mappings. "Everyone should just do what I do" is both a solution to any problem, and also not useful. Everyone's going to disagree on the common thing to do, and oops you're in the exact same position. Fabric as a loader refuses to adopt Mojmap, and Forge is reluctant to wholy and fully embrace it, still maintaining SRG and partially using it for runtime remapping. Mojmap is also, strictly speaking, not an effective intermediary.

One of the core features of an intermediary is a stable naming convention to allow mods to work across trivial version boundaries among other things. But Mojmap maps all the classes, methods, and fields, right? Well, for the most part yes, but these names are not stable, the devs can and do change them whenever they want. Minecraft is a product, not an API.

But more directly, a ton of methods in the game aren't actually mapped by Mojmap. As the bulk, synthetic methods (secret methods generated by java for special language features like lambdas) remain unmapped. You may think modders don't need to touch these, but since both loaders support arbitrary bytecode transformation of vanilla classes at runtime, you may be surprised by a lot of things people do.

As a side note, SRG actually doesn't support this either anymore, so this is more of a point for the textile ecosystems where this concrete stability is expected and relied on.

This combination of technical and social factors means while widespread Mojmap use would benefit everyone, it's also just never going to happen, the same way "Everyone should use Fabric" is wishful thinking. Some people don't like Mojmap names, and will cling to their favorite mappings forever. Personally, I use Yarn in all of my projects, and would go to pretty absurd lengths to continue doing so. So we need a solution that accommodates all the common types of users.

EMI's solution

So we have a publishing problem. How does EMI solve it? Simple! Alongside my Fabric and Forge artifacts, I just need to publish an Intermediary and a Mojmap copy and let the consumers use the one they need. This is only trivially possible because I use Architectury Loom, which supports mapping to Intermediary at all. Then I simply need to document a bunch of example dependencies for various use cases, like this table, but better.

Of course, producing Mojmap artifacts is non-trivial, especially since I use Yarn to develop EMI. This is solved by adding a new subproject that's configured to use Mojmap, grabbing the Intermediary jars from my xplat project, and then asking Loom very nicely to reverse the process. Since the subproject uses Mojmap, you've got a valid VG Mojmap xplat artifact, push on Maven and thrive. Here's one of the tasks my project uses.

task mojmapJar(type: net.fabricmc.loom.task.RemapJarTask) {
	classpath.from loom.getMinecraftJarsCollection(net.fabricmc.loom.api.mappings.layered.MappingsNamespace.INTERMEDIARY)
	dependsOn project(':xplat').remapJar
	
	inputFile = project(':xplat').remapJar.archiveFile
	sourceNamespace = 'intermediary'
	targetNamespace = 'named'
	
	remapperIsolation = true
}

Bam! That's that, do the same with the API and sources jars and you've got everything working, all popular build systems can consume EMI as a library, making it the first mod of its kind.

What's wrong with my sources

That's the end of the story when it comes to mappings, but that's not the end of the story when it comes to Gradle and Maven. I encountered a bunch of roadblocks while working on this, most significant among them was an issue I caught very late on. As a bit of context, when publishing artifacts to Maven for others to use as a dependency, there are 3 common types of artifacts published:

1. First is the normal jar, the mod just made.
2. Second, and optionally, is the API jar. This is a subset of the API visible classes in the primary artifact. The utility of providing this is consumers can depend on it and know for a fact that they're not accidentally importing and using implementation features. A lot of people in the modded community live with the "move fast and break things" pathos, but stick around a while and you'll probably start to appreciate stability.
3. Finally, is the sources jar. While this is a jar in the literal sense, it doesn't have .class files, it is full of .java files. Traditionally, this is the source code used to generate the primary artifact. Not necessarily a reproduceable build, but sources are used by IDEs to view javadocs on methods as well as explore code. In modded environments, sources are often exported in an intermediary format, and the build system handles remapping it as well. This is to prevent confusion. Though, because it's mostly invisible when using the same mapping set, a lot of libraries actually export their sources incorrectly. (Here's a fun game, go on random Mavens for Fabric mods and try to guess if the sources are in Mojmap, Yarn, or Intermediary). Javadocs are pretty important, so while the absence won't prevent compilation, this is basically a necessity for libraries to expose.

So sources are important. And mine were broken. Initially this seemed like a problem with just the xplat projects, but further investigation revealed this was just a sampling bias. The issue I was running into caused my sources to get partially remapped, fail, and just exclude the remaining classes. This is a terrifying (nearly) silent failure state! The processing order was consistent but arbitrary, meaning Forge and Fabric produced significantly more readable sources by random chance. Xplat on the other hand was correctly mapping 14 of about 300 classes for a clean 5%, which is how it got noticed.

The problem

I'd like to tell you what the problem was. I spent 2 hours with a debugger attached to Gradle stepping through method calls reading stack traces and locals to find out. In the end I only had 3 conclusions:

1. My classpath was incomplete for the remap tasks.
2. I knew what classes were causing it to break. This was actually much harder to find out than you'd think. You don't get a nice print when a remap fails, just a pretty java.lang.NullPointerException and a suspiciously roomy jar file.
3. This failure wasn't only occurring in my local project, but was also occurring in dependent projects when they attempted to map my sources.

The problem seemed to be stemming from my usage of another library at compile time only and only in my xplat subproject. JEI is developed using VanillaGradle, so its xplat artifacts are actually unusable by EMI, and instead I need to depend on its Fabric artifacts from xplat and just... be careful to not use the wrong things. If JEI was not on the classpath when Loom tried to remap my sources, it'd just silently fail and give me my beautiful empty jars. This wasn't hard and fast though, plenty of classes using JEI methods got remapped totally fine, and only a handful were causing this issue.

Ultimately, making progress on solving this was taking a ton of time, and fixing the problem on my end wasn't even an option. Even if I did work my magic, it'd still fail to remap for consumers, unless I transitively made them depend on JEI for an internal bit of functionality unrelated to my API (which I'm not going to do). So I did what every reasonable engineer does when going gets tough, I wrote an elegant hack to exclude the problematic classes and accepted the slightly incomplete sources jar since they were implementation classes.

task filteredSourcesJar(type: Jar) {
	classifier = 'filtered-sources'
	dependsOn remapSourcesJar
	from zipTree(remapSourcesJar.archivePath)
	exclude 'dev/emi/emi/jemi/**'
}

I used this task in place of remapSourcesJar for publishing artifacts and for remapping Mojmap sources. This task was also present on the Fabric and Forge sides, but that didn't fix the failures inside of those projects. I found that I need to re-specify all of my compile time dependencies from my xplat project in my loader projects, despite them remaining unused. This likely has to do with how Architectury does its project dependencies, but I don't have a handful of hours staring at a debugger attached to Gradle in me to find out for sure. Following this approach with my Mojmap xplat project would likely have similar results, but it's ultimately uneeded as I need to filter the jar regardless. With all this together, I was generating proper sources on Fabric and Forge too.

Conclusion

EMI can successfully publish its artifacts for all 4 platforms. It publishes 3 artifacts (normal, API, and sources) per platform and has 4 platforms (Forge, Fabric, Intermediary xplat, and Mojmap xplat). A release of a single Minecraft version includes 12 jars. Since EMI currently supports 1.18.2, 1.19.2, 1.19.3, and 1.19.4, and will be supporting 1.20 when it comes out in a matter of days, that'd be 5 versions. EMI updates all Minecraft versions at once, 3 artifacts, 4 platforms, and 5 Minecraft versions permutates for 60 jars per release. That's quite a couple compared to the average library, especially considering it's 60 versions of basically the same thing.

Build systems in the Minecraft ecosystem, especially for the increasingly common xplat use cases, are an absolute mess. Every loader has its own quirks for dev mappings and intermediaries, no one is too interested in coming together, and Quilt in particular is looking to widen the gap. The two primary xplat build systems, Architectury and VanillaGradle, are incompatible with each other, and requires delicate hacking for one way interop.

ForgeGradle is a rigid and untenable mess, and I didn't even get into the issues I ran into where consumers of mods with mixins (like mine!) need to specify seemingly random build flags to not get instant runtime crashes. Loom is more permissive but has serious limitations and shortcomings that require being worked around, as well as subtle failures that need to be caught. Architectury is quite fragile and has its own share of obtuse operations, getting a workspace going is its own challenge. And VanillaGradle is a simple and effective tool for certain situations being used outside of its domain.

Where does this leave us? Well, EMI is available on all common build systems. We now know more about Minecraft build systems and mappings. With this, we're inched ever forward towards global serenity. Maybe in a couple years, things will look better. Until then, we'll make do.