Core Development Memo

~wicdev-wisryt
December 05 2022
core%20memo.png

This is a memo I wrote to the core developers in advance of an offsite earlier this year (in June). Looking back on it, it’s a good thing for everyone in the community to take a look at and be aware of. Being able to chart a clear path from here to ‘barely viable’ is really significant. We didn’t release everything on my ‘before Assembly’ list, but most are already in the release pipeline, and the rest are in progress. It has been a very productive year.


As core infra devs, our whole goal is for Urbit to succeed. It is not to make Tlon succeed, though if Urbit succeeds, Tlon is set up very well to succeed also, and if Urbit doesn't succeed, Tlon can't either. It is not to perpetually build something interesting and compelling to investors, like an academic chasing grants. And it is not to toil away on a spiritual journey of building quality software, like a remnant of monks pursuing the Good for our own edification and to preserve it against a falling world.


Urbit cannot succeed unless it's the best software stack (for personal computing) in the world.


Being the best doesn't guarantee success. Everything else has to fall into place. But, if Urbit isn't the best stack in the world, it will fail. And right now, Urbit isn't the best stack in the world; in fact, it's not even a barely viable stack for a moderately successful product.


Urbit is so good at so many things that the distance from "finally barely viable" to "best in the world" is really small. We have the potential to blow everything else out of the water, and from many perspectives we already do. But we can't seriously claim that a successful product could be built on Urbit as it stands. That other companies are building products on Urbit should not be seen as evidence that we're ready but as great faith in our ability to get there and before too long.


Tlon has two official objectives for 2022. It's impossible to make "landscape as sticky as gm" with the stack as it is now. And it's useless to make the invite flow into hosting good if the base system doesn't get to the point where it can serve as the foundation of a serious product.


In fact, Tlon's product goals have always been gated on core dev work. We've always had product goals, certainly since leaving "unlaunch" status around 2015. During that time, we've continually made sincere efforts at making compelling products, and for many years we utterly failed. It was not possible to build even the most rudimentary product on Urbit.


Then in 2019 that changed, suddenly and almost unexpectedly. We didn't have a good product, but all of a sudden the stack had passed a threshold and there was actually some success. Landscape could be improved somewhat on its own basis, but it still had quite a low ceiling. How good could Landscape be while you're still breaching the network regularly? When ships regularly have to breach themselves because of memory issues? When a hundred ships getting on the network at once brings it to its knees?


The product work done since then has been very important and we've learned a ton. We've built an incredible community as well, and there's a lot of demand for a working Urbit. But when the stack puts a ceiling on growth, it doesn't matter how many hockey sticks you generate.


Here's what the stack lacks to be barely viable as a platform for moderately successful products:


  • Better-than-current scaling
  • Security against DoS and other catastrophic attacks
  • Zero-click maintenance, including across upgrades and in the presence of common errors
  • A mechanism for cross-version compatibility, including across the network
  • A not-insane workflow for building apps

That's pretty much it. You can golf this list by considering the last three to be basically "be correct across upgrades", which is a really deep part of Urbit's architecture — state machines, event logs, nock, hoon, etc are largely chosen because they make this feasible.


What's unique about Urbit in 2022 is that this list of things is quite short. Each of these except security is doable to the barely-viable level in months with little implementation risk.


This used to be a much longer and scarier list. We couldn't even begin to draw up such a list in January 2016, but it would have included at least:


  • Invent some way to make the PKI real
  • Make it so that a single non-networked urbit just sitting there does not inevitably fall over from memory mayhem
  • Make it so we don't wipe all data every couple months
  • Make it so we can push out incompatible upgrades over the air, and support incompatible upgrades to the binary
  • Build even a single web app that anyone would ever want to use for more than 30 seconds
  • Come up with a way for other people to build and distribute apps safely
  • Scale to where more than 200 people can be on the network
  • Prove anybody even cares

The only one of our current problems that reaches the level of the old ones is security, and even that is much easier to get started on than "uh... maybe the pki should be real somehow".


This implies a few things about what we as core devs should do.


First, we can no longer ignore security. We've already taken the first couple steps toward the project of making Urbit secure, and that needs to be one of our top priorities immediately.


Second, we should not be doing any short-term projects which do not advance us along one of those dimensions to a barely viable software stack. Longer-term research projects are reasonable to continue in moderation because they have a long lead time.


Third, we should evaluate our work before Assembly according to this list. This likely includes the following projects:


  • Release content distribution in at least a minimal form (scaling)
  • Release event-log truncation (zero-click maintenance)
  • Build machinery for versioning userspace subscriptions (cross-version compatibility)
  • Make all (including binary) upgrades fully automatic, with better patterns for communicating around them.
  • Make Clay support a reasonable dependency flow (workflow for building apps)
  • Finish subscription reform (workflow for building apps)

If we can accomplish these, then we're most or all the way to four of the five things we need (not security). Even scaling probably clears the bar for a moderately successful app — countless small things will crop up, but they will likely be at a pace that we can keep up with, even in hockey-stick growth.


Implicit in this framing is a fairly unusual mindset toward working on Urbit that's worth calling out. There are several archetypes of teams, and all are good for certain types of projects.


One very common one is the adventurer archetype, where you're a band of travelers, marauders, or conquistadors constantly looking for profitable adventures. You survive by your wits and by constantly adjusting your strategy to the changing landscape as you explore the world. This is the mindset of many startups and should be the mindset of pretty much any product team, including our own. OKRs are purpose-built for this mindset.


Another is the monk archetype, where you're on a spiritual mission to discover something Good. This can be the mindset of researchers and academics. This often relies on superhuman discipline and expertise to determine which is a righteous direction to pursue.


A third is the builder archetype, where your goal is to complete something. This is everything from an author of a book to the builders of the pyramids to NASA landing on the moon to a team of 200 building Windows NT. This isn't a "just do it" mindset — often much of the process of building is figuring out how to do it. However, "Is it done yet?" is a reasonable question to ask a team like this, where it's not with the other archetypes.


Core development on Urbit is most similar to the builder archetype. Landscape and hosting should not have this mindset, but we do need to.


Unlike building the pyramids, we're not simply willing Urbit into existence. There's incredible potential energy in the idea of Urbit, and if we can get it over the hump, then it will become totally self-propelling. This is what some mean by saying that Urbit is inevitable — more precisely, it has the potential to become inevitable if we can just get out of its way. I like the saying that Urbit can't fail, it can only be failed.


This is a common characteristic of technology: once someone invented electric lightbulbs, of course you still needed to sell them, and that took decades. Once they got over the hump, they became inevitable and ceased to need an advocate. Once railroads passed some point, the transcontinental railroad was inevitable. Same for the Internet.


It is possible for that potential to never be realized. The Aztecs knew about wheels and never used them productively. Most other examples we would never have heard about.


Our job is to unleash the potential of Urbit to be the best software stack (for personal computing) in the world. We're close.