Please don't make your compile unit the same size as your namespace.

Before we start, allow me to specify the terminology used in this article. Terms like "package" and "module" tend to differ from language to language, so it's good to settle on one interpretation so we avoid mimatches.

• A package is the unit of code that usually gets distributed atomically. Examples are NPM and Python packages, Java JARs, Rust crates, Go modules.

  When mentioning Go packages (namespaces), it is always next to "Go."

• A compile unit is the collection of code that gets compiled in one call to the compiler. Think whatever files you feed into gcc or javac. Usually, cyclic dependency can only exist within one compilation unit, unless the language contains features like forward declaration.

• A namespace is an organization of code symbols that is logically grouped together. Referencing to symbols (functions, vars, types) within the same namespace is usually easier than referencing those outside. Symbols can have the same name in different namespaces, but usually can't within the same namespace (except overloading).

  Examples are C++ and C# namespaces, Java and Go packages, Rust and OCaml modules. Low-level languages like C don't have namespaces.

Yeah I'm not a Go expert.

I have read a couple of Go code before for researching and debugging, but almost never written them. Take my statements with a grain of salt if you like.

I have been researching about module systems between programming languages recently, because I'm troubled constantly by one specific design decision.

You see, compiled programming languages with namespaces often fall into one of two categories:

• Some have compile units spanning multiple namespaces. An entire package is usually compiled with only one compiler invocation.

  Examples of this are: C#, Java, Dart, Rust, etc.

• Some others have namespaces spanning multiple compile units, usually using one compiler invocation per file. Some of them can also declare smaller namespaces within them.

  Examples of this are: C++, OCaml (with wrapped modules).

Either way (or for some language both ways), the size of a namespace is usually different from the size of a compile unit. The compiler leaves you some room to logically groups parts of your code into smaller units, so you may maintain cleanness while writing namespaces that reference each other.

But there's one language that's a big exception of that -- Go.

In Go, a single folder within the source code repository simultaneously respresent one compile unit and one namespace, which Go calls a "package". Go packages cannot have dependency cycles, and neither does it have any smaller internal structures. All symbols declared within it must reside in one flat namespace, which spans all source code files within it.

But why is Go so special? Can we study Go's implementation and use it in our own ones?

Isn't it because Go's philosophy is to simplify the compiler at every turn, while happily forcing complexity onto the users?

(chuckles) Can't you just shut up and let me explain first?

When Go's solution worked ​

The structure of Go's package should look familiar to many C and C++ programmers. Simply speaking, it's the common C/C++ project structure, sans header files and forward declaration.

In fact, I would suspect that such design directly comes from C projects -- the three main designers of it are all famous senior C/C++ engineers, who have worked large, very large C and C++ projects.

Needless to say, Go's package system works. But it works not because it's good. Rather, it's because it is aligned and deeply coupled with the other design aspects of Go.

Well at least it's better than writing CMakeLists.txts I guess?

And I want to put my argument upfront. Go's package system works now because the following 4 features of Go forms a reasoning cycle that self-supports. When some of them change, the rationale that makes Go packages what they are now no longer holds rigidly.

1. A namespace contains every file in a folder, nothing more or less.
2. Go's interfaces are structural, decoupling it from implementation.
3. go generate generates new files instead of macros modifying existing files.
4. Each file gets to specify its list of imported namespaces.

Namespaces ​

A namespace contains every file in a folder, nothing more or less.

It's direct derivation of classic C projects' structures. A somewhat common pitfall in C is forgetting to implement a forward declaration, so banning them altogether is a valid move (like all other modern languages).

But without forward declaration, how do we handle dependency cycles? Right, we can allow dependency cycles in one (CMake-style C) "library" by considering all symbols declared within it at once. The compile unit gets a little larger than the original C structure, but it's still within control.

From here, to avoid the other common pitfall in C -- name clashing, we make each "library" here its own namespace, so symbols within it won't clash with those outside.

And bam! We get a Go package.

So, a Go package is basically a C library isolated in its own namespace, with forward declaration replaced with cyclic references within the library, and compiled in one unit.

Doing so comes with costs. We'll see how Go's design deals with it.

Interfaces ​

Go's interfaces are structural, decoupling it from implementation.

So, although we can remove cyclic dependencies between Go packages, there's one more pattern we need to handle if we want people to write applications on it -- interfaces.

It's a very common pattern to declare interfaces and methods that use implementations together. Take this for an example:

package pkg.services:
  interface IService
  class Factory:
    method createA() uses ServiceA
    method createB() uses ServiceB

package pkg.services.a:
  class ServiceA impl IService

package pkg.services.b:
  class ServiceB impl IService

Such pattern is very common in applications -- defining high-level interfaces in one namespaces, defining the implementations in the descendants of it.

But sadly, if you're still using nominal interfaces while having everything else behave like Go packages, this won't work. You get a cyclic dependency: pkg.services.Factory -> pkg.services.a.ServiceA -> pkg.services.IService.

I can put IService into a separate Go package, and let all other packages depend on that, I guess?

Yeah, why not.

*staring at dependency graph* Oh man.

So technically you can solve it, but at the cost of creating a lot more Go packages than you want. If each file were their own "package" (like OCaml or TypeScript), you might be able to pull that off (although still quite dirty), but now you're working with folders, which are mentally heavier than files...

Go developers obviously went to the other route: What if our implementation does not depend on the interface?

This takes us to the land of structural interfaces. The implementation ServiceA impl IService is now implicit and hidden to the compiler. Such relationship checked only when you actually perform the cast on the type. In this way, we can now remove the dependency pkg.services.a -> pkg.services.

So, the original downside of Go's package design is now somewhat fixed using structural interfaces, and structural interfaces (not nominal) are justified by the package design. Reasoning cycle 1, complete.

go generate ​

go generate generates new files instead of macros modifying existing files.

Macros. The part of C that programmer hate and love the most. They are powerful enough to save programmers from writing repetitive code, even generating generic data structures at compile time. However (as what they are in C), they are notoriously hard to maintain.

This makes it a clear no-go for Go. (Haha, unintentional wordplay.) So, nothing can modify existing source files now.

Can't they use procedural macros?

You know Go people will decline compile-time code execution ideas like this at first glance.

But programmers still need something to generate code for them. The thing will need to be able to read and parse existing code, and then inject definitions that interact with existing code. Since Go doesn't permit dependency cycles, they will need to directly inject the code into the current Go package, but without touching existing files.

But wait! In Go packages, every file within a package already share the same namespace, so you can just put generated code into the current folder and call it a day!

This is what Go designers settled on. A go generate command that invokes custom generator executables (that can parse and generate Go ASTs), which then generates needed code in new files that gets automagically included into the current package.

But with this definition, they are procedural macros, just manually invoked and generate into another file!

Then every code generator is a procedural macro.

So in short, the Go package design makes generating code in other files painless, and such code generation fixes the issue of not having macros to modify files, making Go's package design necessary. Reasoning cycle 2, complete.

Import list ​

Each file gets to specify its list of imported namespace.

Nobody likes managing dependency, increasingly so when the unit of dependency gets smaller and smaller.

You might be able to manage the dependency list of a NPM package.json with ease. But managing dependencies of libraries in a large project where some of its code are even generated? That's another story.

Not only will you face a dependency graph much more detailed than the package-(not Go packages)-level, but in generated code you will not know what namespaces the generated files depend on, until the compiler complains! -- And it might change from time to time!

This is actually quite common in C/C++ project management -- dependencies are hard -- and it's why many C/C++ libraries are dependency-free and/or header-only.

Since we usually explicitly import namespaces in code, and namespaces are exactly packages in Go, does that mean...

Go takes a clever way to solve that. Instead of the C way of having a CMakeLists.txt-like file in each Go package specifying its dependencies, it just uses an import statement like any other modern languages, and calculates dependencies from those statements.

Because each namespace in Go directly corresponds to a Go package, the union of all import statements are the dependency of the package. This also solves the problem of not knowing the dependencies of generated code -- it's specified in that file, so you don't need to even think about managing them.

So, the design of code generation and Go packages can be justified by gathering dependencies directly from import lists, and using import list as dependency list can only work when packages is equal to namespaces. Reasoning cycle 3, complete.

What do we get now? ​

As can be seen in this dependency graph of features in Go, these features are clearly tightly coupled. In its heart, the feature "a namespace is exactly one compile unit" has its downside solved by other features, and justifies the existence of them.

Wondering if I can change some of them...

When Go's solution doesn't work ​

Say you're designing something of your own. Your personal weekend programming language project or something like that. Can you use Go's solution on the trinity of namespaces, compile units, folder?

As you might have guessed from the graph above -- No. Absolutely no, unless you're making some carbon copy of Go itself. Even then, unless you're absolutely sure it will work in your scenario, just don't.

Let's try doing an experiment. What if we change some of the features above?

Beginning with the structural interface feature. If we replace that with nominal interfaces (which means you need to explicitly implement them), what will happen?

Umm, I will create a new namespace (i.e. "Go" package) for the interface type, so that it can be imported by both its implementors and users?

Might not be that disastrous as you might think, but many small folders would still be pretty annoying to work with. In that case you might better off just use file-scoped namespaces, larger compile units or forward declaration.

Okay. Next up, go generate. What if we remove it?

I will lose the ability to procedurally generate code.

Right, metaprogramming is required for compiled languages like this, or people will find the 100 other workarounds to do metaprogramming without official support for it.

On the other side, if you think of replacing it with real procedural macros... That's a pure boost, but it also means you won't need the generated code to be automagically included in the same namespace as your source code, so the namespace design is useless again.

Finally, what if we remove the import list?

I DON'T WANT TO WRITE CMakeLists.txt AGAIN!!!!!!

That tells. Micromanaging dependencies is never a good idea, so in this way you'll prefer the two mainstream designs again.

And let's not forget the elephant in the room -- large dependency cycles. If you happen to be writing something that has a large inherent complexity, like compilers or large applications, you'll probably encounter one or two very large dependency cycles that just can't be easily reduced without increasing maintenance difficulty.

With Go's namespace design, the tool you have to manage complexity -- smaller namespaces that can be cyclically referenced and partially exported -- does not exist at all. You're forced to put everything in your reference cycle into one big f_cking flat namespace. It'd be nightmare maintaining code like this.

Conclusion ​

Do I need to state this again?!

Go's solution works, but it's because it's Go. Go is special.

Go has its very specific set of trade-offs it made, so that making compile units precisely equal to namespace is possible.

If you're designing something on your own, you're very likely not using the same set of trade-offs as Go does. So, just. Please. DO NOT COPY GO'S NAMESPACE DESIGN.

Take a walk outside. Get some fresh air. Touch the grass. Use anything sane from scripting language's per-file scope, to modern languages' package-level compile units. Caching will always be your friend. Even C/C++'s forward declaration and separated compilation has some merits. But you're not Go, so just don't copy Go.