With a powerful trait system, compile-time constants, and where-and-:-style bounding for types and constants, Rust’s take on generic functions has been a refreshing departure from the anything-goes, Wild Wild West, errors-only-when-called template system of C++. Furthermore, its macro system has been a much needed replacement of C’s underpowered macro system, allowing users to actually generate code in a consistent and dependable manner at compile-time, with the ever-powerful Procedural Macros (“proc macros”) taking care of some of the heaviest language extension tasks. But…

Much like C, Rust perhaps a bit too heavily relies on (proc) macros and code generation techniques to avoid dealing with sincere deficiencies in the language, producing much heavier compile times by frontloading work at inappropriate compilation stages to offset the lack of language and library features. To this end, we have begun working on the specification, formalization, and potential integration (which may not be completed fully as part of this work¹) of a set of core language primitives we are bikeshed-naming introwospection, which we hope to make available under the core::introwospection and std::introwospection modules in Rust.

Primary Motivation

This article will spend a perhaps obscene amount of time reasoning, designing, critiquing, comparing, pontificating, and yearning for this or that feature, functionality, idiom, or capability.

As a general-purpose disclaimer, while we have spoken with a large number of individuals in specific places about how to achieve what we are going to write about and report on here, it should be expressly noted that no individual mentioned here has any recorded support position for this work. Whatever assertions are contained below, it is critical to recognize that we have one and only one goal in mind for this project and that goal was developed solely under our — Shepherd Oasis’s — own ideals and efforts. Our opinions are our own and do not reflect any other entity, individual, project and/or organization mentioned in this report.

One of the biggest things Shepherd’s Oasis wants to enable is infinitely valuable and scalable code. For example, in the primary serialization crate featured across the entire Rust ecosystem, serde, we want to enable David Tolnay (or any one of the 159/future contributors)’s code to be able to handle this:

 0	struct Point {
 1		x: i32,
 2		y: i32,
 3	}
 4
 5	fn main() {
 6		let point = Point { x: 1, y: 2 };
 7
 8		// Convert the Point to a JSON string.
 9		// this still works.
10		let serialized = serde_json::to_string(&point).unwrap();
11
12		// Prints serialized = {"x":1,"y":2}
13		println!("serialized = {}", serialized);
14	}

This is the flagship example from serde, but with a few very important changes made. The code above does not require:

• #[derive(…)];
• impl Serialize for Point { … };
• and, no build.rs, “newtype” idioms, or other shenanigans to generate code or implement interfaces.

In order to do this, we want serde to be able to write something similar to the (non-compiling, using-unimplemented-syntax, not-at-all thoroughly checked) generic code just below as a base implementation for Serialize, its flagship serialization trait for any given struct or enum:

  0	use std::introwospection::*;
  1	use serde::ser::{
  2		Serialize, Serializer,
  3		SerializeTupleStruct, SerializeStruct,
  4		SerializeTupleVariant, SerializeStructVariant,
  5		Error
  6	};
  7
  8	struct DefaultSerializeVisitor<S, T>
  9		where
 10			S: Serializer,
 11			T: Serialize + ?Sized
 12	{
 13		serializer: &mut S,
 14		value: &T,
 15	}
 16
 17	pub trait Serialize {
 18		fn serialize<S, T>(
 19			&self, serializer:
 20			S, value: &T
 21		) -> Result<S::Ok, S::Error>
 22			where S: Serializer,
 23			T: Serialized + ?Sized,
 24		{
 25			let mut visitor = DefaultSerializeVisitor{
 26				serializer,
 27				value: self
 28			};
 29			introwospect(Self, visitor)
 30		}
 31	}
 32
 33	struct DefaultStructSerializeVisitor<S, T>
 34		where
 35			S: Serializer,
 36			T: Serialize + ?Sized
 37	{
 38		serializer: &mut S,
 39		value: &T,
 40		newtype_idiom: bool,
 41		tuple_idiom: Option<(&mut S::SerializeTupleStruct)>,
 42		normal_idiom: Option<(&mut S::SerializeStruct)>,
 43		maybe_error_index: Option<usize>
 44	}
 45
 46	struct DefaultEnumSerializeVisitor<S, T> 
 47	{
 48		serializer: &mut S,
 49		value: &T,
 50		variant_info: Option<(&'static str, bool, usize)>,
 51		tuple_idiom: Option<(&mut S::SerializeTupleVariant)>,
 52		normal_idiom: Option<(&mut S::SerializeStructVariant)>,
 53		maybe_found_index: Option<usize>
 54		maybe_error_index: Option<usize>
 55	}
 56
 57	impl<S: Serializer, T: Serialize + ?Sized> EnumDescriptorVisitor
 58		for DefaultSerializeVisitor<S, T>
 59	{
 60		// Drop into the `enum`eration-style serialization and methods by
 61		// creating, specifically, that visitor type. This provides
 62		// context to the `FieldVisitor`-using methods so we know that at
 63		// the top-level we are working with an `enum`eration.
 64		type Output -> Result<S::Ok, S::Error>
 65
 66		fn visit_enum_mut<Descriptor: 'static>(&mut self) -> Self::Output
 67			where Descriptor: EnumDescriptor
 68		{
 69			let mut visitor = DefaultEnumSerializeVisitor{
 70				serializer: self.serializer,
 71				value: self.value,
 72				variant_info: None,
 73				tuple_idiom: None,
 74				normal_idiom: None,
 75				maybe_found_index: None,
 76				maybe_error_index: None
 77			};
 78			introwospect(T, visitor)
 79		}
 80	}
 81
 82	impl<S: Serializer, T: Serialize + ?Sized> StructDescriptorVisitor
 83		for DefaultSerializeVisitor<S, T>
 84	{
 85		// Drop into the `struct`-style serialization and methods by
 86		// creating, specifically, that visitor type. This provides
 87		// context to the `FieldVisitor`-using methods so we know
 88		// that at the top-level we are working with an `struct`.
 89		type Output -> Result<S::Ok, S::Error>
 90
 91		fn visit_struct_mut<Descriptor: 'static>(&mut self) -> Self::Output
 92			where Descriptor: EnumDescriptor
 93		{
 94			let mut visitor = DefaultStructSerializeVisitor{
 95				serializer: self.serializer,
 96				value: self.value,
 97				newtype_idiom: false,
 98				tuple_idiom: None,
 99				normal_idiom: None,
100				maybe_error_index: None
101			};
102			introwospect(T, visitor)
103		}
104	}
105
106	// … and so much more.

and have it work, in perpetuity, for the rest of their life for the vast majority of Rust types².

Right now, absolutely none of the above code makes sense to the vast majority of people. And that is fine. But, this is the ultimate goal of this combination article-report; we will be going through the constructs above, going to explain what it does, and how it enables less boilerplate, less markup, less “new type idiom” usage, and more doing exactly what you expect and want by-default out of the vast majority of code.

Let’s get started.

introwospect? … introwospection …??

The name is a placeholder. Originally, we wanted to simply call it “reflect” and “reflection”, but both are already reserved in Rust, including in the compiler as a symbol. It was then changed to “uwuflection”, but we are not doing code generation (in the macro sense) with this feature like other reflection facilities in other languages. Thus, it was changed to “introspection”. However, it was requested we make the name at least 11% sillier so there was no question that we honestly do not care what the final name will be, and to avoid what would undoubtedly be an entirely worthless bikeshed session. Enter: introwospection.

Introwospection’s core ideals are as follows:

• does not force the user to pay for what they do not use (if a type is not reflected on, then no information about it whatsoever should show up in the final artifacts);
• will not produce run-time, dynamic allocations, nor will it require it under any circumstances (making it suitable for constrained and resource-starved environments);
• will produce information that can be acted upon by the type system or at compile-time (i.e. const fn time) without exception;
• can be utilized to inspect types and values within generic functions, including values and types of values not owned by the current crate;
• and, cannot be utilized to inspect private or hidden properties that are invisible to code at the current scope, module, or crate.

These core ideals shape what we want out of the API, and how it differs from existing attempts at this in Rust rather powerfully. In particular, current attempts at reflection and similar utilize (procedural) macro programming, defer entirely to run-time/type-erased entities, or some combination of the two in conjunction with hand-crafted wrapper types and methods. Introwospection wants to be able to work with any type whatsoever — much like in the above beefy default Serialize implementation — rather than types that provide a specific implementation or types that we explicitly own. To explain some of this requires us to go back, and talk about the way things work today, including the fact that Rust does not have compile-time reflection like folks originally explained to us when we first started the language a couple of years ago.

Wait, Doesn’t Rust Have Reflection Already?

We held this belief upon first seeing Rust some time ago. Consider, briefly, this code from the rocket.rs front page project as of April 13th, 2023:

 0	#[macro_use] extern crate rocket;
 1
 2	#[get("/hello/<name>/<age>")]
 3	fn hello(name: &str, age: u8) -> String {
 4		format!("Hello, {} year old named {}!", age, name)
 5	}
 6
 7	#[launch]
 8	fn rocket() -> _ {
 9		rocket::build().mount("/", routes![hello])
10	}

This code — apparently, with magic — is capable of understanding that we want a string (&str, in this case, which is a reference to an existing blob of string memory) to designate a name, and an 8-bit integer for the age. It parses that automatically from the <name> and <age> portions of this get attribute-labeled route. It also returns a string, that will then be transported in a basic HTTP-valid form of transportation all the way to the user’s browser. This means that — somehow — Rocket understands this hello function, its parameters, its return types, and — more importantly — the negotiation of these properties to and from a form understood by all web browsers. Is that not reflection? The ability to gaze into normal Rust code as written, and automatically generate the boilerplate and interop? Clearly, Rust had achieved something that C users only dream of, and that they can only produce with externally-orchestrated tools connected by make (or CMake, or meson, or Bazel, or any of the other dozens of build systems holding up a Mt. Everest of code).

Unfortunately, this is only half of the story.

What is happening here is not reflection, which dismayed us greatly as we actually learned the language. It is actually the machinations of a separate system that has been built on top of Rust’s actual programming language. A separate shadow world whose job it is to do the immense heavy lifting that makes code like this possible. And that entire shadow world that’s powering the most elegant Rust code starts with a single crate maintained and propelled forward by David Tolnay and his devout helpers, poetically named syn.

syn

The syn crate (said like the first part of the word “syntax” and exactly like the word “sin”) is the Big Mover and Shaker of Rust. When things get unwieldy to express and complicated to keep typing out in C, one falls back to the preprocessor system or code generators. Similarly, Rust programmers fall back to their own token expansion / generation system, termed “macros”. Exactly like the C counterpart, Rust’s macro programming model does not actually understand anything about the program itself. It receives a stream of tokens, similar to the way invoking a function macro in C (e.g., DO_CALL(A, B, 234, str)) gets a list of tokens to interact with. They are allowed to generate a new sequence of tokens (subject to a handful of rules and hygene requirements that C macros do not have). But, the fun doesn’t stop there for Rust macros; they can be supercharged with even more capabilities that allow them to hook into the “custom attribute” and “custom derive” settings, as well as totally repurpose Rust tokens to create their own languages (delimited within the usual my_macro!(…) or my_macro![…] or my_macro!{ … } invocations).

These enhanced macros — called procedural macros — can do whatever they want by acting as a compiler plugin over that token stream. While C object and function macros are extremely limited in scope and power — despite running on the same conceptual level as Rust macros — Rust macros are so fully-featured that one can reimplement the entire Rust Frontend in their Rust macro feature. Others new to Rust but well-aged in many programming languages — save for the older Lisp veterans — would scoff. If we suggested implementing a C frontend out of token parsing in preprocessor for C, we would be laughed out of the room for even wanting a preprocessor that powerful. But, in this brave new Rust world, not only is Rust’s preprocessor theoretically powerful enough to do that work as an academic on-paper exercise, it is also in-practice exactly that powerful.

syn is the culmination of that very idea. It is a library that parses a stream of Rust tokens into Rust-understood and intimately recognizable constructs such as an syn::Expression, a syn::Type, a syn::AssocType, a syn::DataUnion, and so much more.

The special #[launch] and #[get(…)] macros from the Rocket example are how syn is deployed. Libraries use these attributes as the hooking points and jumping points for their macros and procedural macros. Then, they dip their hands into syn to parse and handle these Rust constructs in order to generate code. This, effectively, means that every macro and procedural macro is re-doing the work of the Rust frontend (up to its AST generation), and then acting on that in order to generate code for constructs (types, associated items, names, traits, etc.) it recognizes. This is how Rocket knows to generate a main function for us with #[launch], knows how to generate the boilerplate that connects a fully-received HTTP Request into something that can talk to our hello function, etc. etc.! It is an amazing feat of engineering. Make no mistake, David Tolnay plus the 80+ contributors to this crate are one of several critical pillars of why Rust is a major system’s programming language worth taking seriously today. It bought a serious amount of time for the language designers and the numerous compiler engineers to focus on other parts of Rust, while use cases involving generating code and similar could be wrapped up in (proc) macros.

As with most engineering approaches born out of necessity, macros using syn to parse Rust source code parsing in a preprocessing language comes with interesting consequences exacerbated by Rust’s programming model. In particular, Rust’s strong ownership rules (not just for resources, but for code concepts) means that macros very quickly hit very particular limits. Sometimes, those limits are billed as advantages, but with our work we have begun to see it as more of a hindrance than a benefit. We will take a slight detour to explain code ownership, specifically in relation to the C model, and expound upon how it relates to compile-time reflection.

Strong Ownership: Not Just for Resources

One of the things that makes C and C++ brutal for code, in conjunction with a source-text #include-based model of programming, is how ownership becomes very hard to define. Multiple translation units may end up with a function, structure, or similar that have identical fully qualified names and name spaces. How C and C++ handle this is effectively a shrug of the shoulders, combined with a rule called the One Definition Rule. This rule states that multiple translation units that do not opt into certain modifying keywords (e.g., static, anonymous top-level namespaces namespace { … }, or extern) promise that any code with identically-named entities shall have the same content, Or Else™. That “Or Else™” is not always enforced, as its placed under either implicit undefined behavior (C) or as Ill-formed, No Diagnostic Required (IFNDR, C++).

These days, some compilers can check these assumptions; for example, GCC or Clang with -Wodr AND Link-Time Optimizations/Link-Time Code Generation turned on can warn on some One Definition Rule collisions. But for the most part, if the code from two or more translation units have identical names, compilers just play a quick game of Russian Roulette and eliminate all but one version of the code. It should not matter if there are multiple versions because we promised that all versions will be the same. This sometimes gets violated (because of different translation unit compilations using different macro definitions, or producing odd compile-time values that go into functions to provoke differing behavior), and it results in occasional hilarity as well as brutal, tear-filled 3 AM debugging sessions.

Rust sidesteps this problem entirely by basing their code concepts not within translation units that get blended into an executable later, but instead with conceptual modules and crates. We will not regale everyone with the details here, but effectively functions, structures, unions, traits, and more all belong to the modules and crates that define them. Code is included through the use of use importations and similar, and this allows strong ownership of code that belongs to one logical entity (the crate, and within a crate, to the module) rather than every single file (translation unit) claiming total control over every single piece of code that gets #included in. This gets rid of quite a few obvious flaws from C, chief among them the need to manically mark every header-implemented function with inline and then spend billions in compute cycles deduplicating the mess.

The drawbacks that start showing up, especially in relation to the macro system and generic programming systems in Rust, are about this strong ownership property.

Traits and Ownership

Consider a trait in a given a crate cats, such as:

0	pub trait Purr {
1		fn do_purr(&self);
2	}

In our own library (which is its own crate), we have a structure called Lion:

0	pub struct Lion {}

We are allowed to implement the trait Purr for our Lion like so, in our own library:

0	use cats::Purr;
1
2	pub struct Lion {}
3
4	impl Purr for Lion {
5		fn do_purr (&self) {
6			// big cat, bigger purrs
7			println!("p u r r")
8		}
9	}

This works fine for our own code in our own programs/libraries. However, consider a different structure named Puma that exists in another library big_animals. It doesn’t have an implementation of Purr on it, but a Puma is a (big) cat, so we’d like to add one. So, in our own library again, we try to import the Puma structure and add the necessary Purr implementation:

0	use big_animals::Puma;
1	use cats::Purr;
2
3	impl Purr for Puma {
4		fn do_purr (&self) {
5			// long, sleek purs
6			println!("purrrrrr")
7		}
8	}

This does not compile. It runs afoul of what’s called the Orphan Rule, which is part of a broader Rust property called coherence. The short definition of the Orphan Rule is as follows:

You cannot provide implementations of a trait for a struct unless you are either the crate that defines the struct, or the crate that defines the trait.

Knowing this, big_animals cannot have someone outside of it add the cats::Purr trait on it, as that runs afoul of both coherence and, specifically, the Orphan Rule. This presents one of the biggest problems that Rocket, rlua, clap, and so many other codebases have to struggle with when they do #[derive(…)] based or Trait-based programming. There are many ways to get around such an issue, but almost every solution requires additional wrappers and special types. For example, for the problem with Puma above, there’s 2 ways to go about this: create a new trait, or create a new type. The latter is the solution that is nominally used, wherein the “new type idiom” is used. It works, fundamentally, like this:

0	use cats::Purr;
1
2	struct MyPuma(big_animals::Puma);
3
4	impl Purr for MyPuma {
5		fn do_purr (&self) {
6			// long, sleek purs
7			println!("purrrrrr")
8		}
9	}

This is, ostensibly, not very workable if we have to pass a Puma into a function to do some work, or receive a Puma back. In either case, we need to “unpack” (easy enough by doing my_puma_thing.0) or we have to construct a MyPuma every time we get it back (e.g., MyPuma(thing_that_makes_a_regular_puma())). This requires a bit of manual labor, and it introduces some compatibility issues in that there is now a (purely syntactic) barrier between the functionality that someone may want/need versus the data type it is implemented over.

Viral Proliferation

Because it is impossible to fully anticipate the myriad of traits that one may need to implement on a structure, union, enumeration, or other type in Rust, many crates need to add “features” onto themselves to accommodate external crates. Going through a handful of crates will reveal {name}-serde features or similar addendums sticking off many of them. This is where “baseline” or otherwise important — but nonetheless entirely external/orthogonal — traits are given necessary implementation by the owners of whatever {name} crate. For many of the serde prefixed and suffixed features, they exist just to pull in the foundational serialization crate serde and then provide baseline definitions without the new type/new trait idiom on the types provided by the crate that added the feature. This may be surprising to some folks, but is actually fairly normal for those of us who participated in the C# and .NET ecosystems for a long time.

That is, C# encounters this same base-level need for its interfaces — and exactly the interfaces defined by a handful of libraries — to be shared all over the ecosystem. Because interfaces are unique to the “assembly” (the closest .NET parallel to a crate), even if they have the same members/methods/properties, one has to implement exactly the interface from that specific assembly. It results in very much the same sort of viral need to continually pull in often times unrelated assemblies and dependencies. It is expressly for the purpose of implementing the interfaces on a given type “properly” so it can be used seamlessly with a given library. Things are generally okay when we’re writing generic functions in C# and working with IEnumerable or a similar interface specifications that come from the standard library, but it gets very dicey when we have to start combining multiple pieces of code that want a function Foo(), and there’s 3 different interface IFoo in the different libraries. There’s a number of ways to solve this problem in Rust, but none are without drawbacks or unfavorable tradeoffs.

This is where the generic system of C++ becomes much better for end-users and — in some cases — the very librarians writing the code themselves. Even with its objectionable template syntax and intensely depressing usage experience (SFINAE, as a starter, and deeply nested error explanations with oft-times missing information), it is incredibly easy to take code that is 5, 10, or even 20 years old and make it work seamlessly with other code. The lack of such strong ownership semantics means that templated functions and templated structures can be modified, specialized, and hacked up to cover a wide variety of types that the library author or application developer has no ownership rights to. Function overloads can be added for new data without impacting any existing overload choices. 13 year old gap buffer implementations sitting on GitHub can have the dust blown off of them and made to work with the existing and future Standard Library algorithms with little to no effort. No IEnumerable interfaces need to be added, nor Iterator traits opted into.

C++ only gets away with this by deeply abusing inline/header-written code, and its lack of code ownership. It toes to the very line what its One Definition Rule means. However, this ultimately leads to generic code written in C++ which provides, both in theory and in significant industry practice, infinite value without actually being fully rewritten. This is in stark contrast to interface-based and trait-based generics, where it must be constantly updated or tweaked or opted into to provide the full range of benefits afforded by the generic system. Combined with the ownership rules for Rust-style trait implementations, it must not only be opted into, but has to be wrapped over and over and over again if the original crate author isn’t willing to make a feature or addendum to their library as they may not consider our use case worthwhile or foundational.

Surely, Others Must Have Solved This Problem by Now?

As with most complex Computer Science topics, the true answer is “it depends”.

There exist a mixed bag of approaches. Many developers use the basic approach with “newtype” and just commit to lots of unwrapping and unboxing. Occasionally, the implementations are so bog-standard that they either rewrite with newtype idioms or just write wrapper functions that call into the “real” functions to do work, hooking them up to the specific implementation. For a small (say, up to even 100) functions and types, such hand-tailored work is not impossible. If the work follows a specific pattern, macros and procedural macros can often wrap up the work. There is still boiler plate to declare, but a lot of that can be somewhat automated since Rust macros are expressly designed to generate and emit code. Much like C macros, however, debugging such expansions and working through any troubles from the created code turns out to be a hassle, though generally the tools are much better than the ones that come with most C compilers.

Still, these are not the only approaches possible. Some shift the burden entirely to procedural macros and try to take it to its logical maximum, such as with reflect.

reflect

David Tolnay’s reflect, at the outset, operates in macro / procedural macro space and wants to solve the problem of robustness when using low/mid-level abstractions in syn and friends. That means that all of the above criticisms about #[derive(…)], custom attributes, and ownership concerns will always apply to whatever comes out of the effort in this space. There will always be some amount of either needing to own the code explicitly so it can be appropriately marked up. Or, an issue with needing to create new types/traits so they can be marked up. Or, trying to replicate enough of rustc’s behavior so that it can look up crates, walk the file system, and attempt to do more of what both cargo and rustc do at (procedural) macro time, which has obvious drawbacks for compilation time and build system sanity.

The proof of concept and the philosophy do make for a very attractive metaprogramming story; namely, it provides an interface that is superficially similar to Zig’s comptime T: type-style of generic programming. You handle reflected values similar to a way you would handle a type value at comptime in Zig. We do not think everything we do with introwospection will be able to cover all of what dtolnay/reflect — or, indeed, (procedural) macro programming in Rust — can do, but we do aim to take a significant chunk of the use cases present here and instead move them into something the Rust language can handle itself. This leaves the other approach to this problem: run-time registration and reflection.

bevy_reflect

The bevy_reflect crate is much more fully featured and production-ready, focusing entirely on the ability to hold onto, transport, and interact with types and methods at run-time. It has advantages for how it interacts with code in that they provide a fixed set of interfaces that allow folks to register types, functions, and even potential trait information at run-time. It forsakes perfect code generation by no longer being compile-time (and, thusly, giving up a meager amount of performance, binary code size, and disk/RAM space) and leans deeply into a well-defined, deeply polished system.

With bevy_reflect, we describe the behavior of a type using specific traits and registration functions and root the entire system in a handful of traits — Reflect being one of them — before using copious amounts of downcast and downcast_ref to get the information we want to, at specific junctures in the system. This means we only register what we want to use, and it is not restricted solely to types we explicitly own as well. Even the examples on bevy_reflect’s tutorials are exceptionally cool and very powerful, from modifying existing fields on a struct hidden behind a run-time type description to iterating over a select list of structures that match a certain casting pattern. This reflection solution feels is the closest to a Java or C#-style of reflection built entirely out of a library, and likely feels familiar to folks who have built similar systems out of void* (C) or BaseClass* (in C++).

Unfortunately, because it goes about this in at run-time, it violates our pretty explicit goal of not making the user pay for things they may not use. Run-time registration that must be cast down and cast up, (slight) run-time cost to the operations being performed, needing to check constantly to ensure things are the type or shape that we would like it to be in; all of these things are both programmatic and performance overhead for our stated goals. Many of these operations are things the language could make safer if it had the ability to look at its own types and functions, so that it did not need constant checking, casting, and unwrap()-ing to get work done.

The std::any::* Module and dyn Trait

std::any::Any, std::any::TypeId, and similar constructs in the std::any module provide ways of handling values through what is effectively a wrapped-up interface. This is further complicated by the Rust language itself, using dyn SomeTrait and similar constructs to allow boxing and virtual table-ing of many entities. This falls in much the same category as bevy_reflect does, as these are run-time powered mechanisms. The focus of this module and for dyn SomeTrait is the ability to add a level of indirection (virtual tables, indirect function calls, and similar techniques) to allow for “hiding” the source of data while allowing a downstream consumer to work with it without strictly typing the interface. Much of this also powers many of the implementation techniques in bevy_reflect, so it’s safe to call this a subset of what bevy_reflect offers to end-users with more general-purpose tools.

Is That Enough?

Well, not really. As our goal is compile-time, no-overhead introspection on existing program types, we come to a bit of an impasse. While dtolnay/reflect can theoretically provide us with no-overhead introspection, it’s tie-in to the procedural macro system prevents us from using it in the normal Rust language, nor on types we do not have rights to (e.g., outside our program/library crate). It also does not fundamentally improve the generic programming situation in Rust, nor give us better primitives to work with in the language itself.

Having seen the myriad of approaches, we now turn to our own attempt which strays from both the procedural macro-heavy approach of Tolnay’s reflect and turns away the run-time powers of bevy_reflect. We may not be able to solve every problem that procedural macros or run-time reflection registration can, but we believe that the below examples will illustrate many of the ways in which we can promote a better way of performing compile-time computation with the purpose of looking back on code. We will start with explaining the API and compiler interface itself, then dive into the reasons for design choices, how it avoids some of the pitfalls of alternatives, and the challenges we will have to face down in order to make it happen. We will also discuss shortcuts which may be applied to sidestep the lack of many unstable, missing, and/or incomplete Rust features.

introwospection at a Low Level

At its core, introwospection attempts to invert the #[derive(MakeSomeTrait)]-style of code generation and instead substitute it with true compile-time, generic-capable introspection, utilizing information the compiler already has. Here is a basic example of the API in a program crate called cats_exec:

 0	#![feature(introwospection)]
 1	use std::introwospection::{FieldDescriptor, StructDescriptor};
 2	use std::mem::size_of;
 3	use std::any::type_name;
 4
 5	pub struct Kitty {
 6		pub is_soft: bool,
 7		pub meows: i64,
 8		destruction_of_all_curtains_trigger: u32
 9	}
10
11	pub fn main () {
12		type KittyDesc = introwospect_type<Kitty>;
13		println!("struct {}, with {} fields {{",
14			<KittyDesc as StructDescriptor>::NAME,
15			<KittyDesc as StructDescriptor>::FIELD_COUNT);
16		println!("\t{} ({}, size {}, at {})",
17			<KittyDesc::Fields as FieldDescriptor<0>>::NAME,
18			std::any::type_name::<<KittyDesc::Fields as FieldDescriptor<0>>::Type>(),
19			std::mem::size_of::<<KittyDesc::Fields as FieldDescriptor<0>>::Type>(),
20			<KittyDesc::Fields as FieldDescriptor<0>>::BYTE_OFFSET);
21		println!("\t{} ({}, size {}, at {})",
22			<KittyDesc::Fields as FieldDescriptor<1>>::NAME,
23			std::any::type_name::<<KittyDesc::Fields as FieldDescriptor<1>>::Type>(),
24			std::mem::size_of::<<KittyDesc::Fields as FieldDescriptor<1>>::Type>(),
25			<KittyDesc::Fields as FieldDescriptor<1>>::BYTE_OFFSET);
26		println!("\t{} ({}, size {}, at {})",
27			<KittyDesc::Fields as FieldDescriptor<2>>::NAME,
28			std::any::type_name::<<KittyDesc::Fields as FieldDescriptor<2>>::Type>(),
29			std::mem::size_of::<<KittyDesc::Fields as FieldDescriptor<2>>::Type>(),
30			<KittyDesc::Fields as FieldDescriptor<2>>::BYTE_OFFSET);
31		println!("}}")
32	}

The integer constant I for FieldDescriptor<I> is the declaration (source code) index. We are accessing each field explicitly, one at a time. In general, each entity in Rust is folded into a trait which has associated types and associated const items, and the collection of these traits are called Descriptors. All of the information can be carried through const fns and/or the type system: in this example, we use introwospect_type<…>, which takes a single type argument to display the which will ultimately print:

0	struct cats_exec::Kitty, with 3 fields {
1		is_soft (bool, size 1, at 12)
2		meows (i64, size 8, at 0)
3		destruction_of_all_curtains_trigger (u32, size 4, at 8)
4	}

Those not used to Rust will note that — when not annotated by a specific kind of #[repr(…)] attribute — byte offsets do not correspond to a linear sequence within the structure. They can be rearranged at will, which is why the API uses the 0/1/…/up-to-::FIELD_COUNT - 1 values that correspond to the unique declaration index rather than any other metric. Each FieldDescriptor<I> trait implementation yields the information about that field on the structure. This API also does not allow access to e.g. crate-hidden or private fields. For example, if Kitty is moved out of cats_exec and is instead in a crate named felines, the code will be unable to access “Field 2” (destruction_of_all_curtains_trigger). So, with felines/src/Kitty.rs:

0	pub struct Kitty {
1		pub is_soft: bool,
2		pub meows: i64,
3		destruction_of_all_curtains_trigger: u32
4	}

And cats_exec/src/main.rs:

 0	#![feature(introwospection)]
 1	use std::introwospection::{FieldDescriptor, StructDescriptor};
 2	use std::mem::size_of;
 3	use std::any::type_name;
 4	use felines::Kitty;
 5
 6	pub fn main () {
 7		type KittyDesc = introwospect_type<Kitty>;
 8		println!("struct {}, with {} fields {{",
 9			<KittyDesc as StructDescriptor>::NAME,
10			<KittyDesc as StructDescriptor>::FIELD_COUNT);
11		println!("\t{} ({}, size {}, at {})",
12			<KittyDesc::Fields as FieldDescriptor<0>>::NAME,
13			type_name::<<KittyDesc::Fields as FieldDescriptor<0>>::Type>(),
14			size_of::<<KittyDesc::Fields as FieldDescriptor<0>>::Type>(),
15			<KittyDesc::Fields as FieldDescriptor<0>>::BYTE_OFFSET);
16		println!("\t{} ({}, size {}, at {})",
17			<KittyDesc::Fields as FieldDescriptor<1>>::NAME,
18			type_name::<<KittyDesc::Fields as FieldDescriptor<1>>::Type>(),
19			size_of::<<KittyDesc::Fields as FieldDescriptor<1>>::Type>(),
20			<KittyDesc::Fields as FieldDescriptor<1>>::BYTE_OFFSET);
21		// Compile-time error
22		// <KittyDesc::Fields as FieldDescriptor<2>>
23		// would error!
24		//
25		//println!("\t{} ({}, size {}, at {})",
26		//	<KittyDesc::Fields as FieldDescriptor<2>>::NAME,
27		//	type_name::<<KittyDesc::Fields as FieldDescriptor<2>>::Type>(),
28		//	size_of::<<KittyDesc::Fields as FieldDescriptor<2>>::Type>(),
29		//	<KittyDesc::Fields as FieldDescriptor<2>>::BYTE_OFFSET);
30		//
31		println!("}}")
32	}

Which would produce:

0	struct felines::Kitty, with 2 fields {
1		is_soft (bool, size 1, at 12)
2		meows (i64, size 8, at 0)
3	}

This means we have a privacy-respecting way of looking at the data for a type which is not part of a crate we authored! There is just one small problem with the above code, unfortunately.

It. Is. U G L Y.

It does not take an opinionated code beautician or a grizzled Staff Engineer to look at the above code and tell us that this has to be the world’s ugliest API in existence. Nobody wants to write the above, nobody wants to hard-code source-level indices, nobody wants to constantly cast some type to a given field descriptor. Rust also does not allow us to have a local type declaration or something similar, like so:

 0	#![feature(introwospection)]
 1	use std::introwospection::{FieldDescriptor, StructDescriptor};
 2	use std::mem::size_of;
 3	use std::any::type_name;
 4	use felines::Kitty;
 5
 6	pub fn main () {
 7		type KittyDesc = introwospect_type<Kitty>;
 8		type trait KittyField0 = <KittyDesc as FieldDescriptor<0>>;
 9		type trait KittyField1 = <KittyDesc as FieldDescriptor<1>>;
10		/* rest of the code… */
11	}

This means that every line has to contain the trait cast, to provide compile-time access to a specific, declaration-indexed field on something. It is genuinely a terrible way to program for reflection, being both non-ergonomic and brittle. The question, however, is why is the interface like this in the first place? Can we not simply use a natural collection for fields or enumeration variants or function arguments or any other thing that boils down to “list of types and other associated information”? It turns out that this is a lot harder than it seems without compromising the goals of introwospection. First, let’s start with using arrays and slices.

What’s wrong with [T; N] or [T]?

One might ask, why we are not using arrays (or slices) to provide this information in a consistent, easy-to-use manner. The simple answer is that arrays require everything to coalesce to a single type. Doing so would mean that it would be impossible to provide type information in a compile-time, non-type-erased, non-potentially-allocating manner with the e.g. Field_at_I::Type. This means that arrays are unsuitable for the task at hand; we want no erasure to be taking place and no run-time checking of types as is done with bevy_reflect. Alternatively, we would have to wrap every list of field descriptors up into an enum of compiler-generated structures that each contain the necessary FieldDescriptor<I> implementation, but that also imposes a compile-time overhead because one would only be able to access the type within a match clause. This is further inhibiting because match clauses over such types do not allow us to perform different kinds of implementations for different Ts within the case arm. The way around this is with something gory like transmute_copy, and praying that it optimizes out. Reportedly, power users of Rust have done something like this and achieved success by depending on the compiler optimizing out unused compiler match arms for a given function and directly accessing the proper memory through the transmute_copy, but this once again is relying on the good graces of the compiler. There is no guarantee a Rust compiler will not just, on a whim one day, fumble the code and start producing some of the most gnarly jumps and conditionals we have seen up until now.

Ultimately, we decided that such an approach was likely to be brittle, and representing lists of types such as function arguments, struct fields, enumeration variants (and the fields on each variant) in this manner was too error prone.

What About (T, U, V, R, S, …)? They are Heterogenous Collections, Right?

Another potential approach is providing a tuple of types, from 0 to Description::FIELD_COUNT - 1. Unfortunately, tuple-based programming is unsophisticated in Rust. To compare, C has no tuples to speak of other than directly creating a structure. C++ has the grungy std::tuple, which is usable — including at compile-time — but deeply costly (to build times) without compiler implementations pouring a LOT of energy into writing heuristics to speed up their access, instantiation, and general usage. Rust has built-in tuples, which is an enormous win for compilation time at instantiation and usage time, but robbed itself of most usability. Accessing a tuple cannot be done programmatically with constant expressions, because the my_tuple.0 syntax is a hardcoded, explicit allowance of the syntax that literally requires a number after the ..

We cannot calculate a const INDEX: usize = /* some computation here */; and then do my_tuple.(INDEX), my_tuple[INDEX], my_tuple[std::num::type_num<INDEX>], or std::tuple::get::<INDEX>(my_tuple). We honestly don’t care what the syntax would look like, and maybe std::tuple::get::<INDEX>(my_tuple) could be implemented, today, in Rust, but it doesn’t exist. There is no way to program over a tuple other than hardcoding not only every tuple from size 0 to N into a several traits, but then doing the combinatoric manual spam to get each value. This contributes to Rust compile-time issues, in the same way that using the excellent P99 library for lots of macro metaprogramming in C contributes to increased compile-times with a high count for the P99 FOR_EACH or similar macros. Even C++ standard libraries would implement variadic std::function with macros in C++, and they suffered compile time loss until they moved to true variadics and compile-time programmability. It turns out that “fake it until you make it” is universally an expensive thing to do no matter what language architecture we build up for ourselves. Needing to parse 0-to-8, 0-to-12, or 0-to-25 faked up things will always be more expensive than having true variadic and tuple support. The language also does not support operations like .map(…)-ing over a tuple to produce a tuple of different types, though as the code from frunk proves we can ostensibly fake it for ourselves. Most attempted faux-implementations — especially naïve ones that take a “Cons and List” style approach — can result in quadratic time algorithms, further exacerbating compile time costs.

Given this, we have decided not to rely directly on tuples to fulfill the metaprogramming needs in conjunction with our desired API for compile-time introspection. This is in the interest of keeping compile times as low as possible from the outset, and also in the interest of not saddling the user with a subpar, map-less API that requires external crates or serious workarounds to handle. In general, type-heterogenous programming and support is fairly weak in Rust, which is why we have come up with the following API below that is a lot more manageable. It is not a perfect API and it cannot handle truly generic, type-heterogenous collections without compiler help, but given how the introwospect_type, introwospect, and introwospect_over keywords will be implemented, it should provide us with the maximum amount of flexibility for the jobs we would like to pursue.

introwospection at the Mid-Level: Visitors

To make things easier, we just have to implement a basic visitor type:

 0	#![feature(introwospection)]
 1	use std::introwospection::{FieldDescriptor, StructDescriptor, };
 2	use std::mem::size_of;
 3	use std::any::type_name;
 4	use felines::Kitty;
 5
 6	struct DescriptorPrinter;
 7
 8	impl FieldDescriptorVisitor for DescriptorPrinter {
 9		
10		type Output = ()
11		
12		fn visit_field<Type: 'static, const INDEX: usize>(&self) -> Self::Output
13			where Type : FieldDescriptor<INDEX>
14		{
15			let type_name = type_name::<Descriptor::Type>();
16			let member_size = size_of::<Descriptor::Type>();
17			println!("\t{} ({}, size {}, at {})",
18				Descriptor::NAME,
19				type_name,
20				member_size,
21				Descriptor::BYTE_OFFSET);
22		}
23	}
24
25	impl StructDescriptorVisitor for DescriptorPrinter {
26		
27		type Output = ()
28
29		fn visit_struct<Type>(&self) -> Self::Output
30			where Type : StructDescriptor
31		{
32			println!("struct {}, with {} fields {{",
33				Descriptor::NAME,
34				Descriptor::FIELD_COUNT);
35			// now, introspect over the fields of this type.
36			( introwospect_over(Descriptor::Type, Descriptor::Fields, self) );
37			println!("}}");
38		}
39	}
40
41	pub fn main () {
42		let visitor = DescriptionPrinter;
43		introwospect(Kitty, visitor);
44	}

Which would produce output identical to the one above:

0	struct felines::Kitty, with 2 fields {
1		is_soft (bool, size 1, at 12)
2		meows (i64, size 8, at 0)
3	}

Visitors are intended to be the mid-level API. They offer a large amount of flexibility and, in conjunction with the introwospect and introwospect_over keywords, allow for easily iterating through the visible fields on structs, unions, enums fns, and other types. This level of compile-time introspection allows us to perform algorithms in a systemic manner over a any type and any field. The visitors that come with the API are as follows. All INDEX names are constant usize expressions that refer to the declaration (source order) index of the entity within whatever context it is applied.

• StructDescriptorVisitor: for observing compile-time information through a generically provided StructDescriptor-implementing type parameter describing either a struct or union type.
• EnumDescriptorVisitor: for observing compile-time information through a generically provided EnumDescriptor-implementing type parameter describing an enum type. If the enumeration has no variants, then the Variants is NoType. Otherwise, it has from INDEX = 0 to INDEX = VARIANT_COUNT - 1 implementations of the trait VariantDescriptor<INDEX>.
• TupleDescriptorVisitor: for observing compile-time information through a generically provided TupleDescriptor-implementing type parameter describing the built-in type (…). The unit type () is covered by aTupleDescriptor whos Fields is NoType (e.g., there are no fields on the tuple). Otherwise, the Fields has from INDEX = 0 to INDEX = FIELD_COUNT - 1 implementations of the trait FieldDescriptor<INDEX>.
• ArrayDescriptorVisitor: for observing compile-time information through a generically provided ArrayDescriptor-implementing type parameter describing the built-in type [T; N]. Uses type ElementType; and const ELEMENT_COUNT: usize; to describe the compile-time size.
• SliceDescriptorVisitor: for observing compile-time information through a generically provided SliceDescriptor-implementing type parameter describing the built-in type [T]. Uses type ElementType; to describe the T of the slice.
• FunctionDescriptorVisitor: for observing compile-time information through a generically provided FunctionDescriptor-implementing type parameter describing the built-in type fn name (Args…); and possibly closure objects as well. Uses type ReturnType; to describe the return type, and const PARAMETER_COUNT: usize; to describe the number of parameters the function has. If it is greater than 0, the type Parameters has implementations from INDEX = 0 to INDEX = PARAMETER_COUNT - 1 of ParameterDescriptor<INDEX>³.
• Two versions for visiting fields at compile-time:
  • FieldDescriptorVisitor: for observing compile-time information through a generically provided FieldDescriptor<INDEX>-implementing type parameter. Use with the std::introwospection::get_field::<…>(…) function and an object whose type matches type Owner to get the field of the tuple, union, enumeration, or structure.
  • FieldDescriptorVisitorAt<INDEX>: identical to FieldDescriptorVisitor, but provides the information for the desired INDEX directly into the *Visitor implementation itself. This allows a different associated type to be chosen for each field on a structure descriptor, enabling heterogenous programming over a visited collection.
• Two versions for visiting variants at compile-time:
  • VariantDescriptorVisitor: for observing compile-time information through a generically provided VariantDescriptor<INDEX>-implementing type parameter. Variants are interesting in that a variant on an enumeration does not represent a type by itself, procuring all sorts of problems for the reflection API as a whole. It sometimes requires bizarre workarounds to grasp at the data optionally contained within a variant. A type Fields; is available on each variant descriptor which has its own FieldDescriptor<INDEX> implementations from INDEX = 0 to INDEX = FIELD_COUNT - 1, provided the variant descriptor’s FIELD_COUNT is greater than 0. Programmatically, one can iterate over each variant and use the DISCRIMINANT field to check if a given enumeration object has the same variant set, and then use std::introwospection::get_field::<…>(…)
  • VariantDescriptorVisitorAt<INDEX>: identical to VariantDescriptorVisitor, but provides the information for the desired INDEX directly into the *Visitor implementation itself. This allows a different associated type to be chosen for each variant on a enum’s descriptor, enabling heterogenous programming over a visited collection.
• Two versions for visiting function parameters at compile-time:
  • ParameterDescriptorVisitor: for observing compile-time information through a generically provided FieldDescriptor<INDEX>-implementing type parameter. Use with the std::introwospection::get_field::<…>(…) function and an object whose type matches type Owner to get the field of the tuple, union, enumeration, or structure.
  • ParameterDescriptorVisitorAt<INDEX>: identical to ParameterDescriptorVisitor, but provides the information for the desired INDEX directly into the *Visitor implementation itself. This allows a different associated type to be chosen for each field on a function descriptor, enabling heterogenous programming over a visited collection.

Each of the highest level *Descriptor types are also paired with an AdtDescriptor trait, which provides the const NAME: &'static str; string name of the type, a const ID: AdtId; identification enumeration of the type for what kind it is, and const ATTRIBUTES: &'static attributes;, which contains the names (and potentially, values) of anything found within a #[introwospect(…)] keyword.

Up until now, we have sort of hand-waved how a lot of this works. This is for good reason: while we were able to create mockups that achieved basic versions of what we have above using existing Rust Compiler Nightly builds with unstable features, we very swiftly ran up against various problems both theoretical and practical with how this could be implemented or provided by Rust. At each point, we had to workaround various issues until we hit some of our penultimate blockers that have prevented us from having a working version of this in unstable, nightly Rust. We will start from the top with our various keywords and what they are meant to do, in conjunction with the semantics they ought to have if “written out” by hand. Then, we will drill down into the individual challenges — many already alluded to — and hone in on these issues.

Basic introwospection and Visitors: The Details

introwospect_type is the first keyword. It signals to the compiler that the type fed into it should be reflected upon. The type that comes out is not well-specified to be any particular type, but it will be something that implements one of the above Descriptor types based on what type was fed into it (a structure, union, enumeration, function (pointer), so on and so forth). For example, using an enumeration type CatHours in conjunction with Kitty for a program crate called cat_time:

 0	#![feature(introwospection)]
 1	use std::time::SystemTime;
 2	use std::introwospection::*;
 3
 4	#[non_exhaustive]
 5	pub struct Kitty {
 6		pub is_soft: bool,
 7		pub meows: i64,
 8		destruction_of_all_curtains_trigger: u32
 9	}
10	
11	pub enum CatHours {
12		ZeroOfThem,
13		OneOfThem(Kitty),
14		TwoOfThem{one: Kitty, two: Kitty},
15		LotsOfThem(&[Kitty]),
16		#[introwospection(lossy)]
17		LostCountOfThem{ last_known: usize, when: SystemTime }
18	}
19
20	pub fn main () {
21		type HoCDesc = introwospect_type<CatHours>;
22		// .. other things here
23	}

This would produce something similar to the following by the compiler:

  0	#![feature(const_discriminant)]
  1	use std::time::SystemTime;
  2	use std::introwospection::*;
  3	use std::mem::{offset_of, Discriminant, discriminant_at}; // for impl
  4	use std::option::Option; // for impl
  5
  6	#[introwospect(are_little_precious_babies = "yes!!")]
  7	#[non_exhaustive]
  8	pub struct Kitty {
  9		pub is_soft: bool,
 10		pub meows: i64,
 11		destruction_of_all_curtains_trigger: u32
 12	}
 13
 14	pub enum CatHours {
 15		ZeroOfThem,
 16		OneOfThem(Kitty),
 17		TwoOfThem{one: Kitty, two: Kitty},
 18		LotsOfThem(&'static [Kitty]),
 19		#[introwospection(lossy)]
 20		LostCountOfThem{ last_known: usize, when: SystemTime }
 21	}
 22
 23	/* COMPILER GENERATION BEGINS HERE! */
 24	// struct Kitty
 25	unsafe impl AdtDescriptor for Kitty {
 26		const ID: AdtId = AdtId::Struct;
 27		const NAME: &'static str = "cat_time::Kitty";
 28		const ATTRIBUTES: &'static [AttributeDescriptor]
 29			= &[
 30				AttributeDescriptor{
 31					name: "are_little_precious_babies",
 32					value: Some("yes!!")
 33				}
 34			];
 35	}
 36	unsafe impl FieldDescriptor<0> for Kitty {
 37		type Owner = Kitty;
 38		type Type = bool;
 39		const NAME: &'static str = "is_soft";
 40		const BYTE_OFFSET: usize = offset_of!(Kitty, is_soft);
 41	}
 42	unsafe impl FieldDescriptor<1> for Kitty {
 43		type Owner = Kitty;
 44		type Type = i32;
 45		const NAME: &'static str = "meows";
 46		const BYTE_OFFSET: usize = offset_of!(Kitty, meows);
 47	}
 48	unsafe impl FieldDescriptor<2> for Kitty {
 49		type Owner = Kitty;
 50		type Type = u32;
 51		const NAME: &'static str = "destruction_of_all_curtains_trigger";
 52		const BYTE_OFFSET: usize
 53			= offset_of!(Kitty, destruction_of_all_curtains_trigger);		
 54	}
 55	unsafe impl StructDescriptor for Kitty {
 56		type Fields
 57			: FieldDescriptor<0> + FieldDescriptor<1> + FieldDescriptor<2>
 58			= Kitty;
 59		const FIELD_COUNT: usize = 3;
 60		// This is NOT a tuple struct (e.g., `Kitty(u64, i32, …)`).
 61		const IS_TUPLE_STRUCT: bool = false;
 62		// private fields are visible to the usage in this context
 63		const HAS_NON_VISIBLE_FIELDS: bool = false;
 64	}
 65	// CatHours
 66	unsafe impl AdtDescriptor for CatHours {
 67		const ID: AdtId = AdtId::Enum;
 68		const NAME: &'static str = "cat_time::CatHours";
 69	}
 70	unsafe impl VariantDescriptor<0> for CatHours {
 71		type Owner = CatHours;
 72		const NAME: &'static str = "ZeroOfThem";
 73		const DISCRIMINANT: &'static Discriminant<CatHours>
 74			= &discriminant_at::<CatHours>(0).unwrap();
 75	}
 76	struct CatHours_Variant1_FieldsType;
 77	unsafe impl FieldDescriptor<0> for CatHours_Variant1_FieldsType {
 78		type Owner = CatHours;
 79		type Type = Kitty;
 80		const NAME: &'static str = "0";
 81		const BYTE_OFFSET: usize = offset_of!(CatHours, OneOfThem.0);
 82	}
 83	unsafe impl VariantDescriptor<1> for CatHours {
 84		type Owner = CatHours;
 85		type Fields
 86			: FieldDecsriptor<0>
 87			= CatHours_Variant1_FieldsType;
 88		const NAME: &'static str = "OneOfThem";
 89		const DISCRIMINANT: &'static Discriminant<CatHours>
 90			= &discriminant_at::<CatHours>(1).unwrap();
 91	}
 92	struct CatHours_Variant2_FieldsType;
 93	unsafe impl FieldDescriptor<0> for CatHours_Variant2_FieldsType {
 94		type Owner = CatHours;
 95		type Type = Kitty;
 96		const NAME: &'static str = "one";
 97		const BYTE_OFFSET: usize = offset_of!(CatHours, TwoOfThem.one);
 98	}
 99	unsafe impl FieldDescriptor<1> for CatHours_Variant2_FieldsType {
100		type Owner = CatHours;
101		type Type = Kitty;
102		const NAME: &'static str = "two";
103		const BYTE_OFFSET: usize = offset_of!(CatHours, TwoOfThem.two);
104	}
105	unsafe impl VariantDescriptor<2> for CatHours {
106		type Owner = CatHours;
107		type Fields
108			: FieldDecsriptor<0> + FieldDecsriptor<1>
109			= CatHours_Variant2_FieldsType;
110		const NAME: &'static str = "TwoOfThem";
111		const DISCRIMINANT: &'static Discriminant<CatHours>
112			= &discriminant_at::<CatHours>(2).unwrap();
113	}
114	struct CatHours_Variant3_FieldsType;
115	unsafe impl FieldDescriptor<0> for CatHours_Variant3_FieldsType {
116		type Owner = CatHours;
117		type Type = &'static [Kitty];
118		const NAME: &'static str = "0";
119		const BYTE_OFFSET: usize = offset_of!(CatHours, LotsOfThem.0);
120	}
121	unsafe impl VariantDescriptor<3> for CatHours {
122		type Owner = CatHours;
123		type Fields
124			: FieldDecsriptor<0>
125			= CatHours_Variant3_FieldsType;
126		const NAME: &'static str = "LotsOfThem";
127		const DISCRIMINANT: &'static Discriminant<CatHours>
128			= &discriminant_at::<CatHours>(3).unwrap();
129	}
130	struct CatHours_Variant4_FieldsType;
131	unsafe impl FieldDescriptor<0> for CatHours_Variant4_FieldsType {
132		type Owner = CatHours;
133		type Type = usize;
134		const NAME: &'static str = "last_known";
135		const BYTE_OFFSET: usize = offset_of!(CatHours, LostCountOfThem.last_known);
136	}
137	unsafe impl FieldDescriptor<1> for CatHours_Variant4_FieldsType {
138		type Owner = CatHours;
139		type Type = SystemTime;
140		const NAME: &'static str = "when";
141		const BYTE_OFFSET: usize = offset_of!(CatHours, LostCountOfThem.when);
142	}
143	unsafe impl VariantDescriptor<4> for CatHours {
144		type Owner = CatHours;
145		type Fields
146			: FieldDecsriptor<0> + FieldDecsriptor<1>
147			= CatHours_Variant4_FieldsType;
148		const NAME: &'static str = "LostCountOfThem";
149		const DISCRIMINANT: &'static Discriminant<CatHours>
150			= &discriminant_at::<CatHours>(4).unwrap();
151		const ATTRIBUTES: &'static [AttributeDescriptor] = &[
152			AttributeDescriptor{
153				name: "lossy",
154				value: None
155			}
156		];
157	}
158	/* COMPILER GENERATION ENDS HERE! */
159
160	pub fn main () {
161		// Description type, in this case, is simply
162		// the type itself!
163		type CatHoursDesc = CatHours;
164		// .. other things here
165	}

This is a lot of boilerplate. introwospect_type gives us a type that implements the appropriate StructDescriptor (for structs), EnumDescriptor (for enums), VariantDescriptor<INDEX> for each variant at source-code index INDEX, and FieldDescriptor<INDEX> for each field within an enum’s variant or a struct at source-code index INDEX. In some cases, the type given is equivalent to the type itself. At other points, it is just a compiler-generated and un-nameable struct that will be decorated with the appropriate implementations to enable the introspection.

There’s a few things that should be highlighted about the expected semantics / magic we are employing in this generated code.

std::mem::offset_of! is Magic™

std::mem::offset_of! has just landed for Rust and will eventually need to be stabilized. As of April 25th, 2023, it also does not have plans for how to get the offsets of the fields of a variant within an enumeration. So, the fanciful std::mem::offset_of!(CatHours, LostCountOfThem.when) syntax we are using above is completely fictitious. We doubt that the official Language, Compiler, or Library teams take blog posts as a sincere form of feedback so at some point we will likely have to participate in the future issue that asks for std::mem::offset_of to work on some form of Rust’s enumerations. This could also be fixed by making CatHours::LostCountOfThem a real type in Rust, rather than a magical entity that may only be named specifically within a match clause, directly (see further below).

std::mem::discriminant_at::<EnumType>(some_usize_index) is Not Real

This function call is not a real intrinsic available in Rust. It is also awkward to program. Because enumerations in Rust cannot directly speak of a given variant inside itself, it is impossible to make a perfectly safe variant of this for the end-user that does not work off an existing enumeration value. This is why there only exists a form of getting a std::mem::Discrimimant<EnumType> by calling std::mem::Discrimimant<EnumType>(enum_type_object) with an existing enum_type_object that is already one of the existing variants; it is the only way to safely get at an enumeration type’s value without returning an Option.

For the desired core function to exist in Rust,

0	pub fn discriminant_at<EnumType>(
1		declaration_variant_index: usize
2	) -> Option<Discriminant<EnumType>>;

it must return a std::option::Option type, as the declaration_variant_index may be outside of the 0 to VARIANT_COUNT for an enumeration, or larger than 0 for other available for the structures and unions⁴. There could also be an unsafe variant, which would make returning and handling the data simpler, but the performance for that use case is generally covered by just returning an Option and simply performing an unwrap_unchecked() within an unsafe { … } block.

All of these options are strictly worse than having the ability to guarantee with the type system that both the enumeration and the variant type are part of one another:

0	pub fn discriminant_for<EnumType, VariantType>() -> Discriminant<EnumType>;

This could only happen if we could write e.g. std::mem::discriminant_for::<CatHours, CatHours::ZeroOfThem>(). But, as stated many times in this article, variant types are not real or touchable outside of match statements.

Enumeration Variants are not Nameable Types

In the generated compiler code, we have to add fictitious CatHours_Variant{INDEX}_FieldsType types for a Fields inside of the VariantDescriptor<INDEX> itself, rather than being able to just make every VariantDescriptor<INDEX> also come with a StructDescriptor implementation. This is because variants cannot be named as real types. This actually adds that extra level of API complexity that is there solely because enumeration variants. Even if they look and smell like other tuples or structures in the language, each variant is actually its own entirely independent and completely different despite looking physically the same. For example:

 0	struct StructMeow {
 1		pub a: i64
 2		pub b: i32
 3		pub c: f64
 4	}
 5
 6	enum EnumMeow {
 7		A(StructMeow),
 8		B{a: i64, b: i32, c:f64}
 9		C((i64, i32, f64))
10		D(i64, i32, f64)
11	}

StructMeow does not require the same alignment, layout, field ordering, or byte offsets as EnumMeow::B. Similarly, the tuple in EnumMeow::C does not have to same alignment, layout, field ordering, or byte offsets as EnumMeow::D. Consider, briefly, that EnumMeow uses a u8 to store the discriminant that tells between all 4 elements. The layout for EnumMeow could place the discriminant at a byte offset of 0. It can then place EnumMeow::B.b at byte offset 4, then place a and c at byte offsets 8 and 16 respectively. Similarly, D can have its .0, .1, and .2 fields rearranged in a way that does not match the contained tuple in C.

Effectively, there is a special type for the Abstract Data Type’s variant that is different from a data type that has the same physical/source code look and appearance. But, it can only be talked about directly in match expressions and pattern matching contexts. It would be helpful to be able to clearly refer to such a type, for both std::mem::discriminant_at’s API and also to solidify the spelling of access for std::mem::offset_of!.

Attribute Introspection is Limited to #[introwospection(…)] Attributes

The attribute #[non_exhaustive] does not appear in the ATTRIBUTES listing for Kitty’s StructDescriptor implementation. This is because we cannot expose any and all forms of attributes on a type, field, or function: it would be too invasive. Therefore, only the values provided by an #[introwospection(…)] attribute will be provided on the Descriptor::ATTRIBUTES associated const item.

This ensures backwards compatibility with existing attributes. It furthermore strongly scopes what attribute-based metaprogramming can do. There may be a future where additional attributes are put into the scope, but for now we want to avoid any potential issues with ascribing meaning to preexisting attributes. If we move in a direction where additional attributes are introspectable, we would likely need to consider an ADDITIONAL_ATTRIBUTES variable with all of the various attributes on a given type.

Fields and Variants are both Difficult to Specify Fully

For the EnumDescriptor (with type Variants) and StructDescriptor/VariantDescriptor (with type Fields), it appears to be incredibly difficult to specify, generically, a working trait bound for the Fields/Variants. To explain, here is the full definition of the pub unsafe trait StructDescriptor part of the current repository with comments:

 0	/// A description of a `struct` type.
 1	pub unsafe trait StructDescriptor: AdtDescriptor {
 2		/// The type of the `struct` that was described.
 3		type Type;
 4		/// A type that represents the fields of this `struct`. If this is
 5		/// `core::introwospection::NoType`, then it has no fields and no field
 6		/// implementations on it.
 7		///
 8		/// NOTE
 9		/// TODO(thephd) Enable a succint way to describe all of the constraints on this type:
10		/// ```
11		/// type Fields :
12		///     (for <const I: usize = 0..Self::FIELD_COUNT> FieldDescriptor<I>)
13		/// = NoType;
14		/// ```
15		/// to specify the proper boundaries to make this type usable in
16		/// generic contexts. (This is bikeshed syntax and subject to change,
17		/// as there is already a `for <T>` trait bounds feature in Rust.)
18		type Fields = NoType;
19		/// The number of fields for this `struct` type.
20		const FIELD_COUNT: usize = 0;
21		/// What kind of syntax was used to encapsulate the fields on this `struct` type.
22		const FIELD_SYNTAX: FieldSyntax = FieldSyntax::Nothing;
23		/// Whether or not there are any fields which are not visible for this type.
24		const NON_VISIBLE_FIELDS: bool = false;
25	}

Effectively, we need to create bounds that are composed of 0 to Self::FIELD_COUNT - 1 FieldDescriptors. The reason these bounds are necessary is so that, in generic code (such as with the *Visitor types in the mid-level API), we have the ability to use these Fieldss and Variantss in those methods. Remember, Rust’s Trait system is not like C++ templates, C’s preprocessor, or Rust macros: they require that you specify up-front all of the necessary actions that can be taken on a given input parameter. We are not allowed to “figure it out” later at usage time (C++ templates) or hack the “step before actually doing the language” so much that it spits out something the base language can understand (C preprocessor, Rust macros). For:

• type Fields on StructDescriptors and VariantDescriptor<INDEX>;
• type Variants on EnumerationDescriptor;
• and, type Parameters on FunctionDescriptor;

we have to be able to use some sort of currently-unknown syntax and specification. We are told that the currently-incomplete feature that needs a lot more time and investment called Generic const Expressions (GCEs) will make this possible/plausible in some fashion, but going through the const repository and documents does not make it immediately obvious to us how we would program such a thing using GCEs. Nevertheless, what this does mean is that within the current trait system, it is entirely and completely inexpressible to do the kind of things we want to do for Rust’s compile-time introspection. So, even though we have finished what will be the library definitions of all of these traits and structures in the library/core/src/introwospection.rs source file for our repository, we will need tons of additional language improvements to reach our goals.

Variadics Do Not Exist in Rust

This is one of the biggest issues and causes some of the worst problems when trying to create an API of this caliber. While Rust has a built-in tuple type — far better than its counterpart in many other languages — tuples are not compile-time or generically programmable. As explained in the above section on tuple syntaxes⁵. Access is hard-coded, and while brilliant folks can make partial solutions (thanks, Alyssa Haroldsen), they are not flexible enough to provide the ergonomics necessary for end-users. This creates an on-going tension: if variadics were a real feature, where there was a language construct representing “0 or more” types or “0 or more values” (both logically consistent with tuples) with a way of accessing those types or values, we would be able to program over the fields of a struct or the variants (and its fields) of an enum. But we do not have any concept of “0 or more” of something in Rust, and therefore it is patently impossible to program over what is effectively a list of “0 or more” fields, variants, types, etc. that come with the territory of performing compile-time introspection.

This is not just a compile-time introspection problem, either: the Rust SIMD Working Group faced similar issues of “how do I work with 0 or more of the same type” when trying to create typical vector types that matched to hardware SIMD/AVX/RISC-V/PowerPC instruction sets. They had to eventually compromise on some of their original design for this, as well as extract a few concessions from the Rust core language in order to finally achieve the current working product of std::simd (but with notable restrictions, still)⁶.

This is one of the primary drivers of the introwospect_over keyword that is likely to be introduced with the compiler work associated with these changes. Since we both cannot express the bounds on e.g. type Fields;, and we cannot write an algorithm which works over what is effectively a heterogenous collection without falling down to writing fairly involved tuple-trait-implementation spam, we instead introduced this keyword. It has a 3-piece syntax. Taking from the example of automatic enumeration and structure serialization from a potentially-future serde², and modifying it for simplicity, we can see introwospect_over’s intended behavior:

 0	use std::introwospection::*;
 1	use serde::{Serializer, Serialize};
 2
 3	pub struct GeneralStructSerializer<S, T>
 4		where S: Serializer, T: Serialize + ?Sized
 5	{
 6		// value being serialized
 7		value: &T,
 8		normal_struct_state: Option<&mut S::SerializeStruct>,
 9		maybe_field_error_index: Option<usize>
10	}
11
12	// Serialization routine for a `struct` type.	
13	impl<S: Serializer, T: Serialize + ?Sized> StructDescriptorVisitor
14		for GeneralStructSerializer<S, T>
15	{
16		type Output -> Result<S::Ok, S::Error>
17
18		fn visit_struct_mut<Descriptor: 'static>(&mut self) -> Self::Output
19			where Descriptor: StructDescriptor
20		{
21			// general structure serialization
22			let mut state = serializer.serialize_struct(
23				Descriptor::NAME, Descriptor::FIELD_COUNT
24			)?;
25			self.normal_struct_state = Some(&state);
26			let results = [
27				// !! USED HERE !!
28				introwospect_over(Descriptor::Type, Descriptor::Fields, self)
29			];
30			self.normal_struct_state = None;
31			if let Some(error_index) = self.maybe_field_error_index {
32				return results[error_index];
33			}
34			return state.end();
35		}
36	}
37
38	// Serialization routine for the fields of a `struct`.
39	impl<S: Serializer, T: Serialize + ?Sized> FieldDescriptorVisitor
40		for DefaultStructSerializeVisitor<S, T>
41	{
42		type Output -> Result<S::Ok, S::Error>
43
44		fn visit_field_mut<Descriptor: 'static, const INDEX: usize>(
45			&mut self
46		) -> Self::Output
47			where Descriptor: FieldDescriptor<INDEX>
48		{
49			if self.maybe_field_error_index.is_some() {
50				return S::Error::custom(
51					"no use: previous field serialization already failed"
52				);
53			}
54			let mut state = self.normal_struct_state.unwrap();
55			// normal structure serializing:
56			// just serialize the field!
57			let result = state.serialize_field(
58				Descriptor::NAME,
59				get_field::<Descriptor, INDEX>(value)
60			);
61			if result.is_err() {
62				self.maybe_error_index = Some(INDEX);
63			}
64			return result;
65		}
66	}

What we are doing here is using introwospect_over to cheat our way into achieving what variadic and const generics could do for us instead. introwospect_over’s entire point is to take something which identifies the owning type (Descriptor::Type), tell us what we want to iterate over (in this case the fields of a structure, so Descriptor::Fields), and finally the object which will be used to get the fields out of. Because we asked for the Fields, we will get a write out of a comma-delimited list of function calls to the visit_…_mut call that every …DescriptorVisitor-implementing type has on it. For example, using the Kitty type defined earlier in this post:

0	#[introwospect(are_little_precious_babies = "yes!!")]
1	#[non_exhaustive]
2	pub struct Kitty {
3		pub is_soft: bool,
4		pub meows: i64,
5		destruction_of_all_curtains_trigger: u32
6	}

calling introwospect_over on it in the above context would produce a comma-delimited list of visit_…_mut calls that take this form:

0	// … code from above
1			let results = [
2				// !! USED HERE !!
3				self.visit_field_mut::<<Kitty as StructDescriptor>::Fields, 0>(),
4				self.visit_field_mut::<<Kitty as StructDescriptor>::Fields, 1>(),
5				self.visit_field_mut::<<Kitty as StructDescriptor>::Fields, 2>()
6			];
7	// … code from above

The INDEX values passed into the visit_…_mut go from 0 to FIELD_COUNT - 1. This allows us to get around the lack of variadic capabilities by just having the compiler expand the list of types out for us and do the function calls we want. introwospect_over also behaves much the same for enums with their variants, just performing visitor.visit_variant_mut::<<Type as EnumDescriptor>::Variants, 0>() from 0 up to VARIANT_COUNT - 1. If there are no fields, then nothing is produced in that spot. This is similar to functions, which produce visitor.visit_parameter_mut::<<Type as FunctionDescriptor>::Parameters, 0>(), all the way up to PARAMETER_COUNT - 1.

Note that the expansion we are performing here is a spiritual equivalence, not an exact or perfect semantic equivalence. type Fields, type Variants, and type Parameters do not have the appropriate trait bounds on it. This regular-looking Rust code is inexpressible in the literal “this is exact compiled source code” sense. But, with introwospect_over, we can commit any action we want to and its perfectly legal because we are the compiler. So long as both the bounds are inexpressible and variadic programming is inexpressible in Rust, we will always need a keyword to basically cover the lack of programmatic access to various aspects of Rust.

introwospect(Type, visitor) works in a similar way. Based on what Type is, introwospect takes the following actions:

0	/// `union`?
1	visitor.visit_union_mut:::<<Type as UnionDescriptor>>()
2	// `struct` ?
3	visitor.visit_struct_mut::<<Type as UnionDescriptor>>()
4	// `enum` ?
5	visitor.visit_enum_mut::<<Type as UnionDescriptor>>()
6	// `fn`/function ?
7	visitor.visit_function_mut::<<Type as FunctionDescriptor>>()

One of the reasons this also needs to be a keyword is because there’s no way to express “do this for a union, do that for an enum, do another thing for a function, and do this for a struct” in Rust generic code. So, switching based on what type of entity we are visiting is paramount to ensuring code compiles. If the core language ever adapts a way to do this in an elegant, Rusty way, we would absolutely welcome it.

The Current Daunting Task

The long list of issues above, even with a hacked-in language feature, is not confidence-inspiring in how simple this task will be. When considering reflection — at compile-time — for C++, Reflection Study Group 7 (SG7), despite having not yet finished a full reflection specification suitable for ISO standardization, never really had the problem of “what we are doing is entirely inexpressible in the language”. Templates were so fully featured that one could do reflection in them given the right language primitives since C++03 (2003). The interface would be absolutely horrendous, but the language was effectively ready for compile-time reflection in one weirdly-shaped way or another and has been for 20 years. That C++ does not have it is a result of competing designs and the emergence of constexpr, pushing the boundaries for what is possible in compile-time contexts in C++. And rightly so: template metaprogramming one’s way to compile-time reflection is grody and disgusting, and this comes from a group — us — who have done far too much of it to great success.

With Rust, the task is far more intimidating. Both the language and the library are wholly incapable of expressing the necessary concepts to do work on these things with normal Rust. The Trait system in Rust is not powerful enough to express necessary constraints in the slightest.

For example, one can recursively express a Fields type in C++ by creating a template template <typename Type, size_t INDEX> struct FieldDescriptor; that takes the typename Type and a size_t INDEX. One can perform recursion with C++ templates, creating what is effectively a stack of types describing each field by doing FieldDescriptor<Type, INDEX - 1>. The stop condition for that template is writing a specialization — something deeply frowned upon in Rust because of soundness issues — and stopping for FieldDescriptor<Type, 0> to prevent blowing out the compiler’s memory through endless recursion.

With Rust traits, not only is INDEX - 1 not exactly a kosher operation in const generics, but there is no way to write a stop condition for FieldDescriptor<0> to tell it to stop its recursion, meaning it would eventually blow out the compiler’s memory stack or just be flagged as a straight up illegal operation. That means it is impossible to do recursive compile-time programming with type information or integer constant expressions in an even remotely normal way in Rust.

This creates an enormous number of problems for the Trait-based world of generics that Rust has built up for itself. Because we cannot properly constraint to have exactly all the field descriptors necessary, it means that generic code cannot write predictable, compile-time computation over an actually generic set of elements. Rust even lacks the ability to select for a specific subset of types that match a specific trait, because it will error if you use more than one bounding trait in the collection of implementations for any given trait.

Things that are trivially expressible in C++, Zig, and other languages become infinitely impossible in Rust’s traits and generics.

This, unfortunately, is a strong driver of how a lot of the mid-level API can and must work in Rust. Keywords introwospect, introwospect_type, and introwospect_over are all driven by deeply-seated inadequacies in Rust’s core syntaxes and language capabilities, each part a fundamental acknowledgement of a sincere issue that must be worked around in order to achieve the end-goal of proper compile-time reflection. And even when we take shortcuts with things like introwospect_type and introwospect_over, it introduces its own severe issues with the ergonomics of the programming models offered to end-users.

The Problem of Visitor-based Programming

The reason we describe the introwospect(...) and introwospect_over(...) APIs as “mid level” is because:

0. there is a better way to program for these constructs and doing so with unconditionally linear calls using 0 or more fields results in needing to remember state from previous function calls to early-exit from other function calls;
1. and, it really does fit the slang term “mid”, as in subpar quality.

For example, consider generic structure serialization we made above that was derived from the full serde implementation prior². In particular, consider this line of code at the top of the visit_field_mut function call:

0			// … rest of the code here!!
1			if self.maybe_field_error_index.is_some() {
2				return S::Error::custom(
3					"no use: previous field serialization already failed"
4				);
5			}
6			// … rest of the code here!!

Here, we have to book-keep that an error has occurred on a prior field and inject that state into this FieldDescriptorVisitor implementation. This is because introwospect_over lays out a list of objects that produces a comma-delimited list, and ultimately every visit_field_mut function for that visitor must return the same type since we put it into an array:

0			// … rest of the code here!!
1			let results = [
2				// !! USED HERE !!
3				introwospect_over(Descriptor::Type, Descriptor::Fields, self)
4			];
5			// … rest of the code here!!

This is the problem with function-based and closure-based code in Rust: by having completely different functions and, effectively, entirely unrelated scopes, objects themselves have to become responsible for book-keeping around function calls, including early cancellation (e.g., stopping if the first field fails to serialize properly).

Can We Do Better?

The better way to fix this is with a programmatic construct that allows you to access each element at compile-time in a generic fashion, such as with a compile-time for loop. For example, a (made-up, not-at-all-real) for const could solve this problem:

 0	// … below is rewritten code to use fanciful, not-real `for const` flow control
 1
 2	// Serialization routine for a `struct` type.	
 3	impl<S: Serializer, T: Serialize + ?Sized> StructDescriptorVisitor
 4		for GeneralStructSerializer<S, T>
 5	{
 6		type Output -> Result<S::Ok, S::Error>
 7
 8		fn visit_struct_mut<Descriptor: 'static>(&mut self) -> Self::Output
 9			where Descriptor: StructDescriptor
10		{
11			// general structure serialization
12			let mut state = serializer.serialize_struct(
13				Descriptor::NAME, Descriptor::FIELD_COUNT
14			)?;
15			for const INDEX in 0..Descriptor::FIELD_COUNT {
16				// normal structure serializing:
17				// just serialize the field!
18				state.serialize_field(
19					Descriptor::NAME,
20					get_field::<Descriptor, INDEX>(value)
21				)?;
22				// no "result" storage
23				// no "normal_struct_state" storage
24				// none of that nonsense!!!
25			}
26			return state.end();
27		}
28	}

Here, we are saying “this for loop runs at compile time”. It is equivalent to effectively performing a compile-time unrolling of the loop, each iteration of the loop effectively it’s own scoped body within the function. The ? usage in here just bails if the serialization fails, without us having to save any intermediate state so that the next function call of a visit_field_mut has to handle it. This is far better than the *DescriptorVisitor-based programming. This is what we would consider a proper “high level” API, that is both high in terms of abstraction level/ease of use, and in terms of quality of the usage experience.

It allows us to stay in our scope, but grasp compile-time values much better. As stated previously, this syntax is not real. It would make everything much simpler, however, and thus would be worth investigating in the long-term for doing better const programming in Rust. In general, providing more constructs which allow this seamless transition between associated const items, types, and behavior would enable not just compile-time programming use cases, but make code for the SIMD project and several other use cases far more elegant, readable, and tractable.

And There’s So Much More

There is a lot more that we could talk about insofar as Rust’s strengths and weaknesses for its generic programming. We have not even gotten to talk about the sneaky way one can introduce post-monomorphization errors to stop someone from serializing a struct with fields that are non-visible to someone outside of your crate. From the full default-serialization serde example²:

 0	// … ELIDED CODE ABOVE
 1
 2	// Private trait to trigger assertion at post-monomorphization time.
 3	trait PostMonomorphizationValidityCheck {
 4		const TRIGGER: ();
 5	}
 6
 7	/// This function takes a list of attributes, and the boolean about whether or not this
 8	// type has non-public fields, and tells whether or not we can serialize this using
 9	// the default serializer at compile-time.
10	const fn is_default_serializable(
11		has_non_visible_fields: bool,
12		attributes: &[AttributeDescriptor],
13	) -> bool {
14		if !has_non_visible_fields {
15			return true;
16		}
17		std::introwospection::contains_attribute("allow_private", attributes)
18	}
19
20	// Serialization routine for a `struct` type.	
21	impl<S: Serializer, T: Serialize + ?Sized> StructDescriptorVisitor
22		for DefaultStructSerializeVisitor<S, T>
23	{
24		type Output -> Result<S::Ok, S::Error>
25
26		fn visit_struct_mut<Descriptor: 'static>(&mut self) -> Self::Output
27			where Descriptor: StructDescriptor
28		{
29			// Implementing an associated constant that fits the trait requirements
30			// allows us to bypass the original trait checks, but defer
31			// the actual compile-time trigger to post-monomorphization time, much
32			// like a C++ template second-stage usage check.
33			struct C<CheckedDescriptor> where CheckedDescriptor: StructDescriptor;
34			impl PostMonomorphizationValidityCheck for C<Type> {
35				const TRIGGER: () = assert!(
36					!is_default_serializable(
37						Descriptor::NON_VISIBLE_FIELDS,
38						Descriptor::ATTRIBUTES
39					),
40					concat!(
41						"We cannot serialize a structure with "
42						"non-visible private fields and no "
43						"`#[introwospection(allow_private)]` attribute."
44					)
45				);
46			}
47			// Trigger the check upon post-monomorphization of this function.
48			const _NO_INACCESSIBLE_FIELDS: () = <C as InaccessibelFieldCheck>::TRIGGER;
49			// … MORE ELIDED CODE BELOW IN FUNCTION
50		}
51	}
52	// … MORE ELIDED CODE BELOW

There are genuinely cool things that can be done in Rust with generics, and genuinely awesome things that a Trait-based system like Rust allows, but at the moment compile-time reflection is an incredibly steep order.

Going Forward

There is a lot that can and should be discussed. For the moment, what Shepherd’s Oasis is going to focus on are the simple library things mentioned above such as

0	pub fn std::mem::discriminant_at<EnumType>(
1		index: usize
2	) -> std::option::Option<std::mem::Discriminant<EnumType>>;

and a few other low-level utilities that will be in service of the code we would like to work on. Implementing the keywords and the rest of the functionality in time for the end of this Grant¹ period — another 2 months and 1 week — seems excessively beyond our capabilities as first-time rustc contributors. Shepherd’s Oasis will also look into aiding Waffle and DrMeepster in pushing and stabilizing std::mem::offset_of! for enumerations, unions, and more to make sure the syntax is solid.

As part of that discussion, one of the things we would like to address up-front is the perception of increased breakable API surface area. Due to providing more information at compile-time, there exists a chance that folks may:

• rely on compile-time computations that feed into other types (but not the same type, as that would be errored by the cycle checker);
• rely on the ordering of fields for internal and external code;
• and, rely on the number of fields available or accessible from an external crate.

This is a very real concern, and is especially powerful when dealing with external crates from e.g. crates.io. While crate-internal and private data types and functions can always be handled with grace, changes to external crates could have strong ripple effects in day-to-day code. If we were working in the C or C++ ecosystem, this is something we would have to consider very seriously as part of a proposal to add such features to the language. Thankfully, for Rust, this is entirely a non-issue. Unlike C and C++ which still rely on — in all perfect reality, honesty, and fairness — grotesque Perl programs, python scripts, and willful amnesia-inducing autoconf scripts mixed in with makefiles, Rust has a robust package management system built into its very core. cargo.toml has strict semantic versioning, feature specifications, and is paired with a language that was developed in an age where source control and repository control is both ubiquitous and personally catered to by crates.io, with an enormously responsive community.

Sage Griffin did not previously co-lead a whole team of the most fearless Ferris-lovers at crates.io to blaze a path of glory for the entire Rust ecosystem just to have folks suddenly be scared of change.

If a crate changes a structure’s public fields, or changes function parameters in a major semantic versioning upgrade, or changes the type of a specific enumeration’s variant, the APIs presented here give you all the power to properly warn/error at compile-time about such changes. And, if you are too scared to make such a change, nothing stops the user from editing their cargo.toml to roll back to an older version if an upgrade is not feasible. We are past the stage where we have to hear horror stories about someone who is building against 25 year old object files because they blew up the only copy of the source code back in the day, and therefore have to maintain exact compatibility with a collection of the world’s worst oldest and worst bundle of static libraries. We have the infrastructure and capability to preserve compatibility locally while pushing for better improvements globally. Most of these tools — from cargo to crates.io — were provided at great burnout and personal cost to elder Rustaceans, battle-hardened software developers, weary SREs, and dedicated computer scientists.

We strong believe individuals scared of such scenarios should use the tools made available to them for their own personal advantage and comfort.

We also believe that we have achieved the right shape for the API we want. It is type-based, and all the values exist at compile-time through the use of associated const items. We may also chime into the team working on const generic expressions and general-purpose Rust const-eval to provide some feedback from time to time as we work on things. We hope that what we wrote here can be useful inspiration for them, and help guide the language to provide APIs and language features that make compile-time programming not just on-par with other languages in their area, but far better than they could ever hope to be.

Appreciation

We would like to thank the following people, who put up with a lot of our inquiries and, in some cases, got us into this trouble in the first place (this is a #[non_exhaustive] list)⁷.

• Manish Goregaokar, for getting us involved in this titanic endeavor in the first place and encouraging us to try for better generic programming and serialization in Rust (Website, GitHub).
• Miguel Young de la Sota, for also getting us involved in this titanic endeavor in the first place and sowing the seeds of chaotic improvement (Website, GitHub, Art).
• Waffle, for helping us get off the ground with building rustc and solving strange errors. Also, for helping DrMeepster merge std::mem::offset_of! even as we were writing this article (Website, GitHub).
• boxy, for being an expert on const generics and putting up with a million questions from us, many of which were very, very basic; she also aided in getting us off the ground for rustc builds (GitHub).
• compiler-errors, for also putting up with a lot of our worst and most boring questions and helping us stop erroring the rustc build (GitHub).
• Callie, for providing a ton of ideas for the implementation and shape of the Descriptor types, working through how Discriminant<T> may be used, helping write a more elegant str_equals in const fn Rust (that still compiles great!) and so much more.
• Jubilee, for spell checking and hearing out random ideas, spell checking a few of our initial pull requests and docs, and pinging us in Zulip (GitHub).
• Nilstrieb, for fielding questions and helping us work through many of the implications of the Trait system and const fn/const generics (GitHub).
• Alyssa Haroldsen, who did a monumental amount of effort in explaining post-monomorphization error techniques, special generic handling, tuple map implementations (and their various compile-time pitfalls), and more (Website, GitHub, Twitter, Fediverse).
• oli-obk, for guidance and encouragement with kicking off with the Rust compiler (GitHub).

That’s All, Folks!

Congratulations on making it to the end of this very long report. We hope reading this was as beneficial as it was for those of us who wrote it.