The new storage system has arrived

I am happy to be able to say that I've wrapped up the first round of development on the new storage & query system. I threw every edge case I could think of at it and was able to, eventually, pass them all.

What's new with the new system?

Learning my lesson from past development, I did my best to make this new version as open-ended and unconstrained as possible.

A proper API. The storage and query API have been cleaned up and condensed to make things easier and more powerful for module authors. (Sorry for the inline images, the OpenCollective editor won't allow me to author code blocks?)

No containers. Unlike the v5 system, there is no hardcoded "container" hierarchy. "Scopes" are arbitrary and defined by the query author, making new or ad hoc scopes trivial to implement.

Fine-grained inheritance. Inheritance in v5 was baked into the system and difficult to modify or work around. The new system allows inheritance to be enabled (or not) between any "scopes", or even on a per-fetch basis.

No more file configurations. This one pleases me greatly: file-level configuration is now no different than anything else. This will make it much easier to implement per-file configuration settings going forward.

No "magic" fields. In v5, certain fields received special processing to enable out-of-order evaluation for situations like the one shown below. This worked for most projects, but not for everyone, and it added extra processing and overhead. The new system is able to handle situations like these as a general case, with no workarounds.

Reduced configuration constraints. It now possible to share projects between workspaces, just as you would any other configuration. Containers which previously required the use of special APIs now work like any other field. Using the v5 scripting syntax (which isn't implemented in the new version), that means you can do things like:

On performance

When I announced that I was working on a new system, several people were quick to raise performance as a critical concern. While it is too soon for performance testing—this is just the "keep it simple; make it work" version—I have tried to design the overall approach for optimizability. The new system is simpler and "flatter", provides more opportunities for caching intermediate results, and should translate to C reasonably well. Once the new system has been proven fit for purpose and there are enough features in place to run a real world test I will loop back to complete those optimizations.

Next steps

All of these improvements and advances are academic until you can actually generate some output, so that's next on my list. In order to cut to the chase and get us to a "v5 vs. v6" decision as quickly as possible, without getting mired in all of the complexities of targeting a specific toolset, I'm planning to build a JSON export module over the new code. (This is something I've wanted in the past to assist with tooling, automation and visualization anyway.) That should allow me to quickly build out the scaffolding and APIs required by all exporters, as well as provide a good testbed for bringing the remaining features online as we move ahead. Feedback on that approach, or alternative suggestions, are welcome.

v5 vs. v6

It's my hope that with an exporter in place folks will have enough to see and touch to consider where things go next. Do we backport the new code to v5 and rework all of the existing actions, potentially breaking existing projects? Or do we flip the bit on v5, mark it "stable", and push ahead with a v6 instead? (I have an opinion, but keeping an open mind.) When the time comes I'll open an RFC issue on GitHub so everyone can have their say on the matter.

Feedback is welcome and appreciated!

These are big changes and I welcome questions, suggestions, and concerns. Everything is up for discussion, from the feature set, to the coding style. You can message or DM me at @premakeapp, email at [email protected], or open an issue on the premake-next GitHub project.

And as we do: a shout out to CitizenFX Collective and Industrious One and everyone else who has helped back us so far. Getting this new system shipped crosses a big dependency off the to-do list, and I'm not sure it ever would have seen the light of day without your help. Many and sincere thanks to all of you! 🙌