Universal processing of references and target frameworks

[ForNeVeR]

Extracted from #947 and comments in #948.

The Problem

When compiling C code, Cesium relies on the following two libraries:

• Cesium.Runtime of an appropriate/compatible version, containing Cesium runtime helper code and various types
• the "system library", meaning either netstandard.dll or System.Runtime.dll¹

Cesium will add references to these libraries to the user application, and they are expected to be available to the user application in runtime. During compilation, Cesium will look up the required types from these libraries for the purpose of the compilation.

These two libraries are "hard-coded" in a sense — Cesium's AssemblyContext knows about them as MscorlibAssembly and AssemblyContext.CesiumRuntimeAssembly. If they are missing, compilation will break.

Cesium extracts the runtime library as Cesium.Runtime.dll from its own directory (so, uh, it will always be the netcore version, even under .NET Framework? another bug, probably) — and has a type Cesium.CodeGen.TargetRuntimeDescriptor to describe "the runtime". We have several runtimes hard-coded: we try to use .NET 6 library version for .NET 6+, .NET Standard 2.0 library for .NET Standard compilation, and (erroneously) netstandard.dll version 4.0 when targeting .NET Framework 4.8. This is a hard-coded set of frameworks; technically, Cesium only knows about these.

This situation has led us to several issues, including but not limited to #947. It is hard to update Cesium to support new target frameworks currently.

The Proposal

I propose a set of changes to how Cesium works with these libraries and project references in general, and also how the distinction between Cesium SDK or Cesium standalone compiler should work.

First, several postulates:

1. When compiling in the command line, it should be easy (no additional arguments required or anything) to target the current framework Cesium compiler runs on. This should be the default option. Currently Cesium runs on .NET 10 — it should generate binaries and .runtimeconfig.json compatible with .NET 10.
2. It should be possible to adjust this behavior to target any other version via the command-line compiler if needed. But some user actions (passing additional reference paths to mscorlib.dll or netstandard.dll or Cesium.Runtime.dll compatible with the current TFM) are acceptable. We shall support .NET Framework and .NET Standard as long as viable, while it's supported by the environment vendor.
3. Usage in a CMake toolchain or command line should be able to work without referencing NuGet packages or anything like that; this kind of usage should be simple if targeting the "current framework".

Then, how we can achieve this.

1. Migrate away from the current explicit assembly paths (AssemblyContext.MscorlibAssembly/CesiumRuntimeAssembly, TargetRuntimeDescriptor) to a general concept of reference assembly list.
2. When resolving any necessary type (be it a type from System or from Cesium.Runtime), go through the list if pre-determined order and resolve the type. We should not hard-code which type comes from where, and go through a generic lookup process instead.
3. The Cesium SDK should pass the actual references from the framework (the set of reference assemblies determined by the SDK), and Cesium.Runtime obtained from a NuGet package to the list of references in the compiler by default. These should be first so that they have priority. I imagine the users might wish to customize these, so is should be possible to opt-out of this behavior .
4. The Cesium compiler (both the command-line one and CMake one) should allow passing one of these:
   • a full list of reference assemblies (so, the user resolves mscorlib.dll and Cesium.Runtime themselves)
   • a short standard framework name (--framework net9.0/net10.0/net48/netstandard2.0) — Cesium should be able to construct a correct set of references for these whenever possible
     • mind that it may not be possible if targeting a previous version of the current runtime — e.g. it might not be possible to target .NET 8 if Cesium runs on .NET 10; we should evaluate this more precisely and perhaps just forbid this kind of use
   • a shorter target framework name (the current trinity NetFramework/NetStandard/Net) — allowing Cesium to choose the "best" version itself
   • nothing (default): in this case, Cesium silently targets the "current runtime", .NET 10 currently

My understanding is that this is more or less whan Roslyn does (in its command-line form and SDK form), so this is what the ecosytem nudges us into.

Implementation Notes

Quoting my text from #947 (comment):

So, together with @pkazakov-dev we found how it works in C#.

During build .NET SDK calculates @(ReferencePathWithRefAssemblies) (mostly calculated during ResolveTargetingPackAssets MSBuild target), and passed that to the C# compiler.

The compiler then uses these assemblies as references/inputs for compilation.

In fact, the same target is part of a compilation based on Cesium SDK as well! Cesium just ignores these items.

We can easily make Cesium to respect @(ReferencePathWithRefAssemblies) in either of two ways.

1. In Cesium SDK's compilation task, we could filter these input assemblies, locate the system runtime DLL among them (System.Runtime.dll, mscorlib.dll, netstandard.dll — which one depends on the target framework), and use it for further compilation.

   We could also include detection of Cesium.Runtime.dll here, although we might skip that for now, and focus on the system runtime assembly only.

2. Alternatively, we could change the way we approach the compilation. Right now, our two referenced assemblies (system runtime & Cesium runtime) are represented as two different hardcoded assemblies — two separate strict arguments.

   We could consider not treating them as two distinct assemblies, but instead implement a generic mechanism to search among referenced assemblies. All places where we currently look up things in the system runtime or in the Cesium runtime would need to be refactored to use a generalized search across references. Then this way of passing inputs to the compiler would work naturally on its own — no more distinct arguments, and just type resolve across the all referenced assemblies.

This issue is inspired by this discussion thread, and is generally a follow-up to #948.

Footnotes

1. A fun fact is that while we technically support SystemAssemblyKind.MsCorLib to use mscorlib.dll, we don't really use that — when targeting .NET Framework, we rely on netstandard.dll. This seems to be a bug. ↩