UE resource management: Engine packaging resource analysis

By default, when a project is packaged in the UE, BuildCookRun will be pulled up to execute a series of processes such as Compile/Cook/Pak/Stage. In UE, only the resources participating in cook will be packaged, but it usually contains many unexpected resources, which may cause trouble. What resources does the engine rely on? And how to manage the resources of UE participating in packaging.

This article starts with analyzing the rules of resource Cook during UE packaging, and studies which resources will be Cook during packaging. Understanding this is very useful for resource management. Based on this, we can realize the custom Cook process, assign Cook tasks to different processes, and even realize the parallelization of machines to accelerate the construction process of UE.

In addition to uasset resources, there are many non asset files during packaging, such as ini, Shader Library, AssetRegistry, or script files added in the project. In the previous article UE hot update: requirement analysis and scheme design As mentioned in, UE does not collect them in the Cook stage (Shader Library and AssetRegistry are generated in the Cook stage). This article will not be discussed for the time being, and a special introduction will be written later.

The resources of UE participating in Cook can be logically divided into this category:

Key resources set by the project, such as StartupMap, GameMode, GameInstance, DefaultTouchInterface, etc., are necessary resources
Several UI directories packaged by default
Resources loaded by code when the engine starts
Cook resources configured in project settings (including resources marked with directory to always cook, PrimaryAssetLabel, etc. to cook)
By executing fgamedelegates:: get() Getcookmodificationdelegate() the resource passed to CookOnTheFlyServer.
Under certain conditions, if no resources are specified, analyze the resources in the project and plug-in directory
Localized resources (UE supports using different resources for different cultures, but it is not commonly used)

The resource analysis process of UE is very complex and scattered in various places, including various condition detection. It is very inconvenient to completely analyze the resources that the project depends on. This is also the reason why it is not convenient to accurately determine which resources are packaged by the UE.

Based on these pain points, I plan to implement a concise and standardized resource collection and packaging process based on HotPatcher. In essence, packaging resources is to package the necessary resources needed by the engine and program running. This is the most fundamental requirement. The resource allocation and detection in UE are so complex to meet it, but we can simplify it for unified analysis and management.

CookCommandlet

The UCookCommandlet is provided in the engine to realize the Cook of resources. In the packaging process, it is pulled up by the UAT. The default Cook command is as follows:

D:/UnrealEngine/Engine/Engine/Binaries/Win64/UE4Editor-Cmd.exe
D:/UnrealProjects/Blank425/Blank425.uproject
-run=Cook
-TargetPlatform=WindowsNoEditor
-fileopenlog
-unversioned
-abslog=D:/UnrealEngine/Engine/Engine/Programs/AutomationTool/Saved/Cook-2021.12.07-15.47.10.txt
-stdout
-CrashForUAT
-unattended
-NoLogTimes
-UTF8Output

It will be executed in the main of UCookCommandlet:

/* UCommandlet interface
 *****************************************************************************/

int32 UCookCommandlet::Main(const FString& CmdLineParams)
{
  COOK_STAT(double CookStartTime = FPlatformTime::Seconds());
  Params = CmdLineParams;
  ParseCommandLine(*Params, Tokens, Switches);
  // ...
}

After some parameter detection, it will pass the execution process to CookByTheBook, create CookOnTheFlyServer, and call startbookbythebook.

When the engine packs, the resources will be cooked in CookOnTheFlyServer and generate shaders and AssetRegistry. It can be said that CookOnTheFlyServer is the process of serializing uasset s in the general format in the editor into platform format in the UE packaging process.

Resource analysis

The idea of loading resources during UE packaging is: first find the local uasset file, convert the path to PackageName, load it, load the dependent resources, and then Cook them together.

StartupPackages

In cookontheflyserver In UCookOnTheFlyServer::Initialize of CPP, add the resources loaded into memory to cookbythebookoptions - > startuppackages:

CookOnTheFlyServer will add them to the Cook list in the subsequent process and handle the redirector.

AllMaps

AllMaps is added to MapIniSections without specifying any maps from the command line:

It is defaulteditor Section in ini:

[AllMaps]
+Map=/Game/Maps/Login
+Map=/Game/Maps/LightSpeed
+Map=/Game/Maps/VFXTest

GlobalShader will be compiled globally before startup:

Get resources through GRedirectCollector:

UI

By default, UE will set baseeditor Add the directory under ContentDirectories in ini to the Cook list: Engine/Config/BaseEditor.ini#L271

The default configuration in the engine is as follows. You can also modify the defaulteditor Ini add other directories:

[UI]
; Directories specifying assets needed by Slate UI, assets in these directories are always cooked even if not referenced
+ContentDirectories=/Game/UI
+ContentDirectories=/Game/Widget
+ContentDirectories=/Game/Widgets
+ContentDirectories=/Engine/MobileResources

Resources in these directories will be packaged: CookOnTheFlyServer.cpp#L5519

//@todo SLATE: This is a hack to ensure all slate referenced assets get cooked.
// Slate needs to be refactored to properly identify required assets at cook time.
// Simply jamming everything in a given directory into the cook list is error-prone
// on many levels - assets not required getting cooked/shipped; assets not put under 
// the correct folder; etc.
if ( !(FilesToCookFlags & ECookByTheBookOptions::NoSlatePackages))
{
  TArray<FString> UIContentPaths;
  TSet <FName> ContentDirectoryAssets; 
  if (GConfig->GetArray(TEXT("UI"), TEXT("ContentDirectories"), UIContentPaths, GEditorIni) > 0)
  {
    for (int32 DirIdx = 0; DirIdx < UIContentPaths.Num(); DirIdx++)
    {
      FString ContentPath = FPackageName::LongPackageNameToFilename(UIContentPaths[DirIdx]);

      TArray<FString> Files;
      IFileManager::Get().FindFilesRecursive(Files, *ContentPath, *(FString(TEXT("*")) + FPackageName::GetAssetPackageExtension()), true, false);
      for (int32 Index = 0; Index < Files.Num(); Index++)
      {
        FString StdFile = Files[Index];
        FName PackageName = FName(*FPackageName::FilenameToLongPackageName(StdFile));
        ContentDirectoryAssets.Add(PackageName);
        FPaths::MakeStandardFilename(StdFile);
        AddFileToCook( FilesInPath, StdFile);
      }
    }
  }

  if (CookByTheBookOptions && CookByTheBookOptions->bGenerateDependenciesForMaps) 
  {
    for (auto& MapDependencyGraph : CookByTheBookOptions->MapDependencyGraphs)
    {
      MapDependencyGraph.Value.Add(FName(TEXT("ContentDirectoryAssets")), ContentDirectoryAssets);
    }
  }
}

Directory to Alway cook

Project Setgings-Directory to Alway cook DirectoriesToAlwaysCook

Maps

AlwaysCookMaps CookOnTheFlyServer.cpp#L5223
MapToCook CookOnTheFlyServer.cpp#L5253

[/Script/UnrealEd.ProjectPackagingSettings]
+MapsToCook=(FilePath="/Game/HandheldAR/Maps/HandheldARBlankMap")

Never Cook CookOnTheFlyServer.cpp#L5317
AllMaps CookOnTheFlyServer.cpp#L5266

Cultures

This does not represent multilingualism, but supports the use of different resources for different cultures. Asset Localization

Acquisition code of Cultures resource: CookOnTheFlyServer.cpp#L6714

Read from project settings

It will traverse all RootPaths, such as / Engine, / Game and the resource root directory of the plug-in:

Such as the resources in the following directory, and the subdirectories will be recursive:

/Game/L10N/en/

DefaultTouchInterface

DefaultTouchInterface is a virtual rocker class configured in the engine. It may not be dependent on other resources, but it also needs to be packaged. Therefore, it will be obtained separately when cooking:

FConfigFile InputIni;
FString InterfaceFile;
FConfigCacheIni::LoadLocalIniFile(InputIni, TEXT("Input"), true);
if (InputIni.GetString(TEXT("/Script/Engine.InputSettings"), TEXT("DefaultTouchInterface"), InterfaceFile))
{
  if (InterfaceFile != TEXT("None") && InterfaceFile != TEXT(""))
  {
    SoftObjectPaths.Emplace(InterfaceFile);
  }
}

GetCookModificationDelegate

Binding agent, which can be passed to the CookCommandlet. Files to Cook:

// allow the game to fill out the asset registry, as well as get a list of objects to always cook
TArray<FString> FilesInPathStrings;
FGameDelegates::Get().GetCookModificationDelegate().ExecuteIfBound(FilesInPathStrings);

Note that the need passed is the absolute path of the uasset file, not the resource path such as / Game/xxx.

AssetManager

Through uassetmanager:: get() Modifycook function to access PrimaryAssetTypeInfo (configuration in project settings, PrimaryAssetLabelId, etc.).

These two are added by default in the new project. ModifyCook will scan all PrimaryAssetId resources and add the specified resources to PackageToCook. Note that only this resource.

If the command line does not explicitly specify any resources, and from fgamedelegates:: get() GetCookModificationDelegate(),UAssetManager::Get(). If modify does not get any resources, add all resources of plug-ins and projects:

The execution conditions are:

DefaultEditor. AlwaysCookMaps in ini is empty, AllMaps is empty
List of maps to include in a packaged build in project settings is empty
DirectoriesToAlwaysCook in project settings is empty
FGameDelegates::Get(). The resource obtained by getcookmodificationdelegate() is empty
UAssetManager::Get(). The resource obtained by modifycook is empty

The / Engine, / Game and all plug-in enabled umap, uasset:

// If no packages were explicitly added by command line or game callback, add all maps
if (FilesInPath.Num() == InitialPackages.Num() || bCookAll)
{
  TArray<FString> Tokens;
  Tokens.Empty(2);
  Tokens.Add(FString("*") + FPackageName::GetAssetPackageExtension());
  Tokens.Add(FString("*") + FPackageName::GetMapPackageExtension());

  uint8 PackageFilter = NORMALIZE_DefaultFlags | NORMALIZE_ExcludeEnginePackages | NORMALIZE_ExcludeLocalizedPackages;
  if (bMapsOnly)
  {
    PackageFilter |= NORMALIZE_ExcludeContentPackages;
  }

  if (bNoDev)
  {
    PackageFilter |= NORMALIZE_ExcludeDeveloperPackages;
  }

  // assume the first token is the map wildcard/pathname
  TArray<FString> Unused;
  for (int32 TokenIndex = 0; TokenIndex < Tokens.Num(); TokenIndex++)
  {
    TArray<FString> TokenFiles;
    if (!NormalizePackageNames(Unused, TokenFiles, Tokens[TokenIndex], PackageFilter))
    {
      UE_LOG(LogCook, Display, TEXT("No packages found for parameter %i: '%s'"), TokenIndex, *Tokens[TokenIndex]);
      continue;
    }

    for (int32 TokenFileIndex = 0; TokenFileIndex < TokenFiles.Num(); ++TokenFileIndex)
    {
      AddFileToCook(FilesInPath, TokenFiles[TokenFileIndex]);
    }
  }
}

However, a filter is added by default, which will exclude resources in the Engine directory (/ Engine) and resources in the localization directory (/ * / L10N /).

In fact, it will add projects, plug-ins and all uasset s and umap s in the L10N directory.

The collected and packaged contents are mainly in the UCookOnTheFlyServer::CollectFilesToCook function: CookOnTheFlyServer.cpp#L5200 Yes.

Map, GameMode and GameInstance set for each platform: CookOnTheFlyServer.cpp#L5450
DefaultTouchInterface in InputIni: CookOnTheFlyServer.cpp#L5499

[/Script/Engine.InputSettings]
DefaultTouchInterface=/Engine/MobileResources/HUD/LeftVirtualJoystickOnly.LeftVirtualJoystickOnly

Dependent loading of Cook

Although the resources that will be included when the engine is packaged are listed above, they are not all, because they are still a single resource or a single directory and do not contain resource dependencies. Therefore, when the UE performs Cook, it will also perform substantive dependency analysis.

Consider the following two questions:

If a C + + implemented Actor is placed in a map and a resource is loaded in its constructor code, how to package it?

AMyActor::AMyActor()
{
  // Set this actor to call Tick() every frame.  You can turn this off to improve performance if you don't need it.
  PrimaryActorTick.bCanEverTick = true;

  UTexture2D* Texture2D = LoadObject<UTexture2D>(nullptr, TEXT("/Game/TextureResources/T_ImportTexture.T_ImportTexture"));
}

Place it in the scene without any dependency:

How to package if there is no resource dependency in the AssetRegistry reference relationship? For example, the BoneCompressionSettings and CruveCompressionSettings configurations of AnimSequence are not in the dependency relationship. They cannot be found when obtaining the dependency of animation sequence from AsssetRegistry.

However, they are recorded in the ImportTable of uasset:

You can also see from the uasset of Cooked:

In this case, why not scan dependencies directly from the Asset's ImportTable?

This is because accessing the ImportTable requires real loading of resources. When the amount of resources is very large, it consumes hardware resources and time, while accessing dependencies from AssetRegistry does not need to load resources into memory, so the speed is very fast.

So how to solve these problems? First, start with the default implementation mechanism of UE. Two points should be made clear:

When the engine starts, CDO will be created and the constructor of the class will be executed
When loading the resources of the UE, the resources it depends on will also be loaded

Therefore, based on this idea, UE implements a scheme when cooking, listens to all created uobjects and adds them to the Cook list to ensure that the resources loaded in the C + + constructor and the dependent resources loaded through ImportTable can also be cooked.

struct FPackageTracker : public FUObjectArray::FUObjectCreateListener, public FUObjectArray::FUObjectDeleteListener
{
  FPackageTracker()
  {
  	for (TObjectIterator<UPackage> It; It; ++It)
	{
		UPackage* Package = *It;
	
		if (Package->GetOuter() == nullptr)
		{
			LoadedPackages.Add(Package);
		}
	}
    GUObjectArray.AddUObjectDeleteListener(this);
    GUObjectArray.AddUObjectCreateListener(this);
  }
  ~FPackageTracker()
  {
    GUObjectArray.RemoveUObjectDeleteListener(this);
    GUObjectArray.RemoveUObjectCreateListener(this);
  }

  virtual void NotifyUObjectCreated(const class UObjectBase *Object, int32 Index) override
  {
    // ...
  }

  virtual void NotifyUObjectDeleted(const class UObjectBase *Object, int32 Index) override
  {
    // ...
  }
};

Therefore, based on the same idea, we only need to obtain the dependency of resources through AssetRegistry, store a rough resource list, and then listen for the creation of UObject during Cook. If it is not in the scanned resource list, it will be added to the Cook queue, so as to realize the complete resource packaging process.

Comparison of Cook results of untracked ImportTable (default Cook results of UE on the left and customized Cook on the right):

Comparison of Cook results of tracking ImportTable (the default Cook results of UE are on the left):

It can be seen that the resources of the Cook that tracks the ImportTable are consistent with the default Cook of the UE.

summary

This article analyzes the analysis process of the default packaged resources of the engine, which can change the resource analysis of UE from black box to deterministic. Through the analysis results of this article, we can obtain a rough list of packaged resources through dependency analysis, and listen to the creation process of UObject during Cook, so as to analyze the resources that are not in the dependency list but actually depend on, The implementation is consistent with the default Cook resource of the UE.

Analyzing the packaging resource analysis process of UE can replace the default Cook process of UE, realize multi process, and even cross machine Cook task allocation, which can greatly improve the packaging efficiency of UE. Later, I will implement the multi Cook mechanism for HotPatcher.

Added by schajee on Thu, 06 Jan 2022 03:07:13 +0200

Programming VIP