Guide to PhyreEngine Part II: The Phyre Asset Pipeline

In the last post we ended with a window which rendered a solid blue screen. In this post I will outline what happens when you build a project and describe what must be done in order to load and render an object to the screen.

The Phyre Asset Pipeline

Models, textures, scripts and just about every other type of asset must be converted to the Phyre format. The .phyre files are platform specific binary versions of your asset. This is not unusual practice, for example Ogre does something similar with .mesh files which are binary versions of your mesh that Ogre can load. What is the point of converting assets to binary formats? To answer this you must understand what a binary file is.

Binary Objects

If you’ve ever opened up an asset file, such as a Collada .dae file you will see that it is human readable XML. This data is then parsed by the engine which looks for attribute names in the XML file and does something with it. If you do not serialize your data to a binary format this means the engine must parse this human readable data every time you start your program.

The best way to explain this (and almost anything!) is with a sample. Below I will describe how you would go about parsing a mesh in your day to day life.

Say you have a Wavefront .obj file. You export your mesh from Maya into an .obj file which contains a “mesh”. What is a mesh? A mesh is just a bunch of vertex positions and maybe you have normals and texture coordinates too but at the most basic level, you must have at least vertex positions. If you open the .obj file you will see something that looks like this:

Sample OBJ data, source: http://en.wikipedia.org/wiki/Wavefront_.obj_file

Notice what we have here? At the start we have identifiers, v for vertex and vt for texture coordinates etc. The simplest way to parse this data would be to scan the first letter in each line, do a string compare:

If (firstChar == “v”)
     vertices.push_back(loaded vertex)
if (firstChar ==”vt”)
     texCoords.push_back(loadedUV)

Assume loadedVertex is a simple struct which has four floats.

This is tedious as heck, but it works. We are loading each vertex and adding it to a list of vertices. This array of vertices is eventually thrown into a vertex buffer which is then sent to the GPU to render. If you do not serialize to binary, you have to do the whole string compare parsing process every single time. 

Every time you parse your data, you will end up with the exact same array which is sent to the GPU. This is very slow, imagine loading a mesh with thousands of vertices; this leads to an absurd amount of string comparisons (which are slow) furthermore, imagine you had to debug something and had to wait for this data to be parsed every time you ran the project. You’d go insane!

Binary serialization allows you to skip the whole text parsing process by essentially writing the array generated from the parsed data to a file. Then instead of parsing text every time, you can just load that array and It is up to the engine to do treat the data correctly. This is very fast because you are just taking a chunk of data from a file and putting it into memory. Of course, there is more to generating the array, different platforms may require different things to be done to the data, this is why Phyre must generate a different binary file for different platforms.

This is the basic idea behind binary formats. I know I’m not the best with words and what not but hopefully from this example you understand the idea and need for binary formats.

Back to Phyre, PhyreEngine generates it’s platform specific binary files as a pre-build event. Let me explain what this means:

PhyreEngine is a multiplatform engine, it supports windows with DirectX and OpenGL, bit, PS3 and PSVita. Depending on which platform you are building for, the engine will generate a binary .phyre file which is optimized for that platform. This is done in a build event. In my past experiences, when I clicked “build and run” Visual Studio would starting building MY project and run my game. PhyreEngine introduced me to the concept of “build events”.

What the heck is a build event? 

Pre-build events in Visual Studio. Note the red box, at different stages of the build you can choose to add additional tasks. The green box is the command to call the PhyreAssetGather tool, I will describe this in detail in the next post.

This means when you click “build” you can tell Visual Studio to do things before it starts running your project. As you can probably infer from the image above, you can tell Visual Studio to do intermediate task at different parts of the build process. PhyreEngine uses a pre-build event to call the AssetGather tool.

The AssetGather uses a .xml file which describes which assets need to be loaded as well as their file paths and generates a header file. Notice in the first line of the XML file below, there is a bit called “pathfix” this means all of your assets need to be under that directory. This directory is known as your media directory and you want all of your assets to be stored relative to that path. It is crucial that you do this because it will save a lot of headaches caused by assets not being found.

Excerpt from our asset XML file.

This XML file is processed and turned into a .h header file

Excerpt from the header generated from the XML file. Note that all assets have been converted to .phyre format. This header is included in our project and is used to load assets from disk. 

This is why the AssetGather tool must be run before your project can be built. This tool generates a header which your project includes. Essentially, you are including a header that does not exist when you click the build button for your project. It is generated as a pre-build event which occurs prior to the compiling of your code.

The AssetGather tool calls the AssetProcessor for each asset defined in the XML file. The AssetProcessor is a beast of its own and at this time I can only describe its interworking’s as black magic, fortunately PhyreEngine’s source is very well commented and one day I plan on looking into it.

Now that we have a better understanding of what the pipeline is like, we can move on in this guide. In the next post, I will demonstrate how to correctly set up your media directories and the pre-build events shown above.