rdMeshTools Tutorial 11 – Creating and Using Masks

Version 1.42 of rdMeshTools introduces a new tool which allows you to create Mask Textures from UVs – each UV island can have it’s own color, allowing masks to be used to reference higher res, shared textures.

This can help a lot with memory optimization along with less streaming data, and single draw-call rendering of meshes.

This tutorial will create some simple Masks from a selection of meshes and render the meshes using shared textures in various ways.

The simplest way in the material is to render the referenced textures using local space positions. The first tutorial mesh is an exterior wall that will always be vertical meaning we can use the Y&Z local space positions to render the high-res reference textures – Values are between 0.0 and 1.0.

This can be expanded to WorldSpace based TriPlanar for more complex meshes, the second tutorial mesh is a cube where each face is a referenced texture. This uses the TriPlanar material function to render them across all axis.

The other way to render when local planar or world space tri-planar won’t suffice is to have multiple UVs in the mesh, have a UV for each mask item for the high-res texture lookup. The tutorial example mesh for this is a PineTree, using an extra 2 UV channels, one for the trunk and one for the leaves.


Step 1. Create the Wall mesh

The tutorial uses Blender to create the meshes, you can use any modelling software or the UE Mesh Editing tools. You can edit existing meshes UV channels too if needed, often their default UV layout is fine to use as-is.

  1. Create a mesh with two planes, both slightly off center in the X axis, and both facing outwards.

2. Edit each of the planes UVs so they occupy half of the UV space each, one per side – make them smaller than the bounds so there is a reasonable gap between each island and the borders – when creating small masks (e.g. 32×32) the gaps also get reduced.

3. Now merge both into one mesh – go to Object mode, select both the meshes in the outliner and press “Ctrl+J” to merge together.

Now save it and import into UE – you should get a single mesh with no materials, but a UV with both side channels combined – this is what the mask is made from. (Note there may be a lightmap UV channel added during importing).

If your UV channels don’t import as expected, try turning off Nanite for the mesh in the import settings and try again.


Step 2. Create the Wall Mask and Material

Now locate the new wall mesh in the Content Browser and create a new Material for the mesh there (right-click ‘Material’). We’ll edit this shortly.

Right-Click on the new mesh and select “rdMesh Tools->Tools->Create UV Masks”

Set the Texture size to 32×32 (can be small as it’s a very simple UV layout), to 8bit GreyScale and give it a name and a folder. We want each island to be a different color, so tick “Unique Islands”, but not “Per Material”. Set the Index Increment to 20 so there’s a dramatic difference. The value ’20’ is using the byte based values (i.e. 0 – 255) – this translates to 0.07843 when expressed as a linear RGB value (0.0 – 1.0).

Also, as we’re going ultra-lores on this, set the Thicken to 1 pixel to give each of the islands a 1 pixel border. This helps to get a solid mask color around the edges of your meshes.

Save that, now open the material you created and drag this texture into the material graph.

Create a shader that looks like the snap below (or open from the example project), the Mask Texture is the newly created mask from the previous step. Set the Wall Meshes material to this and save.


Step 3.

Now for the Cube mesh, create a Cube in Blender (Add->Cube) and move it up so the bottom of the cube is at height 0. This will have a default UV map which we can now edit. Select Each Face, then the UV face and resize them to have good sized gaps in-between each island. Export this and Import into UE.

Now Right-Click on the cube in UE and select “rdMesh Tools->Tools->Create UV Maps”

This time, as the UV Islands are quite close together we must use a higher resolution (64×64) and a 1 pixel thickening – make sure “Unique Islands” is ticked, but not “Per Material”.

Now create a new Material for it, assign it to the mesh and drag the new mask texture into it and create this shader, and save.


Step 4. Create the Tree Mesh, Mask and Material

Now for the last mesh – a tree with trunk, branches and leaves. This one uses a separate UV channel for each of the material types, in this example, it’s just two, the trunk/branches and leaves. This way also allows you to use custom UVs into a shared atlas of textures too which is a great way to store multiple assets textures (best when these assets are all closely grouped for MIP benefits).

The tutorial covers a tree created with the Sapling Generator in Blender, similar workflows exist for other modelling software.

Both the Trunk and Leaves meshes use the default “uvMap” for the mask – the mask is made by selecting all vertices and UV points in UV Edit mode, pressing “u” and selecting the “reset” option – then in the UV panel the default points can be resized into an area within the UV area to define the mask. each mesh occupies a separate area. Then create a new UV map in each mesh for its spatial UV set, create the UVs.

Once this is done, join both meshes together like in Step 1. Make sure there are 3 UV Maps, that uvMap has merged both meshes mask UVs and the other two are correct.

Now Import the mesh into UE (make sure Nanite is unticked or the UV channels won’t import correctly) and create a new material for it, tick the “double sided” option.

This shows a simple way to specify simple colors for the mask areas.


Step 5. Done, Experiment.

That’s it – there’s plenty more you can do, e.g. Normals etc and various other ways you can use the masks.