ParseShader: load extra maps the DarkPlaces way if r_dpMaterial enabled

[illwieckz]

If r_dpMaterial is set to on, load extra textures the DarkPlaces way by looking for extra files with special suffixes like _norm or _gloss.

[illwieckz]

By not working I mean the engine reports to find the extra texture files, but unfortunately it only renders diffuse maps.

[illwieckz]

I think there is a special command to list opened files, does someone remember it?

[illwieckz]

For information this is how it looks on console:

Debug: looking for extra maps for color map: 'textures/phillipk2x/computer/pk02_switches01c' 
Debug: found extra glow map 'textures/phillipk2x/computer/pk02_switches01c_glow' 
Debug: found extra normal map 'textures/phillipk2x/computer/pk02_switches01c_norm' 
Debug: found extra specular map 'textures/phillipk2x/computer/pk02_switches01c_gloss' 

That's why it seems to work: the files are found. But what's painted on screen obviously proves that only diffusemap is rendered.

[illwieckz]

At this point the engine code claims to turn this:

textures/phillipk2x/computer-pk02_switches01c
{
    qer_editorimage textures/phillipk2x/computer/pk02_switches01c

    dpoffsetmapping - .68120712951256245813 match8 89.89403100000000000000
    dpglossintensitymod 3
    dpglossexponentmod 4

    q3map_bouncescale 1.25
    {
        map textures/phillipk2x/computer/pk02_switches01c
    }
    {
        map $lightmap
        rgbgen identity
        tcgen lightmap
        blendfunc filter
    }
}

into this (dp* keywords are ignored with an warning):

textures/phillipk2x/computer-pk02_switches01c
{
    qer_editorimage textures/phillipk2x/computer/pk02_switches01c

    q3map_bouncescale 1.25
    {
        map textures/phillipk2x/computer/pk02_switches01c
    }
    {
        map $lightmap
        rgbgen identity
        tcgen lightmap
        blendfunc filter
    }
    glowMap textures/phillipk2x/computer/pk02_switches01c_glow
    normalMap textures/phillipk2x/computer/pk02_switches01c_norm
    specularMap textures/phillipk2x/computer/pk02_switches01c_gloss
}

Note that the code runs after everything else is parsed, hence right before the ending }.
It means the code runs after the second stage (map $lightmap), perhaps that's the problem ?
I verified that glowMap, normalMap and specularMap are generated as stage 2, 3 and 4 (original ones being stages 0 and 1).

I also verified those files are loaded:

dds/textures/phillipk2x/computer/pk02_switches01c.dds
dds/textures/phillipk2x/computer/pk02_switches01c_gloss.dds
dds/textures/phillipk2x/computer/pk02_switches01c_glow.dds
dds/textures/phillipk2x/computer/pk02_switches01c_norm.dds

[illwieckz]

To get a testbed, people may copy (or symlink) xonotic-20170401-maps.pk3 into a brand new directory, then run this code within that directory:

p="$(ls 'xonotic-'*'-maps.pk3' | tail -n 1)"
b='xonotic-maps'

listMaps () {
    unzip -l "${p}" \
    | tail -n +4 \
    | head -n -2 \
    | egrep '\.bsp$' \
    | sed -e 's|^.* maps/||;s|\.bsp$||'
}

filterSystem () {
    egrep -v '^_'
}

filterInvalid () {
    egrep -v '_'
}

ln -sfv "${p}" "${b}_0.dpk"

for m in $(listMaps | filterSystem | filterInvalid)
do
    d="map-${m}_0.dpkdir"
    mkdir -pv "${d}"
    echo "${b}" > "${d}/DEPS"
done

then run dæmon this way:

daemon -pakpath /path/to/xonotic/pkg \
    -set logs.logLevel.default debug \
    -set language en \
    -set developer 1 \
    +set g_neverEnd 1 \
    +devmap solarium

[illwieckz]

When I print numActiveStages the in CollapseStages() I get the expected amount of stages, for example 6 for this shader:

textures/exomorphx/light-metal_light03
{
	qer_editorimage textures/exomorphx/light/metal_light03
	dpoffsetmapping - 5.05836666682224888188 match8 160.48228200000000000000
	dpglossintensitymod  2
	dpglossexponentmod  4
	dpreflectcube env/exomorph/exomorph
	surfaceparm metalsteps
	q3map_bouncescale 1.50

	{
		map textures/exomorphx/light/metal_light03
	}

	{
		map $lightmap
		rgbGen identity
		tcGen lightmap
		blendfunc filter
	}
}

produces:

Warn: unknown general shader parameter 'dpoffsetmapping' in 'textures/exomorphx/light-metal_light03' 
Warn: unknown general shader parameter 'dpglossintensitymod' in 'textures/exomorphx/light-metal_light03' 
Warn: unknown general shader parameter 'dpglossexponentmod' in 'textures/exomorphx/light-metal_light03' 
Warn: unknown general shader parameter 'dpreflectcube' in 'textures/exomorphx/light-metal_light03' 
Debug: looking for extra maps for color map: 'textures/exomorphx/light/metal_light03' 
Debug: found extra glow map 'textures/exomorphx/light/metal_light03_glow' 
Debug: found extra normal map 'textures/exomorphx/light/metal_light03_norm' 
Debug: found extra specular map 'textures/exomorphx/light/metal_light03_gloss' 
Debug: active stage count: 5 

Five stages because of the two original stages plus the three computed ones:

glowMap textures/exomorphx/light/metal_light03_glow
normalMap textures/exomorphx/light/metal_light03_norm
specularMap textures/exomorphx/light/metal_light03_gloss

[illwieckz]

If I hack stuff by disabling the color map stage and creating a brand new diffuse map stage instead with the same texture path, it works, but it's a hack since we lose all the other keywords that may be there (like tcMod and so on), but it helped me to validate we have to turn the colormap into a diffusemap to get things working.

Note that the lighting is wrong because Xonotic uses sRGB lightmaps and Dæmon does not support them yet.

without hack

with hack

[illwieckz]

Note that since Xonotic ships a heightmap with almost all its normalmap, I enabled temporarily parallax on all shaders just to have a look. It works (and I haven't experienced preformance issue) but I discovered that xonotic heightmaps are upside down, or our engine is.

Perhaps it's a trick to not have to detect the presence of heightmap hidden in alpha channel of normalmap, as a missing alphachannel would mean a flat surface at the good level. Currently our engine displays a flat surface at the wrong level if we enable parallax on a normalmap without heightmap in alpha channel.

[slipher]

If you actually write the second shader in this comment in a file, does it do what you want?

[illwieckz]

I have not tried but I'm now almost sure it won't work because this is a colorMap stage:

	{
		map textures/exomorphx/light/metal_light03
	}

and a diffuseMap stage looks like this:

	diffuseMap textures/exomorphx/light/metal_light03

or

	{
		stage diffuseMap
		map textures/exomorphx/light/metal_light03
	}

and the engine code is explicit: normalMap, glowMap, specularMap and others only work with diffuseMap stage, not colormap stage.

Daemon/src/engine/renderer/tr_shader.cpp

Lines 3768 to 3783 in be89425

	// try to merge diffuse/normal/specular/glow
	if ( stages4.diffuseStage && stages4.normalStage && stages4.specularStage && stages4.glowStage )
	{
	tmpCollapseType = collapseType_t::COLLAPSE_lighting_DBSG;
	shaderStage_t tmpStage = *stages4.diffuseStage;
	tmpStage.type = stageType_t::ST_COLLAPSE_lighting_DBSG;
	tmpStage.bundle[ TB_NORMALMAP ] = stages4.normalStage->bundle[ 0 ];
	tmpStage.bundle[ TB_SPECULARMAP ] = stages4.specularStage->bundle[ 0 ];
	tmpStage.specularExponentMin = stages4.specularStage->specularExponentMin;
	tmpStage.specularExponentMax = stages4.specularStage->specularExponentMax;
	tmpStage.bundle[ TB_GLOWMAP ] = stages4.glowStage->bundle[ 0 ];
	stages[stagesWritten++] = tmpStage;
	j += 4;
	}

So what we need is to transform the colorMap into a diffuseMap. I don't know why but setting the diffuseMap bits in a colorMap stage does not do the trick. The hacky way that works is to disable the colorMap and load a brand new diffuseMap that is activated instead. That's how I produced those pictures above, the problem is that sometime those colorMap stages ship more information that are then lost.

For example the colorMap of the solarium water is this one:

	{
		map textures/map_solarium/water4/water4.tga
		tcmod scale 0.3 0.4
		tcMod scroll 0.05 0.05
		blendfunc add
		alphaGen vertex
	}

If I create a diffuseMap using the textures/map_solarium/water4/water4.tga path and disable the original colorMap, I just lost everything else that came after the name: wobbling, transparency…

Edit: I'm like 99.99% sure of what I just wrote in this comment.

[illwieckz]

I implemented the detection and auto enablement of heightmap in normalmap alpha channel.

I added a hack that reverses r_parallaxDepthScale, it looks to be an engine bug.

[illwieckz]

The code looks OK to me now, I added a r_dpMaterial cvar that is required to be set to on to look for the extra files.

This dpMaterial cvar may be reused later to enable DarkPlaces shader keyword parsing.

I removed from this PR all the related stuff that was useful for testing purpose but must be done in dedicated PR, like attempt to fix parallax mapping or code to detect heightmap in normalmap.

[slipher]

It looks like there are still some commits from one of the other PRs, can you remove those?

[illwieckz]

@slipher, yeah, those commits were required for that patch to be applied, but once they have been merged on master they just disappeared from there.

[illwieckz]

all remarks are now fixed

[illwieckz]

@slipher, do you think this requires the annoying #ifdef BUILD_ENGINE magic too?

[illwieckz]

seems not