The shader thread?

[Mr Whippy]

I'm messing with shaders a bit right now.

Right now I am looking specifically at adding control for specular to the existing reflect shader... on Deyan's Murcielago the unified wheel/tyre texturing poses some problems.

I have to control reflection strength with an alpha map, so use the old trick of adding a syntax in the reflection shader so if we define reflection = -1, the baseCol.a is used to control reflection strength.

ambientColor=FresnelMix(ambientColor,envColor.rgb,(Kr>0?Kr:-Kr*baseCol.a),fresnel);


The problem now is that I have no spare texture slot for specular/shininess control.

So I have added this little fragment of code (from the bump speca shader)

Ks=baseCol*Ks;

This now allows the colouring of the outside of the tyre in the diffuse (rgb) channels, to do some work.



However, because the tyre is still dark in the diffuse texture (near black as it is a tyre), it doesn't have much effect on it's own.

I need to somehow increase the brightness of the texture so Racer pumps up the specular we see.

So I added a *5 multiplier to the Ks as you can see in the image attached.

Now what I don't understand is why the two images look different.

First case:
Ks in car.shd is 8, while Ks = baseCol * Ks * 5 essentially gives us Ks = baseCol * 40

Second case:
Ks in car.shd is 40, while Ks = baseCol * Ks essentially gives us Ks = baseCol * 40


Now, I don't get why they look different. Ks is meant to control the size of the specular too, but in either case they both calculate out to a peak shininess of 40, because the baseCol lookup isn't changing.
The specular settings remain the same.

It appears that multiplying by 5 in the shader means the specular level is pumped up 5x, while the shininess (size of the specular) is retained at 8 in the car.shd...

That is the effect I wanted to achieve, pumping up the actual output of the specular blob, while keeping it soft at a value of 8, but it seems un-intuitive/wrong because the value that really needs bumping up is the specular value, not shininess? (though the specular being multiplied by 5x (2.5 2.5 2.5) looked awful)

Am I missing something obvious here?




Ideally I'd like to control reflection amount, specular AND shininess using one shader here...

Probably using rgb.a for diffuse.reflection strength, and then another 3 channel texture for spec/shine and leaving another slot free for something else (perhaps transparrency?)

Specular/shininess maps add so much to these kinds of materials, so it is a shame that they haven't really been considered in the default shaders very much. I think they should almost be defaults in all materials that err away from being generally reflective (ie, using the envmap)


Bodging in the code I have for the Lambo works quite nicely in this case, but it's obviously not ideal to use the diffuse channel to control specular in this way.



Hmmmm

Dave

 

[Some1]

Theoretically instancing would be nice. But with what editor? Here we have a workflow where artists create the tracks fully in Max. To create trees outside of Max would be costly and a bit awkward since the trees aren't there in Max and you need to place in... TrackEd?
It seems blasting a few hundred polygons to the card is faster these days than trigger a lot of the same geometry. Even in GT5, most trees are only crosses, with a nice shader that is.

You can create instanced objects in Max. Actually, for the most part when creating a track, for any object in the track that has more than one location (copy), I use instances. So, if I place 500 lampposts (using a spacing tool for example, so the posts are evenly placed at a fixed distance of each other along a spline), they are all instances of a single object (which itself is also referenced from another Max file, XRef or XObject or something like that it was called). When I later decide that the lamppost needs to be adjusted in some way, I only need to edit one single object. Brilliant!

Only before I export the track to Racer, I convert the instances to real objects (actually, it is done automatically).

So, in theory, one could write such a tool for Max, which would export only a single object and create instance specifications in the geometry ini for all of that objects instances.

How about, if we could simply override the position and rotation of an object in the geometry.ini?

 

[Mr Whippy]

That is basically it Ruud.

Your old cones were essentially instanced, where you had one cone in that Racer resource folder, and simply told it in what locations to exist.

The way to distribute them is another matter of course, not everyone has 3DS Max, but those who do could benefit from instancing perhaps? It'd certainly reduce the DOF count in the track folder which can end up getting fairly HUGE, especially with 90% of the same data... ie, cones, grass planes, trees and so on.


I'm not fussed either way as Some1 said, Max does instancing for you anyway, and now we have effective Max > Racer tools directly, you don't need to worry... BUT, if there is a memory/speed saving there (high quality renderers like Maxwell Render use instancing for improved performance, it may be the same for realtime?!), then it'd be cool to use it!


Dave

 

[Mr Whippy]

The sun scaling was done because I find normalized values a nice starting point, esp when you consider that if you talk about lumen, you get very high dynamic ratios which seemingly the floats won't handle nicely. So in the future I need to define a HDR range for all lighting to work in. The 2000 was just an empirical value for the Cg shader to look ok, not any real-life value. For those types, I'd rather use 1.0 and for the future have a nice baseline to work from than a funny scale like '2000' which was based on nothing but just looking at a screen.

I'll think about fully parameterisable TOD curves. Sounds ok.

For fresnel, note that CSM ofcourse adds more shadows than it did without shadowmapping! So the ambient side indeed becomes darker since the shadow is multiplied in combination with lightAmbient, whereas we only have lightAmbient in pure HDR (without CSM). I tend to raise ambient to almost 1 1 1. It's just that without CSM we didn't have the added shadowmap darkening things.
It was changed during a test to get a car here more looking like a real Ferrari car photograph. I would have left the defaults alone if they weren't programmable now.
If you don't think that's the cause, try in the car.shd to add fresnel to see if the originals were better:

shader_x
{
fresnel
{
bias=0.15
scale=0.85
power=2.0
}
layer0
{
...
}
}

Sorry that the changelog doesn't capture all, I'll try to do better.

Other than that, I'm trying to add lots of documentation on the v0.9 state of things, so that all of these questions become pointers to racer.nl. Also one of the reasons that I don't want to change more dramatic things like ini paths.

EDIT: oh, the sky. Just updated in the shading_tracks.htm:

"Since the sky Cg shaders take care of the sky color when considered cloudless, texturing the sky should be done with just clouds and alpha to determine where the clouds are visible. Notice the console command 'clouds <n>' with which you can set the amount of cloud dynamically (a scale on the texture's alpha value). So your sky texture should NOT look blue! Just a black & white image of clouds."

Ah, fantastic! A few of us again have been feeling things are too bluey in reflections, so it probably is this very issue here

One thing though, do I have to use a sky dome? Or can it be a box (5 sided, missing bottom, with inward facing normals)? (I'm getting some weird speculars with a box too despite it being welded with just 8 verts and 5 quads (10 tris))


Also, it must be the fresnel change as I am referring to 089/090 with regards to the darkening on the 'dark' side of cars. The reflections on the dark side of the car just looked too dim at steep viewing angles, making that side of the car look darker...



I have some stuff at home to try capture calibrated 'real' sky information to then simulate in Racer... ie, a chessboard type surface, white, black and chrome balls, and then capture them at varying exposures so we can see how the sky/sun look (in the chrome ball), as the white/black balls black out and white out against the sky.
That should help give an idea of what the right balance of all the variables we have to play with is!


All cool stuff!

Dave

 

[camsinny]

Theoretically instancing would be nice. But with what editor? Here we have a workflow where artists create the tracks fully in Max. To create trees outside of Max would be costly and a bit awkward since the trees aren't there in Max and you need to place in... TrackEd?
It seems blasting a few hundred polygons to the card is faster these days than trigger a lot of the same geometry. Even in GT5, most trees are only crosses, with a nice shader that is.

If you want I can write a quick export script for max to take care of this. You could use dummies to specify the position of the object but as Some1 brought up, rotation and then scale come into things, so perhaps selecting your actual instanced objects in max might be better.

It's by no means hard, Dave was asking me to do that geometry.ini script for ages and I ended up finishing it in a few hours...and that was after not touching maxscript for some time.
Speaking of X trees, is proper billboarding possible? It's just another LOD really, you're saving 2 tris per tree in the distance in the end but I guess if we want grass tufts trackside it might be beneficial?

All just thoughts.

###EDIT###
Also, this was brought on by a shader error that popped up in my qlog...

Tue Jun 29 13:01:43 (WARN): Material 'material_25' of geode 'data/cars/###/body.dof' failed to load image 'top.jpg'
Tue Jun 29 13:01:43 (WARN): No shader found for material 'material_25' in 'data/cars/###/body.dof' (default 'top.jpg')
Tue Jun 29 13:01:43 (WARN): d3LoadTextureMap(); can't load 'side.jpg'; creating red/blue stub image
Tue Jun 29 13:01:43 (WARN): Material 'material_26' of geode 'data/cars/###/body.dof' failed to load image 'side.jpg'
Tue Jun 29 13:01:43 (WARN): No shader found for material 'material_26' in 'data/cars/###/body.dof' (default 'side.jpg')
Tue Jun 29 13:01:43 (WARN): d3LoadTextureMap(); can't load 'front.jpg'; creating red/blue stub image
Tue Jun 29 13:01:43 (WARN): Material 'material_27' of geode 'data/cars/###/body.dof' failed to load image 'front.jpg'
Tue Jun 29 13:01:43 (WARN): No shader found for material 'material_27' in 'data/cars/###/body.dof' (default 'front.jpg')
Tue Jun 29 13:01:43 (WARN): d3LoadTextureMap(); can't load 'back.jpg'; creating red/blue stub image
Tue Jun 29 13:01:43 (WARN): Material 'material_28' of geode 'data/cars/###/body.dof' failed to load image 'back.jpg'

Is it possible to have a "clean material" function in modeler? I just exported a car from max > modeler and then exported as a .dof but I get these material warnings in QLOG. There's only 1 object in the .dof file with 1 material. "Body".
These materials shouldn't really be exported from modeler in the first place. (well, they shouldn't be coming out of max when there's no geometry with that material but I doubt Autodesk are going to listen to little old me)

Also, surely there must be a better way of doing ghost cars other than having a seperate folder in the /cars/ subdir of the racer folder. Can't we just have a carghost.shd or something? In the end there's essentially 2 almost EXACT copies of a car, just means more downloads in the end.

 

[boomer541]

1. I made the sky larger than my surfaces and sounds track, Textured with sky.tga (with alpfa) and ran with vf_sky
shader using sky_v and sky_f .cg's. Sky came out black and stayed black while cycle thru 24 hours.
The special.ini "gfx" settings:
gfx
{
clear_color=0
night=0
; Alternative with light sky
sky=1.000000 1.000000 1.000000 0.000000
; Sky exposure
exposure=1
; Day time (overrules sun if specified)
time=1500
; Sun location (XYZ); keep it far away to avoid too much movement
sun=-5000000.0 4000000.0 0.0
{
; Sky Cg shaders
intensity=0.1 ;1.0
intensity=2500
; Paint the sun?
enable=1
; Coloring
ambient=0.5 0.5 0.5 1
diffuse=1.0 0.8 0.6
specular=1 1 1
; No global light
; global=0.15 0.15 0.15 1
}
shadows
{
; Intensity of projected shadows (defaults to 0.3)
intensity_projected=0.5 ;1.0 ;0.5
}
fog
{
density=0.002
start=0
end=1
color=.3 .3 .37 1
; Fog mode; 0=exp, 1=exp^2, 2=linear (start/end only used with linear!)
fog_mode=0
; Sky Cg shaders
rayleigh=0.000008
mie=0.000008
; Extinction factor (<1 = less fog)
extinction_factor=0.5
}
}
The track.shd settings:
shader_sky~vf_sky
{
compression=0
fog=0
sky=1
cull=none
layer0
{
map=sky.tga
texenv=replace
depthwrite=0
}
layer1
{
map=sky_night.tga
texenv=replace
depthwrite=0
}
}
2. Changed vf_sky to sky_daynight_v/f.cg's and ran, cyled thru 24 hours with same result!
3. Changed track shader maps to .jpg's and ran, cycled thru 24 hours and sky cycled normally but was too bright.
4. Changed vf_sky to sky_v/f.cg's and ran as before, Sky stayed daylight and was distorted (not far enuf).
5. Made sky dome, setup same as 2 and 3, textured with .tga(with alpha). Result same as 3.
Conclusions:
1. The cg's don't like .tga's with alpha,
2. Can't adjust brightness
3. Displayed sky not same look as original image.
4. Ruud and/or company screwed up skies!!!!!
5. rayleigh and mie set to 0.000008 does not darken sky.

So why don't thr sky look right?

 

[Mr Whippy]

Rayleigh and mie look very small, which will result in not much atmospheric scattering, resulting in a black sky!

Use "fog mie 0.005" and "fog rayleigh 0.005" in the console in Racer to set these variables more intuitively...!



As per what Cam was saying, I totally agree about the ghost cars, the current system is not ideal. No way to force a material to all objects in car.ini called 'ghost' that is a hardcoded one in Racer?
In theory, this will half the size of the racer/cars folder if all cars have a ghost version!

Hmm

Dave

 

[Ruud]

One thing though, do I have to use a sky dome? Or can it be a box (5 sided, missing bottom, with inward facing normals)? (I'm getting some weird speculars with a box too despite it being welded with just 8 verts and 5 quads (10 tris))

Also, it must be the fresnel change as I am referring to 089/090 with regards to the darkening on the 'dark' side of cars. The reflections on the dark side of the car just looked too dim at steep viewing angles, making that side of the car look darker...

I have some stuff at home to try capture calibrated 'real' sky information to then simulate in Racer... ie, a chessboard type surface, white, black and chrome balls, and then capture them at varying exposures so we can see how the sky/sun look (in the chrome ball), as the white/black balls black out and white out against the sky.

I don't think a box will work, although it might, tesselated ok. I've update http://racer.nl/tutorial/shading_tracks.htm and shown what we use in tesselation. Some colors (rayleigh/mie I think) are calculated per vertex and interpolated over the fragments.

The chrome & white balls are very interesting, I've read about the use in movies and it all makes such simplistic sense. Would be good to have a lighting track with chrome and white balls to play around with lighting.

 

[Ruud]

If you want I can write a quick export script for max to take care of this. You could use dummies to specify the position of the object but as Some1 brought up, rotation and then scale come into things, so perhaps selecting your actual instanced objects in max might be better.

Speaking of X trees, is proper billboarding possible? It's just another LOD really, you're saving 2 tris per tree in the distance in the end but I guess if we want grass tufts trackside it might be beneficial?

...
Tue Jun 29 13:01:43 (WARN): Material 'material_26' of geode 'data/cars/###/body.dof' failed to load image 'side.jpg'
Is it possible to have a "clean material" function in modeler? I just exported a car from max > modeler and then exported as a .dof but I get these material warnings in QLOG. There's only 1 object in the .dof file with 1 material. "Body".

Also, surely there must be a better way of doing ghost cars other than having a seperate folder in the /cars/ subdir of the racer folder. Can't we just have a carghost.shd or something? In the end there's essentially 2 almost EXACT copies of a car, just means more downloads in the end.

I don't think tree instancing would work; given 500 trees, that would mean setting up a VBO 'transfer' (already on the gfxcard) 500 times, with only 4 triangles in them. These days, pushing 2000 polygons to render is much faster than setting up 500 small renders. For non-realtime, it's true that caching these poly's is faster, but for gfxcards, transferring commands/data across to the card is expensive. That's why use_vbo=1 really helps, esp on high-end cards (I've used QuadroPlex and it didn't really seem to matter much without VBO's).

Billboarding for X-trees; not sure if that makes it better. With some added shading (directional sun influence, some nice papers on that) it might be ok. I don't think GTA4 or GT5 does tree billboarding, or does it?

On the ghost car, hm, why didn't we think of that lol. Although it might be a bit tricky with skinning. It's on the todo-list!

EDIT: Modeler: just Optimize (Shift-O) in Modeler! The ASE format is very flexible, so it's read in a really worst-case way to be able to read it all. An optimize is really really needed. And the latest Modeler's also reduce the material table, which is exactly what you want.
I still struggle with this in Swiss Stroll for example; there, the files were stored in .ar I think. Makes it hard to get rid of these superfluous materials... (I can dig out the DOF's ofcourse, but I'm the only one)

 

[Ruud]

Rayleigh and mie look very small, which will result in not much atmospheric scattering, resulting in a black sky!

Use "fog mie 0.005" and "fog rayleigh 0.005" in the console in Racer to set these variables more intuitively...!

Better use "fog mie 0.001" and "fog rayleigh 0.0025" and "sun intensity 1.0". 0.1 is very dark, the ~0.00008 is really zero, making skies dark. Use sky_v.cg sky_f.cg for now; those need clouds on alpha, not a blue sky texture! Re-read the shading_tracks.htm; it now contains recommended settings that work for v0.8.10.

 

[Mr Whippy]

Re-read the shading_tracks.htm; it now contains recommended settings that work for v0.8.10.

Can you direct link to this page, I have no idea where to find it using the on-site navigation, and using www.racer.nl/shading_tracks.htm doesn't work either?!

Thanks

Dave

 

[Mr Whippy]

Better use "fog mie 0.001" and "fog rayleigh 0.0025" and "sun intensity 1.0". 0.1 is very dark, the ~0.00008 is really zero, making skies dark. Use sky_v.cg sky_f.cg for now; those need clouds on alpha, not a blue sky texture! Re-read the shading_tracks.htm; it now contains recommended settings that work for v0.8.10.

Yikes, those values seem pretty intense. At 0730 TOD, or 1200 (midday) the upward-pointing surfaces of my mid-grey car are just solid white and blown out... with clouds 1 it is very white, with clouds 0 it is a bit more bluey, but I still seem to see the car disappearing into a hazy fog at about 200m away. That is a pretty hazy day (almost foggy!)

I'm not sure what real values are, but I think tinkering with them for the specific weather/time etc you are aiming to simulate is key. What works well for one set up is really bad for others... a key reason they need to be able to adjust alongside TOD for TOD to be nice and useful, and why a nice editor (using curves or LUT) for the whole TOD/sky stuff will be really cool

TOD 0530 with half-sensible mie/rayleigh settings looks nothing like this. The setting are pretty specific, from ambient/diffuse in tracks special.ini sun settings, to the mie/rayleigh and so on.

Mmmmmm sunrise


Dave

 

[frank11]

Can you direct link to this page,

It's http://www.racer.nl/tutorial/shading_tracks.htm

 

[Ruud]

It's in the tutorial dir, like most of my current documentation is going to. http://www.racer.nl/tutorial/shading_tracks.htm

 

[Ruud]

Yikes, those values seem pretty intense. At 0730 TOD, or 1200 (midday) the upward-pointing surfaces of my mid-grey car are just solid white and blown out... with clouds 1 it is very white, with clouds 0 it is a bit more bluey, but I still seem to see the car disappearing into a hazy fog at about 200m away. That is a pretty hazy day (almost foggy!)

I'm not sure what real values are, but I think tinkering with them for the specific weather/time etc you are aiming to simulate is key. What works well for one set up is really bad for others... a key reason they need to be able to adjust alongside TOD for TOD to be nice and useful, and why a nice editor (using curves or LUT) for the whole TOD/sky stuff will be really cool

TOD 0530 with half-sensible mie/rayleigh settings looks nothing like this. The setting are pretty specific, from ambient/diffuse in tracks special.ini sun settings, to the mie/rayleigh and so on.

Mmmmmm sunrise


Dave

/
Just google for rayleigh/mie values. What happens if you try those on Carlswood? Here things look pretty normal. Any other settings in special.ini?
Ah btw, noon can indeed go quite white and require a lower exposure. I turned on auto-exposure during experiments which helps (dips down to ~0.7-0.8 at noon).

 

[boomer541]

Images oforiginal sky section. ver 0.0881, ver 0.0810 with the intensity, rayleigh and mie settings.

I'll check out the racer shading tracks tutorial page and see if there is anything there I can use.

See the images in next post!

 

[boomer541]

Images of original sky section, ver 0.0881 and ver 0.0810.
disregard the image below as I have trouble posting images!

 

[Mr Whippy]

/
Just google for rayleigh/mie values. What happens if you try those on Carlswood? Here things look pretty normal. Any other settings in special.ini?
Ah btw, noon can indeed go quite white and require a lower exposure. I turned on auto-exposure during experiments which helps (dips down to ~0.7-0.8 at noon).

I'm not sure on Carlswood, again I think that mie and rayleigh must move around a fair bit depending on lots of atmospheric parameters in real life, and especially cloud cover, and that is also determined by our textures and other variables...

With the settings you suggested on my 7km long moor track, it looks like almost solid luminous fog by about 1000m away

It looks ok on Carlswood because it's so relatively small I guess.



Are there any hard and fast rules for the cloud mapping etc? Is extinctionfactor a tuneable value that we should tweak?

Other fog variables?




Hmmm, copying Carlswood numbers makes it very bright in the sky. Rayleigh 0.002 feels ok. I think maybe my clouds texture is different and so influencing things more than it does on Carlswood.
Also noticed the old fog density makes the soft shadow under the car change intensity too (the old ambient style under-car map)

Should the shadows intensity projected be set to anything specific?


Dave

 

[boomer541]

Looked at the shading track tutorial page and noted gfx.exposure setting info. Tried exposure=0.7, intensity=1.0, rayleigh=0.00025 and mie=0.00008 and the sky came out reasonably dark but the track became too dark.

I use a sky dome textured with a panoramic image of the surounding area. Using the quide in the tutorial (sky_v/f.cg's) there is no night darkening!!

We still need more, better info about how to make skies!!!!!!!!!!!!!!! They work on carlswood but they are not made like I have made mine.

 

[Mr Whippy]

Carlswood uses a different sky for night time.

I'm also not too sure on the exposure setting too much for correcting things later. Why lower exposure when you can lower sun intensity and the diffuse/ambient settings

Pragmatic question there, I know the answer, but we need some hard and fast rules otherwise we will just end up with bad content...


I'll try pull my finger out and take pictures of my chrome/matte balls and some skies over the next few weeks, then at least we can start to try simulate them in Racer and get nicely balanced setups as a guide for content creators. Right now I'm just fiddling with numbers with no idea if they are actually fairly accurate or not to real life (important with so many real life variables in the mix like rayleigh and mie now etc)

Dave

 

[boomer541]

Aha! checked carlswood sky.dof with modeler>information and found that the material that used to be sky.tga is now "generated".

Just what the hell is a generated material (texture) and how is it created on a sky dome? Is it done with 3DMax?

Can we still use panaramic sky images for skies or not?

I have a few more things to try but would appreciate a bit more infor from Ruud!