Super frustrating CPU 99 Occupancy HELP PLEASE

[Hugo Becerra]

I have been experiencing an issue with AC, it started around the time I downloaded and installed CSP .1.75, but since then it happens with every csp build.
Problem is the CPU Occupancy 99% and stuttering (it kind of goes into slow motion, I dont know if stuttering is the right word).
I tried updating SOL 2,1, updating GPU controllers, verifying game integrity, and nothing works
My PC build has been the same siince the start of the year and it has run everything with no problem (A lot of AI, heavy mods, rainfx, etc) Content manager sometimes crashes and makes a report when starting a session, sometimes it does not.

Ive tried every fix that I can find (wheel ffb, face reflections down to 1, low graphics, etc )and havent seen a solution yet, any idea ?


MY PC BUILD

NVIDIA Geforce RTX 2060 SUPER (controller 466.27)
INTEL CORE I7-9700F
31.93 GB RAM


FIXED by using CPU PROCESS LASSO AND FORCING A MULTI-THREADED USE

 

[Lizard Of Bodom]

There has to be some bug that is causing this for some of us. Im sure some ARE limited that get this 99% message, but for some there is a bug of sorts.

For reference when im testing the better CPU, I only swap the MB and CPU. OS and drives stay the same.

Im not sure if I missed that fact or you already eliminated this, but are you aware that 99% CPU occupancy issue also relates to server you are driving on? I can often see it myself on some LAC servers while my 3700X Ryzen stays around 20% of utilisation and its NOT related to my machine but server itself.
Maybe its just simply that?

 

[RasmusP]

Im not sure if I missed that fact or you already eliminated this, but are you aware that 99% CPU occupancy issue also relates to server you are driving on? I can often see it myself on some LAC servers while my 3700X Ryzen stays around 20% of utilisation and its NOT related to my machine but server itself.
Maybe its just simply that?

You can't see single thread bottlenecks with any tool available right now so if assetto only uses 2 big cpu threads and your 3700x has 16 threads for Windows available, the maximum overall load AC can cause, in theory, would be:
(100% / 16) = 6,25%
x2 = 12,5% overall load.

AC has some more little threads and windows manages to get more out of it by shuffling ac around across all cores. Resulting in the 20% I'd say.

So you can be definitely single thread limited at only 20% cpu load without seeing any cpu core at full load.
At 1s refresh rate from Taskmanager, at 4 GHz, 4.000.000.000 cpu cycles are averaged.
Maybe half a second one core was completely loaded. The other half a second not. Would show 50% load but your fps are still limited by that.
(this is a bit inaccurate, but easier to imagine. In reality a cpu core can only "do" or "pause". % load is always the amount of "do" within a certain amount of time)

So I would suggest you check your main_T and physics_late value in the performance stats app when you're seeing the warning.

AC servers don't actually run the content. They just coordinate the coordinates of the cars that are moved around and synchronize the info for everyone.
While not impossible for some servers to be overloaded by massive LAC servers, I highly doubt it.
When I hosted a server, the cpu load by 15 ac servers in parallel was below 1% with about 10 people on each.

 

[Lizard Of Bodom]

...

So I would suggest you check your main_T and physics_late value in the performance stats app when you're seeing the warning.

AC servers don't actually run the content. They just coordinate the coordinates of the cars that are moved around and synchronize the info for everyone.
While not impossible for some servers to be overloaded by massive LAC servers, I highly doubt it.
When I hosted a server, the cpu load by 15 ac servers in parallel was below 1% with about 10 people on each.

I know that bc of game engine my Ryzen mostly yawns thorough the gameplay and fair points above ref multithreading, but what you are actually saying is that this error only ever occurs on local machine and never means server? Or that it can mean either ?

Seen the occupancy error on not-very-crowded LAC server but also while being far away and with no other cars around causing potential issues on my PC, also when checking the per-core utilisation history in Afterburner it didnt show any noticeable spikes so that led me to believe it was simply on server side...

 

[edwinjacobs]

So but something in Lasso solved it for the original poster of this thread... but we can agree it's not that it magically split up the main (heaviest) AC thread in smaller threads.... I still wonder what could have been the quick fix for him in Lasso.

Anyway, in my case it's not server related for sure, I can enduce the error if I do a local race on Monaco 2020 track with a full grid (ACFL 2022).

I will try and take screenshots of the perf both before and while the 99% cpu issue is occuring and post them here later today... thanks already for all for wanting to think and help along! cheers!

 

[pattikins]

thx, but this generic documentation does not say how to "force a multithreaded use for Assetto Corsa."...

Any help appreciated!

Maybe something has been lost in translation. For a short time - a few months - I used Process Lasso when ever I started up AC - up until I solved the stuttering issues... it helped but it was no cure.

The way I used Process Lasso was to dedicate ('force'?) my fastest cores* (2c/4t) to AC, & redirect any other (non-Windows) programs away from those cores:

*Ryzen 3600X - when I say fastest, the cores struggled to hit 4.3ghz.

 

[RasmusP]

I need to take a read on the 99% warning again. Iirc it's about something like one part of the rendering chain that's done on the cpu takes xy amount/% of time of the overall frame time.
Which means it can still happen at 100 fps, if that one part gets delayed inside the cpu while everything else is completed way faster.

I'm not sure that this warning algorithm isn't even coded into the server application. The server is really just a little cmd command prompt window coordinating car coordinates. I can't imagine it creating such a warning but I have no information about it either.

Afterburner only shows exactly the same, averaged, statistic that Taskmanager and all the other monitoring tools show.
To really see what's going on, you'd need a program that takes screenshots of single cpu cycles.
Or better, takes a screenshot every time the load or the process on a core changes.

This way we could see what's going on.

Overall, I'd love the "pc community education", or whatever you'd call it. Basically "what's common knowledge for people who are decently educated about pc stuff":
To think in frame time, intervals and averages instead of "fps".
The transition could be accelerated by games and monitoring tools no longer showing fps but frame time in ms instead.
With afterburner and riva tuner, you can enable a frame time graph that basically is a line that moves. Every frame is one pixel so the higher the fps, the faster the line moves.
The Y axis is the frame time of that frame.
So with an fps limiter, you should see a flat line. But if even only one single frame causes a stutter, you'll see a spike in the line. If it's only 1 frame per second, the "fps" won't change. Average frame time over a full second won't be influenced by a single, little hiccup.
But on this nice little graph, you can instantly see the issue.

Helped me to find out that rocket league stuttered due to my cpu thinking it could clock down because rocket league was causing so little load. Lol...
Switching to high performance power plan solved it.

My fps were always rock solid. Even when setting afterburner to the shortest refresh time (so it would show "fp10ths", frames per 0.1s).


Basically you need to think of this all like this:
For each frame, the cpu gets instructions. Cores can only "do" or "idle".
So you have 3 application threads, 6 cores.
Only 50% of your cpu can "do". The rest sits "idle".
These 3 cores complete one frame in xy amount of cpu cycles and the time is measured.
At 100 fps that's 10ms on average for one frame.
The slowest core decides when the frame is completed to be sent to the gpu.

Then this frame is sent to the graphics card and the same cycle stuff happens. The difference is that gpu stuff can be split into basically infinite amounts of threads so the gpu can work to full extend with everything it got.

If it takes only 5ms, that would be 200 fps but it waits another 5ms to get the next frame from the CPU.
Result is 50% gpu load and a cpu bottleneck.

While the gpu is calculating its frame, the cpu calculates the next frame.
This time other cores are used. They could already feed the instructions from the cache while the other cores were "doing". So you save some time compared to letting it run on the same 3 cores all the time.

This makes it look like all cores would be used to some extend at the same time. They are not.
Depending on the boost clocks and how quickly the clockspeeds can change, Windows will keep the load on certain cores for more cycles.
The first few cores usually have a shorter latency inside the cpu itself so they are used more.
If you overclock the cpu individually, the cores with the highest clock will be used more.
If you do it like me on my Intel, with all cores at 4.8 GHz, you'll see the load almost perfectly spread.


Now for the 99% warning, afaik, this all looks like this:
2 cores take 5 ms to complete their task, the graphics card needs 3ms to complete its task.
But the third cpu core takes 20 ms to complete the frame.
Resulting in momentarily "50 fps".
(Not fps, since it's not over a span of a full second but consistent 20 ms frame time = 50 fps.)

Anyway, assetto corsa would now see that one single core took way longer than everything else for that frame and send you a warning.

Now most of the time, if you have enough cpu cores, this doesn't happen because the lagging behind core would get priority to complete the task in time.

But if you have only 2 cpu cores on a laptop cpu for example, Windows can't do anything about this issue since the 2 cores are simply overloaded.
They "do" all the time and don't get their stuff done in time and therefore ac gives the warning.

If you have more than enough cpu cores, it very rarely happens.
But it means that something in the cpu rendering chain has a hiccup.
Might be due to a USB device causing issues, a Windows update doing stupid stuff, an ac hud app causing delays or whatever.

All we know is that something is holding up one cpu core from being done in time.


And the only thing we can do to get a clue is to either post the logs in the official forums and hope that someone can find out why the warning got triggered.
Or we can enable the two render stats overlay apps and try to find something that seems odd.

If nothing looks suspicious, all you can do is to create a big sheet and test every possibility manually.
Taking all USB decides off, disabling all ac apps, shutting down all unnecessary processes in Taskmanager etc etc

 

[RasmusP]

So but something in Lasso solved it for the original poster of this thread... but we can agree it's not that it magically split up the main (heaviest) AC thread in smaller threads.... I still wonder what could have been the quick fix for him in Lasso.

Yeah that's a good question. It might also be a simple coincidence to be honest.
Maybe he also changed his keyboard or unplugged a USB stick.
Or process lasso put a buggy background task to a cpu core that AC didn't use much.

Maybe something has been lost in translation. For a short time - a few months - I used Process Lasso when ever I started up AC - up until I solved the stuttering issues... it helped but it was no cure.

The way I used Process Lasso was to dedicate ('force'?) my fastest cores* (2c/4t) to AC, & redirect any other (non-Windows) programs away from those cores:
View attachment 551809

*Ryzen 3600X - when I say fastest, the cores struggled to hit 4.3ghz.

Yeah that can sometimes help on CPUs that boost a few cores higher.
In theory it shouldn't matter if a core that is done with its AC frame is doing something else instead of sitting idle but sometimes it seems that keeping these higher boosting cores completely reserved for AC is improving things.

 

[edwinjacobs]

i managed to catch some screenshots of render stats ingame

I have max fps set to 92 in nvidia control panel , and in AC settings I limited framerate to 82.
(Q2 is set to 80 fps )
Before issue occurs (in same offline race , full grid w 19 other ACFL 2022 cars, on Monaco 2020 track):

so 79.7 fps is normal situation

During issue: i see

notice that when the issue occurs , it feels as if everything is in slowmo, but still the render stats say I hit 80 fps (or 79.x which is normal in my case)
The PHY increased alot though: went up from around usually <20% to over 150% when the error started (and at time of the screenshot was even at 300%
But i have not enough knowlegde about anything to make a guess as to what this may revail (if anything - could also be a consequence of instead of a rootcause indication).


Hope someone here can interpret this and give a golden tip to crack this 99percent issue enigma

My specs:
mb: GIGABYTE x570
cpu: Ryzen-9 5950x
gpu: RTX 3070
ps: Corsair 850w Gold
os: Windows 10 Pro version 20H2 (build 19042.1466)
NVIDIA driver version: 511.65
Quest2 via Oculus Link (usb 3.1 cable A to C plugged in a usb 3 type A port of the x570)
neither GPU or CPU overclocked (all stock settings, only activated XMP profile for the mem in bios)

 

[Stereo]

I believe when I looked into the 99% warning, it's simply comparing the system clock to the sim clock. Every physics frame the sim adds 3ms to the clock and does a timestep forward in all the physics (car is moving at 10m/s -> multiply by 0.003s to get 3cm added to its position), then checks the time so it can sleep until the next frame. So if the system clock advanced by 2.97ms while it was calculating, that's the 99% point and it gives you a warning. If it's taking more than 3ms to calculate 3ms of simulation, then your game slows down, which is one thing offline, but online it can't just count its losses, it has to be able to catch up to other users to resynchronize. So either it calculates physics updates as fast as possible or just skips ahead, which either way is gonna be a weird experience.

You can see the 3ms thing on any app that displays the sim time accurately like the comparison to your previous lap which only goes to +0.003 +0.006 etc.

Normally the easiest way to get in this situation is to have more AIs than your CPU can handle; car to car interactions like collisions and aero drafting effects means they can't just run independently, some of the AI code has to be coordinating positions back and forth between all of them.

 

[RasmusP]

I believe when I looked into the 99% warning, it's simply comparing the system clock to the sim clock. Every physics frame the sim adds 3ms to the clock and does a timestep forward in all the physics (car is moving at 10m/s -> multiply by 0.003s to get 3cm added to its position), then checks the time so it can sleep until the next frame. So if the system clock advanced by 2.97ms while it was calculating, that's the 99% point and it gives you a warning. If it's taking more than 3ms to calculate 3ms of simulation, then your game slows down, which is one thing offline, but online it can't just count its losses, it has to be able to catch up to other users to resynchronize. So either it calculates physics updates as fast as possible or just skips ahead, which either way is gonna be a weird experience.

You can see the 3ms thing on any app that displays the sim time accurately like the comparison to your previous lap which only goes to +0.003 +0.006 etc.

Normally the easiest way to get in this situation is to have more AIs than your CPU can handle; car to car interactions like collisions and aero drafting effects means they can't just run independently, some of the AI code has to be coordinating positions back and forth between all of them.

Thanks for the explanation! It was what I was remembering but couldn't recall correctly!

As for the Screenshots from Edwin:
I sadly have absolutely no clue but cpu found out that something goes nuts regarding physics calculation.

 

[edwinjacobs]

interresting... I was tinkering of late with disabling HPET in Bios and device manager and some CMD commands to improve my system performance
(following this video :

)
Maybe that kinda got me into this mess


currently i have Resolution set to 1.000 ms:

So what would be the recommended settings in regards to Timer resolution and HPET on/off?

 

[Stereo]

Don't know what the default is, but I could see that generating a lot of 99% errors if your phy% is regularly in the 50-100% range. Or it might not be relevant at all, who knows with this stuff.

 

[RasmusP]

interresting... I was tinkering of late with disabling HPET in Bios and device manager and some CMD commands to improve my system performance
(following this video :

)
Maybe that kinda got me into this mess


currently i have Resolution set to 1.000 ms:
View attachment 551999

So what would be the recommended settings in regards to Timer resolution and HPET on/off?

Well... I want to read into this or test it since over a year but honestly... Wasn't worth the trouble yet.
It might screw up your whole system

 

[edwinjacobs]

when I check today with the same tool (and not having changed a thing), it shows my timer resolution is now at 0.499ms:

yesterday it was 1.000 ms ... ???

And according to this guide here https://github.com/djdallmann/GamingPCSetup
it's best to keep HPET on in bios ...
pfff too much conflicting information and in the end you don't know what to believe anymore (and I can't understand it all to make my own educated best guess).

Now I understand why so many people shy away from pc gaming and go for consoles...
but then again on console there's not the possibility to scratch your hoarder itch in terms of car mods.

Is there anyone who loves tweaking and for who'm it comes as second nature, and wants to help me (via Discord and/or Teamviewer) to help tweak my pc?
I don't have the energy and brainpower to learn it all myself. (at home with a burnout and covid, I just want to enjoy simracing and not have to worry about resolution timers and frametimes etc. this drains me batteries more then it recharges them!

 

[edwinjacobs]

i reverted back to having HPET on in Bios and device manager, now timer resolution is 0.500ms exact (but still getting the issue )
other things tried to no avail:
- disabling PP (in Settings - Assetto Corsa - Video)
- in bios :
* global c-state control OFF (instead of Auto)
* HPET back on and also in Win10 device manager
=>timer resolution is now always 0.500 ms instead of 0.499 or other
* CPPC -> Disabled (instead of Auto)
* CPPC Preffered Cores -> Disabled (instead of Auto)
- NVIDIA PhysX set to CPU
- disabled Oculus Virtual GamePag emulation bus (in System Devices)

So still the issue occurs almost every time in first lap (w full grid ACFL 2022 on Monaco 2020).
Last time I had the error message (>99%) and slowmo happening, I cancelled the race and got error saying that acs crashed... so I took a look at my assetto log.txt...

It says alot of "WAS LATE, HAD TO LOOP" (see attached files for my log files, remove the .txt ,they are rar archives)
Hope these can be a good indicator for someone to get me on the right track to get rid of this...

cheers, and sorry for my desperate previous post

 

[Stereo]

It says alot of "WAS LATE, HAD TO LOOP" (see attached files for my log files, remove the .txt ,they are rar archives)

That's what PHY_LATE is counting, it logs it any time a physics frame took over 3ms. So it just shows where in the log the game had slowdowns.

 

[pattikins]

i reverted back to having HPET on in Bios and device manager, now timer resolution is 0.500ms exact (but still getting the issue )
other things tried to no avail:
- disabling PP (in Settings - Assetto Corsa - Video)
- in bios :
* global c-state control OFF (instead of Auto)
* HPET back on and also in Win10 device manager
=>timer resolution is now always 0.500 ms instead of 0.499 or other
* CPPC -> Disabled (instead of Auto)
* CPPC Preffered Cores -> Disabled (instead of Auto)
- NVIDIA PhysX set to CPU
- disabled Oculus Virtual GamePag emulation bus (in System Devices)

So still the issue occurs almost every time in first lap (w full grid ACFL 2022 on Monaco 2020).
Last time I had the error message (>99%) and slowmo happening, I cancelled the race and got error saying that acs crashed... so I took a look at my assetto log.txt...

It says alot of "WAS LATE, HAD TO LOOP" (see attached files for my log files, remove the .txt ,they are rar archives)
Hope these can be a good indicator for someone to get me on the right track to get rid of this...

cheers, and sorry for my desperate previous post

I can't work out if you think there is something wrong with your PC or if you're trying to tweak your PC to play nicely with all the mods that you've got installed - looking @ the log, that includes the cars, track, CSP, SOL, CM... If I was testing my PC for possible issues I'd use the stock content, with the minimum devices attached, slowly adding one device/mod @ a time...

It would be interesting to know if the cpu is struggling prior to the slow down in frames, as there are some mods that appear (I don't use CM so I might have misunderstood...) enabled, for instance, 'Extension `Smart Mirror` enabled, settings loaded', that I imagine would trouble an Intel 12900K cpu...

Note - Phy_Late, & use of ASW.

 

[Lizard Of Bodom]

It would be interesting to know if the cpu is struggling prior to the slow down in frames, as there are some mods that appear (I don't use CM so I might have misunderstood...) enabled, for instance, 'Extension `Smart Mirror` enabled, settings loaded', that I imagine would trouble an Intel 12900K cpu...
....

FYI it doesnt impact CPU much at all - using this mod all the time (loving it) and doesnt matter for my Ryzen if its on or off.

 

[RasmusP]

Note - Phy_Late, & use of ASW.

Yikes! The MAIN_T at 23-33% looks fine but at 45 fps that's not unusual. The PHY_LATE at over 8000 and going up constantly though.. Ouch! Hopefully that's only due to him recording and adjusting things and not a constant thing...

FYI it doesnt impact CPU much at all

After all these talks it would be nice to always include specific information what "Doesn't impact my CPU" means.
Overall CPU load in Taskmanager/Afterburner, the MAIN_T not being higher, PHY_LATE still close to 0 after a few laps.
As you can see in the video pattikins posted, everything seems fine but his PHY_LATE is going up like crazy, which leads to issues. Cars jumping around, invisible contacts etc.

 

[Lizard Of Bodom]

After all these talks it would be nice to always include specific information what "Doesn't impact my CPU" means.
Overall CPU load in Taskmanager/Afterburner, the MAIN_T not being higher, PHY_LATE still close to 0 after a few laps.
As you can see in the video pattikins posted, everything seems fine but his PHY_LATE is going up like crazy, which leads to issues. Cars jumping around, invisible contacts etc.

As I didnt notice ANY downside of enabling this mod I obviously didnt do any testing on this so cant give you raw data, was simply referring to "imagine would trouble an Intel 12900K cpu" statement saying that "I dont think it would", based on my experience and my much less powerfull CPU.