Jump to content

So I gathered some information regarding the vkWaitForFences() CTD error


Foreword: I have decided to forego including links to the XPlane forum to avoid breaking any rules. If a mod could please let me know if that's OK to do or not so I can provide better sources. 

To preface this, I wouldn't wish the madness I descended upon to discover what I have discovered about this error. Also the fact that a lot of assumptions are made here based on vague wording, some testing that I admittedly didn't do on other machines because I simply didn't have the access to other machines to do it on. Please take this with a grain of salt, and if you have anything to add, I'd be more than willing to listen to it.

This whole thing started when I migrated to Linux from Windows a couple of weeks ago. Went to fly some jets when I started encountering a weird crash every few matches. I couldn't find anything definitive upon a curosry  Google search to fix the issue and decided to head up on to the Warthunder forums to post about it, when I found out I am definitely not alone with this post. 

 

I posted that I was having the issue as well, and uploaded my .clog files, an inxi output, and a .dmp file as per bug reporting guidelines. Shortly after, this thread was closed and moved to Already Reported & Solved. That being said, I wasn't sure if that meant the problem was solved and pending update to the game, or if the solution was posted elsewhere and it just wasn't mentioned. So I ventured on to the Community Technical Support subforums to find that this definitely seems to be at least a somewhat common issue. 

That's when the madness began

I started combing through each of the posts, seeing if anyone came up with a solid working solution. I found a couple that seemed to solve it for some people, and not for others. For instance there was one in which you pass a launch argument to make sure the game was using the proper Nvidia icd (In their case, it was AMD for their AMD GPU instead of the Radeon one) and that seemed to solve it for them. It improved my FPS for sure, but did nothing to stop the crashes unfortunately. The referenced post being the last reply by Kayne_92 to this topic 

 

Speaking of this topic, I also found the vkQueueSubmit() error to be suspiciously similar to the vkWaitForFences() error. So I decided to look into these specific function calls to shed some light onto the issue gleaming onto the word "failed" in the error report window. For both of them, I found 3 specific returns that would be classified as "failure". VK_ERROR_OUT_OF_HOST_MEMORY, VK_ERROR_OUT_OF_DEVICE_MEMORY, and VK_ERROR_DEVICE_LOST. So I set out to eliminate the out of memory errors. Sources: https://www.khronos.org/registry/vulkan/specs/1.2-extensions/man/html/vkWaitForFences.html | https://www.khronos.org/registry/vulkan/specs/1.2-extensions/man/html/vkQueueSubmit.html

My weapons of choice were htop to monitor CPU/Memory usage, and a Python program called gpustat (I'll edit a link to this program if asked for) to keep an eye on VRAM usage. What I found is that memory usage across the board was fairly normal. No major leaks, not even close to hitting the limit. With a couple browser tabs, Steam, and Warthunder running, my RAM was at 7/16gb, and my VRAM was steady at 4/8GB. So unless there's some serious shenanigans going on with memory allocation, chances are that knocks out the first two as a potential problem.

Upon my Google excursions learning more about these functions, I found a very similar error occurring in XPlane 11 in September. This time with access to more detailed log files (Or at least I have no idea how to look at .clog files) the userbase was able to determine that it was device loss causing the problem. Though I have no idea how the problem got solved, I assume the devs had fixed it in some way. 

Looking for solutions in the XPlane community (A stretch, I know, but you know, madness) I had come to a sudden realization. Most if not all the people reporting this problem without a solid fix the user can implement are people running Nvidia GPUs. Whether or not that has anything to do with Nvidia's current marketshare of the gaming market when it comes to GPU is up for debate, but I decided to turn that particular stone anyways. Unfortunately I did not find anything particularly interesting of note other than an Nvidia library that basically functions similar to Nvidia's Aftermath system for DX by using checkpoints in the drawing stages. 

To make a very long story short. For reasons I don't even know myself, I decided to try and research this issue to try and find my own way of fixing it, but have come up empty handed so far on fixes, but have made some interesting conclusions that I would definitely be interested in other people testing, that being;

The crash is probably the vkWaitForFences() function returning VK_ERROR_DEVICE_LOST, which as I found it in another post on XPlane is something that can be generated in the GPU basically getting caught up in a calculation one way or another, and the watchdog process stopping it, causing the device to reset and chain react an error all the way up. 

There doesn't appear to be a severe RAM or VRAM leak causing the issue, not even really a CPU or GPU usage spike just before.

That being said one thing I have been having which I haven't seen anyone else talk about it my system commonly hangs for 1 or 2 seconds after acknowledging the error. That could just be something unrelated though.

Let me know what you think, if I got something wrong, if I got something right, or if you have an idea to contribute.

I just wanna fly again, man.

  • Thanks 1
  • Upvote 2
medal medal

Share this post


Link to post
Share on other sites
10 hours ago, ConspicuousFox said:

Shortly after, this thread was closed and moved to Already Reported & Solved.

 

I should be called "Already Reported OR Solved" actually. So for Linux bugs it means "We know about it and do no care much".

 

You man seem to be quite desperate, it looks that if you had the source code you would probably fix it for Gaijin. According to @apemax the bug was worked on quite some time ago, but it is hard to tell if it is so time consuming to fix or they have gave up till now.

 

I guess it boils down to how much cash do Linux users bring in, so my advice is not to spend a dime on WT till the bug is fixed.

  • Upvote 2
medal medal medal medal medal medal medal medal

Share this post


Link to post
Share on other sites

I haven't really worked with C, and definitely haven't worked with Vulkan so I'm not sure how much progress I'd make in that scenario lol. Funny you should say that though because NVIDIA does have a tool to track down this exact problem as it is actually a well documented bug in Vulkan that is really hard to track down without the proper tools. 

Not sure I'd call myself desperate, but I get really annoyed when I can't fix the problem myself and answers to it are vague. Also helps that sometimes I hyper fixate. Surprisingly all of this was for the most part found in one night. I was planning on buying a pack because I really do enjoy the game, though there's no point to it with the game being virtually unplayable for me.

Especially sucks because my roommate and I were having a lot of fun flying together. I really hope they didn't just kick it to the side because I'd love to play this game again, I just can't justify reinstalling Windows for one game :/

medal medal

Share this post


Link to post
Share on other sites

So I'm just going to keep dumping information about this crash and the vkQueueSubmit() crash in here so that there is a single point of reference. I saw another thread in which enabling old video card support might help the issue regarding vkQueueSubmit, not sure if it helps vkWaitingForFences 

 

medal medal

Share this post


Link to post
Share on other sites

I noticed after the latest update it's taking a lot longer for the crash to happen for me

 

Edit: It's now happening worse. Can barely get a single match in.

Edited by ConspicuousFox
medal medal

Share this post


Link to post
Share on other sites

Same problem here.


At first I was abble to play tank battles, cos planes get the WaitForFences error. Now I can't even load the game.

My notebook has two very basic GPUs, Geforce 940MX and Intel 950. I'm able to play using the Intel GPU but not the Nvidia one.

 

edit: I'm using Ubuntu 20.04 on a Dell Inspiron 7560 (i7 16GB)

 

Edited by Guitarnan
Adding OS

Share this post


Link to post
Share on other sites

Just to add that enabling old video card support didn't help me either.

 

19 hours ago, ConspicuousFox said:

So I'm just going to keep dumping information about this crash and the vkQueueSubmit() crash in here so that there is a single point of reference. I saw another thread in which enabling old video card support might help the issue regarding vkQueueSubmit, not sure if it helps vkWaitingForFences 

 

 

Share this post


Link to post
Share on other sites
8 hours ago, Guitarnan said:

Same problem here.


At first I was abble to play tank battles, cos planes get the WaitForFences error. Now I can't even load the game.

My notebook has two very basic GPUs, Geforce 940MX and Intel 950. I'm able to play using the Intel GPU but not the Nvidia one.

 

edit: I'm using Ubuntu 20.04 on a Dell Inspiron 7560 (i7 16GB)

 



So you're saying that the game wasn't working at all until you switched to your iGPU (Intel) and the problem ceased entirely? Even in air battles? If so that would definitely support the idea that is is a specific issue related to Nvidia's implementation of Vulkan that devs need to work around. 

@apemax Is there any word on this bug? You've closed a lot of these threads very vaguely in the moderated bug report section. It'd be good to hear just that the devs are working on it because it is making the game unplayable, and it's kind of embarrassing to have a game of this caliber breaking on a platform they claim to support. Switching to the iGPU is not an option for everyone because not everyone runs an Intel CPU or AMD APU for that to be workable. 

medal medal

Share this post


Link to post
Share on other sites

Alright, so I might have found something that might possibly mitigate the problem, but I haven't 100% verified on my system so far. I haven't ran into a crash yet since doing this, but with how unpredictable the crashes on my setup are, I have no idea if this changed anything.

So for those running Ubuntu, the latest proprietary driver is Nvidia 455 but there is an Nvidia 460 production branch version that isn't listed for reason I don't know, could be beta or not very well tested, but I'm not sure yet.

PROCEED WITH CAUTION

Download the Nvidia 460 driver here. This will download a .run file which is not usable in the desktop, you're gonna have to do something a little different.

Referencing the answer to this question, install the driver. It will sit there like it's doing nothing, but just wait. Run some errands, get some coffee, whatever you gotta do. It says restarting is optional, but you just installed a driver, it's best to restart. Afterwards verify that you're running the correct driver by running nvidia-smi in the terminal. At least on my end the gui driver program does not report the driver correctly, but smi will. It should report the driver version being 460 like below. 

 

564133947_Screenshotfrom2021-01-0716-50-

 

Hopefully this helps some people, whether it does or doesn't, please reply with your results. I still haven't 100% confirmed if it helped me or not, but figured it was worth a shot either way because I think they did some changes to the Vulkan implementation on 460

  • Thanks 1
medal medal

Share this post


Link to post
Share on other sites
20 hours ago, ConspicuousFox said:



So you're saying that the game wasn't working at all until you switched to your iGPU (Intel) and the problem ceased entirely? Even in air battles? If so that would definitely support the idea that is is a specific issue related to Nvidia's implementation of Vulkan that devs need to work around.

 

Exactly.
I will try the nvidia 460 as you said.
I already tried that, but I think I got stuck in the installation process. Lets try it again.

Cheers.

 

Edit:

No more issues after installing the newest driver.

 

Edited by Guitarnan
Adding results

Share this post


Link to post
Share on other sites

Looks like my game stabilized too after a handful of sessions a couple hours long. Looks like the 460 driver solves the issue. I'll make a separate post diving more into detail about the fix. 

medal medal

Share this post


Link to post
Share on other sites

Thanks for the help. This WaitForFences error has been driving me nuts for the last couple of days ( applied an NVIDIA drive update to 20.04).


Will now try moving to 460 from 450 and reply

Share this post


Link to post
Share on other sites

@ConspicuousFox,

 

Well as I've already said the issue is known and the Devs have been working on this however as you say it seems like the 460.32.03 driver might have resolved this issue, I have yet to get the crash since upgrading to it today.

Share this post


Link to post
Share on other sites
23 minutes ago, apemax said:

@ConspicuousFox,

 

Well as I've already said the issue is known and the Devs have been working on this however as you say it seems like the 460.32.03 driver might have resolved this issue, I have yet to get the crash since upgrading to it today.


If that's the case across multiple systems, I'd suggest pinning my newest post on installing the 460 driver as a solution to the error. It's possible it may be very difficult for the devs to work around this one as it seems to be an Nvidia specific problem to the driver.

medal medal

Share this post


Link to post
Share on other sites
56 minutes ago, apemax said:

Well as I've already said the issue is known and the Devs have been working on this however as you say it seems like the 460.32.03 driver might have resolved this issue, I have yet to get the crash since upgrading to it today.

 

The question is if Gaijin knew about the issue before switching the game to Vulcan yet decided to go with it anyway or was it just lack of testing of Linux client. I think that most of Linux players are with Nvidia GPUs.

 

It seems that most of players regardless of platform were quite disgruntled with two last major updates as the stability problems overshadowed new features and content. I think that Gaijin would be better off if it took quality assurance more seriously.

As a Linux user I am really afraid of major updates of WT.

 

medal medal medal medal medal medal medal medal

Share this post


Link to post
Share on other sites
53 minutes ago, _azi_ said:

 

The question is if Gaijin knew about the issue before switching the game to Vulcan yet decided to go with it anyway or was it just lack of testing of Linux client. I think that most of Linux players are with Nvidia GPUs.

 

It seems that most of players regardless of platform were quite disgruntled with two last major updates as the stability problems overshadowed new features and content. I think that Gaijin would be better off if it took quality assurance more seriously.

As a Linux user I am really afraid of major updates of WT.

 


I definitely echo azi's concerns. I just moved to the Linux platform myself. Warthunder is the only game I really had major problems with, but this almost does seem like it's a case of either they didn't care or didn't properly test the implementation. Considering this is a pretty openly known about problem with Vulkan and Nvidia, I'm siding on the didn't properly test/research the implementation of Vulkan.

Either way, it's good there's a driver version out that at least fixes this problem. 

medal medal

Share this post


Link to post
Share on other sites
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...