Too much afk causes me to not join a game

Went outside to get some stuff, came back and tried to play, but it wont let me. happened on multiple occasions, any clue on why it happens and how to fix it without reopening the game?

Restart the game usually fix stuff like that.