I am considering performing web scraping on the site at regular intervals. Would scraping every one hours be acceptable, or would this frequency potentially cause issues for the servers?
I reviewed the robots.txt file, my interpretation is that specific paths are disallowed while others are allowed. For instance, /community/leaderboard/ and /community/searchplayers/ is disallowed for scraping. I am interested in scraping the /community/claninfo/ does not appear to be explicitly disallowed in the robots.txt file. Can anyone confirm if scraping this path is permissible?