As some of you may be aware, Amaurice has been winding down AmauriceBot's activity and has been looking for a new bot to take over it's remaining tasks. I have therefore created a script to allow Cresbot to take over the updating of Module:Hiscore counts. This forum seeks to formalise the changeover of that specific task.
Cresbot's source code is available here, and depends upon ceterach (the other non-standard modules can be downloaded from PyPi). At time of writing, there are still a few issues I'd like to resolve in error handling and it also needs something to automatically start the task.
I'd like to start the task midnight UTC each day. AmauriceBot ran the task every 2-3 days, but as there is a rate limit on requesting hiscore pages, which if exceeded too much can cause my IP to be blacklisted and unable to access any hiscore pages, I'd like to be able to minimise the number of requests I have to make with each run. User:Cqm/Signature
Discussion
Strong support name - also the functions are cool too MolMan 13:25, March 3, 2015 (UTC)
Comment - If Amaurice is winding down his bot's activity, is the Charm log bot going to need replacing too? If so, would it be possible for you to add that as well? I support allowing Cresbot to update the hiscore count. I think I wait 2-3 seconds between each request to Jamflax to avoid getting throttled. User:TyA/sig 13:27, March 3, 2015 (UTC)
- Scraping hiscore pages seems much more sensitive, I found 12 seconds gap between requests consistently avoids getting rate limited. I assume it's limited to 5 requests per minute averaged over a number of minutes, probably around 2-5 at a guess, which seems sensible and happens to be more or less the guidelines we use for editing via AWB assuming human usage.
- The charm logs task is continuing for the time being, but I'm hoping to merge it into the charm drop add script as I don't see why we can't just merge the two lots of error checking into one. User:Cqm/Signature
- Sounds good to me. User:TyA/sig 17:40, March 3, 2015 (UTC)
- I've also designed the task implementation to be extensible so it should be relatively simple to add new tasks as required, such as clearing the sandbox or simplifying the process for creating new exchange modules. User:Cqm/Signature
- Oi, can you elaborate this on my talk page. I'd really need to know seeing how I'm pretty much 99% of charm CVU. MolMan 13:06, March 6, 2015 (UTC)
- Sounds good to me. User:TyA/sig 17:40, March 3, 2015 (UTC)
Support - Amaurice talk 10:27, March 5, 2015 (UTC)
|
ThePsionic 
08:59, March 27, 2015 (UTC)
- Right now I'm still working through some bugs that are preventing certain scores from being updated, one in particular is proving a little tricky to catch. I'd rather this stay open until I'm confident it can run automatically without me needing to debug and tweak it every couple of days. User:Cqm/Signature
- Which one? MolMan 13:54, March 27, 2015 (UTC)
- The particular bug that's causing me noticeable issues, it one that causes an infinite loop and eventually results in an error as it exceeds the recursion limit. As I'm using 12 seconds between requests, it takes hours for this limit to be reached and results in a huge stack trace making debugging a little more annoying that usual. User:Cqm/Signature
- Which one? MolMan 13:54, March 27, 2015 (UTC)