Forum:Scraping hiscore data for calculators

Forums: Yew Grove > Scraping hiscore data for calculators

I recently received the suggestion to implement hiscore scraping for JS calculators, like these ones, and I've finished the script; you can try it by pasting

importScript('User:Bluefire2/importStats.js');

here, and then loading any js calculator. It adds a text field and a button to the calculator interface, and when the button is clicked, the script fetches the level (or XP, depending on the selection) of the account whose name was entered in the field from the hiscores, and inserts that number into the corresponding field.

I think it would be a nice addition to the wiki. If you think it's too disruptive, I can replace it with a small checkbox that will summon it only if the user wants to. I've also considered logging the last username the user entered with a cookie, as Joey's QC script does, but, of course, that would need community consensus.

β (t&c)Oil4 I made this 16:19, July 25, 2014 (UTC)

Discussion

Big caek support β (t&c)Oil4 I made this 16:19, July 25, 2014 (UTC)

Support - And I'd like the caekie cookie too Oil4 ^Talk 19:51, July 25, 2014 (UTC)

Comment - Looks pretty good, however I would suggest having this feature be an opt-in for calculators by having a class on the form, rather than adding it to all calculators. ʞooɔ 13:48, July 26, 2014 (UTC)

If all the calculators that you want to have this are modified as such, then this is a very easy modification. Alternatively, we could just have a .js file where we whitelist all the calculators that need this. β (t&c)Oil4 I made this 14:03, July 26, 2014 (UTC)

Whitelisting the calculators is dependent on an admin being around who's confident in doing something like that. Adding a class or a way of autodetecting where it should load is easier to maintain imo. User:Cqm/Signature

Comment - A few notes on your code:

• Use $( func ) over addOnloadHook( func ); The latter was deprecated years ago.
• Would it be possible to not load getHSData.js in your script unless it's not already loaded? We can add it to the existing imports array in the Common.js and import it using importArticles taking advantage of ResourceLoader which minifies it and reduces http requests. We'd need to import getHSData.js before importStats, but RL honours the order of the imports iirc.
• Maybe stick the ES6 polyfill for String.prototype.contains in a conditional for those browsers that do support it and have have a marginally better way of doing it.
• Why did you add jQuery into the closure, but then use jQuery over $ in when you extend jQuery itself?
• Why did you elect to extend jQuery? Surely there's a easier to follow method, such as looking for a class. Maybe altering Q's calc script for something you can hook off?
• You should be defining your functions before using them ideally, which you've followed for those in the local scope, but not for those that end up in the global scope.

This imported script looks good apart from all the globals which you've avoided in the proposed script. You've also extended the String prototype to avoid doing someString.replace( / /g, '\n' ); which saves all of one character in a method you use once... User:Cqm/Signature

1. OK, will do.
2. Er, OK. As long as it's loaded wherever importStats is loaded, that's fine.
3. What?
4. Sorry, that was dumb. I meant to just isolate the code in a closure, and I most of the times I've done that was to extend jQuery. Reflex action, I guess.
5. ^that.
6. All the functions in importStats.js are limited to that closure, so are not global. The only global functions are in getHSData, which is imported at the *top* of importStats.js. Anyway, it doesn't matter where I define them if I use the function keyword... right?
7. I'm just used to extending prototypes whenever I code something, if I need to. Anyway, replaceAll is quite a commonly used extension, so there shouldn't be any conflicts.

β (t&c)Oil4 I made this 09:40, July 28, 2014 (UTC)

This is what I mean by putting the polyfill in a conditional. I originally thought it was a custom string method until I researched it, but apparently it's part of the EMCAScript 6 proposal. So far it's just Firefox that supports it. User:Cqm/Signature

Also if you extend a global in a closure, you extend that global. It only works like you think if you assign a global to a local variable and then extend that local variable:

// this extends jQuery
( function ( $ ) {

    $.foo = function () {
        console.log( 'foo' );
    };

    $.foo();

}( jQuery ) );

// this won't
( function ( $ ) {

    var $local = $;

    $local.foo = function () {
        console.log( 'foo' );
    };

    $local.foo();

}( jQuery ) );

I also made a quick edit to your script here to show you what I meant about the closure thing. Undo my undo if you want to keep it :) User:Cqm/Signature

Ah, I see now. The reason I used jQuery and not $ was because I initially wrote the extension outside of a closure, and it was when I was putting it into one that I carelessly copypasted it in. Now that I've 'accepted' your edit, I see no problem with extending jQuery in this way. β (t&c)Oil4 I made this 10:11, July 28, 2014 (UTC)

This is not true! Both extends jQuery. Try running the last one in your browsers console and after that run `jQuery.foo()`. User:Tyilo/Signature 22:54, August 16, 2014 (UTC)

So it does, my bad. Learn something new every day. User:Cqm/Signature

Comment When I click the link to paste the calculator it tells me the page does not exist. Can somebody explain what I am doing wrong? This is exactly what I think the calculators have been missing. Looks like a good idea. --Yayepicyay (talk) 21:18, August 8, 2014 (UTC)

If Special:MyPage/wikia.js doesn't exist, it means you need to create it. MolMan 21:22, August 8, 2014 (UTC)

Think I must be doing something wrong because the change bluefire mentioned did not appear to have happened. Do I need to do anything special on Special:MyPage/wikia.js to get it to work? Thank you, --Yayepicyay (talk) 21:33, August 8, 2014 (UTC)

Comment - Nothing interesting happens. (Aside from skills not defined error being thrown when importscript is being bad.) I even went and copypasta'd the entire script and still nothing.  @Gaz#7521 00:49, August 9, 2014 (UTC)

Great, it doesn't work now. It worked when I made this thread. I'll take a look at it... β (t&c)Oil4 I made this 07:59, August 9, 2014 (UTC)

Support - It works for me now and it's looking excellent, we should have done this years ago. I like to cookie idea too, just remember to account for display names changing since when the cookie was last set. User:Cqm/Signature

Support - It works now for me as well. What did you change from last version Bluefire? Yayepicyay (talk) 12:58, August 17, 2014 (UTC)

Comment - Doesn't work me for me, I just get 2 alerts with the text "Unable to load data. Please try again later." I can see that my browser tries to load http://anyorigin.com/dev/get?url=http%3A//services.runescape.com/m%3Dhiscore/index_lite.ws%3Fplayer%3DTyilo&callback=jQuery18208853682687040418_1408293345423 which redirects to http://anyorigin.com/dev/get/?url=http%3A//services.runescape.com/m%3Dhiscore/index_lite.ws%3Fplayer%3DTyilo&callback=jQuery18208853682687040418_1408293345423 which returns "jQuery18208853682687040418_1408293345423({"contents":"alarm\n"})" User:Tyilo/Signature 16:38, August 17, 2014 (UTC)

It's working for me Oil4 ^Talk 19:22, August 17, 2014 (UTC)

It works for me too now. I blame anyorigin.com User:Tyilo/Signature 19:24, August 17, 2014 (UTC)

I've had some problems with anyorigin.com myself too when implementing HS fetching in S:C. It happened regularly that the server just replied with a single word "alarm" too, instead of replying with the data. There didn't seem to be any logic behind this reply, and it sometimes even consistently didn't work for one username, but it did work for another username at the same time. Weird stuff. JOEYTJE50TALK pull my finger 11:40, August 20, 2014 (UTC)

Support - It's working for me. Temujin 00:21, August 18, 2014 (UTC)

Problem - It has stopped working for me... When I try using, get error: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://slot1.images.wikia.nocookie.net/_cb1408029593/common/skiins/oasis/fonts/prociono-Regular-webfont.woff. This can be fixed by moving the resource to the same domain or enabling CORS. "alarm " "parsing the data" "executing the callback function"

I have a screenshot of console output... Why is this being inconsistent? It worked before I went to bed... Yayepicyay (talk) 13:22, August 18, 2014 (UTC)

That error is nothing to do with the script, that's a font file loaded by the wikia servers (which interestingly isn't working atm). The number of issues however might warrant some alternative to anyorigin be explored. Are there any plans for that (@Bluefire)? User:Cqm/Signature

I think the best solution would be to use the wiki's server (maybe using a php script?), to do the same as anyorigin. User:Tyilo/Signature 14:51, August 18, 2014 (UTC)

Could we use action=parse of {{#hs:USERNAME}} instead? User:Cqm/Signature

Correct me if I'm wrong as I couldn't find the documentation for {{#hs:USERNAME}}, but it isn't currently possible to extract more than one skill level/xp/rank in one request. Some calculators uses more than one skill, so it should be modified so that it could return all the data and the javascript script could then extract the xp for the relevant skills. Once that is done, the script can simply do something like:

var username = 'Tyilo';
var wikitext = '{{#hs:' + escape(username) + '}}';

$.get('http://runescape.wikia.com/api.php?action=parse&format=json&text=' + encodeURIComponent(wikitext),
	function(data) {
		var totalLevel = Number($(data.parse.text['*'])[0].textContent);
		console.log(username + '\'s total level: ' + totalLevel);
	}
);

User:Tyilo/Signature 00:15, August 19, 2014 (UTC)

Here's the docs although it doesn't appear to support divination (I don't know about minigames either). User:Cqm/Signature

Seeing as it's just pulling data from jagex's api, I think all we need to do to update it for more recent additions is tweak the templates it's used in and update the comments in the source code. Everything's updated on the docs, I just need to get the actual code updated, but it looks like it's limited to 2 calls per page. Is that ok, or do we need to get it increased? User:Cqm/Signature

I don't think the limit applies to the api call (just tested it). Inserting this code:

if($skill == 'data') {
    return $data;
}

just after the lines

if( $status == 200 ) {
	$data = $wgRSHiscoreCache[$player]

should allow us to call {{#hs:Tyilo|data}} to get the raw data from Jagex's hiscores. User:Tyilo/Signature 13:35, August 19, 2014 (UTC)

I've submitted a pull request to get the comments for which numbers are what updated, so hopefully I can do this extra thing without much trouble. I just need to get my devbox booted up to test any code changes before submitting them, so it might be a little longer to get that bit updated. User:Cqm/Signature

Comment - anyorigin.com is buggy and has been for quite some time, and so the data failing to fetch is an issue outside my control. I've made a script that does the same thing on my website, and after some testing I'll put it into this (Joey's QC script already uses it, so it should be fine). Hopefully this will fix the issue. β (t&c)Oil4 I made this 16:27, August 18, 2014 (UTC)

Question - Is there any way we could make the users computer call the runescape server/have the wiki do it? This using an outside server seems kinda sketchy... Yayepicyay (talk) 20:55, August 18, 2014 (UTC)

Sadly, no. β (t&c)Oil4 I made this 12:39, August 19, 2014 (UTC)

We can make the wiki do it, see the discussion between me and Cqm above. User:Tyilo/Signature 13:38, August 19, 2014 (UTC)

Correct me if I'm wrong, but that's an extension coded in php, which is processed before JS. This would only work if I could HTTP call a template and specify a parameter... could I? β (t&c)Oil4 I made this 23:07, August 20, 2014 (UTC)

Yes. But you have to believe. MolMan 23:11, August 20, 2014 (UTC)