Forums: Yew Grove > New bot and protection for Exchange namespace
Replacement filing cabinet
This page or section is an archive.
Please do not edit the contents of this page.
This thread was archived on 15 May 2012 by Cook Me Plox.

We recently discovered that we can get accurate, un-rounded Grand Exchange prices by using the graphing API on Currently, we take the prices from the category API, which are rounded, sometimes by up to 5% of the actual price. It's been a problem that's bugged me for a long time.

User:Σ created a bot that grabs accurate prices from the graph API. More importantly, it runs any time there is an update, instead of whenever the operator feels like running it. User:TyA has volunteered to run it under User:TyBot, and because he has a stable 24/7 connection, it can run any time a price update is detected. The source code is here and its testing can be found here.

User:BrainBot's code has also been updated, and it will run as a backup in case TyBot is unable to operate. Whenever they are both online, they will tackle the page list and get it done in half the time. User:AmauriceBot will also run in case both TyBot and BrainBot malfunction.

This is a vast improvement over the current system, where the bots are run whenever requested, and sometimes they miss a day. This way, not only will we have exact prices (more accurate than any other fansite or the GEDB), but we'll have them added in a timely manner.


This change, if approved, will render human updates to the Exchange namespace almost entirely unnecessary. We have already made it so that without knowledge of how the wiki works, the only thing anons can do is update the prices. The last time we tried to protect it, the main argument against it seemed to be that the prices were sometimes rounded, and oftentimes unregistered editors would correct the prices to the exact amount. That should no longer be a problem if we use these new bots. I am proposing that we semi-protect the entire namespace, remove the price updating script, and unhide the edit button.

I know some people might be opposed to this because it's protecting a huge number of pages that are often updated by lots of editors with a fairly small amount of vandalism. However, the Exchange NS is just a data namespace, an imperfect solution to the fact that MediaWiki doesn't support cURL very well. The only reason that an anon would ever need to update prices would be if the bot hadn't finished updating all the pages, which takes approximately 4-5 hours. There is a small chance that an unregistered user would notice the old price and update accordingly, but I think that benefit is outweighed by the continued chance of vandalism and the probability that the prices these users add are the rounded versions from the GEDB.

Note that the bots will still be able to run even if the second proposal doesn't pass. Protection depends on the approval of the bots, but not the other way around.


Support bots and protection - ʞooɔ 02:22, May 8, 2012 (UTC)

Strong support both - Excellent idea, should cut down on vandalism. Should the first proposal pass, I'd suggest that we officially switch TyBot from AWB account to full-fledged bot. Should the second proposal pass, I'd like to add the "Edit" button back to Exchange pages, as people not knowing what they're doing shouldn't be an issue.

As a second point, would the script for updating the prices be removed if this passes? I would support removing that as it is mostly defunct with bots. --LiquidTalk 02:26, May 8, 2012 (UTC)

Derp, skipped over that sentence. --LiquidTalk 02:29, May 8, 2012 (UTC)

Support both - The bots should indeed make the Exchange namespace accurate, and based on the rationale presented, semi-protection of the entire namespace would be beneficial. I have to look at a few questions I have about the implementation first:

  • What if two bots are updating at the same time? Is there proper edit-conflict handling? Will a bot edit the price or date if the price is the same as before?
  • A corollary of the question above: Is it beneficial for more than three people to run this code? Perhaps any editor with a bot account could be told that they can run the code.
  • Does the bot exit if it receives an Exception?
 a proofreader ▸  02:35, May 8, 2012 (UTC)
To add to the above:
  • What are the contents of gemwdata.itemsdict?
  • What happens if a new item appears in the game? If nothing happens, how must this be dealt with?
 a proofreader ▸  02:42, May 8, 2012 (UTC)
The full source is at w:c:zammy:GEMWbot. There is some additional things that I'll add in later after searching through my IRC logs.
1. <CoalBalls> "in the current revision of the wikibot there is no edit-conflict handling. However, that is unnecessary for this task because both bots will update with the same price."
3. No.
4. It's a dictionary of items. See w:c:zammy:GEMWbot/
5. It has to be manually updated, if not, it won't update that certain item's price. 22px-Logo.svg.pngTyA_userpage_image.png 03:34, May 8, 2012 (UTC)
1. Huh... I suppose, but there's a Date parameter in the Exchange namespace pages which may get updated if the conditions are right. All the bots could end up editing the page with the same price and a different Date exactly once, so with 3 runners there could be 2 extra edits.
3. Okay.
5. Bah, it would work better if the items were gotten automatically. Look at how long it took RuneScript to get potion flasks, for instance: RuneScript forum link. These were probably manually put in after the GEDB site on was modified.  a proofreader ▸  03:54, May 8, 2012 (UTC)
1. The AmauriceBot will only update the page if the price is actually different to the Exchange page, or if the page has not been updated for 7 days (i.e. price has been static for 7 days). Also, AmauriceBot will not update the page if it has already been updated less that 6 hours before (unless the price has been vandalised). AmauriceBot will operate as a backup in other bots fail only. Amaurice talk 11:03, May 8, 2012 (UTC)
Whenever Sigma was stripping his bot base so that he'd feel comfortable sharing it, he realized that there was some built in edit conflict handling. Though you'd have to ask him for the specifics of it. 22px-Logo.svg.pngTyA_userpage_image.png 20:06, May 8, 2012 (UTC)

Strong support both - That will definitely cut down on vandalism. I see no problem with it. Blaze_fire.png12.png 02:30, May 8, 2012 (UTC)

Support both - 22px-Logo.svg.pngTyA_userpage_image.png 02:32, May 8, 2012 (UTC)

Support - Although I am unsure about the API for the graphs, isn't this scraping the page? The Catalog JSON API had a throttle after 100 to 120 requests so every quarter to half hour and I am unsure if changing the API location would do anything differently. If we are going to have the bots edit the Exchange NS, then why not leave the edit button hidden? It's easy to change with the action. The removal of the button was just to limit the amount of misinformation or lack of information added to the data page. Sadly, I am unsure if the GEDB writer will ever add the information we wanted to the API. More important matters to adhere to. Ryan PM 04:26, May 8, 2012 (UTC)

The Graph JSON API proposed also has the same Jagex throttle. The limit is 100 queries + one query every 5 seconds (for given IP address). Amaurice talk 08:13, May 9, 2012 (UTC)

Strong support both - Per the nominator. I wouldn't mind a few of Proof's points taken into strong account though, namely giving some basic edit conflict handling, just in case (even if it's just "wait five seconds and try again, skip if failing once more") and something more efficient than a manual item list (is there no way to grab all pages in the namespace with the proper template?). I was initially unsure of reshowing the edit button, but figure, why not, provided we still have an abuse filter to keep a close eye on any edit a user might make. Hofmic Talk 06:34, May 8, 2012 (UTC)

Support both with slight alteration - I've had discussions with Cook regarding the future of my bot, and I agree that this would be an advantageous move. Perhaps it would be better to use an abuse filter to block edits to the Exchange namespace as it would allow us to display the anons a message explaining why they cannot edit i.e. a bot will work through the pages soon, you haven't been blocked from editing, etc. 222 talk 06:51, May 8, 2012 (UTC)

Super duper strong support - Per all. --Haidro (talk) 07:37, May 8, 2012 (UTC)

Support both - We could merge the charmedit right that rollback+ users have, which allows them to edit the charm namespace, with the thing we would do for the Exchange namespace. This would prevent having to do something like adding a cascading protection to the ExchangeItem template or the use of an abusefilter, and it will protect the pages further than just semiprotection.
I think these new bots would be a great improvement, and I think protecting the namespace would help a lot in preventing vandalism. For the protection, I think the system used for the Charm: namespace is the best. JOEYTJE50TALKpull my finger 08:58, May 8, 2012 (UTC)

Support - Continuing Joey's idea about the charmedit right extension included within rollback and the problem of new items, would it be possible for the list of items needing to be updated to be editable by those with rollback rights, making it easier for us to make sure a new item is added in quickly? cqm  09:40, May 8, 2012 (UTC)

That will not be neccessary because the bot should be able to fetch an up-to-date list of items immediately before updating. If not, this simple feature should be added. 222 talk 09:43, May 8, 2012 (UTC)
Out of interest, where does the bot fetch this list from? cqm  22:49, May 8, 2012 (UTC)

Support - Yes. 18px-Avatar.png Fswe1 26px-Brassica_Prime_symbol.svg.png 13:42, May 8, 2012 (UTC)

Support - Very good. Well done to all involved in making this. Ronan Talk 18:39, May 10, 2012 (UTC)

Support both - per above. Small recharge gem AnselaJonla Slayer-icon 18:40, May 10, 2012 (UTC)

Closed -

  • Wikia will be asked to add "exchangeedit" right to Rollback, Custodian, Administrator, Bot, Wikia Helper and Wikia Staff groups.
  • User:TyBot will become a full bot (cosmetic change only).
  • The edit button will be re-added to the Exchange pages.
  • Exchange namespace AbuseFilters will be removed.
  • The easy-update Exchange script will be removed.

ʞooɔ 02:39, May 15, 2012 (UTC)

Ad blocker interference detected!

Wikia is a free-to-use site that makes money from advertising. We have a modified experience for viewers using ad blockers

Wikia is not accessible if you’ve made further modifications. Remove the custom ad blocker rule(s) and the page will load as expected.