View Single Post
05-14-11, 02:45 PM   #1
Mazaki
A Deviate Faerie Dragon
 
Mazaki's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2011
Posts: 10
Cool Wowhead Parsing with Python

This is just a little Python script I whipped up to parse out data from Wowhead pages. Originally it was just for use with Daily To-Do's, gathering quest data, but I thought I may as well share it.
It parses out the Json embedded in Wowhead pages' source, and turns it into nice Lua tables for use in addons or whatever else.

Code:
import json, urllib, re
from types import *

encoder = json.JSONEncoder()

url = 'http://www.wowhead.com/achievements'
	
p = urllib.urlopen(url)
p = p.read()

json_re = re.compile("new Listview\({template: '\w*', id: '\w*',.*data: (\[.*\])")

match = json_re.search(p)
f = open("parse.lua","w")
if match and f:
	data = match.group(1)
	data = data.strip()
	# parse with json
	
	datadic = json.loads(data)
	
	f.write("parseTable = {\n")
	for category in datadic:
		f.write("{\n")
		for key in category:
			if type(category[key]) is IntType:
				f.write("\t{0} = {1},\n".format(key,category[key]))
			elif type(category[key]) is ListType:
				f.write("\t{0} = {1},\n".format(key,str(category[key]).replace("[","{").replace("]","}")))
			else:
				f.write("\t{0} = \"{1}\",\n".format(key,category[key].replace("\"","\\\"")))
		f.write("},\n")
	f.write("}")
	f.closed
	print "Complete and written to 'parse.lua'!"
	print "Output:"
	f = open("parse.lua","r")
	print f.read()
	f.closed
else:
	print "Error"
You can set the url variable to any Wowhead page that has Json embedded into it and it should magically come out into a nice Lua table you can use in your addons or whatever else you might need it for.
Not sure why I chose to use Python, seeing as prior to this script I had neither used or really seen it before, but whatever.
I'm sure this script is a horrible excuse for programming, but it works I guess.
Hopefully someone might find it useful.

Feel free to leave feedback or something.

NOTE: Wowhead imposes an artificial limit of 200 results, so not all results will be shown if there is more than 200.
__________________
Author of Daily To-Do's - Track all your dailies.

Last edited by Mazaki : 05-14-11 at 02:59 PM. Reason: Left a sentence half finished. Yay me!
  Reply With Quote