Budabot Items Extractor

Other topics not directly related to Budabot (Budabot Items Extractor, AOChatProxy, etc.)
Tyrence
Posts: 1913
Joined: Sat Jan 09, 2010 1:32 am

Budabot Items Extractor

Postby Tyrence » Fri Jan 31, 2014 11:13 pm

Alright. I have updated the tool so it works again. I committed the updated aodb file to Github. You can see the changes here: https://github.com/Budabot/Budabot/comm ... 6f548d0666

To run: download, unzip, run "run.bat", enter the path to your AO install, and hit enter. If you don't have Java installed you will need to install it (http://www.java.com, click Free Java Download). It will generate a file called aodb<version>.sql in the current folder, and you can take that file and drop it into the ITEMS_MODULE as-is. On my development machine it takes about 40s to run. This has only been tested to run on Windows since it needs access to an AO install, but it can run on any platform that supports Java (ie. everything).

The source is available at: https://github.com/Budabot/ItemsExtractor
The binary can be downloaded from here: https://github.com/Budabot/ItemsExtractor/releases

There are three files that control how the tool matches items and you may need to update them if you find problems with the output file:
static_list.txt - this defines matches that the tool cannot determine automatically
delete_list.txt - this defines items that should not be included in the output
nameseperation_list.txt - this defines matches where the low-ql name does not match the high-ql name

Note that you don't necessarily need to update these files when new items are added. Most of the time the tool will be able to handle new items without any changes.

The files are processed in the order listed above and then any remaining items are processed normally which involves several different algorithms for determining how to match items. It writes an output.log file with DETAILED information on every step of the process so if you do find an issue with an item the first step should be looking in that file to see how that item was handled.

When I generate the new aodb file I use a compare tool (ex: http://winmerge.org/) to compare it to the old version to get a quick list of all the changes and to verify that everything is still working as expected. I recommend doing this--otherwise issues start to creep into the items db and it becomes much harder to troubleshoot later. If I find issues I make the necessary changes to the config files and run the tool again. I STRONGLY recommend against updating the sql file directly as it creates a maintenance nightmare down the road.

Some issues can't be fixed immediately because they require more research (for instance, you may need to determine if an item is in-game or how the game matches the low and high ids). I keep track of these issues in a file called fix.txt which is in the ITEMS_MODULE.

If you have any questions or issues let me know.

EDIT: I removed the download from this post. Going forward I will be putting the Budabot Items Extractor releases on GitHub. You can download the latest version from here: https://github.com/Budabot/ItemsExtractor/releases

EDIT2: This produces two files: aodb<version>.sql and weapon_attributes<version>.sql. The aodb file should be put in the ITEMS_MODULE and the weapon_attributes file should be put in the SKILLS_MODULE.
"Those who expect to reap the blessings of freedom, must, like men, undergo the fatigues of supporting it." — Thomas Paine
"Nearly all men stand adversity, but if you want to test a man's character, give him power." — Abraham Lincoln
Budabot Releases and Downloads: https://github.com/Budabot/Budabot/releases
Captank
Member
Posts: 77
Joined: Sun Sep 09, 2012 6:36 pm
antispam: Rimor
Location: irc.funcom.com #budabot

Re: ITEMS_MODULE

Postby Captank » Sun Feb 02, 2014 3:43 pm

ty for posting that :D

and even more ty for the source code ^^
even tho its scala but should be that hard to port to java or python :D

no time to look into it now but will check it later


cap
Tyrence
Posts: 1913
Joined: Sat Jan 09, 2010 1:32 am

Re: Budabot Items Extractor

Postby Tyrence » Tue Jul 07, 2015 2:30 am

I just uploaded Budabot Items Extractor 1.4 which has a few fixes as well as a few changes for AO 18.8.X.

https://github.com/Budabot/ItemsExtractor/releases
"Those who expect to reap the blessings of freedom, must, like men, undergo the fatigues of supporting it." — Thomas Paine
"Nearly all men stand adversity, but if you want to test a man's character, give him power." — Abraham Lincoln
Budabot Releases and Downloads: https://github.com/Budabot/Budabot/releases
Assx
Member
Posts: 11
Joined: Fri May 27, 2016 11:48 am
antispam: Rimor

Re: Budabot Items Extractor

Postby Assx » Sun Jan 22, 2017 5:05 pm

Hi Tyrence,

I have installed maven for windows and downloaded the itemextractor source files.
Shouldn't that be enough to build the release?

Instead I get errors like

Code: Select all

[ERROR] Failed to execute goal on project BudabotItemsExtractor: Could not resolve dependencies for project com.jkbff.ao:BudabotItemsExtractor:jar:1.4: Failure to find com.jkbff.common:scala-util:jar:1.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]


I tried several things like pointing the pom to a location of the scala-util jar which I got from ur 1.4 release bot I just can't make it work.

Could you maybe help to make a setup which allows me to successfully build a working release?
Tyrence
Posts: 1913
Joined: Sat Jan 09, 2010 1:32 am

Re: Budabot Items Extractor

Postby Tyrence » Mon Jan 23, 2017 11:27 am

Yeah, sorry, I meant to fix that but never got around to it.

As a work around, you need to also download this project: https://github.com/bigwheels16/scala-util

Then run

Code: Select all

mvn install
from the scala-util directory which should install the jar into your local maven repo. Then when you go to build the Budabot Items Extractor it should be able to find its dependencies.

It may also make you do that for the TyrLib jar which you can find here: https://github.com/bigwheels16/TyrLib
"Those who expect to reap the blessings of freedom, must, like men, undergo the fatigues of supporting it." — Thomas Paine
"Nearly all men stand adversity, but if you want to test a man's character, give him power." — Abraham Lincoln
Budabot Releases and Downloads: https://github.com/Budabot/Budabot/releases
Assx
Member
Posts: 11
Joined: Fri May 27, 2016 11:48 am
antispam: Rimor

Re: Budabot Items Extractor

Postby Assx » Wed Jan 25, 2017 7:37 am

Great, it's working now.

Any chance to extract all nano programs with that tool?
Tyrence
Posts: 1913
Joined: Sat Jan 09, 2010 1:32 am

Re: Budabot Items Extractor

Postby Tyrence » Wed Jan 25, 2017 6:38 pm

It could be modified to extract any data from the rdb files, but currently it doesn't. I thought about making it modular so that people could add plugins for it like Xyphos did with his tool, but I figured there wasn't enough people familiar with Scala who might want to do that to make it worth it.
"Those who expect to reap the blessings of freedom, must, like men, undergo the fatigues of supporting it." — Thomas Paine
"Nearly all men stand adversity, but if you want to test a man's character, give him power." — Abraham Lincoln
Budabot Releases and Downloads: https://github.com/Budabot/Budabot/releases
Assx
Member
Posts: 11
Joined: Fri May 27, 2016 11:48 am
antispam: Rimor

Re: Budabot Items Extractor

Postby Assx » Wed Jan 25, 2017 9:58 pm

It would be very great if you could help me with this.
Tyrence
Posts: 1913
Joined: Sat Jan 09, 2010 1:32 am

Re: Budabot Items Extractor

Postby Tyrence » Thu Jan 26, 2017 9:56 am

If you can figure out the format of the nano formula data in the rdb files, I can show you how to make the changes to the Budabot Items Extractor.
"Those who expect to reap the blessings of freedom, must, like men, undergo the fatigues of supporting it." — Thomas Paine
"Nearly all men stand adversity, but if you want to test a man's character, give him power." — Abraham Lincoln
Budabot Releases and Downloads: https://github.com/Budabot/Budabot/releases

Return to “Misc/Tools”

Who is online

Users browsing this forum: No registered users and 1 guest