Implemented Automatic Database update and versioning system

Created database revision define, this is located in version.h in common #define CURRENT_BINARY_DATABASE_VERSION 9057
	- This revision define will need to be incremented each time a database update is made
	- Along with a revision define increment, you will need to update the db_update manifest located in:
		- https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
		- An entry needs to be made at the bottom of the manifest, the entry is quite simple
		- Example: 9057|2014_11_13_spells_new_updates.sql|SHOW COLUMNS FROM `spells_new` LIKE 'disallow_sit'|empty|
			- This latest example is checking to see if the spells_new table contains the column 'disallow_sit', if its empty, the update needs to be ran
				- More examples of match types below:
		# Example: Version|Filename.sql|Query_to_Check_Condition_For_Needed_Update|match type|text to match
		#	0 = Database Version
		#	1 = Filename.sql
		#	2 = Query_to_Check_Condition_For_Needed_Update
		#	3 = Match Type - If condition from match type to Value 4 is true, update will flag for needing to be ran
		#		contains = If query results contains text from 4th value
		#		match = If query results matches text from 4th value
		#		missing = If query result is missing text from 4th value
		#		empty = If the query results in no results
		#		not_empty = If the query is not empty
		#	4 = Text to match
	- The manifest contains all database updates 'Required' to be made to the schema, and it will contain a working backport all the way back to SVN -
		currently it is tested and backported through the beginning of our Github repo
	- On world bootup or standalone run of db_update.pl, users will be prompted with a simple menu that we will expand upon later:

		============================================================
			   EQEmu: Automatic Database Upgrade Check
		============================================================
				Operating System is: MSWin32
				(Windows) MySQL is in system path
				Path = C:\Program Files\MariaDB 10.0\bin/mysql
		============================================================
				Binary Database Version: (9057)
				Local Database Version: (9057)

				Database up to Date: Continuing World Bootup...
		============================================================
		Retrieving latest database manifest...
        URL:    https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
        Saved:  db_update/db_update_manifest.txt

Database Management Menu (Please Select):
        1) Backup Database - (Saves to Backups folder)
                Ideal to perform before performing updates
        2) Backup Database Compressed - (Saves to Backups folder)
                Ideal to perform before performing updates
        3) Check for pending Database updates
                Stages updates for automatic upgrade...
        0) Exit

Created db_update.pl, placed in utils/scripts folder, used for the automatic database update routine (Linux/Windows)
	- db_update.pl script created db_version table if not created, if old one is present it will remove it
Created db_dumper.pl, placed in utils/scripts folder, used for the automatic database update routine backups and standalone backups (Linux/Windows)
World will now check the db_update.pl script on bootup, if the db_update.pl script is not present, it will fetch it remotely before running -
	when db_update.pl is done running, world will continue with bootup
world.exe db_version - will report database binary version
This commit is contained in:
Akkadius 2014-11-16 19:26:59 -06:00
parent 9324bdf107
commit 251d4fa3e3
6 changed files with 169 additions and 86 deletions

View File

@ -2,6 +2,60 @@ EQEMu Changelog (Started on Sept 24, 2003 15:50)
-------------------------------------------------------
== 11/16/2014 ==
demonstar55: fix size issue with ControlBoat_Struct and exploit fix in OP_BoardBoat
Akkadius: Implemented Automatic Database update and versioning system
Akkadius: Created database revision define, this is located in version.h in common #define CURRENT_BINARY_DATABASE_VERSION 9057
- This revision define will need to be incremented each time a database update is made
- Along with a revision define increment, you will need to update the db_update manifest located in:
- https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
- An entry needs to be made at the bottom of the manifest, the entry is quite simple
- Example: 9057|2014_11_13_spells_new_updates.sql|SHOW COLUMNS FROM `spells_new` LIKE 'disallow_sit'|empty|
- This latest example is checking to see if the spells_new table contains the column 'disallow_sit', if its empty, the update needs to be ran
- More examples of match types below:
# Example: Version|Filename.sql|Query_to_Check_Condition_For_Needed_Update|match type|text to match
# 0 = Database Version
# 1 = Filename.sql
# 2 = Query_to_Check_Condition_For_Needed_Update
# 3 = Match Type - If condition from match type to Value 4 is true, update will flag for needing to be ran
# contains = If query results contains text from 4th value
# match = If query results matches text from 4th value
# missing = If query result is missing text from 4th value
# empty = If the query results in no results
# not_empty = If the query is not empty
# 4 = Text to match
- The manifest contains all database updates 'Required' to be made to the schema, and it will contain a working backport all the way back to SVN -
currently it is tested and backported through the beginning of our Github repo
- On world bootup or standalone run of db_update.pl, users will be prompted with a simple menu that we will expand upon later:
============================================================
EQEmu: Automatic Database Upgrade Check
============================================================
Operating System is: MSWin32
(Windows) MySQL is in system path
Path = C:\Program Files\MariaDB 10.0\bin/mysql
============================================================
Binary Database Version: (9057)
Local Database Version: (9057)
Database up to Date: Continuing World Bootup...
============================================================
Retrieving latest database manifest...
URL: https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
Saved: db_update/db_update_manifest.txt
Database Management Menu (Please Select):
1) Backup Database - (Saves to Backups folder)
Ideal to perform before performing updates
2) Backup Database Compressed - (Saves to Backups folder)
Ideal to perform before performing updates
3) Check for pending Database updates
Stages updates for automatic upgrade...
0) Exit
Akkadius: Created db_update.pl, placed in utils/scripts folder, used for the automatic database update routine (Linux/Windows)
- db_update.pl script created db_version table if not created, if old one is present it will remove it
Akkadius: Created db_dumper.pl, placed in utils/scripts folder, used for the automatic database update routine backups and standalone backups (Linux/Windows)
Akkadius: World will now check the db_update.pl script on bootup, if the db_update.pl script is not present, it will fetch it remotely before running -
when db_update.pl is done running, world will continue with bootup
== 11/15/2014 ==
Uleat(Natedog): A better fix for OP_ShopPlayerBuy - doesn't cause the issues that I introduced

View File

@ -2084,6 +2084,18 @@ bool Database::CheckDatabaseConversions() {
#endif
/* Fetch Automatic Database Upgrade Script */
if (!std::ifstream("db_update.pl")){
std::cout << "Pulling down automatic database upgrade script...\n" << std::endl;
#ifdef _WIN32
system("perl -MLWP::UserAgent -e \"require LWP::UserAgent; my $ua = LWP::UserAgent->new; $ua->timeout(10); $ua->env_proxy; my $response = $ua->get('https://raw.githubusercontent.com/EQEmu/Server/master/utils/scripts/db_update.pl'); if ($response->is_success){ open(FILE, '> db_update.pl'); print FILE $response->decoded_content; close(FILE); }\"");
#else
system("wget -O db_update.pl https://raw.githubusercontent.com/EQEmu/Server/master/utils/scripts/db_update.pl");
#endif
}
/* Run Automatic Database Upgrade Script */
system("perl db_update.pl ran_from_world");
return true;
}

View File

@ -23,6 +23,14 @@
#define EQEMU_PROTOCOL_VERSION "0.3.10"
#define CURRENT_VERSION "1.0.0"
/*
Everytime a Database SQL is added to Github,
increment CURRENT_BINARY_DATABASE_VERSION number and make sure you update the manifest
Manifest: https://github.com/EQEmu/Server/blob/master/utils/sql/db_update_manifest.txt
*/
#define CURRENT_BINARY_DATABASE_VERSION 9057
#define COMPILE_DATE __DATE__
#define COMPILE_TIME __TIME__
#ifndef WIN32

View File

@ -105,8 +105,8 @@ if($bin_db_ver == $local_db_ver && $ARGV[0] eq "ran_from_world"){
}
print "Retrieving latest database manifest...\n";
#GetRemoteFile("https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt", "db_update/db_update_manifest.txt");
GetRemoteFile("https://dl.dropboxusercontent.com/u/50023467/dl/db_update_manifest.txt", "db_update/db_update_manifest.txt");
GetRemoteFile("https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt", "db_update/db_update_manifest.txt");
# GetRemoteFile("https://dl.dropboxusercontent.com/u/50023467/dl/db_update_manifest.txt", "db_update/db_update_manifest.txt");
if($local_db_ver < $bin_db_ver && $ARGV[0] eq "ran_from_world"){
print "You have missing database updates, type 1 or 2 to backup your database before running them as recommended...\n\n";

View File

@ -1,21 +1,21 @@
5001|1_task_system.sql
5002|2_optional_maxclients.sql
5003|14_optional_merchantlist.sql
# 5002|2_optional_maxclients.sql
# 5003|14_optional_merchantlist.sql
5004|35_task_stepped.sql
5005|42_task_min_maxlevel.sql
5006|55_zone_shutdowndeleay.sql
5007|68_optional_character_maxexplevel.sql
5008|103_optional_chat_rules.sql
# 5007|68_optional_character_maxexplevel.sql
# 5008|103_optional_chat_rules.sql
5009|104_traps.sql
5010|106_optional_proc_rules.sql
# 5010|106_optional_proc_rules.sql
5011|120_damageshieldtypes.sql
5012|125_aggrozone.sql
5013|127_optional_spell_rules.sql
5014|129_optional_shared_plat_rule.sql
5015|131_optional_combat_rules.sql
# 5013|127_optional_spell_rules.sql
# 5014|129_optional_shared_plat_rule.sql
# 5015|131_optional_combat_rules.sql
5016|133_task_repeatable.sql
5017|142_deathpeace_and_lifetap_aas.sql
5018|158_optional_death_exp_loss.sql
# 5018|158_optional_death_exp_loss.sql
5019|176_melody.sql
5020|189_character_.sql
5021|196_trader.sql
@ -28,20 +28,20 @@
5028|247_mail.sql
5029|249_chatchannels.sql
5030|250_bot_spell_update.sql
5031|250_optional_bot_spell_update.sql
5032|285_optional_bot_spell_update.sql
# 5031|250_optional_bot_spell_update.sql
# 5032|285_optional_bot_spell_update.sql
5033|292_augslots.sql
5034|294_merchant_logging.sql
5035|304_faction_list.sql
5036|326_aas.sql
5037|328_bot_management.sql
5038|328_optional_bot_management.sql
# 5038|328_optional_bot_management.sql
5039|340_gm_ips.sql
5040|356_combat.sql
5041|360_peqzone.sql
5042|364_ranged_dist_rule.sql
5043|386_bot_save_raid.sql
5044|434_optional_rest_state_rules.sql
# 5044|434_optional_rest_state_rules.sql
5045|447_sof_startzone_rule.sql
5046|463_altadv_vars.sql
5047|475_aa_actions.sql
@ -77,12 +77,12 @@
5077|754_archery_base_damage_rule.sql
5078|755_sof_altadv_vars_updates.sql
5079|773_monk_rules.sql
5080|853_optional_rule_aaexp.sql
5081|858_optional_rule_ip_limit_by_status.sql
5082|892_optional_bots_table_mod.sql
5083|893_optional_bots_table_mod.sql
# 5080|853_optional_rule_aaexp.sql
# 5081|858_optional_rule_ip_limit_by_status.sql
# 5082|892_optional_bots_table_mod.sql
# 5083|893_optional_bots_table_mod.sql
5084|898_npc_maxlevel_scalerate.sql
5085|902_optional_rule_snareflee.sql
# 5085|902_optional_rule_snareflee.sql
5086|923_spawn2_enabled.sql
5087|962_hot_zone.sql
5088|964_reports.sql
@ -101,17 +101,17 @@
5101|1057_titles.sql
5102|1077_botgroups.sql
5103|1136_spell_globals.sql
5104|1144_optional_rule_return_nodrop.sql
# 5104|1144_optional_rule_return_nodrop.sql
5105|1195_account_suspendeduntil.sql
5106|1259_npc_skill_types.sql
5107|1280_bot_augs.sql
5108|1290_optional_exp_loss_rule.sql
# 5108|1290_optional_exp_loss_rule.sql
5109|1293_guild_bank.sql
5110|1379_loginserver_trusted_server.sql
5111|1392_recipe_learning.sql
5112|1394_optional_rule_sod_hp_mana_end.sql
# 5112|1394_optional_rule_sod_hp_mana_end.sql
5113|1404_faction_list.sql
5114|1410_optional_sod_aas_ht_and_loh.sql
# 5114|1410_optional_sod_aas_ht_and_loh.sql
5115|1436_login_server_table_fix.sql
5116|1446_allowrest_optional.sql
5117|1446_allowrest_required.sql
@ -128,66 +128,66 @@
5128|1586_waypoints_optional.sql
5129|1610_tradeskill_required.sql
5130|1618_zone.sql
5131|1625_optional_rule_class_race_exp_bonus.sql
5132|1672_optional_rules_respawn_window.sql
5133|1679_optional_rules_blocked_buffs.sql
# 5131|1625_optional_rule_class_race_exp_bonus.sql
# 5132|1672_optional_rules_respawn_window.sql
# 5133|1679_optional_rules_blocked_buffs.sql
5134|1696_modify_zone_and_object_tables.sql
5135|1711_account_restricted_aa.sql
5136|1717_optional_rule_bash_stun_chance.sql
5137|1718_optional_rules_mod3s.sql
5138|1719_optional_triggerOnCastAAs.sql
5139|1720_optional_sql_AAs.sql
# 5136|1717_optional_rule_bash_stun_chance.sql
# 5137|1718_optional_rules_mod3s.sql
# 5138|1719_optional_triggerOnCastAAs.sql
# 5139|1720_optional_sql_AAs.sql
5140|1720_required_sql_AA_effects_update.sql
5141|1721_optional_sql_drakkin_breath_update.sql
# 5141|1721_optional_sql_drakkin_breath_update.sql
5142|1721_required_sql_altadv_vars_update.sql
5143|1723_optional_sql_new_stats_window_rule.sql
# 5143|1723_optional_sql_new_stats_window_rule.sql
5144|1723_required_sql_corruption.sql
5145|1736_optional_sql_feral_swipe.sql
# 5145|1736_optional_sql_feral_swipe.sql
5146|1737_required_sql_rule_and_aa_update.sql
5147|1746_optional_sql_bot_manaregen.sql
5148|1747_optional_HoT_zone_and_zonepoints.sql
5149|1750_optional_sql_reflect_rule.sql
5150|1753_optional_haste_cap_rule.sql
# 5147|1746_optional_sql_bot_manaregen.sql
# 5148|1747_optional_HoT_zone_and_zonepoints.sql
# 5149|1750_optional_sql_reflect_rule.sql
# 5150|1753_optional_haste_cap_rule.sql
5151|1753_required_sql_healing_adept_aa.sql
5152|1754_required_sql_healing_adept_aa_fix.sql
5153|1755_required_sql_fear_resist_aas.sql
5154|1784_optional_corpsedrag_rules.sql
# 5154|1784_optional_corpsedrag_rules.sql
5155|1786_required_update_to_aas.sql
5156|1790_required_aa_required_level_cost.sql
5157|1793_resist_adjust.sql
5158|1799_optional_rest_regen_endurance_rule.sql
# 5158|1799_optional_rest_regen_endurance_rule.sql
5159|1802_required_doppelganger.sql
5160|1803_required_tasks_xpreward_signed.sql
5161|1804_required_ae_melee_updates.sql
5162|1809_optional_rules.sql
# 5162|1809_optional_rules.sql
5163|1813_required_doppelganger_npcid_change.sql
5164|1817_optional_npc_archery_bonus_rule.sql
5165|1823_optional_delay_death.sql
# 5164|1817_optional_npc_archery_bonus_rule.sql
# 5165|1823_optional_delay_death.sql
5166|1847_required_doors_dest_zone_size_32.sql
5167|1859_optional_item_casts_use_focus_rule.sql
5168|1884_optional_bot_spells_update.sql
5169|1885_optional_rules_fv_pvp_expansions.sql
5170|1889_optional_skill_cap_rule.sql
# 5167|1859_optional_item_casts_use_focus_rule.sql
# 5168|1884_optional_bot_spells_update.sql
# 5169|1885_optional_rules_fv_pvp_expansions.sql
# 5170|1889_optional_skill_cap_rule.sql
5171|1908_required_npc_types_definitions.sql
5172|1926_optional_stat_cap.sql
# 5172|1926_optional_stat_cap.sql
5173|1944_spawn2.sql
5174|1946_doors.sql
5175|1960_optional_console_timeout_rule.sql
5176|1962_optional_guild_creation_window_rules.sql
5177|1963_optional_rule_live_like_focuses.sql
5178|1968_optional_enrage_rules.sql
5179|1972_optional_extradmg_item_cap.sql
# 5175|1960_optional_console_timeout_rule.sql
# 5176|1962_optional_guild_creation_window_rules.sql
# 5177|1963_optional_rule_live_like_focuses.sql
# 5178|1968_optional_enrage_rules.sql
# 5179|1972_optional_extradmg_item_cap.sql
5180|1974_required_bot_spells_update.sql
5181|1977_underwater.sql
5182|1998_optional_intoxication_and_looting_rules.sql
# 5182|1998_optional_intoxication_and_looting_rules.sql
5183|2004_charges_alt_currency.sql
5184|2015_optional_specialization_training_rule.sql
5185|2016_optional_rule_bot_aa_expansion.sql
5186|2023_optional_mysqlcli.sql
5187|2024_optional_update_crystals.sql
# 5184|2015_optional_specialization_training_rule.sql
# 5185|2016_optional_rule_bot_aa_expansion.sql
# 5186|2023_optional_mysqlcli.sql
# 5187|2024_optional_update_crystals.sql
5188|2024_required_update.sql
5189|2057_required_discovered_items.sql
5190|2058_optional_rule_discovered_items.sql
# 5190|2058_optional_rule_discovered_items.sql
5191|2062_required_version_changes.sql
5192|2069_required_pets.sql
5193|2079_player_speech.sql
@ -200,59 +200,59 @@
5200|2133_required_faction_loot_despawn.sql
5201|2136_extended_targets.sql
5202|2142_emotes.sql
5203|2154_optional_rule_spell_procs_resists_falloff.sql
5204|2156_optional_charm_break_rule.sql
5205|2159_optional_defensiveproc_rules.sql
# 5203|2154_optional_rule_spell_procs_resists_falloff.sql
# 5204|2156_optional_charm_break_rule.sql
# 5205|2159_optional_defensiveproc_rules.sql
5206|2164_require_bots_bottimers.sql
5207|2171_optional_SpecialAttackACBonus_rule.sql
5208|2176_optional_aa_expansion_SOF_fix.sql
5209|2176_optional_FrenzyBonus_rule.sql
# 5207|2171_optional_SpecialAttackACBonus_rule.sql
# 5208|2176_optional_aa_expansion_SOF_fix.sql
# 5209|2176_optional_FrenzyBonus_rule.sql
5210|2176_required_aa_updates.sql
5211|2178_required_aa_updates.sql
5212|2183_optional_bot_xp_rule.sql
5213|2185_optional_NPCFlurryChacne_rule
5214|2185_optional_NPCFlurryChacne_rule.sql
5215|2185_optional_NPCFlurryChance_rule.sql
# 5212|2183_optional_bot_xp_rule.sql
# 5213|2185_optional_NPCFlurryChacne_rule
# 5214|2185_optional_NPCFlurryChacne_rule.sql
# 5215|2185_optional_NPCFlurryChance_rule.sql
5216|2185_required_aa_updates
5217|2185_required_aa_updates.sql
5218|2188_optional_miscspelleffect_rules
5219|2188_optional_miscspelleffect_rules.sql
# 5218|2188_optional_miscspelleffect_rules
# 5219|2188_optional_miscspelleffect_rules.sql
5220|2188_required_aa_updates
5221|2188_required_aa_updates.sql
5222|2189_optional_taunt_rules
5223|2189_optional_taunt_rules.sql
# 5222|2189_optional_taunt_rules
# 5223|2189_optional_taunt_rules.sql
5224|2195_required_sharedplatupdates.sql
5225|2208_optional_aa_stacking_rule.sql
5226|2208_optional_EnableSoulAbrasionAA.sql
# 5225|2208_optional_aa_stacking_rule.sql
# 5226|2208_optional_EnableSoulAbrasionAA.sql
5227|2208_required_aa_updates.sql
5228|2209_optional_additive_bonus_rule.sql
# 5228|2209_optional_additive_bonus_rule.sql
5229|2213_loot_changes.sql
5230|2214_faction_list_mod.sql
5231|2215_required_aa_updates.sql
5232|2243_optional_char_max_level_rule.sql
# 5232|2243_optional_char_max_level_rule.sql
5233|2260_probability.sql
5234|2262_required_pet_discipline_update.sql
5235|2264_required_aa_updates.sql
5236|2268_QueryServ.sql
5237|2268_required_updates.sql
5238|2274_optional_rule_iplimitdisconnectall.sql
5239|2278_optional_rule_targetableswarmpet.sql
5240|2280_optional_rule_targetableswarmpet-rename.sql
# 5238|2274_optional_rule_iplimitdisconnectall.sql
# 5239|2278_optional_rule_targetableswarmpet.sql
# 5240|2280_optional_rule_targetableswarmpet-rename.sql
5241|2283_required_npc_changes.sql
5242|2299_required_inspectmessage_fields.sql
5243|2300_optional_loot_changes.sql
# 5243|2300_optional_loot_changes.sql
5244|2304_QueryServ.sql
5245|2340_required_maxbuffslotspet.sql
5246|2361_QueryServ.sql
5247|2361_required_qs_rule_values.sql
5248|2370_required_aa_updates.sql
5249|2376_required_aa_updates.sql
5250|2380_optional_merc_data.sql
5251|2380_optional_merc_merchant_npctypes_update.sql
5252|2380_optional_merc_rules.sql
# 5250|2380_optional_merc_data.sql
# 5251|2380_optional_merc_merchant_npctypes_update.sql
# 5252|2380_optional_merc_rules.sql
5253|2383_required_group_ismerc.sql
5254|2428_optional_levelbasedexpmods.sql
5255|2448_optional_stun_proc_aggro_rule.sql
# 5254|2428_optional_levelbasedexpmods.sql
# 5255|2448_optional_stun_proc_aggro_rule.sql
5256|2471_required_aa_updates.sql
5257|2482_required_start_zones.sql
5258|2504_required_aa_updates.sql

View File

@ -111,6 +111,15 @@ int main(int argc, char** argv) {
RegisterExecutablePlatform(ExePlatformWorld);
set_exception_handler();
/* Database Version Check */
uint32 Database_Version = CURRENT_BINARY_DATABASE_VERSION;
if (argc >= 2) {
if (strcasecmp(argv[1], "db_version") == 0) {
std::cout << "Binary Database Version: " << Database_Version << std::endl;
return 0;
}
}
// Load server configuration
_log(WORLD__INIT, "Loading server configuration..");
if (!WorldConfig::LoadConfig()) {