Implemented Automatic Database update and versioning system

Created database revision define, this is located in version.h in common #define CURRENT_BINARY_DATABASE_VERSION 9057
	- This revision define will need to be incremented each time a database update is made
	- Along with a revision define increment, you will need to update the db_update manifest located in:
		- https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
		- An entry needs to be made at the bottom of the manifest, the entry is quite simple
		- Example: 9057|2014_11_13_spells_new_updates.sql|SHOW COLUMNS FROM `spells_new` LIKE 'disallow_sit'|empty|
			- This latest example is checking to see if the spells_new table contains the column 'disallow_sit', if its empty, the update needs to be ran
				- More examples of match types below:
		# Example: Version|Filename.sql|Query_to_Check_Condition_For_Needed_Update|match type|text to match
		#	0 = Database Version
		#	1 = Filename.sql
		#	2 = Query_to_Check_Condition_For_Needed_Update
		#	3 = Match Type - If condition from match type to Value 4 is true, update will flag for needing to be ran
		#		contains = If query results contains text from 4th value
		#		match = If query results matches text from 4th value
		#		missing = If query result is missing text from 4th value
		#		empty = If the query results in no results
		#		not_empty = If the query is not empty
		#	4 = Text to match
	- The manifest contains all database updates 'Required' to be made to the schema, and it will contain a working backport all the way back to SVN -
		currently it is tested and backported through the beginning of our Github repo
	- On world bootup or standalone run of db_update.pl, users will be prompted with a simple menu that we will expand upon later:

		============================================================
			   EQEmu: Automatic Database Upgrade Check
		============================================================
				Operating System is: MSWin32
				(Windows) MySQL is in system path
				Path = C:\Program Files\MariaDB 10.0\bin/mysql
		============================================================
				Binary Database Version: (9057)
				Local Database Version: (9057)

				Database up to Date: Continuing World Bootup...
		============================================================
		Retrieving latest database manifest...
        URL:    https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
        Saved:  db_update/db_update_manifest.txt

Database Management Menu (Please Select):
        1) Backup Database - (Saves to Backups folder)
                Ideal to perform before performing updates
        2) Backup Database Compressed - (Saves to Backups folder)
                Ideal to perform before performing updates
        3) Check for pending Database updates
                Stages updates for automatic upgrade...
        0) Exit

Created db_update.pl, placed in utils/scripts folder, used for the automatic database update routine (Linux/Windows)
	- db_update.pl script created db_version table if not created, if old one is present it will remove it
Created db_dumper.pl, placed in utils/scripts folder, used for the automatic database update routine backups and standalone backups (Linux/Windows)
World will now check the db_update.pl script on bootup, if the db_update.pl script is not present, it will fetch it remotely before running -
	when db_update.pl is done running, world will continue with bootup
world.exe db_version - will report database binary version

changelog.txt

@@ -2,6 +2,60 @@ EQEMu Changelog (Started on Sept 24, 2003 15:50)
 -------------------------------------------------------
 == 11/16/2014 ==
 demonstar55: fix size issue with ControlBoat_Struct and exploit fix in OP_BoardBoat
+Akkadius: Implemented Automatic Database update and versioning system
+Akkadius: Created database revision define, this is located in version.h in common #define CURRENT_BINARY_DATABASE_VERSION 9057
+	- This revision define will need to be incremented each time a database update is made
+	- Along with a revision define increment, you will need to update the db_update manifest located in:
+		- https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
+		- An entry needs to be made at the bottom of the manifest, the entry is quite simple
+		- Example: 9057|2014_11_13_spells_new_updates.sql|SHOW COLUMNS FROM `spells_new` LIKE 'disallow_sit'|empty|
+			- This latest example is checking to see if the spells_new table contains the column 'disallow_sit', if its empty, the update needs to be ran
+				- More examples of match types below:
+		# Example: Version|Filename.sql|Query_to_Check_Condition_For_Needed_Update|match type|text to match
+		#	0 = Database Version
+		#	1 = Filename.sql
+		#	2 = Query_to_Check_Condition_For_Needed_Update
+		#	3 = Match Type - If condition from match type to Value 4 is true, update will flag for needing to be ran
+		#		contains = If query results contains text from 4th value
+		#		match = If query results matches text from 4th value
+		#		missing = If query result is missing text from 4th value
+		#		empty = If the query results in no results
+		#		not_empty = If the query is not empty
+		#	4 = Text to match
+	- The manifest contains all database updates 'Required' to be made to the schema, and it will contain a working backport all the way back to SVN - 
+		currently it is tested and backported through the beginning of our Github repo
+	- On world bootup or standalone run of db_update.pl, users will be prompted with a simple menu that we will expand upon later:
+	
+		============================================================
+			   EQEmu: Automatic Database Upgrade Check
+		============================================================
+				Operating System is: MSWin32
+				(Windows) MySQL is in system path
+				Path = C:\Program Files\MariaDB 10.0\bin/mysql
+		============================================================
+				Binary Database Version: (9057)
+				Local Database Version: (9057)
+
+				Database up to Date: Continuing World Bootup...
+		============================================================
+		Retrieving latest database manifest...
+        URL:    https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
+        Saved:  db_update/db_update_manifest.txt
+
+Database Management Menu (Please Select):
+        1) Backup Database - (Saves to Backups folder)
+                Ideal to perform before performing updates
+        2) Backup Database Compressed - (Saves to Backups folder)
+                Ideal to perform before performing updates
+        3) Check for pending Database updates
+                Stages updates for automatic upgrade...
+        0) Exit
+
+Akkadius: Created db_update.pl, placed in utils/scripts folder, used for the automatic database update routine (Linux/Windows)
+	- db_update.pl script created db_version table if not created, if old one is present it will remove it
+Akkadius: Created db_dumper.pl, placed in utils/scripts folder, used for the automatic database update routine backups and standalone backups (Linux/Windows)
+Akkadius: World will now check the db_update.pl script on bootup, if the db_update.pl script is not present, it will fetch it remotely before running - 
+	when db_update.pl is done running, world will continue with bootup
 
 == 11/15/2014 ==
 Uleat(Natedog): A better fix for OP_ShopPlayerBuy - doesn't cause the issues that I introduced

common/database.cpp

@@ -2084,6 +2084,18 @@ bool Database::CheckDatabaseConversions() {
 
 #endif
 
+	/* Fetch Automatic Database Upgrade Script */
+	if (!std::ifstream("db_update.pl")){
+		std::cout << "Pulling down automatic database upgrade script...\n" << std::endl;
+#ifdef _WIN32
+		system("perl -MLWP::UserAgent -e \"require LWP::UserAgent;  my $ua = LWP::UserAgent->new; $ua->timeout(10); $ua->env_proxy; my $response = $ua->get('https://raw.githubusercontent.com/EQEmu/Server/master/utils/scripts/db_update.pl'); if ($response->is_success){ open(FILE, '> db_update.pl'); print FILE $response->decoded_content; close(FILE); }\"");
+#else
+		system("wget -O db_update.pl https://raw.githubusercontent.com/EQEmu/Server/master/utils/scripts/db_update.pl");
+#endif
+	}
+	/* Run Automatic Database Upgrade Script */
+	system("perl db_update.pl ran_from_world"); 
+
 	return true;
 }
 

common/version.h

@@ -23,6 +23,14 @@
 #define EQEMU_PROTOCOL_VERSION "0.3.10"
 
 #define CURRENT_VERSION "1.0.0"
+
+/* 
+	Everytime a Database SQL is added to Github, 
+		increment CURRENT_BINARY_DATABASE_VERSION number and make sure you update the manifest
+	Manifest: https://github.com/EQEmu/Server/blob/master/utils/sql/db_update_manifest.txt	
+*/
+
+#define CURRENT_BINARY_DATABASE_VERSION 9057
 #define COMPILE_DATE	__DATE__
 #define COMPILE_TIME	__TIME__
 #ifndef WIN32

utils/scripts/db_update.pl

@@ -105,8 +105,8 @@ if($bin_db_ver == $local_db_ver && $ARGV[0] eq "ran_from_world"){
 }
 
 print "Retrieving latest database manifest...\n";
-#GetRemoteFile("https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt", "db_update/db_update_manifest.txt");
+GetRemoteFile("https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt", "db_update/db_update_manifest.txt");
-GetRemoteFile("https://dl.dropboxusercontent.com/u/50023467/dl/db_update_manifest.txt", "db_update/db_update_manifest.txt");
+# GetRemoteFile("https://dl.dropboxusercontent.com/u/50023467/dl/db_update_manifest.txt", "db_update/db_update_manifest.txt");
 
 if($local_db_ver < $bin_db_ver && $ARGV[0] eq "ran_from_world"){
 	print "You have missing database updates, type 1 or 2 to backup your database before running them as recommended...\n\n";

utils/sql/db_update_manifest.txt

@@ -1,21 +1,21 @@
 5001|1_task_system.sql
-5002|2_optional_maxclients.sql
+# 5002|2_optional_maxclients.sql
-5003|14_optional_merchantlist.sql
+# 5003|14_optional_merchantlist.sql
 5004|35_task_stepped.sql
 5005|42_task_min_maxlevel.sql
 5006|55_zone_shutdowndeleay.sql
-5007|68_optional_character_maxexplevel.sql
+# 5007|68_optional_character_maxexplevel.sql
-5008|103_optional_chat_rules.sql
+# 5008|103_optional_chat_rules.sql
 5009|104_traps.sql
-5010|106_optional_proc_rules.sql
+# 5010|106_optional_proc_rules.sql
 5011|120_damageshieldtypes.sql
 5012|125_aggrozone.sql
-5013|127_optional_spell_rules.sql
+# 5013|127_optional_spell_rules.sql
-5014|129_optional_shared_plat_rule.sql
+# 5014|129_optional_shared_plat_rule.sql
-5015|131_optional_combat_rules.sql
+# 5015|131_optional_combat_rules.sql
 5016|133_task_repeatable.sql
 5017|142_deathpeace_and_lifetap_aas.sql
-5018|158_optional_death_exp_loss.sql
+# 5018|158_optional_death_exp_loss.sql
 5019|176_melody.sql
 5020|189_character_.sql
 5021|196_trader.sql
@@ -28,20 +28,20 @@
 5028|247_mail.sql
 5029|249_chatchannels.sql
 5030|250_bot_spell_update.sql
-5031|250_optional_bot_spell_update.sql
+# 5031|250_optional_bot_spell_update.sql
-5032|285_optional_bot_spell_update.sql
+# 5032|285_optional_bot_spell_update.sql
 5033|292_augslots.sql
 5034|294_merchant_logging.sql
 5035|304_faction_list.sql
 5036|326_aas.sql
 5037|328_bot_management.sql
-5038|328_optional_bot_management.sql
+# 5038|328_optional_bot_management.sql
 5039|340_gm_ips.sql
 5040|356_combat.sql
 5041|360_peqzone.sql
 5042|364_ranged_dist_rule.sql
 5043|386_bot_save_raid.sql
-5044|434_optional_rest_state_rules.sql
+# 5044|434_optional_rest_state_rules.sql
 5045|447_sof_startzone_rule.sql
 5046|463_altadv_vars.sql
 5047|475_aa_actions.sql
@@ -77,12 +77,12 @@
 5077|754_archery_base_damage_rule.sql
 5078|755_sof_altadv_vars_updates.sql
 5079|773_monk_rules.sql
-5080|853_optional_rule_aaexp.sql
+# 5080|853_optional_rule_aaexp.sql
-5081|858_optional_rule_ip_limit_by_status.sql
+# 5081|858_optional_rule_ip_limit_by_status.sql
-5082|892_optional_bots_table_mod.sql
+# 5082|892_optional_bots_table_mod.sql
-5083|893_optional_bots_table_mod.sql
+# 5083|893_optional_bots_table_mod.sql
 5084|898_npc_maxlevel_scalerate.sql
-5085|902_optional_rule_snareflee.sql
+# 5085|902_optional_rule_snareflee.sql
 5086|923_spawn2_enabled.sql
 5087|962_hot_zone.sql
 5088|964_reports.sql
@@ -101,17 +101,17 @@
 5101|1057_titles.sql
 5102|1077_botgroups.sql
 5103|1136_spell_globals.sql
-5104|1144_optional_rule_return_nodrop.sql
+# 5104|1144_optional_rule_return_nodrop.sql
 5105|1195_account_suspendeduntil.sql
 5106|1259_npc_skill_types.sql
 5107|1280_bot_augs.sql
-5108|1290_optional_exp_loss_rule.sql
+# 5108|1290_optional_exp_loss_rule.sql
 5109|1293_guild_bank.sql
 5110|1379_loginserver_trusted_server.sql
 5111|1392_recipe_learning.sql
-5112|1394_optional_rule_sod_hp_mana_end.sql
+# 5112|1394_optional_rule_sod_hp_mana_end.sql
 5113|1404_faction_list.sql
-5114|1410_optional_sod_aas_ht_and_loh.sql
+# 5114|1410_optional_sod_aas_ht_and_loh.sql
 5115|1436_login_server_table_fix.sql
 5116|1446_allowrest_optional.sql
 5117|1446_allowrest_required.sql
@@ -128,66 +128,66 @@
 5128|1586_waypoints_optional.sql
 5129|1610_tradeskill_required.sql
 5130|1618_zone.sql
-5131|1625_optional_rule_class_race_exp_bonus.sql
+# 5131|1625_optional_rule_class_race_exp_bonus.sql
-5132|1672_optional_rules_respawn_window.sql
+# 5132|1672_optional_rules_respawn_window.sql
-5133|1679_optional_rules_blocked_buffs.sql
+# 5133|1679_optional_rules_blocked_buffs.sql
 5134|1696_modify_zone_and_object_tables.sql
 5135|1711_account_restricted_aa.sql
-5136|1717_optional_rule_bash_stun_chance.sql
+# 5136|1717_optional_rule_bash_stun_chance.sql
-5137|1718_optional_rules_mod3s.sql
+# 5137|1718_optional_rules_mod3s.sql
-5138|1719_optional_triggerOnCastAAs.sql
+# 5138|1719_optional_triggerOnCastAAs.sql
-5139|1720_optional_sql_AAs.sql
+# 5139|1720_optional_sql_AAs.sql
 5140|1720_required_sql_AA_effects_update.sql
-5141|1721_optional_sql_drakkin_breath_update.sql
+# 5141|1721_optional_sql_drakkin_breath_update.sql
 5142|1721_required_sql_altadv_vars_update.sql
-5143|1723_optional_sql_new_stats_window_rule.sql
+# 5143|1723_optional_sql_new_stats_window_rule.sql
 5144|1723_required_sql_corruption.sql
-5145|1736_optional_sql_feral_swipe.sql
+# 5145|1736_optional_sql_feral_swipe.sql
 5146|1737_required_sql_rule_and_aa_update.sql
-5147|1746_optional_sql_bot_manaregen.sql
+# 5147|1746_optional_sql_bot_manaregen.sql
-5148|1747_optional_HoT_zone_and_zonepoints.sql
+# 5148|1747_optional_HoT_zone_and_zonepoints.sql
-5149|1750_optional_sql_reflect_rule.sql
+# 5149|1750_optional_sql_reflect_rule.sql
-5150|1753_optional_haste_cap_rule.sql
+# 5150|1753_optional_haste_cap_rule.sql
 5151|1753_required_sql_healing_adept_aa.sql
 5152|1754_required_sql_healing_adept_aa_fix.sql
 5153|1755_required_sql_fear_resist_aas.sql
-5154|1784_optional_corpsedrag_rules.sql
+# 5154|1784_optional_corpsedrag_rules.sql
 5155|1786_required_update_to_aas.sql
 5156|1790_required_aa_required_level_cost.sql
 5157|1793_resist_adjust.sql
-5158|1799_optional_rest_regen_endurance_rule.sql
+# 5158|1799_optional_rest_regen_endurance_rule.sql
 5159|1802_required_doppelganger.sql
 5160|1803_required_tasks_xpreward_signed.sql
 5161|1804_required_ae_melee_updates.sql
-5162|1809_optional_rules.sql
+# 5162|1809_optional_rules.sql
 5163|1813_required_doppelganger_npcid_change.sql
-5164|1817_optional_npc_archery_bonus_rule.sql
+# 5164|1817_optional_npc_archery_bonus_rule.sql
-5165|1823_optional_delay_death.sql
+# 5165|1823_optional_delay_death.sql
 5166|1847_required_doors_dest_zone_size_32.sql
-5167|1859_optional_item_casts_use_focus_rule.sql
+# 5167|1859_optional_item_casts_use_focus_rule.sql
-5168|1884_optional_bot_spells_update.sql
+# 5168|1884_optional_bot_spells_update.sql
-5169|1885_optional_rules_fv_pvp_expansions.sql
+# 5169|1885_optional_rules_fv_pvp_expansions.sql
-5170|1889_optional_skill_cap_rule.sql
+# 5170|1889_optional_skill_cap_rule.sql
 5171|1908_required_npc_types_definitions.sql
-5172|1926_optional_stat_cap.sql
+# 5172|1926_optional_stat_cap.sql
 5173|1944_spawn2.sql
 5174|1946_doors.sql
-5175|1960_optional_console_timeout_rule.sql
+# 5175|1960_optional_console_timeout_rule.sql
-5176|1962_optional_guild_creation_window_rules.sql
+# 5176|1962_optional_guild_creation_window_rules.sql
-5177|1963_optional_rule_live_like_focuses.sql
+# 5177|1963_optional_rule_live_like_focuses.sql
-5178|1968_optional_enrage_rules.sql
+# 5178|1968_optional_enrage_rules.sql
-5179|1972_optional_extradmg_item_cap.sql
+# 5179|1972_optional_extradmg_item_cap.sql
 5180|1974_required_bot_spells_update.sql
 5181|1977_underwater.sql
-5182|1998_optional_intoxication_and_looting_rules.sql
+# 5182|1998_optional_intoxication_and_looting_rules.sql
 5183|2004_charges_alt_currency.sql
-5184|2015_optional_specialization_training_rule.sql
+# 5184|2015_optional_specialization_training_rule.sql
-5185|2016_optional_rule_bot_aa_expansion.sql
+# 5185|2016_optional_rule_bot_aa_expansion.sql
-5186|2023_optional_mysqlcli.sql
+# 5186|2023_optional_mysqlcli.sql
-5187|2024_optional_update_crystals.sql
+# 5187|2024_optional_update_crystals.sql
 5188|2024_required_update.sql
 5189|2057_required_discovered_items.sql
-5190|2058_optional_rule_discovered_items.sql
+# 5190|2058_optional_rule_discovered_items.sql
 5191|2062_required_version_changes.sql
 5192|2069_required_pets.sql
 5193|2079_player_speech.sql
@@ -200,59 +200,59 @@
 5200|2133_required_faction_loot_despawn.sql
 5201|2136_extended_targets.sql
 5202|2142_emotes.sql
-5203|2154_optional_rule_spell_procs_resists_falloff.sql
+# 5203|2154_optional_rule_spell_procs_resists_falloff.sql
-5204|2156_optional_charm_break_rule.sql
+# 5204|2156_optional_charm_break_rule.sql
-5205|2159_optional_defensiveproc_rules.sql
+# 5205|2159_optional_defensiveproc_rules.sql
 5206|2164_require_bots_bottimers.sql
-5207|2171_optional_SpecialAttackACBonus_rule.sql
+# 5207|2171_optional_SpecialAttackACBonus_rule.sql
-5208|2176_optional_aa_expansion_SOF_fix.sql
+# 5208|2176_optional_aa_expansion_SOF_fix.sql
-5209|2176_optional_FrenzyBonus_rule.sql
+# 5209|2176_optional_FrenzyBonus_rule.sql
 5210|2176_required_aa_updates.sql
 5211|2178_required_aa_updates.sql
-5212|2183_optional_bot_xp_rule.sql
+# 5212|2183_optional_bot_xp_rule.sql
-5213|2185_optional_NPCFlurryChacne_rule
+# 5213|2185_optional_NPCFlurryChacne_rule
-5214|2185_optional_NPCFlurryChacne_rule.sql
+# 5214|2185_optional_NPCFlurryChacne_rule.sql
-5215|2185_optional_NPCFlurryChance_rule.sql
+# 5215|2185_optional_NPCFlurryChance_rule.sql
 5216|2185_required_aa_updates
 5217|2185_required_aa_updates.sql
-5218|2188_optional_miscspelleffect_rules
+# 5218|2188_optional_miscspelleffect_rules
-5219|2188_optional_miscspelleffect_rules.sql
+# 5219|2188_optional_miscspelleffect_rules.sql
 5220|2188_required_aa_updates
 5221|2188_required_aa_updates.sql
-5222|2189_optional_taunt_rules
+# 5222|2189_optional_taunt_rules
-5223|2189_optional_taunt_rules.sql
+# 5223|2189_optional_taunt_rules.sql
 5224|2195_required_sharedplatupdates.sql
-5225|2208_optional_aa_stacking_rule.sql
+# 5225|2208_optional_aa_stacking_rule.sql
-5226|2208_optional_EnableSoulAbrasionAA.sql
+# 5226|2208_optional_EnableSoulAbrasionAA.sql
 5227|2208_required_aa_updates.sql
-5228|2209_optional_additive_bonus_rule.sql
+# 5228|2209_optional_additive_bonus_rule.sql
 5229|2213_loot_changes.sql
 5230|2214_faction_list_mod.sql
 5231|2215_required_aa_updates.sql
-5232|2243_optional_char_max_level_rule.sql
+# 5232|2243_optional_char_max_level_rule.sql
 5233|2260_probability.sql
 5234|2262_required_pet_discipline_update.sql
 5235|2264_required_aa_updates.sql
 5236|2268_QueryServ.sql
 5237|2268_required_updates.sql
-5238|2274_optional_rule_iplimitdisconnectall.sql
+# 5238|2274_optional_rule_iplimitdisconnectall.sql
-5239|2278_optional_rule_targetableswarmpet.sql
+# 5239|2278_optional_rule_targetableswarmpet.sql
-5240|2280_optional_rule_targetableswarmpet-rename.sql
+# 5240|2280_optional_rule_targetableswarmpet-rename.sql
 5241|2283_required_npc_changes.sql
 5242|2299_required_inspectmessage_fields.sql
-5243|2300_optional_loot_changes.sql
+# 5243|2300_optional_loot_changes.sql
 5244|2304_QueryServ.sql
 5245|2340_required_maxbuffslotspet.sql
 5246|2361_QueryServ.sql
 5247|2361_required_qs_rule_values.sql
 5248|2370_required_aa_updates.sql
 5249|2376_required_aa_updates.sql
-5250|2380_optional_merc_data.sql
+# 5250|2380_optional_merc_data.sql
-5251|2380_optional_merc_merchant_npctypes_update.sql
+# 5251|2380_optional_merc_merchant_npctypes_update.sql
-5252|2380_optional_merc_rules.sql
+# 5252|2380_optional_merc_rules.sql
 5253|2383_required_group_ismerc.sql
-5254|2428_optional_levelbasedexpmods.sql
+# 5254|2428_optional_levelbasedexpmods.sql
-5255|2448_optional_stun_proc_aggro_rule.sql
+# 5255|2448_optional_stun_proc_aggro_rule.sql
 5256|2471_required_aa_updates.sql
 5257|2482_required_start_zones.sql
 5258|2504_required_aa_updates.sql

world/net.cpp

@@ -111,6 +111,15 @@ int main(int argc, char** argv) {
 	RegisterExecutablePlatform(ExePlatformWorld);
 	set_exception_handler();
 
+	/* Database Version Check */
+	uint32 Database_Version = CURRENT_BINARY_DATABASE_VERSION;
+	if (argc >= 2) { 
+		if (strcasecmp(argv[1], "db_version") == 0) {
+			std::cout << "Binary Database Version: " << Database_Version << std::endl;
+			return 0;
+		}
+	}
+
 	// Load server configuration
 	_log(WORLD__INIT, "Loading server configuration..");
 	if (!WorldConfig::LoadConfig()) {