Created database revision define, this is located in version.h in common #define CURRENT_BINARY_DATABASE_VERSION 9057
- This revision define will need to be incremented each time a database update is made
- Along with a revision define increment, you will need to update the db_update manifest located in:
- https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
- An entry needs to be made at the bottom of the manifest, the entry is quite simple
- Example: 9057|2014_11_13_spells_new_updates.sql|SHOW COLUMNS FROM `spells_new` LIKE 'disallow_sit'|empty|
- This latest example is checking to see if the spells_new table contains the column 'disallow_sit', if its empty, the update needs to be ran
- More examples of match types below:
# Example: Version|Filename.sql|Query_to_Check_Condition_For_Needed_Update|match type|text to match
# 0 = Database Version
# 1 = Filename.sql
# 2 = Query_to_Check_Condition_For_Needed_Update
# 3 = Match Type - If condition from match type to Value 4 is true, update will flag for needing to be ran
# contains = If query results contains text from 4th value
# match = If query results matches text from 4th value
# missing = If query result is missing text from 4th value
# empty = If the query results in no results
# not_empty = If the query is not empty
# 4 = Text to match
- The manifest contains all database updates 'Required' to be made to the schema, and it will contain a working backport all the way back to SVN -
currently it is tested and backported through the beginning of our Github repo
- On world bootup or standalone run of db_update.pl, users will be prompted with a simple menu that we will expand upon later:
============================================================
EQEmu: Automatic Database Upgrade Check
============================================================
Operating System is: MSWin32
(Windows) MySQL is in system path
Path = C:\Program Files\MariaDB 10.0\bin/mysql
============================================================
Binary Database Version: (9057)
Local Database Version: (9057)
Database up to Date: Continuing World Bootup...
============================================================
Retrieving latest database manifest...
URL: https://raw.githubusercontent.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
Saved: db_update/db_update_manifest.txt
Database Management Menu (Please Select):
1) Backup Database - (Saves to Backups folder)
Ideal to perform before performing updates
2) Backup Database Compressed - (Saves to Backups folder)
Ideal to perform before performing updates
3) Check for pending Database updates
Stages updates for automatic upgrade...
0) Exit
Created db_update.pl, placed in utils/scripts folder, used for the automatic database update routine (Linux/Windows)
- db_update.pl script created db_version table if not created, if old one is present it will remove it
Created db_dumper.pl, placed in utils/scripts folder, used for the automatic database update routine backups and standalone backups (Linux/Windows)
World will now check the db_update.pl script on bootup, if the db_update.pl script is not present, it will fetch it remotely before running -
when db_update.pl is done running, world will continue with bootup
world.exe db_version - will report database binary version
############################################################
#::: Script: DB_Dumper.pl
#::: Purpose: Utility to easily manage database backups and compress.
#::: Export Individual DB Tables...
#::: Export specific databases...
#::: Built for both Windows and Linux
#::: Windows uses WinRar or 7-Zip for compression
#::: Linux uses tar for compression
#::: Author: Akkadius
############################################################
Tribute uninitialized value database conversion fix
Tribute load fix (Lost in translation)
Changed Tribute 0 value to 0xFFFFFFFF instead of 0
Sanity check for unitialized tribute value on load
#resetaa will wipe all AA data, refund the spent points into the available points and send character to character select properly
Removed #refundaa
Removed a lot of debug code for blob conversion
Changed status logging for loads/saves to Debug category
At the end of conversion, character_ table will be renamed to character_old so that it does not get hit with trying to convert next world bootup
Added some file logging during the initial conversion routine
Corrected some character name reservation logic
Adjusted telnet console so it does not throw 'Command Unknown' when simply hitting enter key
Adjusted System MSG for worldshutdown to use minutes instead of seconds
Added warning for when Shared platinum is disabled at the rule level for players to NOT put platinum in the shared bank when they insert platinum in the shared platinum slot
Changed a place in the code where disciplines were trained, someone had uint16 allocated to CharacterID and this was causing issues for characters with large ID's in the database
Added a fix for bind points getting overwritten with invalid zone data, setting everything to 0,0,0,0 - Since I've added this change I've had no issues for players
Fixed some spell book swap logic in the code, removed swap function.
Fixed issue with guild ranks not loading properly
Commented out some of the non-working tell-que code
Took out some of the auto increment settings in the auto database conversion that don't affect anything anyways
Added some additional escape string sequences for queries that needed them
Added ThrowDBError logging to catch any potential query issues with saves or load functions
- Ported character leadership abilities to character_leadership_abilities
- Removed player profile debug printing
- Refactored total time entitled on account to load from the sum of time_played from all characters in character_data
- Need to convert a list of functions and columns and should be ready to start intensive testing phase
- All preliminary tests show things working great
- All of player profile is saved and loaded from the database
- DBAsync has been completely removed from all code
- Removed zone/dbasync.cpp/.h
- Removed common/dbasync.cpp/.h
- Removed dbasync from cmake commmon and zone
- Cleaned up a ton of functions
- Added several tables to world CheckDatabaseConversions script:
- `character_skills`
- `character_languages`
- `character_bind`
- `character_alternate_abilities`
- `character_currency`
- `character_data`
- `character_spells`
- `character_memmed_spells`
- `character_disciplines`
- `character_material`
- `character_tribute`
- `character_bandolier`
- `character_potionbelt`
- Character select now loads from `character_data`
- Character creation now creates to `character_data`
- Updated function Database::UpdateName to use `character_data`
- Updated function Database::CheckUsedName to use `character_data`
- Updated function Database::MoveCharacterToZone to use `character_data`
- Updated function Database::SetLoginFlags to use `character_data`
- Updated function Database::SetFirstLogon to use `character_data`
- Updated function Database::SetLFG to use `character_data`
- Removed CopyCharacter functions and commands, to be recreated later since it never worked to begin with
- Removed SharedDatabase::SetPlayerProfile
- Trimmed down redundant case switch statements for World sendpackets to QueryServ
- Added Character Methods to Database class:
Loads:
bool LoadCharacterBandolier(uint32 character_id, PlayerProfile_Struct* pp);
bool LoadCharacterTribute(uint32 character_id, PlayerProfile_Struct* pp);
bool LoadCharacterPotions(uint32 character_id, PlayerProfile_Struct* pp);
Saves:
bool SaveCharacterBindPoint(uint32 character_id, uint32 zone_id, uint32 instance_id, float x, float y, float z, float heading, uint8 is_home);
bool SaveCharacterCurrency(uint32 character_id, PlayerProfile_Struct* pp);
bool SaveCharacterData(uint32 character_id, uint32 account_id, PlayerProfile_Struct* pp);
bool SaveCharacterAA(uint32 character_id, uint32 aa_id, uint32 current_level);
bool SaveCharacterSpellSwap(uint32 character_id, uint32 spell_id, uint32 from_slot, uint32 to_slot);
bool SaveCharacterSpell(uint32 character_id, uint32 spell_id, uint32 slot_id);
bool SaveCharacterMemorizedSpell(uint32 character_id, uint32 spell_id, uint32 slot_id);
bool SaveCharacterMaterialColor(uint32 character_id, uint32 slot_id, uint32 color);
bool SaveCharacterSkill(uint32 character_id, uint32 skill_id, uint32 value);
bool SaveCharacterLanguage(uint32 character_id, uint32 lang_id, uint32 value);
bool SaveCharacterDisc(uint32 character_id, uint32 slot_id, uint32 disc_id);
bool SaveCharacterTribute(uint32 character_id, PlayerProfile_Struct* pp);
bool SaveCharacterBandolier(uint32 character_id, uint8 bandolier_id, uint8 bandolier_slot, uint32 item_id, uint32 icon, const char* bandolier_name);
bool SaveCharacterPotionBelt(uint32 character_id, uint8 potion_id, uint32 item_id, uint32 icon);
Deletes:
bool DeleteCharacterSpell(uint32 character_id, uint32 spell_id, uint32 slot_id);
bool DeleteCharacterMemorizedSpell(uint32 character_id, uint32 spell_id, uint32 slot_id);
bool DeleteCharacterDisc(uint32 character_id, uint32 slot_id);
bool DeleteCharacterBandolier(uint32 character_id, uint32 band_id);
Basic character data, currency and AA are being loaded/saved from the database, currently working on the rest right now.
- Character blob removed from load for testing. Lots of cleanup yet to be done so don't judge code yet.
Saves:
- Two FULL saves when looting a corpse, this has been reduced to just currency saves on initial loot and trimmed to one save since AddToMoneyPP did it already
- Every time a player moves coin with any situation (Splits/Trades/Merchant/Skills/Bank Coin Exchange/Coin Moves), a full save is made, this is now just a currency save
- Every time a player skilled up at a skill vendor, a full blob save hit was made, this is not just a currency hit
- A 2.4GB character_ table will take 7 seconds to query on a SSD versus .1s on the character_lookup table
- This also causes applications like Magelo to burst reads of the entire character table because of the blob fields that come with the reads, as much as 500-600MB/s even if a indexed id filter is provided
- This field is synchronized on player save and has 0.001s DB hit
- When we split out from the blob, ideally this table can be removed, but it really does no harm in mirroring data when a 2.6GB character table mirrors everything subtracting blob data down to 8MB
- Required SQL: utils\sql\git\required\2014_08_24_character_lookup.sql
Most of the following changes are QueryServ related, fully implemented its original functionality to be able to offload
intensive or metric based logging to a remote server process that could exist on another server entirely
Implemented Player Event Logging Types (Go to table `qs_player_events`):
1 = Player_Log_Quest,
2 = Player_Log_Zoning,
3 = Player_Log_Deaths,
4 = Player_Log_Connect_State,
5 = Player_Log_Levels,
6 = Player_Log_Keyring_Addition,
7 = Player_Log_QGlobal_Update,
8 = Player_Log_Task_Updates,
9 = Player_Log_AA_Purchases,
10 = Player_Log_Trade_Skill_Events,
11 = Player_Log_Issued_Commands,
12 = Player_Log_Money_Transactions,
13 = Player_Log_Alternate_Currency_Transactions,
- All QueryServ logging will be implemented with a front end in EoC 2.0 very soon
Changed all QS Error related logging to 'QUERYSERV__ERROR'
(Natedog) (Crash Fix) Legacy MySQL bug revert for loading AA's COALESCE( from COALESCE (
Implemented Perl Quest objects (LUA still needed to be exported):
- quest::qs_send_query("MySQL query") - Will send a raw query to the QueryServ process, useful for custom logging
- quest::qs_player_event(char_id, event_desc); - Will process a quest type event to table `qs_player_events`
Added MySQL Tables:
- `qs_player_aa_rate_hourly`
- `qs_player_events`
- Source table structures from:
- utils\sql\git\queryserv\required\08_23_2014_player_events_and_player_aa_rate_hourly
To get the complete QueryServ schema, source from here:
- utils\sql\git\queryserv\required\Complete_QueryServ_Table_Structures.sql
Added rules for each logging type, source rules here with them enabled by default:
- utils\sql\git\queryserv\required\Complete_QueryServ_Rules_Enabled.sql
Spawn related logging cleanup
General code cleanup
Added queryserv.cpp and queryserv.h with QueryServ class
- You need to have rule 'Chat:SuppressCommandErrors' set to true, this is set to false by default
- Required SQL: 2014_04_18_Suppress_Command_Error.sql
for custom servers that have balanced their content around having more than 1 aug proc on weapons. By having this rule set to 'false' you revert this functionality.
This rule is set to 'true' by default as the original functionality from Live was intended to be
see that the query 'SELECT varname, value, unix_timestamp() FROM variables where unix_timestamp(ts) >= timestamp' is called every 10 seconds. This function is loading
variables that are initially loaded on World and Zone bootup. When running a large amount of zone servers, the amount of MySQL chatter that is produced is enormous and
unnecessary. For example, if I ran 400 zone servers, I would see 3,456,000 unnecessary queries from all idle or active zone processes in a 24 hour interval.
Akkadius