[Player Event Logs] Migrate and Deprecate QS Legacy Logging (#4542)

* First pass of player_event_loot_items

* Second pass of player_event_loot_items

* Third pass of player_event_loot_items

* Example without RecordDetailEvent template

* Cleanup the removal of the template

* Fourth Pass

Add retention for etl tables
Rename tables/fields to etl nomenclature
Combine database work to one atomic load

* Reposition to reduce db tasks

* Refactor etl processing for easier additions

* Add merchant purchase event
testing passed though appears that the event itself has a few bugs.  Will fix them in another commit

* Fix PlayerEventMerchantPurchase in client_packet.cpp

* WIP - Handin

* Handin Event added

* Cleanup

* All a rentention period of 0 days which deletes all current records.

* Updates

Cleanup and refactor a few items.

* Cleanup and Formatting

Cleanup and Formatting

* Add etl for
Playerevent::Trade
PlayerEvent::Speech (new event to mirror functionality of qs_speech

* Add etl for
Playerevent::KilledNPC, KilledNamedNPC and KilledRaidNPC

* Add etl for Playerevent::AA_purchase

Add etl for Playerevent::AA_purchase

* Cleanup before PR

* Review comment updates.

* Add world cli etl:settings to output a json on all player event details.

* Add reserve for all etl_queues
Correct a failed test case for improper next id for etl tables when table is first created.

* Potential solution for a dedicated database connection for player events.

* Simple thread for player_events.  Likely there is a better way to do this.

* Add zone to qs communications for recordplayerevents

First pass of enabling zone to qs direct transport to allow for PlayerEvents to bypass world.

* Cleanup a linux compile issue

* Add augments to LOOT ITEM and DESTROY ITEM

* Add augments to ITEMCREATION, FORAGESUCCESS, FISHSUCCESS, DESTROYITEM, LOOTITEM, DROPPEDITEM, TRADERPURCHASE, TRADERSELL, GUILDTRIBUTEDONATE and cleaned up the naming convention of augments

* Formatting fixes

* Swap out GetNextTableId

* Statically load counter

* Add counter.clear() since the counter is static

* Upload optional QS conversion scripts

* Remove all qs_tables and code referencing them

* Update database.cpp

* Simplify ProcessBatchQueue

* Simplify PorcessBatchQueue

* Simplify event truncation

* Build event truncation to bulk query by retention groups

* Post rebase

* Update player_events.h

* Fix build

* Update npc.cpp

* First pass of direct zone to qs sending for player events

* Remove keepalive logic

* Fix event ordering

* Cleanup

* Update player_event_logs.cpp

* Wipe event data after ETL processed

* Split up database connections, hot reload logs for QS

* Load rules from database vs qs_database

* Update player_event_logs.cpp

* Hot toggle queryserv connect

---------

Co-authored-by: Akkadius <akkadius1@gmail.com>

common/events/player_event_logs.h

@@ -1,19 +1,38 @@
 #ifndef EQEMU_PLAYER_EVENT_LOGS_H
 #define EQEMU_PLAYER_EVENT_LOGS_H
 
-#include "../repositories/player_event_log_settings_repository.h"
-#include "player_events.h"
-#include "../servertalk.h"
-#include "../repositories/player_event_logs_repository.h"
-#include "../timer.h"
-#include "../json/json_archive_single_line.h"
 #include <cereal/archives/json.hpp>
 #include <mutex>
+#include "../json/json_archive_single_line.h"
+#include "../servertalk.h"
+#include "../timer.h"
+#include "../eqemu_config.h"
+
+#include "../repositories/player_event_log_settings_repository.h"
+#include "../repositories/player_event_logs_repository.h"
+#include "../repositories/player_event_loot_items_repository.h"
+#include "../repositories/player_event_merchant_purchase_repository.h"
+#include "../repositories/player_event_merchant_sell_repository.h"
+#include "../repositories/player_event_npc_handin_repository.h"
+#include "../repositories/player_event_npc_handin_entries_repository.h"
+#include "../repositories/player_event_trade_repository.h"
+#include "../repositories/player_event_trade_entries_repository.h"
+#include "../repositories/player_event_speech_repository.h"
+#include "../repositories/player_event_killed_npc_repository.h"
+#include "../repositories/player_event_killed_named_npc_repository.h"
+#include "../repositories/player_event_killed_raid_npc_repository.h"
+#include "../repositories/player_event_aa_purchase_repository.h"
+
+
 
 class PlayerEventLogs {
 public:
+	Database player_event_database{};
+
 	void Init();
+	bool LoadDatabaseConnection();
 	void ReloadSettings();
+	void LoadEtlIds();
 	PlayerEventLogs *SetDatabase(Database *db);
 	bool ValidateDatabaseConnection();
 	bool IsEventEnabled(PlayerEvent::EventType event);
@@ -21,7 +40,7 @@ public:
 	void Process();
 
 	// batch queue
-	void AddToQueue(const PlayerEventLogsRepository::PlayerEventLogs &logs);
+	void AddToQueue(PlayerEventLogsRepository::PlayerEventLogs &logs);
 
 	// main event record generic function
 	// can ingest any struct event types
@@ -59,7 +78,29 @@ public:
 	std::string GetDiscordWebhookUrlFromEventType(int32_t event_type_id);
 
 	static std::string GetDiscordPayloadFromEvent(const PlayerEvent::PlayerEventContainer &e);
+
+	struct EtlQueues {
+		std::vector<PlayerEventLootItemsRepository::PlayerEventLootItems>               loot_items;
+		std::vector<PlayerEventMerchantPurchaseRepository::PlayerEventMerchantPurchase> merchant_purchase;
+		std::vector<PlayerEventMerchantSellRepository::PlayerEventMerchantSell>         merchant_sell;
+		std::vector<PlayerEventNpcHandinRepository::PlayerEventNpcHandin>               npc_handin;
+		std::vector<PlayerEventNpcHandinEntriesRepository::PlayerEventNpcHandinEntries> npc_handin_entries;
+		std::vector<PlayerEventTradeRepository::PlayerEventTrade>                       trade;
+		std::vector<PlayerEventTradeEntriesRepository::PlayerEventTradeEntries>         trade_entries;
+		std::vector<PlayerEventSpeechRepository::PlayerEventSpeech>                     speech;
+		std::vector<PlayerEventKilledNpcRepository::PlayerEventKilledNpc>               killed_npc;
+		std::vector<PlayerEventKilledNamedNpcRepository::PlayerEventKilledNamedNpc>     killed_named_npc;
+		std::vector<PlayerEventKilledRaidNpcRepository::PlayerEventKilledRaidNpc>       killed_raid_npc;
+		std::vector<PlayerEventAaPurchaseRepository::PlayerEventAaPurchase>             aa_purchase;
+	};
+
 private:
+	struct EtlSettings {
+		bool        enabled;
+		std::string table_name;
+		int64       next_id;
+	};
+
 	Database                                                 *m_database; // reference to database
 	PlayerEventLogSettingsRepository::PlayerEventLogSettings m_settings[PlayerEvent::EventType::MAX]{};
 
@@ -69,6 +110,8 @@ private:
 	static std::unique_ptr<ServerPacket>
 	BuildPlayerEventPacket(const PlayerEvent::PlayerEventContainer &e);
 
+	std::map<PlayerEvent::EventType, EtlSettings>  m_etl_settings{};
+
 	// timers
 	Timer m_process_batch_events_timer; // events processing timer
 	Timer m_process_retention_truncation_timer; // timer for truncating events based on retention settings
@@ -78,6 +121,9 @@ private:
 	void ProcessBatchQueue();
 	void ProcessRetentionTruncation();
 	void SetSettingsDefaults();
+
+public:
+	std::map<PlayerEvent::EventType, EtlSettings> &GetEtlSettings() { return m_etl_settings;}
 };
 
 extern PlayerEventLogs player_event_logs;