Build Improvements (#5033)
Build / Linux (push) Waiting to run
Build / Windows (push) Waiting to run

* Start rewrite, add vcpkg

* Simple vcpkg manifest, will almost certainly need tweaking

* Remove cmake ext we wont be using anymore

* Update vcpkg to no longer be from 2022, update cmake lists (wip)

* Add finds to the toplevel cmakelists

* WIP, luabind and perlbind build.  Common only partially builds.

* Fix common build.

* shared_memory compiles

* client files compile

* Tests and more cmake version updates

* World, had to swap out zlib-ng for now because it wasn't playing nicely along side the zlib install.  May revisit.

* UCS compiles now too!

* queryserv and eqlaunch

* loginserver works

* Zone works but is messy, tomorrow futher cleanup!

* Cleanup main file

* remove old zlibng, remove perlwrap, remove hc

* More cleanup

* vcpkg baseline set for CI

* Remove pkg-config, it's the suggested way to use luajit with vcpkg but it causes issues with CI and might be a pain point for windows users

* Actually add file

* Set perlbind include dir

* Perl link got lost

* PERL_SET_INTERP causes an issue on newer versions of perl on windows because a symbol is not properly exported in their API, change the lines so it's basically what it used to be

* Remove static unix linking, we dont do automated released anymore and this was tightly coupled to that.  Can explore this again if we decide to change that.

* Remove unused submodules, set cmake policy for boost

* Fix some cereal includes

* Improve some boilerplate, I'd still like to do better about getting linker stuff set.

* Going through and cleaning up the build.

* Fix world, separate out data_buckets.

* add fixes for other servers

* fix zone

* Fix client files, loginserver and tests

* Newer versions of libmariadb default to tls forced on, return to the default of not forcing that.
auto_login were breaking on linux builds
loginserver wasn't setting proper openssl compile flag

* Move set out of a giant cpp file include.

* Convert show

* convert find

* Add uuid to unix builds

* Remove some cpp includes.

* Restructure to remove more things.

* change db update manifest to header
change build yml

* Move world CLI include cpps to cmake.

* Move zone cli out of source and into cmake

* Sidecar stuff wont directly include cpp files now too.

* Fix uuid-dev missing on linux runner

* Reorg common cmake file

* Some cleanup

* Fix libsodium support (oops). Fix perl support (more oops)

* Change doc

---------

Co-authored-by: KimLS <KimLS@peqtgc.com>
This commit is contained in:
Alex
2025-12-13 19:56:37 -08:00
committed by GitHub
parent 9b3f9f356d
commit c84df0d5ba
510 changed files with 1505 additions and 47046 deletions
+319 -349
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -16,6 +16,7 @@
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/
#include <fmt/format.h>
#include <fmt/ranges.h>
#include "../common/global_define.h"
#include "../common/classes.h"
#include "data_verification.h"
+1
View File
@@ -19,6 +19,7 @@
*/
#include <fmt/format.h>
#include <fmt/ranges.h>
#include "eqemu_command_handler.h"
#include "terminal_color.hpp"
#include "../platform.h"
+32 -42
View File
@@ -9,17 +9,7 @@ using json = nlohmann::json;
const std::string NESTED_KEY_DELIMITER = ".";
std::vector<DataBucketsRepository::DataBuckets> g_data_bucket_cache = {};
#if defined(ZONE)
#include "../zone/zonedb.h"
extern ZoneDatabase database;
#elif defined(WORLD)
#include "../world/worlddb.h"
extern WorldDatabase database;
#else
#error "You must define either ZONE or WORLD"
#endif
void DataBucket::SetData(const std::string &bucket_key, const std::string &bucket_value, std::string expires_time)
void DataBucket::SetData(SharedDatabase* database, const std::string &bucket_key, const std::string &bucket_value, std::string expires_time)
{
auto k = DataBucketKey{
.key = bucket_key,
@@ -27,10 +17,10 @@ void DataBucket::SetData(const std::string &bucket_key, const std::string &bucke
.expires = expires_time,
};
DataBucket::SetData(k);
DataBucket::SetData(database, k);
}
void DataBucket::SetData(const DataBucketKey &k_)
void DataBucket::SetData(SharedDatabase *database, const DataBucketKey &k_)
{
DataBucketKey k = k_; // copy the key so we can modify it
bool is_nested = k.key.find(NESTED_KEY_DELIMITER) != std::string::npos;
@@ -39,7 +29,7 @@ void DataBucket::SetData(const DataBucketKey &k_)
}
auto b = DataBucketsRepository::NewEntity();
auto r = GetData(k, true);
auto r = GetData(database, k, true);
// if we have an entry, use it
if (r.id > 0) {
b = r;
@@ -149,10 +139,10 @@ void DataBucket::SetData(const DataBucketKey &k_)
}
}
DataBucketsRepository::UpdateOne(database, b);
DataBucketsRepository::UpdateOne(*database, b);
}
else {
b = DataBucketsRepository::InsertOne(database, b);
b = DataBucketsRepository::InsertOne(*database, b);
// add to cache if it doesn't exist
if (CanCache(k) && !ExistsInCache(b)) {
@@ -162,9 +152,9 @@ void DataBucket::SetData(const DataBucketKey &k_)
}
}
std::string DataBucket::GetData(const std::string &bucket_key)
std::string DataBucket::GetData(SharedDatabase* database, const std::string &bucket_key)
{
return GetData(DataBucketKey{.key = bucket_key}).value;
return GetData(database, DataBucketKey{.key = bucket_key}).value;
}
DataBucketsRepository::DataBuckets DataBucket::ExtractNestedValue(
@@ -214,7 +204,7 @@ DataBucketsRepository::DataBuckets DataBucket::ExtractNestedValue(
// if the bucket doesn't exist, it will be added to the cache as a miss
// if ignore_misses_cache is true, the bucket will not be added to the cache as a miss
// the only place we should be ignoring the misses cache is on the initial read during SetData
DataBucketsRepository::DataBuckets DataBucket::GetData(const DataBucketKey &k_, bool ignore_misses_cache)
DataBucketsRepository::DataBuckets DataBucket::GetData(SharedDatabase* database, const DataBucketKey &k_, bool ignore_misses_cache)
{
DataBucketKey k = k_; // Copy the key so we can modify it
@@ -244,7 +234,7 @@ DataBucketsRepository::DataBuckets DataBucket::GetData(const DataBucketKey &k_,
if (CheckBucketMatch(e, k)) {
if (e.expires > 0 && e.expires < std::time(nullptr)) {
LogDataBuckets("Attempted to read expired key [{}] removing from cache", e.key_);
DeleteData(k);
DeleteData(database, k);
return DataBucketsRepository::NewEntity();
}
@@ -261,7 +251,7 @@ DataBucketsRepository::DataBuckets DataBucket::GetData(const DataBucketKey &k_,
// Fetch the value from the database
auto r = DataBucketsRepository::GetWhere(
database,
*database,
fmt::format(
" {} `key` = '{}' LIMIT 1",
DataBucket::GetScopedDbFilters(k),
@@ -310,7 +300,7 @@ DataBucketsRepository::DataBuckets DataBucket::GetData(const DataBucketKey &k_,
// If the entry has expired, delete it
if (bucket.expires > 0 && bucket.expires < static_cast<long long>(std::time(nullptr))) {
DeleteData(k);
DeleteData(database, k);
return DataBucketsRepository::NewEntity();
}
@@ -337,22 +327,22 @@ DataBucketsRepository::DataBuckets DataBucket::GetData(const DataBucketKey &k_,
return bucket;
}
std::string DataBucket::GetDataExpires(const std::string &bucket_key)
std::string DataBucket::GetDataExpires(SharedDatabase* database, const std::string &bucket_key)
{
return GetDataExpires(DataBucketKey{.key = bucket_key});
return GetDataExpires(database, DataBucketKey{.key = bucket_key});
}
std::string DataBucket::GetDataRemaining(const std::string &bucket_key)
std::string DataBucket::GetDataRemaining(SharedDatabase* database, const std::string &bucket_key)
{
return GetDataRemaining(DataBucketKey{.key = bucket_key});
return GetDataRemaining(database, DataBucketKey{.key = bucket_key});
}
bool DataBucket::DeleteData(const std::string &bucket_key)
bool DataBucket::DeleteData(SharedDatabase* database, const std::string &bucket_key)
{
return DeleteData(DataBucketKey{.key = bucket_key});
return DeleteData(database, DataBucketKey{.key = bucket_key});
}
bool DataBucket::DeleteData(const DataBucketKey &k)
bool DataBucket::DeleteData(SharedDatabase* database, const DataBucketKey &k)
{
bool is_nested_key = k.key.find(NESTED_KEY_DELIMITER) != std::string::npos;
@@ -374,7 +364,7 @@ bool DataBucket::DeleteData(const DataBucketKey &k)
// Regular key deletion, no nesting involved
return DataBucketsRepository::DeleteWhere(
database,
*database,
fmt::format("{} `key` = '{}'", DataBucket::GetScopedDbFilters(k), k.key)
);
}
@@ -384,7 +374,7 @@ bool DataBucket::DeleteData(const DataBucketKey &k)
DataBucketKey top_level_k = k;
top_level_k.key = top_level_key;
auto r = GetData(top_level_k);
auto r = GetData(database, top_level_k);
if (r.id == 0 || r.value.empty() || !Strings::IsValidJson(r.value)) {
LogDataBuckets("Attempted to delete nested key [{}] but parent key [{}] does not exist or is invalid JSON", k.key, top_level_key);
return false;
@@ -444,14 +434,14 @@ bool DataBucket::DeleteData(const DataBucketKey &k)
}
return DataBucketsRepository::DeleteWhere(
database,
*database,
fmt::format("{} `key` = '{}'", DataBucket::GetScopedDbFilters(k), top_level_key)
);
}
// Otherwise, update the existing JSON without the deleted key
r.value = json_value.dump();
DataBucketsRepository::UpdateOne(database, r);
DataBucketsRepository::UpdateOne(*database, r);
// Update cache
if (CanCache(k)) {
@@ -466,7 +456,7 @@ bool DataBucket::DeleteData(const DataBucketKey &k)
return true;
}
std::string DataBucket::GetDataExpires(const DataBucketKey &k)
std::string DataBucket::GetDataExpires(SharedDatabase* database, const DataBucketKey &k)
{
LogDataBuckets(
"Getting bucket expiration key [{}] bot_id [{}] account_id [{}] character_id [{}] npc_id [{}]",
@@ -477,7 +467,7 @@ std::string DataBucket::GetDataExpires(const DataBucketKey &k)
k.npc_id
);
auto r = GetData(k);
auto r = GetData(database, k);
if (r.id == 0) {
return {};
}
@@ -485,7 +475,7 @@ std::string DataBucket::GetDataExpires(const DataBucketKey &k)
return std::to_string(r.expires);
}
std::string DataBucket::GetDataRemaining(const DataBucketKey &k)
std::string DataBucket::GetDataRemaining(SharedDatabase* database, const DataBucketKey &k)
{
LogDataBuckets(
"Getting bucket remaining key [{}] bot_id [{}] account_id [{}] character_id [{}] npc_id [{}] bot_id [{}] zone_id [{}] instance_id [{}]",
@@ -499,7 +489,7 @@ std::string DataBucket::GetDataRemaining(const DataBucketKey &k)
k.instance_id
);
auto r = GetData(k);
auto r = GetData(database, k);
if (r.id == 0) {
return "0";
}
@@ -565,10 +555,10 @@ bool DataBucket::CheckBucketMatch(const DataBucketsRepository::DataBuckets &dbe,
);
}
void DataBucket::LoadZoneCache(uint16 zone_id, uint16 instance_id)
void DataBucket::LoadZoneCache(SharedDatabase* database, uint16 zone_id, uint16 instance_id)
{
const auto &l = DataBucketsRepository::GetWhere(
database,
*database,
fmt::format(
"zone_id = {} AND instance_id = {} AND (`expires` > {} OR `expires` = 0)",
zone_id,
@@ -608,7 +598,7 @@ void DataBucket::LoadZoneCache(uint16 zone_id, uint16 instance_id)
);
}
void DataBucket::BulkLoadEntitiesToCache(DataBucketLoadType::Type t, std::vector<uint32> ids)
void DataBucket::BulkLoadEntitiesToCache(SharedDatabase* database, DataBucketLoadType::Type t, std::vector<uint32> ids)
{
if (ids.empty()) {
return;
@@ -653,7 +643,7 @@ void DataBucket::BulkLoadEntitiesToCache(DataBucketLoadType::Type t, std::vector
}
const auto &l = DataBucketsRepository::GetWhere(
database,
*database,
fmt::format(
"{} IN ({}) AND (`expires` > {} OR `expires` = 0)",
column,
@@ -843,4 +833,4 @@ bool DataBucket::CanCache(const DataBucketKey &key)
}
return false;
}
}
+13 -12
View File
@@ -5,6 +5,7 @@
#include "types.h"
#include "repositories/data_buckets_repository.h"
#include "json/json_archive_single_line.h"
#include "shareddb.h"
struct DataBucketKey {
std::string key;
@@ -38,26 +39,26 @@ namespace DataBucketLoadType {
class DataBucket {
public:
// non-scoped bucket methods (for global buckets)
static void SetData(const std::string &bucket_key, const std::string &bucket_value, std::string expires_time = "");
static bool DeleteData(const std::string &bucket_key);
static std::string GetData(const std::string &bucket_key);
static std::string GetDataExpires(const std::string &bucket_key);
static std::string GetDataRemaining(const std::string &bucket_key);
static void SetData(SharedDatabase *database, const std::string &bucket_key, const std::string &bucket_value, std::string expires_time = "");
static bool DeleteData(SharedDatabase *database, const std::string &bucket_key);
static std::string GetData(SharedDatabase *database, const std::string &bucket_key);
static std::string GetDataExpires(SharedDatabase *database, const std::string &bucket_key);
static std::string GetDataRemaining(SharedDatabase *database, const std::string &bucket_key);
// scoped bucket methods
static void SetData(const DataBucketKey &k_);
static bool DeleteData(const DataBucketKey &k);
static DataBucketsRepository::DataBuckets GetData(const DataBucketKey &k_, bool ignore_misses_cache = false);
static std::string GetDataExpires(const DataBucketKey &k);
static std::string GetDataRemaining(const DataBucketKey &k);
static void SetData(SharedDatabase *database, const DataBucketKey &k_);
static bool DeleteData(SharedDatabase *database, const DataBucketKey &k);
static DataBucketsRepository::DataBuckets GetData(SharedDatabase *database, const DataBucketKey &k_, bool ignore_misses_cache = false);
static std::string GetDataExpires(SharedDatabase *database, const DataBucketKey &k);
static std::string GetDataRemaining(SharedDatabase *database, const DataBucketKey &k);
static std::string GetScopedDbFilters(const DataBucketKey &k);
// bucket repository versus key matching
static bool CheckBucketMatch(const DataBucketsRepository::DataBuckets &dbe, const DataBucketKey &k);
static bool ExistsInCache(const DataBucketsRepository::DataBuckets &entry);
static void LoadZoneCache(uint16 zone_id, uint16 instance_id);
static void BulkLoadEntitiesToCache(DataBucketLoadType::Type t, std::vector<uint32> ids);
static void LoadZoneCache(SharedDatabase* database, uint16 zone_id, uint16 instance_id);
static void BulkLoadEntitiesToCache(SharedDatabase* database, DataBucketLoadType::Type t, std::vector<uint32> ids);
static void DeleteCachedBuckets(DataBucketLoadType::Type type, uint32 id, uint32 secondary_id = 0);
static void DeleteFromMissesCache(DataBucketsRepository::DataBuckets e);
+3 -4
View File
@@ -5,11 +5,10 @@
#include "../strings.h"
#include "../rulesys.h"
#include "../http/httplib.h"
#include "database_update_manifest.cpp"
#include "database_update_manifest_custom.cpp"
#include "database_update_manifest_bots.cpp"
#include "database_dump_service.h"
#include "database_update_manifest.h"
#include "database_update_manifest_custom.h"
#include "database_update_manifest_bots.h"
constexpr int BREAK_LENGTH = 70;
@@ -1,4 +1,4 @@
#include "database_update.h"
#pragma once
std::vector<ManifestEntry> manifest_entries = {
ManifestEntry{
@@ -1,4 +1,4 @@
#include "database_update.h"
#pragma once
std::vector<ManifestEntry> bot_manifest_entries = {
ManifestEntry{
@@ -1,4 +1,4 @@
#include "database_update.h"
#pragma once
std::vector<ManifestEntry> manifest_entries_custom = {
ManifestEntry{
+8
View File
@@ -258,9 +258,17 @@ bool DBcore::Open(uint32 *errnum, char *errbuf)
if (pCompress) {
flags |= CLIENT_COMPRESS;
}
//todo: we need to revisit this ssl handling later
//the whole connect code is ancient and tls is starting to come as a default requirement for many db setups
if (pSSL) {
flags |= CLIENT_SSL;
}
else {
int off = 0;
mysql_options(mysql, MYSQL_OPT_SSL_ENFORCE, &off);
mysql_options(mysql, MYSQL_OPT_SSL_VERIFY_SERVER_CERT, &off);
}
if (mysql_real_connect(mysql, pHost, pUser, pPassword, pDatabase, pPort, 0, flags)) {
pStatus = Connected;
+1
View File
@@ -3,6 +3,7 @@
#include "rulesys.h"
#include "util/uuid.h"
#include <fmt/format.h>
#include <fmt/ranges.h>
#include <cereal/types/chrono.hpp>
DzLockout::DzLockout(std::string uuid, std::string expedition, std::string event, uint64_t expire_time, uint32_t duration)
+3 -3
View File
@@ -23,9 +23,9 @@
#include <string.h>
#include <string>
#include <time.h>
#include "../cereal/include/cereal/archives/binary.hpp"
#include "../cereal/include/cereal/types/string.hpp"
#include "../cereal/include/cereal/types/vector.hpp"
#include <cereal/archives/binary.hpp>
#include <cereal/types/string.hpp>
#include <cereal/types/vector.hpp>
#include "../common/version.h"
#include "emu_constants.h"
#include "textures.h"
+1
View File
@@ -23,6 +23,7 @@
#include "path_manager.h"
#include <fstream>
#include <fmt/format.h>
#include <fmt/ranges.h>
struct LoginConfig {
std::string LoginHost;
+1
View File
@@ -34,6 +34,7 @@
#endif
#include <fmt/format.h>
#include <fmt/ranges.h>
#include "types.h"
namespace Logs {
@@ -3,6 +3,7 @@
#include "../json/json_archive_single_line.h"
#include <vector>
#include <fmt/format.h>
#include <fmt/ranges.h>
#include <cereal/archives/json.hpp>
#include <cereal/types/vector.hpp>
+1
View File
@@ -36,6 +36,7 @@
#endif
#include <fmt/format.h>
#include <fmt/ranges.h>
#include <filesystem>
#include <iostream>
#include <sys/stat.h>
+1
View File
@@ -20,6 +20,7 @@
#include <cstring>
#include <fmt/format.h>
#include <fmt/ranges.h>
#include <csignal>
#include <vector>
#include "ip_util.h"
+1
View File
@@ -1,6 +1,7 @@
#include "console_server.h"
#include "../strings.h"
#include <fmt/format.h>
#include <fmt/ranges.h>
EQ::Net::ConsoleServer::ConsoleServer(const std::string &addr, int port)
{
+1
View File
@@ -5,6 +5,7 @@
#include "../servertalk.h"
#include "../rulesys.h"
#include <fmt/format.h>
#include <fmt/ranges.h>
EQ::Net::ConsoleServerConnection::ConsoleServerConnection(ConsoleServer *parent, std::shared_ptr<TCPConnection> connection)
{
+1
View File
@@ -2,6 +2,7 @@
#include "endian.h"
#include <cctype>
#include <fmt/format.h>
#include <fmt/ranges.h>
void EQ::Net::Packet::PutInt8(size_t offset, int8_t value)
{
@@ -4,6 +4,7 @@
#include "crc32.h"
#include <zlib.h>
#include <fmt/format.h>
#include <fmt/ranges.h>
// observed client receive window is 300 packets, 140KB
constexpr size_t MAX_CLIENT_RECV_PACKETS_PER_WINDOW = 300;
+1
View File
@@ -2,6 +2,7 @@
#include "../event/event_loop.h"
#include "../event/timer.h"
#include <fmt/format.h>
#include <fmt/ranges.h>
#include <map>
#include <unordered_set>
#include <array>
@@ -4,6 +4,7 @@
#include "../util/uuid.h"
#include <sstream>
#include <fmt/format.h>
#include <fmt/ranges.h>
struct EQ::Net::WebsocketServerConnection::Impl {
WebsocketServer *parent;
+1 -1
View File
@@ -38,7 +38,7 @@
#include "../raid.h"
#include "../guilds.h"
//#include "../repositories/trader_repository.h"
#include "../cereal/include/cereal/types/vector.hpp"
#include <cereal/types/vector.hpp>
#include <iostream>
#include <sstream>
+1
View File
@@ -2,6 +2,7 @@
#include <memory>
#include "process.h"
#include <fmt/format.h>
#include <fmt/ranges.h>
std::string Process::execute(const std::string &cmd)
{
+1 -1
View File
@@ -23,7 +23,7 @@
#include <string>
#include <list>
#include <fmt/format.h>
#include <fmt/ranges.h>
class DBcore;
+2 -2
View File
@@ -7,8 +7,8 @@
#include "items_repository.h"
#include "../../common/item_data.h"
#include "../../common/races.h"
#include "../cereal/include/cereal/archives/binary.hpp"
#include "../cereal/include/cereal/types/string.hpp"
#include <cereal/archives/binary.hpp>
#include <cereal/types/string.hpp>
class TraderRepository : public BaseTraderRepository {
public:
+1
View File
@@ -23,6 +23,7 @@
#include <cstdlib>
#include <cstring>
#include <fmt/format.h>
#include <fmt/ranges.h>
#include "../common/repositories/rule_sets_repository.h"
#include "../common/repositories/rule_values_repository.h"
+1
View File
@@ -19,6 +19,7 @@
#include <iostream>
#include <cstring>
#include <fmt/format.h>
#include <fmt/ranges.h>
#if defined(_MSC_VER) && _MSC_VER >= 1800
#include <algorithm>
+1 -5
View File
@@ -36,6 +36,7 @@
#include "strings.h"
#include <cereal/external/rapidjson/document.h>
#include <fmt/format.h>
#include <fmt/ranges.h>
#include <algorithm>
#include <cctype>
@@ -47,14 +48,9 @@
#include <random>
#include <string>
//Const char based
#include "strings_legacy.cpp" // legacy c functions
#include "strings_misc.cpp" // anything non "Strings" scoped
#ifdef _WINDOWS
#include <ctype.h>
#include <functional>
#include <algorithm>
#endif
std::string Strings::Random(size_t length)
+1
View File
@@ -2,6 +2,7 @@
#include <ios>
#include <fmt/format.h>
#include <fmt/ranges.h>
#ifdef _WIN32
#include <objbase.h>