Commit Graph

54 Commits

Author SHA1 Message Date
Clemens Schwaighofer
d29f827fc9 Add a function to Log system to update the console formatter dynamically. 2025-11-19 15:17:25 +09:00
Clemens Schwaighofer
5996bb1fc0 Add Log ConsoleFormatSettings.from_string static method to get settings by name with default option
To help set from config or command line with fallback
2025-11-19 13:45:26 +09:00
Clemens Schwaighofer
1280b2f855 Log switch to bitwise flag settings for console format type
Has the following settings
TIME, TIME_SECONDS, TIME_MILLISECONDS, TIME_MICROSECONDS: enable time output in different formats
TIME and TIME_MILLISECONDS are equivalent, if multiple are set the smallest precision wins
TIMEZONE: add time zone to time output
NAME: log group name
FILE: short file name
FUNCTION: function name
LINENO: line number

There is a class with quick grouped settings
ConsoleFormatSettings
ALL: all options enabled, time is in milliseconds
CONDENSED: time without time zone, file and line number
MINIMAL: only time without time zone
BARE: only the message, no other info
2025-11-19 11:25:49 +09:00
Clemens Schwaighofer
592652cff1 Update logging with console output format changes
"console_format_type" with "normal", "condensed", "minimal" options
This sets the format of the console output, controlling the amount of detail shown.
normal show log title, file, function and line number
condensed show file and line number only
minimal shows only timestamp, log level and message
Default is normal

"console_iso_precision" with "seconds", "milliseconds", "microseconds" options
This sets the precision of the ISO timestamp in console logs.
Default is milliseconds

The timestamp output is now ISO8601 formatatted with time zone.
2025-11-18 15:31:16 +09:00
Clemens Schwaighofer
4e78d83092 Add checks for BOM encoding in files 2025-11-06 18:21:32 +09:00
Clemens Schwaighofer
c5ab4352e3 Fix name dict_helper to dict_helpers
So we have the same name for everyhing
2025-10-27 10:40:12 +09:00
Clemens Schwaighofer
fb4fdb6857 iterator tests added 2025-10-24 16:36:42 +09:00
Clemens Schwaighofer
d642a13b6e file handling tests, move progress to script handling
Progress is not only file, but process progress in a script
2025-10-24 16:07:47 +09:00
Clemens Schwaighofer
a76eae0cc7 Add datetime helpers and move all time/date time datetime_handling folder
previous string_handling located datetime and timestamp files have been moved
to the datetime handling folder

Update readme file with more information about currently covered areas
2025-10-24 10:03:04 +09:00
Clemens Schwaighofer
53cf2a6f48 Add prepare_url_slash to string_helpers.py and tests
Function cleans up url paths (without domain) by ensuring they start with a single slash and removing double slashes.
2025-10-23 15:47:19 +09:00
Clemens Schwaighofer
fe69530b38 Add a simple add key entry to dictionary 2025-10-23 15:31:52 +09:00
Clemens Schwaighofer
84ce43ab93 Add SQLite IO class
This is a very basic class without many helper functions added yet
Add to the CoreLibs so when we develop it further it can be used by all projects
2025-10-23 15:22:12 +09:00
Clemens Schwaighofer
5e0765ee24 Rename the enum_test to enum_base for the test run file 2025-10-23 14:32:52 +09:00
Clemens Schwaighofer
30bf9c1bcb Add Enum base class
A helper class for handling enum classes with various lookup helpers
2025-10-23 13:47:13 +09:00
Clemens Schwaighofer
2544fad9ce Add json helper function json_replace
Function can replace content for a json path string in a dictionary
2025-10-23 13:20:40 +09:00
Clemens Schwaighofer
543e9766a1 Add symmetric encryption and tests 2025-10-23 11:47:41 +09:00
Clemens Schwaighofer
dadc14563a jmespath search check update 2025-10-09 16:42:41 +09:00
Clemens Schwaighofer
b858936c68 Add test file for datetime helpers 2025-10-08 16:08:23 +09:00
Clemens Schwaighofer
bb60a570d0 Change the TimestampStrings check to check for str instead of not ZoneInfo.
This fixes the pytest problem which threw:
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

during Mocking
2025-09-25 15:36:47 +09:00
Clemens Schwaighofer
38bae7fb46 TimestampStrings allows ZoneInfo object as time_zone parameter
So we can use pre-parsed data

Some tests for parsing settings, timestamp output
2025-09-25 15:14:40 +09:00
Clemens Schwaighofer
ef5981b473 convert_to_seconds allow negative time strings and add pytests 2025-09-24 15:25:53 +09:00
Clemens Schwaighofer
b32887a6d8 Add time in seconds convert to human readable format 2025-09-19 09:57:51 +09:00
Clemens Schwaighofer
f8ae6609c7 Allow empty config settings for settings loader if only loading is needed 2025-08-26 14:38:55 +09:00
Clemens Schwaighofer
f8905a176c Fix settings loader
Remove all class vars for vars that are only used in the loader itsef
- entry_split_char
- entry_convert
- entry_set_empty

The self.settings varr was never used, removed

The config file path exists check is moved to the config data loader

The internal _check_settings_abort is now __check_settings_abort to make it private

lock file updates
2025-08-26 14:29:52 +09:00
Clemens Schwaighofer
0df049d453 Add per run log rotate flag
This flag will use the normal file handler with a file name that has date + time + milliseconds
to create a new file each time the script is run
2025-08-01 16:01:50 +09:00
Clemens Schwaighofer
2f08ecabbf For convert time string, skip convert if incoming value is a number of any type
Any float number will be rounded, and everything that is any kind of number will be then converted to int and returned
The rest will be converted to string and normal convert is run
2025-07-29 09:29:38 +09:00
Clemens Schwaighofer
a586cf65e2 Convert string with time units to seconds 2025-07-29 09:13:36 +09:00
Clemens Schwaighofer
4f9c2b9d5f Add exception stack caller and add this to the logger exception call
So we get the location of the exception in the console log too
2025-07-28 15:26:23 +09:00
Clemens Schwaighofer
f1e3bc8559 For Log exception write to ERROR, add the stack trace too 2025-07-28 14:32:14 +09:00
Clemens Schwaighofer
2cfbf4bb90 Update data search for iterators
array_search name is deprecated
use find_in_array_from_list
- change parameter order
data (search in) comes before search_params list
- created a TypedDict for the array search params dict entry
2025-07-25 15:48:37 +09:00
Clemens Schwaighofer
fd5e1db22b Change Log class and add simple Logger class without the init work
This Logger class can be used after the main Log class has been setup, eg in workers
inside pool/fork/thread runs

A new parent class holds all the public methods

init Logger class with "get_logger_settings" or with a dictionary Type LoggerInit that has
logger with type logging.Logger as mandatory and optional Queue entry
2025-07-18 14:15:00 +09:00
Clemens Schwaighofer
39e23faf7f dict mask helper test code update 2025-07-17 15:29:49 +09:00
Clemens Schwaighofer
280e5fa861 Update the mask dict helper
It now goes recursive, is case insenstivie for keys and mask keys requests
Checks not for equal but for start/end or inside with edge character set

pytests added
2025-07-17 15:17:57 +09:00
Clemens Schwaighofer
694712ed2e logging update for exceptions with special error log to console
When calling log.exception it automatically writes a short message to the console as error
This error message is tagged <=EXCEPTION> and contains only the message without the stack trace
2025-07-15 17:56:56 +09:00
Clemens Schwaighofer
948b0dd5e7 Settings loader add more checks
string.domain.with-localhost
string.domain.with-localhost.port
string.domain
string.date
2025-07-15 10:32:19 +09:00
Clemens Schwaighofer
a626b738a9 Move dump_data from iterator folder to debug folder 2025-07-15 09:54:23 +09:00
Clemens Schwaighofer
c69076f517 Add set default if empty/not set in settings
With new empty: block if just like this set to None if not set (empty), can also be any value,
if list, skip setting default
2025-07-14 17:21:04 +09:00
Clemens Schwaighofer
648ab001b6 Settings loader fix for not set range check entries
If we have a range or length check and the value is not set, skip, and do not convert either
Not set is None
2025-07-14 17:00:25 +09:00
Clemens Schwaighofer
0770ac0bb4 Better error handling in the settings loader for entry not found in block 2025-07-14 16:49:37 +09:00
Clemens Schwaighofer
58c8447531 Settings loader mandatory fixes
- mandatory empty check if empty list ([''])
- skip regex check if replace value is None -> allowed empty as empty if not mandatory
2025-07-14 16:23:55 +09:00
Clemens Schwaighofer
6c2637ad34 Settings loader update with basic email check, and on check abort if not valid
In the settings checker, if a regex_clean is set as None then we will abort the script with error
if the regex is not matching

Add regex check for email basic

Also add a regex_constants list with regex entries (not compiled and compiled)
2025-07-14 15:57:19 +09:00
Clemens Schwaighofer
4ca45ebc73 Move var helpers into their own file, log update with additional levels
Add levels for ALERT, EMERGENCY to be syslog compatible
Add direct wrappers for all, but they are not yet fully usable because the stack fix is not yet implemented

Add a new debug helepr to get the stack as a string
2025-07-11 19:09:22 +09:00
Clemens Schwaighofer
6902768fed Make sure in log that adding handlers only works before logging is initialized
For future: if we add handlers later, the queue / logger must be re-intialized
2025-07-11 15:50:17 +09:00
Clemens Schwaighofer
3f9f2ceaac Log level setter now uses LoggingLevel for levels, set/get log level
Add flush for queue flushing

Add set/get level for handler

Allow adding handlers during launch, handlers cannot be added afterwards at the moment

Add testing for LoggingLevel enum
2025-07-11 15:35:34 +09:00
Clemens Schwaighofer
c559a6bafb Move list_dict_handling to iterator_handling, add settings parser, add list helpers, add some string helpers
list_helpers:
convert to list, any input, output is always a list
compare to lists, check what elements from A are not in B, type safe

string helpers
add is_int, is_float checker
add string to bool converter for true/True/false/False strings

config reader with parsing and checking
The simple config reader is now in the corelibs with the basic content check, convert to list for entries, convert to value for entries, etc

log updates:
Add Log type Enum for better log level checks and convert
Add a get int for requested log level, and return default if not found
Make the validate log level a static function

Add tests for list helpers and new string helpers
2025-07-11 10:58:35 +09:00
Clemens Schwaighofer
19d7e9b5ed Make critical errors as red bold for logging to console 2025-07-10 14:34:06 +09:00
Clemens Schwaighofer
424c91945a Update log and add Log Enum for Logging levels, add log level as in from Log
Add all logging levels as Enum class type (same as for the PHP CoreLibs Logging)

Add a new method to get the log level as int with fallback via the LoggingLevel Enum
2025-07-10 13:43:35 +09:00
Clemens Schwaighofer
c657dc564e Log: Make the validate log level a static function 2025-07-09 18:54:03 +09:00
Clemens Schwaighofer
084ecc01e0 Bug fix for logging queue listener and ignoring of per handler set levels
in the QueueLister launcher add "respect_handler_level=True" so it respects the previous set log levels per handler

Also split all logging tests into their own file
2025-07-09 17:14:48 +09:00
Clemens Schwaighofer
08cb994d8d PyProject version update, testing logging 2025-07-09 16:16:02 +09:00