Commit Graph

58 Commits

Author SHA1 Message Date
Clemens Schwaighofer
4f9c2b9d5f Add exception stack caller and add this to the logger exception call
So we get the location of the exception in the console log too
2025-07-28 15:26:23 +09:00
Clemens Schwaighofer
f1e3bc8559 For Log exception write to ERROR, add the stack trace too 2025-07-28 14:32:14 +09:00
Clemens Schwaighofer
d1ea9874da Add HTTP basic auth builder 2025-07-26 11:26:09 +09:00
Clemens Schwaighofer
582937b866 dump_data is now ANY, we do the detail dump type in the run later 2025-07-26 10:51:37 +09:00
Clemens Schwaighofer
abf4b7ac89 Bug fix for find_in_array_from_list because of keys order 2025-07-25 15:57:48 +09:00
Clemens Schwaighofer
2cfbf4bb90 Update data search for iterators
array_search name is deprecated
use find_in_array_from_list
- change parameter order
data (search in) comes before search_params list
- created a TypedDict for the array search params dict entry
2025-07-25 15:48:37 +09:00
Clemens Schwaighofer
24798f19ca Add CSV Exceptions 2025-07-25 10:23:52 +09:00
Clemens Schwaighofer
dcefa564da Fix stack stack traceback call
It now works correct with start and skip_last settings, the method is now called "call_stack"

Also added auto reset if no output (start too hight) and optional stack separator
2025-07-24 10:52:39 +09:00
Clemens Schwaighofer
edd35dccea Comment update for log class 2025-07-22 19:32:21 +09:00
Clemens Schwaighofer
fd5e1db22b Change Log class and add simple Logger class without the init work
This Logger class can be used after the main Log class has been setup, eg in workers
inside pool/fork/thread runs

A new parent class holds all the public methods

init Logger class with "get_logger_settings" or with a dictionary Type LoggerInit that has
logger with type logging.Logger as mandatory and optional Queue entry
2025-07-18 14:15:00 +09:00
Clemens Schwaighofer
280e5fa861 Update the mask dict helper
It now goes recursive, is case insenstivie for keys and mask keys requests
Checks not for equal but for start/end or inside with edge character set

pytests added
2025-07-17 15:17:57 +09:00
Clemens Schwaighofer
472d3495b5 Add missing typedefs for regex constants 2025-07-17 13:32:35 +09:00
Clemens Schwaighofer
2778ac6870 in Log use the defined default levels for console and file 2025-07-16 11:06:38 +09:00
Clemens Schwaighofer
694712ed2e logging update for exceptions with special error log to console
When calling log.exception it automatically writes a short message to the console as error
This error message is tagged <=EXCEPTION> and contains only the message without the stack trace
2025-07-15 17:56:56 +09:00
Clemens Schwaighofer
da68818d4f Move the domain regex to the regex constant file 2025-07-15 11:13:23 +09:00
Clemens Schwaighofer
948b0dd5e7 Settings loader add more checks
string.domain.with-localhost
string.domain.with-localhost.port
string.domain
string.date
2025-07-15 10:32:19 +09:00
Clemens Schwaighofer
a626b738a9 Move dump_data from iterator folder to debug folder 2025-07-15 09:54:23 +09:00
Clemens Schwaighofer
5763f57830 In settings loader do the raise ValueRror on the error, fix stack level, loggin fix stack level
Settings loader: all errors are thrown where the error happens and not in the print function
The print function if to log will add +1 to the stack level so the error is shown

In the log class in the log wrapper calls add +1 to the stack level to have the error line in the correct place
-> this fixes the stack trace part for now but we still want to have an auto full stack trace simple added
2025-07-15 09:44:29 +09:00
Clemens Schwaighofer
acbe1ac692 Settings load add info for future settings/options argument 2025-07-14 18:15:07 +09:00
Clemens Schwaighofer
99bca2c467 Allow settings block to not exist via call setting 2025-07-14 18:14:33 +09:00
Clemens Schwaighofer
8082ab78a1 Merge branch 'development' 2025-07-14 17:21:28 +09:00
Clemens Schwaighofer
c69076f517 Add set default if empty/not set in settings
With new empty: block if just like this set to None if not set (empty), can also be any value,
if list, skip setting default
2025-07-14 17:21:04 +09:00
Clemens Schwaighofer
648ab001b6 Settings loader fix for not set range check entries
If we have a range or length check and the value is not set, skip, and do not convert either
Not set is None
2025-07-14 17:00:25 +09:00
Clemens Schwaighofer
447034046e v0.12.3: settings loader error message improvement 2025-07-14 16:50:36 +09:00
Clemens Schwaighofer
0770ac0bb4 Better error handling in the settings loader for entry not found in block 2025-07-14 16:49:37 +09:00
Clemens Schwaighofer
58c8447531 Settings loader mandatory fixes
- mandatory empty check if empty list ([''])
- skip regex check if replace value is None -> allowed empty as empty if not mandatory
2025-07-14 16:23:55 +09:00
Clemens Schwaighofer
e9ccfe7ad2 Rebame the regex constants file name to not have compiled inside the name 2025-07-14 15:59:34 +09:00
Clemens Schwaighofer
6c2637ad34 Settings loader update with basic email check, and on check abort if not valid
In the settings checker, if a regex_clean is set as None then we will abort the script with error
if the regex is not matching

Add regex check for email basic

Also add a regex_constants list with regex entries (not compiled and compiled)
2025-07-14 15:57:19 +09:00
Clemens Schwaighofer
7183d05dd6 Update log method documentation 2025-07-14 14:29:42 +09:00
Clemens Schwaighofer
4ca45ebc73 Move var helpers into their own file, log update with additional levels
Add levels for ALERT, EMERGENCY to be syslog compatible
Add direct wrappers for all, but they are not yet fully usable because the stack fix is not yet implemented

Add a new debug helepr to get the stack as a string
2025-07-11 19:09:22 +09:00
Clemens Schwaighofer
6902768fed Make sure in log that adding handlers only works before logging is initialized
For future: if we add handlers later, the queue / logger must be re-intialized
2025-07-11 15:50:17 +09:00
Clemens Schwaighofer
3f9f2ceaac Log level setter now uses LoggingLevel for levels, set/get log level
Add flush for queue flushing

Add set/get level for handler

Allow adding handlers during launch, handlers cannot be added afterwards at the moment

Add testing for LoggingLevel enum
2025-07-11 15:35:34 +09:00
Clemens Schwaighofer
c559a6bafb Move list_dict_handling to iterator_handling, add settings parser, add list helpers, add some string helpers
list_helpers:
convert to list, any input, output is always a list
compare to lists, check what elements from A are not in B, type safe

string helpers
add is_int, is_float checker
add string to bool converter for true/True/false/False strings

config reader with parsing and checking
The simple config reader is now in the corelibs with the basic content check, convert to list for entries, convert to value for entries, etc

log updates:
Add Log type Enum for better log level checks and convert
Add a get int for requested log level, and return default if not found
Make the validate log level a static function

Add tests for list helpers and new string helpers
2025-07-11 10:58:35 +09:00
Clemens Schwaighofer
19d7e9b5ed Make critical errors as red bold for logging to console 2025-07-10 14:34:06 +09:00
Clemens Schwaighofer
424c91945a Update log and add Log Enum for Logging levels, add log level as in from Log
Add all logging levels as Enum class type (same as for the PHP CoreLibs Logging)

Add a new method to get the log level as int with fallback via the LoggingLevel Enum
2025-07-10 13:43:35 +09:00
Clemens Schwaighofer
c657dc564e Log: Make the validate log level a static function 2025-07-09 18:54:03 +09:00
Clemens Schwaighofer
084ecc01e0 Bug fix for logging queue listener and ignoring of per handler set levels
in the QueueLister launcher add "respect_handler_level=True" so it respects the previous set log levels per handler

Also split all logging tests into their own file
2025-07-09 17:14:48 +09:00
Clemens Schwaighofer
67f1a6688d Update Colors to have static colors that we can reset to 2025-07-09 15:22:31 +09:00
Clemens Schwaighofer
efb7968e93 time zone general names: upper case for constant 2025-07-09 15:09:12 +09:00
Clemens Schwaighofer
fe7c7db004 Change all __class__ to self for class global var 2025-07-09 15:06:27 +09:00
Clemens Schwaighofer
d500b7d473 Log update with listener Queues and color highlight for console
enables Queues if multiprocessing.Queue() is set in the "log_queue" setting

Now a logger "Log.init_worker_logging" can be attached to the ProcessPoolExecutor,
the init args must be the log_queue set

example:

```py
with concurrent.futures.ProcessPoolExecutor(
    max_workers=max_forks,
    initializer=Log.init_worker_logging,
    initargs=(log_queue,)
) as executor:
````

Move all settings into a settings argument, the structure is defined in LogSettings.
Default settings are in Log.DEFAULT_LOG_SETTINGS

Only log path and log name are parameters

Color output for console is on default enabled, disable via "console_color_output_enabled"
The complete console output can be stopped with "console_enabled"
2025-07-09 14:41:53 +09:00
Clemens Schwaighofer
2d197134f1 log splitter output make length configurable 2025-07-09 09:44:55 +09:00
Clemens Schwaighofer
717080a009 Add Text ANSI colors class 2025-07-09 08:54:29 +09:00
Clemens Schwaighofer
b88e0fe564 Add tests for string helpers too, update timestamp strings tests
Fix string helpers calls for some edge cases
2025-07-08 15:58:37 +09:00
Clemens Schwaighofer
060e3b4afe Move test runs into the test-run folder, add TimestampStrings, add basic tests via pytest
pytest added for dev.
Move all the test run python scripts into the "test-run" folder, so we can use the tests/ folder for just tests.

Setup one test for the TimestampStrings class with pytests in tests/unit/string_handling/test_timestamp_strings.py
2025-07-08 14:54:26 +09:00
Clemens Schwaighofer
2fa031f6ee Comment out log handlers until we rebuild the logging class 2025-07-08 10:01:09 +09:00
Clemens Schwaighofer
f38cce1c1d Rename src CoreLibs to corelibs 2025-07-08 09:58:33 +09:00
Clemens Schwaighofer
52dd1e7b73 Fix base folder name, must be lower case 2025-07-08 09:56:43 +09:00
Clemens Schwaighofer
661a182655 Fix __init__.py 2025-07-08 09:47:55 +09:00
Clemens Schwaighofer
d803de312d Change log file formatter order 2025-07-08 09:46:00 +09:00