Commit Graph

42 Commits

Author SHA1 Message Date
Clemens Schwaighofer
2e0d5aeb51 Move all debug handling into their own packages
dump data: corelibs_dump_data
stack trace: corelibs_stack_trace
profiling, timing, etc: corelibs_debug
2026-02-03 10:48:59 +09:00
Clemens Schwaighofer
84286593f6 Log fix bug where log consosle format set to None would throw an exception
Also add prefix "[SettingsLoader] " to print statements in SettingsLoader if we do not write to log
2026-01-23 15:14:31 +09:00
Clemens Schwaighofer
2748bc19be Log, add get console formatter method
Returns current flags set for console formatter
2026-01-23 11:33:38 +09:00
Clemens Schwaighofer
acd61e825e Add Log method "any handler is minimum level" with tests
Checks if a given handler is set for any current active handler
2025-12-04 14:37:55 +09:00
Clemens Schwaighofer
90e5179980 Remove text color handling from corelibs and use corelibs_text_colors instead
Also update enum with proper pyi file for deprecation warnings
2025-11-20 10:59:44 +09:00
Clemens Schwaighofer
f1788f057f Log skip format change it format flags have not changed 2025-11-19 17:42:47 +09:00
Clemens Schwaighofer
51e9b1ce7c Add "LEVEL" option to console log format
So we can set output to onle the message without any information (NONE),
only level (BARE), time and level (MINIMAL), time, file, line and level (CONDENSED) or
(ALL) full information.
2025-11-19 17:35:27 +09:00
Clemens Schwaighofer
d29f827fc9 Add a function to Log system to update the console formatter dynamically. 2025-11-19 15:17:25 +09:00
Clemens Schwaighofer
5996bb1fc0 Add Log ConsoleFormatSettings.from_string static method to get settings by name with default option
To help set from config or command line with fallback
2025-11-19 13:45:26 +09:00
Clemens Schwaighofer
1280b2f855 Log switch to bitwise flag settings for console format type
Has the following settings
TIME, TIME_SECONDS, TIME_MILLISECONDS, TIME_MICROSECONDS: enable time output in different formats
TIME and TIME_MILLISECONDS are equivalent, if multiple are set the smallest precision wins
TIMEZONE: add time zone to time output
NAME: log group name
FILE: short file name
FUNCTION: function name
LINENO: line number

There is a class with quick grouped settings
ConsoleFormatSettings
ALL: all options enabled, time is in milliseconds
CONDENSED: time without time zone, file and line number
MINIMAL: only time without time zone
BARE: only the message, no other info
2025-11-19 11:25:49 +09:00
Clemens Schwaighofer
52919cbc49 Log: move process id to front of pathname in log format
The previous filename:pid has been removed, the filename is part of the pathname.
No need for double filename info and wasting space in the log line.
2025-11-18 15:49:46 +09:00
Clemens Schwaighofer
592652cff1 Update logging with console output format changes
"console_format_type" with "normal", "condensed", "minimal" options
This sets the format of the console output, controlling the amount of detail shown.
normal show log title, file, function and line number
condensed show file and line number only
minimal shows only timestamp, log level and message
Default is normal

"console_iso_precision" with "seconds", "milliseconds", "microseconds" options
This sets the precision of the ISO timestamp in console logs.
Default is milliseconds

The timestamp output is now ISO8601 formatatted with time zone.
2025-11-18 15:31:16 +09:00
Clemens Schwaighofer
d0a1673965 Add pytest for logging 2025-10-24 18:33:25 +09:00
Clemens Schwaighofer
f85fbb86af Add iso datetime create with time zone support
The time zone check is for short mappings is limited, it is recommended
to use full TZ names like "Europe/Berlin", "Asia/Tokyo", "America/New_York"
2025-10-08 15:57:57 +09:00
Clemens Schwaighofer
446d9d5217 Log documentation updates 2025-08-18 14:35:14 +09:00
Clemens Schwaighofer
bc23006a34 disable the auto close of the log queue
This causes problems with logger clean up
2025-08-05 16:20:13 +09:00
Clemens Schwaighofer
60db747d6d More fixes for the queue clean up
Changed that we call stop_listener and not _cleanup on exit
Then call _cleanup from the stop listener
We only need that if we have listeners (queue) anyway
2025-08-05 13:22:54 +09:00
Clemens Schwaighofer
2b04cbe239 Remove Log __del__ cleanup 2025-08-05 10:36:49 +09:00
Clemens Schwaighofer
80319385f0 Add Log exist queue clean up if queue is set
to avoid hung threads on errors
2025-08-05 10:32:33 +09:00
Clemens Schwaighofer
0df049d453 Add per run log rotate flag
This flag will use the normal file handler with a file name that has date + time + milliseconds
to create a new file each time the script is run
2025-08-01 16:01:50 +09:00
Clemens Schwaighofer
4f9c2b9d5f Add exception stack caller and add this to the logger exception call
So we get the location of the exception in the console log too
2025-07-28 15:26:23 +09:00
Clemens Schwaighofer
f1e3bc8559 For Log exception write to ERROR, add the stack trace too 2025-07-28 14:32:14 +09:00
Clemens Schwaighofer
dcefa564da Fix stack stack traceback call
It now works correct with start and skip_last settings, the method is now called "call_stack"

Also added auto reset if no output (start too hight) and optional stack separator
2025-07-24 10:52:39 +09:00
Clemens Schwaighofer
edd35dccea Comment update for log class 2025-07-22 19:32:21 +09:00
Clemens Schwaighofer
fd5e1db22b Change Log class and add simple Logger class without the init work
This Logger class can be used after the main Log class has been setup, eg in workers
inside pool/fork/thread runs

A new parent class holds all the public methods

init Logger class with "get_logger_settings" or with a dictionary Type LoggerInit that has
logger with type logging.Logger as mandatory and optional Queue entry
2025-07-18 14:15:00 +09:00
Clemens Schwaighofer
2778ac6870 in Log use the defined default levels for console and file 2025-07-16 11:06:38 +09:00
Clemens Schwaighofer
694712ed2e logging update for exceptions with special error log to console
When calling log.exception it automatically writes a short message to the console as error
This error message is tagged <=EXCEPTION> and contains only the message without the stack trace
2025-07-15 17:56:56 +09:00
Clemens Schwaighofer
5763f57830 In settings loader do the raise ValueRror on the error, fix stack level, loggin fix stack level
Settings loader: all errors are thrown where the error happens and not in the print function
The print function if to log will add +1 to the stack level so the error is shown

In the log class in the log wrapper calls add +1 to the stack level to have the error line in the correct place
-> this fixes the stack trace part for now but we still want to have an auto full stack trace simple added
2025-07-15 09:44:29 +09:00
Clemens Schwaighofer
7183d05dd6 Update log method documentation 2025-07-14 14:29:42 +09:00
Clemens Schwaighofer
4ca45ebc73 Move var helpers into their own file, log update with additional levels
Add levels for ALERT, EMERGENCY to be syslog compatible
Add direct wrappers for all, but they are not yet fully usable because the stack fix is not yet implemented

Add a new debug helepr to get the stack as a string
2025-07-11 19:09:22 +09:00
Clemens Schwaighofer
6902768fed Make sure in log that adding handlers only works before logging is initialized
For future: if we add handlers later, the queue / logger must be re-intialized
2025-07-11 15:50:17 +09:00
Clemens Schwaighofer
3f9f2ceaac Log level setter now uses LoggingLevel for levels, set/get log level
Add flush for queue flushing

Add set/get level for handler

Allow adding handlers during launch, handlers cannot be added afterwards at the moment

Add testing for LoggingLevel enum
2025-07-11 15:35:34 +09:00
Clemens Schwaighofer
19d7e9b5ed Make critical errors as red bold for logging to console 2025-07-10 14:34:06 +09:00
Clemens Schwaighofer
424c91945a Update log and add Log Enum for Logging levels, add log level as in from Log
Add all logging levels as Enum class type (same as for the PHP CoreLibs Logging)

Add a new method to get the log level as int with fallback via the LoggingLevel Enum
2025-07-10 13:43:35 +09:00
Clemens Schwaighofer
c657dc564e Log: Make the validate log level a static function 2025-07-09 18:54:03 +09:00
Clemens Schwaighofer
084ecc01e0 Bug fix for logging queue listener and ignoring of per handler set levels
in the QueueLister launcher add "respect_handler_level=True" so it respects the previous set log levels per handler

Also split all logging tests into their own file
2025-07-09 17:14:48 +09:00
Clemens Schwaighofer
fe7c7db004 Change all __class__ to self for class global var 2025-07-09 15:06:27 +09:00
Clemens Schwaighofer
d500b7d473 Log update with listener Queues and color highlight for console
enables Queues if multiprocessing.Queue() is set in the "log_queue" setting

Now a logger "Log.init_worker_logging" can be attached to the ProcessPoolExecutor,
the init args must be the log_queue set

example:

```py
with concurrent.futures.ProcessPoolExecutor(
    max_workers=max_forks,
    initializer=Log.init_worker_logging,
    initargs=(log_queue,)
) as executor:
````

Move all settings into a settings argument, the structure is defined in LogSettings.
Default settings are in Log.DEFAULT_LOG_SETTINGS

Only log path and log name are parameters

Color output for console is on default enabled, disable via "console_color_output_enabled"
The complete console output can be stopped with "console_enabled"
2025-07-09 14:41:53 +09:00
Clemens Schwaighofer
2d197134f1 log splitter output make length configurable 2025-07-09 09:44:55 +09:00
Clemens Schwaighofer
2fa031f6ee Comment out log handlers until we rebuild the logging class 2025-07-08 10:01:09 +09:00
Clemens Schwaighofer
f38cce1c1d Rename src CoreLibs to corelibs 2025-07-08 09:58:33 +09:00
Clemens Schwaighofer
52dd1e7b73 Fix base folder name, must be lower case 2025-07-08 09:56:43 +09:00