CoreLibs for Python

Warning

This is pre-production, location of methods and names of paths can change

This will be split up into modules per file and this will be just a collection holder

This is a pip package that can be installed into any project and covers the following parts

• logging update with exception logs
• requests wrapper for easier auth pass on access
• dict fingerprinting
• sending email
• jmespath search
• json helpers for conten replace and output
• dump outputs for data for debugging
• progress printing
• string formatting, time creation, byte formatting
• Enum base class
• SQLite simple IO class
• Symmetric encryption

Current list

• config_handling: simple INI config file data loader with check/convert/etc
• csv_interface: csv dict writer/reader helper
• debug_handling: various debug helpers like data dumper, timer, utilization, etc
• db_handling: SQLite interface class
• encyption_handling: symmetric encryption
• email_handling: simple email sending
• file_handling: crc handling for file content and file names, progress bar
• json_handling: jmespath support and json date support, replace content in dict with json paths
• iterator_handling: list and dictionary handling support (search, fingerprinting, etc)
• logging_handling: extend log and also error message handling
• requests_handling: requests wrapper for better calls with auth headers
• script_handling: pid lock file handling, abort timer
• string_handling: byte format, datetime format, datetime compare, hashing, string formats for numbers, double byte string format, etc
• var_handling: var type checkers, enum base class

Unfinished

• csv_handling/csv_interface: The CSV DictWriter interface is just in a very basic way implemented
• script_handling/script_helpers: No idea if there is need for this, tests are written but not finished

UV setup

uv must be installed

How to publish

Have the following setup in project.toml

[[tool.uv.index]]
name = "opj-pypi"
url = "https://git.egplusww.jp/api/packages/PyPI/pypi/simple/"
publish-url = "https://git.egplusww.jp/api/packages/PyPI/pypi"
explicit = true

uv build
uv publish --index opj-pypi --token <gitea token>

Use package

We must set the full index URL here because we run with "--no-project"

uv run --with corelibs --index opj-pypi=https://git.egplusww.jp/api/packages/PyPI/pypi/simple/ --no-project -- python -c "import corelibs"

Python tests

All python tests are the tests/ folder. They are structured by the source folder layout

run them with

uv run pytest

Get a coverate report

uv run pytest --cov=corelibs
uv run pytest --cov=corelibs --cov-report=term-missing

Other tests

In the test-run folder usage and run tests are located, runt them below

uv run test-run/<script>

How to install in another project

This will also add the index entry

uv add corelibs --index opj-pypi=https://git.egplusww.jp/api/packages/PyPI/pypi/simple/

Python venv setup

After clone, run the command below to install all dependenciss

uv sync

NOTE on TLS problems

[!warning] TLS problems with Netskope

If the Netskope service is running all uv runs will fail unless either --native-tls is set or the enviroment variable SSL_CERT_FILE is set, see blow

export SSL_CERT_FILE='/Library/Application Support/Netskope/STAgent/data/nscacert_combined.pem'