Add missing OPT_REMOTE to commands

check, compact commands had OPT_REMOTE missing
Test for different command lookup check
2023-05-10 16:30:23 +09:00 · 2022-11-22 09:36:30 +09:00 · 2022-11-14 20:27:42 +09:00 · 2022-08-30 10:00:05 +09:00 · 2022-08-30 09:56:20 +09:00 · 2022-08-08 14:51:26 +09:00
17 changed files with 1440 additions and 486 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -6,3 +6,4 @@ borg.backup.*.include
 borg.backup.*.exclude
 borg.backup.*.schema-only
 borg.backup.*.init
+borg.backup.*.compact
--- a/Readme.md
+++ b/Readme.md
@@ -4,6 +4,31 @@ These scripts are wrappers around the main borg backup scripts.

 Modules for plain file backup, mysql and postgresql backup exists.

+## IMPORTANT NOTICE FOR UPGRADE TO VERSION 4.0 OR HIGHER
+
+*VERSION 4.0* CHANGE
+
+Version 4.0 introduces default borg repository name with `-file` for the `file` module. The repository has to be renamed manual before the next backup or the backup will fail.
+
+*Example:*
+
+Old backup name
+
+```sh
+BACKUP_FILE="some-backup-data.borg"
+```
+
+Then the file need to be renamed the following way:
+`mv some-backup-data.borg some-backup-data-file.borg`
+
+Below changes have to be done after the `file` module backup has been renamed.
+
+With Version 4.0 all backup sets are prefixed with the module name and a comma. For exmaple the files backup will have backup set "file,YYYY-MM-DD" as base name.
+
+To make sure prune of archives will work the `_borg_backup_set_prefix_cleanup.sh` script has to be run once. It has the same config (-c), debug (-d) and dry run (-n) options like the main scripts. It is recommended to run with the dry-run script first and see that the list of chagnes matches the expectation.
+
+The zabbix module has the prefix changed from `zabbix-settings-` to `zabbix,settings-` to match the new archive set rules
+
 ## Recommended setup

 git clone this repostory into the target folder:
@@ -16,6 +41,84 @@ Now the core scripts can be updated with a simple

 No settings files will be overwritten

+## Possible command line options
+
+### `-c <config folder>`
+
+if this is not given, /usr/local/scripts/borg/ is used
+
+### `-L <log folder>`
+
+override config set and default log folder
+
+### `-T <tag>`
+
+create one time stand alone backup prefixed with tag name
+
+### `-D <tag backup set>`
+
+remove a tagged backup set, full name must be given
+
+### `-b <borg executable>`
+
+override the default borg executable found in path
+
+### `-P`
+
+print list of archives created
+
+### `-V`
+
+verify if repository exists, if not abort
+
+### `-e`
+
+exit after running verify `-V`
+
+### `-I`
+
+init repository (must be run first)
+
+### `-Z`
+
+run `borg compact` over given repository
+
+### `-C`
+
+run `borg check` over given repository
+
+#### `-y`
+
+Add `--verify-data` to `borg check`. Only works with `-C`
+
+#### `-p <prefix|glob>`
+
+Only `borg check` data that has given prefix or glob (with *). Only works with `-C`
+
+### `-i`
+
+print out only info
+
+### `-l`
+
+list files during backup
+
+### `-v`
+
+be verbose
+
+### `-d`
+
+debug output all commands
+
+### `-n`
+
+only do dry run
+
+### `-h`
+
+this help page
+
 ## Basic Settings

 `borg.backup.settings`
@@ -27,7 +130,6 @@ LOG_FOLDER: default `/var/log/borg.backup/`
 TARGET_FOLDER: must be set to a path where the backups can be written
 BACKUP_FILE: the folder inside the TARGET_FOLDER that is the target for borg. Must end with `.borg`

-
 Note: BACKUP_FILE is the base name. For all except file (current) a module suffix will be added:

 eg:
@@ -35,24 +137,42 @@ eg:

 If `FILE_REPOSITORY_COMPATIBLE` is set to `false` in the borg.backup.file.settings then the file borg name will have `-file` added too. Currently this is not added to stay compatible with older scripts

-All module settings files can have the following prefixed with `SUB_` to override master settings:
- * SUB_BACKUP_FILE
- * SUB_COMPRESSION
- * SUB_COMPRESSION_LEVEL
- * SUB_BACKUP_SET
- * SUB_KEEP_LAST
- * SUB_KEEP_HOURS
- * SUB_KEEP_DAYS
- * SUB_KEEP_WEEKS
- * SUB_KEEP_MONTHS
- * SUB_KEEP_YEARS
- * SUB_KEEP_WITHIN
+All below have default values if not set in the main settings file

-## Setup backup via SSH to remote host
+* COMPRESSION: zstd
+* COMPRESSION_LEVEL: 3
+* ENCRYPTION: none
+* FORCE_VERIFY: false
+* COMPACT_INTERVAL: 1
+* CHECK_INTERVAL: none
+* KEEP_LAST: 0
+* KEEP_HOURS: 0
+* KEEP_DAYS: 7
+* KEEP_WEEKS: 4
+* KEEP_MONTHS: 6
+* KEEP_YEARS: 1
+
+All module settings files can have the following prefixed with `SUB_` to override master settings:
+
+* SUB_BACKUP_FILE
+* SUB_COMPRESSION
+* SUB_COMPRESSION_LEVEL
+* SUB_COMPACT_INTERVAL
+* SUB_CHECK_INTERVAL
+* SUB_BACKUP_SET
+* SUB_KEEP_LAST
+* SUB_KEEP_HOURS
+* SUB_KEEP_DAYS
+* SUB_KEEP_WEEKS
+* SUB_KEEP_MONTHS
+* SUB_KEEP_YEARS
+* SUB_KEEP_WITHIN
+
+## Setup backup via SSH to remote host on `borg.backup.settings`

 For this the following settings are from interest

-```
+```sh
 TARGET_USER="";
 TARGET_HOST="";
 TARGET_PORT="";
@@ -62,14 +182,29 @@ Note that if `.ssh/config` is used only `TARGET_HOST` needs to be set. Recommene

 and `TARGET_BORG_PATH="";` if the target borg is in a non default path

+## Override borg executable in `borg.backup.settings`
+
+`BORG_EXECUTABLE="<full path to borg>"`
+
+## Note on CHECK_INTERVAL and SUB_CHECK_INTERVAL
+
+If set to empty or 0 it will not run an automatic check. If set to 1 it will run a check after each backup. Any other value means days differente to the last check.
+
+Running check manually (`-C`) will not reset the last check timestamp.
+
+Automatic checks always add `--verify-data`, with manual `-C` the option `-y` has to be set.
+
 ## File backup settings

+On new setups it is recommended to use the `borg.backup.file.setings` and set
+`FILE_REPOSITORY_COMPATIBLE`
+to `true`
+
 ### Config variables

-
 ### Control files

-```
+```sh
 backup.borg.file.include
 backup.borg.file.exclude
 ```
@@ -83,16 +218,16 @@ backup.borg.file.exclude
 This script must be run as the postgres user, normaly `postgres`.
 The postgres user must be added to the backup group for this, so that the basic init file can be created in the borg base folder.

-### Config variables
+### PostgreSQL Config variables

 Variable | Default | Description
 | - | - | - |
 DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema
 DATABASE_USER | | overide username to connect to database

-### Control files
+### PostgreSQL Control files

-```
+```sh
 backup.borg.pgsql.include
 backup.borg.pgsql.exclude
 backup.borg.pgsql.schema-only
@@ -102,16 +237,16 @@ backup.borg.pgsql.schema-only

 If non root ident authentication run is used, be sure that the `mysql` user is in the backup group.

-### Config variables
+### MySQL Config variables

 Variable | Default | Description
 | - | - | - |
 DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema
 MYSQL_DB_CONFIG | | override file for connection. In modern mariaDB installations it is rcommended to run the script as root or mysql user and use the ident authentication instead.

-### Control files
+### MySQLControl files

-```
+```sh
 backup.borg.mysql.include
 backup.borg.mysql.exclude
 backup.borg.mysql.schema-only
@@ -122,7 +257,7 @@ backup.borg.mysql.schema-only
 Note that the backup needs the GIT_USER set that runs gitea.
 This user is neede to create the temporary dump folder and access for the git files and database.

-### Config Variables
+### gitea Config Variables

 Variable | Default | Description
 | - | - | - |
@@ -131,14 +266,13 @@ GITEA_TMP | /tmp/gitea/ | Where the temporary dump files from the backup are sto
 GITEA_BIN | /usr/local/bin/gitea | Where the gitea binary is located |
 GITEA_CONFIG | /etc/gitea/app.ini | The configuration file for gitea |

-
-### Control files
+### gitea Control files

 There are no control files for gitea backup

 ## zabbix config backup settings

-### Config Variables
+### zabbix Config Variables

 Variable | Default | Description
 | - | - | - |
@@ -147,6 +281,24 @@ ZABBIX_DATABASE | '' | Must be set as either psql or mysql
 ZABBIX_CONFIG | '' | if not set uses default location
 ZABBIX_UNKNOWN_TABLES | '' | if set, changed to -f (force)

-### Control files
+### zabbix Control files

 There are no control files for zabbix settings backup
+
+## File connection
+
+Running any of the commands below
+
+* borg.backup.file.sh
+* borg.backup.gitea.sh
+* borg.backup.mysql.sh
+* borg.backup.pgsql.sh
+* borg.backup.zabbix.sh
+
+1) Run `borg.backup.functions.init.sh` (always)
+2) Run `borg.backup.functions.verify.sh` (always)
+3) (other code in "file" module)
+4) Run `borg.backup.functions.info.sh` (always)
+5) Run `borg.backup.functions.compact.sh` (not if one time tag)
+6) Run `borg.backup.functions.check.sh` (not if one time tag)
+7) Run `borg.backup.functions.close.sh` (always, can be called on error too)
--- a/_borg_backup_set_prefix_cleanup.sh
+++ b/_borg_backup_set_prefix_cleanup.sh
@@ -0,0 +1,140 @@
+#!/usr/bin/env bash
+
+# this will fix backup sets name
+# must have a target call for
+# file
+# gitea
+# mysql
+# pgsql
+# zabbix-settings-
+
+# debug and dry run
+DEBUG=0;
+DRYRUN=0;
+PGSQL_SUDO=1;
+# options
+OPT_REMOTE="";
+PGSQL_SUDO_USER="postgres";
+# basic settings needed
+TARGET_USER="";
+TARGET_HOST="";
+TARGET_PORT="";
+TARGET_BORG_PATH="";
+TARGET_FOLDER="";
+# base folder
+BASE_FOLDER="/usr/local/scripts/borg/";
+# those are the valid modules
+MODULE_LIST="file gitea mysql pgsql zabbix"
+
+# basic options
+# -c for config file override
+# -n for dry run test
+while getopts ":c:snd" opt; do
+	case "${opt}" in
+		c|config)
+			BASE_FOLDER=${OPTARG};
+			;;
+		s|nosudo)
+			PGSQL_SUDO=0;
+			;;
+		d|debug)
+			DEBUG=1;
+			;;
+		n|dryrun)
+			DRYRUN=1;
+			;;
+		:)
+			echo "Option -$OPTARG requires an argument."
+			;;
+		\?)
+			echo -e "\n Option does not exist: ${OPTARG}\n";
+			usage;
+			exit 1;
+			;;
+	esac;
+done;
+
+[[ "${BASE_FOLDER}" != */ ]] && BASE_FOLDER=${BASE_FOLDER}"/";
+if [ ! -d "${BASE_FOLDER}" ]; then
+	echo "Base folder not found: ${BASE_FOLDER}";
+	exit 1;
+fi;
+SETTINGS_FILE="borg.backup.settings";
+if [ ! -f "${BASE_FOLDER}${SETTINGS_FILE}" ]; then
+	echo "Could not find: ${BASE_FOLDER}${SETTINGS_FILE}";
+	exit;
+fi;
+. "${BASE_FOLDER}${SETTINGS_FILE}";
+
+if [ ! -z "${TARGET_BORG_PATH}" ]; then
+	OPT_REMOTE="--remote-path="$(printf "%q" "${TARGET_BORG_PATH}");
+fi;
+export BORG_BASE_DIR="${BASE_FOLDER}";
+export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes";
+export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes";
+
+ORIG_BACKUPFILE=${BACKUP_FILE};
+for MODULE in ${MODULE_LIST}; do
+	echo "************* MODULE: ${MODULE}";
+	BACKUP_FILE=${ORIG_BACKUPFILE};
+	BACKUP_FILE=${BACKUP_FILE/.borg/-${MODULE,,}.borg};
+	TARGET_FOLDER=${TARGET_FOLDER%/}
+	TARGET_FOLDER=${TARGET_FOLDER#/}
+	# and add slash front and back and escape the path
+	TARGET_FOLDER=$(printf "%q" "/${TARGET_FOLDER}/");
+	if [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then
+		TARGET_SERVER="ssh://${TARGET_USER}@${TARGET_HOST}:${TARGET_PORT}/";
+	# host/port
+	elif [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then
+		TARGET_SERVER="ssh://${TARGET_HOST}:${TARGET_PORT}/";
+	# user/host
+	elif [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ]; then
+		TARGET_SERVER="${TARGET_USER}@${TARGET_HOST}:";
+	# host
+	elif [ ! -z "${TARGET_HOST}" ]; then
+		TARGET_SERVER="${TARGET_HOST}:";
+	fi;
+	# we dont allow special characters, so we don't need to special escape it
+	REPOSITORY="${TARGET_SERVER}${TARGET_FOLDER}${BACKUP_FILE}";
+	# set sudo prefix for postgres so the cache folder stays the same
+	# if run as root then the foloders below have to have the user set to postgres again
+	# .config/borg/security/<postgresql repo id>
+	# .cache/borg/<postgresql repo id>
+	CMD_PREFIX="";
+	# only sudo to pgsql if sudo is set to true
+	if [ "${MODULE}" = "pgsql" ] && [ "${PGSQL_SUDO}" = "1" ]; then
+		CMD_PREFIX="sudo -E -u ${PGSQL_SUDO_USER} ";
+	fi;
+	echo "==== REPOSITORY: ${REPOSITORY}";
+	borg list ${OPT_REMOTE} --format '{archive}{NL}' ${REPOSITORY}|grep -v "${MODULE},"|
+	while read i; do
+		# for gitea, zabbix we do not ADD we RENAME
+		if [ "${MODULE}" = "gitea" ]; then
+			# if just date, add gitea,
+			# else rename
+			if [ ! -z "${i##gitea*}" ]; then
+				target_name="${MODULE},${i}";
+			else
+				target_name=$(echo $i | sed -e 's/gitea-/gitea,/');
+			fi;
+		elif [ "${MODULE}" = "zabbix" ]; then
+			# if zabbix is missing, prefix
+			if [ ! -z "${i##zabbix*}" ]; then
+				target_name="${MODULE},${i}";
+			else
+				target_name=$(echo $i | sed -e 's/zabbix-settings-/zabbix,settings-/');
+			fi;
+		else
+			target_name="${MODULE},${i}";
+		fi;
+		echo "- Rename from: ${i} to: ${target_name}";
+		if [ ${DEBUG} -eq 1 ]; then
+			echo "${CMD_PREFIX}borg rename ${OPT_REMOTE} -p -v ${REPOSITORY}::${i} ${target_name}";
+		fi;
+		if [ ${DRYRUN} -eq 0 ]; then
+			${CMD_PREFIX}borg rename ${OPT_REMOTE} -p -v ${REPOSITORY}::${i} ${target_name};
+		fi;
+	done;
+done;
+
+# __END__
--- a/borg.backup.file.settings-default
+++ b/borg.backup.file.settings-default
@@ -2,8 +2,8 @@

 # rename to borg.backup.file.settings to use

-# set to false to use -file, current default is "true"
-#FILE_REPOSITORY_COMPATIBLE="false"
+# set to true to use old setting without -file in backup name
+#FILE_REPOSITORY_COMPATIBLE="true"

 # override settings in borg.backup.settings with SUB_ prefix
 # valid for BACKUP_FILE, BACKUP_SET, COMPRESSION*, KEEP_*
--- a/borg.backup.file.sh
+++ b/borg.backup.file.sh
@@ -1,30 +1,35 @@
 #!/usr/bin/env bash

-# Run -I first to initialize repository
-# There are no automatic repository checks unless -C is given
+# Plain file backup

 # set last edit date + time
 MODULE="file";
-MODULE_VERSION="1.0.0";
+MODULE_VERSION="1.2.3";

 DIR="${BASH_SOURCE%/*}"
 if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 . "${DIR}/borg.backup.functions.init.sh";

 # include and exclude file
-INCLUDE_FILE="borg.backup.file.include";
-EXCLUDE_FILE="borg.backup.file.exclude";
-# init check file
-BACKUP_INIT_CHECK="borg.backup.file.init";
+INCLUDE_FILE="borg.backup.${MODULE}.include";
+EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
+# init verify, compact and check file
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";

-. "${DIR}/borg.backup.functions.check.sh";
+# verify valid data
+. "${DIR}/borg.backup.functions.verify.sh";

 # exit if include file is missing
 if [ ! -f "${BASE_FOLDER}${INCLUDE_FILE}" ]; then
 	echo "[! $(date +'%F %T')] The include folder file ${INCLUDE_FILE} is missing";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
-echo "--- [INCLUDE: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+printf "${PRINTF_SUB_BLOCK}" "INCLUDE" "$(date +'%F %T')" "${MODULE}";
 # folders to backup
 FOLDERS=();
 # this if for debug output with quoted folders
@@ -87,9 +92,9 @@ while read include_folder; do
 	fi;
 done<"${BASE_FOLDER}${INCLUDE_FILE}";

-# exclude list
-if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
-	echo "--- [EXCLUDE: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+# exclude list, only if file exists and is larger than zero
+if [ -s "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
+	printf "${PRINTF_SUB_BLOCK}" "EXCLUDE" "$(date +'%F %T')" "${MODULE}";
 	# check that the folders in that exclude file are actually valid,
 	# remove non valid ones and warn
 	#TMP_EXCLUDE_FILE=$(mktemp --tmpdir ${EXCLUDE_FILE}.XXXXXXXX); # non mac
@@ -142,20 +147,23 @@ if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
 		fi;
 	done<"${BASE_FOLDER}${EXCLUDE_FILE}";
 	# avoid blank file add by checking if the tmp file has a size >0
-	if [ -s "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
+	if [ -s "${TMP_EXCLUDE_FILE}" ]; then
 		OPT_EXCLUDE="--exclude-from=${TMP_EXCLUDE_FILE}";
 	fi;
 fi;
+
+# set a special file prefix
+BACKUP_SET_PREFIX="${MODULE},";
 # add the repository set before we add the folders
 # base command
-COMMAND="borg create -v ${OPT_LIST} ${OPT_PROGRESS} ${OPT_COMPRESSION} -s ${OPT_REMOTE} ${OPT_EXCLUDE} ";
+COMMAND="${BORG_COMMAND} create -v ${OPT_LIST} ${OPT_PROGRESS} ${OPT_COMPRESSION} -s ${OPT_REMOTE} ${OPT_EXCLUDE} ";
 # add repoistory, after that the folders will be added on call
-COMMAND=${COMMAND}${REPOSITORY}::${BACKUP_SET};
+COMMAND=${COMMAND}${REPOSITORY}::${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${BACKUP_SET};
 # if info print info and then abort run
 . "${DIR}/borg.backup.functions.info.sh";

 if [ $FOLDER_OK -eq 1 ]; then
-	echo "--- [BACKUP: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "BACKUP" "$(date +'%F %T')" "${MODULE}";
 	# show command
 	if [ ${DEBUG} -eq 1 ]; then
 		echo $(echo ${COMMAND} | sed -e 's/[ ][ ]*/ /g') ${FOLDERS_Q[*]};
@@ -172,19 +180,28 @@ if [ $FOLDER_OK -eq 1 ]; then
 	fi;
 else
 	echo "[! $(date +'%F %T')] No folders where set for the backup";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;

-# clean up, always verbose
-echo "--- [PRUNE : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
-# build command
-COMMAND="borg prune ${OPT_REMOTE} -v -s --list ${PRUNE_DEBUG} ${KEEP_OPTIONS[*]} ${REPOSITORY}";
-echo "Prune repository with keep${KEEP_INFO:1}";
-if [ ${DEBUG} -eq 1 ]; then
+# clean up, always verbose, but only if we do not run one time tag
+if [ -z "${ONE_TIME_TAG}" ]; then
+	printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}";
+	# build command
+	COMMAND="${BORG_COMMAND} prune ${OPT_REMOTE} -v --list ${OPT_PROGRESS} ${DRY_RUN_STATS} -P ${BACKUP_SET_PREFIX} ${KEEP_OPTIONS[*]} ${REPOSITORY}";
+	echo "Prune repository with keep${KEEP_INFO:1}";
+	if [ ${DEBUG} -eq 1 ]; then
 		echo "${COMMAND//#/ }" | sed -e 's/[ ][ ]*/ /g';
+	fi;
+	# for the IFS="#" to work we need to replace options spaces with exactly ONE #
+	$(echo "${COMMAND}" | sed -e 's/[ ][ ]*/#/g') 2>&1 || echo "[!] Borg prune aborted";
+	# if this is borg version >1.2 we need to run compact after prune
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
+else
+	echo "[#] No prune with tagged backup";
 fi;
-# for the IFS="#" to work we need to replace options spaces with exactly ONE #
-$(echo "${COMMAND}" | sed -e 's/[ ][ ]*/#/g') 2>&1 || echo "[!] Borg prune aborted";

 . "${DIR}/borg.backup.functions.close.sh";

--- a/borg.backup.functions.check.sh
+++ b/borg.backup.functions.check.sh
@@ -1,320 +1,83 @@
 #!/usr/bin/env bash

-# start time in seconds
-START=$(date +'%s');
-# start logging from here
-exec &> >(tee -a "${LOG}");
-echo "=== [START : $(date +'%F %T')] ==[${MODULE}]====================================>";
-# show info for version always
-echo "Script version: ${VERSION}";
-# show type
-echo "Backup module : ${MODULE}";
-echo "Module version: ${MODULE_VERSION}";
-# show base folder always
-echo "Base folder   : ${BASE_FOLDER}";
-
-# if force check is true set CHECK to 1unless INFO is 1
-# Needs bash 4.0 at lesat for this
-if [ "${FORCE_CHECK,,}" = "true" ] && [ ${INFO} -eq 0 ]; then
-	CHECK=1;
-	if [ ${DEBUG} -eq 1 ]; then
-		echo "Force repository check";
-	fi;
-fi;
-
-# remote borg path
-if [ ! -z "${TARGET_BORG_PATH}" ]; then
-	if [[ "${TARGET_BORG_PATH}" =~ \ |\' ]]; then
-		echo "Space found in ${TARGET_BORG_PATH}. Aborting";
-		echo "There are issues with passing on paths with spaces"
-		echo "as parameters"
-		exit;
-	fi;
-	OPT_REMOTE="--remote-path="$(printf "%q" "${TARGET_BORG_PATH}");
-fi;
-
-if [ -z "${TARGET_FOLDER}" ]; then
-	echo "[! $(date +'%F %T')] No target folder has been set yet";
-	exit 1;
-else
-	# There are big issues with TARGET FOLDERS with spaces
-	# we should abort anything with this
-	if [[ "${TARGET_FOLDER}" =~ \ |\' ]]; then
-		echo "Space found in ${TARGET_FOLDER}. Aborting";
-		echo "There is some problem with passing paths with spaces as";
-		echo "repository base folder"
-		exit;
-	fi;
-
-	# This does not care for multiple trailing or leading slashes
-	# it just makes sure we have at least one set
-	# for if we have a single slash, remove it
-	TARGET_FOLDER=${TARGET_FOLDER%/}
-	TARGET_FOLDER=${TARGET_FOLDER#/}
-	# and add slash front and back and escape the path
-	TARGET_FOLDER=$(printf "%q" "/${TARGET_FOLDER}/");
-fi;
-
-# if we have user/host then we build the ssh command
-TARGET_SERVER='';
-# allow host only (if full setup in .ssh/config)
-# user@host OR ssh://user@host:port/ IF TARGET_PORT is set
-# user/host/port
-if [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then
-	TARGET_SERVER="ssh://${TARGET_USER}@${TARGET_HOST}:${TARGET_PORT}/";
-# host/port
-elif [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then
-	TARGET_SERVER="ssh://${TARGET_HOST}:${TARGET_PORT}/";
-# user/host
-elif [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ]; then
-	TARGET_SERVER="${TARGET_USER}@${TARGET_HOST}:";
-# host
-elif [ ! -z "${TARGET_HOST}" ]; then
-	TARGET_SERVER="${TARGET_HOST}:";
-fi;
-# we dont allow special characters, so we don't need to special escape it
-REPOSITORY="${TARGET_SERVER}${TARGET_FOLDER}${BACKUP_FILE}";
-echo "Repository    : ${REPOSITORY}";
-
-# check compression if given is valid and check compression level is valid if given
-if [ ! -z "${COMPRESSION}" ]; then
-	# valid compression
-	if [ "${COMPRESSION}" = "lz4" ] || [ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ] || [ "${COMPRESSION}" = "zstd" ]; then
-		OPT_COMPRESSION="-C=${COMPRESSION}";
-		# if COMPRESSION_LEVEL, check it is a valid regex
-		# for zlib, zstd, lzma
-		if [ ! -z "${COMPRESSION_LEVEL}" ] && ([ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ] || [ "${COMPRESSION}" = "zstd" ]); then
-			MIN_COMPRESSION=0;
-			MAX_COMPRESSION=0;
-			case "${COMPRESSION}" in
-				zlib|lzma)
-					MIN_COMPRESSION=0;
-					MAX_COMPRESSION=9;
-					;;
-				zstd)
-					MIN_COMPRESSION=1;
-					MAX_COMPRESSION=22;
-					;;
-				*)
-					MIN_COMPRESSION=0;
-					MAX_COMPRESSION=0;
-					;;
-			esac;
-			# if [ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ]
-			# 	MIN_COMPRESSION=0;
-			# 	MAX_COMPRESSION=9;
-			# elif [ "${COMPRESSION}" = "zstd" ]; then
-			# 	MIN_COMPRESSION=1;
-			# 	MAX_COMPRESSION=22;
-			# fi;
-			error_message="[! $(date +'%F %T')] Compression level for ${COMPRESSION} needs to be a numeric value between ${MIN_COMPRESSION} and ${MAX_COMPRESSION}: ${COMPRESSION_LEVEL}";
-			if ! [[ "${COMPRESSION_LEVEL}" =~ ${REGEX_NUMERIC} ]]; then
-				echo ${error_message};
-				exit 1;
-			elif [ ${COMPRESSION_LEVEL} -lt ${MIN_COMPRESSION} ] || [ ${COMPRESSION_LEVEL} -gt ${MAX_COMPRESSION} ]; then
-				echo ${error_message};
-				exit 1;
-			else
-				OPT_COMPRESSION=${OPT_COMPRESSION}","${COMPRESSION_LEVEL};
-			fi;
-		fi;
-	else
-		echo "[! $(date +'%F %T')] Compress setting need to be lz4, zstd, zlib or lzma. Or empty for no compression: ${COMPRESSION}";
-		exit 1;
-	fi;
-fi;
-
-# home folder, needs to be set if there is eg a HOME=/ in the crontab
-if [ ! -w "${HOME}" ] || [ "${HOME}" = '/' ]; then
-	HOME=$(eval echo "$(whoami)");
-fi;
-
-# build options and info string,
-# also flag BACKUP_SET check if hourly is set
-KEEP_OPTIONS=();
-KEEP_INFO="";
-BACKUP_SET_CHECK=0;
-if [ ${KEEP_LAST} -gt 0 ]; then
-	KEEP_OPTIONS+=("--keep-last=${KEEP_LAST}");
-	KEEP_INFO="${KEEP_INFO}, last: ${KEEP_LAST}";
-fi;
-if [ ${KEEP_HOURS} -gt 0 ]; then
-	KEEP_OPTIONS+=("--keep-hourly=${KEEP_HOURS}");
-	KEEP_INFO="${KEEP_INFO}, hourly: ${KEEP_HOURS}";
-	BACKUP_SET_CHECK=1;
-fi;
-if [ ${KEEP_DAYS} -gt 0 ]; then
-	KEEP_OPTIONS+=("--keep-daily=${KEEP_DAYS}");
-	KEEP_INFO="${KEEP_INFO}, daily: ${KEEP_DAYS}";
-fi;
-if [ ${KEEP_WEEKS} -gt 0 ]; then
-	KEEP_OPTIONS+=("--keep-weekly=${KEEP_WEEKS}");
-	KEEP_INFO="${KEEP_INFO}, weekly: ${KEEP_WEEKS}";
-fi;
-if [ ${KEEP_MONTHS} -gt 0 ]; then
-	KEEP_OPTIONS+=("--keep-monthly=${KEEP_MONTHS}");
-	KEEP_INFO="${KEEP_INFO}, monthly: ${KEEP_MONTHS}";
-fi;
-if [ ${KEEP_YEARS} -gt 0 ]; then
-	KEEP_OPTIONS+=("--keep-yearly=${KEEP_YEARS}");
-	KEEP_INFO="${KEEP_INFO}, yearly: ${KEEP_YEARS}";
-fi;
-if [ ! -z "${KEEP_WITHIN}" ]; then
-	# check for invalid string. can only be number + H|d|w|m|y
-	if [[ "${KEEP_WITHIN}" =~ ^[0-9]+[Hdwmy]{1}$ ]]; then
-		KEEP_OPTIONS+=("--keep-within=${KEEP_WITHIN}");
-		KEEP_INFO="${KEEP_INFO}, within: ${KEEP_WITHIN}";
-		if [[ "${KEEP_WITHIN}" == *"H"* ]]; then
-			BACKUP_SET_CHECK=1;
-		fi;
-	else
-		echo "[! $(date +'%F %T')] KEEP_WITHIN has invalid string.";
-		exit 1;
-	fi;
-fi;
-# abort if KEEP_OPTIONS is empty
-if [ -z "${KEEP_OPTIONS}" ]; then
-	echo "[! $(date +'%F %T')] It seems no KEEP_* entries where set in a valid format.";
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
 	exit 1;
 fi;
-# set BACKUP_SET if empty, set to Year-month-day
-if [ -z "${BACKUP_SET}" ]; then
-	BACKUP_SET="{now:%Y-%m-%d}";
-fi;
-# backup set check, and there is no hour entry (%H) in the archive string
-# we add T%H:%M:%S in this case, before the last }
-if [ ${BACKUP_SET_CHECK} -eq 1 ] && [[ "${BACKUP_SET}" != *"%H"* ]]; then
-	BACKUP_SET=$(echo "${BACKUP_SET}" | sed -e "s/}/T%H:%M:%S}/");
+
+# run borg check (NOT REPAIR)
+
+RUN_CHECK=0;
+if [ $# -ge 1 ] && [ "$1" = "auto" ]; then
+	# strip any spaces and convert to int
+	CHECK_INTERVAL=$(echo "${CHECK_INTERVAL}" | sed -e 's/ //g');
+	# not a valid check interval, do no check
+	if [ -z "${CHECK_INTERVAL##*[!0-9]*}" ]; then
+		CHECK_INTERVAL=0;
+	fi;
+	# get current date timestmap
+	CURRENT_DATE=$(date +%s);
+	# if =1 always ok
+	if [ ${CHECK_INTERVAL} -eq 1 ]; then
+		RUN_CHECK=1;
+		# always add verify data for automatic check
+		OPT_CHECK_VERIFY_DATA="--verify-data";
+		# set new check time here
+		echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_CHECK_FILE}";
+	elif [ ${CHECK_INTERVAL} -gt 1 ]; then
+		# else load last timestamp and check if today - last time stamp > days
+		if [ -z "${LAST_CHECK_DATE}" ]; then
+			LAST_CHECK_DATE=$(cat "${BASE_FOLDER}${BACKUP_CHECK_FILE}" 2>/dev/null | sed -e 's/ //g');
+		fi;
+		# file date is not a timestamp
+		if [ -z "${LAST_CHECK_DATE##*[!0-9]*}" ]; then
+			LAST_CHECK_DATE=0;
+		fi;
+		# if the difference greate than check date, run. CHECK INTERVAL is in days
+		if [ $(($CURRENT_DATE-$LAST_CHECK_DATE)) -ge $((${CHECK_INTERVAL}*86400)) ]; then
+			RUN_CHECK=1;
+			# always add verify data for automatic check
+			OPT_CHECK_VERIFY_DATA="--verify-data";
+			# set new check time here
+			echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_CHECK_FILE}";
+		fi;
+	fi;
+elif [ ${CHECK} -eq 1 ]; then
+	RUN_CHECK=1;
 fi;

-# for folders list split set to "#" and keep the old setting as is
-_IFS=${IFS};
-IFS="#";
-# turn off for non file
-if [ "${MODULE}" != "file" ]; then
+if [ ${RUN_CHECK} -eq 1 ]; then
+	# run borg check command
 	IFS=${_IFS};
-fi;
-
-# borg call, replace ##...## parts during run
-# used in all modules, except 'file'
-_BORG_CALL="borg create ${OPT_REMOTE} -v ${OPT_LIST} ${OPT_PROGRESS} ${OPT_COMPRESSION} -s --stdin-name ##FILENAME## ${REPOSITORY}::##BACKUP_SET## -";
-_BORG_PRUNE="borg prune ${OPT_REMOTE} -v -s --list ${PRUNE_DEBUG} -P ##BACKUP_SET_PREFIX## ${KEEP_OPTIONS[*]} ${REPOSITORY}";
-
-# general borg settings
-# set base path to config directory to keep cache/config separated
-export BORG_BASE_DIR="${BASE_FOLDER}";
-# ignore non encrypted access
-export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=${_BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK};
-# ignore moved repo access
-export BORG_RELOCATED_REPO_ACCESS_IS_OK=${_BORG_RELOCATED_REPO_ACCESS_IS_OK};
-# and for debug print that tout
-if [ ${DEBUG} -eq 1 ]; then
-	echo "export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=${_BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK};";
-	echo "export BORG_RELOCATED_REPO_ACCESS_IS_OK=${_BORG_RELOCATED_REPO_ACCESS_IS_OK};";
-	echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
-fi;
-# prepare debug commands only
-COMMAND_EXPORT="export BORG_BASE_DIR=\"${BASE_FOLDER}\";"
-COMMAND_INFO="${COMMAND_EXPORT}borg info ${OPT_REMOTE} ${REPOSITORY}";
-# if the is not there, call init to create it
-# if this is user@host, we need to use ssh command to check if the file is there
-# else a normal check is ok
-# unless explicit given, check is skipped
-if [ ${CHECK} -eq 1 ] || [ ${INIT} -eq 1 ]; then
-	echo "--- [CHECK : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
-	if [ ! -z "${TARGET_SERVER}" ]; then
-		if [ ${DEBUG} -eq 1 ]; then
-			echo "borg info ${OPT_REMOTE} ${REPOSITORY} 2>&1|grep \"Repository ID:\"";
+	printf "${PRINTF_SUB_BLOCK}" "CHECK" "$(date +'%F %T')" "${MODULE}";
+	# repare command
+	OPT_GLOB="";
+	if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then
+		OPT_GLOB="-a '${CHECK_PREFIX}'"
+	elif [ ! -z "${CHECK_PREFIX}" ]; then
+		OPT_GLOB="-P ${CHECK_PREFIX}";
 	fi;
-		# use borg info and check if it returns "Repository ID:" in the first line
-		REPO_CHECK=$(borg info ${OPT_REMOTE} ${REPOSITORY} 2>&1|grep "Repository ID:");
-		# this is currently a hack to work round the error code in borg info
-		# this checks if REPO_CHECK holds this error message and then starts init
-		regex="^Some part of the script failed with an error:";
-		if [[ -z "${REPO_CHECK}" ]] || [[ "${REPO_CHECK}" =~ ${regex} ]]; then
-			INIT_REPOSITORY=1;
-		fi;
-	elif [ ! -d "${REPOSITORY}" ]; then
-		INIT_REPOSITORY=1;
-	fi;
-	# if check but no init and repo is there but init file is missing set it
-	if [ ${CHECK} -eq 1 ] && [ ${INIT} -eq 0 ] && [ ${INIT_REPOSITORY} -eq 0 ] &&
-		[ ! -f "${BASE_FOLDER}${BACKUP_INIT_CHECK}" ]; then
-		# write init file
-		echo "[!] Add missing init check file";
-		echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_CHECK}";
-	fi;
-	# end if checked but repository is not here
-	if [ ${CHECK} -eq 1 ] && [ ${INIT} -eq 0 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then
-		echo "[! $(date +'%F %T')] No repository. Please run with -I flag to initialze repository";
-		exit 1;
-	fi;
-	if [ ${EXIT} -eq 1 ] && [ ${CHECK} -eq 1 ] && [ ${INIT} -eq 0 ]; then
-		echo "Repository exists";
-		echo "For more information run:"
-		echo "${COMMAND_INFO}";
-		echo "=== [END  : $(date +'%F %T')] ==[${MODULE}]====================================>";
-		exit;
-	fi;
-fi;
-if [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then
-	echo "--- [INIT  : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	# debug/dryrun
 	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
-		echo "borg init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY}";
-	fi
-	if [ ${DRYRUN} -eq 0 ]; then
-		# should trap and exit properly here
-		borg init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY};
-		# write init file
-		echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_CHECK}";
-		echo "Repository initialized";
-		echo "For more information run:"
-		echo "${COMMAND_INFO}";
-	fi
-	echo "=== [END  : $(date +'%F %T')] ==[${MODULE}]====================================>";
-	# exit after init
-	exit;
-elif [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 0 ]; then
-	echo "[! $(date +'%F %T')] Repository already initialized";
-	echo "For more information run:"
-	echo "${COMMAND_INFO}";
-	exit 1;
-fi;
-
-# check for init file
-if [ ! -f "${BASE_FOLDER}${BACKUP_INIT_CHECK}" ]; then
-	echo "[! $(date +'%F %T')] It seems the repository has never been initialized."
-	echo "Please run -I to initialize or if already initialzed run with -C for init update."
-	exit 1;
-fi;
-
-# PRINT OUT current data, only do this if REPO exists
-if [ ${PRINT} -eq 1 ]; then
-	echo "--- [PRINT : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
-	FORMAT="{archive} {comment:6} {start} - {end} [{id}] ({username}@{hostname}){NL}"
-	# show command on debug or dry run
-	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
-		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";borg list ${OPT_REMOTE} --format ${FORMAT} ${REPOSITORY}";
+		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY}";
 	fi;
 	# run info command if not a dry drun
 	if [ ${DRYRUN} -eq 0 ]; then
-		borg list ${OPT_REMOTE} --format "${FORMAT}" ${REPOSITORY} ;
-	fi;
-	if [ ${VERBOSE} -eq 1 ]; then
-		echo "";
-		echo "Base command info:"
-		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";borg [COMMAND] ${OPT_REMOTE} ${REPOSITORY}::[BACKUP] [PATH]";
-		echo "Replace [COMMAND] with list for listing or extract for restoring backup data."
-		echo "Replace [BACKUP] with archive name."
-		echo "If no [PATH] is given then all files will be restored."
-		echo "Before extracting -n (dry run) is recommended to use."
-		echo "If archive size is needed the info command with archive name has to be used."
-		echo "When listing (list) data the --format command can be used."
-		echo "Example: \"{mode} {user:6} {group:6} {size:8d} {csize:8d} {dsize:8d} {dcsize:8d} {mtime} {path}{extra} [{health}]{NL}\""
+		# if glob add glob command directly
+		if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then
+		${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} -a "${CHECK_PREFIX}" ${REPOSITORY};
 		else
-		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";borg [COMMAND] ${OPT_REMOTE} [FORMAT] ${REPOSITORY}::[BACKUP] [PATH]";
+			${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY};
+		fi;
+	fi;
+	# print additional info for use --repair command
+	# but only for manual checks
+	if [ ${VERBOSE} -eq 1 ] && [ ${CHECK} -eq 1 ]; then
+		echo "";
+		echo "In case of needed repair: "
+		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} --repair ${OPT_GLOB} ${REPOSITORY}";
+		echo "Before running repair, a copy from the backup should be made because repair might damage a backup"
 	fi;
-	exit;
 fi;

 # __END__
--- a/borg.backup.functions.close.sh
+++ b/borg.backup.functions.close.sh
@@ -1,9 +1,24 @@
 #!/usr/bin/env bash

+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
 # unset borg settings
 unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK
-DURATION=$[ $(date +'%s')-$START ];
-echo "=== [Run time: $(convert_time ${DURATION})]";
-echo "=== [END  : $(date +'%F %T')] ==[${MODULE}]===================>";
+# delete lock file if it exists
+if [ -f "${BASE_FOLDER}${BACKUP_LOCK_FILE}" ]; then
+	rm "${BASE_FOLDER}${BACKUP_LOCK_FILE}";
+fi;
+# error abort without duration and error notice
+if [ $# -ge 1 ] && [ "$1" = "1" ]; then
+	printf "${PRINTF_MASTER_BLOCK}" "ERROR" "$(date +'%F %T')" "${MODULE}";
+else
+	# running time calculation
+	DURATION=$[ $(date +'%s')-$START ];
+	echo "=== [Run time: $(convert_time ${DURATION})]";
+	printf "${PRINTF_MASTER_BLOCK}" "END" "$(date +'%F %T')" "${MODULE}";
+fi;

 # __END__
--- a/borg.backup.functions.compact.sh
+++ b/borg.backup.functions.compact.sh
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
+# compact (only if BORG COMPACT is set)
+# only for borg 1.2
+if [ $(version $BORG_VERSION) -ge $(version "1.2.0") ]; then
+	RUN_COMPACT=0;
+	if [ $# -ge 1 ] && [ "$1" = "auto" ]; then
+		# strip any spaces and convert to int
+		COMPACT_INTERVAL=$(echo "${COMPACT_INTERVAL}" | sed -e 's/ //g');
+		# not a valid compact interval, do no compact
+		if [ -z "${COMPACT_INTERVAL##*[!0-9]*}" ]; then
+			COMPACT_INTERVAL=0;
+		fi;
+		# get current date timestmap
+		CURRENT_DATE=$(date +%s);
+		if [ ${COMPACT_INTERVAL} -eq 1 ]; then
+			RUN_COMPACT=1;
+			# set new compact time here
+			echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_COMPACT_FILE}";
+		elif [ ${COMPACT_INTERVAL} -gt 1 ]; then
+			# else load last timestamp and check if today - last time stamp > days
+			if [ -z "${LAST_COMPACT_DATE}" ]; then
+				LAST_COMPACT_DATE=$(cat "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" 2>/dev/null | sed -e 's/ //g');
+			fi;
+			# file date is not a timestamp
+			if [ -z "${LAST_COMPACT_DATE##*[!0-9]*}" ]; then
+				LAST_COMPACT_DATE=0;
+			fi;
+			# if the difference greate than compact date, run. COMPACT INTERVAL is in days
+			if [ $(($CURRENT_DATE-$LAST_COMPACT_DATE)) -ge $((${COMPACT_INTERVAL}*86400)) ]; then
+				RUN_COMPACT=1;
+				# set new compact time here
+				echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_COMPACT_FILE}";
+			fi;
+		fi;
+	elif [ ${COMPACT} -eq 1 ]; then
+		RUN_COMPACT=1;
+	fi;
+
+	if [ ${RUN_COMPACT} -eq 1 ]; then
+		# reset to normal IFS, so command works here
+		IFS=${_IFS};
+		printf "${PRINTF_SUB_BLOCK}" "COMPACT" "$(date +'%F %T')" "${MODULE}";
+		BORG_COMPACT="${BORG_COMMAND} compact ${OPT_REMOTE} -v ${OPT_PROGRESS} ${REPOSITORY}";
+		if [ ${DEBUG} -eq 1 ]; then
+				echo "${BORG_COMPACT}";
+		fi;
+		if [ ${DRYRUN} -eq 0 ]; then
+			${BORG_COMPACT};
+		fi;
+	fi;
+fi;
+
+# __END__
--- a/borg.backup.functions.info.sh
+++ b/borg.backup.functions.info.sh
@@ -1,14 +1,19 @@
 #!/usr/bin/env bash

+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
 if [ ${INFO} -eq 1 ]; then
-	echo "--- [INFO  : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "INFO" "$(date +'%F %T')" "${MODULE}";
 	# show command on debug or dry run
 	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
-		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";borg info ${OPT_REMOTE} ${REPOSITORY}";
+		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY}";
 	fi;
 	# run info command if not a dry drun
 	if [ ${DRYRUN} -eq 0 ]; then
-		borg info ${OPT_REMOTE} ${REPOSITORY};
+		${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY};
 	fi;
 	if [ "${MODULE}" = "files" ]; then
 		if [ $FOLDER_OK -eq 1 ]; then
@@ -23,7 +28,7 @@ if [ ${INFO} -eq 1 ]; then
 			rm -f "${TMP_EXCLUDE_FILE}";
 		fi;
 	fi;
-	echo "=== [END  : $(date +'%F %T')] ==[${MODULE}]====================================>";
+	. "${DIR}/borg.backup.functions.close.sh";
 	exit;
 fi;

--- a/borg.backup.functions.init.sh
+++ b/borg.backup.functions.init.sh
@@ -1,5 +1,10 @@
 #!/usr/bin/env bash

+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
 set -ETu #-e -o pipefail
 trap cleanup SIGINT SIGTERM ERR

@@ -13,16 +18,25 @@ cleanup() {
 }
 # on exit unset any exported var
 trap "unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK" EXIT;
+# for version compare
+function version {
+	echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }';
+}

 # version for all general files
-VERSION="3.0.2";
+VERSION="4.4.0";

+# borg version and borg comamnd
+BORG_VERSION="";
+BORG_COMMAND="borg";
 # default log folder if none are set in config or option
 _LOG_FOLDER="/var/log/borg.backup/";
 # log file name is set based on BACKUP_FILE, .log is added
 LOG_FOLDER="";
 # should be there on everything
 TEMPDIR="/tmp/";
+# HOSTNAME (as set on server)
+HOSTNAME=$(hostname);
 # creates borg backup based on the include/exclude files
 # if base borg folder (backup files) does not exist, it will automatically init it
 # base folder
@@ -32,15 +46,28 @@ SETTINGS_FILE="borg.backup.settings";
 # include files
 INCLUDE_FILE="";
 EXCLUDE_FILE="";
-# backup folder initialzed check
-BACKUP_INIT_CHECK="";
-# debug/verbose
+# backup folder initialzed verify
+BACKUP_INIT_FILE="";
+BACKUP_INIT_DATE="";
+# file with last compact date
+BACKUP_COMPACT_FILE="";
+# file with last check timestamp
+BACKUP_CHECK_FILE="";
+# one time backup prefix tag, if set will use <tag>.<prefix>-Y-M-DTh:m:s type backup prefix
+ONE_TIME_TAG="";
+DELETE_ONE_TIME_TAG="";
+# check command prefix/glob
+CHECK_PREFIX="";
+# debug/verbose/other flags
 VERBOSE=0;
 LIST=0;
 DEBUG=0;
 DRYRUN=0;
 INFO=0;
+VERIFY=0;
 CHECK=0;
+CHECK_VERIFY_DATA=0;
+COMPACT=0;
 INIT=0;
 EXIT=0;
 PRINT=0;
@@ -51,17 +78,24 @@ _BORG_RELOCATED_REPO_ACCESS_IS_OK="yes";
 # NOTE: to keep the old .borg repository name for file module set this to true
 # if set to false (future) it will add -file to the repository name like for other
 # modules
-FILE_REPOSITORY_COMPATIBLE="true";
+FILE_REPOSITORY_COMPATIBLE="false";
 # other variables
 TARGET_SERVER="";
 REGEX="";
 REGEX_COMMENT="^[\ \t]*#";
 REGEX_GLOB='\*';
 REGEX_NUMERIC="^[0-9]{1,2}$";
+REGEX_ERROR="^Some part of the script failed with an error:";
 PRUNE_DEBUG="";
 INIT_REPOSITORY=0;
 FOLDER_OK=0;
 TMP_EXCLUDE_FILE="";
+# printf strings
+PRINTF_INFO_STRING="%-23s: %s\n";
+PRINTF_MASTER_BLOCK="=== [%-8s: %19s] ==[%s]====================================>\n";
+PRINTF_SUB_BLOCK="|-- [%-8s: %19s] --[%s]------------------------------------>\n";
+PRINTF_SUBEXT_BLOCK="|-- [%-8s: %s: %19s] --[%s]------------------------------------>\n";
+PRINTF_DB_SUB_BLOCK=">>- [%-8s: %s] =======================[%s]====================================>\n";
 # opt flags
 OPT_VERBOSE="";
 OPT_PROGRESS="";
@@ -69,6 +103,7 @@ OPT_LIST="";
 OPT_REMOTE="";
 OPT_LOG_FOLDER="";
 OPT_EXCLUDE="";
+OPT_CHECK_VERIFY_DATA="";
 # config variables (will be overwritten from .settings file)
 TARGET_USER="";
 TARGET_HOST="";
@@ -77,15 +112,36 @@ TARGET_BORG_PATH="";
 TARGET_FOLDER="";
 BACKUP_FILE="";
 SUB_BACKUP_FILE="";
+# OPT is for options set
+OPT_BORG_EXECUTEABLE="";
+# which overrides BORG_EXECUTABLE that can be set in the settings file
+BORG_EXECUTEABLE="";
 # lz4, zstd 1-22 (3), zlib 0-9 (6), lzma 0-9 (6)
-COMPRESSION="zstd";
-COMPRESSION_LEVEL=3;
+DEFAULT_COMPRESSION="zstd";
+DEFAULT_COMPRESSION_LEVEL=3;
+COMPRESSION="";
+COMPRESSION_LEVEL="";
 SUB_COMPRESSION="";
 SUB_COMPRESSION_LEVEL="";
 # encryption settings
-ENCRYPTION="none";
-# force check always
-FORCE_CHECK="false";
+DEFAULT_ENCRYPTION="none";
+ENCRYPTION="";
+# force verify always
+DEFAULT_FORCE_VERIFY="false";
+FORCE_VERIFY="";
+FORCE_CHECK=""; # Deprecated name, use FORCE_VERIFY
+# compact
+DEFAULT_COMPACT_INTERVAL="1";
+LAST_COMPACT_DATE="";
+COMPACT_INTERVAL="";
+SUB_COMPACT_INTERVAL="";
+# check
+# default interval is none
+DEFAULT_CHECK_INTERVAL="";
+LAST_CHECK_DATE="";
+CHECK_INTERVAL="";
+SUB_CHECK_INTERVAL="";
+# backup set names
 BACKUP_SET="";
 SUB_BACKUP_SET="";
 # for database backup only
@@ -113,12 +169,18 @@ OPT_ZABBIX_UNKNOWN_TABLES="";
 # BACKUP SET that includes hour and minute information
 # IF BACKUP_SET is empty, this is automatically added
 # general keep last, if only this is set only last n will be kept
-KEEP_LAST=0;
-KEEP_HOURS=0;
-KEEP_DAYS=7;
-KEEP_WEEKS=4;
-KEEP_MONTHS=6;
-KEEP_YEARS=1;
+DEFAULT_KEEP_LAST=0;
+DEFAULT_KEEP_HOURS=0;
+DEFAULT_KEEP_DAYS=7;
+DEFAULT_KEEP_WEEKS=4;
+DEFAULT_KEEP_MONTHS=6;
+DEFAULT_KEEP_YEARS=1;
+KEEP_LAST="";
+KEEP_HOURS="";
+KEEP_DAYS="";
+KEEP_WEEKS="";
+KEEP_MONTHS="";
+KEEP_YEARS="";
 # in the format of nY|M|d|h|m|s
 KEEP_WITHIN="";
 # sub override init to empty
@@ -137,13 +199,20 @@ function usage()

 	-c <config folder>: if this is not given, ${BASE_FOLDER} is used
 	-L <log folder>: override config set and default log folder
+	-T <tag>: create one time stand alone backup prefixed with tag name
+	-D <tag backup set>: remove a tagged backup set, full name must be given
+	-b <borg executable>: override default path
+	-Z: run compress after prune/backup. Only for borg 1.2 or newer
+	-C: run borg check if repository is ok
+	-y: in combination with -C: add --verify-data
+	-p <archive prefix|glob>: in combinatio with -C: only check archives with prefix or glob
 	-P: print list of archives created
-	-C: check if repository exists, if not abort
-	-E: exit after check
+	-V: verify if repository exists, if not abort
+	-e: exit after verify
 	-I: init repository (must be run first)
-	-v: be verbose
 	-i: print out only info
 	-l: list files during backup
+	-v: be verbose
 	-d: debug output all commands
 	-n: only do dry run
 	-h: this help page
@@ -155,26 +224,52 @@ function usage()
 }

 # set options
-while getopts ":c:L:vldniCEIPh" opt; do
+while getopts ":c:L:T:D:b:p:vldniCVeIPyZh" opt; do
 	case "${opt}" in
 		c|config)
 			BASE_FOLDER=${OPTARG};
 			;;
-		L|log)
+		L|Log)
 			OPT_LOG_FOLDER=${OPTARG};
 			;;
+		T|Tag)
+			ONE_TIME_TAG=${OPTARG};
+			;;
+		D|Delete)
+			DELETE_ONE_TIME_TAG=${OPTARG};
+			;;
+		b|borg)
+			OPT_BORG_EXECUTEABLE=${OPTARG};
+			;;
+		Z|Compact)
+			# will run compact alone
+			COMPACT=1;
+			;;
 		C|Check)
-			# will check if repo is there and abort if not
+			# will run borg check
+			# alt modes --repository-only, --archives-only,
+			# add mode --verify-data
+			# note that --repair has to be called manually is it might damange backups
 			CHECK=1;
 			;;
-		E|Exit)
-			# exit after check
+		y|Verify-Data)
+			CHECK_VERIFY_DATA=1;
+			;;
+		p|prefix-glob)
+			CHECK_PREFIX=${OPTARG};
+			;;
+		V|Verify)
+			# will verify if repo is there and abort if not
+			VERIFY=1;
+			;;
+		e|exit)
+			# exit after verify or init (default off)
 			EXIT=1;
 			;;
 		I|Init)
 			# will check if there is a repo and init it
 			# previoous this was default
-			CHECK=1;
+			VERIFY=1;
 			INIT=1;
 			;;
 		P|Print)
@@ -229,13 +324,55 @@ if [ ! -w "${BASE_FOLDER}" ]; then
 fi;

 # info -i && -C/-I cannot be run together
-if [ ${CHECK} -eq 1 ] || [ ${INIT} -eq 1 ] && [ ${INFO} -eq 1 ]; then
-	echo "Cannot have -i info option and -C check or -I initialize option at the same time";
+if [ ${VERIFY} -eq 1 ] || [ ${INIT} -eq 1 ] && [ ${INFO} -eq 1 ]; then
+	echo "Cannot have -i info option and -V verify or -I initialize option at the same time";
 	exit 1;
 fi;
 # print -P cannot be run with -i/-C/-I together
-if [ ${PRINT} -eq 1 ] || [ ${INIT} -eq 1 ] && [ ${CHECK} -eq 1 ] && [ ${INFO} -eq 1 ]; then
-	echo "Cannot have -P print option and -i info, -C check or -I initizalize option at the same time";
+if [ ${PRINT} -eq 1 ] && ([ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]); then
+	echo "Cannot have -P print option and -i info, -V verify or -I initizalize option at the same time";
+	exit 1;
+fi;
+# if tag is set, you can't have init, verify, info, etc
+if [ ! -z "${ONE_TIME_TAG}" ] && ([ ${PRINT} -eq 1 ] || [ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]); then
+	echo "Cannot have -T '${ONE_TIME_TAG}' option with -i info, -V verify, -I initialize or -P print option at the same time";
+	exit 1;
+fi;
+# verify only alphanumeric, no spaces, only underscore and dash
+if [ ! -z "${ONE_TIME_TAG}" ] && ! [[ "${ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+$ ]]; then
+	echo "One time tag '${ONE_TIME_TAG}' must be alphanumeric with dashes and underscore only.";
+	exit 1;
+elif [ ! -z "${ONE_TIME_TAG}" ]; then
+	# all ok, attach . at the end
+	ONE_TIME_TAG=${ONE_TIME_TAG}".";
+fi;
+# if -D, cannot be with -T, -i, -C, -I, -P
+if [ ! -z "${DELETE_ONE_TIME_TAG}" ] && ([ ! -z "${ONE_TIME_TAG}" ] || [ ${PRINT} -eq 1 ] || [ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]); then
+	echo "Cannot have -D delete tag option with -T one time tag, -i info, -V verify, -I initialize or -P print option at the same time";
+	exit 1;
+fi;
+# -D also must be in valid backup set format
+# ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},(\*-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T\*$ ]]
+if [ ! -z "${DELETE_ONE_TIME_TAG}" ] && ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},([A-Za-z0-9_-]+-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}$ ]] && ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},(\*-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T\*$ ]]; then
+	echo "Delete one time tag '${DELETE_ONE_TIME_TAG}' is in an invalid format. "
+	echo "Please verify existing tags with -P option."
+	echo "For a globing be sure it is in the format of: TAG.MODULE,*-YYYY-MM-DDT*";
+	echo "Note the dash (-) after the first *, also time (T) is a globa (*) must."
+	exit 1;
+fi;
+# -y can't be set without -C
+if [ ${CHECK_VERIFY_DATA} -eq 1 ] && [ ${CHECK} -eq 0 ]; then
+	echo "-y (verify-data) cannot be run without -C (Check) option";
+	exit 1;
+fi;
+# -p can't be set without -C
+if [ ! -z "${CHECK_PREFIX}" ] && [ ${CHECK} -eq 0 ]; then
+	echo "-p (pattern|glob) for check cannot be run without -C (Check) options";
+	exit 1;
+fi;
+# can't have -e if VERIFY or INIT is not set
+if [ ${EXIT} -eq 1 ] && [ ${VERIFY} -eq 0 ] && [ ${INIT} -eq 0 ]; then
+	echo "-e (exit) can only be used with -V (Verify) and -I (Init)";
 	exit 1;
 fi;

@@ -248,12 +385,91 @@ fi;
 if [ ${LIST} -eq 1 ]; then
 	OPT_LIST="--list";
 fi;
+# If dry run, the stats (-s) option cannot be used
 if [ ${DRYRUN} -eq 1 ]; then
-	PRUNE_DEBUG="--dry-run";
+	DRY_RUN_STATS="-n";
+else
+	DRY_RUN_STATS="-s";
+fi;
+if [ ${CHECK_VERIFY_DATA} -eq 1 ] && [ ${CHECK} -eq 1 ]; then
+	OPT_CHECK_VERIFY_DATA="--verify-data";
 fi;

 # read config file
 . "${BASE_FOLDER}${SETTINGS_FILE}";
+
+# if OPTION SET overrides ALL others
+if [ ! -z "${OPT_BORG_EXECUTEABLE}" ]; then
+	BORG_COMMAND="${OPT_BORG_EXECUTEABLE}";
+	if [ ! -f "${BORG_COMMAND}" ]; then
+		echo "borg command not found with option -b: ${BORG_COMMAND}";
+		exit;
+	fi;
+# if in setting file, use this
+elif [ ! -z "${BORG_EXECUTEABLE}" ]; then
+	BORG_COMMAND="${BORG_EXECUTEABLE}";
+	if [ ! -f "${BORG_COMMAND}" ]; then
+		echo "borg command not found with setting: ${BORG_COMMAND}";
+		exit;
+	fi;
+elif ! command -v borg &> /dev/null; then
+# elif [ -z ($command -v borg) ]; then
+	echo "borg backup seems not to be installed, please verify paths";
+	exit;
+fi;
+# verify that this is a borg executable, no detail check
+_BORG_COMMAND_VERIFY=$(${BORG_COMMAND} -V | grep "borg");
+if [[ "${_BORG_COMMAND_VERIFY}" =~ ${REGEX_ERROR} ]]; then
+	echo "Cannot extract borg info from command, is this a valid borg executable?: ${BORG_COMMAND}";
+	exit;
+fi;
+# extract actually borg version from here
+# alt sed to get only numbes: sed -e 's/.* \([0-9]*\.[0-9]*\.[0-9]*\)$/\1/g'
+# or use cut -d " " -f 2 and assume NO space in the first part
+BORG_VERSION=$(${BORG_COMMAND} -V | sed -e 's/borg.* //') 2>&1 || echo "[!] Borg version not estable";
+
+# load default settings for fileds not set
+if [ -z "${COMPRESSION}" ]; then
+	COMPRESSION="${DEFAULT_COMPRESSION}";
+fi;
+if [ -z "${COMPRESSION_LEVEL}" ]; then
+	COMPRESSION_LEVEL="${DEFAULT_COMPRESSION_LEVEL}";
+fi;
+if [ -z "${ENCRYPTION}" ]; then
+	ENCRYPTION="${DEFAULT_ENCRYPTION}";
+fi;
+# check interval override
+if [ -z "${COMPACT_INTERVAL}" ]; then
+	COMPACT_INTERVAL="${DEFAULT_COMPACT_INTERVAL}";
+fi;
+if [ -z "${CHECK_INTERVAL}" ]; then
+	CHECK_INTERVAL="${DEFAULT_CHECK_INTERVAL}";
+fi;
+# deprecated name FORCE_CHECK, use FORCE_VERIFY instead
+if [ ! -z "${FORCE_CHECK}" ]; then
+	FORCE_VERIFY="${FORCE_CHECK}";
+fi;
+if [ -z "${FORCE_VERIFY}" ]; then
+	FORCE_VERIFY="${DEFAULT_FORCE_VERIFY}";
+fi;
+if [ -z "${KEEP_LAST}" ]; then
+	KEEP_LAST="${DEFAULT_KEEP_LAST}";
+fi;
+if [ -z "${KEEP_HOURS}" ]; then
+	KEEP_HOURS="${DEFAULT_KEEP_HOURS}";
+fi;
+if [ -z "${KEEP_DAYS}" ]; then
+	KEEP_DAYS="${DEFAULT_KEEP_DAYS}";
+fi;
+if [ -z "${KEEP_WEEKS}" ]; then
+	KEEP_WEEKS="${DEFAULT_KEEP_WEEKS}";
+fi;
+if [ -z "${KEEP_MONTHS}" ]; then
+	KEEP_MONTHS="${DEFAULT_KEEP_MONTHS}";
+fi;
+if [ -z "${KEEP_YEARS}" ]; then
+	KEEP_YEARS="${DEFAULT_KEEP_YEARS}";
+fi;
 # ** SUB LOAD
 # a settings file always end in .settings, replace that with lower case MODULE.settings
 SETTINGS_FILE_SUB=$(echo "${SETTINGS_FILE}" | sed -e "s/\.settings/\.${MODULE,,}\.settings/");
@@ -275,6 +491,14 @@ if [ -f "${BASE_FOLDER}${SETTINGS_FILE_SUB}" ]; then
 	if [ ! -z "${SUB_COMPRESSION_LEVEL}" ]; then
 		COMPRESSION_LEVEL=${SUB_COMPRESSION_LEVEL};
 	fi;
+	# compact interval override
+	if [ ! -z "${SUB_COMPACT_INTERVAL}" ]; then
+		COMPACT_INTERVAL="${SUB_COMPACT_INTERVAL}";
+	fi;
+	# override check interval
+	if [ ! -z "${SUB_CHECK_INTERVAL}" ]; then
+		CHECK_INTERVAL="${SUB_CHECK_INTERVAL}";
+	fi;
 	# check override for keep time
 	if [ ! -z "${SUB_KEEP_LAST}" ]; then
 		KEEP_LAST=${SUB_KEEP_LAST};
@@ -300,7 +524,7 @@ if [ -f "${BASE_FOLDER}${SETTINGS_FILE_SUB}" ]; then
 fi;
 # add module name to backup file, always
 # except if FILE module and FILE_REPOSITORY_COMPATIBLE="true"
-if ([ "${FILE_REPOSITORY_COMPATIBLE}" = "false" ] && [ "${MODULE,,}" = "file" ]) || [ "${MODULE,,}" != "file" ]; then
+if [ "${FILE_REPOSITORY_COMPATIBLE}" != "true" ] || [ "${MODULE,,}" != "file" ]; then
 	BACKUP_FILE=${BACKUP_FILE/.borg/-${MODULE,,}.borg};
 fi;
 # backup file must be set
@@ -337,6 +561,9 @@ if [[ ${TARGET_FOLDER} =~ ^~\/ ]]; then
 	exit 1;
 fi

+# COMPACT_INTERVAL must be a number from -1 to 365
+# CHECK_INTERVAL must be a number from -1 to 365
+
 # log file set and check
 # option folder overrides all other folders
 if [ ! -z "${OPT_LOG_FOLDER}" ]; then
@@ -365,11 +592,13 @@ if [[ -f "${LOG}" && ! -w "${LOG}" ]] || [[ ! -f "${LOG}" && ! -w "${LOG_FOLDER}
 fi;

 # if ENCRYPTION is empty or not in the valid list fall back to none
-if [ -z "${ENCRYPTION}" ]; then
-	ENCRYPTION="none";
+# NOTE This is currently set in default and doesn't need to be set on empty
+# only ivalid should be checked
+#if [ -z "${ENCRYPTION}" ]; then
+#	ENCRYPTION="none";
 #else
 	# TODO check for invalid encryption string
-fi;
+#fi;

 ## FUNCTIONS

--- a/borg.backup.functions.verify.sh
+++ b/borg.backup.functions.verify.sh
@@ -0,0 +1,447 @@
+#!/usr/bin/env bash
+
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
+# start time in seconds
+START=$(date +'%s');
+# set init date, or today if not file is set
+BACKUP_INIT_DATE='';
+if [ -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
+	BACKUP_INIT_DATE=$(printf '%(%c)T' $(cat "${BASE_FOLDER}${BACKUP_INIT_FILE}" 2>/dev/null));
+fi;
+# start logging from here
+exec &> >(tee -a "${LOG}");
+printf "${PRINTF_MASTER_BLOCK}" "START" "$(date +'%F %T')" "${MODULE}";
+# show info for version always
+printf "${PRINTF_INFO_STRING}" "Script version" "${VERSION}";
+# show type
+printf "${PRINTF_INFO_STRING}" "Backup module" "${MODULE}";
+printf "${PRINTF_INFO_STRING}" "Module version" "${MODULE_VERSION}";
+# borg version
+printf "${PRINTF_INFO_STRING}" "Borg version" "${BORG_VERSION}";
+# host name
+printf "${PRINTF_INFO_STRING}" "Hostname" "${HOSTNAME}";
+# show base folder always
+printf "${PRINTF_INFO_STRING}" "Base folder" "${BASE_FOLDER}";
+# Module init date (when init file was writen)
+printf "${PRINTF_INFO_STRING}" "Module init date" "${BACKUP_INIT_DATE}";
+# print last compact date if positive integer
+# only if borg > 1.2
+if [ $(version $BORG_VERSION) -ge $(version "1.2.0") ]; then
+	if [ "${COMPACT_INTERVAL##*[!0-9]*}" ]; then
+		printf "${PRINTF_INFO_STRING}" "Module compact interval" "${COMPACT_INTERVAL}";
+		if [ -f "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" ]; then
+			LAST_COMPACT_DATE=$(cat "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" 2>/dev/null);
+			printf "${PRINTF_INFO_STRING}" "Module last compact" \
+				"$(printf '%(%c)T' ${LAST_COMPACT_DATE}) ($(convert_time $(($(date +%s)-${LAST_COMPACT_DATE}))) ago)";
+		else
+			printf "${PRINTF_INFO_STRING}" "Module last compact" "No compact run yet"
+		fi;
+	fi;
+fi;
+# print last check date if positive integer
+if [ "${CHECK_INTERVAL##*[!0-9]*}" ]; then
+	printf "${PRINTF_INFO_STRING}" "Module check interval" "${CHECK_INTERVAL}";
+	# get last check date
+	if [ -f "${BASE_FOLDER}${BACKUP_CHECK_FILE}" ]; then
+		LAST_CHECK_DATE=$(cat "${BASE_FOLDER}${BACKUP_CHECK_FILE}" 2>/dev/null);
+		printf "${PRINTF_INFO_STRING}" "Module last check" \
+			"$(printf '%(%c)T' ${LAST_CHECK_DATE}) ($(convert_time $(($(date +%s)-${LAST_CHECK_DATE}))) ago)";
+	else
+		printf "${PRINTF_INFO_STRING}" "Module last check" "No check run yet";
+	fi;
+fi;
+
+# if force verify is true set VERIFY to 1 unless INFO is 1
+# Needs bash 4.0 at lesat for this
+if [ "${FORCE_VERIFY,,}" = "true" ] && [ ${INFO} -eq 0 ]; then
+	VERIFY=1;
+	if [ ${DEBUG} -eq 1 ]; then
+		echo "Force repository verify";
+	fi;
+fi;
+
+# remote borg path
+if [ ! -z "${TARGET_BORG_PATH}" ]; then
+	if [[ "${TARGET_BORG_PATH}" =~ \ |\' ]]; then
+		echo "Space found in ${TARGET_BORG_PATH}. Aborting";
+		echo "There are issues with passing on paths with spaces"
+		echo "as parameters"
+		exit;
+	fi;
+	OPT_REMOTE="--remote-path="$(printf "%q" "${TARGET_BORG_PATH}");
+fi;
+
+if [ -z "${TARGET_FOLDER}" ]; then
+	echo "[! $(date +'%F %T')] No target folder has been set yet";
+	exit 1;
+else
+	# There are big issues with TARGET FOLDERS with spaces
+	# we should abort anything with this
+	if [[ "${TARGET_FOLDER}" =~ \ |\' ]]; then
+		echo "Space found in ${TARGET_FOLDER}. Aborting";
+		echo "There is some problem with passing paths with spaces as";
+		echo "repository base folder"
+		exit;
+	fi;
+
+	# This does not care for multiple trailing or leading slashes
+	# it just makes sure we have at least one set
+	# for if we have a single slash, remove it
+	TARGET_FOLDER=${TARGET_FOLDER%/}
+	TARGET_FOLDER=${TARGET_FOLDER#/}
+	# and add slash front and back and escape the path
+	TARGET_FOLDER=$(printf "%q" "/${TARGET_FOLDER}/");
+fi;
+
+# if we have user/host then we build the ssh command
+TARGET_SERVER='';
+# allow host only (if full setup in .ssh/config)
+# user@host OR ssh://user@host:port/ IF TARGET_PORT is set
+# user/host/port
+if [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then
+	TARGET_SERVER="ssh://${TARGET_USER}@${TARGET_HOST}:${TARGET_PORT}/";
+# host/port
+elif [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then
+	TARGET_SERVER="ssh://${TARGET_HOST}:${TARGET_PORT}/";
+# user/host
+elif [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ]; then
+	TARGET_SERVER="${TARGET_USER}@${TARGET_HOST}:";
+# host
+elif [ ! -z "${TARGET_HOST}" ]; then
+	TARGET_SERVER="${TARGET_HOST}:";
+fi;
+# we dont allow special characters, so we don't need to special escape it
+REPOSITORY="${TARGET_SERVER}${TARGET_FOLDER}${BACKUP_FILE}";
+printf "${PRINTF_INFO_STRING}" "Repository" "${REPOSITORY}";
+
+# check if given compression name and level are valid
+OPT_COMPRESSION='';
+if [ ! -z "${COMPRESSION}" ]; then
+	# valid compression
+	if [ "${COMPRESSION}" = "lz4" ] || [ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ] || [ "${COMPRESSION}" = "zstd" ]; then
+		OPT_COMPRESSION="-C=${COMPRESSION}";
+		# if COMPRESSION_LEVEL, check it is a valid regex
+		# for zlib, zstd, lzma
+		if [ ! -z "${COMPRESSION_LEVEL}" ] && ([ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ] || [ "${COMPRESSION}" = "zstd" ]); then
+			MIN_COMPRESSION=0;
+			MAX_COMPRESSION=0;
+			case "${COMPRESSION}" in
+				zlib|lzma)
+					MIN_COMPRESSION=0;
+					MAX_COMPRESSION=9;
+					;;
+				zstd)
+					MIN_COMPRESSION=1;
+					MAX_COMPRESSION=22;
+					;;
+				*)
+					MIN_COMPRESSION=0;
+					MAX_COMPRESSION=0;
+					;;
+			esac;
+			# if [ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ]
+			# 	MIN_COMPRESSION=0;
+			# 	MAX_COMPRESSION=9;
+			# elif [ "${COMPRESSION}" = "zstd" ]; then
+			# 	MIN_COMPRESSION=1;
+			# 	MAX_COMPRESSION=22;
+			# fi;
+			error_message="[! $(date +'%F %T')] Compression level for ${COMPRESSION} needs to be a numeric value between ${MIN_COMPRESSION} and ${MAX_COMPRESSION}: ${COMPRESSION_LEVEL}";
+			if ! [[ "${COMPRESSION_LEVEL}" =~ ${REGEX_NUMERIC} ]]; then
+				echo ${error_message};
+				exit 1;
+			elif [ ${COMPRESSION_LEVEL} -lt ${MIN_COMPRESSION} ] || [ ${COMPRESSION_LEVEL} -gt ${MAX_COMPRESSION} ]; then
+				echo ${error_message};
+				exit 1;
+			else
+				OPT_COMPRESSION=${OPT_COMPRESSION}","${COMPRESSION_LEVEL};
+			fi;
+		fi;
+	else
+		echo "[! $(date +'%F %T')] Compress setting need to be lz4, zstd, zlib or lzma. Or empty for no compression: ${COMPRESSION}";
+		exit 1;
+	fi;
+fi;
+
+# home folder, needs to be set if there is eg a HOME=/ in the crontab
+if [ ! -w "${HOME}" ] || [ "${HOME}" = '/' ]; then
+	HOME=$(eval echo "$(whoami)");
+fi;
+
+# keep optionfs (for files)
+KEEP_OPTIONS=();
+# keep info string (for files)
+KEEP_INFO="";
+# override standard keep for tagged backups
+if [ ! -z "${ONE_TIME_TAG}" ]; then
+	BACKUP_SET="{now:%Y-%m-%dT%H:%M:%S}";
+	# set empty to avoid problems
+	KEEP_OPTIONS=("");
+else
+	# build options and info string,
+	# also flag BACKUP_SET check if hourly is set
+	BACKUP_SET_VERIFY=0;
+	if [ ${KEEP_LAST} -gt 0 ]; then
+		KEEP_OPTIONS+=("--keep-last=${KEEP_LAST}");
+		KEEP_INFO="${KEEP_INFO}, last: ${KEEP_LAST}";
+	fi;
+	if [ ${KEEP_HOURS} -gt 0 ]; then
+		KEEP_OPTIONS+=("--keep-hourly=${KEEP_HOURS}");
+		KEEP_INFO="${KEEP_INFO}, hourly: ${KEEP_HOURS}";
+		BACKUP_SET_VERIFY=1;
+	fi;
+	if [ ${KEEP_DAYS} -gt 0 ]; then
+		KEEP_OPTIONS+=("--keep-daily=${KEEP_DAYS}");
+		KEEP_INFO="${KEEP_INFO}, daily: ${KEEP_DAYS}";
+	fi;
+	if [ ${KEEP_WEEKS} -gt 0 ]; then
+		KEEP_OPTIONS+=("--keep-weekly=${KEEP_WEEKS}");
+		KEEP_INFO="${KEEP_INFO}, weekly: ${KEEP_WEEKS}";
+	fi;
+	if [ ${KEEP_MONTHS} -gt 0 ]; then
+		KEEP_OPTIONS+=("--keep-monthly=${KEEP_MONTHS}");
+		KEEP_INFO="${KEEP_INFO}, monthly: ${KEEP_MONTHS}";
+	fi;
+	if [ ${KEEP_YEARS} -gt 0 ]; then
+		KEEP_OPTIONS+=("--keep-yearly=${KEEP_YEARS}");
+		KEEP_INFO="${KEEP_INFO}, yearly: ${KEEP_YEARS}";
+	fi;
+	if [ ! -z "${KEEP_WITHIN}" ]; then
+		# check for invalid string. can only be number + H|d|w|m|y
+		if [[ "${KEEP_WITHIN}" =~ ^[0-9]+[Hdwmy]{1}$ ]]; then
+			KEEP_OPTIONS+=("--keep-within=${KEEP_WITHIN}");
+			KEEP_INFO="${KEEP_INFO}, within: ${KEEP_WITHIN}";
+			if [[ "${KEEP_WITHIN}" == *"H"* ]]; then
+				BACKUP_SET_VERIFY=1;
+			fi;
+		else
+			echo "[! $(date +'%F %T')] KEEP_WITHIN has invalid string.";
+			exit 1;
+		fi;
+	fi;
+	# abort if KEEP_OPTIONS is empty
+	if [ "${#KEEP_OPTIONS[@]}" -eq "0" ]; then
+		echo "[! $(date +'%F %T')] It seems no KEEP_* entries where set in a valid format.";
+		exit 1;
+	fi;
+	# set BACKUP_SET if empty, set to Year-month-day
+	if [ -z "${BACKUP_SET}" ]; then
+		BACKUP_SET="{now:%Y-%m-%d}";
+	fi;
+	# backup set check, and there is no hour entry (%H) in the archive string
+	# we add T%H:%M:%S in this case, before the last }
+	if [ ${BACKUP_SET_VERIFY} -eq 1 ] && [[ "${BACKUP_SET}" != *"%H"* ]]; then
+		BACKUP_SET=$(echo "${BACKUP_SET}" | sed -e "s/}/T%H:%M:%S}/");
+	fi;
+fi;
+
+# check if we have lock file, check pid in lock file, if no matching pid found
+# running remove lock file
+if [ -f "${BASE_FOLDER}${BACKUP_LOCK_FILE}" ]; then
+	LOCK_PID=$(cat "${BASE_FOLDER}${BACKUP_LOCK_FILE}" 2>/dev/null);
+	# check if lock file pid has an active program attached to it
+	if [ -f /proc/${LOCK_PID}/cmdline ]; then
+		echo "Script is already running on PID: ${$}";
+		. "${DIR}/borg.backup.functions.close.sh" 1;
+		exit 1;
+	else
+		echo "[#] Clean up stale lock file for PID: ${LOCK_PID}";
+		rm "${BASE_FOLDER}${BACKUP_LOCK_FILE}";
+	fi;
+fi;
+echo "${$}" > "${BASE_FOLDER}${BACKUP_LOCK_FILE}";
+
+# for folders list split set to "#" and keep the old setting as is
+_IFS=${IFS};
+IFS="#";
+# turn off for non file
+if [ "${MODULE}" != "file" ]; then
+	IFS=${_IFS};
+fi;
+
+# borg call, replace ##...## parts during run
+# used in all modules, except 'file'
+_BORG_CALL="${BORG_COMMAND} create ${OPT_REMOTE} -v ${OPT_LIST} ${OPT_PROGRESS} ${OPT_COMPRESSION} -s --stdin-name ##FILENAME## ${REPOSITORY}::##BACKUP_SET## -";
+_BORG_PRUNE="${BORG_COMMAND} prune ${OPT_REMOTE} -v --list ${OPT_PROGRESS} ${DRY_RUN_STATS} -P ##BACKUP_SET_PREFIX## ${KEEP_OPTIONS[*]} ${REPOSITORY}";
+
+# general borg settings
+# set base path to config directory to keep cache/config separated
+export BORG_BASE_DIR="${BASE_FOLDER}";
+# ignore non encrypted access
+export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=${_BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK};
+# ignore moved repo access
+export BORG_RELOCATED_REPO_ACCESS_IS_OK=${_BORG_RELOCATED_REPO_ACCESS_IS_OK};
+# and for debug print that tout
+if [ ${DEBUG} -eq 1 ]; then
+	echo "export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=${_BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK};";
+	echo "export BORG_RELOCATED_REPO_ACCESS_IS_OK=${_BORG_RELOCATED_REPO_ACCESS_IS_OK};";
+	echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
+fi;
+# prepare debug commands only
+COMMAND_EXPORT="export BORG_BASE_DIR=\"${BASE_FOLDER}\";"
+COMMAND_INFO="${COMMAND_EXPORT}${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY}";
+# if the is not there, call init to create it
+# if this is user@host, we need to use ssh command to verify if the file is there
+# else a normal verify is ok
+# unless explicit given, verify is skipped
+if [ ${VERIFY} -eq 1 ] || [ ${INIT} -eq 1 ]; then
+	printf "${PRINTF_SUB_BLOCK}" "VERIFY" "$(date +'%F %T')" "${MODULE}";
+	if [ ! -z "${TARGET_SERVER}" ]; then
+		if [ ${DEBUG} -eq 1 ]; then
+			echo "${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY} 2>&1|grep \"Repository ID:\"";
+		fi;
+		# use borg info and verify if it returns "Repository ID:" in the first line
+		REPO_VERIFY=$(${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY} 2>&1|grep "Repository ID:");
+		# this is currently a hack to work round the error code in borg info
+		# this checks if REPO_VERIFY holds this error message and then starts init
+		if [[ -z "${REPO_VERIFY}" ]] || [[ "${REPO_VERIFY}" =~ ${REGEX_ERROR} ]]; then
+			INIT_REPOSITORY=1;
+		fi;
+	elif [ ! -d "${REPOSITORY}" ]; then
+		INIT_REPOSITORY=1;
+	fi;
+	# if verrify but no init and repo is there but init file is missing set it
+	if [ ${VERIFY} -eq 1 ] && [ ${INIT} -eq 0 ] && [ ${INIT_REPOSITORY} -eq 0 ] &&
+		[ ! -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
+		# write init file
+		echo "[!] Add missing init verify file";
+		echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_FILE}";
+	fi;
+	# end if verified but repository is not here
+	if [ ${VERIFY} -eq 1 ] && [ ${INIT} -eq 0 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then
+		echo "[! $(date +'%F %T')] No repository. Please run with -I flag to initialze repository";
+		. "${DIR}/borg.backup.functions.close.sh" 1;
+		exit 1;
+	fi;
+	if [ ${EXIT} -eq 1 ] && [ ${VERIFY} -eq 1 ] && [ ${INIT} -eq 0 ]; then
+		echo "Repository exists";
+		echo "For more information run:"
+		echo "${COMMAND_INFO}";
+		. "${DIR}/borg.backup.functions.close.sh";
+		exit;
+	fi;
+fi;
+if [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then
+	printf "${PRINTF_SUB_BLOCK}" "INIT" "$(date +'%F %T')" "${MODULE}";
+	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
+		echo "${BORG_COMMAND} init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY}";
+	fi
+	if [ ${DRYRUN} -eq 0 ]; then
+		# should trap and exit properly here
+		${BORG_COMMAND} init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY};
+		# write init file
+		echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_FILE}";
+		echo "Repository initialized";
+		echo "For more information run:"
+		echo "${COMMAND_INFO}";
+	fi
+	. "${DIR}/borg.backup.functions.close.sh";
+	# exit after init
+	exit;
+elif [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 0 ]; then
+	echo "[! $(date +'%F %T')] Repository already initialized";
+	echo "For more information run:"
+	echo "${COMMAND_INFO}";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
+	exit 1;
+fi;
+
+# verify for init file
+if [ ! -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
+	echo "[! $(date +'%F %T')] It seems the repository has never been initialized."
+	echo "Please run -I to initialize or if already initialzed run with -C for init update."
+	. "${DIR}/borg.backup.functions.close.sh" 1;
+	exit 1;
+fi;
+
+# PRINT OUT current data, only do this if REPO exists
+if [ ${PRINT} -eq 1 ]; then
+	printf "${PRINTF_SUB_BLOCK}" "PRINT" "$(date +'%F %T')" "${MODULE}";
+	FORMAT="{archive:<45} {comment:6} {start} - {end} [{id}] ({username}@{hostname}){NL}"
+	# show command on debug or dry run
+	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
+		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} list ${OPT_REMOTE} --format ${FORMAT} ${REPOSITORY}";
+	fi;
+	# run info command if not a dry drun
+	if [ ${DRYRUN} -eq 0 ]; then
+		${BORG_COMMAND} list ${OPT_REMOTE} --format "${FORMAT}" ${REPOSITORY} ;
+	fi;
+	if [ ${VERBOSE} -eq 1 ]; then
+		echo "";
+		echo "Base command info:"
+		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} [COMMAND] ${OPT_REMOTE} ${REPOSITORY}::[BACKUP] [PATH]";
+		echo "Replace [COMMAND] with list for listing or extract for restoring backup data."
+		echo "Replace [BACKUP] with archive name."
+		echo "If no [PATH] is given then all files will be restored."
+		echo "Before extracting -n (dry run) is recommended to use."
+		echo "If archive size is needed the info command with archive name has to be used."
+		echo "When listing files in an archive set (::SET) the --format command can be used."
+		echo "Example: \"{mode} {user:6} {group:6} {size:8d} {csize:8d} {dsize:8d} {dcsize:8d} {mtime} {path}{extra} [{health}]{NL}\""
+	else
+		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} [COMMAND] ${OPT_REMOTE} [FORMAT] ${REPOSITORY}::[BACKUP] [PATH]";
+	fi;
+	. "${DIR}/borg.backup.functions.close.sh";
+	exit;
+fi;
+
+# run borg compact command and exit
+if [ ${COMPACT} -eq 1 ]; then
+	. "${DIR}/borg.backup.functions.compact.sh";
+	. "${DIR}/borg.backup.functions.close.sh";
+	exit;
+fi;
+
+# run borg check command and exit
+if [ ${CHECK} -eq 1 ]; then
+	. "${DIR}/borg.backup.functions.check.sh";
+	. "${DIR}/borg.backup.functions.close.sh";
+	exit;
+fi;
+
+# DELETE ONE TIME TAG
+if [ ! -z "${DELETE_ONE_TIME_TAG}" ]; then
+	printf "${PRINTF_SUB_BLOCK}" "DELETE" "$(date +'%F %T')" "${MODULE}";
+	# if a "*" is inside we don't do ONE archive, but globbing via -a option
+	DELETE_ARCHIVE=""
+	OPT_GLOB="";
+	# this is more or less for debug only
+	if [[ "${DELETE_ONE_TIME_TAG}" =~ $REGEX_GLOB ]]; then
+		OPT_GLOB="-a '${DELETE_ONE_TIME_TAG}'"
+	else
+		DELETE_ARCHIVE="::"${DELETE_ONE_TIME_TAG};
+	fi
+	# if this is borg <1.2 OPT_LIST does not work
+	if [ $(version $BORG_VERSION) -lt $(version "1.2.0") ]; then
+		OPT_LIST="";
+	fi;
+	# if exists, delete and exit
+	# show command on debug or dry run
+	if [ ${DEBUG} -eq 1 ]; then
+		echo "${BORG_COMMAND} delete ${OPT_REMOTE} ${OPT_LIST} -s ${OPT_GLOB} ${REPOSITORY}${DELETE_ARCHIVE}";
+	fi;
+	# run delete command if not a dry drun
+	# NOTE seems to be glob is not working if wrapped into another variable
+	if [[ "${DELETE_ONE_TIME_TAG}" =~ $REGEX_GLOB ]]; then
+		${BORG_COMMAND} delete ${OPT_REMOTE} ${OPT_LIST} ${DRY_RUN_STATS} -a "${DELETE_ONE_TIME_TAG}" ${REPOSITORY};
+	else
+		${BORG_COMMAND} delete ${OPT_REMOTE} ${OPT_LIST} ${DRY_RUN_STATS} ${REPOSITORY}${DELETE_ARCHIVE};
+	fi;
+	# if not a dry run, compact repository after delete
+	# not that compact only works on borg 1.2
+	if [ $(version $BORG_VERSION) -ge $(version "1.2.0") ]; then
+		if [ ${DRYRUN} -eq 0 ]; then
+			${BORG_COMMAND} compact ${OPT_REMOTE} ${REPOSITORY};
+		fi;
+		if [ ${DEBUG} -eq 1 ]; then
+			echo "${BORG_COMMAND} compact ${OPT_REMOTE} ${REPOSITORY}";
+		fi;
+	fi;
+	. "${DIR}/borg.backup.functions.close.sh";
+	exit;
+fi;
+
+# __END__
--- a/borg.backup.gitea.sh
+++ b/borg.backup.gitea.sh
@@ -1,18 +1,24 @@
 #!/usr/bin/env bash

+# Backup gitea database, all git folders and gitea settings
+
 MODULE="gitea"
-MODULE_VERSION="1.0.0";
+MODULE_VERSION="1.1.4";

 DIR="${BASH_SOURCE%/*}"
 if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 # init system
 . "${DIR}/borg.backup.functions.init.sh";

-# init check file
-BACKUP_INIT_CHECK="borg.backup.gitea.init";
+# init verify, compact and check file
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";

-# check valid data
-. "${DIR}/borg.backup.functions.check.sh";
+# verify valid data
+. "${DIR}/borg.backup.functions.verify.sh";
 # if info print info and then abort run
 . "${DIR}/borg.backup.functions.info.sh";

@@ -35,25 +41,29 @@ if [ -z "${GITEA_CONFIG}" ]; then
 fi;
 if [ ! -f "${GITEA_BIN}" ]; then
 	echo "[! $(date +'%F %T')] Cannot find gitea binary";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
 if [ ! -f "${GITEA_CONFIG}" ]; then
 	echo "[! $(date +'%F %T')] Cannot find gitea config";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
 # Filename
 FILENAME="gitea.backup.zip";
 # backup set and prefix
-BACKUP_SET_PREFIX="gitea-";
-BACKUP_SET_NAME="${BACKUP_SET_PREFIX}${BACKUP_SET}";
+BACKUP_SET_PREFIX="${MODULE},";
+BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${BACKUP_SET}";

 # borg call
 BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/");
 BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/");
-echo "--- [git data and database: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+printf "${PRINTF_SUB_BLOCK}" "BACKUP: git data and database" "$(date +'%F %T')" "${MODULE}";
 if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 	echo "sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_TMP} -L -f - | ${BORG_CALL}";
+	if [ -z "${ONE_TIME_TAG}" ]; then
 		echo "${BORG_PRUNE}";
+	fi;
 fi;
 if [ ${DRYRUN} -eq 0 ]; then
 	(
@@ -67,10 +77,17 @@ if [ ${DRYRUN} -eq 0 ]; then
 		# this needs to be run in a folder that can be stat by git user
 		cd "${GITEA_TMP}";
 		sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_TMP} -L -f - | ${BORG_CALL};
-	) | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g' # remove all ESC strings
+	) 2>&1 | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g' # remove all ESC strings
+fi;
+if [ -z "${ONE_TIME_TAG}" ]; then
+	printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}";
+	echo "Prune repository with keep${KEEP_INFO:1}";
+	${BORG_PRUNE};
+	# if this is borg version >1.2 we need to run compact after prune
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
 fi;
-echo "Prune repository with keep${KEEP_INFO:1}";
-${BORG_PRUNE};

 . "${DIR}/borg.backup.functions.close.sh";

--- a/borg.backup.mysql.sh
+++ b/borg.backup.mysql.sh
@@ -5,12 +5,9 @@
 # config override set in borg.backup.mysql.settings
 # if run as mysql user, be sure user is in the backup group

-# Run -I first to initialize repository
-# There are no automatic repository checks unless -C is given
-
 # set last edit date + time
 MODULE="mysql"
-MODULE_VERSION="1.0.0";
+MODULE_VERSION="1.1.2";

 DIR="${BASH_SOURCE%/*}"
 if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
@@ -18,14 +15,18 @@ if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 . "${DIR}/borg.backup.functions.init.sh";

 # include and exclude file
-INCLUDE_FILE="borg.backup.mysql.include";
-EXCLUDE_FILE="borg.backup.mysql.exclude";
-SCHEMA_ONLY_FILE="borg.backup.mysql.schema-only";
-# init check file
-BACKUP_INIT_CHECK="borg.backup.mysql.init";
+INCLUDE_FILE="borg.backup.${MODULE}.include";
+EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
+SCHEMA_ONLY_FILE="borg.backup.${MODULE}.schema-only";
+# init verify, compact and check file
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";

-# check valid data
-. "${DIR}/borg.backup.functions.check.sh";
+# verify valid data
+. "${DIR}/borg.backup.functions.verify.sh";
 # if info print info and then abort run
 . "${DIR}/borg.backup.functions.info.sh";

@@ -42,18 +43,21 @@ MYSQL_CMD=${MYSQL_BASE_PATH}'mysql';
 # no dump or mysql, bail
 if [ ! -f "${MYSQL_DUMP}" ]; then
 	echo "[! $(date +'%F %T')] mysqldump binary not found";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
 if [ ! -f "${MYSQL_CMD}" ]; then
 	echo "[! $(date +'%F %T')] mysql binary not found";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
-# check that the user can actually do, else abort here
+# verify that the user can actually do, else abort here
 # note: this is the only way to not error
-_MYSQL_CHECK=$(mysqladmin ${MYSQL_DB_CONFIG_PARAM} ping 2>&1);
-_MYSQL_OK=$(echo "${_MYSQL_CHECK}" | grep "is alive");
+_MYSQL_VERIFY=$(mysqladmin ${MYSQL_DB_CONFIG_PARAM} ping 2>&1);
+_MYSQL_OK=$(echo "${_MYSQL_VERIFY}" | grep "is alive");
 if [ -z "${_MYSQL_OK}" ]; then
 	echo "[! $(date +'%F %T')] Current user has no access right to mysql database";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
 # below is for file name only
@@ -89,35 +93,42 @@ if [ ! -z "${DATABASE_FULL_DUMP}" ]; then
 		SCHEMA_ONLY='--no-data';
 		schema_flag='schema';
 	fi;
-	echo "--- [all databases: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "all databases" "$(date +'%F %T')" "${MODULE}";
 	# We only do a full backup and not per table backup here
 	# Filename
 	FILENAME="all-${schema_flag}-${DB_TYPE}_${DB_VERSION}_${DB_HOST}_${DB_PORT}.sql"
 	# backup set:
-	BACKUP_SET_PREFIX="all-";
-	BACKUP_SET_NAME="${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
+	BACKUP_SET_PREFIX="${MODULE},all-";
+	BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
 	# borg call
 	BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/");
 	BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/");
 	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
 		echo "${MYSQL_DUMP} ${MYSQL_DB_CONFIG_PARAM} --all-databases --create-options --add-drop-database --events ${SCHEMA_ONLY} | ${BORG_CALL}";
+		if [ -z "${ONE_TIME_TAG}" ]; then
 			echo "${BORG_PRUNE}";
 		fi;
+	fi;
 	if [ ${DRYRUN} -eq 0 ]; then
 		${MYSQL_DUMP} ${MYSQL_DB_CONFIG_PARAM} --all-databases --create-options --add-drop-database --events ${SCHEMA_ONLY} | ${BORG_CALL};
 		_backup_error=$?;
 		if [ $_backup_error -ne 0 ]; then
 			echo "[! $(date +'%F %T')] Backup creation failed for full dump with error code: ${_backup_error}";
+			. "${DIR}/borg.backup.functions.close.sh" 1;
 			exit $_backup_error;
 		fi;
 	fi;
+	if [ -z "${ONE_TIME_TAG}" ]; then
+		printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "all databases" "$(date +'%F %T')" "${MODULE}";
 		echo "Prune repository with keep${KEEP_INFO:1}";
 		${BORG_PRUNE};
+	fi;
 else
 	${MYSQL_CMD} ${MYSQL_DB_CONFIG_PARAM} -B -N -e "show databases" |
 	while read db; do
-		echo "--- [${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+		printf "${PRINTF_DB_SUB_BLOCK}" "DB" "${db}" "${MODULE}";
+		printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
 		# exclude checks
 		include=0;
 		if [ -s "${BASE_FOLDER}${INCLUDE_FILE}" ]; then
@@ -174,8 +185,8 @@ else
 			# prepare borg calls
 			FILENAME="${db}-${schema_flag}-${DB_TYPE}_${DB_VERSION}_${DB_HOST}_${DB_PORT}.sql"
 			# backup set:
-			BACKUP_SET_PREFIX="${db}-"
-			BACKUP_SET_NAME="${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
+			BACKUP_SET_PREFIX="${MODULE},${db}-"
+			BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
 			# borg call
 			BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/");
 			BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/");
@@ -190,16 +201,27 @@ else
 				_backup_error=$?;
 				if [ $_backup_error -ne 0 ]; then
 					echo "[! $(date +'%F %T')] Backup creation failed for ${db} dump with error code: ${_backup_error}";
+					. "${DIR}/borg.backup.functions.close.sh" 1;
 					exit $_backup_error;
 				fi;
 			fi;
+			if [ -z "${ONE_TIME_TAG}" ]; then
+				printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "${db}" "$(date +'%F %T')" "${MODULE}";
 				echo "Prune repository prefixed ${BACKUP_SET_PREFIX} with keep${KEEP_INFO:1}";
 				${BORG_PRUNE};
+			fi;
 		else
 			echo "- [E] ${db}";
 		fi;
 	done;
 fi;
+# run compact at the end if not a dry run
+if [ -z "${ONE_TIME_TAG}" ]; then
+	# if this is borg version >1.2 we need to run compact after prune
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
+fi;

 . "${DIR}/borg.backup.functions.close.sh";

--- a/borg.backup.pgsql.sh
+++ b/borg.backup.pgsql.sh
@@ -5,12 +5,9 @@
 # config override set in borg.backup.pgsql.settings
 # if run as postgres user, be sure user is in the backup group

-# Run -I first to initialize repository
-# There are no automatic repository checks unless -C is given
-
 # set last edit date + time
 MODULE="pgsql"
-MODULE_VERSION="1.0.1";
+MODULE_VERSION="1.2.2";


 DIR="${BASH_SOURCE%/*}"
@@ -19,14 +16,19 @@ if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 . "${DIR}/borg.backup.functions.init.sh";

 # include and exclude file
-INCLUDE_FILE="borg.backup.pgsql.include";
-EXCLUDE_FILE="borg.backup.pgsql.exclude";
-SCHEMA_ONLY_FILE="borg.backup.pgsql.schema-only";
-# init check file
-BACKUP_INIT_CHECK="borg.backup.pgsql.init";
+INCLUDE_FILE="borg.backup.${MODULE}.include";
+EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
+SCHEMA_ONLY_FILE="borg.backup.${MODULE}.schema-only";
+DATA_ONLY_FILE="borg.backup.${MODULE}.data-only";
+# init verify, compact and check file
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";

-# check valid data
-. "${DIR}/borg.backup.functions.check.sh";
+# verify valid data
+. "${DIR}/borg.backup.functions.verify.sh";
 # if info print info and then abort run
 . "${DIR}/borg.backup.functions.info.sh";

@@ -43,6 +45,7 @@ _PATH_PG_VERSION=${PG_VERSION};
 _backup_error=$?;
 if [ $_backup_error -ne 0 ] || [ -z "${PG_VERSION}" ]; then
 	echo "[! $(date +'%F %T')] Cannot get PostgreSQL server version: ${_backup_error}";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit $_backup_error;
 fi;

@@ -58,6 +61,7 @@ if [ ! -f "${PG_BASE_PATH}${_PATH_PG_VERSION}/bin/psql" ]; then
 		_PATH_PG_VERSION=$(echo "${PG_VERSION}" | sed -e 's/\.//');
 		if [ ! -f "${PG_BASE_PATH}${_PATH_PG_VERSION}/bin/psql" ]; then
 			echo "[! $(date +'%F %T')] PostgreSQL not found in any paths";
+			. "${DIR}/borg.backup.functions.close.sh" 1;
 			exit 1;
 		fi;
 	fi;
@@ -69,14 +73,17 @@ PG_DUMPALL=${PG_PATH}'pg_dumpall';
 # check that command are here
 if [ ! -f "${PG_PSQL}" ]; then
 	echo "[! $(date +'%F %T')] psql binary not found in ${PG_PATH}";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
 if [ ! -f "${PG_DUMP}" ]; then
 	echo "[! $(date +'%F %T')] pg_dump binary not found in ${PG_PATH}";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
 if [ ! -f "${PG_DUMPALL}" ]; then
 	echo "[! $(date +'%F %T')] pg_dumpall binary not found in ${PG_PATH}";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;

@@ -95,58 +102,70 @@ if [ ! -z "${DATABASE_FULL_DUMP}" ]; then
 		SCHEMA_ONLY='-s';
 		schema_flag='schema';
 	fi;
-	echo "--- [all databases: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "all databases" "$(date +'%F %T')" "${MODULE}";
 	# Filename
 	FILENAME-"all.${DB_USER}.NONE.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql"
 	# backup set:
-	BACKUP_SET_PREFIX="all-";
-	BACKUP_SET_NAME="${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
+	BACKUP_SET_PREFIX="${MODULE},all-";
+	BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
 	# borg call
 	BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/");
 	BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/");
 	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
 		echo "${PG_DUMPALL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} ${SCHEMA_ONLY} -c | ${BORG_CALL}";
+		if [ -z "${ONE_TIME_TAG}" ]; then
 			echo "${BORG_PRUNE}";
 		fi;
+	fi;
 	if [ ${DRYRUN} -eq 0 ]; then
 		$(${PG_DUMPALL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} ${SCHEMA_ONLY} -c | ${BORG_CALL});
 		_backup_error=$?;
 		if [ $_backup_error -ne 0 ]; then
 			echo "[! $(date +'%F %T')] Backup creation failed for full dump with error code: ${_backup_error}";
+			. "${DIR}/borg.backup.functions.close.sh" 1;
 			exit $_backup_error;
 		fi;
 	fi;
+	if [ -z "${ONE_TIME_TAG}" ]; then
+		printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "all databases" "$(date +'%F %T')" "${MODULE}";
 		echo "Prune repository with keep${KEEP_INFO:1}";
 		${BORG_PRUNE};
+	fi;
 else
 	# dump globals first
 	db="pg_globals";
 	schema_flag="data";
-	echo "--- [${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
 	# Filename
 	FILENAME="${db}.${DB_USER}.NONE.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql"
 	# backup set:
-	BACKUP_SET_PREFIX="${db}-";
-	BACKUP_SET_NAME="${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
+	BACKUP_SET_PREFIX="${MODULE},${db}-";
+	BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
 	# borg call
 	BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/");
 	BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/");
 	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
 		echo "${PG_DUMPALL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} --globals-only | ${BORG_CALL}";
+		if [ -z "${ONE_TIME_TAG}" ]; then
 			echo "${BORG_PRUNE}";
 		fi;
+	fi;
 	if [ ${DRYRUN} -eq 0 ]; then
 		${PG_DUMPALL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} --globals-only | ${BORG_CALL};
 		_backup_error=$?;
 		if [ $_backup_error -ne 0 ]; then
 			echo "[! $(date +'%F %T')] Backup creation failed for ${db} dump with error code: ${_backup_error}";
+			. "${DIR}/borg.backup.functions.close.sh" 1;
 			exit $_backup_error;
 		fi;
 	fi;
+	if [ -z "${ONE_TIME_TAG}" ]; then
+		printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "${db}" "$(date +'%F %T')" "${MODULE}";
 		echo "Prune repository with keep${KEEP_INFO:1}";
 		${BORG_PRUNE};
+	fi;

 	# get list of tables
 	for owner_db in $(${PG_PSQL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} -d template1 -t -A -F "," -X -q -c "SELECT pg_catalog.pg_get_userbyid(datdba) AS owner, datname, pg_catalog.pg_encoding_to_char(encoding) AS encoding FROM pg_catalog.pg_database WHERE datname "\!"~ 'template(0|1)' ORDER BY datname;"); do
@@ -154,7 +173,8 @@ else
 		owner=$(echo ${owner_db} | cut -d "," -f 1);
 		db=$(echo ${owner_db} | cut -d "," -f 2);
 		encoding=$(echo ${owner_db} | cut -d "," -f 3);
-		echo "--- [${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+		printf "${PRINTF_DB_SUB_BLOCK}" "DB" "${db}" "${MODULE}";
+		printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
 		include=0;
 		if [ -s "${BASE_FOLDER}${INCLUDE_FILE}" ]; then
 			while read incl_db; do
@@ -177,9 +197,13 @@ else
 		fi;
 		if [ ${include} -eq 1 ] && [ ${exclude} -eq 0 ]; then
 			# set dump type
-			SCHEMA_ONLY=''; # empty for all
-			schema_flag='data'; # or data
+			SCHEMA_ONLY='';
+			schema_flag=''; # schema or data
+			# schema exclude over data exclude, can't have both
 			if [ -s "${BASE_FOLDER}${SCHEMA_ONLY_FILE}" ]; then
+				# default is data dump
+				SCHEMA_ONLY='';
+				schema_flag='data';
 				while read schema_db; do
 					if [ "${db}" = "${schema_db}" ]; then
 						SCHEMA_ONLY='-s';
@@ -188,14 +212,31 @@ else
 						break;
 					fi;
 				done<"${BASE_FOLDER}${SCHEMA_ONLY_FILE}";
+			elif [ -s "${BASE_FOLDER}${DATA_ONLY_FILE}" ]; then
+				# default to schema, unless in data list
+				SCHEMA_ONLY='-s';
+				schema_flag='schema';
+				while read data_db; do
+					if [ "${db}" = "${data_db}" ]; then
+						SCHEMA_ONLY='';
+						schema_flag='data';
+						# skip out
+						break;
+					fi;
+				done<"${BASE_FOLDER}${DATA_ONLY_FILE}";
+			fi;
+			# if nothing is set, default to data
+			if [ -z "${schema_flag}" ]; then
+				SCHEMA_ONLY=''
+				schema_flag="data";
 			fi;
 			# Filename
 			# Database.User.Encoding.pgsql|data|schema-Version_Host_Port_YearMonthDay_HourMinute_Counter.Fromat(c).sql
 			FILENAME="${db}.${owner}.${encoding}.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql"
-			# backup set:
-			BACKUP_SET_NAME="${db}-${schema_flag}-${BACKUP_SET}";
 			# PER db either data or schema
-			BACKUP_SET_PREFIX="${db}-";
+			BACKUP_SET_PREFIX="${MODULE},${db}-";
+			# backup set:
+			BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
 			# borg call
 			BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/");
 			# borg prune
@@ -203,23 +244,36 @@ else
 			if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 				echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
 				echo "${PG_DUMP} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} -c ${SCHEMA_ONLY} --format=c ${db} | ${BORG_CALL}";
+				if [ -z "${ONE_TIME_TAG}" ]; then
 					echo "${BORG_PRUNE}";
 				fi;
+			fi;
 			if [ ${DRYRUN} -eq 0 ]; then
 				${PG_DUMP} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} -c ${SCHEMA_ONLY} --format=c ${db} | ${BORG_CALL};
 				_backup_error=$?;
 				if [ $_backup_error -ne 0 ]; then
 					echo "[! $(date +'%F %T')] Backup creation failed for ${db} dump with error code: ${_backup_error}";
+					. "${DIR}/borg.backup.functions.close.sh" 1;
 					exit $_backup_error;
 				fi;
 			fi;
+			if [ -z "${ONE_TIME_TAG}" ]; then
+				printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "${db}" "$(date +'%F %T')" "${MODULE}";
 				echo "Prune repository prefixed ${BACKUP_SET_PREFIX} with keep${KEEP_INFO:1}";
 				${BORG_PRUNE};
+			fi;
 		else
 			echo "- [E] ${db}";
 		fi;
 	done;
 fi;
+# run compact at the end if not a dry run
+if [ -z "${ONE_TIME_TAG}" ]; then
+	# if this is borg version >1.2 we need to run compact after prune
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
+fi;

 . "${DIR}/borg.backup.functions.close.sh";

--- a/borg.backup.settings-default
+++ b/borg.backup.settings-default
@@ -27,8 +27,19 @@ COMPRESSION_LEVEL="";
 # Blank passwords allowed for only key (if used, use keyfile)
 # See: http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically
 ENCRYPTION="";
-# force repository check, default is off, set to true for check
-FORCE_CHECK="";
+# force repository verify, default is off, set to true for verify on every run
+FORCE_VERIFY="";
+# compact interval, only if using borg 1.2 or higher
+# after how many days to run compress on data
+# default is 1 for run after each prune
+# set to 0 or negative to turn off
+# or any other value for every n days
+COMPACT_INTERVAL="";
+# check interval, if 0 or negative number, no check will ever run
+# if empty fall back to default set
+# if set to 1 then every time the script runs
+# any other value it means ever n days, eg 90 would be every 90 days
+CHECK_INTERVAL="";
 # default is %Y-%m-%d
 # todays date, if more than one per day add -%H%M for hour/minute
 # it can also be "{hostname}-{user}-{now:%Y-%m-%dT%H:%M:%S.%f}"
--- a/borg.backup.zabbix.sh
+++ b/borg.backup.zabbix.sh
@@ -3,18 +3,22 @@
 # Backup zabbix config and settings only

 MODULE="zabbix"
-MODULE_VERSION="1.0.1";
+MODULE_VERSION="1.1.3";

 DIR="${BASH_SOURCE%/*}"
 if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 # init system
 . "${DIR}/borg.backup.functions.init.sh";

-# init check file
-BACKUP_INIT_CHECK="borg.backup.zabbix.init";
+# init verify, compact and check file
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";

-# check valid data
-. "${DIR}/borg.backup.functions.check.sh";
+# verify valid data
+. "${DIR}/borg.backup.functions.verify.sh";
 # if info print info and then abort run
 . "${DIR}/borg.backup.functions.info.sh";

@@ -24,7 +28,8 @@ if [ -z "${ZABBIX_DUMP_BIN}" ]; then
 fi;
 if [ ! -z "${ZABBIX_CONFIG}" ] && [ ! -f "${ZABBIX_CONFIG}" ]; then
 	echo "[! $(date +'%F %T')] Cannot find zabbix config: ${ZABBIX_CONFIG}";
-	exit;
+	. "${DIR}/borg.backup.functions.close.sh" 1;
+	exit 1;
 fi;
 if [ -f "${ZABBIX_CONFIG}" ]; then
 	OPT_ZABBIX_CONFIG="-z ${ZABBIX_CONFIG}";
@@ -34,10 +39,12 @@ if [ "${ZABBIX_DATABASE}" = "psql" ]; then
 fi;
 if [ "${ZABBIX_DATABASE}" != "psql" ] && [ "${ZABBIX_DATABASE}" != "mysql" ]; then
 	echo "[! $(date +'%F %T')] Zabbix dump must have database set to either psql or mysql";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
 if [ ! -f "${ZABBIX_DUMP_BIN}" ]; then
 	echo "[! $(date +'%F %T')] Zabbix dump script could not be found: ${ZABBIX_DUMP_BIN}";
+	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
 # -i (ignore)/ -f (backup)
@@ -50,8 +57,8 @@ fi;
 # Filename
 FILENAME="zabbix-config.c.sql";
 # backup set:
-BACKUP_SET_PREFIX="zabbix-settings-";
-BACKUP_SET_NAME="${BACKUP_SET_PREFIX}${BACKUP_SET}";
+BACKUP_SET_PREFIX="${MODULE},settings-";
+BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${BACKUP_SET}";

 # borg call
 BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/");
@@ -61,13 +68,25 @@ if [ -z "${BACKUP_SET_PREFIX}" ]; then
 	BORG_PRUNE=$(echo "${BORG_PRUNE}" | sed -e 's/-P //');
 fi;

-echo "--- [zabbix settings: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+printf "${PRINTF_SUB_BLOCK}" "BACKUP: zabbix settings" "$(date +'%F %T')" "${MODULE}";
 if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 	echo "${ZABBIX_DUMP_BIN} -t ${ZABBIX_DATABASE} ${OPT_ZABBIX_UNKNOWN_TABLES} ${OPT_ZABBIX_DUMP} ${OPT_ZABBIX_CONFIG} -o - | ${BORG_CALL}"
+	if [ -z "${ONE_TIME_TAG}" ]; then
 		echo "${BORG_PRUNE}";
+	fi;
 fi;
 if [ ${DRYRUN} -eq 0 ]; then
 	${ZABBIX_DUMP_BIN} -t ${ZABBIX_DATABASE} ${OPT_ZABBIX_UNKNOWN_TABLES} ${OPT_ZABBIX_DUMP} ${OPT_ZABBIX_CONFIG} -o - | ${BORG_CALL};
 fi;
-echo "Prune repository with keep${KEEP_INFO:1}";
-${BORG_PRUNE};
+if [ -z "${ONE_TIME_TAG}" ]; then
+	printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}";
+	${BORG_PRUNE};
+	# if this is borg version >1.2 we need to run compact after prune
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
+fi;
+
+. "${DIR}/borg.backup.functions.close.sh";
+
+# __EMD__
--- a/borg.mount.sh
+++ b/borg.mount.sh
@@ -1,5 +1,8 @@
 #!/usr/bin/env bash

+echo "${0} Currently not checked";
+exit;
+
 set -e -u -o pipefail

 # mount this servers borg backup to a folder
@@ -13,7 +16,7 @@ SETTINGS_FILE="borg.backup.settings";
 # base mount path (default)
 MOUNT_PATH="/mnt/restore/";
 # backup path to borg storage
-ATTIC_BACKUP_FILE='';
+BORG_BACKUP_FILE='';
 # if we are mount or unmount (default is mount)
 UMOUNT=0;

@@ -42,7 +45,7 @@ while getopts ":c:m:uf:h" opt do
 			UMOUNT=1;
 			;;
 		f|file)
-			ATTIC_BACKUP_FILE=${OPTARG};
+			BORG_BACKUP_FILE=${OPTARG};
 			;;
 		h|help)
 			usage;
@@ -69,7 +72,7 @@ fi;

 if [ ${UMOUNT} -eq 0 ]; then
 	TARGET_SERVER='';
-	if [ -z "${ATTIC_BACKUP_FILE}" ]; then
+	if [ -z "${BORG_BACKUP_FILE}" ]; then
 		if [ ! -f "${BASE_FOLDER}${SETTINGS_FILE}" ]; then
 			echo "Cannot find ${BASE_FOLDER}${SETTINGS_FILE}";
 			exit 0;
@@ -82,7 +85,7 @@ if [ ${UMOUNT} -eq 0 ]; then
 		fi;
 		REPOSITORY=${TARGET_SERVER}${TARGET_FOLDER}${BACKUP_FILE};
 	else
-		REPOSITORY=${ATTIC_BACKUP_FILE};
+		REPOSITORY=${BORG_BACKUP_FILE};
 	fi;

 	# check that the repostiory exists
@@ -112,4 +115,4 @@ else
 	borg umount "${MOUNT_PATH}";
 fi;

-## END
+# __END__
Author	SHA1	Message	Date
Clemens Schwaighofer	beffbee20c	Add missing OPT_REMOTE to commands check, compact commands had OPT_REMOTE missing	2023-05-10 16:30:23 +09:00
Clemens Schwaighofer	72f3f86eb6	Test for different command lookup check	2022-11-22 09:36:30 +09:00
Clemens Schwaighofer	c0f9634442	add *.compact to git ignore file	2022-11-14 20:27:42 +09:00
Clemens Schwaighofer	0e379d6ce0	Bump borg file module version number	2022-08-30 10:00:05 +09:00
Clemens Schwaighofer	bba8be6a42	Exclude File check is if file is >0 and fix wrong check for tmp file Exclude file was only checked if it exists, even if it was empty it would start processing. In the exclude check there was also a wrong check for the tmp exclude file, it actually checked if the main exclude file was >0 instead of the tmp file.	2022-08-30 09:56:20 +09:00
Clemens Schwaighofer	ce3cdc7627	Readme update to fix Marktdown lint issues	2022-08-08 14:51:26 +09:00
Clemens Schwaighofer	67c0b00268	Initialize LAST COMPACT/CHECK date variable	2022-06-07 19:57:36 +09:00
Clemens Schwaighofer	abbcfa883f	Add compact auto run in intervals like check archive Same settings like check archive in days. Auto set to 1 to run after each prune. Can be controlled per module.	2022-06-06 14:58:36 +09:00
Clemens Schwaighofer	9739436589	Redirect STDERR output from borg backup to STDIN for ANSI code cleanup gitea dump prints out info/status messages to STDERR. The whole subprocess then returns this on STDERR and so is not parsed through the sed regex command. Redirect STDERR to STDIN	2022-05-31 09:54:56 +09:00
Clemens Schwaighofer	bf0364a0e7	Move lock file clean up to before any exit info is printed Just try to clean lock file always on exit	2022-04-25 08:57:01 +09:00
Clemens Schwaighofer	8e87503528	Add auto check functionality CHECK_INTERVAL or SUB_CHECK_INTERVAL in module settings. If set to 1, will check everytime. Ever other number is for n days after last check. FORCE_CHECK for check if repository has been setup os now renamed FORCE_VERIFY but FORCE_CHECK is currently still honored but deprecated all function shell scripts will abort if run on their own Print info on last check time if set in info block Internal updates: All separator lines are now printf strings for central control. All module used config/settings file names use $MODULE name Check functionality is in its own file	2022-04-25 06:49:57 +09:00
Clemens Schwaighofer	34277483e9	Borg check -p (progress) is only set if -v (verbose) flag is set Avoid any progress output if run from script	2022-04-18 16:00:41 +09:00
Clemens Schwaighofer	dffb7c6450	Fix borg check prefix settings	2022-04-18 15:58:03 +09:00
Clemens Schwaighofer	2ae05f5302	Add borg check and rename options Because we added borg check functionality, some of the Options have been renamed: -C -> -V -E -> -e (as it is a sub) -C: check -y: --verify-data -p: prefix or glob for check Internal variables with CHECK have been renamed or changed to VERIFY Borg -C without any extra parameters is equal to borg check. -y adds the --verify-data and -p is a mix of the -P and -a options. If there is a "*" in the option then -a will be used, else -P Note that repair command has to be run manually. Run -C with -v (verbose) to see the repair command structure. borg check can take a long time on very large repositories.	2022-04-18 15:41:28 +09:00
Clemens Schwaighofer	b5ead9a2e1	Show host name and module init date in the start info block	2022-04-01 11:16:38 +09:00
Clemens Schwaighofer	fa77876440	Update Version ID to 4.2.4	2022-03-31 11:09:11 +09:00
Clemens Schwaighofer	f128b7ebc4	Bug fix for no keep options set check	2022-03-31 11:01:02 +09:00
Clemens Schwaighofer	77977207c8	Change close parameter check to pure string type	2022-03-31 09:21:23 +09:00
Clemens Schwaighofer	c1f6bb443a	Fix for close with empty parameter	2022-03-31 09:18:23 +09:00
Clemens Schwaighofer	86b0fa122a	Add empty KEEP option for error with -T	2022-03-30 20:27:46 +09:00
Clemens Schwaighofer	32c320be27	Fix close call int compare	2022-03-30 14:48:52 +09:00
Clemens Schwaighofer	500ab01790	Bug fix for unset var in close call	2022-03-30 11:35:19 +09:00
Clemens Schwaighofer	ab58ab3ad0	Update postgresql module version	2022-03-30 09:48:37 +09:00
Clemens Schwaighofer	7767eb58df	Override sudo for postgresql in upgrade script	2022-03-30 09:32:57 +09:00
Clemens Schwaighofer	38f467de96	PostgreSQL backup add schema or data dump in either default direction	2022-03-29 11:21:06 +09:00
Clemens Schwaighofer	8f91690f6a	Upgrade script for zabbix fix if no zabbix prefix set at all	2022-03-29 10:53:59 +09:00
Clemens Schwaighofer	e860573e0c	Fix upgrade script gitea module rename check	2022-03-29 10:28:26 +09:00
Clemens Schwaighofer	f990e86949	update script gitea block fix for missing module name	2022-03-29 10:25:12 +09:00
Clemens Schwaighofer	c929987900	variable name wrong for module check in upgrade script	2022-03-29 10:20:52 +09:00
Clemens Schwaighofer	6cb941818c	Repository ID via config only works with local repos, drop that and keep repo ID in -i option only	2022-03-29 10:01:00 +09:00
Clemens Schwaighofer	edaf41f1af	Repository ID info, borg command call via variable The main info block gets Repository ID as info too (for cache/etc folders check) Missing borg command as variable for borg list command.	2022-03-29 09:57:12 +09:00
Clemens Schwaighofer	c7f2197614	add -E to sudo for upgrade script to keep enviroment variables for postgres run	2022-03-29 09:16:03 +09:00
Clemens Schwaighofer	88ea600e1d	Mising remote path option in update script	2022-03-29 08:55:19 +09:00
Clemens Schwaighofer	f396032728	Add remote override path for borg backup in update script	2022-03-29 08:53:37 +09:00
Clemens Schwaighofer	18cbcea2b0	Archive print page make archive space 45 chars wide	2022-03-29 07:01:31 +09:00
Clemens Schwaighofer	a0537a24d3	PostgreSQL sudo for upgrade script	2022-03-29 06:46:23 +09:00
Clemens Schwaighofer	89897eb676	Add base BORG export variables in upgrade script	2022-03-29 06:36:06 +09:00
Clemens Schwaighofer	e1787fcfb3	Add missing DEBUG var setting for borg update wrapper script	2022-03-28 12:37:33 +09:00
Clemens Schwaighofer	0ce5442bcf	Bug fix for update fix script Add missing / for BASE_FOLDER if not set. Dropped old file module borg name	2022-03-28 11:54:21 +09:00
Clemens Schwaighofer	d9346c84a7	Borg Backup wrapper Version 4.0 update * file backup borg folder has now -file name inside. Old folder must be manuall renamed * All modules have the module id name as prefix in the backup set, _borg_backup_set_prefix_cleanup.sh needs to be run before to clean up all names or prune will not correctly delete old entries New -T for one time target backup with custom prefix to have backups outside the automated prune. -D option to delete this set Add borg 1.2 support for compact which is called after delete and prune to actually clean up the space. -b borg executable and BORG_EXECUTEABLE override setting if borg is not in path or another borg executable should be used	2022-03-28 11:27:35 +09:00
Clemens Schwaighofer	828a59c984	Set proper default values for core settings. If nothing set in the settings file some entries need to be default set: COMPRESSION COMPRESSION_LEVEL ENCRYPTION FORCE_CHECK KEEP_LAST KEEP_HOURS KEEP_DAYS KEEP_WEEKS KEEP_MONTHS KEEP_YEARS If these variables are empty after settings file is ready they are set to the default value. After that the sub configs override those settings	2022-03-08 13:56:16 +09:00
Clemens Schwaighofer	ee1b3b23ab	Fixed missing init for OPT_COMPRESSION variable in functions check	2022-03-03 10:23:39 +09:00