Missed out duration calculation in postgresql run time output

Gitea dump has now settings for
- temp folder (GITEA_TEMP_DIR)
- new name for working dir (GITEA_WORKING_DIR) from old (GITEA_TMP)

MySQL and PostgreSQL exports have a run time output per db dump set and not only at the end of the full run

.gitignore

@@ -6,3 +6,4 @@ borg.backup.*.include
 borg.backup.*.exclude
 borg.backup.*.schema-only
 borg.backup.*.init
+borg.backup.*.compact

Readme.md

@@ -13,9 +13,11 @@ Version 4.0 introduces default borg repository name with `-file` for the `file`
 *Example:*
 
 Old backup name
+
 ```sh
 BACKUP_FILE="some-backup-data.borg"
 ```
+
 Then the file need to be renamed the following way:
 `mv some-backup-data.borg some-backup-data-file.borg`
 
@@ -42,57 +44,79 @@ No settings files will be overwritten
 ## Possible command line options
 
 ### `-c <config folder>`
+
 if this is not given, /usr/local/scripts/borg/ is used
 
 ### `-L <log folder>`
+
 override config set and default log folder
 
 ### `-T <tag>`
+
 create one time stand alone backup prefixed with tag name
 
 ### `-D <tag backup set>`
+
 remove a tagged backup set, full name must be given
 
 ### `-b <borg executable>`
+
 override the default borg executable found in path
 
 ### `-P`
+
 print list of archives created
 
 ### `-V`
+
 verify if repository exists, if not abort
 
 ### `-e`
-exit after check
+
+exit after running verify `-V`
 
 ### `-I`
+
 init repository (must be run first)
 
+### `-Z`
+
+run `borg compact` over given repository
+
 ### `-C`
+
 run `borg check` over given repository
 
 #### `-y`
+
 Add `--verify-data` to `borg check`. Only works with `-C`
 
 #### `-p <prefix|glob>`
+
 Only `borg check` data that has given prefix or glob (with *). Only works with `-C`
 
 ### `-i`
+
 print out only info
 
 ### `-l`
+
 list files during backup
 
 ### `-v`
+
 be verbose
 
 ### `-d`
+
 debug output all commands
 
 ### `-n`
+
 only do dry run
 
 ### `-h`
+
 this help page
 
 ## Basic Settings
@@ -106,7 +130,6 @@ LOG_FOLDER: default `/var/log/borg.backup/`
 TARGET_FOLDER: must be set to a path where the backups can be written
 BACKUP_FILE: the folder inside the TARGET_FOLDER that is the target for borg. Must end with `.borg`
 
-
 Note: BACKUP_FILE is the base name. For all except file (current) a module suffix will be added:
 
 eg:
@@ -115,35 +138,41 @@ eg:
 If `FILE_REPOSITORY_COMPATIBLE` is set to `false` in the borg.backup.file.settings then the file borg name will have `-file` added too. Currently this is not added to stay compatible with older scripts
 
 All below have default values if not set in the main settings file
- * COMPRESSION: zstd
+
- * COMPRESSION_LEVEL: 3
+* COMPRESSION: zstd
- * ENCRYPTION: none
+* COMPRESSION_LEVEL: 3
- * FORCE_CHECK: false
+* ENCRYPTION: none
- * KEEP_LAST: 0
+* FORCE_VERIFY: false
- * KEEP_HOURS: 0
+* COMPACT_INTERVAL: 1
- * KEEP_DAYS: 7
+* CHECK_INTERVAL: none
- * KEEP_WEEKS: 4
+* KEEP_LAST: 0
- * KEEP_MONTHS: 6
+* KEEP_HOURS: 0
- * KEEP_YEARS: 1
+* KEEP_DAYS: 7
+* KEEP_WEEKS: 4
+* KEEP_MONTHS: 6
+* KEEP_YEARS: 1
 
 All module settings files can have the following prefixed with `SUB_` to override master settings:
- * SUB_BACKUP_FILE
+
- * SUB_COMPRESSION
+* SUB_BACKUP_FILE
- * SUB_COMPRESSION_LEVEL
+* SUB_COMPRESSION
- * SUB_BACKUP_SET
+* SUB_COMPRESSION_LEVEL
- * SUB_KEEP_LAST
+* SUB_COMPACT_INTERVAL
- * SUB_KEEP_HOURS
+* SUB_CHECK_INTERVAL
- * SUB_KEEP_DAYS
+* SUB_BACKUP_SET
- * SUB_KEEP_WEEKS
+* SUB_KEEP_LAST
- * SUB_KEEP_MONTHS
+* SUB_KEEP_HOURS
- * SUB_KEEP_YEARS
+* SUB_KEEP_DAYS
- * SUB_KEEP_WITHIN
+* SUB_KEEP_WEEKS
+* SUB_KEEP_MONTHS
+* SUB_KEEP_YEARS
+* SUB_KEEP_WITHIN
 
 ## Setup backup via SSH to remote host on `borg.backup.settings`
 
 For this the following settings are from interest
 
-```
+```sh
 TARGET_USER="";
 TARGET_HOST="";
 TARGET_PORT="";
@@ -157,6 +186,14 @@ and `TARGET_BORG_PATH="";` if the target borg is in a non default path
 
 `BORG_EXECUTABLE="<full path to borg>"`
 
+## Note on CHECK_INTERVAL and SUB_CHECK_INTERVAL
+
+If set to empty or 0 it will not run an automatic check. If set to 1 it will run a check after each backup. Any other value means days differente to the last check.
+
+Running check manually (`-C`) will not reset the last check timestamp.
+
+Automatic checks always add `--verify-data`, with manual `-C` the option `-y` has to be set.
+
 ## File backup settings
 
 On new setups it is recommended to use the `borg.backup.file.setings` and set
@@ -165,10 +202,9 @@ to `true`
 
 ### Config variables
 
-
 ### Control files
 
-```
+```sh
 backup.borg.file.include
 backup.borg.file.exclude
 ```
@@ -182,16 +218,16 @@ backup.borg.file.exclude
 This script must be run as the postgres user, normaly `postgres`.
 The postgres user must be added to the backup group for this, so that the basic init file can be created in the borg base folder.
 
-### Config variables
+### PostgreSQL Config variables
 
-Variable | Default | Description
+| Variable | Default | Description |
 | - | - | - |
-DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema
+| DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema |
-DATABASE_USER | | overide username to connect to database
+| DATABASE_USER | | overide username to connect to database |
 
-### Control files
+### PostgreSQL Control files
 
-```
+```sh
 backup.borg.pgsql.include
 backup.borg.pgsql.exclude
 backup.borg.pgsql.schema-only
@@ -201,16 +237,16 @@ backup.borg.pgsql.schema-only
 
 If non root ident authentication run is used, be sure that the `mysql` user is in the backup group.
 
-### Config variables
+### MySQL Config variables
 
-Variable | Default | Description
+| Variable | Default | Description |
 | - | - | - |
-DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema
+| DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema |
-MYSQL_DB_CONFIG | | override file for connection. In modern mariaDB installations it is rcommended to run the script as root or mysql user and use the ident authentication instead.
+| MYSQL_DB_CONFIG | | override file for connection. In modern mariaDB installations it is rcommended to run the script as root or mysql user and use the ident authentication instead. |
 
-### Control files
+### MySQLControl files
 
-```
+```sh
 backup.borg.mysql.include
 backup.borg.mysql.exclude
 backup.borg.mysql.schema-only
@@ -221,31 +257,49 @@ backup.borg.mysql.schema-only
 Note that the backup needs the GIT_USER set that runs gitea.
 This user is neede to create the temporary dump folder and access for the git files and database.
 
-### Config Variables
+### gitea Config Variables
 
-Variable | Default | Description
+| Variable | Default | Description |
 | - | - | - |
-GIT_USER | git | The user that runs gitea |
+| GIT_USER | git | The user that runs gitea |
-GITEA_TMP | /tmp/gitea/ | Where the temporary dump files from the backup are stored, as user git |
+| GITEA_WORKING_DIR | /var/tmp/gitea/ | Where the temporary dump files from the backup are stored, as user git |
-GITEA_BIN | /usr/local/bin/gitea | Where the gitea binary is located |
+| GITEA_TEMP_DIR | /var/tmp/ | General temporary folder |
-GITEA_CONFIG | /etc/gitea/app.ini | The configuration file for gitea |
+| GITEA_BIN | /usr/local/bin/gitea | Where the gitea binary is located |
+| GITEA_CONFIG | /etc/gitea/app.ini | The configuration file for gitea |
 
-
+### gitea Control files
-### Control files
 
 There are no control files for gitea backup
 
 ## zabbix config backup settings
 
-### Config Variables
+### zabbix Config Variables
 
-Variable | Default | Description
+| Variable | Default | Description |
 | - | - | - |
-ZABBIX_DUMP | /usr/local/bin/zabbix-dump |
+| ZABBIX_DUMP | /usr/local/bin/zabbix-dump | |
-ZABBIX_DATABASE | '' | Must be set as either psql or mysql
+| ZABBIX_DATABASE | '' | Must be set as either psql or mysql |
-ZABBIX_CONFIG | '' | if not set uses default location
+| ZABBIX_CONFIG | '' | if not set uses default location |
-ZABBIX_UNKNOWN_TABLES | '' | if set, changed to -f (force)
+| ZABBIX_UNKNOWN_TABLES | '' | if set, changed to -f (force) |
 
-### Control files
+### zabbix Control files
 
 There are no control files for zabbix settings backup
+
+## File connection
+
+Running any of the commands below
+
+* borg.backup.file.sh
+* borg.backup.gitea.sh
+* borg.backup.mysql.sh
+* borg.backup.pgsql.sh
+* borg.backup.zabbix.sh
+
+1) Run `borg.backup.functions.init.sh` (always)
+2) Run `borg.backup.functions.verify.sh` (always)
+3) (other code in "file" module)
+4) Run `borg.backup.functions.info.sh` (always)
+5) Run `borg.backup.functions.compact.sh` (not if one time tag)
+6) Run `borg.backup.functions.check.sh` (not if one time tag)
+7) Run `borg.backup.functions.close.sh` (always, can be called on error too)

borg.backup.file.sh

@@ -4,17 +4,21 @@
 
 # set last edit date + time
 MODULE="file";
-MODULE_VERSION="1.2.1";
+MODULE_VERSION="1.2.3";
 
 DIR="${BASH_SOURCE%/*}"
 if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 . "${DIR}/borg.backup.functions.init.sh";
 
 # include and exclude file
-INCLUDE_FILE="borg.backup.file.include";
+INCLUDE_FILE="borg.backup.${MODULE}.include";
-EXCLUDE_FILE="borg.backup.file.exclude";
+EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
-# init verify file
+# init verify, compact and check file
-BACKUP_INIT_VERIFY="borg.backup.file.init";
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
 
 # verify valid data
 . "${DIR}/borg.backup.functions.verify.sh";
@@ -25,7 +29,7 @@ if [ ! -f "${BASE_FOLDER}${INCLUDE_FILE}" ]; then
 	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
-echo "--- [INCLUDE: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+printf "${PRINTF_SUB_BLOCK}" "INCLUDE" "$(date +'%F %T')" "${MODULE}";
 # folders to backup
 FOLDERS=();
 # this if for debug output with quoted folders
@@ -88,9 +92,9 @@ while read include_folder; do
 	fi;
 done<"${BASE_FOLDER}${INCLUDE_FILE}";
 
-# exclude list
+# exclude list, only if file exists and is larger than zero
-if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
+if [ -s "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
-	echo "--- [EXCLUDE: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "EXCLUDE" "$(date +'%F %T')" "${MODULE}";
 	# check that the folders in that exclude file are actually valid,
 	# remove non valid ones and warn
 	#TMP_EXCLUDE_FILE=$(mktemp --tmpdir ${EXCLUDE_FILE}.XXXXXXXX); # non mac
@@ -143,7 +147,7 @@ if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
 		fi;
 	done<"${BASE_FOLDER}${EXCLUDE_FILE}";
 	# avoid blank file add by checking if the tmp file has a size >0
-	if [ -s "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
+	if [ -s "${TMP_EXCLUDE_FILE}" ]; then
 		OPT_EXCLUDE="--exclude-from=${TMP_EXCLUDE_FILE}";
 	fi;
 fi;
@@ -159,7 +163,7 @@ COMMAND=${COMMAND}${REPOSITORY}::${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${BACKUP_SET
 . "${DIR}/borg.backup.functions.info.sh";
 
 if [ $FOLDER_OK -eq 1 ]; then
-	echo "--- [BACKUP: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "BACKUP" "$(date +'%F %T')" "${MODULE}";
 	# show command
 	if [ ${DEBUG} -eq 1 ]; then
 		echo $(echo ${COMMAND} | sed -e 's/[ ][ ]*/ /g') ${FOLDERS_Q[*]};
@@ -182,7 +186,7 @@ fi;
 
 # clean up, always verbose, but only if we do not run one time tag
 if [ -z "${ONE_TIME_TAG}" ]; then
-	echo "--- [PRUNE : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}";
 	# build command
 	COMMAND="${BORG_COMMAND} prune ${OPT_REMOTE} -v --list ${OPT_PROGRESS} ${DRY_RUN_STATS} -P ${BACKUP_SET_PREFIX} ${KEEP_OPTIONS[*]} ${REPOSITORY}";
 	echo "Prune repository with keep${KEEP_INFO:1}";
@@ -192,7 +196,9 @@ if [ -z "${ONE_TIME_TAG}" ]; then
 	# for the IFS="#" to work we need to replace options spaces with exactly ONE #
 	$(echo "${COMMAND}" | sed -e 's/[ ][ ]*/#/g') 2>&1 || echo "[!] Borg prune aborted";
 	# if this is borg version >1.2 we need to run compact after prune
-	. "${DIR}/borg.backup.functions.compact.sh";
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
 else
 	echo "[#] No prune with tagged backup";
 fi;

borg.backup.functions.check.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
+# run borg check (NOT REPAIR)
+
+RUN_CHECK=0;
+if [ $# -ge 1 ] && [ "$1" = "auto" ]; then
+	# strip any spaces and convert to int
+	CHECK_INTERVAL=$(echo "${CHECK_INTERVAL}" | sed -e 's/ //g');
+	# not a valid check interval, do no check
+	if [ -z "${CHECK_INTERVAL##*[!0-9]*}" ]; then
+		CHECK_INTERVAL=0;
+	fi;
+	# get current date timestmap
+	CURRENT_DATE=$(date +%s);
+	# if =1 always ok
+	if [ ${CHECK_INTERVAL} -eq 1 ]; then
+		RUN_CHECK=1;
+		# always add verify data for automatic check
+		OPT_CHECK_VERIFY_DATA="--verify-data";
+		# set new check time here
+		echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_CHECK_FILE}";
+	elif [ ${CHECK_INTERVAL} -gt 1 ]; then
+		# else load last timestamp and check if today - last time stamp > days
+		if [ -z "${LAST_CHECK_DATE}" ]; then
+			LAST_CHECK_DATE=$(cat "${BASE_FOLDER}${BACKUP_CHECK_FILE}" 2>/dev/null | sed -e 's/ //g');
+		fi;
+		# file date is not a timestamp
+		if [ -z "${LAST_CHECK_DATE##*[!0-9]*}" ]; then
+			LAST_CHECK_DATE=0;
+		fi;
+		# if the difference greate than check date, run. CHECK INTERVAL is in days
+		if [ $(($CURRENT_DATE-$LAST_CHECK_DATE)) -ge $((${CHECK_INTERVAL}*86400)) ]; then
+			RUN_CHECK=1;
+			# always add verify data for automatic check
+			OPT_CHECK_VERIFY_DATA="--verify-data";
+			# set new check time here
+			echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_CHECK_FILE}";
+		fi;
+	fi;
+elif [ ${CHECK} -eq 1 ]; then
+	RUN_CHECK=1;
+fi;
+
+if [ ${RUN_CHECK} -eq 1 ]; then
+	# run borg check command
+	IFS=${_IFS};
+	printf "${PRINTF_SUB_BLOCK}" "CHECK" "$(date +'%F %T')" "${MODULE}";
+	# repare command
+	OPT_GLOB="";
+	if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then
+		OPT_GLOB="-a '${CHECK_PREFIX}'"
+	elif [ ! -z "${CHECK_PREFIX}" ]; then
+		OPT_GLOB="-P ${CHECK_PREFIX}";
+	fi;
+	# debug/dryrun
+	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
+		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY}";
+	fi;
+	# run info command if not a dry drun
+	if [ ${DRYRUN} -eq 0 ]; then
+		# if glob add glob command directly
+		if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then
+		${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} -a "${CHECK_PREFIX}" ${REPOSITORY};
+		else
+			${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY};
+		fi;
+	fi;
+	# print additional info for use --repair command
+	# but only for manual checks
+	if [ ${VERBOSE} -eq 1 ] && [ ${CHECK} -eq 1 ]; then
+		echo "";
+		echo "In case of needed repair: "
+		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} --repair ${OPT_GLOB} ${REPOSITORY}";
+		echo "Before running repair, a copy from the backup should be made because repair might damage a backup"
+	fi;
+fi;
+
+# __END__

borg.backup.functions.close.sh

@@ -1,14 +1,24 @@
 #!/usr/bin/env bash
 
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
 # unset borg settings
 unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK
+# delete lock file if it exists
+if [ -f "${BASE_FOLDER}${BACKUP_LOCK_FILE}" ]; then
+	rm "${BASE_FOLDER}${BACKUP_LOCK_FILE}";
+fi;
 # error abort without duration and error notice
 if [ $# -ge 1 ] && [ "$1" = "1" ]; then
-	echo "=== [ERROR: $(date +'%F %T')] ==[${MODULE}]====================================>";
+	printf "${PRINTF_MASTER_BLOCK}" "ERROR" "$(date +'%F %T')" "${MODULE}";
 else
+	# running time calculation
 	DURATION=$[ $(date +'%s')-$START ];
 	echo "=== [Run time: $(convert_time ${DURATION})]";
-	echo "=== [END  : $(date +'%F %T')] ==[${MODULE}]====================================>";
+	printf "${PRINTF_MASTER_BLOCK}" "END" "$(date +'%F %T')" "${MODULE}";
 fi;
 
 # __END__

borg.backup.functions.compact.sh

@@ -1,18 +1,58 @@
 #!/usr/bin/env bash
 
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
 # compact (only if BORG COMPACT is set)
 # only for borg 1.2
-
-# reset to normal IFS, so command works here
-IFS=${_IFS};
 if [ $(version $BORG_VERSION) -ge $(version "1.2.0") ]; then
-	echo "--- [COMPACT:$(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	RUN_COMPACT=0;
-	BORG_COMPACT="${BORG_COMMAND} compact -v ${OPT_PROGRESS} ${REPOSITORY}";
+	if [ $# -ge 1 ] && [ "$1" = "auto" ]; then
-	if [ ${DEBUG} -eq 1 ]; then
+		# strip any spaces and convert to int
-			echo "${BORG_COMPACT}";
+		COMPACT_INTERVAL=$(echo "${COMPACT_INTERVAL}" | sed -e 's/ //g');
+		# not a valid compact interval, do no compact
+		if [ -z "${COMPACT_INTERVAL##*[!0-9]*}" ]; then
+			COMPACT_INTERVAL=0;
+		fi;
+		# get current date timestmap
+		CURRENT_DATE=$(date +%s);
+		if [ ${COMPACT_INTERVAL} -eq 1 ]; then
+			RUN_COMPACT=1;
+			# set new compact time here
+			echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_COMPACT_FILE}";
+		elif [ ${COMPACT_INTERVAL} -gt 1 ]; then
+			# else load last timestamp and check if today - last time stamp > days
+			if [ -z "${LAST_COMPACT_DATE}" ]; then
+				LAST_COMPACT_DATE=$(cat "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" 2>/dev/null | sed -e 's/ //g');
+			fi;
+			# file date is not a timestamp
+			if [ -z "${LAST_COMPACT_DATE##*[!0-9]*}" ]; then
+				LAST_COMPACT_DATE=0;
+			fi;
+			# if the difference greate than compact date, run. COMPACT INTERVAL is in days
+			if [ $(($CURRENT_DATE-$LAST_COMPACT_DATE)) -ge $((${COMPACT_INTERVAL}*86400)) ]; then
+				RUN_COMPACT=1;
+				# set new compact time here
+				echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_COMPACT_FILE}";
+			fi;
+		fi;
+	elif [ ${COMPACT} -eq 1 ]; then
+		RUN_COMPACT=1;
 	fi;
-	if [ ${DRYRUN} -eq 0 ]; then
+
-		${BORG_COMPACT};
+	if [ ${RUN_COMPACT} -eq 1 ]; then
+		# reset to normal IFS, so command works here
+		IFS=${_IFS};
+		printf "${PRINTF_SUB_BLOCK}" "COMPACT" "$(date +'%F %T')" "${MODULE}";
+		BORG_COMPACT="${BORG_COMMAND} compact ${OPT_REMOTE} -v ${OPT_PROGRESS} ${REPOSITORY}";
+		if [ ${DEBUG} -eq 1 ]; then
+				echo "${BORG_COMPACT}";
+		fi;
+		if [ ${DRYRUN} -eq 0 ]; then
+			${BORG_COMPACT};
+		fi;
 	fi;
 fi;
 

borg.backup.functions.info.sh

@@ -1,7 +1,12 @@
 #!/usr/bin/env bash
 
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
 if [ ${INFO} -eq 1 ]; then
-	echo "--- [INFO  : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "INFO" "$(date +'%F %T')" "${MODULE}";
 	# show command on debug or dry run
 	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY}";

borg.backup.functions.init.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
 
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
 set -ETu #-e -o pipefail
 trap cleanup SIGINT SIGTERM ERR
 
@@ -19,7 +24,7 @@ function version {
 }
 
 # version for all general files
-VERSION="4.3.0";
+VERSION="4.4.0";
 
 # borg version and borg comamnd
 BORG_VERSION="";
@@ -42,8 +47,12 @@ SETTINGS_FILE="borg.backup.settings";
 INCLUDE_FILE="";
 EXCLUDE_FILE="";
 # backup folder initialzed verify
-BACKUP_INIT_VERIFY="";
+BACKUP_INIT_FILE="";
 BACKUP_INIT_DATE="";
+# file with last compact date
+BACKUP_COMPACT_FILE="";
+# file with last check timestamp
+BACKUP_CHECK_FILE="";
 # one time backup prefix tag, if set will use <tag>.<prefix>-Y-M-DTh:m:s type backup prefix
 ONE_TIME_TAG="";
 DELETE_ONE_TIME_TAG="";
@@ -58,6 +67,7 @@ INFO=0;
 VERIFY=0;
 CHECK=0;
 CHECK_VERIFY_DATA=0;
+COMPACT=0;
 INIT=0;
 EXIT=0;
 PRINT=0;
@@ -80,6 +90,13 @@ PRUNE_DEBUG="";
 INIT_REPOSITORY=0;
 FOLDER_OK=0;
 TMP_EXCLUDE_FILE="";
+# printf strings
+PRINTF_INFO_STRING="%-23s: %s\n";
+PRINTF_MASTER_BLOCK="=== [%-8s: %19s] ==[%s]====================================>\n";
+PRINTF_SUB_BLOCK="|-- [%-8s: %19s] --[%s]------------------------------------>\n";
+PRINTF_SUBEXT_BLOCK="|-- [%-8s: %s: %19s] --[%s]------------------------------------>\n";
+PRINTF_DB_SUB_BLOCK=">>- [%-8s: %s] ==[%s]=======================>\n";
+PRINTF_DB_RUN_TIME_SUB_BLOCK=">>- [%-8s: %s] ==[%s]==[Run time: %s:]=======================>\n";
 # opt flags
 OPT_VERBOSE="";
 OPT_PROGRESS="";
@@ -114,6 +131,18 @@ ENCRYPTION="";
 DEFAULT_FORCE_VERIFY="false";
 FORCE_VERIFY="";
 FORCE_CHECK=""; # Deprecated name, use FORCE_VERIFY
+# compact
+DEFAULT_COMPACT_INTERVAL="1";
+LAST_COMPACT_DATE="";
+COMPACT_INTERVAL="";
+SUB_COMPACT_INTERVAL="";
+# check
+# default interval is none
+DEFAULT_CHECK_INTERVAL="";
+LAST_CHECK_DATE="";
+CHECK_INTERVAL="";
+SUB_CHECK_INTERVAL="";
+# backup set names
 BACKUP_SET="";
 SUB_BACKUP_SET="";
 # for database backup only
@@ -124,9 +153,11 @@ MYSQL_DB_CONFIG="";
 MYSQL_DB_CONFIG_PARAM="";
 # gitea module
 GIT_USER="";
-GITEA_TMP="";
+GITEA_WORKING_DIR="";
+GITEA_TEMP_DIR="";
 GITEA_BIN="";
 GITEA_CONFIG="";
+GITEA_EXPORT_TYPE="";
 # zabbix module
 ZABBIX_DUMP_BIN="";
 ZABBIX_CONFIG="";
@@ -174,6 +205,7 @@ function usage()
 	-T <tag>: create one time stand alone backup prefixed with tag name
 	-D <tag backup set>: remove a tagged backup set, full name must be given
 	-b <borg executable>: override default path
+	-Z: run compress after prune/backup. Only for borg 1.2 or newer
 	-C: run borg check if repository is ok
 	-y: in combination with -C: add --verify-data
 	-p <archive prefix|glob>: in combinatio with -C: only check archives with prefix or glob
@@ -195,7 +227,7 @@ function usage()
 }
 
 # set options
-while getopts ":c:L:T:D:b:p:vldniCVeIPyh" opt; do
+while getopts ":c:L:T:D:b:p:vldniCVeIPyZh" opt; do
 	case "${opt}" in
 		c|config)
 			BASE_FOLDER=${OPTARG};
@@ -212,6 +244,10 @@ while getopts ":c:L:T:D:b:p:vldniCVeIPyh" opt; do
 		b|borg)
 			OPT_BORG_EXECUTEABLE=${OPTARG};
 			;;
+		Z|Compact)
+			# will run compact alone
+			COMPACT=1;
+			;;
 		C|Check)
 			# will run borg check
 			# alt modes --repository-only, --archives-only,
@@ -380,6 +416,7 @@ elif [ ! -z "${BORG_EXECUTEABLE}" ]; then
 		exit;
 	fi;
 elif ! command -v borg &> /dev/null; then
+# elif [ -z ($command -v borg) ]; then
 	echo "borg backup seems not to be installed, please verify paths";
 	exit;
 fi;
@@ -404,6 +441,13 @@ fi;
 if [ -z "${ENCRYPTION}" ]; then
 	ENCRYPTION="${DEFAULT_ENCRYPTION}";
 fi;
+# check interval override
+if [ -z "${COMPACT_INTERVAL}" ]; then
+	COMPACT_INTERVAL="${DEFAULT_COMPACT_INTERVAL}";
+fi;
+if [ -z "${CHECK_INTERVAL}" ]; then
+	CHECK_INTERVAL="${DEFAULT_CHECK_INTERVAL}";
+fi;
 # deprecated name FORCE_CHECK, use FORCE_VERIFY instead
 if [ ! -z "${FORCE_CHECK}" ]; then
 	FORCE_VERIFY="${FORCE_CHECK}";
@@ -450,6 +494,14 @@ if [ -f "${BASE_FOLDER}${SETTINGS_FILE_SUB}" ]; then
 	if [ ! -z "${SUB_COMPRESSION_LEVEL}" ]; then
 		COMPRESSION_LEVEL=${SUB_COMPRESSION_LEVEL};
 	fi;
+	# compact interval override
+	if [ ! -z "${SUB_COMPACT_INTERVAL}" ]; then
+		COMPACT_INTERVAL="${SUB_COMPACT_INTERVAL}";
+	fi;
+	# override check interval
+	if [ ! -z "${SUB_CHECK_INTERVAL}" ]; then
+		CHECK_INTERVAL="${SUB_CHECK_INTERVAL}";
+	fi;
 	# check override for keep time
 	if [ ! -z "${SUB_KEEP_LAST}" ]; then
 		KEEP_LAST=${SUB_KEEP_LAST};
@@ -512,6 +564,9 @@ if [[ ${TARGET_FOLDER} =~ ^~\/ ]]; then
 	exit 1;
 fi
 
+# COMPACT_INTERVAL must be a number from -1 to 365
+# CHECK_INTERVAL must be a number from -1 to 365
+
 # log file set and check
 # option folder overrides all other folders
 if [ ! -z "${OPT_LOG_FOLDER}" ]; then

borg.backup.functions.verify.sh

@@ -1,25 +1,59 @@
 #!/usr/bin/env bash
 
+if [ -z "${MODULE}" ]; then
+	echo "Script cannot be run on its own";
+	exit 1;
+fi;
+
 # start time in seconds
 START=$(date +'%s');
 # set init date, or today if not file is set
-BACKUP_INIT_DATE=$(printf '%(%c)T' $(cat "${BASE_FOLDER}${BACKUP_INIT_VERIFY}" 2>/dev/null));
+BACKUP_INIT_DATE='';
+if [ -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
+	BACKUP_INIT_DATE=$(printf '%(%c)T' $(cat "${BASE_FOLDER}${BACKUP_INIT_FILE}" 2>/dev/null));
+fi;
 # start logging from here
 exec &> >(tee -a "${LOG}");
-echo "=== [START : $(date +'%F %T')] ==[${MODULE}]====================================>";
+printf "${PRINTF_MASTER_BLOCK}" "START" "$(date +'%F %T')" "${MODULE}";
 # show info for version always
-echo "Script version  : ${VERSION}";
+printf "${PRINTF_INFO_STRING}" "Script version" "${VERSION}";
 # show type
-echo "Backup module   : ${MODULE}";
+printf "${PRINTF_INFO_STRING}" "Backup module" "${MODULE}";
-echo "Module version  : ${MODULE_VERSION}";
+printf "${PRINTF_INFO_STRING}" "Module version" "${MODULE_VERSION}";
 # borg version
-echo "Borg version    : ${BORG_VERSION}";
+printf "${PRINTF_INFO_STRING}" "Borg version" "${BORG_VERSION}";
 # host name
-echo "Hostname        : ${HOSTNAME}";
+printf "${PRINTF_INFO_STRING}" "Hostname" "${HOSTNAME}";
 # show base folder always
-echo "Base folder     : ${BASE_FOLDER}";
+printf "${PRINTF_INFO_STRING}" "Base folder" "${BASE_FOLDER}";
 # Module init date (when init file was writen)
-echo "Module init date: ${BACKUP_INIT_DATE}";
+printf "${PRINTF_INFO_STRING}" "Module init date" "${BACKUP_INIT_DATE}";
+# print last compact date if positive integer
+# only if borg > 1.2
+if [ $(version $BORG_VERSION) -ge $(version "1.2.0") ]; then
+	if [ "${COMPACT_INTERVAL##*[!0-9]*}" ]; then
+		printf "${PRINTF_INFO_STRING}" "Module compact interval" "${COMPACT_INTERVAL}";
+		if [ -f "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" ]; then
+			LAST_COMPACT_DATE=$(cat "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" 2>/dev/null);
+			printf "${PRINTF_INFO_STRING}" "Module last compact" \
+				"$(printf '%(%c)T' ${LAST_COMPACT_DATE}) ($(convert_time $(($(date +%s)-${LAST_COMPACT_DATE}))) ago)";
+		else
+			printf "${PRINTF_INFO_STRING}" "Module last compact" "No compact run yet"
+		fi;
+	fi;
+fi;
+# print last check date if positive integer
+if [ "${CHECK_INTERVAL##*[!0-9]*}" ]; then
+	printf "${PRINTF_INFO_STRING}" "Module check interval" "${CHECK_INTERVAL}";
+	# get last check date
+	if [ -f "${BASE_FOLDER}${BACKUP_CHECK_FILE}" ]; then
+		LAST_CHECK_DATE=$(cat "${BASE_FOLDER}${BACKUP_CHECK_FILE}" 2>/dev/null);
+		printf "${PRINTF_INFO_STRING}" "Module last check" \
+			"$(printf '%(%c)T' ${LAST_CHECK_DATE}) ($(convert_time $(($(date +%s)-${LAST_CHECK_DATE}))) ago)";
+	else
+		printf "${PRINTF_INFO_STRING}" "Module last check" "No check run yet";
+	fi;
+fi;
 
 # if force verify is true set VERIFY to 1 unless INFO is 1
 # Needs bash 4.0 at lesat for this
@@ -82,7 +116,7 @@ elif [ ! -z "${TARGET_HOST}" ]; then
 fi;
 # we dont allow special characters, so we don't need to special escape it
 REPOSITORY="${TARGET_SERVER}${TARGET_FOLDER}${BACKUP_FILE}";
-echo "Repository      : ${REPOSITORY}";
+printf "${PRINTF_INFO_STRING}" "Repository" "${REPOSITORY}";
 
 # check if given compression name and level are valid
 OPT_COMPRESSION='';
@@ -205,6 +239,22 @@ else
 	fi;
 fi;
 
+# check if we have lock file, check pid in lock file, if no matching pid found
+# running remove lock file
+if [ -f "${BASE_FOLDER}${BACKUP_LOCK_FILE}" ]; then
+	LOCK_PID=$(cat "${BASE_FOLDER}${BACKUP_LOCK_FILE}" 2>/dev/null);
+	# check if lock file pid has an active program attached to it
+	if [ -f /proc/${LOCK_PID}/cmdline ]; then
+		echo "Script is already running on PID: ${$}";
+		. "${DIR}/borg.backup.functions.close.sh" 1;
+		exit 1;
+	else
+		echo "[#] Clean up stale lock file for PID: ${LOCK_PID}";
+		rm "${BASE_FOLDER}${BACKUP_LOCK_FILE}";
+	fi;
+fi;
+echo "${$}" > "${BASE_FOLDER}${BACKUP_LOCK_FILE}";
+
 # for folders list split set to "#" and keep the old setting as is
 _IFS=${IFS};
 IFS="#";
@@ -239,7 +289,7 @@ COMMAND_INFO="${COMMAND_EXPORT}${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY}"
 # else a normal verify is ok
 # unless explicit given, verify is skipped
 if [ ${VERIFY} -eq 1 ] || [ ${INIT} -eq 1 ]; then
-	echo "--- [VERIFY: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "VERIFY" "$(date +'%F %T')" "${MODULE}";
 	if [ ! -z "${TARGET_SERVER}" ]; then
 		if [ ${DEBUG} -eq 1 ]; then
 			echo "${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY} 2>&1|grep \"Repository ID:\"";
@@ -256,10 +306,10 @@ if [ ${VERIFY} -eq 1 ] || [ ${INIT} -eq 1 ]; then
 	fi;
 	# if verrify but no init and repo is there but init file is missing set it
 	if [ ${VERIFY} -eq 1 ] && [ ${INIT} -eq 0 ] && [ ${INIT_REPOSITORY} -eq 0 ] &&
-		[ ! -f "${BASE_FOLDER}${BACKUP_INIT_VERIFY}" ]; then
+		[ ! -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
 		# write init file
 		echo "[!] Add missing init verify file";
-		echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_VERIFY}";
+		echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_FILE}";
 	fi;
 	# end if verified but repository is not here
 	if [ ${VERIFY} -eq 1 ] && [ ${INIT} -eq 0 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then
@@ -276,7 +326,7 @@ if [ ${VERIFY} -eq 1 ] || [ ${INIT} -eq 1 ]; then
 	fi;
 fi;
 if [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then
-	echo "--- [INIT  : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "INIT" "$(date +'%F %T')" "${MODULE}";
 	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 		echo "${BORG_COMMAND} init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY}";
 	fi
@@ -284,7 +334,7 @@ if [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then
 		# should trap and exit properly here
 		${BORG_COMMAND} init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY};
 		# write init file
-		echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_VERIFY}";
+		echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_FILE}";
 		echo "Repository initialized";
 		echo "For more information run:"
 		echo "${COMMAND_INFO}";
@@ -301,7 +351,7 @@ elif [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 0 ]; then
 fi;
 
 # verify for init file
-if [ ! -f "${BASE_FOLDER}${BACKUP_INIT_VERIFY}" ]; then
+if [ ! -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
 	echo "[! $(date +'%F %T')] It seems the repository has never been initialized."
 	echo "Please run -I to initialize or if already initialzed run with -C for init update."
 	. "${DIR}/borg.backup.functions.close.sh" 1;
@@ -310,7 +360,7 @@ fi;
 
 # PRINT OUT current data, only do this if REPO exists
 if [ ${PRINT} -eq 1 ]; then
-	echo "--- [PRINT : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "PRINT" "$(date +'%F %T')" "${MODULE}";
 	FORMAT="{archive:<45} {comment:6} {start} - {end} [{id}] ({username}@{hostname}){NL}"
 	# show command on debug or dry run
 	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
@@ -338,43 +388,23 @@ if [ ${PRINT} -eq 1 ]; then
 	exit;
 fi;
 
-# run borg check command
+# run borg compact command and exit
+if [ ${COMPACT} -eq 1 ]; then
+	. "${DIR}/borg.backup.functions.compact.sh";
+	. "${DIR}/borg.backup.functions.close.sh";
+	exit;
+fi;
+
+# run borg check command and exit
 if [ ${CHECK} -eq 1 ]; then
-	echo "--- [CHECK : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	. "${DIR}/borg.backup.functions.check.sh";
-	# repare command
-	OPT_GLOB="";
-	if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then
-		OPT_GLOB="-a '${CHECK_PREFIX}'"
-	elif [ ! -z "${CHECK_PREFIX}" ]; then
-		OPT_GLOB="-P ${CHECK_PREFIX}";
-	fi;
-	# debug/dryrun
-	if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
-		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY}";
-	fi;
-	# run info command if not a dry drun
-	if [ ${DRYRUN} -eq 0 ]; then
-		# if glob add glob command directly
-		if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then
-		${BORG_COMMAND} check ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} -a "${CHECK_PREFIX}" ${REPOSITORY};
-		else
-			${BORG_COMMAND} check ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY};
-		fi;
-	fi;
-	# print additional info for use --repair command
-	if [ ${VERBOSE} -eq 1 ]; then
-		echo "";
-		echo "In case of needed repair: "
-		echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_PROGRESS} --repair ${OPT_GLOB} ${REPOSITORY}";
-		echo "Before running repair, a copy from the backup should be made because repair might damage a backup"
-	fi;
 	. "${DIR}/borg.backup.functions.close.sh";
 	exit;
 fi;
 
 # DELETE ONE TIME TAG
 if [ ! -z "${DELETE_ONE_TIME_TAG}" ]; then
-	echo "--- [DELETE: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "DELETE" "$(date +'%F %T')" "${MODULE}";
 	# if a "*" is inside we don't do ONE archive, but globbing via -a option
 	DELETE_ARCHIVE=""
 	OPT_GLOB="";
@@ -404,10 +434,10 @@ if [ ! -z "${DELETE_ONE_TIME_TAG}" ]; then
 	# not that compact only works on borg 1.2
 	if [ $(version $BORG_VERSION) -ge $(version "1.2.0") ]; then
 		if [ ${DRYRUN} -eq 0 ]; then
-			${BORG_COMMAND} compact ${REPOSITORY};
+			${BORG_COMMAND} compact ${OPT_REMOTE} ${REPOSITORY};
 		fi;
 		if [ ${DEBUG} -eq 1 ]; then
-			echo "${BORG_COMMAND} compact ${REPOSITORY}";
+			echo "${BORG_COMMAND} compact ${OPT_REMOTE} ${REPOSITORY}";
 		fi;
 	fi;
 	. "${DIR}/borg.backup.functions.close.sh";

borg.backup.gitea.settings-default

@@ -2,10 +2,8 @@
 
 # rename to borg.backup.gitea.settings to use
 
-# override settings in borg.backup.settings with SUB_ prefix
-# valid for BACKUP_FILE, BACKUP_SET, COMPRESSION*, KEEP_*
-
 GIT_USER="";
-GITEA_TMP="";
+GITEA_WORKING_DIR="";
+GITEA_TEMP_DIR="";
 GITEA_BIN="";
 GITEA_CONFIG="";

borg.backup.gitea.sh

@@ -3,15 +3,19 @@
 # Backup gitea database, all git folders and gitea settings
 
 MODULE="gitea"
-MODULE_VERSION="1.1.2";
+MODULE_VERSION="1.2.0";
 
 DIR="${BASH_SOURCE%/*}"
 if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 # init system
 . "${DIR}/borg.backup.functions.init.sh";
 
-# init verify file
+# init verify, compact and check file
-BACKUP_INIT_VERIFY="borg.backup.gitea.init";
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
 
 # verify valid data
 . "${DIR}/borg.backup.functions.verify.sh";
@@ -25,9 +29,13 @@ if [ -z "${GIT_USER}" ]; then
 	GIT_USER="git";
 fi;
 # set GITEA_* if not set
-if [ -z "${GITEA_TMP}" ]; then
+if [ -z "${GITEA_WORKING_DIR}" ]; then
 	# run gitea backup (user mktemp?)
-	GITEA_TMP="/tmp/gitea/";
+	GITEA_WORKING_DIR="/var/tmp/gitea/";
+fi;
+# general temp folder for temporary data storage, this is not working output folder
+if [ -z "${GITEA_TEMP_DIR}"]; then
+	GITEA_TEMP_DIR="/var/tmp";
 fi;
 if [ -z "${GITEA_BIN}" ]; then
 	GITEA_BIN="/usr/local/bin/gitea";
@@ -35,6 +43,10 @@ fi;
 if [ -z "${GITEA_CONFIG}" ]; then
 	GITEA_CONFIG="/etc/gitea/app.ini"
 fi;
+# This one is not advertised in the config file as it is not recommended to change
+if [ -z "${GITEA_EXPORT_TYPE}" ]; then
+	GITEA_EXPORT_TYPE="zip";
+fi;
 if [ ! -f "${GITEA_BIN}" ]; then
 	echo "[! $(date +'%F %T')] Cannot find gitea binary";
 	. "${DIR}/borg.backup.functions.close.sh" 1;
@@ -45,6 +57,16 @@ if [ ! -f "${GITEA_CONFIG}" ]; then
 	. "${DIR}/borg.backup.functions.close.sh" 1;
 	exit 1;
 fi;
+# some basic checks with abort
+if [ ! -d "${GITEA_TEMP_DIR}" ]; then
+	echo "Temp directory does not exist: ${GITEA_TEMP_DIR}";
+	exit;
+fi;
+# we should check GITEA_EXPORT_TYPE too at some point for an allow list
+# At the moment warn if not zip
+if [ "${GITEA_EXPORT_TYPE}" != "zip" ]; then
+	echo "[!!!!] The gitea export type has been changed from 'zip' to '${GITEA_EXPORT_TYPE}'. This can either break or make exports take very ling";
+fi;
 # Filename
 FILENAME="gitea.backup.zip";
 # backup set and prefix
@@ -54,9 +76,9 @@ BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${BACKUP_SET}";
 # borg call
 BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/");
 BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/");
-echo "--- [git data and database: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+printf "${PRINTF_SUB_BLOCK}" "BACKUP: git data and database" "$(date +'%F %T')" "${MODULE}";
 if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
-	echo "sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_TMP} -L -f - | ${BORG_CALL}";
+	echo "sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_WORKING_DIR} -t ${GITEA_TEMP_DIR} --type ${GITEA_EXPORT_TYPE} -L -f - | ${BORG_CALL}";
 	if [ -z "${ONE_TIME_TAG}" ]; then
 		echo "${BORG_PRUNE}";
 	fi;
@@ -66,21 +88,23 @@ if [ ${DRYRUN} -eq 0 ]; then
 		# below was an old workaround
 		#export USER="${LOGNAME}" # workaround for broken gitea EUID check
 		# make sure temp folder is there and is set as git. user
-		if [ ! -d "${GITEA_TMP}" ]; then
+		if [ ! -d "${GITEA_WORKING_DIR}" ]; then
-			mkdir -p "${GITEA_TMP}";
+			mkdir -p "${GITEA_WORKING_DIR}";
 		fi;
-		chown -R ${GIT_USER}. "${GITEA_TMP}";
+		chown -R ${GIT_USER}. "${GITEA_WORKING_DIR}";
 		# this needs to be run in a folder that can be stat by git user
-		cd "${GITEA_TMP}";
+		cd "${GITEA_WORKING_DIR}";
-		sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_TMP} -L -f - | ${BORG_CALL};
+		sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_WORKING_DIR} -t ${GITEA_TEMP_DIR} --type ${GITEA_EXPORT_TYPE} -L -f - | ${BORG_CALL};
-	) | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g' # remove all ESC strings
+	) 2>&1 | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g' # remove all ESC strings
 fi;
 if [ -z "${ONE_TIME_TAG}" ]; then
-	echo "--- [PRUNE : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}";
 	echo "Prune repository with keep${KEEP_INFO:1}";
 	${BORG_PRUNE};
 	# if this is borg version >1.2 we need to run compact after prune
-	. "${DIR}/borg.backup.functions.compact.sh";
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
 fi;
 
 . "${DIR}/borg.backup.functions.close.sh";

borg.backup.mysql.sh

@@ -7,7 +7,7 @@
 
 # set last edit date + time
 MODULE="mysql"
-MODULE_VERSION="1.1.1";
+MODULE_VERSION="1.1.3";
 
 DIR="${BASH_SOURCE%/*}"
 if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
@@ -15,11 +15,15 @@ if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 . "${DIR}/borg.backup.functions.init.sh";
 
 # include and exclude file
-INCLUDE_FILE="borg.backup.mysql.include";
+INCLUDE_FILE="borg.backup.${MODULE}.include";
-EXCLUDE_FILE="borg.backup.mysql.exclude";
+EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
-SCHEMA_ONLY_FILE="borg.backup.mysql.schema-only";
+SCHEMA_ONLY_FILE="borg.backup.${MODULE}.schema-only";
-# init verify file
+# init verify, compact and check file
-BACKUP_INIT_VERIFY="borg.backup.mysql.init";
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
 
 # verify valid data
 . "${DIR}/borg.backup.functions.verify.sh";
@@ -89,7 +93,8 @@ if [ ! -z "${DATABASE_FULL_DUMP}" ]; then
 		SCHEMA_ONLY='--no-data';
 		schema_flag='schema';
 	fi;
-	echo "---[BACKUP: all databases: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	LOCAL_START=$(date +'%s');
+	printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "all databases" "$(date +'%F %T')" "${MODULE}";
 	# We only do a full backup and not per table backup here
 	# Filename
 	FILENAME="all-${schema_flag}-${DB_TYPE}_${DB_VERSION}_${DB_HOST}_${DB_PORT}.sql"
@@ -116,15 +121,18 @@ if [ ! -z "${DATABASE_FULL_DUMP}" ]; then
 		fi;
 	fi;
 	if [ -z "${ONE_TIME_TAG}" ]; then
-		echo "--- [PRUNE : all databases: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+		printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "all databases" "$(date +'%F %T')" "${MODULE}";
 		echo "Prune repository with keep${KEEP_INFO:1}";
 		${BORG_PRUNE};
 	fi;
+	DURATION=$[ $(date +'%s')-$LOCAL_START ];
+	printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "databases" "${MODULE}" "$(convert_time ${DURATION})";
 else
 	${MYSQL_CMD} ${MYSQL_DB_CONFIG_PARAM} -B -N -e "show databases" |
 	while read db; do
-		echo "========[DB: ${db}]========================[${MODULE}]====================================>";
+		LOCAL_START=$(date +'%s');
-		echo "--- [BACKUP: ${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+		printf "${PRINTF_DB_SUB_BLOCK}" "DB" "${db}" "${MODULE}";
+		printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
 		# exclude checks
 		include=0;
 		if [ -s "${BASE_FOLDER}${INCLUDE_FILE}" ]; then
@@ -202,19 +210,23 @@ else
 				fi;
 			fi;
 			if [ -z "${ONE_TIME_TAG}" ]; then
-				echo "--- [PRUNE : ${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+				printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "${db}" "$(date +'%F %T')" "${MODULE}";
 				echo "Prune repository prefixed ${BACKUP_SET_PREFIX} with keep${KEEP_INFO:1}";
 				${BORG_PRUNE};
 			fi;
 		else
 			echo "- [E] ${db}";
 		fi;
+		DURATION=$[ $(date +'%s')-$LOCAL_START ];
+		printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "${db}" "${MODULE}" "$(convert_time ${DURATION})";
 	done;
 fi;
 # run compact at the end if not a dry run
 if [ -z "${ONE_TIME_TAG}" ]; then
 	# if this is borg version >1.2 we need to run compact after prune
-	. "${DIR}/borg.backup.functions.compact.sh";
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
 fi;
 
 . "${DIR}/borg.backup.functions.close.sh";

borg.backup.pgsql.sh

@@ -7,7 +7,7 @@
 
 # set last edit date + time
 MODULE="pgsql"
-MODULE_VERSION="1.2.1";
+MODULE_VERSION="1.2.4";
 
 
 DIR="${BASH_SOURCE%/*}"
@@ -16,12 +16,16 @@ if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 . "${DIR}/borg.backup.functions.init.sh";
 
 # include and exclude file
-INCLUDE_FILE="borg.backup.pgsql.include";
+INCLUDE_FILE="borg.backup.${MODULE}.include";
-EXCLUDE_FILE="borg.backup.pgsql.exclude";
+EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
-SCHEMA_ONLY_FILE="borg.backup.pgsql.schema-only";
+SCHEMA_ONLY_FILE="borg.backup.${MODULE}.schema-only";
-DATA_ONLY_FILE="borg.backup.pgsql.data-only";
+DATA_ONLY_FILE="borg.backup.${MODULE}.data-only";
-# init verify file
+# init verify, compact and check file
-BACKUP_INIT_VERIFY="borg.backup.pgsql.init";
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
 
 # verify valid data
 . "${DIR}/borg.backup.functions.verify.sh";
@@ -98,7 +102,8 @@ if [ ! -z "${DATABASE_FULL_DUMP}" ]; then
 		SCHEMA_ONLY='-s';
 		schema_flag='schema';
 	fi;
-	echo "--- [BACKUP: all databases: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	LOCAL_START=$(date +'%s');
+	printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "all databases" "$(date +'%F %T')" "${MODULE}";
 	# Filename
 	FILENAME-"all.${DB_USER}.NONE.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql"
 	# backup set:
@@ -124,15 +129,18 @@ if [ ! -z "${DATABASE_FULL_DUMP}" ]; then
 		fi;
 	fi;
 	if [ -z "${ONE_TIME_TAG}" ]; then
-		echo "--- [PRUNE : all databases: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+		printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "all databases" "$(date +'%F %T')" "${MODULE}";
 		echo "Prune repository with keep${KEEP_INFO:1}";
 		${BORG_PRUNE};
 	fi;
+	DURATION=$[ $(date +'%s')-$LOCAL_START ];
+	printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "databases" "${MODULE}" "$(convert_time ${DURATION})";
 else
 	# dump globals first
 	db="pg_globals";
 	schema_flag="data";
-	echo "--- [BACKUP: ${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	LOCAL_START=$(date +'%s');
+	printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
 	# Filename
 	FILENAME="${db}.${DB_USER}.NONE.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql"
 	# backup set:
@@ -158,19 +166,22 @@ else
 		fi;
 	fi;
 	if [ -z "${ONE_TIME_TAG}" ]; then
-		echo "--- [PRUNE : ${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+		printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "${db}" "$(date +'%F %T')" "${MODULE}";
 		echo "Prune repository with keep${KEEP_INFO:1}";
 		${BORG_PRUNE};
 	fi;
+	DURATION=$[ $(date +'%s')-$LOCAL_START ];
+	printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "BACKUP" "${db}" "${MODULE}" "$(convert_time ${DURATION})";
 
 	# get list of tables
 	for owner_db in $(${PG_PSQL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} -d template1 -t -A -F "," -X -q -c "SELECT pg_catalog.pg_get_userbyid(datdba) AS owner, datname, pg_catalog.pg_encoding_to_char(encoding) AS encoding FROM pg_catalog.pg_database WHERE datname "\!"~ 'template(0|1)' ORDER BY datname;"); do
+		LOCAL_START=$(date +'%s');
 		# get the user who owns the DB too
 		owner=$(echo ${owner_db} | cut -d "," -f 1);
 		db=$(echo ${owner_db} | cut -d "," -f 2);
 		encoding=$(echo ${owner_db} | cut -d "," -f 3);
-		echo "========[DB: ${db}]========================[${MODULE}]====================================>";
+		printf "${PRINTF_DB_SUB_BLOCK}" "DB" "${db}" "${MODULE}";
-		echo "--- [BACKUP: ${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+		printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
 		include=0;
 		if [ -s "${BASE_FOLDER}${INCLUDE_FILE}" ]; then
 			while read incl_db; do
@@ -254,19 +265,23 @@ else
 				fi;
 			fi;
 			if [ -z "${ONE_TIME_TAG}" ]; then
-				echo "--- [PRUNE : ${db}: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+				printf "${PRINTF_SUBEXT_BLOCK}" "PRUNE" "${db}" "$(date +'%F %T')" "${MODULE}";
 				echo "Prune repository prefixed ${BACKUP_SET_PREFIX} with keep${KEEP_INFO:1}";
 				${BORG_PRUNE};
 			fi;
 		else
 			echo "- [E] ${db}";
 		fi;
+		DURATION=$[ $(date +'%s')-$LOCAL_START ];
+		printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "${db}" "${MODULE}" "$(convert_time ${DURATION})";
 	done;
 fi;
 # run compact at the end if not a dry run
 if [ -z "${ONE_TIME_TAG}" ]; then
 	# if this is borg version >1.2 we need to run compact after prune
-	. "${DIR}/borg.backup.functions.compact.sh";
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
 fi;
 
 . "${DIR}/borg.backup.functions.close.sh";

borg.backup.settings-default

@@ -27,8 +27,19 @@ COMPRESSION_LEVEL="";
 # Blank passwords allowed for only key (if used, use keyfile)
 # See: http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically
 ENCRYPTION="";
-# force repository check, default is off, set to true for check
+# force repository verify, default is off, set to true for verify on every run
-FORCE_CHECK="";
+FORCE_VERIFY="";
+# compact interval, only if using borg 1.2 or higher
+# after how many days to run compress on data
+# default is 1 for run after each prune
+# set to 0 or negative to turn off
+# or any other value for every n days
+COMPACT_INTERVAL="";
+# check interval, if 0 or negative number, no check will ever run
+# if empty fall back to default set
+# if set to 1 then every time the script runs
+# any other value it means ever n days, eg 90 would be every 90 days
+CHECK_INTERVAL="";
 # default is %Y-%m-%d
 # todays date, if more than one per day add -%H%M for hour/minute
 # it can also be "{hostname}-{user}-{now:%Y-%m-%dT%H:%M:%S.%f}"

borg.backup.zabbix.sh

@@ -3,15 +3,19 @@
 # Backup zabbix config and settings only
 
 MODULE="zabbix"
-MODULE_VERSION="1.1.2";
+MODULE_VERSION="1.1.3";
 
 DIR="${BASH_SOURCE%/*}"
 if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
 # init system
 . "${DIR}/borg.backup.functions.init.sh";
 
-# init verify file
+# init verify, compact and check file
-BACKUP_INIT_VERIFY="borg.backup.zabbix.init";
+BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
+BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
+BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
+# lock file
+BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
 
 # verify valid data
 . "${DIR}/borg.backup.functions.verify.sh";
@@ -64,7 +68,7 @@ if [ -z "${BACKUP_SET_PREFIX}" ]; then
 	BORG_PRUNE=$(echo "${BORG_PRUNE}" | sed -e 's/-P //');
 fi;
 
-echo "--- [BACKUP: zabbix settings: $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+printf "${PRINTF_SUB_BLOCK}" "BACKUP: zabbix settings" "$(date +'%F %T')" "${MODULE}";
 if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
 	echo "${ZABBIX_DUMP_BIN} -t ${ZABBIX_DATABASE} ${OPT_ZABBIX_UNKNOWN_TABLES} ${OPT_ZABBIX_DUMP} ${OPT_ZABBIX_CONFIG} -o - | ${BORG_CALL}"
 	if [ -z "${ONE_TIME_TAG}" ]; then
@@ -75,10 +79,12 @@ if [ ${DRYRUN} -eq 0 ]; then
 	${ZABBIX_DUMP_BIN} -t ${ZABBIX_DATABASE} ${OPT_ZABBIX_UNKNOWN_TABLES} ${OPT_ZABBIX_DUMP} ${OPT_ZABBIX_CONFIG} -o - | ${BORG_CALL};
 fi;
 if [ -z "${ONE_TIME_TAG}" ]; then
-	echo "--- [PRUNE : $(date +'%F %T')] --[${MODULE}]------------------------------------>";
+	printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}";
 	${BORG_PRUNE};
 	# if this is borg version >1.2 we need to run compact after prune
-	. "${DIR}/borg.backup.functions.compact.sh";
+	. "${DIR}/borg.backup.functions.compact.sh" "auto";
+	# check in auto mode
+	. "${DIR}/borg.backup.functions.check.sh" "auto";
 fi;
 
 . "${DIR}/borg.backup.functions.close.sh";

borg.mount.sh

@@ -115,4 +115,4 @@ else
 	borg umount "${MOUNT_PATH}";
 fi;
 
-## END
+# __END__