Compare commits

...

48 Commits

Author SHA1 Message Date
Clemens Schwaighofer
924014688e Update gitignore file with backup lock files 2025-12-04 09:57:54 +09:00
Clemens Schwaighofer
f9a58171cc Bump version to v4.8.0 2025-12-04 09:35:08 +09:00
Clemens Schwaighofer
1b52af909a Update PostgreSQL module to run as normal user with sudo for PostgreSQL
also clean up all PostgreSQL command calls as array type calls
clean up binary finder for postgresql installations
and add port/host override options
2025-12-04 09:28:57 +09:00
Clemens Schwaighofer
32a75626bb Version number fixes 2025-12-03 15:49:45 +09:00
Clemens Schwaighofer
12ff648504 PostgreSQL backup hat wrong table selection 2025-12-03 15:31:55 +09:00
Clemens Schwaighofer
69be76e34c Default settings file update for BACKUP_FILE 2025-12-03 14:48:54 +09:00
Clemens Schwaighofer
4a95049d4f Default config file info update 2025-12-03 14:12:24 +09:00
Clemens Schwaighofer
d0e96a82e0 Clean up PostgreSQL call, clean up varify error block, ERR trap fixes
ERR trap is now returning last error so $? can bet used
Verify catches $? error now for INIT repository check
2025-12-03 14:00:49 +09:00
Clemens Schwaighofer
e5c5df6013 Fix error trap with return error to STD ERR 2025-12-03 13:08:35 +09:00
Clemens Schwaighofer
6023fac636 Fix trap clanup that also trapped ERR 2025-12-03 13:05:11 +09:00
Clemens Schwaighofer
9e0db8bae9 Fix the error trap with exit to stop calling it again with itself 2025-12-03 13:03:46 +09:00
Clemens Schwaighofer
8a8de773df Change borg repository verify error message collection 2025-12-03 12:47:49 +09:00
Clemens Schwaighofer
f41dd1b723 Change verify command error check 2025-12-03 12:45:56 +09:00
Clemens Schwaighofer
c5d88a64e1 Fix last command check for verify 2025-12-03 12:43:52 +09:00
Clemens Schwaighofer
943d1c551e Stop trapping errors und unset enviromant vars
On error just print error, but do not reset env vars, if this is done on verify error the init afterwards will store all settings in the wrong path
2025-12-03 12:38:01 +09:00
Clemens Schwaighofer
45e1e29d22 Update readme file with info about encryption 2025-12-02 10:01:37 +09:00
Clemens Schwaighofer
1f8d38720c On info if verbose is set, print out the key too 2025-12-01 17:05:45 +09:00
Clemens Schwaighofer
7e4dc8d500 Clean up bash shell code and add proper ENCRYPTION support
- check for invalid encryption setting
- switch from none to keyfile type (without password)
- error for no password for repokey or authentication type encryption
- update init and print out the key file data
2025-12-01 16:53:50 +09:00
Clemens Schwaighofer
297c745df7 Borg MySQL backup: fix version detection for MariaDB variant
This one is from Debian 13
2025-09-18 09:58:51 +09:00
Clemens Schwaighofer
a2bf6b3331 Readme update for zabbix backup settings change 2025-08-21 10:40:56 +09:00
Clemens Schwaighofer
07896af1ae Zabbix port override settings fix
use OPT var prefix for options
add missing init in functions init
2025-08-20 22:37:37 +09:00
Clemens Schwaighofer
8de971b922 Fix Zabbix db port override settings 2025-08-20 22:16:49 +09:00
Clemens Schwaighofer
e7699015c5 Allow override of zabbix db port 2025-08-20 22:14:19 +09:00
Clemens Schwaighofer
28dd6cb911 Merge branch 'master' into development 2024-09-04 10:38:58 +09:00
Clemens Schwaighofer
ab47692848 borg backup default settings file sample update 2024-09-04 10:37:36 +09:00
Clemens Schwaighofer
4267cb5b32 Bug Fix in gitea backup for temp dir check 2024-09-04 10:37:16 +09:00
Clemens Schwaighofer
f24daba195 Add shellcheckrc 2024-09-03 12:40:39 +09:00
Clemens Schwaighofer
98890b87f5 gitea module fix chown call
do not use ".", use ":" for username and group separator
2024-06-11 10:33:47 +09:00
Clemens Schwaighofer
a5bc90bb89 Text fix in PostgreSQL backup module 2024-06-10 09:52:06 +09:00
Clemens Schwaighofer
5af6a84d5f Mysql/PostgreSQL module version string update 2024-06-07 15:08:49 +09:00
Clemens Schwaighofer
14d71f3053 Naming update for full update info text 2024-06-07 15:08:20 +09:00
Clemens Schwaighofer
24d4fb694f Typo in postgresql module for full dump filename 2024-06-07 14:25:00 +09:00
Clemens Schwaighofer
f9fbbcc6de Version fix for master script 2024-06-07 09:57:21 +09:00
Clemens Schwaighofer
64e876a9a4 Fix run time output template string 2024-06-07 09:56:51 +09:00
Clemens Schwaighofer
694e54ca9c Missed out duration calculation in postgresql run time output 2024-06-07 09:16:52 +09:00
Clemens Schwaighofer
09bd6015ac Version update for modules gitea, mysql, postgresql 2024-06-07 09:12:04 +09:00
Clemens Schwaighofer
5b8b0afec2 Fix not initialized new gitea variable names 2024-06-07 09:09:02 +09:00
Clemens Schwaighofer
0cc7e574fe Borg backup update for gitea dump, mysql/postgresql info block
Gitea dump has now settings for
- temp folder (GITEA_TEMP_DIR)
- new name for working dir (GITEA_WORKING_DIR) from old (GITEA_TMP)

MySQL and PostgreSQL exports have a run time output per db dump set and not only at the end of the full run
2024-06-06 10:30:48 +09:00
Clemens Schwaighofer
beffbee20c Add missing OPT_REMOTE to commands
check, compact commands had OPT_REMOTE missing
2023-05-10 16:30:23 +09:00
Clemens Schwaighofer
72f3f86eb6 Test for different command lookup check 2022-11-22 09:36:30 +09:00
Clemens Schwaighofer
c0f9634442 add *.compact to git ignore file 2022-11-14 20:27:42 +09:00
Clemens Schwaighofer
0e379d6ce0 Bump borg file module version number 2022-08-30 10:00:05 +09:00
Clemens Schwaighofer
bba8be6a42 Exclude File check is if file is >0 and fix wrong check for tmp file
Exclude file was only checked if it exists, even if it was empty it
would start processing.
In the exclude check there was also a wrong check for the tmp exclude
file, it actually checked if the main exclude file was >0 instead of the
tmp file.
2022-08-30 09:56:20 +09:00
Clemens Schwaighofer
ce3cdc7627 Readme update to fix Marktdown lint issues 2022-08-08 14:51:26 +09:00
Clemens Schwaighofer
67c0b00268 Initialize LAST COMPACT/CHECK date variable 2022-06-07 19:57:36 +09:00
Clemens Schwaighofer
abbcfa883f Add compact auto run in intervals like check archive
Same settings like check archive in days. Auto set to 1 to run after
each prune. Can be controlled per module.
2022-06-06 14:58:36 +09:00
Clemens Schwaighofer
9739436589 Redirect STDERR output from borg backup to STDIN for ANSI code cleanup
gitea dump prints out info/status messages to STDERR. The whole
subprocess then returns this on STDERR and so is not parsed through the
sed regex command.
Redirect STDERR to STDIN
2022-05-31 09:54:56 +09:00
Clemens Schwaighofer
bf0364a0e7 Move lock file clean up to before any exit info is printed
Just try to clean lock file always on exit
2022-04-25 08:57:01 +09:00
20 changed files with 703 additions and 345 deletions

3
.gitignore vendored
View File

@@ -1,8 +1,11 @@
.cache/ .cache/
.config/ .config/
# various borg files that need to be ignored
borg.backup.settings borg.backup.settings
borg.backup.*.settings borg.backup.*.settings
borg.backup.*.include borg.backup.*.include
borg.backup.*.exclude borg.backup.*.exclude
borg.backup.*.schema-only borg.backup.*.schema-only
borg.backup.*.init borg.backup.*.init
borg.backup.*.compact
borg.backup.*.lock

2
.shellcheckrc Normal file
View File

@@ -0,0 +1,2 @@
shell=bash
external-sources=true

155
Readme.md
View File

@@ -13,9 +13,11 @@ Version 4.0 introduces default borg repository name with `-file` for the `file`
*Example:* *Example:*
Old backup name Old backup name
```sh ```sh
BACKUP_FILE="some-backup-data.borg" BACKUP_FILE="some-backup-data.borg"
``` ```
Then the file need to be renamed the following way: Then the file need to be renamed the following way:
`mv some-backup-data.borg some-backup-data-file.borg` `mv some-backup-data.borg some-backup-data-file.borg`
@@ -42,57 +44,79 @@ No settings files will be overwritten
## Possible command line options ## Possible command line options
### `-c <config folder>` ### `-c <config folder>`
if this is not given, /usr/local/scripts/borg/ is used if this is not given, /usr/local/scripts/borg/ is used
### `-L <log folder>` ### `-L <log folder>`
override config set and default log folder override config set and default log folder
### `-T <tag>` ### `-T <tag>`
create one time stand alone backup prefixed with tag name create one time stand alone backup prefixed with tag name
### `-D <tag backup set>` ### `-D <tag backup set>`
remove a tagged backup set, full name must be given remove a tagged backup set, full name must be given
### `-b <borg executable>` ### `-b <borg executable>`
override the default borg executable found in path override the default borg executable found in path
### `-P` ### `-P`
print list of archives created print list of archives created
### `-V` ### `-V`
verify if repository exists, if not abort verify if repository exists, if not abort
### `-e` ### `-e`
exit after running verify `-V` exit after running verify `-V`
### `-I` ### `-I`
init repository (must be run first) init repository (must be run first)
### `-Z`
run `borg compact` over given repository
### `-C` ### `-C`
run `borg check` over given repository run `borg check` over given repository
#### `-y` #### `-y`
Add `--verify-data` to `borg check`. Only works with `-C` Add `--verify-data` to `borg check`. Only works with `-C`
#### `-p <prefix|glob>` #### `-p <prefix|glob>`
Only `borg check` data that has given prefix or glob (with *). Only works with `-C` Only `borg check` data that has given prefix or glob (with *). Only works with `-C`
### `-i` ### `-i`
print out only info print out only info
### `-l` ### `-l`
list files during backup list files during backup
### `-v` ### `-v`
be verbose be verbose
### `-d` ### `-d`
debug output all commands debug output all commands
### `-n` ### `-n`
only do dry run only do dry run
### `-h` ### `-h`
this help page this help page
## Basic Settings ## Basic Settings
@@ -106,7 +130,6 @@ LOG_FOLDER: default `/var/log/borg.backup/`
TARGET_FOLDER: must be set to a path where the backups can be written TARGET_FOLDER: must be set to a path where the backups can be written
BACKUP_FILE: the folder inside the TARGET_FOLDER that is the target for borg. Must end with `.borg` BACKUP_FILE: the folder inside the TARGET_FOLDER that is the target for borg. Must end with `.borg`
Note: BACKUP_FILE is the base name. For all except file (current) a module suffix will be added: Note: BACKUP_FILE is the base name. For all except file (current) a module suffix will be added:
eg: eg:
@@ -115,37 +138,41 @@ eg:
If `FILE_REPOSITORY_COMPATIBLE` is set to `false` in the borg.backup.file.settings then the file borg name will have `-file` added too. Currently this is not added to stay compatible with older scripts If `FILE_REPOSITORY_COMPATIBLE` is set to `false` in the borg.backup.file.settings then the file borg name will have `-file` added too. Currently this is not added to stay compatible with older scripts
All below have default values if not set in the main settings file All below have default values if not set in the main settings file
* COMPRESSION: zstd
* COMPRESSION_LEVEL: 3 * COMPRESSION: zstd
* ENCRYPTION: none * COMPRESSION_LEVEL: 3
* FORCE_VERIFY: false * ENCRYPTION: keyfile (with no password)
* CHECK_INTERVAL: none * FORCE_VERIFY: false
* KEEP_LAST: 0 * COMPACT_INTERVAL: 1
* KEEP_HOURS: 0 * CHECK_INTERVAL: none
* KEEP_DAYS: 7 * KEEP_LAST: 0
* KEEP_WEEKS: 4 * KEEP_HOURS: 0
* KEEP_MONTHS: 6 * KEEP_DAYS: 7
* KEEP_YEARS: 1 * KEEP_WEEKS: 4
* KEEP_MONTHS: 6
* KEEP_YEARS: 1
All module settings files can have the following prefixed with `SUB_` to override master settings: All module settings files can have the following prefixed with `SUB_` to override master settings:
* SUB_BACKUP_FILE
* SUB_COMPRESSION * SUB_BACKUP_FILE
* SUB_COMPRESSION_LEVEL * SUB_COMPRESSION
* SUB_CHECK_INTERVAL * SUB_COMPRESSION_LEVEL
* SUB_BACKUP_SET * SUB_COMPACT_INTERVAL
* SUB_KEEP_LAST * SUB_CHECK_INTERVAL
* SUB_KEEP_HOURS * SUB_BACKUP_SET
* SUB_KEEP_DAYS * SUB_KEEP_LAST
* SUB_KEEP_WEEKS * SUB_KEEP_HOURS
* SUB_KEEP_MONTHS * SUB_KEEP_DAYS
* SUB_KEEP_YEARS * SUB_KEEP_WEEKS
* SUB_KEEP_WITHIN * SUB_KEEP_MONTHS
* SUB_KEEP_YEARS
* SUB_KEEP_WITHIN
## Setup backup via SSH to remote host on `borg.backup.settings` ## Setup backup via SSH to remote host on `borg.backup.settings`
For this the following settings are from interest For this the following settings are from interest
``` ```sh
TARGET_USER=""; TARGET_USER="";
TARGET_HOST=""; TARGET_HOST="";
TARGET_PORT=""; TARGET_PORT="";
@@ -175,10 +202,9 @@ to `true`
### Config variables ### Config variables
### Control files ### Control files
``` ```sh
backup.borg.file.include backup.borg.file.include
backup.borg.file.exclude backup.borg.file.exclude
``` ```
@@ -192,16 +218,16 @@ backup.borg.file.exclude
This script must be run as the postgres user, normaly `postgres`. This script must be run as the postgres user, normaly `postgres`.
The postgres user must be added to the backup group for this, so that the basic init file can be created in the borg base folder. The postgres user must be added to the backup group for this, so that the basic init file can be created in the borg base folder.
### Config variables ### PostgreSQL Config variables
Variable | Default | Description | Variable | Default | Description |
| - | - | - | | - | - | - |
DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema | DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema |
DATABASE_USER | | overide username to connect to database | DATABASE_USER | | overide username to connect to database |
### Control files ### PostgreSQL Control files
``` ```sh
backup.borg.pgsql.include backup.borg.pgsql.include
backup.borg.pgsql.exclude backup.borg.pgsql.exclude
backup.borg.pgsql.schema-only backup.borg.pgsql.schema-only
@@ -211,16 +237,16 @@ backup.borg.pgsql.schema-only
If non root ident authentication run is used, be sure that the `mysql` user is in the backup group. If non root ident authentication run is used, be sure that the `mysql` user is in the backup group.
### Config variables ### MySQL Config variables
Variable | Default | Description | Variable | Default | Description |
| - | - | - | | - | - | - |
DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema | DATABASE_FULL_DUMP | | if empty, dump per databse, if set dump all in one file, if set to schema dump only schema |
MYSQL_DB_CONFIG | | override file for connection. In modern mariaDB installations it is rcommended to run the script as root or mysql user and use the ident authentication instead. | MYSQL_DB_CONFIG | | override file for connection. In modern mariaDB installations it is rcommended to run the script as root or mysql user and use the ident authentication instead. |
### Control files ### MySQLControl files
``` ```sh
backup.borg.mysql.include backup.borg.mysql.include
backup.borg.mysql.exclude backup.borg.mysql.exclude
backup.borg.mysql.schema-only backup.borg.mysql.schema-only
@@ -231,31 +257,52 @@ backup.borg.mysql.schema-only
Note that the backup needs the GIT_USER set that runs gitea. Note that the backup needs the GIT_USER set that runs gitea.
This user is neede to create the temporary dump folder and access for the git files and database. This user is neede to create the temporary dump folder and access for the git files and database.
### Config Variables ### gitea Config Variables
Variable | Default | Description | Variable | Default | Description |
| - | - | - | | - | - | - |
GIT_USER | git | The user that runs gitea | | GIT_USER | git | The user that runs gitea |
GITEA_TMP | /tmp/gitea/ | Where the temporary dump files from the backup are stored, as user git | | GITEA_WORKING_DIR | /var/tmp/gitea/ | Where the temporary dump files from the backup are stored, as user git |
GITEA_BIN | /usr/local/bin/gitea | Where the gitea binary is located | | GITEA_TEMP_DIR | /var/tmp/ | General temporary folder |
GITEA_CONFIG | /etc/gitea/app.ini | The configuration file for gitea | | GITEA_BIN | /usr/local/bin/gitea | Where the gitea binary is located |
| GITEA_CONFIG | /etc/gitea/app.ini | The configuration file for gitea |
### gitea Control files
### Control files
There are no control files for gitea backup There are no control files for gitea backup
## zabbix config backup settings ## zabbix config backup settings
### Config Variables The `zabbix-dump` dump script must be installed from: <https://github.com/gullevek/zabbix-backup>
Variable | Default | Description ### zabbix Config Variables
| Variable | Default | Description |
| - | - | - | | - | - | - |
ZABBIX_DUMP | /usr/local/bin/zabbix-dump | | ZABBIX_DUMP | /usr/local/bin/zabbix-dump | |
ZABBIX_DATABASE | '' | Must be set as either psql or mysql | ZABBIX_DATABASE | '' | Must be set as either psql or mysql |
ZABBIX_CONFIG | '' | if not set uses default location | ZABBIX_CONFIG | '' | if not set uses default location |
ZABBIX_UNKNOWN_TABLES | '' | if set, changed to -f (force) | ZABBIX_UNKNOWN_TABLES | '' | if set, changed to -f (force) |
| ZABBIX_DB_PORT | '' | if set, sets -P parameter on zabbix backup script to override auto get from zabbix server config |
### Control files ### zabbix Control files
There are no control files for zabbix settings backup There are no control files for zabbix settings backup
## File connection
Running any of the commands below
* borg.backup.file.sh
* borg.backup.gitea.sh
* borg.backup.mysql.sh
* borg.backup.pgsql.sh
* borg.backup.zabbix.sh
1) Run `borg.backup.functions.init.sh` (always)
2) Run `borg.backup.functions.verify.sh` (always)
3) (other code in "file" module)
4) Run `borg.backup.functions.info.sh` (always)
5) Run `borg.backup.functions.compact.sh` (not if one time tag)
6) Run `borg.backup.functions.check.sh` (not if one time tag)
7) Run `borg.backup.functions.close.sh` (always, can be called on error too)

View File

@@ -66,7 +66,7 @@ if [ ! -f "${BASE_FOLDER}${SETTINGS_FILE}" ]; then
fi; fi;
. "${BASE_FOLDER}${SETTINGS_FILE}"; . "${BASE_FOLDER}${SETTINGS_FILE}";
if [ ! -z "${TARGET_BORG_PATH}" ]; then if [ -n "${TARGET_BORG_PATH}" ]; then
OPT_REMOTE="--remote-path="$(printf "%q" "${TARGET_BORG_PATH}"); OPT_REMOTE="--remote-path="$(printf "%q" "${TARGET_BORG_PATH}");
fi; fi;
export BORG_BASE_DIR="${BASE_FOLDER}"; export BORG_BASE_DIR="${BASE_FOLDER}";
@@ -82,16 +82,16 @@ for MODULE in ${MODULE_LIST}; do
TARGET_FOLDER=${TARGET_FOLDER#/} TARGET_FOLDER=${TARGET_FOLDER#/}
# and add slash front and back and escape the path # and add slash front and back and escape the path
TARGET_FOLDER=$(printf "%q" "/${TARGET_FOLDER}/"); TARGET_FOLDER=$(printf "%q" "/${TARGET_FOLDER}/");
if [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then if [ -n "${TARGET_USER}" ] && [ -n "${TARGET_HOST}" ] && [ -n "${TARGET_PORT}" ]; then
TARGET_SERVER="ssh://${TARGET_USER}@${TARGET_HOST}:${TARGET_PORT}/"; TARGET_SERVER="ssh://${TARGET_USER}@${TARGET_HOST}:${TARGET_PORT}/";
# host/port # host/port
elif [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then elif [ -n "${TARGET_HOST}" ] && [ -n "${TARGET_PORT}" ]; then
TARGET_SERVER="ssh://${TARGET_HOST}:${TARGET_PORT}/"; TARGET_SERVER="ssh://${TARGET_HOST}:${TARGET_PORT}/";
# user/host # user/host
elif [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ]; then elif [ -n "${TARGET_USER}" ] && [ -n "${TARGET_HOST}" ]; then
TARGET_SERVER="${TARGET_USER}@${TARGET_HOST}:"; TARGET_SERVER="${TARGET_USER}@${TARGET_HOST}:";
# host # host
elif [ ! -z "${TARGET_HOST}" ]; then elif [ -n "${TARGET_HOST}" ]; then
TARGET_SERVER="${TARGET_HOST}:"; TARGET_SERVER="${TARGET_HOST}:";
fi; fi;
# we dont allow special characters, so we don't need to special escape it # we dont allow special characters, so we don't need to special escape it
@@ -112,14 +112,14 @@ for MODULE in ${MODULE_LIST}; do
if [ "${MODULE}" = "gitea" ]; then if [ "${MODULE}" = "gitea" ]; then
# if just date, add gitea, # if just date, add gitea,
# else rename # else rename
if [ ! -z "${i##gitea*}" ]; then if [ -n "${i##gitea*}" ]; then
target_name="${MODULE},${i}"; target_name="${MODULE},${i}";
else else
target_name=$(echo $i | sed -e 's/gitea-/gitea,/'); target_name=$(echo $i | sed -e 's/gitea-/gitea,/');
fi; fi;
elif [ "${MODULE}" = "zabbix" ]; then elif [ "${MODULE}" = "zabbix" ]; then
# if zabbix is missing, prefix # if zabbix is missing, prefix
if [ ! -z "${i##zabbix*}" ]; then if [ -n "${i##zabbix*}" ]; then
target_name="${MODULE},${i}"; target_name="${MODULE},${i}";
else else
target_name=$(echo $i | sed -e 's/zabbix-settings-/zabbix,settings-/'); target_name=$(echo $i | sed -e 's/zabbix-settings-/zabbix,settings-/');

View File

@@ -1,10 +1,13 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
# Plain file backup # Plain file backup
# set last edit date + time # set last edit date + time
MODULE="file"; MODULE="file";
MODULE_VERSION="1.2.1"; MODULE_VERSION="1.2.3";
DIR="${BASH_SOURCE%/*}" DIR="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
@@ -13,8 +16,9 @@ if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
# include and exclude file # include and exclude file
INCLUDE_FILE="borg.backup.${MODULE}.include"; INCLUDE_FILE="borg.backup.${MODULE}.include";
EXCLUDE_FILE="borg.backup.${MODULE}.exclude"; EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
# init verify and check file # init verify, compact and check file
BACKUP_INIT_FILE="borg.backup.${MODULE}.init"; BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
BACKUP_CHECK_FILE="borg.backup.${MODULE}.check"; BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
# lock file # lock file
BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock"; BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
@@ -34,7 +38,7 @@ FOLDERS=();
# this if for debug output with quoted folders # this if for debug output with quoted folders
FOLDERS_Q=(); FOLDERS_Q=();
# include list # include list
while read include_folder; do while read -r include_folder; do
# strip any leading spaces from that folder # strip any leading spaces from that folder
include_folder=$(echo "${include_folder}" | sed -e 's/^[ \t]*//'); include_folder=$(echo "${include_folder}" | sed -e 's/^[ \t]*//');
# check that those folders exist, warn on error, # check that those folders exist, warn on error,
@@ -44,7 +48,7 @@ while read include_folder; do
echo "# [C] Comment: '${include_folder}'"; echo "# [C] Comment: '${include_folder}'";
else else
# skip if it is empty # skip if it is empty
if [ ! -z "${include_folder}" ]; then if [ -n "${include_folder}" ]; then
# if this is a glob, do a double check that the base folder actually exists (?) # if this is a glob, do a double check that the base folder actually exists (?)
if [[ "${include_folder}" =~ $REGEX_GLOB ]]; then if [[ "${include_folder}" =~ $REGEX_GLOB ]]; then
# if this is */ then allow it # if this is */ then allow it
@@ -91,14 +95,14 @@ while read include_folder; do
fi; fi;
done<"${BASE_FOLDER}${INCLUDE_FILE}"; done<"${BASE_FOLDER}${INCLUDE_FILE}";
# exclude list # exclude list, only if file exists and is larger than zero
if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then if [ -s "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
printf "${PRINTF_SUB_BLOCK}" "EXCLUDE" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "EXCLUDE" "$(date +'%F %T')" "${MODULE}";
# check that the folders in that exclude file are actually valid, # check that the folders in that exclude file are actually valid,
# remove non valid ones and warn # remove non valid ones and warn
#TMP_EXCLUDE_FILE=$(mktemp --tmpdir ${EXCLUDE_FILE}.XXXXXXXX); # non mac #TMP_EXCLUDE_FILE=$(mktemp --tmpdir ${EXCLUDE_FILE}.XXXXXXXX); # non mac
TMP_EXCLUDE_FILE=$(mktemp "${TEMPDIR}${EXCLUDE_FILE}".XXXXXXXX); TMP_EXCLUDE_FILE=$(mktemp "${TEMPDIR}${EXCLUDE_FILE}".XXXXXXXX);
while read exclude_folder; do while read -r exclude_folder; do
# strip any leading spaces from that folder # strip any leading spaces from that folder
exclude_folder=$(echo "${exclude_folder}" | sed -e 's/^[ \t]*//'); exclude_folder=$(echo "${exclude_folder}" | sed -e 's/^[ \t]*//');
# folder or any type of file is ok # folder or any type of file is ok
@@ -107,10 +111,10 @@ if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
echo "# [C] Comment: '${exclude_folder}'"; echo "# [C] Comment: '${exclude_folder}'";
else else
# skip if it is empty # skip if it is empty
if [ ! -z "${exclude_folder}" ]; then if [ -n "${exclude_folder}" ]; then
# if it DOES NOT start with a / we assume free folder and add as is # if it DOES NOT start with a / we assume free folder and add as is
if [[ "${exclude_folder}" != /* ]]; then if [[ "${exclude_folder}" != /* ]]; then
echo "${exclude_folder}" >> ${TMP_EXCLUDE_FILE}; echo "${exclude_folder}" >> "${TMP_EXCLUDE_FILE}";
echo "+ [E] General exclude: '${exclude_folder}'"; echo "+ [E] General exclude: '${exclude_folder}'";
# if this is a glob, do a double check that the base folder actually exists (?) # if this is a glob, do a double check that the base folder actually exists (?)
elif [[ "${exclude_folder}" =~ $REGEX_GLOB ]]; then elif [[ "${exclude_folder}" =~ $REGEX_GLOB ]]; then
@@ -120,7 +124,7 @@ if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
if [ ! -d "${_exclude_folder}" ]; then if [ ! -d "${_exclude_folder}" ]; then
echo "- [E] Exclude folder with glob '${exclude_folder}' does not exist or is not accessable"; echo "- [E] Exclude folder with glob '${exclude_folder}' does not exist or is not accessable";
else else
echo "${exclude_folder}" >> ${TMP_EXCLUDE_FILE}; echo "${exclude_folder}" >> "${TMP_EXCLUDE_FILE}";
echo "+ [E] Exclude folder with glob '${exclude_folder}'"; echo "+ [E] Exclude folder with glob '${exclude_folder}'";
fi; fi;
# do a warning for a possible invalid folder # do a warning for a possible invalid folder
@@ -128,7 +132,7 @@ if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
elif [ ! -d "${exclude_folder}" ] && [ ! -e "${exclude_folder}" ]; then elif [ ! -d "${exclude_folder}" ] && [ ! -e "${exclude_folder}" ]; then
echo "- [E] Exclude folder or file '${exclude_folder}' does not exist or is not accessable"; echo "- [E] Exclude folder or file '${exclude_folder}' does not exist or is not accessable";
else else
echo "${exclude_folder}" >> ${TMP_EXCLUDE_FILE}; echo "${exclude_folder}" >> "${TMP_EXCLUDE_FILE}";
# if it is a folder, remove the last / or the symlink check will not work # if it is a folder, remove the last / or the symlink check will not work
if [ -d "${exclude_folder}" ]; then if [ -d "${exclude_folder}" ]; then
_exclude_folder=${exclude_folder%/*}; _exclude_folder=${exclude_folder%/*};
@@ -146,7 +150,7 @@ if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
fi; fi;
done<"${BASE_FOLDER}${EXCLUDE_FILE}"; done<"${BASE_FOLDER}${EXCLUDE_FILE}";
# avoid blank file add by checking if the tmp file has a size >0 # avoid blank file add by checking if the tmp file has a size >0
if [ -s "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then if [ -s "${TMP_EXCLUDE_FILE}" ]; then
OPT_EXCLUDE="--exclude-from=${TMP_EXCLUDE_FILE}"; OPT_EXCLUDE="--exclude-from=${TMP_EXCLUDE_FILE}";
fi; fi;
fi; fi;
@@ -195,7 +199,7 @@ if [ -z "${ONE_TIME_TAG}" ]; then
# for the IFS="#" to work we need to replace options spaces with exactly ONE # # for the IFS="#" to work we need to replace options spaces with exactly ONE #
$(echo "${COMMAND}" | sed -e 's/[ ][ ]*/#/g') 2>&1 || echo "[!] Borg prune aborted"; $(echo "${COMMAND}" | sed -e 's/[ ][ ]*/#/g') 2>&1 || echo "[!] Borg prune aborted";
# if this is borg version >1.2 we need to run compact after prune # if this is borg version >1.2 we need to run compact after prune
. "${DIR}/borg.backup.functions.compact.sh"; . "${DIR}/borg.backup.functions.compact.sh" "auto";
# check in auto mode # check in auto mode
. "${DIR}/borg.backup.functions.check.sh" "auto"; . "${DIR}/borg.backup.functions.check.sh" "auto";
else else

View File

@@ -1,5 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
if [ -z "${MODULE}" ]; then if [ -z "${MODULE}" ]; then
echo "Script cannot be run on its own"; echo "Script cannot be run on its own";
exit 1; exit 1;
@@ -18,13 +21,13 @@ if [ $# -ge 1 ] && [ "$1" = "auto" ]; then
# get current date timestmap # get current date timestmap
CURRENT_DATE=$(date +%s); CURRENT_DATE=$(date +%s);
# if =1 always ok # if =1 always ok
if [ ${CHECK_INTERVAL} -eq 1 ]; then if [ "${CHECK_INTERVAL}" -eq 1 ]; then
RUN_CHECK=1; RUN_CHECK=1;
# always add verify data for automatic check # always add verify data for automatic check
OPT_CHECK_VERIFY_DATA="--verify-data"; OPT_CHECK_VERIFY_DATA="--verify-data";
# set new check time here # set new check time here
echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_CHECK_FILE}"; echo "${CURRENT_DATE}" > "${BASE_FOLDER}${BACKUP_CHECK_FILE}";
elif [ ${CHECK_INTERVAL} -gt 1 ]; then elif [ "${CHECK_INTERVAL}" -gt 1 ]; then
# else load last timestamp and check if today - last time stamp > days # else load last timestamp and check if today - last time stamp > days
if [ -z "${LAST_CHECK_DATE}" ]; then if [ -z "${LAST_CHECK_DATE}" ]; then
LAST_CHECK_DATE=$(cat "${BASE_FOLDER}${BACKUP_CHECK_FILE}" 2>/dev/null | sed -e 's/ //g'); LAST_CHECK_DATE=$(cat "${BASE_FOLDER}${BACKUP_CHECK_FILE}" 2>/dev/null | sed -e 's/ //g');
@@ -34,19 +37,19 @@ if [ $# -ge 1 ] && [ "$1" = "auto" ]; then
LAST_CHECK_DATE=0; LAST_CHECK_DATE=0;
fi; fi;
# if the difference greate than check date, run. CHECK INTERVAL is in days # if the difference greate than check date, run. CHECK INTERVAL is in days
if [ $(($CURRENT_DATE-$LAST_CHECK_DATE)) -ge $((${CHECK_INTERVAL}*86400)) ]; then if [ $((CURRENT_DATE - LAST_CHECK_DATE)) -ge $((CHECK_INTERVAL * 86400)) ]; then
RUN_CHECK=1; RUN_CHECK=1;
# always add verify data for automatic check # always add verify data for automatic check
OPT_CHECK_VERIFY_DATA="--verify-data"; OPT_CHECK_VERIFY_DATA="--verify-data";
# set new check time here # set new check time here
echo ${CURRENT_DATE} > "${BASE_FOLDER}${BACKUP_CHECK_FILE}"; echo "${CURRENT_DATE}" > "${BASE_FOLDER}${BACKUP_CHECK_FILE}";
fi; fi;
fi; fi;
elif [ ${CHECK} -eq 1 ]; then elif [ "${CHECK}" -eq 1 ]; then
RUN_CHECK=1; RUN_CHECK=1;
fi; fi;
if [ ${RUN_CHECK} -eq 1 ]; then if [ "${RUN_CHECK}" -eq 1 ]; then
# run borg check command # run borg check command
IFS=${_IFS}; IFS=${_IFS};
printf "${PRINTF_SUB_BLOCK}" "CHECK" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "CHECK" "$(date +'%F %T')" "${MODULE}";
@@ -54,28 +57,28 @@ if [ ${RUN_CHECK} -eq 1 ]; then
OPT_GLOB=""; OPT_GLOB="";
if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then
OPT_GLOB="-a '${CHECK_PREFIX}'" OPT_GLOB="-a '${CHECK_PREFIX}'"
elif [ ! -z "${CHECK_PREFIX}" ]; then elif [ -n "${CHECK_PREFIX}" ]; then
OPT_GLOB="-P ${CHECK_PREFIX}"; OPT_GLOB="-P ${CHECK_PREFIX}";
fi; fi;
# debug/dryrun # debug/dryrun
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ] || [ "${DRYRUN}" -eq 1 ]; then
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY}"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY}";
fi; fi;
# run info command if not a dry drun # run info command if not a dry drun
if [ ${DRYRUN} -eq 0 ]; then if [ "${DRYRUN}" -eq 0 ]; then
# if glob add glob command directly # if glob add glob command directly
if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then if [[ "${CHECK_PREFIX}" =~ $REGEX_GLOB ]]; then
${BORG_COMMAND} check ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} -a "${CHECK_PREFIX}" ${REPOSITORY}; ${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} -a "${CHECK_PREFIX}" "${REPOSITORY}";
else else
${BORG_COMMAND} check ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} ${REPOSITORY}; ${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} ${OPT_CHECK_VERIFY_DATA} ${OPT_GLOB} "${REPOSITORY}";
fi; fi;
fi; fi;
# print additional info for use --repair command # print additional info for use --repair command
# but only for manual checks # but only for manual checks
if [ ${VERBOSE} -eq 1 ] && [ ${CHECK} -eq 1 ]; then if [ "${VERBOSE}" -eq 1 ] && [ "${CHECK}" -eq 1 ]; then
echo ""; echo "";
echo "In case of needed repair: " echo "In case of needed repair: "
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_PROGRESS} --repair ${OPT_GLOB} ${REPOSITORY}"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} check ${OPT_REMOTE} ${OPT_PROGRESS} --repair ${OPT_GLOB} ${REPOSITORY}";
echo "Before running repair, a copy from the backup should be made because repair might damage a backup" echo "Before running repair, a copy from the backup should be made because repair might damage a backup"
fi; fi;
fi; fi;

View File

@@ -1,5 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
if [ -z "${MODULE}" ]; then if [ -z "${MODULE}" ]; then
echo "Script cannot be run on its own"; echo "Script cannot be run on its own";
exit 1; exit 1;
@@ -7,16 +10,16 @@ fi;
# unset borg settings # unset borg settings
unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK
# delete lock file if it exists
if [ -f "${BASE_FOLDER}${BACKUP_LOCK_FILE}" ]; then
rm "${BASE_FOLDER}${BACKUP_LOCK_FILE}";
fi;
# error abort without duration and error notice # error abort without duration and error notice
if [ $# -ge 1 ] && [ "$1" = "1" ]; then if [ $# -ge 1 ] && [ "$1" = "1" ]; then
printf "${PRINTF_MASTER_BLOCK}" "ERROR" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_MASTER_BLOCK}" "ERROR" "$(date +'%F %T')" "${MODULE}";
else else
# delete lock file
if [ -f "${BASE_FOLDER}${BACKUP_LOCK_FILE}" ]; then
rm "${BASE_FOLDER}${BACKUP_LOCK_FILE}";
fi;
# running time calculation # running time calculation
DURATION=$[ $(date +'%s')-$START ]; DURATION=$(( $(date +'%s') - START ));
echo "=== [Run time: $(convert_time ${DURATION})]"; echo "=== [Run time: $(convert_time ${DURATION})]";
printf "${PRINTF_MASTER_BLOCK}" "END" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_MASTER_BLOCK}" "END" "$(date +'%F %T')" "${MODULE}";
fi; fi;

View File

@@ -1,5 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
if [ -z "${MODULE}" ]; then if [ -z "${MODULE}" ]; then
echo "Script cannot be run on its own"; echo "Script cannot be run on its own";
exit 1; exit 1;
@@ -7,18 +10,53 @@ fi;
# compact (only if BORG COMPACT is set) # compact (only if BORG COMPACT is set)
# only for borg 1.2 # only for borg 1.2
if [ "$(version "$BORG_VERSION")" -ge "$(version "1.2.0")" ]; then
RUN_COMPACT=0;
if [ $# -ge 1 ] && [ "$1" = "auto" ]; then
# strip any spaces and convert to int
COMPACT_INTERVAL=$(echo "${COMPACT_INTERVAL}" | sed -e 's/ //g');
# not a valid compact interval, do no compact
if [ -z "${COMPACT_INTERVAL##*[!0-9]*}" ]; then
COMPACT_INTERVAL=0;
fi;
# get current date timestmap
CURRENT_DATE=$(date +%s);
if [ "${COMPACT_INTERVAL}" -eq 1 ]; then
RUN_COMPACT=1;
# set new compact time here
echo "${CURRENT_DATE}" > "${BASE_FOLDER}${BACKUP_COMPACT_FILE}";
elif [ "${COMPACT_INTERVAL}" -gt 1 ]; then
# else load last timestamp and check if today - last time stamp > days
if [ -z "${LAST_COMPACT_DATE}" ]; then
LAST_COMPACT_DATE=$(cat "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" 2>/dev/null | sed -e 's/ //g');
fi;
# file date is not a timestamp
if [ -z "${LAST_COMPACT_DATE##*[!0-9]*}" ]; then
LAST_COMPACT_DATE=0;
fi;
# if the difference greate than compact date, run. COMPACT INTERVAL is in days
if [ $((CURRENT_DATE - LAST_COMPACT_DATE)) -ge $((COMPACT_INTERVAL * 86400)) ]; then
RUN_COMPACT=1;
# set new compact time here
echo "${CURRENT_DATE}" > "${BASE_FOLDER}${BACKUP_COMPACT_FILE}";
fi;
fi;
elif [ "${COMPACT}" -eq 1 ]; then
RUN_COMPACT=1;
fi;
# reset to normal IFS, so command works here if [ "${RUN_COMPACT}" -eq 1 ]; then
IFS=${_IFS}; # reset to normal IFS, so command works here
if [ $(version $BORG_VERSION) -ge $(version "1.2.0") ]; then IFS=${_IFS};
printf "${PRINTF_SUB_BLOCK}" "COMPACT" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "COMPACT" "$(date +'%F %T')" "${MODULE}";
BORG_COMPACT="${BORG_COMMAND} compact -v ${OPT_PROGRESS} ${REPOSITORY}"; BORG_COMPACT="${BORG_COMMAND} compact ${OPT_REMOTE} -v ${OPT_PROGRESS} ${REPOSITORY}";
if [ ${DEBUG} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ]; then
echo "${BORG_COMPACT}"; echo "${BORG_COMPACT}";
fi; fi;
if [ ${DRYRUN} -eq 0 ]; then if [ "${DRYRUN}" -eq 0 ]; then
${BORG_COMPACT}; ${BORG_COMPACT};
fi; fi;
fi;
fi; fi;
# __END__ # __END__

View File

@@ -1,25 +1,33 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
if [ -z "${MODULE}" ]; then if [ -z "${MODULE}" ]; then
echo "Script cannot be run on its own"; echo "Script cannot be run on its own";
exit 1; exit 1;
fi; fi;
if [ ${INFO} -eq 1 ]; then if [ "${INFO}" -eq 1 ]; then
printf "${PRINTF_SUB_BLOCK}" "INFO" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "INFO" "$(date +'%F %T')" "${MODULE}";
# show command on debug or dry run # show command on debug or dry run
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ] || [ "${DRYRUN}" -eq 1 ]; then
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY}"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY}";
fi; fi;
# run info command if not a dry drun # run info command if not a dry drun
if [ ${DRYRUN} -eq 0 ]; then if [ "${DRYRUN}" -eq 0 ]; then
${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY}; ${BORG_COMMAND} info ${OPT_REMOTE} "${REPOSITORY}";
if [ "${VERBOSE}" -eq 1 ]; then
# print key information
echo "------------------------------------------------------------------------------";
${BORG_COMMAND} key export "${REPOSITORY}";
fi;
fi; fi;
if [ "${MODULE}" = "files" ]; then if [ "${MODULE}" = "files" ]; then
if [ $FOLDER_OK -eq 1 ]; then if [ "${FOLDER_OK}" -eq 1 ]; then
echo "--- [Run command]:"; echo "--- [Run command]:";
#IFS="#"; #IFS="#";
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${COMMAND} "${FOLDERS_Q[*]}; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${COMMAND} ${FOLDERS_Q[*]}";
else else
echo "[!] No folders where set for the backup"; echo "[!] No folders where set for the backup";
fi; fi;

View File

@@ -5,16 +5,24 @@ if [ -z "${MODULE}" ]; then
exit 1; exit 1;
fi; fi;
# E: inherit trap ERR
# T: DEBUG and RETURN traps are inherited
# u: unset variables ere error
set -ETu #-e -o pipefail set -ETu #-e -o pipefail
trap cleanup SIGINT SIGTERM ERR trap _catch ERR
trap _cleanup SIGINT SIGTERM
cleanup() { _cleanup() {
# script cleanup here # script cleanup here
echo "Some part of the script failed with an error: $? @LINE: $(caller)"; echo "Script abort: $? @LINE: $(caller)";
# unset exported vars # unset exported vars
unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK; unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK;
# end trap # end trap
trap - SIGINT SIGTERM ERR trap - SIGINT SIGTERM
}
_catch() {
local last_exit_code=$?;
echo "Some part of the script failed with ERROR: $last_exit_code @COMMAND: '$BASH_COMMAND' @LINE: $(caller)" >&2;
} }
# on exit unset any exported var # on exit unset any exported var
trap "unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK" EXIT; trap "unset BORG_BASE_DIR BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK BORG_RELOCATED_REPO_ACCESS_IS_OK" EXIT;
@@ -24,7 +32,7 @@ function version {
} }
# version for all general files # version for all general files
VERSION="4.3.0"; VERSION="4.8.0";
# borg version and borg comamnd # borg version and borg comamnd
BORG_VERSION=""; BORG_VERSION="";
@@ -49,7 +57,9 @@ EXCLUDE_FILE="";
# backup folder initialzed verify # backup folder initialzed verify
BACKUP_INIT_FILE=""; BACKUP_INIT_FILE="";
BACKUP_INIT_DATE=""; BACKUP_INIT_DATE="";
# fiel with last check timestamp # file with last compact date
BACKUP_COMPACT_FILE="";
# file with last check timestamp
BACKUP_CHECK_FILE=""; BACKUP_CHECK_FILE="";
# one time backup prefix tag, if set will use <tag>.<prefix>-Y-M-DTh:m:s type backup prefix # one time backup prefix tag, if set will use <tag>.<prefix>-Y-M-DTh:m:s type backup prefix
ONE_TIME_TAG=""; ONE_TIME_TAG="";
@@ -65,6 +75,7 @@ INFO=0;
VERIFY=0; VERIFY=0;
CHECK=0; CHECK=0;
CHECK_VERIFY_DATA=0; CHECK_VERIFY_DATA=0;
COMPACT=0;
INIT=0; INIT=0;
EXIT=0; EXIT=0;
PRINT=0; PRINT=0;
@@ -82,17 +93,20 @@ REGEX="";
REGEX_COMMENT="^[\ \t]*#"; REGEX_COMMENT="^[\ \t]*#";
REGEX_GLOB='\*'; REGEX_GLOB='\*';
REGEX_NUMERIC="^[0-9]{1,2}$"; REGEX_NUMERIC="^[0-9]{1,2}$";
REGEX_ERROR="^Some part of the script failed with an error:"; # port regex, but only approximately
REGEX_PORT="^[0-9]{2,5}$";
REGEX_ERROR="^Some part of the script failed with ERROR:";
PRUNE_DEBUG=""; PRUNE_DEBUG="";
INIT_REPOSITORY=0; INIT_REPOSITORY=0;
FOLDER_OK=0; FOLDER_OK=0;
TMP_EXCLUDE_FILE=""; TMP_EXCLUDE_FILE="";
# printf strings # printf strings
PRINTF_INFO_STRING="%-21s: %s\n"; PRINTF_INFO_STRING="%-23s: %s\n";
PRINTF_MASTER_BLOCK="=== [%-8s: %19s] ==[%s]====================================>\n"; PRINTF_MASTER_BLOCK="=== [%-8s: %19s] ==[%s]====================================>\n";
PRINTF_SUB_BLOCK="|-- [%-8s: %19s] --[%s]------------------------------------>\n"; PRINTF_SUB_BLOCK="|-- [%-8s: %19s] --[%s]------------------------------------>\n";
PRINTF_SUBEXT_BLOCK="|-- [%-8s: %s: %19s] --[%s]------------------------------------>\n"; PRINTF_SUBEXT_BLOCK="|-- [%-8s: %s: %19s] --[%s]------------------------------------>\n";
PRINTF_DB_SUB_BLOCK=">>- [%-8s: %s] =======================[%s]====================================>\n"; PRINTF_DB_SUB_BLOCK="|>- [%-8s: %s] ==[%s]=======================>\n";
PRINTF_DB_RUN_TIME_SUB_BLOCK=">>- [%-8s: %s] ==[%s]==[Run time: %s]=======================>\n";
# opt flags # opt flags
OPT_VERBOSE=""; OPT_VERBOSE="";
OPT_PROGRESS=""; OPT_PROGRESS="";
@@ -121,14 +135,21 @@ COMPRESSION_LEVEL="";
SUB_COMPRESSION=""; SUB_COMPRESSION="";
SUB_COMPRESSION_LEVEL=""; SUB_COMPRESSION_LEVEL="";
# encryption settings # encryption settings
DEFAULT_ENCRYPTION="none"; DEFAULT_ENCRYPTION="keyfile";
ENCRYPTION=""; ENCRYPTION="";
# force verify always # force verify always
DEFAULT_FORCE_VERIFY="false"; DEFAULT_FORCE_VERIFY="false";
FORCE_VERIFY=""; FORCE_VERIFY="";
FORCE_CHECK=""; # Deprecated name, use FORCE_VERIFY FORCE_CHECK=""; # Deprecated name, use FORCE_VERIFY
# compact
DEFAULT_COMPACT_INTERVAL="1";
LAST_COMPACT_DATE="";
COMPACT_INTERVAL="";
SUB_COMPACT_INTERVAL="";
# check
# default interval is none # default interval is none
DEFAULT_CHECK_INTERVAL=""; DEFAULT_CHECK_INTERVAL="";
LAST_CHECK_DATE="";
CHECK_INTERVAL=""; CHECK_INTERVAL="";
SUB_CHECK_INTERVAL=""; SUB_CHECK_INTERVAL="";
# backup set names # backup set names
@@ -137,22 +158,30 @@ SUB_BACKUP_SET="";
# for database backup only # for database backup only
DATABASE_FULL_DUMP=""; DATABASE_FULL_DUMP="";
DATABASE_USER=""; DATABASE_USER="";
DATABASE_USE_SUDO="";
DATABASE_SUDO_USER="";
DATABASE_PORT="";
DATABASE_HOST="";
# only for mysql old config file # only for mysql old config file
MYSQL_DB_CONFIG=""; MYSQL_DB_CONFIG="";
MYSQL_DB_CONFIG_PARAM=""; MYSQL_DB_CONFIG_PARAM="";
# gitea module # gitea module
GIT_USER=""; GIT_USER="";
GITEA_TMP=""; GITEA_WORKING_DIR="";
GITEA_TEMP_DIR="";
GITEA_BIN=""; GITEA_BIN="";
GITEA_CONFIG=""; GITEA_CONFIG="";
GITEA_EXPORT_TYPE="";
# zabbix module # zabbix module
ZABBIX_DUMP_BIN=""; ZABBIX_DUMP_BIN="";
ZABBIX_CONFIG=""; ZABBIX_CONFIG="";
ZABBIX_DATABASE=""; ZABBIX_DATABASE="";
ZABBIX_UNKNOWN_TABLES=""; ZABBIX_UNKNOWN_TABLES="";
ZABBIX_DB_PORT="";
OPT_ZABBIX_DUMP=""; OPT_ZABBIX_DUMP="";
OPT_ZABBIX_CONFIG=""; OPT_ZABBIX_CONFIG="";
OPT_ZABBIX_UNKNOWN_TABLES=""; OPT_ZABBIX_UNKNOWN_TABLES="";
OPT_ZABBIX_DB_PORT="";
# default keep 7 days, 4 weeks, 6 months, 1 year # default keep 7 days, 4 weeks, 6 months, 1 year
# if set 0, ignore/off # if set 0, ignore/off
# note that for last/hourly it is needed to create a different # note that for last/hourly it is needed to create a different
@@ -192,6 +221,7 @@ function usage()
-T <tag>: create one time stand alone backup prefixed with tag name -T <tag>: create one time stand alone backup prefixed with tag name
-D <tag backup set>: remove a tagged backup set, full name must be given -D <tag backup set>: remove a tagged backup set, full name must be given
-b <borg executable>: override default path -b <borg executable>: override default path
-Z: run compress after prune/backup. Only for borg 1.2 or newer
-C: run borg check if repository is ok -C: run borg check if repository is ok
-y: in combination with -C: add --verify-data -y: in combination with -C: add --verify-data
-p <archive prefix|glob>: in combinatio with -C: only check archives with prefix or glob -p <archive prefix|glob>: in combinatio with -C: only check archives with prefix or glob
@@ -213,7 +243,7 @@ function usage()
} }
# set options # set options
while getopts ":c:L:T:D:b:p:vldniCVeIPyh" opt; do while getopts ":c:L:T:D:b:p:vldniCVeIPyZh" opt; do
case "${opt}" in case "${opt}" in
c|config) c|config)
BASE_FOLDER=${OPTARG}; BASE_FOLDER=${OPTARG};
@@ -230,6 +260,10 @@ while getopts ":c:L:T:D:b:p:vldniCVeIPyh" opt; do
b|borg) b|borg)
OPT_BORG_EXECUTEABLE=${OPTARG}; OPT_BORG_EXECUTEABLE=${OPTARG};
;; ;;
Z|Compact)
# will run compact alone
COMPACT=1;
;;
C|Check) C|Check)
# will run borg check # will run borg check
# alt modes --repository-only, --archives-only, # alt modes --repository-only, --archives-only,
@@ -314,32 +348,32 @@ if [ ${VERIFY} -eq 1 ] || [ ${INIT} -eq 1 ] && [ ${INFO} -eq 1 ]; then
exit 1; exit 1;
fi; fi;
# print -P cannot be run with -i/-C/-I together # print -P cannot be run with -i/-C/-I together
if [ ${PRINT} -eq 1 ] && ([ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]); then if [ ${PRINT} -eq 1 ] && { [ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]; }; then
echo "Cannot have -P print option and -i info, -V verify or -I initizalize option at the same time"; echo "Cannot have -P print option and -i info, -V verify or -I initizalize option at the same time";
exit 1; exit 1;
fi; fi;
# if tag is set, you can't have init, verify, info, etc # if tag is set, you can't have init, verify, info, etc
if [ ! -z "${ONE_TIME_TAG}" ] && ([ ${PRINT} -eq 1 ] || [ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]); then if [ -n "${ONE_TIME_TAG}" ] && { [ ${PRINT} -eq 1 ] || [ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]; }; then
echo "Cannot have -T '${ONE_TIME_TAG}' option with -i info, -V verify, -I initialize or -P print option at the same time"; echo "Cannot have -T '${ONE_TIME_TAG}' option with -i info, -V verify, -I initialize or -P print option at the same time";
exit 1; exit 1;
fi; fi;
# verify only alphanumeric, no spaces, only underscore and dash # verify only alphanumeric, no spaces, only underscore and dash
if [ ! -z "${ONE_TIME_TAG}" ] && ! [[ "${ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+$ ]]; then if [ -n "${ONE_TIME_TAG}" ] && ! [[ "${ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+$ ]]; then
echo "One time tag '${ONE_TIME_TAG}' must be alphanumeric with dashes and underscore only."; echo "One time tag '${ONE_TIME_TAG}' must be alphanumeric with dashes and underscore only.";
exit 1; exit 1;
elif [ ! -z "${ONE_TIME_TAG}" ]; then elif [ -n "${ONE_TIME_TAG}" ]; then
# all ok, attach . at the end # all ok, attach '.' at the end
ONE_TIME_TAG=${ONE_TIME_TAG}"."; ONE_TIME_TAG=${ONE_TIME_TAG}".";
fi; fi;
# if -D, cannot be with -T, -i, -C, -I, -P # if -D, cannot be with -T, -i, -C, -I, -P
if [ ! -z "${DELETE_ONE_TIME_TAG}" ] && ([ ! -z "${ONE_TIME_TAG}" ] || [ ${PRINT} -eq 1 ] || [ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]); then if [ -n "${DELETE_ONE_TIME_TAG}" ] && { [ -n "${ONE_TIME_TAG}" ] || [ ${PRINT} -eq 1 ] || [ ${INIT} -eq 1 ] || [ ${VERIFY} -eq 1 ] || [ ${INFO} -eq 1 ]; }; then
echo "Cannot have -D delete tag option with -T one time tag, -i info, -V verify, -I initialize or -P print option at the same time"; echo "Cannot have -D delete tag option with -T one time tag, -i info, -V verify, -I initialize or -P print option at the same time";
exit 1; exit 1;
fi; fi;
# -D also must be in valid backup set format # -D also must be in valid backup set format
# ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},(\*-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T\*$ ]] # ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},(\*-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T\*$ ]]
if [ ! -z "${DELETE_ONE_TIME_TAG}" ] && ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},([A-Za-z0-9_-]+-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}$ ]] && ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},(\*-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T\*$ ]]; then if [ -n "${DELETE_ONE_TIME_TAG}" ] && ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},([A-Za-z0-9_-]+-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}$ ]] && ! [[ "${DELETE_ONE_TIME_TAG}" =~ ^[A-Za-z0-9_-]+\.${MODULE},(\*-)?[0-9]{4}-[0-9]{2}-[0-9]{2}T\*$ ]]; then
echo "Delete one time tag '${DELETE_ONE_TIME_TAG}' is in an invalid format. " echo "Delete one time tag '${DELETE_ONE_TIME_TAG}' is in an invalid format."
echo "Please verify existing tags with -P option." echo "Please verify existing tags with -P option."
echo "For a globing be sure it is in the format of: TAG.MODULE,*-YYYY-MM-DDT*"; echo "For a globing be sure it is in the format of: TAG.MODULE,*-YYYY-MM-DDT*";
echo "Note the dash (-) after the first *, also time (T) is a globa (*) must." echo "Note the dash (-) after the first *, also time (T) is a globa (*) must."
@@ -351,7 +385,7 @@ if [ ${CHECK_VERIFY_DATA} -eq 1 ] && [ ${CHECK} -eq 0 ]; then
exit 1; exit 1;
fi; fi;
# -p can't be set without -C # -p can't be set without -C
if [ ! -z "${CHECK_PREFIX}" ] && [ ${CHECK} -eq 0 ]; then if [ -n "${CHECK_PREFIX}" ] && [ ${CHECK} -eq 0 ]; then
echo "-p (pattern|glob) for check cannot be run without -C (Check) options"; echo "-p (pattern|glob) for check cannot be run without -C (Check) options";
exit 1; exit 1;
fi; fi;
@@ -384,20 +418,21 @@ fi;
. "${BASE_FOLDER}${SETTINGS_FILE}"; . "${BASE_FOLDER}${SETTINGS_FILE}";
# if OPTION SET overrides ALL others # if OPTION SET overrides ALL others
if [ ! -z "${OPT_BORG_EXECUTEABLE}" ]; then if [ -n "${OPT_BORG_EXECUTEABLE}" ]; then
BORG_COMMAND="${OPT_BORG_EXECUTEABLE}"; BORG_COMMAND="${OPT_BORG_EXECUTEABLE}";
if [ ! -f "${BORG_COMMAND}" ]; then if [ ! -f "${BORG_COMMAND}" ]; then
echo "borg command not found with option -b: ${BORG_COMMAND}"; echo "borg command not found with option -b: ${BORG_COMMAND}";
exit; exit;
fi; fi;
# if in setting file, use this # if in setting file, use this
elif [ ! -z "${BORG_EXECUTEABLE}" ]; then elif [ -n "${BORG_EXECUTEABLE}" ]; then
BORG_COMMAND="${BORG_EXECUTEABLE}"; BORG_COMMAND="${BORG_EXECUTEABLE}";
if [ ! -f "${BORG_COMMAND}" ]; then if [ ! -f "${BORG_COMMAND}" ]; then
echo "borg command not found with setting: ${BORG_COMMAND}"; echo "borg command not found with setting: ${BORG_COMMAND}";
exit; exit;
fi; fi;
elif ! command -v borg &> /dev/null; then elif ! command -v borg &> /dev/null; then
# elif [ -z ($command -v borg) ]; then
echo "borg backup seems not to be installed, please verify paths"; echo "borg backup seems not to be installed, please verify paths";
exit; exit;
fi; fi;
@@ -422,11 +457,15 @@ fi;
if [ -z "${ENCRYPTION}" ]; then if [ -z "${ENCRYPTION}" ]; then
ENCRYPTION="${DEFAULT_ENCRYPTION}"; ENCRYPTION="${DEFAULT_ENCRYPTION}";
fi; fi;
# check interval override
if [ -z "${COMPACT_INTERVAL}" ]; then
COMPACT_INTERVAL="${DEFAULT_COMPACT_INTERVAL}";
fi;
if [ -z "${CHECK_INTERVAL}" ]; then if [ -z "${CHECK_INTERVAL}" ]; then
CHECK_INTERVAL="${DEFAULT_CHECK_INTERVAL}"; CHECK_INTERVAL="${DEFAULT_CHECK_INTERVAL}";
fi; fi;
# deprecated name FORCE_CHECK, use FORCE_VERIFY instead # deprecated name FORCE_CHECK, use FORCE_VERIFY instead
if [ ! -z "${FORCE_CHECK}" ]; then if [ -n "${FORCE_CHECK}" ]; then
FORCE_VERIFY="${FORCE_CHECK}"; FORCE_VERIFY="${FORCE_CHECK}";
fi; fi;
if [ -z "${FORCE_VERIFY}" ]; then if [ -z "${FORCE_VERIFY}" ]; then
@@ -457,44 +496,48 @@ SETTINGS_FILE_SUB=$(echo "${SETTINGS_FILE}" | sed -e "s/\.settings/\.${MODULE,,}
if [ -f "${BASE_FOLDER}${SETTINGS_FILE_SUB}" ]; then if [ -f "${BASE_FOLDER}${SETTINGS_FILE_SUB}" ]; then
. "${BASE_FOLDER}${SETTINGS_FILE_SUB}"; . "${BASE_FOLDER}${SETTINGS_FILE_SUB}";
# if SUB_ set override master # if SUB_ set override master
if [ ! -z "${SUB_BACKUP_FILE}" ]; then if [ -n "${SUB_BACKUP_FILE}" ]; then
BACKUP_FILE=${SUB_BACKUP_FILE} BACKUP_FILE=${SUB_BACKUP_FILE}
fi; fi;
# if sub backup set it set, override current # if sub backup set it set, override current
if [ ! -z "${SUB_BACKUP_SET}" ]; then if [ -n "${SUB_BACKUP_SET}" ]; then
BACKUP_SET=${SUB_BACKUP_SET}; BACKUP_SET=${SUB_BACKUP_SET};
fi; fi;
# ovrride compression # ovrride compression
if [ ! -z "${SUB_COMPRESSION}" ]; then if [ -n "${SUB_COMPRESSION}" ]; then
COMPRESSION=${SUB_COMPRESSION}; COMPRESSION=${SUB_COMPRESSION};
fi; fi;
if [ ! -z "${SUB_COMPRESSION_LEVEL}" ]; then if [ -n "${SUB_COMPRESSION_LEVEL}" ]; then
COMPRESSION_LEVEL=${SUB_COMPRESSION_LEVEL}; COMPRESSION_LEVEL=${SUB_COMPRESSION_LEVEL};
fi; fi;
# compact interval override
if [ -n "${SUB_COMPACT_INTERVAL}" ]; then
COMPACT_INTERVAL="${SUB_COMPACT_INTERVAL}";
fi;
# override check interval # override check interval
if [ ! -z "${SUB_CHECK_INTERVAL}" ]; then if [ -n "${SUB_CHECK_INTERVAL}" ]; then
CHECK_INTERVAL="${SUB_CHECK_INTERVAL}"; CHECK_INTERVAL="${SUB_CHECK_INTERVAL}";
fi; fi;
# check override for keep time # check override for keep time
if [ ! -z "${SUB_KEEP_LAST}" ]; then if [ -n "${SUB_KEEP_LAST}" ]; then
KEEP_LAST=${SUB_KEEP_LAST}; KEEP_LAST=${SUB_KEEP_LAST};
fi; fi;
if [ ! -z "${SUB_KEEP_HOURS}" ]; then if [ -n "${SUB_KEEP_HOURS}" ]; then
KEEP_HOURS=${SUB_KEEP_HOURS}; KEEP_HOURS=${SUB_KEEP_HOURS};
fi; fi;
if [ ! -z "${SUB_KEEP_DAYS}" ]; then if [ -n "${SUB_KEEP_DAYS}" ]; then
KEEP_DAYS=${SUB_KEEP_DAYS}; KEEP_DAYS=${SUB_KEEP_DAYS};
fi; fi;
if [ ! -z "${SUB_KEEP_WEEKS}" ]; then if [ -n "${SUB_KEEP_WEEKS}" ]; then
KEEP_WEEKS=${SUB_KEEP_WEEKS}; KEEP_WEEKS=${SUB_KEEP_WEEKS};
fi; fi;
if [ ! -z "${SUB_KEEP_MONTHS}" ]; then if [ -n "${SUB_KEEP_MONTHS}" ]; then
KEEP_MONTHS=${SUB_KEEP_MONTHS}; KEEP_MONTHS=${SUB_KEEP_MONTHS};
fi; fi;
if [ ! -z "${SUB_KEEP_YEARS}" ]; then if [ -n "${SUB_KEEP_YEARS}" ]; then
KEEP_YEARS=${SUB_KEEP_YEARS}; KEEP_YEARS=${SUB_KEEP_YEARS};
fi; fi;
if [ ! -z "${SUB_KEEP_WITHIN}" ]; then if [ -n "${SUB_KEEP_WITHIN}" ]; then
KEEP_WITHIN=${SUB_KEEP_WITHIN}; KEEP_WITHIN=${SUB_KEEP_WITHIN};
fi; fi;
fi; fi;
@@ -537,11 +580,12 @@ if [[ ${TARGET_FOLDER} =~ ^~\/ ]]; then
exit 1; exit 1;
fi fi
# COMPACT_INTERVAL must be a number from -1 to 365
# CHECK_INTERVAL must be a number from -1 to 365 # CHECK_INTERVAL must be a number from -1 to 365
# log file set and check # log file set and check
# option folder overrides all other folders # option folder overrides all other folders
if [ ! -z "${OPT_LOG_FOLDER}" ]; then if [ -n "${OPT_LOG_FOLDER}" ]; then
LOG_FOLDER="${OPT_LOG_FOLDER}"; LOG_FOLDER="${OPT_LOG_FOLDER}";
fi; fi;
# if empty folder set to default folder # if empty folder set to default folder
@@ -569,11 +613,26 @@ fi;
# if ENCRYPTION is empty or not in the valid list fall back to none # if ENCRYPTION is empty or not in the valid list fall back to none
# NOTE This is currently set in default and doesn't need to be set on empty # NOTE This is currently set in default and doesn't need to be set on empty
# only ivalid should be checked # only ivalid should be checked
#if [ -z "${ENCRYPTION}" ]; then if
# ENCRYPTION="none"; [ "${ENCRYPTION}" = "authenticated" ] ||
#else [ "${ENCRYPTION}" = "repokey" ] ||
# TODO check for invalid encryption string [ "${ENCRYPTION}" = "authenticated-blake2" ] ||
#fi; [ "${ENCRYPTION}" = "repokey-blake2" ] ;
then
# if "authenticated" or "repokey" a password must be set
if [[ ! -v BORG_PASSPHRASE ]] && [[ ! -v BORG_PASSCOMMAND ]] && [[ ! -v BORG_PASSPHRASE_FD ]]; then
echo "Encryption method '${ENCRYPTION}' requires a BORG_PASSPHRASE, BORG_PASSCOMMAND or BORG_PASSPHRASE_FD to be set.";
exit 1;
fi;
elif [ "${ENCRYPTION}" = "keyfile" ] || [ "${ENCRYPTION}" = "keyfile-blake2" ]; then
# if no password, set empty password
if [[ ! -v BORG_PASSPHRASE ]] && [[ ! -v BORG_PASSCOMMAND ]] && [[ ! -v BORG_PASSPHRASE_FD ]]; then
export BORG_PASSPHRASE="";
fi;
elif [ "${ENCRYPTION}" != "none" ]; then
echo "Encryption method '${ENCRYPTION}' is not valid.";
exit 1;
fi;
## FUNCTIONS ## FUNCTIONS
@@ -588,30 +647,30 @@ function convert_time
{ {
timestamp=${1}; timestamp=${1};
# round to four digits for ms # round to four digits for ms
timestamp=$(printf "%1.4f" $timestamp); timestamp=$(printf "%1.4f" "$timestamp");
# get the ms part and remove any leading 0 # get the ms part and remove any leading 0
ms=$(echo ${timestamp} | cut -d "." -f 2 | sed -e 's/^0*//'); ms=$(echo "${timestamp}" | cut -d "." -f 2 | sed -e 's/^0*//');
timestamp=$(echo ${timestamp} | cut -d "." -f 1); timestamp=$(echo "${timestamp}" | cut -d "." -f 1);
timegroups=(86400 3600 60 1); # day, hour, min, sec timegroups=(86400 3600 60 1); # day, hour, min, sec
timenames=("d" "h" "m" "s"); # day, hour, min, sec timenames=("d" "h" "m" "s"); # day, hour, min, sec
output=( ); output=( );
time_string=; time_string=;
for timeslice in ${timegroups[@]}; do for timeslice in "${timegroups[@]}"; do
# floor for the division, push to output # floor for the division, push to output
output[${#output[*]}]=$(awk "BEGIN {printf \"%d\", ${timestamp}/${timeslice}}"); output[${#output[*]}]=$(awk "BEGIN {printf \"%d\", ${timestamp}/${timeslice}}");
timestamp=$(awk "BEGIN {printf \"%d\", ${timestamp}%${timeslice}}"); timestamp=$(awk "BEGIN {printf \"%d\", ${timestamp}%${timeslice}}");
done; done;
for ((i=0; i<${#output[@]}; i++)); do for ((i=0; i<${#output[@]}; i++)); do
if [ ${output[$i]} -gt 0 ] || [ ! -z "$time_string" ]; then if [ "${output[$i]}" -gt 0 ] || [ -n "$time_string" ]; then
if [ ! -z "${time_string}" ]; then if [ -n "${time_string}" ]; then
time_string=${time_string}" "; time_string=${time_string}" ";
fi; fi;
time_string=${time_string}${output[$i]}${timenames[$i]}; time_string=${time_string}${output[$i]}${timenames[$i]};
fi; fi;
done; done;
if [ ! -z ${ms} ] && [ "${ms}" != "nan" ] && [ ${ms} -gt 0 ]; then if [ -n "${ms}" ] && [ "${ms}" != "nan" ] && [ "${ms}" -gt 0 ]; then
time_string=${time_string}" "${ms}"ms"; time_string="${time_string} ${ms}ms";
fi; fi;
# just in case the time is 0 # just in case the time is 0
if [ -z "${time_string}" ]; then if [ -z "${time_string}" ]; then

View File

@@ -1,5 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
if [ -z "${MODULE}" ]; then if [ -z "${MODULE}" ]; then
echo "Script cannot be run on its own"; echo "Script cannot be run on its own";
exit 1; exit 1;
@@ -10,16 +13,7 @@ START=$(date +'%s');
# set init date, or today if not file is set # set init date, or today if not file is set
BACKUP_INIT_DATE=''; BACKUP_INIT_DATE='';
if [ -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then if [ -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
BACKUP_INIT_DATE=$(printf '%(%c)T' $(cat "${BASE_FOLDER}${BACKUP_INIT_FILE}" 2>/dev/null)); BACKUP_INIT_DATE=$(printf '%(%c)T' "$(cat "${BASE_FOLDER}${BACKUP_INIT_FILE}" 2>/dev/null)");
fi;
# last check date if set
BACKUP_LAST_CHECK_DATE='';
LAST_CHECK_DATE='';
CONVERT_TIME='';
if [ -f "${BASE_FOLDER}${BACKUP_CHECK_FILE}" ]; then
LAST_CHECK_DATE=$(cat "${BASE_FOLDER}${BACKUP_CHECK_FILE}" 2>/dev/null);
BACKUP_LAST_CHECK_DATE=$(printf '%(%c)T' ${LAST_CHECK_DATE});
CONVERT_TIME=$(convert_time $(($(date +%s)-${LAST_CHECK_DATE})));
fi; fi;
# start logging from here # start logging from here
exec &> >(tee -a "${LOG}"); exec &> >(tee -a "${LOG}");
@@ -37,23 +31,44 @@ printf "${PRINTF_INFO_STRING}" "Hostname" "${HOSTNAME}";
printf "${PRINTF_INFO_STRING}" "Base folder" "${BASE_FOLDER}"; printf "${PRINTF_INFO_STRING}" "Base folder" "${BASE_FOLDER}";
# Module init date (when init file was writen) # Module init date (when init file was writen)
printf "${PRINTF_INFO_STRING}" "Module init date" "${BACKUP_INIT_DATE}"; printf "${PRINTF_INFO_STRING}" "Module init date" "${BACKUP_INIT_DATE}";
# print last compact date if positive integer
# only if borg > 1.2
if [ "$(version "$BORG_VERSION")" -ge "$(version "1.2.0")" ]; then
if [ "${COMPACT_INTERVAL##*[!0-9]*}" ]; then
printf "${PRINTF_INFO_STRING}" "Module compact interval" "${COMPACT_INTERVAL}";
if [ -f "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" ]; then
LAST_COMPACT_DATE=$(cat "${BASE_FOLDER}${BACKUP_COMPACT_FILE}" 2>/dev/null);
printf "${PRINTF_INFO_STRING}" "Module last compact" \
"$(printf '%(%c)T' "${LAST_COMPACT_DATE}") ($(convert_time $(($(date +%s) - LAST_COMPACT_DATE))) ago)";
else
printf "${PRINTF_INFO_STRING}" "Module last compact" "No compact run yet"
fi;
fi;
fi;
# print last check date if positive integer # print last check date if positive integer
if [ "${CHECK_INTERVAL##*[!0-9]*}" ]; then if [ "${CHECK_INTERVAL##*[!0-9]*}" ]; then
printf "${PRINTF_INFO_STRING}" "Module check interval" "${CHECK_INTERVAL}"; printf "${PRINTF_INFO_STRING}" "Module check interval" "${CHECK_INTERVAL}";
printf "${PRINTF_INFO_STRING}" "Module last check" "${BACKUP_LAST_CHECK_DATE} (${CONVERT_TIME} ago)"; # get last check date
if [ -f "${BASE_FOLDER}${BACKUP_CHECK_FILE}" ]; then
LAST_CHECK_DATE=$(cat "${BASE_FOLDER}${BACKUP_CHECK_FILE}" 2>/dev/null);
printf "${PRINTF_INFO_STRING}" "Module last check" \
"$(printf '%(%c)T' "${LAST_CHECK_DATE}") ($(convert_time $(($(date +%s) - LAST_CHECK_DATE))) ago)";
else
printf "${PRINTF_INFO_STRING}" "Module last check" "No check run yet";
fi;
fi; fi;
# if force verify is true set VERIFY to 1 unless INFO is 1 # if force verify is true set VERIFY to 1 unless INFO is 1
# Needs bash 4.0 at lesat for this # Needs bash 4.0 at lesat for this
if [ "${FORCE_VERIFY,,}" = "true" ] && [ ${INFO} -eq 0 ]; then if [ "${FORCE_VERIFY,,}" = "true" ] && [ "${INFO}" -eq 0 ]; then
VERIFY=1; VERIFY=1;
if [ ${DEBUG} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ]; then
echo "Force repository verify"; echo "Force repository verify";
fi; fi;
fi; fi;
# remote borg path # remote borg path
if [ ! -z "${TARGET_BORG_PATH}" ]; then if [ -n "${TARGET_BORG_PATH}" ]; then
if [[ "${TARGET_BORG_PATH}" =~ \ |\' ]]; then if [[ "${TARGET_BORG_PATH}" =~ \ |\' ]]; then
echo "Space found in ${TARGET_BORG_PATH}. Aborting"; echo "Space found in ${TARGET_BORG_PATH}. Aborting";
echo "There are issues with passing on paths with spaces" echo "There are issues with passing on paths with spaces"
@@ -90,16 +105,16 @@ TARGET_SERVER='';
# allow host only (if full setup in .ssh/config) # allow host only (if full setup in .ssh/config)
# user@host OR ssh://user@host:port/ IF TARGET_PORT is set # user@host OR ssh://user@host:port/ IF TARGET_PORT is set
# user/host/port # user/host/port
if [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then if [ -n "${TARGET_USER}" ] && [ -n "${TARGET_HOST}" ] && [ -n "${TARGET_PORT}" ]; then
TARGET_SERVER="ssh://${TARGET_USER}@${TARGET_HOST}:${TARGET_PORT}/"; TARGET_SERVER="ssh://${TARGET_USER}@${TARGET_HOST}:${TARGET_PORT}/";
# host/port # host/port
elif [ ! -z "${TARGET_HOST}" ] && [ ! -z "${TARGET_PORT}" ]; then elif [ -n "${TARGET_HOST}" ] && [ -n "${TARGET_PORT}" ]; then
TARGET_SERVER="ssh://${TARGET_HOST}:${TARGET_PORT}/"; TARGET_SERVER="ssh://${TARGET_HOST}:${TARGET_PORT}/";
# user/host # user/host
elif [ ! -z "${TARGET_USER}" ] && [ ! -z "${TARGET_HOST}" ]; then elif [ -n "${TARGET_USER}" ] && [ -n "${TARGET_HOST}" ]; then
TARGET_SERVER="${TARGET_USER}@${TARGET_HOST}:"; TARGET_SERVER="${TARGET_USER}@${TARGET_HOST}:";
# host # host
elif [ ! -z "${TARGET_HOST}" ]; then elif [ -n "${TARGET_HOST}" ]; then
TARGET_SERVER="${TARGET_HOST}:"; TARGET_SERVER="${TARGET_HOST}:";
fi; fi;
# we dont allow special characters, so we don't need to special escape it # we dont allow special characters, so we don't need to special escape it
@@ -108,13 +123,13 @@ printf "${PRINTF_INFO_STRING}" "Repository" "${REPOSITORY}";
# check if given compression name and level are valid # check if given compression name and level are valid
OPT_COMPRESSION=''; OPT_COMPRESSION='';
if [ ! -z "${COMPRESSION}" ]; then if [ -n "${COMPRESSION}" ]; then
# valid compression # valid compression
if [ "${COMPRESSION}" = "lz4" ] || [ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ] || [ "${COMPRESSION}" = "zstd" ]; then if [ "${COMPRESSION}" = "lz4" ] || [ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ] || [ "${COMPRESSION}" = "zstd" ]; then
OPT_COMPRESSION="-C=${COMPRESSION}"; OPT_COMPRESSION="-C=${COMPRESSION}";
# if COMPRESSION_LEVEL, check it is a valid regex # if COMPRESSION_LEVEL, check it is a valid regex
# for zlib, zstd, lzma # for zlib, zstd, lzma
if [ ! -z "${COMPRESSION_LEVEL}" ] && ([ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ] || [ "${COMPRESSION}" = "zstd" ]); then if [ -n "${COMPRESSION_LEVEL}" ] && { [ "${COMPRESSION}" = "zlib" ] || [ "${COMPRESSION}" = "lzma" ] || [ "${COMPRESSION}" = "zstd" ]; }; then
MIN_COMPRESSION=0; MIN_COMPRESSION=0;
MAX_COMPRESSION=0; MAX_COMPRESSION=0;
case "${COMPRESSION}" in case "${COMPRESSION}" in
@@ -140,10 +155,10 @@ if [ ! -z "${COMPRESSION}" ]; then
# fi; # fi;
error_message="[! $(date +'%F %T')] Compression level for ${COMPRESSION} needs to be a numeric value between ${MIN_COMPRESSION} and ${MAX_COMPRESSION}: ${COMPRESSION_LEVEL}"; error_message="[! $(date +'%F %T')] Compression level for ${COMPRESSION} needs to be a numeric value between ${MIN_COMPRESSION} and ${MAX_COMPRESSION}: ${COMPRESSION_LEVEL}";
if ! [[ "${COMPRESSION_LEVEL}" =~ ${REGEX_NUMERIC} ]]; then if ! [[ "${COMPRESSION_LEVEL}" =~ ${REGEX_NUMERIC} ]]; then
echo ${error_message}; echo "${error_message}";
exit 1; exit 1;
elif [ ${COMPRESSION_LEVEL} -lt ${MIN_COMPRESSION} ] || [ ${COMPRESSION_LEVEL} -gt ${MAX_COMPRESSION} ]; then elif [ "${COMPRESSION_LEVEL}" -lt "${MIN_COMPRESSION}" ] || [ "${COMPRESSION_LEVEL}" -gt "${MAX_COMPRESSION}" ]; then
echo ${error_message}; echo "${error_message}";
exit 1; exit 1;
else else
OPT_COMPRESSION=${OPT_COMPRESSION}","${COMPRESSION_LEVEL}; OPT_COMPRESSION=${OPT_COMPRESSION}","${COMPRESSION_LEVEL};
@@ -165,7 +180,7 @@ KEEP_OPTIONS=();
# keep info string (for files) # keep info string (for files)
KEEP_INFO=""; KEEP_INFO="";
# override standard keep for tagged backups # override standard keep for tagged backups
if [ ! -z "${ONE_TIME_TAG}" ]; then if [ -n "${ONE_TIME_TAG}" ]; then
BACKUP_SET="{now:%Y-%m-%dT%H:%M:%S}"; BACKUP_SET="{now:%Y-%m-%dT%H:%M:%S}";
# set empty to avoid problems # set empty to avoid problems
KEEP_OPTIONS=(""); KEEP_OPTIONS=("");
@@ -173,32 +188,32 @@ else
# build options and info string, # build options and info string,
# also flag BACKUP_SET check if hourly is set # also flag BACKUP_SET check if hourly is set
BACKUP_SET_VERIFY=0; BACKUP_SET_VERIFY=0;
if [ ${KEEP_LAST} -gt 0 ]; then if [ "${KEEP_LAST}" -gt 0 ]; then
KEEP_OPTIONS+=("--keep-last=${KEEP_LAST}"); KEEP_OPTIONS+=("--keep-last=${KEEP_LAST}");
KEEP_INFO="${KEEP_INFO}, last: ${KEEP_LAST}"; KEEP_INFO="${KEEP_INFO}, last: ${KEEP_LAST}";
fi; fi;
if [ ${KEEP_HOURS} -gt 0 ]; then if [ "${KEEP_HOURS}" -gt 0 ]; then
KEEP_OPTIONS+=("--keep-hourly=${KEEP_HOURS}"); KEEP_OPTIONS+=("--keep-hourly=${KEEP_HOURS}");
KEEP_INFO="${KEEP_INFO}, hourly: ${KEEP_HOURS}"; KEEP_INFO="${KEEP_INFO}, hourly: ${KEEP_HOURS}";
BACKUP_SET_VERIFY=1; BACKUP_SET_VERIFY=1;
fi; fi;
if [ ${KEEP_DAYS} -gt 0 ]; then if [ "${KEEP_DAYS}" -gt 0 ]; then
KEEP_OPTIONS+=("--keep-daily=${KEEP_DAYS}"); KEEP_OPTIONS+=("--keep-daily=${KEEP_DAYS}");
KEEP_INFO="${KEEP_INFO}, daily: ${KEEP_DAYS}"; KEEP_INFO="${KEEP_INFO}, daily: ${KEEP_DAYS}";
fi; fi;
if [ ${KEEP_WEEKS} -gt 0 ]; then if [ "${KEEP_WEEKS}" -gt 0 ]; then
KEEP_OPTIONS+=("--keep-weekly=${KEEP_WEEKS}"); KEEP_OPTIONS+=("--keep-weekly=${KEEP_WEEKS}");
KEEP_INFO="${KEEP_INFO}, weekly: ${KEEP_WEEKS}"; KEEP_INFO="${KEEP_INFO}, weekly: ${KEEP_WEEKS}";
fi; fi;
if [ ${KEEP_MONTHS} -gt 0 ]; then if [ "${KEEP_MONTHS}" -gt 0 ]; then
KEEP_OPTIONS+=("--keep-monthly=${KEEP_MONTHS}"); KEEP_OPTIONS+=("--keep-monthly=${KEEP_MONTHS}");
KEEP_INFO="${KEEP_INFO}, monthly: ${KEEP_MONTHS}"; KEEP_INFO="${KEEP_INFO}, monthly: ${KEEP_MONTHS}";
fi; fi;
if [ ${KEEP_YEARS} -gt 0 ]; then if [ "${KEEP_YEARS}" -gt 0 ]; then
KEEP_OPTIONS+=("--keep-yearly=${KEEP_YEARS}"); KEEP_OPTIONS+=("--keep-yearly=${KEEP_YEARS}");
KEEP_INFO="${KEEP_INFO}, yearly: ${KEEP_YEARS}"; KEEP_INFO="${KEEP_INFO}, yearly: ${KEEP_YEARS}";
fi; fi;
if [ ! -z "${KEEP_WITHIN}" ]; then if [ -n "${KEEP_WITHIN}" ]; then
# check for invalid string. can only be number + H|d|w|m|y # check for invalid string. can only be number + H|d|w|m|y
if [[ "${KEEP_WITHIN}" =~ ^[0-9]+[Hdwmy]{1}$ ]]; then if [[ "${KEEP_WITHIN}" =~ ^[0-9]+[Hdwmy]{1}$ ]]; then
KEEP_OPTIONS+=("--keep-within=${KEEP_WITHIN}"); KEEP_OPTIONS+=("--keep-within=${KEEP_WITHIN}");
@@ -232,7 +247,7 @@ fi;
if [ -f "${BASE_FOLDER}${BACKUP_LOCK_FILE}" ]; then if [ -f "${BASE_FOLDER}${BACKUP_LOCK_FILE}" ]; then
LOCK_PID=$(cat "${BASE_FOLDER}${BACKUP_LOCK_FILE}" 2>/dev/null); LOCK_PID=$(cat "${BASE_FOLDER}${BACKUP_LOCK_FILE}" 2>/dev/null);
# check if lock file pid has an active program attached to it # check if lock file pid has an active program attached to it
if [ -f /proc/${LOCK_PID}/cmdline ]; then if [ -f "/proc/${LOCK_PID}/cmdline" ]; then
echo "Script is already running on PID: ${$}"; echo "Script is already running on PID: ${$}";
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
exit 1; exit 1;
@@ -260,11 +275,11 @@ _BORG_PRUNE="${BORG_COMMAND} prune ${OPT_REMOTE} -v --list ${OPT_PROGRESS} ${DRY
# set base path to config directory to keep cache/config separated # set base path to config directory to keep cache/config separated
export BORG_BASE_DIR="${BASE_FOLDER}"; export BORG_BASE_DIR="${BASE_FOLDER}";
# ignore non encrypted access # ignore non encrypted access
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=${_BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK}; export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="${_BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK}";
# ignore moved repo access # ignore moved repo access
export BORG_RELOCATED_REPO_ACCESS_IS_OK=${_BORG_RELOCATED_REPO_ACCESS_IS_OK}; export BORG_RELOCATED_REPO_ACCESS_IS_OK="${_BORG_RELOCATED_REPO_ACCESS_IS_OK}";
# and for debug print that tout # and for debug print that tout
if [ ${DEBUG} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ]; then
echo "export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=${_BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK};"; echo "export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=${_BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK};";
echo "export BORG_RELOCATED_REPO_ACCESS_IS_OK=${_BORG_RELOCATED_REPO_ACCESS_IS_OK};"; echo "export BORG_RELOCATED_REPO_ACCESS_IS_OK=${_BORG_RELOCATED_REPO_ACCESS_IS_OK};";
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
@@ -276,36 +291,34 @@ COMMAND_INFO="${COMMAND_EXPORT}${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY}"
# if this is user@host, we need to use ssh command to verify if the file is there # if this is user@host, we need to use ssh command to verify if the file is there
# else a normal verify is ok # else a normal verify is ok
# unless explicit given, verify is skipped # unless explicit given, verify is skipped
if [ ${VERIFY} -eq 1 ] || [ ${INIT} -eq 1 ]; then # MARK: VERIFY / INFO
if [ "${VERIFY}" -eq 1 ] || [ "${INIT}" -eq 1 ]; then
printf "${PRINTF_SUB_BLOCK}" "VERIFY" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "VERIFY" "$(date +'%F %T')" "${MODULE}";
if [ ! -z "${TARGET_SERVER}" ]; then if [ "${DEBUG}" -eq 1 ]; then
if [ ${DEBUG} -eq 1 ]; then echo "${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY} 2>&1 ";
echo "${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY} 2>&1|grep \"Repository ID:\"";
fi; fi;
# use borg info and verify if it returns "Repository ID:" in the first line # use borg info and verify if it returns "Repository ID:" in the first line
REPO_VERIFY=$(${BORG_COMMAND} info ${OPT_REMOTE} ${REPOSITORY} 2>&1|grep "Repository ID:"); REPO_VERIFY=$(${BORG_COMMAND} info ${OPT_REMOTE} "${REPOSITORY}" 2>&1);
# this is currently a hack to work round the error code in borg info __last_error=$?;
# this checks if REPO_VERIFY holds this error message and then starts init # on any error in verify command force new INIT
if [[ -z "${REPO_VERIFY}" ]] || [[ "${REPO_VERIFY}" =~ ${REGEX_ERROR} ]]; then if [[ $__last_error -ne 0 ]]; then
INIT_REPOSITORY=1; echo "[!] Repository verify error: ${REPO_VERIFY}";
fi;
elif [ ! -d "${REPOSITORY}" ]; then
INIT_REPOSITORY=1; INIT_REPOSITORY=1;
fi; fi;
# if verrify but no init and repo is there but init file is missing set it # if verrify but no init and repo is there but init file is missing set it
if [ ${VERIFY} -eq 1 ] && [ ${INIT} -eq 0 ] && [ ${INIT_REPOSITORY} -eq 0 ] && if [ "${VERIFY}" -eq 1 ] && [ "${INIT}" -eq 0 ] && [ "${INIT_REPOSITORY}" -eq 0 ] &&
[ ! -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then [ ! -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
# write init file # write init file
echo "[!] Add missing init verify file"; echo "[!] Add missing init verify file";
echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_FILE}"; date +%s > "${BASE_FOLDER}${BACKUP_INIT_FILE}";
fi; fi;
# end if verified but repository is not here # end if verified but repository is not here
if [ ${VERIFY} -eq 1 ] && [ ${INIT} -eq 0 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then if [ "${VERIFY}" -eq 1 ] && [ "${INIT}" -eq 0 ] && [ "${INIT_REPOSITORY}" -eq 1 ]; then
echo "[! $(date +'%F %T')] No repository. Please run with -I flag to initialze repository"; echo "[! $(date +'%F %T')] No repository. Please run with -I flag to initialze repository";
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
exit 1; exit 1;
fi; fi;
if [ ${EXIT} -eq 1 ] && [ ${VERIFY} -eq 1 ] && [ ${INIT} -eq 0 ]; then if [ "${EXIT}" -eq 1 ] && [ "${VERIFY}" -eq 1 ] && [ "${INIT}" -eq 0 ]; then
echo "Repository exists"; echo "Repository exists";
echo "For more information run:" echo "For more information run:"
echo "${COMMAND_INFO}"; echo "${COMMAND_INFO}";
@@ -313,16 +326,31 @@ if [ ${VERIFY} -eq 1 ] || [ ${INIT} -eq 1 ]; then
exit; exit;
fi; fi;
fi; fi;
if [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then # MARK: INIT
if [ "${INIT}" -eq 1 ] && [ "${INIT_REPOSITORY}" -eq 1 ]; then
printf "${PRINTF_SUB_BLOCK}" "INIT" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "INIT" "$(date +'%F %T')" "${MODULE}";
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ] || [ "${DRYRUN}" -eq 1 ]; then
echo "${BORG_COMMAND} init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY}"; echo "${BORG_COMMAND} init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY}";
echo "${BORG_COMMAND} key export ${REPOSITORY}";
echo "${BORG_COMMAND} key export --paper ${REPOSITORY}";
fi fi
if [ ${DRYRUN} -eq 0 ]; then if [ "${DRYRUN}" -eq 0 ]; then
# should trap and exit properly here # should trap and exit properly here
${BORG_COMMAND} init ${OPT_REMOTE} -e ${ENCRYPTION} ${OPT_VERBOSE} ${REPOSITORY}; ${BORG_COMMAND} init ${OPT_REMOTE} -e "${ENCRYPTION}" ${OPT_VERBOSE} "${REPOSITORY}";
# show the key file
if [ "${ENCRYPTION}" = "keyfile" ]; then
echo "--- [ENCRYPTION KEY] --[START]-------------------------------------------------->";
echo "Store the key and password in a safe place";
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";borg key export [--paper] ${REPOSITORY}";
echo "----[BORG KEY] -------------------------------->";
${BORG_COMMAND} key export "${REPOSITORY}";
echo "----[BORG KEY:paper] -------------------------->";
${BORG_COMMAND} key export --paper "${REPOSITORY}";
echo "--- [ENCRYPTION KEY] --[END ]-------------------------------------------------->";
fi;
# write init file # write init file
echo "$(date +%s)" > "${BASE_FOLDER}${BACKUP_INIT_FILE}"; date +%s > "${BASE_FOLDER}${BACKUP_INIT_FILE}";
echo "Repository initialized"; echo "Repository initialized";
echo "For more information run:" echo "For more information run:"
echo "${COMMAND_INFO}"; echo "${COMMAND_INFO}";
@@ -330,8 +358,8 @@ if [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 1 ]; then
. "${DIR}/borg.backup.functions.close.sh"; . "${DIR}/borg.backup.functions.close.sh";
# exit after init # exit after init
exit; exit;
elif [ ${INIT} -eq 1 ] && [ ${INIT_REPOSITORY} -eq 0 ]; then elif [ "${INIT}" -eq 1 ] && [ "${INIT_REPOSITORY}" -eq 0 ]; then
echo "[! $(date +'%F %T')] Repository already initialized"; echo "[!] ($(date +'%F %T')) Repository already initialized";
echo "For more information run:" echo "For more information run:"
echo "${COMMAND_INFO}"; echo "${COMMAND_INFO}";
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
@@ -340,25 +368,26 @@ fi;
# verify for init file # verify for init file
if [ ! -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then if [ ! -f "${BASE_FOLDER}${BACKUP_INIT_FILE}" ]; then
echo "[! $(date +'%F %T')] It seems the repository has never been initialized." echo "[!] ($(date +'%F %T')) It seems the repository has never been initialized."
echo "Please run -I to initialize or if already initialzed run with -C for init update." echo "Please run -I to initialize or if already initialzed run with -C for init update."
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
exit 1; exit 1;
fi; fi;
# MARK: LIST / PRINT
# PRINT OUT current data, only do this if REPO exists # PRINT OUT current data, only do this if REPO exists
if [ ${PRINT} -eq 1 ]; then if [ "${PRINT}" -eq 1 ]; then
printf "${PRINTF_SUB_BLOCK}" "PRINT" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "PRINT" "$(date +'%F %T')" "${MODULE}";
FORMAT="{archive:<45} {comment:6} {start} - {end} [{id}] ({username}@{hostname}){NL}" FORMAT="{archive:<45} {comment:6} {start} - {end} [{id}] ({username}@{hostname}){NL}"
# show command on debug or dry run # show command on debug or dry run
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ] || [ "${DRYRUN}" -eq 1 ]; then
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} list ${OPT_REMOTE} --format ${FORMAT} ${REPOSITORY}"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} list ${OPT_REMOTE} --format ${FORMAT} ${REPOSITORY}";
fi; fi;
# run info command if not a dry drun # run info command if not a dry drun
if [ ${DRYRUN} -eq 0 ]; then if [ "${DRYRUN}" -eq 0 ]; then
${BORG_COMMAND} list ${OPT_REMOTE} --format "${FORMAT}" ${REPOSITORY} ; ${BORG_COMMAND} list ${OPT_REMOTE} --format "${FORMAT}" "${REPOSITORY}" ;
fi; fi;
if [ ${VERBOSE} -eq 1 ]; then if [ "${VERBOSE}" -eq 1 ]; then
echo ""; echo "";
echo "Base command info:" echo "Base command info:"
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} [COMMAND] ${OPT_REMOTE} ${REPOSITORY}::[BACKUP] [PATH]"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";${BORG_COMMAND} [COMMAND] ${OPT_REMOTE} ${REPOSITORY}::[BACKUP] [PATH]";
@@ -376,15 +405,22 @@ if [ ${PRINT} -eq 1 ]; then
exit; exit;
fi; fi;
# run borg compact command and exit
if [ "${COMPACT}" -eq 1 ]; then
. "${DIR}/borg.backup.functions.compact.sh";
. "${DIR}/borg.backup.functions.close.sh";
exit;
fi;
# run borg check command and exit # run borg check command and exit
if [ ${CHECK} -eq 1 ]; then if [ "${CHECK}" -eq 1 ]; then
. "${DIR}/borg.backup.functions.check.sh"; . "${DIR}/borg.backup.functions.check.sh";
. "${DIR}/borg.backup.functions.close.sh"; . "${DIR}/borg.backup.functions.close.sh";
exit; exit;
fi; fi;
# DELETE ONE TIME TAG # DELETE ONE TIME TAG
if [ ! -z "${DELETE_ONE_TIME_TAG}" ]; then if [ -n "${DELETE_ONE_TIME_TAG}" ]; then
printf "${PRINTF_SUB_BLOCK}" "DELETE" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "DELETE" "$(date +'%F %T')" "${MODULE}";
# if a "*" is inside we don't do ONE archive, but globbing via -a option # if a "*" is inside we don't do ONE archive, but globbing via -a option
DELETE_ARCHIVE="" DELETE_ARCHIVE=""
@@ -396,12 +432,12 @@ if [ ! -z "${DELETE_ONE_TIME_TAG}" ]; then
DELETE_ARCHIVE="::"${DELETE_ONE_TIME_TAG}; DELETE_ARCHIVE="::"${DELETE_ONE_TIME_TAG};
fi fi
# if this is borg <1.2 OPT_LIST does not work # if this is borg <1.2 OPT_LIST does not work
if [ $(version $BORG_VERSION) -lt $(version "1.2.0") ]; then if [ "$(version "$BORG_VERSION")" -lt "$(version "1.2.0")" ]; then
OPT_LIST=""; OPT_LIST="";
fi; fi;
# if exists, delete and exit # if exists, delete and exit
# show command on debug or dry run # show command on debug or dry run
if [ ${DEBUG} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ]; then
echo "${BORG_COMMAND} delete ${OPT_REMOTE} ${OPT_LIST} -s ${OPT_GLOB} ${REPOSITORY}${DELETE_ARCHIVE}"; echo "${BORG_COMMAND} delete ${OPT_REMOTE} ${OPT_LIST} -s ${OPT_GLOB} ${REPOSITORY}${DELETE_ARCHIVE}";
fi; fi;
# run delete command if not a dry drun # run delete command if not a dry drun
@@ -413,12 +449,12 @@ if [ ! -z "${DELETE_ONE_TIME_TAG}" ]; then
fi; fi;
# if not a dry run, compact repository after delete # if not a dry run, compact repository after delete
# not that compact only works on borg 1.2 # not that compact only works on borg 1.2
if [ $(version $BORG_VERSION) -ge $(version "1.2.0") ]; then if [ "$(version "$BORG_VERSION")" -ge "$(version "1.2.0")" ]; then
if [ ${DRYRUN} -eq 0 ]; then if [ "${DRYRUN}" -eq 0 ]; then
${BORG_COMMAND} compact ${REPOSITORY}; ${BORG_COMMAND} compact ${OPT_REMOTE} "${REPOSITORY}";
fi; fi;
if [ ${DEBUG} -eq 1 ]; then if [ "${DEBUG}" -eq 1 ]; then
echo "${BORG_COMMAND} compact ${REPOSITORY}"; echo "${BORG_COMMAND} compact ${OPT_REMOTE} ${REPOSITORY}";
fi; fi;
fi; fi;
. "${DIR}/borg.backup.functions.close.sh"; . "${DIR}/borg.backup.functions.close.sh";

View File

@@ -2,10 +2,8 @@
# rename to borg.backup.gitea.settings to use # rename to borg.backup.gitea.settings to use
# override settings in borg.backup.settings with SUB_ prefix
# valid for BACKUP_FILE, BACKUP_SET, COMPRESSION*, KEEP_*
GIT_USER=""; GIT_USER="";
GITEA_TMP=""; GITEA_WORKING_DIR="";
GITEA_TEMP_DIR="";
GITEA_BIN=""; GITEA_BIN="";
GITEA_CONFIG=""; GITEA_CONFIG="";

View File

@@ -1,17 +1,21 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
# Backup gitea database, all git folders and gitea settings # Backup gitea database, all git folders and gitea settings
MODULE="gitea" MODULE="gitea"
MODULE_VERSION="1.1.2"; MODULE_VERSION="1.2.2";
DIR="${BASH_SOURCE%/*}" DIR="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
# init system # init system
. "${DIR}/borg.backup.functions.init.sh"; . "${DIR}/borg.backup.functions.init.sh";
# init verify and check file # init verify, compact and check file
BACKUP_INIT_FILE="borg.backup.${MODULE}.init"; BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
BACKUP_CHECK_FILE="borg.backup.${MODULE}.check"; BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
# lock file # lock file
BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock"; BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
@@ -28,9 +32,13 @@ if [ -z "${GIT_USER}" ]; then
GIT_USER="git"; GIT_USER="git";
fi; fi;
# set GITEA_* if not set # set GITEA_* if not set
if [ -z "${GITEA_TMP}" ]; then if [ -z "${GITEA_WORKING_DIR}" ]; then
# run gitea backup (user mktemp?) # run gitea backup (user mktemp?)
GITEA_TMP="/tmp/gitea/"; GITEA_WORKING_DIR="/var/tmp/gitea/";
fi;
# general temp folder for temporary data storage, this is not working output folder
if [ -z "${GITEA_TEMP_DIR}" ]; then
GITEA_TEMP_DIR="/var/tmp";
fi; fi;
if [ -z "${GITEA_BIN}" ]; then if [ -z "${GITEA_BIN}" ]; then
GITEA_BIN="/usr/local/bin/gitea"; GITEA_BIN="/usr/local/bin/gitea";
@@ -38,6 +46,10 @@ fi;
if [ -z "${GITEA_CONFIG}" ]; then if [ -z "${GITEA_CONFIG}" ]; then
GITEA_CONFIG="/etc/gitea/app.ini" GITEA_CONFIG="/etc/gitea/app.ini"
fi; fi;
# This one is not advertised in the config file as it is not recommended to change
if [ -z "${GITEA_EXPORT_TYPE}" ]; then
GITEA_EXPORT_TYPE="zip";
fi;
if [ ! -f "${GITEA_BIN}" ]; then if [ ! -f "${GITEA_BIN}" ]; then
echo "[! $(date +'%F %T')] Cannot find gitea binary"; echo "[! $(date +'%F %T')] Cannot find gitea binary";
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
@@ -48,6 +60,16 @@ if [ ! -f "${GITEA_CONFIG}" ]; then
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
exit 1; exit 1;
fi; fi;
# some basic checks with abort
if [ ! -d "${GITEA_TEMP_DIR}" ]; then
echo "Temp directory does not exist: ${GITEA_TEMP_DIR}";
exit;
fi;
# we should check GITEA_EXPORT_TYPE too at some point for an allow list
# At the moment warn if not zip
if [ "${GITEA_EXPORT_TYPE}" != "zip" ]; then
echo "[!!!!] The gitea export type has been changed from 'zip' to '${GITEA_EXPORT_TYPE}'. This can either break or make exports take very ling";
fi;
# Filename # Filename
FILENAME="gitea.backup.zip"; FILENAME="gitea.backup.zip";
# backup set and prefix # backup set and prefix
@@ -59,7 +81,7 @@ BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e
BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/"); BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/");
printf "${PRINTF_SUB_BLOCK}" "BACKUP: git data and database" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "BACKUP: git data and database" "$(date +'%F %T')" "${MODULE}";
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
echo "sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_TMP} -L -f - | ${BORG_CALL}"; echo "sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_WORKING_DIR} -t ${GITEA_TEMP_DIR} --type ${GITEA_EXPORT_TYPE} -L -f - | ${BORG_CALL}";
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
echo "${BORG_PRUNE}"; echo "${BORG_PRUNE}";
fi; fi;
@@ -69,21 +91,21 @@ if [ ${DRYRUN} -eq 0 ]; then
# below was an old workaround # below was an old workaround
#export USER="${LOGNAME}" # workaround for broken gitea EUID check #export USER="${LOGNAME}" # workaround for broken gitea EUID check
# make sure temp folder is there and is set as git. user # make sure temp folder is there and is set as git. user
if [ ! -d "${GITEA_TMP}" ]; then if [ ! -d "${GITEA_WORKING_DIR}" ]; then
mkdir -p "${GITEA_TMP}"; mkdir -p "${GITEA_WORKING_DIR}";
fi; fi;
chown -R ${GIT_USER}. "${GITEA_TMP}"; chown -R ${GIT_USER}: "${GITEA_WORKING_DIR}";
# this needs to be run in a folder that can be stat by git user # this needs to be run in a folder that can be stat by git user
cd "${GITEA_TMP}"; cd "${GITEA_WORKING_DIR}" || exit 1;
sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_TMP} -L -f - | ${BORG_CALL}; sudo -u ${GIT_USER} ${GITEA_BIN} dump -c ${GITEA_CONFIG} -w ${GITEA_WORKING_DIR} -t ${GITEA_TEMP_DIR} --type ${GITEA_EXPORT_TYPE} -L -f - | ${BORG_CALL};
) | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g' # remove all ESC strings ) 2>&1 | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g' # remove all ESC strings
fi; fi;
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}";
echo "Prune repository with keep${KEEP_INFO:1}"; echo "Prune repository with keep${KEEP_INFO:1}";
${BORG_PRUNE}; ${BORG_PRUNE};
# if this is borg version >1.2 we need to run compact after prune # if this is borg version >1.2 we need to run compact after prune
. "${DIR}/borg.backup.functions.compact.sh"; . "${DIR}/borg.backup.functions.compact.sh" "auto";
# check in auto mode # check in auto mode
. "${DIR}/borg.backup.functions.check.sh" "auto"; . "${DIR}/borg.backup.functions.check.sh" "auto";
fi; fi;

View File

@@ -1,5 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
# Backup MySQL/MariaDB # Backup MySQL/MariaDB
# default is per table dump, can be set to one full dump # default is per table dump, can be set to one full dump
# config override set in borg.backup.mysql.settings # config override set in borg.backup.mysql.settings
@@ -7,7 +10,7 @@
# set last edit date + time # set last edit date + time
MODULE="mysql" MODULE="mysql"
MODULE_VERSION="1.1.1"; MODULE_VERSION="1.1.5";
DIR="${BASH_SOURCE%/*}" DIR="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
@@ -18,8 +21,9 @@ if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
INCLUDE_FILE="borg.backup.${MODULE}.include"; INCLUDE_FILE="borg.backup.${MODULE}.include";
EXCLUDE_FILE="borg.backup.${MODULE}.exclude"; EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
SCHEMA_ONLY_FILE="borg.backup.${MODULE}.schema-only"; SCHEMA_ONLY_FILE="borg.backup.${MODULE}.schema-only";
# init verify and check file # init verify, compact and check file
BACKUP_INIT_FILE="borg.backup.${MODULE}.init"; BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
BACKUP_CHECK_FILE="borg.backup.${MODULE}.check"; BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
# lock file # lock file
BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock"; BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
@@ -61,10 +65,11 @@ if [ -z "${_MYSQL_OK}" ]; then
fi; fi;
# below is for file name only # below is for file name only
# set DB_VERSION (Distrib n.n.n-type) # set DB_VERSION (Distrib n.n.n-type)
# NEW: mysql Ver 15.1 Distrib 10.5.12-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper
# OLD: mysql Ver 14.14 Distrib 5.7.35, for Linux (x86_64) using EditLine wrapper # OLD: mysql Ver 14.14 Distrib 5.7.35, for Linux (x86_64) using EditLine wrapper
# NEW: mysql Ver 15.1 Distrib 10.5.12-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper
# VARIANT: mysql from 11.8.3-MariaDB, client 15.2 for debian-linux-gnu (x86_64) using EditLine wrapper
_DB_VERSION_TYPE=$("${MYSQL_CMD}" --version); _DB_VERSION_TYPE=$("${MYSQL_CMD}" --version);
_DB_VERSION=$(echo "${_DB_VERSION_TYPE}" | sed 's/.*Distrib \([0-9]\{1,\}\.[0-9]\{1,\}\)\.[0-9]\{1,\}.*/\1/'); _DB_VERSION=$(echo "${_DB_VERSION_TYPE}" | sed 's/.*\(Distrib\|from\) \([0-9]\{1,\}\.[0-9]\{1,\}\)\.[0-9]\{1,\}.*/\2/');
DB_VERSION=$(echo "${_DB_VERSION}" | cut -d " " -f 1); DB_VERSION=$(echo "${_DB_VERSION}" | cut -d " " -f 1);
# temporary until correct type detection is set # temporary until correct type detection is set
DB_TYPE="mysql"; DB_TYPE="mysql";
@@ -85,13 +90,14 @@ EVENTDB="mysql"
EVENTS="--events" EVENTS="--events"
# ALL IN ONE FILE or PER DATABASE FLAG # ALL IN ONE FILE or PER DATABASE FLAG
if [ ! -z "${DATABASE_FULL_DUMP}" ]; then if [ -n "${DATABASE_FULL_DUMP}" ]; then
SCHEMA_ONLY=''; SCHEMA_ONLY='';
schema_flag='data'; schema_flag='data';
if [ "${DATABASE_FULL_DUMP}" = "schema" ]; then if [ "${DATABASE_FULL_DUMP}" = "schema" ]; then
SCHEMA_ONLY='--no-data'; SCHEMA_ONLY='--no-data';
schema_flag='schema'; schema_flag='schema';
fi; fi;
LOCAL_START=$(date +'%s');
printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "all databases" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "all databases" "$(date +'%F %T')" "${MODULE}";
# We only do a full backup and not per table backup here # We only do a full backup and not per table backup here
# Filename # Filename
@@ -123,9 +129,12 @@ if [ ! -z "${DATABASE_FULL_DUMP}" ]; then
echo "Prune repository with keep${KEEP_INFO:1}"; echo "Prune repository with keep${KEEP_INFO:1}";
${BORG_PRUNE}; ${BORG_PRUNE};
fi; fi;
DURATION=$(( $(date +'%s') - LOCAL_START ));
printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "all databases" "${MODULE}" "$(convert_time ${DURATION})";
else else
${MYSQL_CMD} ${MYSQL_DB_CONFIG_PARAM} -B -N -e "show databases" | ${MYSQL_CMD} ${MYSQL_DB_CONFIG_PARAM} -B -N -e "show databases" |
while read db; do while read -r db; do
LOCAL_START=$(date +'%s');
printf "${PRINTF_DB_SUB_BLOCK}" "DB" "${db}" "${MODULE}"; printf "${PRINTF_DB_SUB_BLOCK}" "DB" "${db}" "${MODULE}";
printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
# exclude checks # exclude checks
@@ -142,7 +151,7 @@ else
fi; fi;
exclude=0; exclude=0;
if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
while read excl_db; do while read -r excl_db; do
if [ "${db}" = "${excl_db}" ]; then if [ "${db}" = "${excl_db}" ]; then
exclude=1; exclude=1;
break; break;
@@ -172,7 +181,7 @@ else
SCHEMA_ONLY=''; # empty for all SCHEMA_ONLY=''; # empty for all
schema_flag='data'; # or data schema_flag='data'; # or data
if [ -s "${BASE_FOLDER}${SCHEMA_ONLY_FILE}" ]; then if [ -s "${BASE_FOLDER}${SCHEMA_ONLY_FILE}" ]; then
while read schema_db; do while read -r schema_db; do
if [ "${db}" = "${schema_db}" ]; then if [ "${db}" = "${schema_db}" ]; then
SCHEMA_ONLY='--no-data'; SCHEMA_ONLY='--no-data';
schema_flag='schema'; schema_flag='schema';
@@ -212,12 +221,14 @@ else
else else
echo "- [E] ${db}"; echo "- [E] ${db}";
fi; fi;
DURATION=$(( $(date +'%s') - LOCAL_START ));
printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "${db}" "${MODULE}" "$(convert_time ${DURATION})";
done; done;
fi; fi;
# run compact at the end if not a dry run # run compact at the end if not a dry run
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
# if this is borg version >1.2 we need to run compact after prune # if this is borg version >1.2 we need to run compact after prune
. "${DIR}/borg.backup.functions.compact.sh"; . "${DIR}/borg.backup.functions.compact.sh" "auto";
# check in auto mode # check in auto mode
. "${DIR}/borg.backup.functions.check.sh" "auto"; . "${DIR}/borg.backup.functions.check.sh" "auto";
fi; fi;

View File

@@ -9,5 +9,15 @@
# note that with this databases that have been dropped need to be pruned manually # note that with this databases that have been dropped need to be pruned manually
# if 'schema' word is used, only schema data is dumped # if 'schema' word is used, only schema data is dumped
DATABASE_FULL_DUMP=""; DATABASE_FULL_DUMP="";
# override default postgres user # override default postgres user and sudo user for all postgres commands
# All commands must be run as the postgres admin user
DATABASE_USER=""; DATABASE_USER="";
# disable sudo usage by setting to "0", default is to use sudo
DATABASE_USE_SUDO="";
# set the sudo user, if not set postgres or DATABASE_USER is used
DATABASE_SUDO_USER="";
# override port
DATABASE_PORT="";
# override database host, if set to local, localhost or 127.0.0.1 it will use sockets to connect
# for socket connection ident or sudo has to be used
DATABASE_HOST="";

View File

@@ -1,5 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
# Backup PostgreSQL # Backup PostgreSQL
# default is per table dump, can be set to one full dump # default is per table dump, can be set to one full dump
# config override set in borg.backup.pgsql.settings # config override set in borg.backup.pgsql.settings
@@ -7,11 +10,13 @@
# set last edit date + time # set last edit date + time
MODULE="pgsql" MODULE="pgsql"
MODULE_VERSION="1.2.1"; MODULE_VERSION="1.3.0";
DIR="${BASH_SOURCE%/*}" DIR="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
DEFAULT_DATABASE_USE_SUDO="1";
DEFAULT_DATABASE_USER="postgres";
# init system # init system
. "${DIR}/borg.backup.functions.init.sh"; . "${DIR}/borg.backup.functions.init.sh";
@@ -20,8 +25,9 @@ INCLUDE_FILE="borg.backup.${MODULE}.include";
EXCLUDE_FILE="borg.backup.${MODULE}.exclude"; EXCLUDE_FILE="borg.backup.${MODULE}.exclude";
SCHEMA_ONLY_FILE="borg.backup.${MODULE}.schema-only"; SCHEMA_ONLY_FILE="borg.backup.${MODULE}.schema-only";
DATA_ONLY_FILE="borg.backup.${MODULE}.data-only"; DATA_ONLY_FILE="borg.backup.${MODULE}.data-only";
# init verify and check file # init verify, compact and check file
BACKUP_INIT_FILE="borg.backup.${MODULE}.init"; BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
BACKUP_CHECK_FILE="borg.backup.${MODULE}.check"; BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
# lock file # lock file
BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock"; BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
@@ -31,15 +37,41 @@ BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
# if info print info and then abort run # if info print info and then abort run
. "${DIR}/borg.backup.functions.info.sh"; . "${DIR}/borg.backup.functions.info.sh";
if [ ! -z "${DATABASE_USER}" ]; then # set db and sudo user
DB_USER=${DATABASE_USER}; if [ -n "${DATABASE_USER}" ]; then
DB_USER="${DATABASE_USER}";
else else
DB_USER='postgres'; DB_USER="${DEFAULT_DATABASE_USER}";
fi; fi;
if [ -n "${DATABASE_SUDO_USER}" ]; then
DB_SUDO_USER="${DATABASE_SUDO_USER}";
else
DB_SUDO_USER="${DB_USER}";
fi;
# set flag if we should use sudo
if [ -n "${DATABASE_USE_SUDO}" ]; then
USE_SUDO="${DATABASE_USE_SUDO}";
else
USE_SUDO="${DEFAULT_DATABASE_USE_SUDO}";
fi;
# sudo command, or none if not requested
SUDO_COMMAND_A=("sudo" "-u" "${DB_SUDO_USER}");
if [ "${USE_SUDO}" -eq "0" ]; then
SUDO_COMMAND_A=()
fi;
# get current pgsql version first # get current pgsql version first
# if first part <10 then user full, else only first part # if first part <10 then user full, else only first part
# eg 9.4 -> 9.4, 12.5 -> 12 # eg 9.4 -> 9.4, 12.5 -> 12
PG_VERSION=$(pgv=$(psql -U ${DB_USER} -d template1 -t -A -F "," -X -q -c 'select version();' | sed -e 's/^PostgreSQL \([0-9]\{1,\}\.[0-9]\{1,\}\).*/\1/'); if [[ $(echo "${pgv}" | cut -d "." -f 1) -ge 10 ]]; then echo "${pgv}" | cut -d "." -f 1; else echo "${pgv}" | cut -d "." -f 1,2; fi ); PG_VERSION_CMD=("${SUDO_COMMAND_A[@]}" "psql" "-U" "${DB_USER}" "-d" "template1" "-t" "-A" "-F" "," "-X" "-q" "-c" "select version();");
PG_VERSION=$(
pgv=$("${PG_VERSION_CMD[@]}" | sed -e 's/^PostgreSQL \([0-9]\{1,\}\.[0-9]\{1,\}\).*/\1/');
if [[ $(echo "${pgv}" | cut -d "." -f 1) -ge 10 ]]; then
echo "${pgv}" | cut -d "." -f 1;
else
echo "${pgv}" | cut -d "." -f 1,2;
fi;
);
_PATH_PG_VERSION=${PG_VERSION}; _PATH_PG_VERSION=${PG_VERSION};
_backup_error=$?; _backup_error=$?;
if [ $_backup_error -ne 0 ] || [ -z "${PG_VERSION}" ]; then if [ $_backup_error -ne 0 ] || [ -z "${PG_VERSION}" ]; then
@@ -49,76 +81,111 @@ if [ $_backup_error -ne 0 ] || [ -z "${PG_VERSION}" ]; then
fi; fi;
# path set per Distribution type and current running DB version # path set per Distribution type and current running DB version
# Debian/Ubuntu: PG_BASE_PATH='/usr/lib/postgresql/';
# Redhat: PG_BASE_PATH='/usr/pgsql-'; # Redhat: PG_BASE_PATH='/usr/pgsql-';
# AWS 1: PG_BASE_PATH='/usr/lib64/pgsql'; PG_BASE_PATH_LIST=("/usr/lib/postgresql" "/usr/lib64/pgsql");
# Debian: PG_BASE_PATH='/usr/lib/postgresql/'; PG_BASE_PATH="";
PG_BASE_PATH='/usr/lib/postgresql/'; for path in "${PG_BASE_PATH_LIST[@]}"; do
if [ ! -f "${PG_BASE_PATH}${_PATH_PG_VERSION}/bin/psql" ]; then if [ -f "${path}/${_PATH_PG_VERSION}/bin/psql" ]; then
PG_BASE_PATH='/usr/pgsql-'; PG_BASE_PATH="${path}/";
if [ ! -f "${PG_BASE_PATH}${_PATH_PG_VERSION}/bin/psql" ]; then break;
PG_BASE_PATH='/usr/lib64/pgsql'; fi;
_PATH_PG_VERSION=$(echo "${PG_VERSION}" | sed -e 's/\.//'); done;
if [ ! -f "${PG_BASE_PATH}${_PATH_PG_VERSION}/bin/psql" ]; then if [ -z "${PG_BASE_PATH}" ]; then
echo "[! $(date +'%F %T')] PostgreSQL not found in any paths"; echo "[! $(date +'%F %T')] PostgreSQL not found in any paths";
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
exit 1; exit 1;
fi;
fi;
fi; fi;
PG_PATH=${PG_BASE_PATH}${_PATH_PG_VERSION}'/bin/'; PG_PATH="${PG_BASE_PATH}${_PATH_PG_VERSION}/bin/";
PG_PSQL=${PG_PATH}'psql'; PG_PSQL=("${PG_PATH}psql");
PG_DUMP=${PG_PATH}'pg_dump'; PG_DUMP=("${PG_PATH}pg_dump");
PG_DUMPALL=${PG_PATH}'pg_dumpall'; PG_DUMPALL=("${PG_PATH}pg_dumpall");
PG_ERROR=0
# check that command are here # check that command are here
if [ ! -f "${PG_PSQL}" ]; then if [ ! -f "${PG_PSQL[0]}" ]; then
echo "[! $(date +'%F %T')] psql binary not found in ${PG_PATH}"; echo "[!] ($(date +'%F %T')) psql binary not found in ${PG_PATH}";
. "${DIR}/borg.backup.functions.close.sh" 1; PG_ERROR=1;
exit 1;
fi; fi;
if [ ! -f "${PG_DUMP}" ]; then if [ ! -f "${PG_DUMP[0]}" ]; then
echo "[! $(date +'%F %T')] pg_dump binary not found in ${PG_PATH}"; echo "[!] ($(date +'%F %T')) pg_dump binary not found in ${PG_PATH}";
. "${DIR}/borg.backup.functions.close.sh" 1; PG_ERROR=1;
exit 1;
fi; fi;
if [ ! -f "${PG_DUMPALL}" ]; then if [ ! -f "${PG_DUMPALL[0]}" ]; then
echo "[! $(date +'%F %T')] pg_dumpall binary not found in ${PG_PATH}"; echo "[!] ($(date +'%F %T')) pg_dumpall binary not found in ${PG_PATH}";
PG_ERROR=1;
fi;
if [ ${PG_ERROR} -ne 0 ]; then
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
exit 1; exit ${PG_ERROR};
fi;
# prefix with sudo, if sudo command is requested
if [ "${USE_SUDO}" -ne "0" ]; then
PG_PSQL=("${SUDO_COMMAND_A[@]}" "${PG_PSQL[@]}");
PG_DUMP=("${SUDO_COMMAND_A[@]}" "${PG_DUMP[@]}");
PG_DUMPALL=("${SUDO_COMMAND_A[@]}" "${PG_DUMPALL[@]}");
fi; fi;
DB_VERSION=${PG_VERSION}; DB_VERSION=${PG_VERSION};
# TODO override port/host info # override for port or host name
DB_PORT='5432'; if [ -n "${DATABASE_PORT}" ] && [[ "${DATABASE_PORT}" =~ ${REGEX_PORT} ]]; then
DB_HOST='local'; # or <host> DB_PORT="${DATABASE_PORT}";
CONN_DB_HOST=''; # -h <host> else
CONN_DB_PORT=''; # -p <port> DB_PORT="5432";
fi;
if [ -n "${DATABASE_HOST}" ]; then
DB_HOST="${DATABASE_HOST}";
else
DB_HOST="local";
fi;
# use socket for local
if [ "${DB_HOST}" = "local" ] || [ "${DB_HOST}" = "localhost" ] || [ "${DB_HOST}" = "127.0.0.1" ]; then
DB_HOST='local';
CONN_DB_HOST=();
CONN_DB_PORT=();
else
CONN_DB_HOST=("-p" "${DB_HOST}"); # -h <host>
CONN_DB_PORT=("-h" "${DB_HOSTNAME}"); # -p <port>
fi;
# now add user, con, host to all commands
PG_PSQL=("${PG_PSQL[@]}" "-U" "${DB_USER}" "${CONN_DB_HOST[@]}" "${CONN_DB_PORT[@]}");
PG_DUMP=("${PG_DUMP[@]}" "-U" "${DB_USER}" "${CONN_DB_HOST[@]}" "${CONN_DB_PORT[@]}");
PG_DUMPALL=("${PG_DUMPALL[@]}" "-U" "${DB_USER}" "${CONN_DB_HOST[@]}" "${CONN_DB_PORT[@]}");
# ALL IN ONE FILE or PER DATABASE FLAG # ALL IN ONE FILE or PER DATABASE FLAG
if [ ! -z "${DATABASE_FULL_DUMP}" ]; then if [ -n "${DATABASE_FULL_DUMP}" ]; then
SCHEMA_ONLY=''; SCHEMA_ONLY='';
schema_flag='data'; schema_flag='data';
if [ "${DATABASE_FULL_DUMP}" = "schema" ]; then if [ "${DATABASE_FULL_DUMP}" = "schema" ]; then
SCHEMA_ONLY='-s'; SCHEMA_ONLY='-s';
schema_flag='schema'; schema_flag='schema';
fi; fi;
LOCAL_START=$(date +'%s');
printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "all databases" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "all databases" "$(date +'%F %T')" "${MODULE}";
# Filename # Filename
FILENAME-"all.${DB_USER}.NONE.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql" FILENAME="all.${DB_USER}.NONE.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql"
# backup set: # backup set:
BACKUP_SET_PREFIX="${MODULE},all-"; BACKUP_SET_PREFIX="${MODULE},all-";
BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}"; BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
# borg call # borg call
BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/"); BORG_CALL=$(
BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/"); echo "${_BORG_CALL}" |
sed -e "s/##FILENAME##/${FILENAME}/" |
sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/"
);
BORG_PRUNE=$(
echo "${_BORG_PRUNE}" |
sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/"
);
PG_DUMPALL_CMD=("${PG_DUMPALL[@]}" "${SCHEMA_ONLY}" "-c");
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
echo "${PG_DUMPALL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} ${SCHEMA_ONLY} -c | ${BORG_CALL}"; echo "${PG_DUMPALL_CMD[*]} | ${BORG_CALL}";
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
echo "${BORG_PRUNE}"; echo "${BORG_PRUNE}";
fi; fi;
fi; fi;
if [ ${DRYRUN} -eq 0 ]; then if [ ${DRYRUN} -eq 0 ]; then
$(${PG_DUMPALL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} ${SCHEMA_ONLY} -c | ${BORG_CALL}); "${PG_DUMPALL_CMD[@]}" | ${BORG_CALL};
_backup_error=$?; _backup_error=$?;
if [ $_backup_error -ne 0 ]; then if [ $_backup_error -ne 0 ]; then
echo "[! $(date +'%F %T')] Backup creation failed for full dump with error code: ${_backup_error}"; echo "[! $(date +'%F %T')] Backup creation failed for full dump with error code: ${_backup_error}";
@@ -131,10 +198,13 @@ if [ ! -z "${DATABASE_FULL_DUMP}" ]; then
echo "Prune repository with keep${KEEP_INFO:1}"; echo "Prune repository with keep${KEEP_INFO:1}";
${BORG_PRUNE}; ${BORG_PRUNE};
fi; fi;
DURATION=$(( $(date +'%s') - LOCAL_START ));
printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "all databases" "${MODULE}" "$(convert_time ${DURATION})";
else else
# dump globals first # dump globals first
db="pg_globals"; db="pg_globals";
schema_flag="data"; schema_flag="data";
LOCAL_START=$(date +'%s');
printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
# Filename # Filename
FILENAME="${db}.${DB_USER}.NONE.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql" FILENAME="${db}.${DB_USER}.NONE.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql"
@@ -142,17 +212,25 @@ else
BACKUP_SET_PREFIX="${MODULE},${db}-"; BACKUP_SET_PREFIX="${MODULE},${db}-";
BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}"; BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
# borg call # borg call
BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/"); BORG_CALL=$(
BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/"); echo "${_BORG_CALL}" |
sed -e "s/##FILENAME##/${FILENAME}/" |
sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/"
);
BORG_PRUNE=$(
echo "${_BORG_PRUNE}" |
sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/"
);
PG_DUMPALL_CMD=("${PG_DUMPALL[@]}" "--globals-only");
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
echo "${PG_DUMPALL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} --globals-only | ${BORG_CALL}"; echo "${PG_DUMPALL_CMD[*]} | ${BORG_CALL}";
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
echo "${BORG_PRUNE}"; echo "${BORG_PRUNE}";
fi; fi;
fi; fi;
if [ ${DRYRUN} -eq 0 ]; then if [ ${DRYRUN} -eq 0 ]; then
${PG_DUMPALL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} --globals-only | ${BORG_CALL}; "${PG_DUMPALL_CMD[@]}" | ${BORG_CALL};
_backup_error=$?; _backup_error=$?;
if [ $_backup_error -ne 0 ]; then if [ $_backup_error -ne 0 ]; then
echo "[! $(date +'%F %T')] Backup creation failed for ${db} dump with error code: ${_backup_error}"; echo "[! $(date +'%F %T')] Backup creation failed for ${db} dump with error code: ${_backup_error}";
@@ -165,18 +243,25 @@ else
echo "Prune repository with keep${KEEP_INFO:1}"; echo "Prune repository with keep${KEEP_INFO:1}";
${BORG_PRUNE}; ${BORG_PRUNE};
fi; fi;
DURATION=$(( $(date +'%s') - LOCAL_START ));
printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "${db}" "${MODULE}" "$(convert_time ${DURATION})";
# get list of tables # get list of tables
for owner_db in $(${PG_PSQL} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} -d template1 -t -A -F "," -X -q -c "SELECT pg_catalog.pg_get_userbyid(datdba) AS owner, datname, pg_catalog.pg_encoding_to_char(encoding) AS encoding FROM pg_catalog.pg_database WHERE datname "\!"~ 'template(0|1)' ORDER BY datname;"); do GET_DB_CMD=(
"${PG_PSQL[@]}" "-d" "template1" "-t" "-A" "-F" "," "-X" "-q" "-c"
"SELECT pg_catalog.pg_get_userbyid(datdba) AS owner, datname, pg_catalog.pg_encoding_to_char(encoding) AS encoding FROM pg_catalog.pg_database WHERE datname !~ 'template(0|1)' ORDER BY datname;"
);
for owner_db in $("${GET_DB_CMD[@]}"); do
LOCAL_START=$(date +'%s');
# get the user who owns the DB too # get the user who owns the DB too
owner=$(echo ${owner_db} | cut -d "," -f 1); owner=$(echo "${owner_db}" | cut -d "," -f 1);
db=$(echo ${owner_db} | cut -d "," -f 2); db=$(echo "${owner_db}" | cut -d "," -f 2);
encoding=$(echo ${owner_db} | cut -d "," -f 3); encoding=$(echo "${owner_db}" | cut -d "," -f 3);
printf "${PRINTF_DB_SUB_BLOCK}" "DB" "${db}" "${MODULE}"; printf "${PRINTF_DB_SUB_BLOCK}" "DB" "${db}" "${MODULE}";
printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUBEXT_BLOCK}" "BACKUP" "${db}" "$(date +'%F %T')" "${MODULE}";
include=0; include=0;
if [ -s "${BASE_FOLDER}${INCLUDE_FILE}" ]; then if [ -s "${BASE_FOLDER}${INCLUDE_FILE}" ]; then
while read incl_db; do while read -r incl_db; do
if [ "${db}" = "${incl_db}" ]; then if [ "${db}" = "${incl_db}" ]; then
include=1; include=1;
break; break;
@@ -187,7 +272,7 @@ else
fi; fi;
exclude=0; exclude=0;
if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then if [ -f "${BASE_FOLDER}${EXCLUDE_FILE}" ]; then
while read excl_db; do while read -r excl_db; do
if [ "${db}" = "${excl_db}" ]; then if [ "${db}" = "${excl_db}" ]; then
exclude=1; exclude=1;
break; break;
@@ -195,17 +280,15 @@ else
done<"${BASE_FOLDER}${EXCLUDE_FILE}"; done<"${BASE_FOLDER}${EXCLUDE_FILE}";
fi; fi;
if [ ${include} -eq 1 ] && [ ${exclude} -eq 0 ]; then if [ ${include} -eq 1 ] && [ ${exclude} -eq 0 ]; then
PG_DUMP_CMD=("${PG_DUMP[@]}" "-c" "--format=c");
# set dump type # set dump type
SCHEMA_ONLY='';
schema_flag=''; # schema or data schema_flag=''; # schema or data
# schema exclude over data exclude, can't have both # schema exclude over data exclude, can't have both
if [ -s "${BASE_FOLDER}${SCHEMA_ONLY_FILE}" ]; then if [ -s "${BASE_FOLDER}${SCHEMA_ONLY_FILE}" ]; then
# default is data dump # default is data dump
SCHEMA_ONLY='';
schema_flag='data'; schema_flag='data';
while read schema_db; do while read -r schema_db; do
if [ "${db}" = "${schema_db}" ]; then if [ "${db}" = "${schema_db}" ]; then
SCHEMA_ONLY='-s';
schema_flag='schema'; schema_flag='schema';
# skip out # skip out
break; break;
@@ -213,11 +296,9 @@ else
done<"${BASE_FOLDER}${SCHEMA_ONLY_FILE}"; done<"${BASE_FOLDER}${SCHEMA_ONLY_FILE}";
elif [ -s "${BASE_FOLDER}${DATA_ONLY_FILE}" ]; then elif [ -s "${BASE_FOLDER}${DATA_ONLY_FILE}" ]; then
# default to schema, unless in data list # default to schema, unless in data list
SCHEMA_ONLY='-s';
schema_flag='schema'; schema_flag='schema';
while read data_db; do while read -r data_db; do
if [ "${db}" = "${data_db}" ]; then if [ "${db}" = "${data_db}" ]; then
SCHEMA_ONLY='';
schema_flag='data'; schema_flag='data';
# skip out # skip out
break; break;
@@ -226,9 +307,13 @@ else
fi; fi;
# if nothing is set, default to data # if nothing is set, default to data
if [ -z "${schema_flag}" ]; then if [ -z "${schema_flag}" ]; then
SCHEMA_ONLY=''
schema_flag="data"; schema_flag="data";
fi; fi;
# set schema flag if schmea only is requested
if [ $schema_flag = "schema" ]; then
PG_DUMP_CMD+=("-s");
fi;
PG_DUMP_CMD+=("${db}");
# Filename # Filename
# Database.User.Encoding.pgsql|data|schema-Version_Host_Port_YearMonthDay_HourMinute_Counter.Fromat(c).sql # Database.User.Encoding.pgsql|data|schema-Version_Host_Port_YearMonthDay_HourMinute_Counter.Fromat(c).sql
FILENAME="${db}.${owner}.${encoding}.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql" FILENAME="${db}.${owner}.${encoding}.${schema_flag}-${DB_VERSION}_${DB_HOST}_${DB_PORT}.c.sql"
@@ -237,18 +322,25 @@ else
# backup set: # backup set:
BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}"; BACKUP_SET_NAME="${ONE_TIME_TAG}${BACKUP_SET_PREFIX}${schema_flag}-${BACKUP_SET}";
# borg call # borg call
BORG_CALL=$(echo "${_BORG_CALL}" | sed -e "s/##FILENAME##/${FILENAME}/" | sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/"); BORG_CALL=$(
echo "${_BORG_CALL}" |
sed -e "s/##FILENAME##/${FILENAME}/" |
sed -e "s/##BACKUP_SET##/${BACKUP_SET_NAME}/"
);
# borg prune # borg prune
BORG_PRUNE=$(echo "${_BORG_PRUNE}" | sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/"); BORG_PRUNE=$(
echo "${_BORG_PRUNE}" |
sed -e "s/##BACKUP_SET_PREFIX##/${BACKUP_SET_PREFIX}/"
);
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";"; echo "export BORG_BASE_DIR=\"${BASE_FOLDER}\";";
echo "${PG_DUMP} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} -c ${SCHEMA_ONLY} --format=c ${db} | ${BORG_CALL}"; echo "${PG_DUMP_CMD[*]} | ${BORG_CALL}";
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
echo "${BORG_PRUNE}"; echo "${BORG_PRUNE}";
fi; fi;
fi; fi;
if [ ${DRYRUN} -eq 0 ]; then if [ ${DRYRUN} -eq 0 ]; then
${PG_DUMP} -U ${DB_USER} ${CONN_DB_HOST} ${CONN_DB_PORT} -c ${SCHEMA_ONLY} --format=c ${db} | ${BORG_CALL}; "${PG_DUMP_CMD[@]}" | ${BORG_CALL};
_backup_error=$?; _backup_error=$?;
if [ $_backup_error -ne 0 ]; then if [ $_backup_error -ne 0 ]; then
echo "[! $(date +'%F %T')] Backup creation failed for ${db} dump with error code: ${_backup_error}"; echo "[! $(date +'%F %T')] Backup creation failed for ${db} dump with error code: ${_backup_error}";
@@ -264,12 +356,14 @@ else
else else
echo "- [E] ${db}"; echo "- [E] ${db}";
fi; fi;
DURATION=$(( $(date +'%s') - LOCAL_START ));
printf "${PRINTF_DB_RUN_TIME_SUB_BLOCK}" "DONE" "${db}" "${MODULE}" "$(convert_time ${DURATION})";
done; done;
fi; fi;
# run compact at the end if not a dry run # run compact at the end if not a dry run
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
# if this is borg version >1.2 we need to run compact after prune # if this is borg version >1.2 we need to run compact after prune
. "${DIR}/borg.backup.functions.compact.sh"; . "${DIR}/borg.backup.functions.compact.sh" "auto";
# check in auto mode # check in auto mode
. "${DIR}/borg.backup.functions.check.sh" "auto"; . "${DIR}/borg.backup.functions.check.sh" "auto";
fi; fi;

View File

@@ -13,33 +13,43 @@ TARGET_PORT="";
TARGET_BORG_PATH=""; TARGET_BORG_PATH="";
# folder where the backup folder will be created # folder where the backup folder will be created
TARGET_FOLDER=""; TARGET_FOLDER="";
# the backup file (folder) for this host $(hostname), must end with .borg # the backup file (folder) for this backup, avoid dynamic variables in here, must end with .borg
BACKUP_FILE=""; BACKUP_FILE="";
# compression settings (empty for none, lz4, zstd, zlib, lzma) # compression settings (none, lz4, zstd, zlib, lzma)
# level, if empty then default, else number between 0 and 9, or 1 to 22 for zstd # empty is default zstd
# default is zstd, 3
COMPRESSION=""; COMPRESSION="";
# compression level, if empty then default 3, else number between 0 and 9, or 1 to 22 for zstd
COMPRESSION_LEVEL=""; COMPRESSION_LEVEL="";
# encryption settings: # encryption settings:
# SHA-256: 'none', 'authenticated', 'repokey', 'keyfile' # SHA-256: 'none', 'authenticated', 'repokey', 'keyfile'
# BLAKE2b: 'authenticated-blake2', 'repokey-blake2', 'keyfile-blake2' # BLAKE2b: 'authenticated-blake2', 'repokey-blake2', 'keyfile-blake2'
# Note: none or empty does not encrypt # Note: none does not encrypt and is not recommended
# Blank passwords allowed for only key (if used, use keyfile) # Default is keyfile
# Blank passwords allowed for only keyfile, else passwords have to be set via BORG_PASSPHRASE or BORG_PASSCOMMAND
# See: http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically # See: http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically
ENCRYPTION=""; ENCRYPTION=""
# force repository verirfy, default is off, set to true for verify on every run # force repository verify, default is off, set to true for verify on every run
FORCE_VERIFY=""; FORCE_VERIFY="";
# compact interval, only if using borg 1.2 or higher
# after how many days to run compress on data
# default is 1 for run after each prune
# set to 0 or negative to turn off
# or any other value for every n days
COMPACT_INTERVAL="";
# check interval, if 0 or negative number, no check will ever run # check interval, if 0 or negative number, no check will ever run
# if empty fall back to default set # if empty fall back to default set
# if set to 1 then every time the script runs # if set to 1 then every time the script runs
# any other value it means ever n days, eg 90 would be every 90 days # any other value it means ever n days, eg 90 would be every 90 days
CHECK_INTERVAL=""; CHECK_INTERVAL="";
# default is %Y-%m-%d # default is {now:%Y-%m-%d}
# todays date, if more than one per day add -%H%M for hour/minute # todays date, if more than one per day add -%H%M for hour/minute
# it can also be "{hostname}-{user}-{now:%Y-%m-%dT%H:%M:%S.%f}" # it can also be "{hostname}-{user}-{now:%Y-%m-%dT%H:%M:%S.%f}"
BACKUP_SET=""; BACKUP_SET="";
# prune times, how many are kept in each time frame # prune times, how many are kept in each time frame
KEEP_LAST="";
KEEP_HOURS="";
KEEP_DAYS=""; KEEP_DAYS="";
KEEP_WEEKS=""; KEEP_WEEKS="";
KEEP_MONTHS=""; KEEP_MONTHS="";
KEEP_YEARS=""; KEEP_YEARS="";
KEEP_WITHIN="";

View File

@@ -15,3 +15,5 @@ ZABBIX_DATABASE="";
ZABBIX_CONFIG=""; ZABBIX_CONFIG="";
# unknown tables, default ignore, is set to true, will force backup them # unknown tables, default ignore, is set to true, will force backup them
ZABBIX_UNKNOWN_TABLES=""; ZABBIX_UNKNOWN_TABLES="";
# overide zabbix db port read from zabbix config, if not set read from zabbix config
ZABBIX_DB_PORT="";

View File

@@ -1,17 +1,21 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# allow variables in printf format string
# shellcheck disable=SC2059
# Backup zabbix config and settings only # Backup zabbix config and settings only
MODULE="zabbix" MODULE="zabbix"
MODULE_VERSION="1.1.2"; MODULE_VERSION="1.1.4";
DIR="${BASH_SOURCE%/*}" DIR="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
# init system # init system
. "${DIR}/borg.backup.functions.init.sh"; . "${DIR}/borg.backup.functions.init.sh";
# init verify and check file # init verify, compact and check file
BACKUP_INIT_FILE="borg.backup.${MODULE}.init"; BACKUP_INIT_FILE="borg.backup.${MODULE}.init";
BACKUP_COMPACT_FILE="borg.backup.${MODULE}.compact";
BACKUP_CHECK_FILE="borg.backup.${MODULE}.check"; BACKUP_CHECK_FILE="borg.backup.${MODULE}.check";
# lock file # lock file
BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock"; BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
@@ -25,7 +29,7 @@ BACKUP_LOCK_FILE="borg.backup.${MODULE}.lock";
if [ -z "${ZABBIX_DUMP_BIN}" ]; then if [ -z "${ZABBIX_DUMP_BIN}" ]; then
ZABBIX_DUMP_BIN="/usr/local/bin/zabbix-dump"; ZABBIX_DUMP_BIN="/usr/local/bin/zabbix-dump";
fi; fi;
if [ ! -z "${ZABBIX_CONFIG}" ] && [ ! -f "${ZABBIX_CONFIG}" ]; then if [ -n "${ZABBIX_CONFIG}" ] && [ ! -f "${ZABBIX_CONFIG}" ]; then
echo "[! $(date +'%F %T')] Cannot find zabbix config: ${ZABBIX_CONFIG}"; echo "[! $(date +'%F %T')] Cannot find zabbix config: ${ZABBIX_CONFIG}";
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
exit 1; exit 1;
@@ -36,6 +40,10 @@ fi;
if [ "${ZABBIX_DATABASE}" = "psql" ]; then if [ "${ZABBIX_DATABASE}" = "psql" ]; then
OPT_ZABBIX_DUMP="-C"; OPT_ZABBIX_DUMP="-C";
fi; fi;
OPT_ZABBIX_DB_PORT="";
if [ -n "${ZABBIX_DB_PORT}" ]; then
OPT_ZABBIX_DB_PORT="-P ${ZABBIX_DB_PORT}";
fi;
if [ "${ZABBIX_DATABASE}" != "psql" ] && [ "${ZABBIX_DATABASE}" != "mysql" ]; then if [ "${ZABBIX_DATABASE}" != "psql" ] && [ "${ZABBIX_DATABASE}" != "mysql" ]; then
echo "[! $(date +'%F %T')] Zabbix dump must have database set to either psql or mysql"; echo "[! $(date +'%F %T')] Zabbix dump must have database set to either psql or mysql";
. "${DIR}/borg.backup.functions.close.sh" 1; . "${DIR}/borg.backup.functions.close.sh" 1;
@@ -69,19 +77,19 @@ fi;
printf "${PRINTF_SUB_BLOCK}" "BACKUP: zabbix settings" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "BACKUP: zabbix settings" "$(date +'%F %T')" "${MODULE}";
if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then if [ ${DEBUG} -eq 1 ] || [ ${DRYRUN} -eq 1 ]; then
echo "${ZABBIX_DUMP_BIN} -t ${ZABBIX_DATABASE} ${OPT_ZABBIX_UNKNOWN_TABLES} ${OPT_ZABBIX_DUMP} ${OPT_ZABBIX_CONFIG} -o - | ${BORG_CALL}" echo "${ZABBIX_DUMP_BIN} ${OPT_ZABBIX_DB_PORT} -t ${ZABBIX_DATABASE} ${OPT_ZABBIX_UNKNOWN_TABLES} ${OPT_ZABBIX_DUMP} ${OPT_ZABBIX_CONFIG} -o - | ${BORG_CALL}"
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
echo "${BORG_PRUNE}"; echo "${BORG_PRUNE}";
fi; fi;
fi; fi;
if [ ${DRYRUN} -eq 0 ]; then if [ ${DRYRUN} -eq 0 ]; then
${ZABBIX_DUMP_BIN} -t ${ZABBIX_DATABASE} ${OPT_ZABBIX_UNKNOWN_TABLES} ${OPT_ZABBIX_DUMP} ${OPT_ZABBIX_CONFIG} -o - | ${BORG_CALL}; ${ZABBIX_DUMP_BIN} ${OPT_ZABBIX_DB_PORT} -t ${ZABBIX_DATABASE} ${OPT_ZABBIX_UNKNOWN_TABLES} ${OPT_ZABBIX_DUMP} ${OPT_ZABBIX_CONFIG} -o - | ${BORG_CALL};
fi; fi;
if [ -z "${ONE_TIME_TAG}" ]; then if [ -z "${ONE_TIME_TAG}" ]; then
printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}"; printf "${PRINTF_SUB_BLOCK}" "PRUNE" "$(date +'%F %T')" "${MODULE}";
${BORG_PRUNE}; ${BORG_PRUNE};
# if this is borg version >1.2 we need to run compact after prune # if this is borg version >1.2 we need to run compact after prune
. "${DIR}/borg.backup.functions.compact.sh"; . "${DIR}/borg.backup.functions.compact.sh" "auto";
# check in auto mode # check in auto mode
. "${DIR}/borg.backup.functions.check.sh" "auto"; . "${DIR}/borg.backup.functions.check.sh" "auto";
fi; fi;

View File

@@ -33,7 +33,7 @@ function usage ()
} }
# set options # set options
while getopts ":c:m:uf:h" opt do while getopts ":c:m:uf:h" opt; do
case "${opt}" in case "${opt}" in
c|config) c|config)
BASE_FOLDER=${OPTARG}; BASE_FOLDER=${OPTARG};