the backup command

[pascal]

Hello,

today I backup all my siteworx accounts in one shot on a NAS storage by using the interworx-cp backup command


/home/interworx/bin/backup.pex --domains=all --scp --transfer-ini=/root/scripts/xxx/transfer.ini --email=email@domain.com


(I also use a sftp cmd to delete all existing backup file on the NAS storage, like this I have only the latest backup, but anyway....)

My problem is that, yet, the backup process run during 2 hours. I have more than 120 accounts to backup

Is it possible to use an asterix (*) in the --domains= ?

My idea is to run this backup every hours and to backup all the domains that begin by a letter every hours

For example if I could do something like this

/home/interworx/bin/backup.pex --domains=a* b* --scp --transfer-ini=/root/scripts/xxx/transfer.ini --email=email@domain.com


I could backup all the domains with a repartition of the backup during all the day (all domain that begin by these 2 letters per hour)

Is it possible ?

Is there an other solution ?

Thanks

Pascal

[pascal]

Hello,

It becomes urgent for me to have an answer.

Indeed, now the backup process in one shot take more than 3 hours and half and during this time the usage of the box is really bad. The cpu usage is near 80%, the load average reach 5 etc....

I absolutly need to find a solution to backup the siteworx account but not in one shot

Thanks to tell me if may use something like --domains=a* b*

any others users experiences about this are really welcome

Thanks

Pascal

[IWorx-Chris]

Pascal,

You can backup all domains, a single domain or a comma separated list of domains. The "a*, b*" can be handled by shell globbing in your backup script to expand out the domains themselves or via some other means scripted by you.

Hope this helps.

Chris

[pascal]

Pascal,

You can backup all domains, a single domain or a comma separated list of domains. The "a*, b*" can be handled by shell globbing in your backup script to expand out the domains themselves or via some other means scripted by you.

Hope this helps.

Chris

Oups...

Really sorry Chris but my english is so poor.
What do yu mean by

shell globbing in your backup script to expand out the domains themselves


You mean I'll have to add in my script something to manage the "A*" ?

Something like :

PATH2CONF=/etc/httpd/conf.d
RSLT=`ls -1 domain.com.conf_to_ignore /$PATH2CONF/a*`

for tobackup in $RSLT; do
Remove the .conf
launch the backup command
done


I don't know if it is the best solution. How to remove the .conf and I don't know how to test the current hour. I mean the best way would be to do soemthing like

CAse current hour = 01 then DOM = a*
Case current hour = 02 then DOM = b*
etc ...
then have a generic
RSLT=`ls -1 domain.com.conf_to_ignore /$PATH2CONF/$DOM`


Is seome Interworx customers did soemthing like this ?
How Interworx customers backup ALL their siteworx accounts ?
I can't continue backup them in one shot, so what couzld be the best solutions to backups all siteworx account ?

Thanks for your advises and comment about this subject

Pascal

[IWorx-Socheat]

This may be a little more useful in generating the domains you need:

find /home/*/var/a* -type d -maxdepth 0 -mindepth 0
find /home/*/var/b* -type d -maxdepth 0 -mindepth 0
...
find /home/*/var/i* -type d -maxdepth 0 -mindepth 0 | grep -v "interworx"
etc, etc

This will search the /home directories for subdirectories in var starting with a, b, etc. The grep -v "interworx" is to exclude over the "interworx" directory from the results when doing "i*", since you aren't interested in the InterWorx directory.

I only quickly tested this on one of our smaller boxes, so it may need some tweaking.

[pascal]

Cool so I may do something like

HOUR = date +%H

case $HOUR in
00)
DOM = a*
IGNORE =
;;
01)
DOM = i*
IGNORE = "interworx"
;;
...
...
23)
DOM = z*
;;

esac

RSLT = find /home/*/var/$DOM -type d -maxdepth 0 -mindepth 0 | grep -V $IGNORE

for TOBACKUP in $RSLT; do
backup.pex --domain=$TOBACKUP
done


I'll test this

heu ??? just a question what about multiple PATTERNS in $IGNORE for the grep -v
is it possible to have more than one pattern ?

Thanks Socheat

Pascal

[IWorx-Socheat]

You'll need to do a little more parsing than that. The output of those find commands will be full paths. e.g.:

/home/abccom/var/abc.com
/home/mydomain/var/mydomain.com
/home/myother/var/myotherdomain.com
etc,etc

You'll need to use something like awk to just grab the domain name.

[pascal]

I was trying something like


#!/bin/bash
PATH2CONF=/etc/httpd/conf.d
HOUR=$(date +%H)
IGNORE=--igonre php.conf --ignore iworx.conf --ignore watch.conf --ignore ssl.conf --ignore security.conf --ignore namevirtualhost.conf --ignore directoryindex.conf

case $HOUR in
00)
sftp -b /root/sftpfile -o IdentityFile=/root/key/padawan squalito@storage1.sagonet.com ;; #to delete previous day backup files
01)
DOM=a*
DOM2=b*
DOM3=u*
;;
02)
DOM=c*
DOM2=d*
DOM3=o*
;;
03)
DOM=e*
DOM2=f*
DOM3=p*
;;
04)
DOM=s*
DOM2=t*
DOM3=x*
;;
05)
DOM=i*
DOM2=j*
DOM3=q*
;;
06)
DOM=k*
DOM2=l*
DOM3=r*
;;
07)
DOM=m*
DOM2=n*
;;
15)
DOM=v*
DOM2=w*
;;
16)
DOM=y*
DOM2=z*
;;
17)
DOM=g*
DOM2=h*
;;

esac


RSLT=`ls -1 $IGNORE /$PATH2CONF/$DOM`

for TOBACKUP in $RSLT; do
DOMTOBACKUP = ${TOBACKUP%.conf$}
/home/interworx/bin/backup.pex --domains=$DOMTOBACKUP --scp --transfer-ini=/root/transfer.ini #to store new backup files
done

RSLT=`ls -1 $IGNORE /$PATH2CONF/$DOM2`

for TOBACKUP in $RSLT; do
DOMTOBACKUP = ${TOBACKUP%.conf$}
/home/interworx/bin/backup.pex --domains=$DOMTOBACKUP --scp --transfer-ini=/root/transfer.ini #to store new backup files
done

RSLT=`ls -1 $IGNORE /$PATH2CONF/$DOM3`

for TOBACKUP in $RSLT; do
DOMTOBACKUP = ${TOBACKUP%.conf$}
/home/interworx/bin/backup.pex --domains=$DOMTOBACKUP --scp --transfer-ini=/root/transfer.ini #to store new backup files
done


but $DOMTOBACKUP was empty (it was not finished of course...)

[pascal]

Ok

yet I did this


#!/bin/bash
PATH2CONF=/etc/httpd/conf.d
cd $PATH2CONF
HOUR=$(date +%H)
#IGNORE=`php.conf --ignore iworx.conf --ignore watch.conf --ignore ssl.conf --ignore security.conf --ignore namevirtualhost.conf --ignore directoryindex.conf`

case $HOUR in
00)
sftp -b /root/sftpfile -o IdentityFile=/root/key/padawan squalito@storage1.sagonet.com ;; #to delete previous day backup files
01)
DOM=a*
DOM2=b*
DOM3=u*
;;
02)
DOM=c*
DOM2=d*
DOM3=o*
;;
03)
DOM=e*
DOM2=f*
DOM3=p*
;;
04)
DOM=s*
DOM2=t*
DOM3=x*
;;
05)
DOM=i*
DOM2=j*
DOM3=q*
;;
06)
DOM=k*
DOM2=l*
DOM3=r*
;;
07)
DOM=m*
DOM2=n*
;;
15)
DOM=v*
DOM2=w*
;;
16)
DOM=y*
DOM2=z*
;;
17)
DOM=g*
DOM2=h*
;;

esac


RSLT=`ls -1 --ignore php.conf --ignore iworx.conf --ignore watch.conf --ignore ssl.conf --ignore security.conf --ignore namevirtualhost.conf --ignore directoryindex.conf $DOM`

for TOBACKUP in $RSLT; do
DOMTOBACKUP= echo {TOBACKUP%.conf$}
/home/interworx/bin/backup.pex --domains=$DOMTOBACKUP --scp --transfer-ini=/root/transfer.ini #to store new backup files
done

RSLT=`ls -1 --ignore php.conf --ignore iworx.conf --ignore watch.conf --ignore ssl.conf --ignore security.conf --ignore namevirtualhost.conf --ignore directoryindex.conf $DOM2`

for TOBACKUP in $RSLT; do
DOMTOBACKUP= echo ${TOBACKUP%.conf$}
/home/interworx/bin/backup.pex --domains=$DOMTOBACKUP --scp --transfer-ini=/root/transfer.ini #to store new backup files
done

RSLT=`ls -1 --ignore php.conf --ignore iworx.conf --ignore watch.conf --ignore ssl.conf --ignore security.conf --ignore namevirtualhost.conf --ignore directoryindex.conf $DOM3`

for TOBACKUP in $RSLT; do
DOMTOBACKUP=echo ${TOBACKUP%.conf$}
/home/interworx/bin/backup.pex --domains=$DOMTOBACKUP --scp --transfer-ini=/root/transfer.ini #to store new backup files
done


but I have an empty DOMTOBACKUP

the result is


script begin
backup.php --domains= --scp --transfer-ini=/root/transfer.ini
backup filename was not provided
SiteWorx account for '--scp' does not exist!
script end

hebergement-siteweb.fr.conf

script begin
backup.php --domains= --scp --transfer-ini=/root/transfer.ini
backup filename was not provided
SiteWorx account for '--scp' does not exist!
script end

histo1956.com.conf

script begin
backup.php --domains= --scp --transfer-ini=/root/transfer.ini
backup filename was not provided
SiteWorx account for '--scp' does not exist!
script end

hoopsleagues.com.conf
etc....


so my pbm is here


DOMTOBACKUP= echo {TOBACKUP%.conf$}


do you know how to remove the .conf at the end of domaine.com.conf

Thanks

Pascal

[IWorx-Socheat]

Check out 'basename':

NAME
basename - strip directory and suffix from filenames

SYNOPSIS
basename NAME [SUFFIX]
basename OPTION

DESCRIPTION
Print NAME with any leading directory components removed. If speci-
fied, also remove a trailing SUFFIX.

[pascal]

Thanks Socheat

But the pbm is that command PRINT the result

So I can't do something like
DOMTOBACKUP=basename $TOBACKUP .conf

Is somebody know how to redirect the print command of basename in a var ?

erff I'm so bad with awk. I'm pretty sure it should be very easy to remove .conf at the end of the VAR with awk

I'll search

Anyway, thanks a ton Socheat for your help

Pascal

[IWorx-Socheat]

You can do this:


DOMTOBACKUP=$(basename $TOBACKUP .conf)


The $() tells bash to execute the command between the parenthesis first, before assigning it to the DOMTOBACKUP variable.

[CMI]

Here is what I did.
#!/bin/sh

BASE="/root/backup"
BACKUP=/home/interworx/bin/backup.pex

cd $BASE

datestring=$(date +%F)

# Backup *all* domains and snarf backup file names
backupfiles=$($BACKUP --siteworx --domains=all | grep "Backup File: " | cut -d " " -f 3 )

# Remove currently stored backups and copy in backup files
rm -f /root/backup/domains/*
for file in $backupfiles; do
mv $file /root/backup/domains/
done

I'm reluctant to keep extra copies (although I want to) because there was recently a huge jump in the combined size of backups.

Seems to work OK but I have yet to try a restore.

[IWorx-Socheat]

I'm reluctant to keep extra copies (although I want to) because there was recently a huge jump in the combined size of backups.


CMI, can you clarify what you mean by this?

[CMI]

CMI, can you clarify what you mean by this?

I mean I used to keep about 2 months of backups from user's and IWorx itself, but recently the box got clogged with backups more quickly than it should. Someone its probably eating more diskspace, so keeping too many copies of backups is going to cause problems again.

I'll likley save backups in folders based on the current week number, keeping the current and previous week, and deleting prior ones. Hopefuly that will keep things reasonable.

That reminds me... I need to go add another feature request :)