Migration questions

[timryberg]

I am moving my server from a P4 2.0 with InterWorx to a P4 3.0 with InterWorx. So many other people are looking at doing this I will be authoring a Wite Paper on my experience to assist others.

I have made a copy of the file system of my old server on that server's secondary hard drive and had that drive transferred to the new server (which also comes with a secondary hard drive). This way I know I have ALL of the files I will need.

Per Chris's suggestion I am in the process of re-creating my SiteWorx accounts on the new server rather than trying to "restore" InterWorx since he indicated the probable existance of database errors in the process..

Basically I have two Mozilla tabs open (one for each server) to the NodeWorx => SiteWorx screen. I am creating these in the (reverse or the) same order listed on the old server since this is the order that they were originally created in and want to make sure that the system user for each SiteWprx accunt (and hopefully the UID and GID) will be the same.

After this I will go through the domains that have websites or email (a number of them are just "parked" so this isn't all of them) and writing down the email boxex and passwords, subdomains, pointer domains, databases and database users.

Is there an easy way to back up the database users and their passwords or do I need to re-create them from scratch? The reason I ask is that I didn't use the same database password for each database and InterWorx does not display the passwords in the interface. I have a lot of PHP scripts (phpBB and others) installed on the server and the only way I know to find out these paswords is to look at each script's configuration file or use new passwords and manually edit the config file. Is there an easier way to do this?

Also, when I get to moving my files over what attributes to the CP command do I need to preserve a files's readable/writable status?

I am sure I will have other questions as I go further in this.

[IWorx-Chris]

Basically I have two Mozilla tabs open (one for each server) to the NodeWorx => SiteWorx screen. I am creating these in the (reverse or the) same order listed on the old server since this is the order that they were originally created in and want to make sure that the system user for each SiteWprx accunt (and hopefully the UID and GID) will be the same.


Actually, tar stores the username in the archive and not the numeric UID/GID, which is good :). As long as the usernames are the same (and they should be) you'll be fine here.


After this I will go through the domains that have websites or email (a number of them are just "parked" so this isn't all of them) and writing down the email boxex and passwords, subdomains, pointer domains, databases and database users.


I apologize in advance for this Tim, and yes, that's how it's done now. This will be fixed "soon" as in the next release if we can swing it. A SiteWorx backup -> restore is coming that will do this for you.

And now that I saw this let me see if we can get you a BETA version of this script now, to save you the headache.

Let me see where it's at and I'll post / contact you today if I can.

Chris

[timryberg]

Thanks Chris. Like I said this is going to be a headache. I have about 15 of these scripts (phpbb, phpnuke, etc.) installed on the server and while only a couple of them are "critical" to get up and running now, that's still a lot of work.

[IWorx-Socheat]

Hi Tim,

If you give me your box information, I'll install the backup/restore script we've written to handle backing up and migrating Siteworx accounts, as well as give you instructions on how it's used. It's only been tested between internal Siteworx boxes, but it has been working well in the tests we've run. Thanks for testing it out for us. :)

Socheat

[IWorx-Socheat]

Actually, it would be better if you opened a support ticket and supplied your information there instead.

[timryberg]

Actually, it would be better if you opened a support ticket and supplied your information there instead.

Done. I assume you need info for both boxes. It's no problem doing a test for you. In fact it's you who are helping me out.

Tim

[pascal]

Hello,

I know that you do not give ETA information about your releases but as I just reveived my root acces for my new Centos/interworx 1.8.0 server, I'd like to know if I begin the migration without your Backup/restore release or if I wait for it (what it should be more easy)

Tim :
Finally, what did you use ? tar, cp, rsync ? and could you give me the list of folders you copied to able me to compare the list I'm planning to backup (just to be sure I do not forget something)
A last thing, did they migrate the old IP to your new server ? or do yu have new IP ?

Thanks
Pascal

[IWorx-Chris]

We're still working with Tim on this Pascal and if the backup/restore script works (it is still very beta) then we'll of course let you use it as well. We should know more today or tomorrow Pascal and we'll update this thread with how things go.

Chris

[timryberg]

Tim :
Finally, what did you use ? tar, cp, rsync ? and could you give me the list of folders you copied to able me to compare the list I'm planning to backup (just to be sure I do not forget something)
A last thing, did they migrate the old IP to your new server ? or do yu have new IP ?

I used tar to do a complete backup of / to my second hard drive. Then had Sago move the second drive to the second server (both come buldled with two hard drives).

In order to move the IP's Sago would have to move the box to a different rack. Since the new box is in the NEW part of the dataceneter I don't want that (even if I could convince someone to do; as I understand it's an arduous chore to change the IP block on a set up server).

As you may be aware I use my own name servers ns1. and ns2.transwarphosting.net and will simply go to my registrar (domainsite) and change the IP's assigned to them when i am sure everything is setup correctly.

I am writing something about my experiences so anyone interested in it may learn from it but I don't expect to be done with this untull late tomorrow (Tuesday) evening. I have things going onthis evening and work days tomorrow.

[timryberg]

We're still working with Tim on this Pascal and if the backup/restore script works (it is still very beta) then we'll of course let you use it as well. We should know more today or tomorrow Pascal and we'll update this thread with how things go.

Chris

As Chris says this is very BETA and COMMAND-LINE-ONLY but so far the backups are going fine. Will keep you updated.

[pascal]

Cool !!!!

I'm not hurry.

Thanks to share your knowledge :)

Good luck

Pascal

[timryberg]

Cool !!!!

I'm not hurry.

Thanks to share your knowledge :)

Good luck

Pascal

There are some glitches in the restore script and they are working on it. They know what is causing the errors and are working on finding out why and fixing it.

If you are using regular HTML websites that doesn't use databases, pointer domains, and other fancy features, I'd advise just moving your sites over by hand. The command line is akward for someone not as familiar with doing it that way and you may find the other way easier and faster. You have to specify the name of the domain you want to back up and what you want to call the backup file as well as the path to the script itself. I'm okay with this but users wo are less comfortable in the shell may find it easier to do it the old fashioned way.

If you prefer to wait that's okay too. The script is nice when it works :-)

And as I've told Socheat I'm okay with all of this I knew full well what Beta is when I agreed to do this and have backups of all of my data on a second hard drive. Even when they do get the bugs worked out I'd advise anyone doing things this way make a backup "just in case."

We'll keep you posted.

Tim

[pascal]

Thanks Tim,

Well, now I have my 2nd HDD installed and mounted on my old server.
I've done a rsync --delete -a /home /my2ndHDD

It keeps rights authority,

I think that I will backup all the /home/. directories and use the interworx backup/restore utility too (for mails, iworx database, user database)

Then on my new server Restore both rsync and interworx backup files.

For all 'home' softwares (spamassassin, clam, ircd, ...) I think I will reinstall them manualy (wait to know if in the next release spamassassin will be installed).

Could yu list all the directories yu backup/restore and that are absolutly necessary

Hope the next iworx release ETA will be for this week-end (as I'm not at home during the few next week)

Pascal

[timryberg]

Thanks Tim,



NP



Well, now I have my 2nd HDD installed and mounted on my old server.
I've done a rsync --delete -a /home /my2ndHDD

It keeps rights authority,

I think that I will backup all the /home/. directories and use the interworx backup/restore utility too (for mails, iworx database, user database)



At minimum I would make a backup of

/home -- your sites

/etc -- your databases and config files (you don't need all of it but I'd rather have more than I need than not enough)

I will be posting about my experiences with the restore utility in a moment.




Then on my new server Restore both rsync and interworx backup files.

For all 'home' softwares (spamassassin, clam, ircd, ...) I think I will reinstall them manualy (wait to know if in the next release spamassassin will be installed).



That is advisable. RH9 and CentOS may appear the same on the surface but they are different OS's and you may run into problems if you just transfer files. I intend to reinstall mailman from source.



Could yu list all the directories yu backup/restore and that are absolutly necessary.



With the restore untility all you need to do is set up InterWorx the way you want it (name it, crate packages, name servers, etc.)

The Backup/Restore utility will save all of the web, database and other files you need in one convenient archive, but I strongly advise you to also have a regular backup "just in case"

This is done on a _domain_by_domain basis_, and you save an archive in .tar.gz format.

You need to transfer the file to the other server (via scp, sftp, or wget -- your chouce but you get permissions problems with wget) run a command on the new server and follow the prompts specifying the archive name and location the IP you want it on, etc. It will restore everything.




Hope the next iworx release ETA will be for this week-end (as I'm not at home during the few next week)

Pascal

I'm sure 1.8.1 will be released when they think they have fixed all of the buggs in 1.8.0. If you set the auto-update to ON in NodeWorx it will install it for you if it comes while you are away from home.

[timryberg]

As I've insinuated I completed the backup/restore of my old/new server and things appear to have been successfull. So far so good. The domains all resolve as near as I can tell and the database seem to be working. Everything is where it should be.

Fair warning: This has NOT been completely smooth. There were bugs in the backup and restore scripts that Socheat and Chris have spent a good amount of time fixing. I know I re-created the backup archives at least twice and Socheat did it at least once. At one time of another over the last week or so Socheat, Chris, and Paul all worked on my server.

I THINK the buggs are probably fixed but last I heard they were still examining code. I'm not sure when they will feel it's ready to try on another server.

Thanks guys for letting me try this. It was fun. :)

I'll keep you posted as to any other oddities that I find.

[pascal]

héhé glad to know everything is fine for you

Iworx team is really good :)

Well here is how I plan to migrate my data. I have changed my plan to take in account your advises. Thanks tim :)

just waiting for the backup/restore iworx tools

I plan to do this :
1 - on my new server change the dns serveur to a false one (ns1.toto.com)

2- Create on the new server all Siteworx/shell accounts

3- Backup siteworx account from the old one to my 2nd hdd

4- rsync /home and /etc from the old one to the 2nd HDD

5- Install 2nd HDD on new server

6- Restore siteworx backup on the new one (from my 2nd Hdd)

7- Restore the data of the shell accounts (from my 2nd hdd)

8- Install 'home' software manually on the new one (wait to know if the next iworx release will have spamassassin, ETA ???)

9- Modify the IP dns serveur at my registrar

10- modify the dns serveur on the new one to the real dns serveurs

11- test everything

12- drop my old server

13- enjoye


Hmm it's ok I'm not supersticious, I'm sure I'll enjoye on the 13 tasks :-p

Do you think I forget something ?
Does it sound fine ?

Thanks
Pascal

[IWorx-Chris]

I have to comment to that many of the bugs weren't actually with the backup/restore script but the system itself. The interworx install wasn't originally "clean" and there were some mail, dns and other issues with permissions that we had to fix after the fact. On a "clean" install these problems wouldn't take the time they did with Tim's system.

Tim's help with this process was much appreciated as well as we had to muddle through some parts because of the original system install. I concur that his system should be all set now and thanks Tim for participating!

Chris

[pascal]

one more time this post show how much how the "collaborative working" is one of the best way :)

Chris, do yu have any comments about my plan ?

Pascal

[IWorx-Chris]

I'll check it out and post later this evening pascal, I don't want to rush to post without reading/thinking.

Chris

[pascal]

no pbm and sorry if I'm 'always asking'

[timryberg]

I have to comment to that many of the bugs weren't actually with the backup/restore script but the system itself. The interworx install wasn't originally "clean" and there were some mail, dns and other issues with permissions that we had to fix after the fact. On a "clean" install these problems wouldn't take the time they did with Tim's system.


Right, I get the impression that mine was one of the first CentOS/InterWorx boxes that the Sago tean set up. I had problems with DNS and qMail that did not surface until I had transferred the domains and DNS server to the new box.




Tim's help with this process was much appreciated as well as we had to muddle through some parts because of the original system install. I concur that his system should be all set now and thanks Tim for participating!

Chris

NP Chris. It was fun :-)

[timryberg]

I'm sure Chris will have more to say on this but I do have a few things.

héhé glad to know everything is fine for you

Iworx team is really good :)

Well here is how I plan to migrate my data. I have changed my plan to take in account your advises. Thanks tim :)

just waiting for the backup/restore iworx tools

I plan to do this :
1 - on my new server change the dns serveur to a false one (ns1.toto.com)


There is no need to do this. What it calls itself makes no difference until you move name server to these IP's. You can leave it at Sago's default or change it to your new name at this juncture so you don't forget. It doesn't matter.



2- Create on the new server all Siteworx/shell accounts



You can do this if you want, but, but it isn't necessary. The restore script creates the SiteWorx account if it doesn't already exist.

Just make sure to create the SW accounts in the same order as they were created on the old server if you have more than one domain with a similar SW account name to make sure the SW account names are the same

For example I have:

Transwarphosting.net (transwar)
Transwarp2000.net (transwa1)
Transwarplists.net (transwa2).

If I had done this in a different order I might have gotten:

Transwarp2000.net (transwar)
Transwarplists.net (transwa1)
Transwarphosting.net (transwa2)

See the difference?

Creating your other shell users at this juncture is probably appropriate.



3- Backup siteworx account from the old one to my 2nd hdd

4- rsync /home and /etc from the old one to the 2nd HDD

5- Install 2nd HDD on new server

6- Restore siteworx backup on the new one (from my 2nd Hdd)

7- Restore the data of the shell accounts (from my 2nd hdd)

8- Install 'home' software manually on the new one (wait to know if the next iworx release will have spamassassin, ETA ???)

9- Modify the IP dns serveur at my registrar

10- modify the dns serveur on the new one to the real dns serveurs

11- test everything

12- drop my old server

13- enjoye



The rest of this looks fine to me.

[IWorx-Chris]

Based on Tim's back/restore I'd advise the following and Tim has already covered some of this:

1) backup each siteworx account on the old server using our backup script.

2) restore each account to the new box (order is less important now Tim as we embed a suggestion for the unix user/uniqname in the siteworx backup file)

3) at this point you have the core data on the new box, I'd test the DNS server on the new box to make sure everthing is resolving. Use dig or nslookup to perform queries on the new server. If DNS works I'd change the old box's DNS for each domain to point to the new server, this will speed up your resolution.

4) If #3 goes ok then you *should* be ok to switch where you DNS servers point at the registrar level (i.e. change what IPs ns1.yourdomain.com points to to the new server).

5) That *should* be most of it, the siteworx backup / restore script does most of the work and based on Tim's experience we've tweaked it both during and after his conversion.

I'm writing this from memory so please ask questions if something seems out of place.

Chris

[pascal]

1) backup each siteworx account on the old server using our backup script.


Does it create account ? or should I create them

I'm afraid that if I restore all the accounts on the new box, with a good name server configure on it, I will have no site accessible. ??? is there any pbm ?



3) at this point you have the core data on the new box, I'd test the DNS server on the new box to make sure everthing is resolving. Use dig or nslookup to perform queries on the new server. If DNS works I'd change the old box's DNS for each domain to point to the new server, this will speed up your resolution.


Do I have to setup my box with my own name server ? (the same which is on the old server)
There will not have corrupted resolution ?


4) If #3 goes ok then you *should* be ok to switch where you DNS servers point at the registrar level (i.e. change what IPs ns1.yourdomain.com points to to the new server).


ok for that


5) That *should* be most of it, the siteworx backup / restore script does most of the work and based on Tim's experience we've tweaked it both during and after his conversion



ok, great, I'm waiting for it !!! (let's play backup/restore) :rolleyes: ;)

Another thing, strange, when I do DIG ns1.myserver.com on the old box I have :

;; ANSWER SECTION:
ns2.myserver.com. 167319 IN A xx.xxx.xx.40
ns2.myserver.com. 167319 IN A xx.xxx.xx.42


but if I do the same command on my new box

;; ANSWER SECTION:
ns2.myserver.com. 167319 IN A xx.xxx.xx.42


this last one is the good one


Thanks Chris, Tim for your advises

Pascal

[IWorx-Socheat]

Does it create account ? or should I create them

I'm afraid that if I restore all the accounts on the new box, with a good name server configure on it, I will have no site accessible. ??? is there any pbm ?

No, it is not necessary to create the account on the new box. If the account does not exist, the restore script will automatically add the account for you. If the account does exist, then the restore script will ask you if it's ok to delete it first.

[pascal]

ok thanks

And for the other points ?

[timryberg]

Does it create account ? or should I create them

I'm afraid that if I restore all the accounts on the new box, with a good name server configure on it, I will have no site accessible. ??? is there any pbm ?

Do I have to setup my box with my own name server ? (the same which is on the old server)
There will not have corrupted resolution ?


No there won't be a problem. Untill you go to your registrar and change the IP address of your name server to the new box the only one that recognizes the new server as ns1.myserver.com and ns2.myserver.com is your server. It's no different than creating a domain/siteworx account on your server. It may say it's mywebsite.com but until you register the domain and point the domain to your name servers the only computer on the Internet that knows the site exists is your server.




ok, great, I'm waiting for it !!! (let's play backup/restore) :rolleyes: ;)



I'm sure the InterWorx team will contact you when they are reado to do this on another server. you've made your willingness to be the next guinea pig quite obvious ;-)



Another thing, strange, when I do DIG ns1.myserver.com on the old box I have :

;; ANSWER SECTION:
ns2.myserver.com. 167319 IN A xx.xxx.xx.40
ns2.myserver.com. 167319 IN A xx.xxx.xx.42


but if I do the same command on my new box

;; ANSWER SECTION:
ns2.myserver.com. 167319 IN A xx.xxx.xx.42


this last one is the good one



Is that first one a typo? you show ns2 twice on two different IP addresses.

I'm not completely sure but I think the dig command checks the DNS info included in your own server's DNS server which is not necessarily what the rest of the Internet sees until you change your IP at your registrar.



Thanks Chris, Tim for your advises

Pascal

NP

[pascal]

lol

Sorry tim and others, I missed one tim's post, so I have asked questions already answered by tim. Sorry for that.

I'd like to thanks a ton all of you for your precious advises.

Well, now everything is clear in my head, so let's do it when you want.

Regards

Pascal

[timryberg]

Good Luck!

[pascal]

Hi everybody

I'd like to tell thank you to all the iworx team involved in my migration (successfull migration :-p )

Indeed, all the iworx team (with a special dedicace to socheat and Chris) help me to migrate my data from my old box to the new one.

I used the great futur 1.8.1 bckp/restore script with the precious socheat's helps.

So only in few words : THANKS A TON

Thank to Tim too, who does it before me, and gives me the good advises !!!

So thanks to everybody, good job guys, I've got it
Pascal

[timryberg]

Hi everybody

I'd like to tell thank you to all the iworx team involved in my migration (successfull migration :-p )

<SNIP>

Thank to Tim too, who does it before me, and gives me the good advises !!!

So thanks to everybody, good job guys, I've got it
Pascal

NP Pascal, glad things worked out for you. Hopeflly we worked out all of the bugs on my box first :-)

Just an update things are still runnung smootely for me and as near as I can tell everything that should have been changed over was.

Mailman is now reinstalled (Thanks Paul!) but on a different domain than it was so I have mailing list ability back while I work on Sympa, a much better solution, but a lot more complicated.

Tim

[IWorx-Chris]

Glad it all worked out for both of you. These things will only get smoother as time goes on :).

Chris

[Justec]

I was going to create a new thread for this, but doing a search first I found this one and thought this would be a good place to start since what I’m looking for is the same thing just applied differently.

I know a lot of people with 2 hard drives do backups (copy the files, which they could automate by writing a script and running it with cron) of their server to the 2nd hard drive, but I like doing remote backups (or maybe I just like them because I only have one hard drive :D ). Right now I'm doing manual backups for each SiteWorx (through SiteWorx backup feature) account and saving them on my computer which is backed up to a USB hard drive. But even for people that have 2 hard drives, it isn’t a bad idea to also have an offsite backup.

Basically what I want to accomplish is a remote backup of all clients (SiteWorx) accounts and if possible a backup of selected system folders. I know I could create a script similar to the above 2nd hard drive backup and just gzip the files and download, but I want to see what the plans are for NodeWorx before I move forward with that. I want to know if what I’m thinking is just something I would have to implement myself or if it’s something that would be useful to others and will be incorporated into NodeWorx in the future.

My Ideas:
Not sure how far along you guys are, but just want to throw in my thoughts and see if they help. First off I’m assuming the backup/restore script for version 1.8.1 will be incorporated directly into NodeWorx as a “Backup SiteWorx Accounts” tab or something similar to that.

The way I see it working after that is this new NodeWorx tab will be a list of all the SiteWorx accounts (like the one you see when you click the SiteWorx account tab now) and you can check off which accounts you would like to backup (and what you want to backup, ie. Mail, web, db) and it would make a really big :eek: zip file to download the same way the SiteWorx backup works now.

Also, another cool feature would be having a more automated way of doing this. You could create something like “Backup Jobs” in NodeWorx where you can have NodeWorx perform scheduled backups of SiteWorx accounts (and system folders) and have them automatically transferred to a remote server or copied to a directory on the server (or to a folder on that second hard drive :rolleyes: ).

Thanks for reading,

Justin

[timryberg]

I'm sure Chris or Socheat will want to respond to this as well but having beta-tested the backup/restore and talked to Socheat about it quite a bit I have a few things to contribute.


My Ideas:
Not sure how far along you guys are, but just want to throw in my thoughts and see if they help. First off I’m assuming the backup/restore script for version 1.8.1 will be incorporated directly into NodeWorx as a “Backup SiteWorx Accounts” tab or something similar to that.



I think doing it this was is a good one, but as I understood it the backup was part of SiteWorx (just like the old backup script). Of course, as the NodeWorx user you have access to all of the SiteWorx accounts even if you don't know a user's password.



The way I see it working after that is this new NodeWorx tab will be a list of all the SiteWorx accounts (like the one you see when you click the SiteWorx account tab now) and you can check off which accounts you would like to backup (and what you want to backup, ie. Mail, web, db) and it would make a really big :eek: zip file to download the same way the SiteWorx backup works now.



Socheat told me that the script is a complete backup/restore in this version and will be selective in the next version. Is that still the case?



Also, another cool feature would be having a more automated way of doing this. You could create something like “Backup Jobs” in NodeWorx where you can have NodeWorx perform scheduled backups of SiteWorx accounts (and system folders) and have them automatically transferred to a remote server or copied to a directory on the server (or to a folder on that second hard drive :rolleyes: ).

Those are good ideas, but I suspect this is a ways off. They want a stable and functional backup for now, and will work on features later, as I understand it.

[Justec]

I think doing it this was is a good one, but as I understood it the backup was part of SiteWorx (just like the old backup script). Of course, as the NodeWorx user you have access to all of the SiteWorx accounts even if you don't know a user's password.
So if I'm understanding what you are saying correctly this new Backup/Restore that is going to be added in 1.8.1 isn't going to be something along the lines of a NodeWorx upgrade, but rather will just be adding a restore feature to the SiteWorx page.

From what I understood of the "script version" that you tested this was a complete backup restore of all SiteWorx accounts. Maybe I misunderstood how it worked and you still have to do each one individually (ie. iworxbackup siteworxaccount1, iworxrestore siteworxaccount1.tar.gz) and it was just through the shell you had access to the restore.

Socheat told me that the script is a complete backup/restore in this version and will be selective in the next version. Is that still the case?This confuses me with the first quote unless the script will be a systemwide backup/restore, but will be available to use command line ONLY and a restore feature will be added to SiteWorx... is that right?

Those are good ideas, but I suspect this is a ways off. They want a stable and functional backup for now, and will work on features later, as I understand it.Yes, I agree, I'm sure there are more important things to work on although backing up data is a very important they do have something that works (just not as flexable as what I'm thinking).
Maybe a midway point is I can just use the same script you used that the Iworx team created and backup the SiteWorx accounts command line so I can do them all at once instead of what I'm doing now going one by one by one. Then maybe I could create another script that calls the Iworx script and then handles all the FTP'n (or however I transfer it) and that way I could automate it with a cron job.

- Justin

[timryberg]

So if I'm understanding what you are saying correctly this new Backup/Restore that is going to be added in 1.8.1 isn't going to be something along the lines of a NodeWorx upgrade, but rather will just be adding a restore feature to the SiteWorx page.



As I understand it, it will be replacing the existing backup script and adding a restore, yes. That's how I understand it but I may be wrong. They may work in both SW and NW. Since all I had was a command line version, Socheat and I never really talked about that :-)



From what I understood of the "script version" that you tested this was a complete backup restore of all SiteWorx accounts. Maybe I misunderstood how it worked and you still have to do each one individually (ie. iworxbackup siteworxaccount1, iworxrestore siteworxaccount1.tar.gz) and it was just through the shell you had access to the restore.



Yes, each account was backed up and restored individually from the command line. I used this to back up and restore all of my sw accounts. NodeWorx settings (e.g name servers, pakcages, etc) had to be restored by hand.

~/bin/newbakup.pex (or something like that)
~/bin/restore.pex filename.tar.gz

all by hand.



This confuses me with the first quote unless the script will be a systemwide backup/restore, but will be available to use command line ONLY and a restore feature will be added to SiteWorx... is that right?



The only reason the one I used was command line only was because it was VERY BETA and they hadn't written the web interface yet. I expect that they will eliminate command line access since it is sort of complicated permissions wise (you have to su - to the InterWorx user iworx). It makes sexne to give both NodeWorx and SIteWorx access to this but I'm not sure if that is what they are going to do.



Yes, I agree, I'm sure there are more important things to work on although backing up data is a very important they do have something that works (just not as flexable as what I'm thinking).



I imagine them developing this to something along these lines at some point but I expect the priority now is to get it stabke and working so they can get any remaining bugs worked out of it before expanding it.



Maybe a midway point is I can just use the same script you used that the Iworx team created and backup the SiteWorx accounts command line so I can do them all at once instead of what I'm doing now going one by one by one. Then maybe I could create another script that calls the Iworx script and then handles all the FTP'n (or however I transfer it) and that way I could automate it with a cron job.

- Justin

Possibly but I think it would be more complicated than you think because of permissions. I don't know, maybe they will set it up the way you inagine eventually if you ask. Really I'd waith for Chris or Socheat to respond to this thread since they know more about their specific plans than I do.

[IWorx-Socheat]

Sorry for the late reply guys. I'll address several posts at once, comments beneath the quoted text.

Basically what I want to accomplish is a remote backup of all clients (SiteWorx) accounts and if possible a backup of selected system folders.
It depends on what you mean by "selected system folders". If you are talking about the "var" directory that is within each SiteWorx account, then this is already handled by the backup program. If you want to back up, for example, "/etc", you'll have to do that on your own.


First off I’m assuming the backup/restore script for version 1.8.1 will be incorporated directly into NodeWorx as a “Backup SiteWorx Accounts” tab or something similar to that.
Yes, that is the plan. :)

The way I see it working after that is this new NodeWorx tab will be a list of all the SiteWorx accounts (like the one you see when you click the SiteWorx account tab now) and you can check off which accounts you would like to backup (and what you want to backup, ie. Mail, web, db) and it would make a really big :eek: zip file to download the same way the SiteWorx backup works now.
As Tim has pointed out, the next version that will be released only does complete backups and complete restores. Selective backup/restores will be in a future release. But yes, you will be able to choose which SiteWorx accounts you would like a full backup of. It will create a tar.gz file for each SiteWorx account.

Also, another cool feature would be having a more automated way of doing this. You could create something like “Backup Jobs” in NodeWorx where you can have NodeWorx perform scheduled backups of SiteWorx accounts (and system folders) and have them automatically transferred to a remote server or copied to a directory on the server
This is also planned, and may make it into this next release, but no promises at this time. :o

So if I'm understanding what you are saying correctly this new Backup/Restore that is going to be added in 1.8.1 isn't going to be something along the lines of a NodeWorx upgrade, but rather will just be adding a restore feature to the SiteWorx page.
Right now, the primary restore interface will be in NodeWorx. Because of how the backups are restored right now, it's not possible to do a SiteWorx restore from within the SiteWorx interface. This will be possible once the selective restore is implemented. Backups can be done from either NodeWorx or from within SiteWorx.

Maybe I misunderstood how it worked and you still have to do each one individually (ie. iworxbackup siteworxaccount1, iworxrestore siteworxaccount1.tar.gz) and it was just through the shell you had access to the restore.
This is only true for the command line interface backup program. It may change in the future, if enough people like the command line interface better, but as it stands, you have to run the backup script on each SiteWorx account. Again, the web interface will allow you to backup multiple accounts at once.

This confuses me with the first quote unless the script will be a systemwide backup/restore, but will be available to use command line ONLY and a restore feature will be added to SiteWorx... is that right?
To clarify, there will be a web interface for both backup and restore.

Maybe a midway point is I can just use the same script you used that the Iworx team created and backup the SiteWorx accounts command line so I can do them all at once instead of what I'm doing now going one by one by one.
I'm always looking for backup/restore beta testers. :) However, part of the problem with those who have beta tested previous versions is that there are a significant amount of files that have changed between the stable/released version and the development branch the backup/restore code is in. We copy over the relevant files, but it could break other functions outside of backup/restore. So then time is spent figuring out if a.) the backup/restore code is buggy b.) the dependent files are buggy c.) the stable code is buggy d.) the mix of stable/devel code is buggy d.) the box has something else wrong with it (bad install, hardware, etc) or 3.) a mix of all of the above. :)

In a nutshell, it would be easier for you to test the updated command line code, because that requires less files to be copied over. I'm not sure how feasible it would be to have you test out the web interface, but if it is, I'll definately let you know.

As I understand it, it will be replacing the existing backup script and adding a restore, yes. That's how I understand it but I may be wrong.
Yes, the old backup system in SiteWorx will be replaced with this new backup system. The exact same backup system will also be mirrored in NodeWorx to allow the NodeWorx admin to backup SiteWorx accounts. However, as I stated above, the restore feature will initially only be available in NodeWorx.

I expect that they will eliminate command line access since it is sort of complicated permissions wise (you have to su - to the InterWorx user iworx).
Just a comment on su'ing to the iworx user. This isn't absolutely necessary. The backup and restore script can be run as the root user, but as they both do things to the filesystem, I advised everyone to run the script as the iworx user, so in the event code was buggy it would minimize the possible damage. The version Tim tested was more complex than the current version. His required several command line parameters, whereas this newer version will prompt you for required information from the command line.

Hope that answers your questions Justin. Again, sorry for the late reply. :(

[Justec]

As Tim has pointed out, the next version that will be released only does complete backups and complete restores. Selective backup/restores will be in a future release. But yes, you will be able to choose which SiteWorx accounts you would like a full backup of. It will create a tar.gz file for each SiteWorx account.Thanks for the very detailed response. My only question now is to clarify full/selective backups.

At first I thought you meant you could only backup / restore all the siteworx accounts on the server and only restore all of them together. So for example if you have 5 accounts and 1 of them has a problem you would then have to restore all 5 accounts to get that one account back.

But then I remember the web, database, mail selective backups. So because my first idea doesn't make much sense and because you said it creates seperate tar.gz for each account I assume you mean the backup just backups everything in each SiteWorx account without the option to unselect "mail" for example (Run-on..... :D )

So is my second assumption correct? You could select which accounts to backup and restore, but just not what to backup inside each account?

I just reread what I quoted above and it pretty much answered my question. "But yes, you will be able to choose which SiteWorx accounts you would like a full backup of." I thought I would not delete this post just in case someone else missed that the first time :o

Justin

[IWorx-Socheat]

Yes, your second assumption is correct. Right now, full backup means "Full backup of an individual SiteWorx account, which includes everything (mail, db, ftp users, cronjobs, ssl, sub/pointer domains, etc)". The web interface will automate backing up more than one SiteWorx account, but you will still be able to pick and choose which ones and you will have a seperate backup file for each account.

When we say selective backup, we mean "from a given full SiteWorx backup file, restore only the mail, only the db, etc".

[timryberg]

Thanks for all of the detailed answers, Socheat. It certainly looks like the backup/restore will be something really great. Good job, guys! Your hard work is greatly appreichated :-)

[Justec]

Thanks for all of the detailed answers, Socheat. It certainly looks like the backup/restore will be something really great. Good job, guys! Your hard work is greatly appreichated :-)Couldn't have said it better myself, so why bother when you can just hit "QUOTE" :p