[sip-comm-dev] Improving MCL storage. Brought to u by Ben!


#1

Hey Ben,

I just saw your commit on the MetaContactList storage algorithm. Now
this is a nice surprise! :slight_smile:

For those of you that have not followed, here's the situation. For some
time now, many users (especially those running SIP Communicator on Mac
OS X) had been seeing problems with the meta contact list. In some cases
it would disappear and then you lose all contacts that you have been
merging. The problem would also appear on Linux though more rarely.

We believe that in a vast majority of the cases, such failures are
caused by the fact that the application would exit and shut down the VM
while storing the list so the xml file would remain incomplete.

The solution that Ben has implemented consists in always writing the
contact list file twice. This way if a write on the main file fails,
next time we start SIP Communicator we would be able to reload from the
backup file.

Good job Ben!

I only have a question. Maybe I am missing something, but I don't quite
understand why you delete the backup file after you create it. Did you
commit this line by mistake?

Anyways. I am really glad you committed this!

Cheers
Emil

路路路

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#2

Hey Emil, all,

Thanks for your mail and your explications, I should have write it but I thought that a such little modification didn't need an explication but thanks anyway for finishing my work :slight_smile:

I'm deleting the backup file only to simplify the work when SC restarts:
If a backup is present, we use it because it means that the last write didn't finish correctly, else we do the normal job.
If the backup file is never deleted it is more complex to determine if the last write was successful: maybe the original file is correct and contains the modification, maybe the shutdown occurred just before the write or worst, maybe the write is only partial. Then, when SC restarts, it become very difficult to determine what file to use.

I also would like to complete your explications by saying that this little patch is only a first step because this problem also occur sometimes with the sip-comunicator.xml file and perhaps others so it'll probably be a new plugin or service able to handle secured write to important files.

Cheers,
Ben

Emil Ivov a 锟絚rit :

路路路

Hey Ben,

I just saw your commit on the MetaContactList storage algorithm. Now
this is a nice surprise! :slight_smile:

For those of you that have not followed, here's the situation. For some
time now, many users (especially those running SIP Communicator on Mac
OS X) had been seeing problems with the meta contact list. In some cases
it would disappear and then you lose all contacts that you have been
merging. The problem would also appear on Linux though more rarely.

We believe that in a vast majority of the cases, such failures are
caused by the fact that the application would exit and shut down the VM
while storing the list so the xml file would remain incomplete.

The solution that Ben has implemented consists in always writing the
contact list file twice. This way if a write on the main file fails,
next time we start SIP Communicator we would be able to reload from the
backup file.

Good job Ben!

I only have a question. Maybe I am missing something, but I don't quite
understand why you delete the backup file after you create it. Did you
commit this line by mistake?

Anyways. I am really glad you committed this!

Cheers
Emil

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#3

I've downloaded the source code into Eclipse and used the .classpath file provided with the download.

Eclipse can't find org.jibble.pircbot

Where is it located?
Is it part of a jar file?

Thanks

Charlie

路路路

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#4

Hi Ben,

Benoit Pradelle wrote:

Hey Emil, all,

Thanks for your mail and your explications, I should have write it but I
thought that a such little modification didn't need an explication but
thanks anyway for finishing my work :slight_smile:

No problem :).

I'm deleting the backup file only to simplify the work when SC restarts:
If a backup is present, we use it because it means that the last write
didn't finish correctly, else we do the normal job.
If the backup file is never deleted it is more complex to determine if
the last write was successful: maybe the original file is correct and
contains the modification, maybe the shutdown occurred just before the
write or worst, maybe the write is only partial. Then, when SC restarts,
it become very difficult to determine what file to use.

I am not sure I get this. The way that things are currently implemented
we have:

1) Create original; 2) Create copy; 3) Delete copy

After a normal execution of the sequence we would end-up with only the
original and no copy because it would be deleted.

Then let's say that the program exits unexpectedly and the process gets
interrupted in step 1. Step 2 would never happen since the program died
and the backup file from a previous execution of the sequence would have
been deleted. So next time we start SIP Communicator we'd still be
unable to reload the contact list since there won't be any backup.

If you don't delete the copy however everything falls into place. If the
process is cut in Step 1, you would have the backup from the previous run.

If a shutdown happens normally in this case you would indeed have an
unnecessary backup but this won't be a problem. I don't think this makes
it difficult to determine which file to use. You always load the primary
file and only use the backup if loading the first one has failed.

Does this make any sense?

I also would like to complete your explications by saying that this
little patch is only a first step because this problem also occur
sometimes with the sip-comunicator.xml file and perhaps others so it'll
probably be a new plugin or service able to handle secured write to
important files.

Cool! Glad to know that this is also in your plans! After we have tested
this in the meta contact list we could actually try to bring it out in a
service. WDYT?

Cheers
Emil

路路路

Cheers,
Ben

Emil Ivov a 锟絚rit :

Hey Ben,

I just saw your commit on the MetaContactList storage algorithm. Now
this is a nice surprise! :slight_smile:

For those of you that have not followed, here's the situation. For some
time now, many users (especially those running SIP Communicator on Mac
OS X) had been seeing problems with the meta contact list. In some cases
it would disappear and then you lose all contacts that you have been
merging. The problem would also appear on Linux though more rarely.

We believe that in a vast majority of the cases, such failures are
caused by the fact that the application would exit and shut down the VM
while storing the list so the xml file would remain incomplete.

The solution that Ben has implemented consists in always writing the
contact list file twice. This way if a write on the main file fails,
next time we start SIP Communicator we would be able to reload from the
backup file.

Good job Ben!

I only have a question. Maybe I am missing something, but I don't quite
understand why you delete the backup file after you create it. Did you
commit this line by mistake?

Anyways. I am really glad you committed this!

Cheers
Emil

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#5

Hi Charlie,

you could find the pircbot.jar in PROJECT_DIR/lib/installer-exclude/pircbot.jar. I've updated also the .classpath file in ide/eclipse.

Yana

Charlie Kelly wrote:

路路路

I've downloaded the source code into Eclipse and used the .classpath file provided with the download.

Eclipse can't find org.jibble.pircbot

Where is it located?
Is it part of a jar file?

Thanks

Charlie

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#6

I Emil,

Emil Ivov a 锟絚rit :

Hi Ben,
聽聽

I'm deleting the backup file only to simplify the work when SC restarts:
If a backup is present, we use it because it means that the last write didn't finish correctly, else we do the normal job.
If the backup file is never deleted it is more complex to determine if the last write was successful: maybe the original file is correct and contains the modification, maybe the shutdown occurred just before the write or worst, maybe the write is only partial. Then, when SC restarts, it become very difficult to determine what file to use.
聽聽聽聽
I am not sure I get this. The way that things are currently implemented
we have:

1) Create original; 2) Create copy; 3) Delete copy

After a normal execution of the sequence we would end-up with only the
original and no copy because it would be deleted.

Then let's say that the program exits unexpectedly and the process gets
interrupted in step 1. Step 2 would never happen since the program died
and the backup file from a previous execution of the sequence would have
been deleted. So next time we start SIP Communicator we'd still be
unable to reload the contact list since there won't be any backup.

If you don't delete the copy however everything falls into place. If the
process is cut in Step 1, you would have the backup from the previous run.

If a shutdown happens normally in this case you would indeed have an
unnecessary backup but this won't be a problem. I don't think this makes
it difficult to determine which file to use. You always load the primary
file and only use the backup if loading the first one has failed.

Does this make any sense?
聽聽
In fact the algorithm is a little bit different than what you described:
1) copy the original contact list
2) modify the contact list
3) delete the backup file

this way, if a backup file is present it contains an older but valid contactlist.

However, while I'm writting these lines I'm thinking at a possible error case: if the stop occurs during the copy of the original contact list, the backup will be loaded at the next SC start even if it isn't correct. So it seems that I still got some work on it :slight_smile:

I also would like to complete your explications by saying that this little patch is only a first step because this problem also occur sometimes with the sip-comunicator.xml file and perhaps others so it'll probably be a new plugin or service able to handle secured write to important files.
聽聽聽聽
Cool! Glad to know that this is also in your plans! After we have tested
this in the meta contact list we could actually try to bring it out in a
service. WDYT?
聽聽
It's exactly what I was thinking :slight_smile:

Cheers,
Ben

路路路

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#7

Hi everyone,

the question in hand is very interesting - what to do when an error occurs while replacing an existing (and potentially important) file. Maybe we can create a sort of a generic API that can emulate transactional behavior on top of a general FS. I am thinking about something like this (if we are following Java Transaction API http://java.sun.com/products/jta/):

File f = new File("/test.xml");
Transaction trans;
try {
聽聽trans = FileTransaction.beginTransaction(f);
聽聽// some actions with f
聽聽trans.commit();
}
catch(Throwable e) {
聽聽trans.rollback();
聽聽throw e;
}

Following JTA would require that the originating file is copied when beginTransaction is called. On commit() the backup file should be deleted, whereas on rollback() the backup file should replace the invalidated "f". Here, there are some potential issues, but I think that they can be addressed relatively easily. The default behavior would be to automatically rollback a transaction. This way, one may encapsulate all kinds of file operations in transactions and in case of unexpected exceptions, kernel panics, premature application termination, etc. data corruption would be kept to minimum.

A simpler approach would be to write all changes to a secondary file, and once done - replace the original. An example would be like:

File f = new File("/test.xml");
Transaction trans;
try {
聽聽trans = FileTransaction.beginTransaction(f);
聽聽File newFile = trans.getProtectedFile();
聽聽// write to newFile and potentially read from f
聽聽trans.commit();
}
catch(Throwable e) {
聽聽trans.rollback();
聽聽throw e;
}

An example behavior of the different functions could be:

getProtectedFile():
create a file "/test.xml.tmp03942940192.transactionbackup"

commit():
1) Rename "/test.xml" to "/test.xml.tmp01293093.original"
2) Rename "/test.xml.tmp03942940192.transactionbackup" to "/text.xml"
3) Delete "/test.xml.tmp01293093.original"

Of course, one would always have to access a file in a transaction (or at least call a function that would search for eventual "....original" and ".....transactionbackup" files and perform appropriate actions.

What do you think ?

Alex

路路路

Le 16 sept. 07 脿 13:39, Benoit Pradelle a 茅crit :

I Emil,

Emil Ivov a 茅crit :

Hi Ben,

I'm deleting the backup file only to simplify the work when SC restarts:
If a backup is present, we use it because it means that the last write didn't finish correctly, else we do the normal job.
If the backup file is never deleted it is more complex to determine if the last write was successful: maybe the original file is correct and contains the modification, maybe the shutdown occurred just before the write or worst, maybe the write is only partial. Then, when SC restarts, it become very difficult to determine what file to use.

I am not sure I get this. The way that things are currently implemented
we have:

1) Create original; 2) Create copy; 3) Delete copy

After a normal execution of the sequence we would end-up with only the
original and no copy because it would be deleted.

Then let's say that the program exits unexpectedly and the process gets
interrupted in step 1. Step 2 would never happen since the program died
and the backup file from a previous execution of the sequence would have
been deleted. So next time we start SIP Communicator we'd still be
unable to reload the contact list since there won't be any backup.

If you don't delete the copy however everything falls into place. If the
process is cut in Step 1, you would have the backup from the previous run.

If a shutdown happens normally in this case you would indeed have an
unnecessary backup but this won't be a problem. I don't think this makes
it difficult to determine which file to use. You always load the primary
file and only use the backup if loading the first one has failed.

Does this make any sense?

In fact the algorithm is a little bit different than what you described:
1) copy the original contact list
2) modify the contact list
3) delete the backup file

this way, if a backup file is present it contains an older but valid contactlist.

However, while I'm writting these lines I'm thinking at a possible error case: if the stop occurs during the copy of the original contact list, the backup will be loaded at the next SC start even if it isn't correct. So it seems that I still got some work on it :slight_smile:

I also would like to complete your explications by saying that this little patch is only a first step because this problem also occur sometimes with the sip-comunicator.xml file and perhaps others so it'll probably be a new plugin or service able to handle secured write to important files.

Cool! Glad to know that this is also in your plans! After we have tested
this in the meta contact list we could actually try to bring it out in a
service. WDYT?

It's exactly what I was thinking :slight_smile:

Cheers,
Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#8

Hi Ben,

Benoit Pradelle wrote:

In fact the algorithm is a little bit different than what you described:
1) copy the original contact list
2) modify the contact list
3) delete the backup file

this way, if a backup file is present it contains an older but valid
contactlist.

Oh, ok, I see now. Indeed I hadn't looked at your storage procedure very
carefully. Yes this should work and it is better than what I suggested
because it would spare us the redundancy of always keeping two contact
list files. Nice!

However, while I'm writting these lines I'm thinking at a possible error
case: if the stop occurs during the copy of the original contact list,
the backup will be loaded at the next SC start even if it isn't correct.
So it seems that I still got some work on it :slight_smile:

Well, how about changing your read procedure this way:

1. You always try to load the original file first and you don't care
whether or not there is a backup.
2. If, and only if loading the original fails, you look for a backup. If
there is one, you move it over the corrupt original and go back to 1.
Does this make sense?

Cheers
Emil

路路路

I also would like to complete your explications by saying that this
little patch is only a first step because this problem also occur
sometimes with the sip-comunicator.xml file and perhaps others so it'll
probably be a new plugin or service able to handle secured write to
important files.
聽聽聽聽

Cool! Glad to know that this is also in your plans! After we have tested
this in the meta contact list we could actually try to bring it out in a
service. WDYT?
聽聽
It's exactly what I was thinking :slight_smile:

Cheers,
Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#9

Hi again,

I wanted to add that we could modify a little bit the FileAccessService adding the mentioned FileTransaction to it. This way, the moment one invokes getPersistentFile, the FileAccessService will look up for remaining "..tmpXXXXXX.original" and "...tmpXXXXXX.transactionbackup" files and perform a cleanup. The cleanup procedure would be the following:

1) If exist FILENAME + FILENAME.tmpXXXX.transactionbackup -> DISCARD FILENAME.tmpXXXX.transactionbackup
2) If exist FILENAME.tmpXXXX.original + FILENAME.tmpXXXX.transactionbackup -> RENAME FILENAME.tmpXXXX.transactionbackup to FILENAME and DISCARD FILENAME.tmpXXXX.original
3) If exist FILENAME + FILENAME.tmpXXXX.original -> DISCARD FILENAME.tmpXXXX.original
4) In all other cases keep FILENAME.

WDYT?

Alex

路路路

Le 16 sept. 07 脿 14:30, Alexander Pelov a 茅crit :

Hi everyone,

the question in hand is very interesting - what to do when an error occurs while replacing an existing (and potentially important) file. Maybe we can create a sort of a generic API that can emulate transactional behavior on top of a general FS. I am thinking about something like this (if we are following Java Transaction API http://java.sun.com/products/jta/):

File f = new File("/test.xml");
Transaction trans;
try {
聽聽trans = FileTransaction.beginTransaction(f);
聽聽// some actions with f
聽聽trans.commit();
}
catch(Throwable e) {
聽聽trans.rollback();
聽聽throw e;
}

Following JTA would require that the originating file is copied when beginTransaction is called. On commit() the backup file should be deleted, whereas on rollback() the backup file should replace the invalidated "f". Here, there are some potential issues, but I think that they can be addressed relatively easily. The default behavior would be to automatically rollback a transaction. This way, one may encapsulate all kinds of file operations in transactions and in case of unexpected exceptions, kernel panics, premature application termination, etc. data corruption would be kept to minimum.

A simpler approach would be to write all changes to a secondary file, and once done - replace the original. An example would be like:

File f = new File("/test.xml");
Transaction trans;
try {
聽聽trans = FileTransaction.beginTransaction(f);
聽聽File newFile = trans.getProtectedFile();
聽聽// write to newFile and potentially read from f
聽聽trans.commit();
}
catch(Throwable e) {
聽聽trans.rollback();
聽聽throw e;
}

An example behavior of the different functions could be:

getProtectedFile():
create a file "/test.xml.tmp03942940192.transactionbackup"

commit():
1) Rename "/test.xml" to "/test.xml.tmp01293093.original"
2) Rename "/test.xml.tmp03942940192.transactionbackup" to "/text.xml"
3) Delete "/test.xml.tmp01293093.original"

Of course, one would always have to access a file in a transaction (or at least call a function that would search for eventual "....original" and ".....transactionbackup" files and perform appropriate actions.

What do you think ?

Alex

Le 16 sept. 07 脿 13:39, Benoit Pradelle a 茅crit :

I Emil,

Emil Ivov a 茅crit :

Hi Ben,

I'm deleting the backup file only to simplify the work when SC restarts:
If a backup is present, we use it because it means that the last write didn't finish correctly, else we do the normal job.
If the backup file is never deleted it is more complex to determine if the last write was successful: maybe the original file is correct and contains the modification, maybe the shutdown occurred just before the write or worst, maybe the write is only partial. Then, when SC restarts, it become very difficult to determine what file to use.

I am not sure I get this. The way that things are currently implemented
we have:

1) Create original; 2) Create copy; 3) Delete copy

After a normal execution of the sequence we would end-up with only the
original and no copy because it would be deleted.

Then let's say that the program exits unexpectedly and the process gets
interrupted in step 1. Step 2 would never happen since the program died
and the backup file from a previous execution of the sequence would have
been deleted. So next time we start SIP Communicator we'd still be
unable to reload the contact list since there won't be any backup.

If you don't delete the copy however everything falls into place. If the
process is cut in Step 1, you would have the backup from the previous run.

If a shutdown happens normally in this case you would indeed have an
unnecessary backup but this won't be a problem. I don't think this makes
it difficult to determine which file to use. You always load the primary
file and only use the backup if loading the first one has failed.

Does this make any sense?

In fact the algorithm is a little bit different than what you described:
1) copy the original contact list
2) modify the contact list
3) delete the backup file

this way, if a backup file is present it contains an older but valid contactlist.

However, while I'm writting these lines I'm thinking at a possible error case: if the stop occurs during the copy of the original contact list, the backup will be loaded at the next SC start even if it isn't correct. So it seems that I still got some work on it :slight_smile:

I also would like to complete your explications by saying that this little patch is only a first step because this problem also occur sometimes with the sip-comunicator.xml file and perhaps others so it'll probably be a new plugin or service able to handle secured write to important files.

Cool! Glad to know that this is also in your plans! After we have tested
this in the meta contact list we could actually try to bring it out in a
service. WDYT?

It's exactly what I was thinking :slight_smile:

Cheers,
Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#10

Hi again,

Emil Ivov a 锟絚rit :

Hi Ben,

Benoit Pradelle wrote:
聽聽

In fact the algorithm is a little bit different than what you described:
1) copy the original contact list
2) modify the contact list
3) delete the backup file

this way, if a backup file is present it contains an older but valid contactlist.
聽聽聽聽
Oh, ok, I see now. Indeed I hadn't looked at your storage procedure very
carefully. Yes this should work and it is better than what I suggested
because it would spare us the redundancy of always keeping two contact
list files. Nice!

However, while I'm writting these lines I'm thinking at a possible error case: if the stop occurs during the copy of the original contact list, the backup will be loaded at the next SC start even if it isn't correct. So it seems that I still got some work on it :slight_smile:
聽聽聽聽
Well, how about changing your read procedure this way:

1. You always try to load the original file first and you don't care
whether or not there is a backup.
2. If, and only if loading the original fails, you look for a backup. If
there is one, you move it over the corrupt original and go back to 1.
Does this make sense?
聽聽
It may be a good idea but I had another idea which may avoid us the double parse if a backup is present and avoid having a copy of every important file on the disk.

In fact my idea is pretty simple:
1) copy "file" in "file.bak"
2) rename "file.bak" in "file.bakup"
3) write on "file"
4) delete "file.backup"

This way, if a failure occur before file.bak is renamed (totally copied so in a coherent state) it'll be totally ignored at the next restart. Of course, as before, if a .backup file is present, we load it without considering the original file.

WDYT ?

Ben.

路路路

Cheers
Emil
聽聽

I also would like to complete your explications by saying that this little patch is only a first step because this problem also occur sometimes with the sip-comunicator.xml file and perhaps others so it'll probably be a new plugin or service able to handle secured write to important files.
聽聽聽聽

Cool! Glad to know that this is also in your plans! After we have tested
this in the meta contact list we could actually try to bring it out in a
service. WDYT?
聽聽

It's exactly what I was thinking :slight_smile:

Cheers,
Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#11

Hi alexander, all,

Alexander Pelov a 锟絚rit :

Hi again,

I wanted to add that we could modify a little bit the FileAccessService adding the mentioned FileTransaction to it. This way, the moment one invokes getPersistentFile, the FileAccessService will look up for remaining "..tmpXXXXXX.original" and "...tmpXXXXXX.transactionbackup" files and perform a cleanup. The cleanup procedure would be the following:

1) If exist FILENAME + FILENAME.tmpXXXX.transactionbackup -> DISCARD FILENAME.tmpXXXX.transactionbackup
2) If exist FILENAME.tmpXXXX.original + FILENAME.tmpXXXX.transactionbackup -> RENAME FILENAME.tmpXXXX.transactionbackup to FILENAME and DISCARD FILENAME.tmpXXXX.original
3) If exist FILENAME + FILENAME.tmpXXXX.original -> DISCARD FILENAME.tmpXXXX.original
4) In all other cases keep FILENAME.

WDYT?

Alex

Le 16 sept. 07 锟 14:30, Alexander Pelov a 锟絚rit :

Hi everyone,

the question in hand is very interesting - what to do when an error occurs while replacing an existing (and potentially important) file. Maybe we can create a sort of a generic API that can emulate transactional behavior on top of a general FS. I am thinking about something like this (if we are following Java Transaction API http://java.sun.com/products/jta/):

File f = new File("/test.xml");
Transaction trans;
try {
聽聽聽聽trans = FileTransaction.beginTransaction(f);
聽聽聽聽// some actions with f
聽聽聽聽trans.commit();
}
catch(Throwable e) {
聽聽聽聽trans.rollback();
聽聽聽聽throw e;
}

Following JTA would require that the originating file is copied when beginTransaction is called. On commit() the backup file should be deleted, whereas on rollback() the backup file should replace the invalidated "f". Here, there are some potential issues, but I think that they can be addressed relatively easily. The default behavior would be to automatically rollback a transaction. This way, one may encapsulate all kinds of file operations in transactions and in case of unexpected exceptions, kernel panics, premature application termination, etc. data corruption would be kept to minimum.

A simpler approach would be to write all changes to a secondary file, and once done - replace the original. An example would be like:

File f = new File("/test.xml");
Transaction trans;
try {
聽聽聽聽trans = FileTransaction.beginTransaction(f);
聽聽聽聽File newFile = trans.getProtectedFile();
聽聽聽聽// write to newFile and potentially read from f
聽聽聽聽trans.commit();
}
catch(Throwable e) {
聽聽聽聽trans.rollback();
聽聽聽聽throw e;
}

An example behavior of the different functions could be:

getProtectedFile():
create a file "/test.xml.tmp03942940192.transactionbackup"

commit():
1) Rename "/test.xml" to "/test.xml.tmp01293093.original"
2) Rename "/test.xml.tmp03942940192.transactionbackup" to "/text.xml"
3) Delete "/test.xml.tmp01293093.original"

Of course, one would always have to access a file in a transaction (or at least call a function that would search for eventual "....original" and ".....transactionbackup" files and perform appropriate actions.

What do you think ?

Alex

Basically it's what I was thinking at but instead of using a transaction model I was thinking at a SecuredFile which extends File and by overloading basic I/O operations of the File class we could use these SecuredFiles really more easily and without any large modifications. However, it's just an idea for the moment, I don't really know if it is really possible.

The main drawback of this solution is that sometimes a lot of little write are done in an operation and it'll be difficult to determine when all these little write end. I was thinking on something like a buffer for all the operations and a writing thread which periodically commit the modifications. It's probably less efficient but this may avoid large code modifications.

I don't know what we should prefer ? I don't really know how many class will use these secured files and what they'll do with it...

Cheers,
Ben

路路路

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#12

Hi Ben,

Benoit Pradelle wrote:

Hi again,

Emil Ivov a 锟絚rit :

Hi Ben,

Benoit Pradelle wrote:
聽聽

In fact the algorithm is a little bit different than what you described:
1) copy the original contact list
2) modify the contact list
3) delete the backup file

this way, if a backup file is present it contains an older but valid
contactlist.
聽聽聽聽

Oh, ok, I see now. Indeed I hadn't looked at your storage procedure very
carefully. Yes this should work and it is better than what I suggested
because it would spare us the redundancy of always keeping two contact
list files. Nice!

However, while I'm writting these lines I'm thinking at a possible error
case: if the stop occurs during the copy of the original contact list,
the backup will be loaded at the next SC start even if it isn't correct.
So it seems that I still got some work on it :slight_smile:
聽聽聽聽

Well, how about changing your read procedure this way:

1. You always try to load the original file first and you don't care
whether or not there is a backup.
2. If, and only if loading the original fails, you look for a backup. If
there is one, you move it over the corrupt original and go back to 1.
Does this make sense?
聽聽
It may be a good idea but I had another idea which may avoid us the
double parse if a backup is present and avoid having a copy of every
important file on the disk.

In fact my idea is pretty simple:
1) copy "file" in "file.bak"
2) rename "file.bak" in "file.bakup"
3) write on "file"
4) delete "file.backup"

This way, if a failure occur before file.bak is renamed (totally copied
so in a coherent state) it'll be totally ignored at the next restart. Of
course, as before, if a .backup file is present, we load it without
considering the original file.

WDYT ?

Alright, sounds reasonable.

Emil

P.S. I'd probably go for more descriptive extension names though.
Firefox for example adds a "part" suffix for unfinished downloads. We
could also use "tmp" or "writing" or something like that. Would make
debugging easier if we need to play with it some day.

路路路

Ben.

Cheers
Emil
聽聽

I also would like to complete your explications by saying that this
little patch is only a first step because this problem also occur
sometimes with the sip-comunicator.xml file and perhaps others so it'll
probably be a new plugin or service able to handle secured write to
important files.
聽聽聽聽

Cool! Glad to know that this is also in your plans! After we have tested
this in the meta contact list we could actually try to bring it out in a
service. WDYT?
聽聽

It's exactly what I was thinking :slight_smile:

Cheers,
Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#13

Hi Ben, everyone,

If I correctly understand the idea behind SecuredFile, it is a kind of a "failsafe storage" - that is, it almost always guarantees that whatever gets written will not be lost. This is extremely useful, but I think that it's going to be a bit difficult to generalize (as you point out creating a buffer may aggregate several requests, but what would be the size of the buffer + one should have to use some user defined file access streams).

I'd still go with the transactional model, as it has the advantage that the "user process" controls what's happening with its data. Indeed, it would be difficult for an external library to guess when to "commit" the accumulated changes (for example, it wouldn't be very nice if the SecuredFile to decides to "commit" a semi-generated XML file AND a runtime error kills the application just afterwards).

I've already used a 'transactional' file access functionality and I must say that it's extremely useful. You don't think about file integrity, about exceptions that could occur half way through some dirty file generation, etc. - if an error occurs you simply do a "rollback" and as if nothing happened.

Alex

路路路

Le 16 sept. 07 脿 20:24, Benoit Pradelle a 茅crit :

Hi alexander, all,

Alexander Pelov a 茅crit :

Hi again,

I wanted to add that we could modify a little bit the FileAccessService adding the mentioned FileTransaction to it. This way, the moment one invokes getPersistentFile, the FileAccessService will look up for remaining "..tmpXXXXXX.original" and "...tmpXXXXXX.transactionbackup" files and perform a cleanup. The cleanup procedure would be the following:

1) If exist FILENAME + FILENAME.tmpXXXX.transactionbackup -> DISCARD FILENAME.tmpXXXX.transactionbackup
2) If exist FILENAME.tmpXXXX.original + FILENAME.tmpXXXX.transactionbackup -> RENAME FILENAME.tmpXXXX.transactionbackup to FILENAME and DISCARD FILENAME.tmpXXXX.original
3) If exist FILENAME + FILENAME.tmpXXXX.original -> DISCARD FILENAME.tmpXXXX.original
4) In all other cases keep FILENAME.

WDYT?

Alex

Le 16 sept. 07 脿 14:30, Alexander Pelov a 茅crit :

Hi everyone,

the question in hand is very interesting - what to do when an error occurs while replacing an existing (and potentially important) file. Maybe we can create a sort of a generic API that can emulate transactional behavior on top of a general FS. I am thinking about something like this (if we are following Java Transaction API http://java.sun.com/products/jta/):

File f = new File("/test.xml");
Transaction trans;
try {
聽聽聽聽trans = FileTransaction.beginTransaction(f);
聽聽聽聽// some actions with f
聽聽聽聽trans.commit();
}
catch(Throwable e) {
聽聽聽聽trans.rollback();
聽聽聽聽throw e;
}

Following JTA would require that the originating file is copied when beginTransaction is called. On commit() the backup file should be deleted, whereas on rollback() the backup file should replace the invalidated "f". Here, there are some potential issues, but I think that they can be addressed relatively easily. The default behavior would be to automatically rollback a transaction. This way, one may encapsulate all kinds of file operations in transactions and in case of unexpected exceptions, kernel panics, premature application termination, etc. data corruption would be kept to minimum.

A simpler approach would be to write all changes to a secondary file, and once done - replace the original. An example would be like:

File f = new File("/test.xml");
Transaction trans;
try {
聽聽聽聽trans = FileTransaction.beginTransaction(f);
聽聽聽聽File newFile = trans.getProtectedFile();
聽聽聽聽// write to newFile and potentially read from f
聽聽聽聽trans.commit();
}
catch(Throwable e) {
聽聽聽聽trans.rollback();
聽聽聽聽throw e;
}

An example behavior of the different functions could be:

getProtectedFile():
create a file "/test.xml.tmp03942940192.transactionbackup"

commit():
1) Rename "/test.xml" to "/test.xml.tmp01293093.original"
2) Rename "/test.xml.tmp03942940192.transactionbackup" to "/text.xml"
3) Delete "/test.xml.tmp01293093.original"

Of course, one would always have to access a file in a transaction (or at least call a function that would search for eventual "....original" and ".....transactionbackup" files and perform appropriate actions.

What do you think ?

Alex

Basically it's what I was thinking at but instead of using a transaction model I was thinking at a SecuredFile which extends File and by overloading basic I/O operations of the File class we could use these SecuredFiles really more easily and without any large modifications. However, it's just an idea for the moment, I don't really know if it is really possible.

The main drawback of this solution is that sometimes a lot of little write are done in an operation and it'll be difficult to determine when all these little write end. I was thinking on something like a buffer for all the operations and a writing thread which periodically commit the modifications. It's probably less efficient but this may avoid large code modifications.

I don't know what we should prefer ? I don't really know how many class will use these secured files and what they'll do with it...

Cheers,
Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net


#14

Hi,

Alexander Pelov a 锟絚rit :

Hi Ben, everyone,

If I correctly understand the idea behind SecuredFile, it is a kind of a "failsafe storage" - that is, it almost always guarantees that whatever gets written will not be lost. This is extremely useful, but I think that it's going to be a bit difficult to generalize (as you point out creating a buffer may aggregate several requests, but what would be the size of the buffer + one should have to use some user defined file access streams).

I'd still go with the transactional model, as it has the advantage that the "user process" controls what's happening with its data. Indeed, it would be difficult for an external library to guess when to "commit" the accumulated changes (for example, it wouldn't be very nice if the SecuredFile to decides to "commit" a semi-generated XML file AND a runtime error kills the application just afterwards).

I've already used a 'transactional' file access functionality and I must say that it's extremely useful. You don't think about file integrity, about exceptions that could occur half way through some dirty file generation, etc. - if an error occurs you simply do a "rollback" and as if nothing happened.

Alex

Yep, you're right, in our case it seems that the "explicit" transactional model is the best solution.

thanks for your explications :slight_smile:

Ben

路路路

Le 16 sept. 07 锟 20:24, Benoit Pradelle a 锟絚rit :

Hi alexander, all,

Alexander Pelov a 锟絚rit :

Hi again,

I wanted to add that we could modify a little bit the FileAccessService adding the mentioned FileTransaction to it. This way, the moment one invokes getPersistentFile, the FileAccessService will look up for remaining "..tmpXXXXXX.original" and "...tmpXXXXXX.transactionbackup" files and perform a cleanup. The cleanup procedure would be the following:

1) If exist FILENAME + FILENAME.tmpXXXX.transactionbackup -> DISCARD FILENAME.tmpXXXX.transactionbackup
2) If exist FILENAME.tmpXXXX.original + FILENAME.tmpXXXX.transactionbackup -> RENAME FILENAME.tmpXXXX.transactionbackup to FILENAME and DISCARD FILENAME.tmpXXXX.original
3) If exist FILENAME + FILENAME.tmpXXXX.original -> DISCARD FILENAME.tmpXXXX.original
4) In all other cases keep FILENAME.

WDYT?

Alex

Le 16 sept. 07 锟 14:30, Alexander Pelov a 锟絚rit :

Hi everyone,

the question in hand is very interesting - what to do when an error occurs while replacing an existing (and potentially important) file. Maybe we can create a sort of a generic API that can emulate transactional behavior on top of a general FS. I am thinking about something like this (if we are following Java Transaction API http://java.sun.com/products/jta/):

File f = new File("/test.xml");
Transaction trans;
try {
聽聽聽聽trans = FileTransaction.beginTransaction(f);
聽聽聽聽// some actions with f
聽聽聽聽trans.commit();
}
catch(Throwable e) {
聽聽聽聽trans.rollback();
聽聽聽聽throw e;
}

Following JTA would require that the originating file is copied when beginTransaction is called. On commit() the backup file should be deleted, whereas on rollback() the backup file should replace the invalidated "f". Here, there are some potential issues, but I think that they can be addressed relatively easily. The default behavior would be to automatically rollback a transaction. This way, one may encapsulate all kinds of file operations in transactions and in case of unexpected exceptions, kernel panics, premature application termination, etc. data corruption would be kept to minimum.

A simpler approach would be to write all changes to a secondary file, and once done - replace the original. An example would be like:

File f = new File("/test.xml");
Transaction trans;
try {
聽聽聽聽trans = FileTransaction.beginTransaction(f);
聽聽聽聽File newFile = trans.getProtectedFile();
聽聽聽聽// write to newFile and potentially read from f
聽聽聽聽trans.commit();
}
catch(Throwable e) {
聽聽聽聽trans.rollback();
聽聽聽聽throw e;
}

An example behavior of the different functions could be:

getProtectedFile():
create a file "/test.xml.tmp03942940192.transactionbackup"

commit():
1) Rename "/test.xml" to "/test.xml.tmp01293093.original"
2) Rename "/test.xml.tmp03942940192.transactionbackup" to "/text.xml"
3) Delete "/test.xml.tmp01293093.original"

Of course, one would always have to access a file in a transaction (or at least call a function that would search for eventual "....original" and ".....transactionbackup" files and perform appropriate actions.

What do you think ?

Alex

Basically it's what I was thinking at but instead of using a transaction model I was thinking at a SecuredFile which extends File and by overloading basic I/O operations of the File class we could use these SecuredFiles really more easily and without any large modifications. However, it's just an idea for the moment, I don't really know if it is really possible.

The main drawback of this solution is that sometimes a lot of little write are done in an operation and it'll be difficult to determine when all these little write end. I was thinking on something like a buffer for all the operations and a writing thread which periodically commit the modifications. It's probably less efficient but this may avoid large code modifications.

I don't know what we should prefer ? I don't really know how many class will use these secured files and what they'll do with it...

Cheers,
Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@sip-communicator.dev.java.net
For additional commands, e-mail: dev-help@sip-communicator.dev.java.net