These questions are about Mirror backups, in particular about the algorithm used by FBackup to decide whether a file on the source drive has "changed" (and thus needs to be copied to the destination drive).
1. If the source file's Modified Date has changed but its content and size have not changed, will FBackup overwrite the copy on the destination drive or will it simply adjust the Modified Date of the copy? (The latter is more efficient.)
2. If the source file's content has changed but its size and Modified Date have not changed, will FBackup overwrite the copy on the destination drive, or will it be fooled into thinking the copy is already up to date?
3. If the source file's filename has changed but its size, Modified Date and content have not changed, will FBackup overwrite the copy on the destination drive or will it simply rename the copy? (The latter is more efficient.)
4. If the source file has been moved to a different folder (that's included in the source specification) but its filename, size, Modified Date and content have not changed, will FBackup copy the file to the different folder on the destination drive and delete the old copy from the old folder, or will it simply move the copy to the different folder? (The latter is more efficient.)
I may have some follow-up questions.
How does FBackup decide whether a file has changed?
-
Adrian (Softland)
- Posts: 1993
- Joined: Thu May 23, 2013 7:57 am
Hi,
The comparison criteria used by FBackup includes: date modified, date created, read-only, archived, hidden and system.
1. Yes, FBackup will overwrite the file. Please note if the size is the same, it doesn't mean the file was not modified. Try to compare two text files with a single different letter inside. The size is the same but the content could be different.
2. The file will be overwritten.
3. The file will be backed up again with the new name. FBackup does not track your actions to have a history of that file. It just sees a new file and will back it up.
4. The moved files will be backed up again. The same reason as above.
The comparison criteria used by FBackup includes: date modified, date created, read-only, archived, hidden and system.
1. Yes, FBackup will overwrite the file. Please note if the size is the same, it doesn't mean the file was not modified. Try to compare two text files with a single different letter inside. The size is the same but the content could be different.
2. The file will be overwritten.
3. The file will be backed up again with the new name. FBackup does not track your actions to have a history of that file. It just sees a new file and will back it up.
4. The moved files will be backed up again. The same reason as above.
Do you know you can monitor your backups remotely with Backup4all Monitor? You can read more here: https://www.backup4all.com/backup4all-monitor.html
I have some follow-up questions about Mirror backups, prompted by the answers provided in Admin's post above.
1. If a source file's "read-only" or "archived" or "hidden" or "system" attribute bit has changed but its "date modified" and "date created" have not changed, does FBackup overwrite the destination copy or does it simply change the corresponding attribute bit(s) of the destination copy? (The latter would be faster.) If the former, why not the latter?
2. Admin's 2nd answer above implies FBackup will detect change of a source file's content even when none of the source file's attributes (such as "date modified") have changed. How does FBackup determine that the source file's content has changed?
3. What info is stored in the catalog file and how is it used? None of the comparison criteria listed above by Admin--date modified, date created, read-only, archived, hidden and system--require storage in a catalog file because they can be read directly from Windows' file system. (One reason I'm asking about the catalog file is that FBackup will unnecessarily overwrite all destination files--which takes many hours--if the catalog doesn't exist or if the user modifies some properties of the backup job such as the destination folder. Another reason is that if the catalog file contains a hash value for each destination file, FBackup could use the hash values as described in question #5 below.)
4. Admin's list of comparison criteria (see above) does not include "file contents" or "filename," which means Admin's 2nd and 3rd answers (see above) appear inconsistent with Admin's list of comparison criteria. Is the list incomplete? If so, what is the complete list of comparison criteria that FBackup uses to determine when a file needs to be re-copied to the mirror destination?
5. Admin wrote above that the reason FBackup cannot simply rename or move a destination file to match a renamed or moved source file is because FBackup does not track "renaming" or "moving" actions. Can't it be done without tracking such actions? Here is a way it can be done, which assumes only that the catalog file maintains a hash value for each destination file: Suppose, for example, that the user renamed a source file. During the mirror operation, FBackup can see the corresponding destination file with the old filename, which tentatively appears to no longer exist in the source, and FBackup can see the source file, which tentatively appears not to exist yet in the destination. If the pair of files have the same size, FBackup can compare their hash values to determine whether their contents are the same, and if their contents are the same then the destination file can simply be renamed (instead of deleted and backing up the source file again). So, at the beginning of the mirror operation--before copying or deleting any files--FBackup would construct a list of source files that tentatively appear not to exist yet in the destination, and a list of destination files that tentatively appear to no longer exist in the source. These two lists can be quickly sorted by filesize, and FBackup can compare the hash values of all pairs of same-size files to determine whether their contents are the same. When identical, the pair of files would be removed from the lists and the destination file would be renamed or moved to match the name and location of the source file. For efficiency, only files larger than some threshold would need to be compared. (The optimal size threshold should be based on the speed of the relevant disk access operations.)
6. If the user modifies or deletes a destination file, will FBackup detect that during the mirror operation, or will FBackup incorrectly assume the catalog file's data about the destination files is still correct?
Regards!
1. If a source file's "read-only" or "archived" or "hidden" or "system" attribute bit has changed but its "date modified" and "date created" have not changed, does FBackup overwrite the destination copy or does it simply change the corresponding attribute bit(s) of the destination copy? (The latter would be faster.) If the former, why not the latter?
2. Admin's 2nd answer above implies FBackup will detect change of a source file's content even when none of the source file's attributes (such as "date modified") have changed. How does FBackup determine that the source file's content has changed?
3. What info is stored in the catalog file and how is it used? None of the comparison criteria listed above by Admin--date modified, date created, read-only, archived, hidden and system--require storage in a catalog file because they can be read directly from Windows' file system. (One reason I'm asking about the catalog file is that FBackup will unnecessarily overwrite all destination files--which takes many hours--if the catalog doesn't exist or if the user modifies some properties of the backup job such as the destination folder. Another reason is that if the catalog file contains a hash value for each destination file, FBackup could use the hash values as described in question #5 below.)
4. Admin's list of comparison criteria (see above) does not include "file contents" or "filename," which means Admin's 2nd and 3rd answers (see above) appear inconsistent with Admin's list of comparison criteria. Is the list incomplete? If so, what is the complete list of comparison criteria that FBackup uses to determine when a file needs to be re-copied to the mirror destination?
5. Admin wrote above that the reason FBackup cannot simply rename or move a destination file to match a renamed or moved source file is because FBackup does not track "renaming" or "moving" actions. Can't it be done without tracking such actions? Here is a way it can be done, which assumes only that the catalog file maintains a hash value for each destination file: Suppose, for example, that the user renamed a source file. During the mirror operation, FBackup can see the corresponding destination file with the old filename, which tentatively appears to no longer exist in the source, and FBackup can see the source file, which tentatively appears not to exist yet in the destination. If the pair of files have the same size, FBackup can compare their hash values to determine whether their contents are the same, and if their contents are the same then the destination file can simply be renamed (instead of deleted and backing up the source file again). So, at the beginning of the mirror operation--before copying or deleting any files--FBackup would construct a list of source files that tentatively appear not to exist yet in the destination, and a list of destination files that tentatively appear to no longer exist in the source. These two lists can be quickly sorted by filesize, and FBackup can compare the hash values of all pairs of same-size files to determine whether their contents are the same. When identical, the pair of files would be removed from the lists and the destination file would be renamed or moved to match the name and location of the source file. For efficiency, only files larger than some threshold would need to be compared. (The optimal size threshold should be based on the speed of the relevant disk access operations.)
6. If the user modifies or deletes a destination file, will FBackup detect that during the mirror operation, or will FBackup incorrectly assume the catalog file's data about the destination files is still correct?
Regards!
-
Adrian (Softland)
- Posts: 1993
- Joined: Thu May 23, 2013 7:57 am
Hi,
1. The file will be overwritten, this is how the application was designed to work.
2. FBackup uses CRC to determine which files were changed.
3. The backup catalog includes information about the backed up files, settings and configurations. All options visible in Backup Properties are stored in catalog. The backup job can be recreated using the catalog.
4. Just because there isn't an option visible in Backup4all Properties->Filters, it doesn't mean the filters does not exist in application. You can try our commercial edition: Backup4all and you'll have access to all options and settings.
5. The ideea cannot be applied because the application was not designed to work this way and because it involves very long backup times. You need to think the following situation: you have 1,000,000 files in sources and you change the destination. You will also have 100,000 files new, 100,000 removed and few more renamed. That will take a very long time comparing each source file with the files in destination in order to match them in pairs.
6. FBackup uses the information stored in catalog to determine which files will be backed up. It does not look in destination for missing files. Manually deleting a file from destination is not a good choise anyway as you would be without backup from the deletion moment until you run again the backup.
1. The file will be overwritten, this is how the application was designed to work.
2. FBackup uses CRC to determine which files were changed.
3. The backup catalog includes information about the backed up files, settings and configurations. All options visible in Backup Properties are stored in catalog. The backup job can be recreated using the catalog.
4. Just because there isn't an option visible in Backup4all Properties->Filters, it doesn't mean the filters does not exist in application. You can try our commercial edition: Backup4all and you'll have access to all options and settings.
5. The ideea cannot be applied because the application was not designed to work this way and because it involves very long backup times. You need to think the following situation: you have 1,000,000 files in sources and you change the destination. You will also have 100,000 files new, 100,000 removed and few more renamed. That will take a very long time comparing each source file with the files in destination in order to match them in pairs.
6. FBackup uses the information stored in catalog to determine which files will be backed up. It does not look in destination for missing files. Manually deleting a file from destination is not a good choise anyway as you would be without backup from the deletion moment until you run again the backup.
Do you know you can monitor your backups remotely with Backup4all Monitor? You can read more here: https://www.backup4all.com/backup4all-monitor.html
Re: How does FBackup decide whether a file has changed?
I have a related question....
When I copied a bunch of files to my hard drive from another computer, FBackup did not backup those files. It remained unbackedup until I edited them.
Is this because FBackup is not comparing source and destination folders?
Never mind.....I solved it. FBackup will not copy those types of files during a forced run. It only copied them during the scheduled run which I have set @ 4 hrs daily.
When I copied a bunch of files to my hard drive from another computer, FBackup did not backup those files. It remained unbackedup until I edited them.
Is this because FBackup is not comparing source and destination folders?
Never mind.....I solved it. FBackup will not copy those types of files during a forced run. It only copied them during the scheduled run which I have set @ 4 hrs daily.
-
Adrian (Softland)
- Posts: 1993
- Joined: Thu May 23, 2013 7:57 am
Re: How does FBackup decide whether a file has changed?
Hi,
It could be the user is different when manually run the backup and it does not have enough permissions to access those files.
It could be the user is different when manually run the backup and it does not have enough permissions to access those files.
Do you know you can monitor your backups remotely with Backup4all Monitor? You can read more here: https://www.backup4all.com/backup4all-monitor.html