(Dieser Artikel ist auch auf deutsch verfügbar.)
This article refers to a predecessor article concerning Time Machine backups on network shares in Leopard.
In the comments on this article a thread in Apple’s support forums was mentioned which describes problems with Time Machine and network shares (thanks to John M for that remark).
To explain the problem you need to be aware of how sparse bundle images (the ones Time Machine uses for network backups) work. Sparse bundle images and sparse images are special forms of disk images with certain advantages. To say it in simple terms a disk image is just a file that is treated like a drive (e. g. a hard disk). The image file is contained in a real drive (locally or on the network) which I will call container.
You can mount the image and use it like a real drive. Mac OS X takes care of writing the data into the image file. In the simplest case the image is a file that has exactly the size of the virtual drive’s maximum capacity. If you create an image with 100 GB capacity you will have a 100 GB file even if the image only contains some MB of data.
And here the sparse images come into play. They occupy only the space the data in the image occupies. Therefore, you can create an image with 100 GB capacity which as a file is only a few MB big. The image file grows as it is filled with data.
However, there is a catch in it. Deleting files in the image does not result in a smaller image file. For example: If you create a 10 GB file in the image the image file will increase to a size of 10 GB as well. Free space in the image will be 90 GB. Afterwards you delete the file and the free space in the image will be 100 GB again. Without further intervention the image file however will keep its size of 10 GB*. Inside the image everything is in order. Thus, while using the image its file will at most keep its size, mostly grow in size (due to creating further files in the image), but never shrink by itself. This is one part of the problem.
The other part of the problem is a behaviour of Mac OS X which is actually really useful. Mac OS X does not prevent the user from creating a sparse image with a maximum capacity bigger than the free space or even the maximum capacity of the container. This may turn out to be a problem. For example take our fresh image with 100 GB capacity. Assume its in a container with 20 GB free space. The user might want to create a 30 GB file in the image which naturally will not work.
Mac OS X acts rather clever here. If an image is mounted Mac OS X checks how much free space is available in the container (here 20 GB). If this value is smaller than the theoretically free space in the image (here approx. 100 GB) Mac OS X will limit the free space in the image to the free space of the container. In our example: The image has a maximum capacity of 100 GB and 20 GB free space even if there are no files on the image!
In itself this behaviour makes sense. In combination with Time Machine it can lead to fatal consequences. For example: In our previously mentioned image you create an 18 GB file. The image file will grow to a size of 18 GB. Afterwards free space in the container and the image will be 2 GB. Now you delete the 18 GB file. Theoretically the free space in the image will increase by 18 GB.
Now both above mentioned effects come into play. The image file will keep its size of approx. 18 GB. Hence the free space in the container will still be 2 GB. Therefore, the free size in the image will as well be only 2 GB. In this case deleting a file in the image does not create further free space in it. If you want to create a new 18 GB file Mac OS X will not allow that. The problem can be avoided by always keeping enough free space in the container. In our case we would have to free at least 16 GB in the container*.
Assume in our initial situation above free space in the container would not be 20 GB but 120 GB. Mac OS X will then always specify the free space regarding only the image (100 GB capacity). Deleting the 18 GB file will indeed increase the free space to 100 GB. This also shows how fatal it is, if the maximum capacity (or the maximum free space respectively) of the container is less than the capacity of the image. In this case the free space of the image will always be specified regarding the container. At some point the image will become unusable because the image file grows inexorably while the apparent free space of the image shrinks.
This is exactly the trap Time Machine falls into, if you do not pay attention to the free space on the network drive. Assume the Time Machine image has less free space than the next backup needs. Time Machine will then automatically delete old backups to free up some additional space. If the free space on the network share is greater than that in the image (i. e. the free space in the image is not affected by the free space in the container) everything will work smoothly. With every deleted old backup the free size in the image will grow a little.
However, if the free size in the image is limited by the container deleting backups will not create additional free space. Time Machine will futilely keep on deleting all backups but the last one in order to clear space. Afterwards it will fail due to lack of free space (like in the above mentioned support thread).
Possible trouble-shootings:
- Make sure there is enough free space on the network drive in the first place and keep it free permanently (i. e. at least the maximum size of the image). The image capacity must not exceed the free space of the container.
- Regularly free up additional space on the network share so that there is always enough free space for the next backup. In the end this leads to option 1 and needs more attention. However, while the backup image is still small you can use the free space for other things. The image capacity must not exceed the maximum free space of the container.
- Regularly delete old backups in Time Machine manually and reclaim freed space via hdiutil. Most complicated but also the most flexible solution.
Disclaimer: I have never experienced these problems in Time Machine myself. Therefore, I cannot guarantee that the above stated solutions actually work. In my case there is enough free space on the network share so that backups are created smoothly. Your own reports and experiences are very appreciated.
You should always be aware that Time Machine backups on network shares are not officially supported by Mac OS X (except Time Capsule). In doubt you should not trust a solution like this.
* There is a possibility to reclaim unoccupied space via hdiutil. However, as Time Machine does not utilize it I will ignore this possibility here.
In this very interesting article you say “The image capacity must not exceed the free space of the container.” Now consider the situation of two or more Macs using Time Machine to back up to images on a shared server. In this case should the rule be “The total image capacity of all images should not exceed the free space of the container”?
@Dana: Your assumption is right.
Hallo Flo,
Du solltest vor dem Artikel noch einen Hinweis auf die deutsche Übersetzung einfügen. Habe mich soeben durch das Englisch gekämpft und hernach erst den deutschen Artikel entdeckt.
Danke für die Infos!
Hi there,
Thanks for the clear explanation of how the sparse images make Time Machine delete all the old backups.
I’ve been running Time Machine against an SMB share for a while now (since the beginning of January) and this was my last outstanding issue. However, I was pleasantly surprised to find out that the latest Time Machine and AirPort Updates v1.0 fixes the problem. Apple have changed Time Machine so it now compacts the sparse bundle after it deletes each old backup.
Cheers,
Adam
@Adam: That’s indeed good news. Made my article somewhat obsolete though.
Not obsolete if you have this problem (which I ran into and still don’t have a fix for it. The free space on the Time Machine drive didn’t give back its free space after deleting the backup data my moving it to the trash and emptying it. Normally I would simply reformat the drive, but it has some data on it that I need.
While I’m glad the “new” software from Apple fixes the from the problem from this point forward, it’s not clear or likely that it will go box and update the free space to agree with the Capacity - Used = Free. In other words, despite deleting Time Machine bakcup files (/Backups.backupb//{folder1,…,folderN), when free space was down to 40 GB (on a 600 GB drive), with 200 GB of other files, the free space is still only 40 GB! This suggests not using a drive for Time Machine to keep any other data, at the least.
So I’ll find another drive on which to move the data I need (200 GB), reformat, and then think about using Time Machine again. While the visual part of Time Machine is a nice way to understand the timeframe backups, there’s no easy way to reclaim the free space or need to understand why, without some technical background.
Thanks very much for posting the findings, information, and fixes, and good news for those who want to continue using Time Machine. I’m now going to use a Time Capsule and not put anything else on it. If Apple had spent just a little more time testing these boundary conditions, instead of leaving it to the customer, we all may not have had to suffer the discovery and lost of time and data. Pushing out a complex feature like this without fully flushing out such a major problem is just silly.
PS Of course after posting “there’s no easy way…”, I found this on Mac OS X Hints (http://www.macosxhints.com/article.php?story=20071101005330768)
10.5: How to delete objects from Time Machine backups
Wed, Nov 14 ‘07 at 7:30AM PST • Submitted by iGrouch
This is a something that Apple has hidden away, leaving some Mac users out there to go through all sorts of lengths to remove items from their Time Machine backups.
The solution is really easy, though.
Open time Machine. Go to the top level — the first window; the one with your hard drive. Navigate to the backup you want to delete and use the Actions menu (gear icon) in the Finder’s toolbar. It has an option to delete the backup. Also, when using the Actions menu, with a file or folder selected, there is an option to delete all instances of the file/folder from all backups.
I’m finding the wording in the original post a bit unclear. Let me try to restate:
It sounds like the formula for calculating free space in a mounted
sparse image is flawed in Leopard. Instead of adding the free
space from the containing volume to the free space inside the
image, Leopard uses the _minimum_ of free space in the Volume
and the mounted image. I can imagine that the erroneous free
space calculation results in certain processes running out of
space when there is really enough space. Perhaps higher level software is using a correct free space calculation and then gets stopped while trying to write a file for which there is really enough free space. Maybe the lower level image management/driver code likely throws an error thinking there isn’t enough space?
Does this make sense?
My problems has to do with trying to maintain backups from two different desktop macs on the same hard drive which is network mounted via an Airport Extreme. Perhaps the free space showing in Finder is leading me to believe there is room for all backups while Time Machine thinks it’s out of space due to this free space calculation error?
hi
you mention that
“There is a possibility to reclaim unoccupied space via hdiutil. However, as Time Machine does not utilize it I will ignore this possibility here.” and “Regularly delete old backups in Time Machine manually and reclaim freed space via hdiutil. Most complicated but also the most flexible solution.”
would you have a hint as to how this is done? i.e. what is the terminal command to use to free that space? thanks a lot, as i am not very versed in terminal…
sorry, hab gleich gesehen das das problem im update gelöst ist…nevermind.
I am confused: I created a sparsebundle from old backups from a USB-attached harddrive to use wirelessly via AirPort Extreme (based on this: http://rolf.haynberg.de/?p=83). This sparsebundle on the network drive has a capacity of 465.4GB (I guess this is because my old hard drive was 500GB).
My new hard drive (the one which has the sparsebundle on it and is attached to the AirPort Extreme) is 1TB. What is going to happen if my backups add up to the capacity? Is time machine expanding the capacity, up to the capacity of the drive? Or is it starting to delete old backups although there is 0.5TB still available on the drive?
Please help!
@Andre: If the sparsebundle has a max capacity of about 500 GB it can be smaller but will never become bigger even if there is lots of space available on the drive.
You could try to resize the sparse bundle via Disk Utility or create a new, bigger sparse bundle and copy the files (although I’m not sure if this works).
Thanks for the quick reply. How do I create a new sparsebundle (i.e. which options do I have to use in Disk Utility)?
And, how could I try to resize the sparsebundle?
Thanks again!