Recover sparse image after upgrading to Tahoe

An encrypted disk image is a good way to provide an extra layer of security. I keep my financial data (e.g., tax records) in one saved on my internal storage. I mount it when i need the files, eject it when I’m done.

There are several types of disk images. Standard images fully use their allocated space, i.e., if you create a 500 MB image, that’s the space it takes on your internal drive even if you only have 10 MB in it. ‘Sparse’ images have a maximum specified size but only use what they need, i.e., if you create a 500 MB image but only have 10 MB in it, it takes only 10 MB of your storage.

See:

Create a disk image using Disk Utility on Mac - Apple Support

With Tahoe, Apple updated the sparse image format. Some users have reported issues accessing sparse disk images in the old format on Tahoe, but the disk images mount fine on a Mac running Sequoia.

Regarding the disappearing file, if the disk image was deleted, a won’t open it. Hopefully you can locate the original image file (.dmg or .sparseimage) in a backup.

hollistonma wrote:

The original disc was in my Documents folder. The actual sparse image file/disc is missing from the Documents folder (and not found anywhere else on my hard drive).

The Tahoe update shouldn't delete any files in your documents folder. Was this the only file that was deleted?

Regarding disk image vs. sparse file, I believe it was actually a sparse disc b/c it would have to be mounted of me to access files/folders on the drive.

They both act the same way. The only difference is that a sparse image format is composed of many small files. The idea is that if you change the data, you only have to change a relatively small number of files. Plus, the disk image only takes up as much space as you have data. A 100 GB sparse image file with 100 MB of data only takes up about 100 MB of space. A 100 GB non-spare disk image file would need 100 GB of storage.

Unfortunately, all those extra files are also lots more opportunities for corruption. I've tried sparse image files a couple of times. My eventual corruption rate is 100%.

I'll see if I can access the files on my backups.

Also you need to verify if your backups have the rest of your data. Random files don't just randomly disappear after an OS update. Either you've lost all of your files, or there is something extra special about this one file.

Regarding it being a bad idea, I created it to add an extra layer of security for files/folders on my MBP. One has to enter a password to mount the drive in the first place. If this is a bad idea, what's the best way to provide another level of protection for sensitive information on my MBP?

If you want to encrypt your data, the best way to do that is with FileVault.

You can certainly add additional encrypted disk images if you want. But I strongly recommend that you keep the data somewhere else (that isn't a disk image) as a backup. The operating system is better about enforcing file system and encryption formats on the boot drive. Next best is an external, non-boot drive. Encrypted disk images (and even non-encrypted disk images) are the least reliable because people don't realize that Apple eventually stops supporting them. Then, unless you have a backup or an older computer, your data is gone forever. Encrypted sparse images are the least reliable of all.

Note that it's not for archival purposes....use the folders regularly to store items that a casual user of my MBP shouldn't be able to access.

I'm not sure what you mean by "casual user". There's no such thing. Someone who has access to your computer has access to all of your files. They may be unsophisticated, and not know how to access the encrypted data, but that's a limitation of the user, not the technology. Any rationale you might have about why they couldn't, or wouldn't want to, access your data also defeats your rationale for encrypting it in the first place.

There are two different things going on.

First of all. That item on your desktop was an alias. Where is the original? It's unfortunate if an alias gets broken, but aliases were never very reliable. Otherwise, they wouldn't have needed to build in an entire recovery user interface for it.

You referenced: "no 'files.sparseimage' file to be found". What were you actually doing there? Seems like you missed some steps. Where is the original file? That's a very important question. If you don't know, that's a very bad answer.

And finally, sparse images are a very bad idea indeed. If you can recover it, you should immediately move it to a newly created (non-sparse) disk image. Even regular disk images simply aren't reliable. You'll regularly need to recreate them every few years after they stop working. Hopefully you'll have an older system handy that can still open it. If you don't always plan to have an older system available, then any use of disk images is extremely dangerous. Apple regularly re-writes all the disk image logic and removes support for older versions. This can easily happen in a 5-7 year timeframe. Apple is not in the archival media business.

etresoft wrote:

Regarding disk image vs. sparse file, I believe it was actually a sparse disc b/c it would have to be mounted of me to access files/folders on the drive.

They both act the same way. The only difference is that a sparse image format is composed of many small files. The idea is that if you change the data, you only have to change a relatively small number of files.

Sounds like you're confusing a sparse bundle image with a sparse image. A sparse image is has a variable size, a sparse bundle divides that variable size into the required number of 8.4 MB bands.

https://eclecticlight.co/2023/01/12/which-disk-image-format/

I'm not sure what you mean by "casual user". There's no such thing. Someone who has access to your computer has access to all of your files.

I understand the OP's point. When my computer is at home, sometimes I will walk away from it leaving it unlocked. It will sit there for 10 minutes until the screensaver starts up, at which point it locks. Yes, I could change that to a shorter time, or just move the cursor to a hot corner before getting up from my chair. But it's a compromise between security and convenience. Additionally, sometimes I have a shopping link or something like that open, and I want someone else to use the browser to compare some things, and I don't want to breathe down their neck. But still, maybe I don't want my kids to know my annual salary, etc., so putting those tax returns in an encrypted disk image makes sense even though the whole internal storage is encrypted by Secure Enclave regardless of whether FielVault is on (which in my case, it is...but all that does is incorporate my login password into the cryptographic key, it doesn't result in 'better' encryption than would be present on a T2 or M series Mac without FileVault).

hollistonma wrote:

It was originally in my Documents folder as "Files.sparseimage". Still there on the backup but not on the hard drive after the MBP was upgraded to Tahoe.

Is it possible that iCloud is involved? If you had turned on iCloud Drive > Documents & Data, then this data would be in iCloud.

But iCloud is quite flaky during the upgrade process. Are there any other files you could check?

Unfortunately, there's no simple, easy answer here. An upgrade is not going locate one important file on your hard drive and delete it. Maybe there's something unique about that file, or maybe you've lost a bunch of files. But you're the only one who has the computer in front of them. All I can tell you is that certain things never, ever happen, and certain other things can and do happen on a regular basis.

The files wasn't singularly deleted. That doesn't happen. End of story.

Maybe you were trying something clever. Maybe iCloud is involved. Maybe your entire home directory is scrambled due to a dozen different things that seemed clever at the time. I just don't know. I do know that the upgrade didn't just delete one file and then upgrade the OS. That doesn't happen.

At this point it seems as if I should mount the file from back up and copy the contents to Tahoe. Not sure how to verify if other files are missing so perhaps I'll "retire" this backup in case I find other files missing.

Maybe inquire with the developers of that 3rd party backup tool.

hollistonma wrote:

My backups are made with Carbon Copy Cloner...verified that backup was successful.

I'm not sure what you mean by "verified that backup was successful". Does this verification process ensure that all files on the hard drive also exist on the backup? Are there any "optimizations" or "exclusions"? Did you perform this verification at the time you backed up the data? Or just now? Because it sounds like the file is gone now. So if the file didn't exist on the backup, then it would be verified because it doesn't exist on the disk too.

So far, you haven't answered the question of whether or not any other files are missing. It's possible that you've had some unknown, catastrophic upgrade failure and are just refusing to look. But it's more likely that you had excluded that particularly sensitive file from backups in the first place. Since you were using an alias, the original is elsewhere.

Most likely, the file is still somewhere on your hard drive. Perhaps you've forgotten where. And that old alias just didn't survive the upgrade process. I recommend you look harder to find it on the hard drive while you still can.

Yes, my hard drive is encrypted with FileVault. Will use an external, password protected USB drive once I get my files back.

Don't forget to keep a back up of that.

I think so, however what's the best way to verify? Or is the CCC verification sufficient?

I have no clue about 3rd party products like Carbon Copy Cloner.

Is there any way you can look at the backup to see if that one particular file exists on it?

neuroanatomist wrote:

Sounds like you're confusing a sparse bundle image with a sparse image. A sparse image is has a variable size, a sparse bundle divides that variable size into the required number of 8.4 MB bands.

I was referring to the fact that all disk images must be mounted to access the files and folders on the drive. The OP referenced both "sparse bundle" and "sparseimage".

I was referring to sparse bundles, but it really doesn't make much difference. The last time I had to deal with a corrupt disk image was like two weeks ago. And that was just a standard read-only archive of a folder. Nothing fancy in any way, other than the way it permanently killed the Finder on Sequoia, requiring a restart.

https://eclecticlight.co/2023/01/12/which-disk-image-format/

Please don't quote social media influencers at me. If I want obsolete, 3 year-old information, I can find that myself. If I want advice on how to sow fear, uncertainty, and doubt, or advice on how to contradict myself within the span of 24 hours, knowing my gullible readers will never notice, then I'm well-acquainted with that site.

I understand the OP's point. When my computer is at home, sometimes I will walk away from it leaving it unlocked. It will sit there for 10 minutes until the screensaver starts up, at which point it locks. Yes, I could change that to a shorter time, or just move the cursor to a hot corner before getting up from my chair. But it's a compromise between security and convenience. Additionally, sometimes I have a shopping link or something like that open, and I want someone else to use the browser to compare some things, and I don't want to breathe down their neck. But still, maybe I don't want my kids to know my annual salary, etc., so putting those tax returns in an encrypted disk image makes sense even though the whole internal storage is encrypted by Secure Enclave regardless of whether FielVault is on (which in my case, it is...but all that does is incorporate my login password into the cryptographic key, it doesn't result in 'better' encryption than would be present on a T2 or M series Mac without FileVault).

I don't know what situation the OP was describing.

Regardless of what level of trust you have at home, it's an established fact that children are immature and often irresponsible. I wouldn't let them have unsupervised access to my computer regardless of encryption levels.

The Secure Enclave doesn't provide any encryption on its own. It's just a secure co-processor, isolated from the rest of the system. If FileVault is turned off, then your data isn't encrypted. I understand the point you're trying to make about how the encryption works internally. But if the password is publicly available, then it's not encrypted. If FileVault is turned off, then the password is publicly available.

etresoft wrote:

The Secure Enclave doesn't provide any encryption on its own. It's just a secure co-processor, isolated from the rest of the system. If FileVault is turned off, then your data isn't encrypted. I understand the point you're trying to make about how the encryption works internally. But if the password is publicly available, then it's not encrypted. If FileVault is turned off, then the password is publicly available.

I'm not sure your information is current. Or maybe you know more about macOS and Apple hardware than Apple, but you state without FileVault the data are not encrypted and Apple states (Protect data on your Mac with FileVault - Apple Support):

"If you have a Mac with Apple silicon or an Apple T2 Security Chip, your data is encrypted automatically. Turning on FileVault provides an extra layer of security by keeping someone from decrypting or getting access to your data without entering your login password. If you use a Mac that doesn’t have Apple silicon or the T2 chip, you need to turn on FileVault to encrypt your data."

Elsewhere, Apple states (Volume encryption with FileVault in macOS - Apple Support):

"If FileVault isn’t turned on in a Mac with Apple silicon or a Mac with the T2 chip during the initial Setup Assistant process, the volume is still encrypted but the volume encryption key is protected only by the hardware UID in the Secure Enclave."

Apple clearly states the data are encrypted without FileVault for Macs with T2 or Apple Silicon (i.e., any Mac running Tahoe). Not sure why you disagree with the folks designing the hardware and writing the software.

Not sure what you mean by 'the password is publicly available'. The root cryptographic key for Secure Enclave is a 256-bit AES key that’s burned into each processor at manufacture, can’t be read by firmware or software, and is used only by the processor’s hardware AES Engine. How is that publicly available?

neuroanatomist wrote:

Not sure what you mean by 'the password is publicly available'. The root cryptographic key for Secure Enclave is a 256-bit AES key that’s burned into each processor at manufacture, can’t be read by firmware or software, and is used only by the processor’s hardware AES Engine. How is that publicly available?

It's right there in the part you quoted:

Turning on FileVault provides an extra layer of security by keeping someone from decrypting or getting access to your data without entering your login password.

So if you don't turn on FileVault, the implication is that someone could decrypt or get access to your data without entering your login password. All they have to do is follow the instructions at the end of these instructions: If you forgot your Mac login password

This is the same way it has always worked. You can reset a login account password and get access to almost all the data. The only data that would be inaccessible is data that was already encrypted, such as disk images or the keychain. Disk images provide even more security because, eventually, you won't be able to get into them even if you do know the password. 😄

etresoft wrote:

This is the same way it has always worked. You can reset a login account password and get access to almost all the data.

Thanks, I learned something today. But...wow. So Apple talks up their security, but anyone who finds your Mac can just reset the password and gain access to everything on it (with limited exceptions)? Technically, the data are encrypted and Apple says that because of Secure Enclave even if someone tries to remove the soldered-on SSDs and access them elsewhere that's prevented by Secure Enclave...but if they could remove the SSDs then they have the Mac, and all they have to do is boot in Recovery, open Terminal and type a single command. Again...wow. Holy gaping security hole, Batman.

Well, I have always used FileVault. But in the context of Tahoe where that's enabled by default, based on statements from others I've assumed that the internal drive was encrypted anyway so FV was basically superfluous. But the drive is technically encrypted but practically freely accessible, and I'll certainly stop repeating that poor advice.

neuroanatomist wrote:

So Apple talks up their security, but anyone who finds your Mac can just reset the password and gain access to everything on it (with limited exceptions)?

Apple's default security settings are plenty secure for most people. Most real-world security problems stem from people purposefully disabling those built-in security settings.

There are higher levels of security available, including FileVault, Lockdown, Find My Mac, and even more strict controls for enterprise customers.

Again...wow. Holy gaping security hole, Batman.

Well, not really. This is why I get so annoyed at social media influencers. They always repost theoretical "evil maid" exploits, funded by US government intelligence agencies, with dire warnings about North Korean and Chinese state-sponsored hacking.

But I have no doubt that the number of daily technical support requests that Apple fields for forgotten passwords far exceeds the total number of "evil maids".

Well, I have always used FileVault. But in the context of Tahoe where that's enabled by default, based on statements from others I've assumed that the internal drive was encrypted anyway so FV was basically superfluous.

Just look at posts here in the forums. How many people are complaining about password reset procedures? Now how many people complained when Tahoe accidentally enabled FileVault?