Bug report using Find command in Terminal with md5 getting duplicate md5 hash tags for completely different photos

Question

Level 1

10 points

Bug report using Find command in Terminal with md5 getting duplicate md5 hash tags for completely different photos

I am using the find command in terminal for getting unique md5 tags for all my photos in my personal directory (not Photos APP)

the commands I am using are:

find "$(pwd)" -type f -print0 | xargs -0 md5 -r > md5listing.txt

and

find "$(pwd)" -type f $ -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" -o -iname "*.gif" -o -iname "*.tiff" -o -iname "*.bmp" -o -iname "*.heic" $ -print0 | xargs -0 md5 -r 2>/dev/null > md5listing.txt

While I was eyeballing the listed output from the Cat md5listing.txt command I noticed that there were a bunch of entries that had identical md5 tags??? I checked the actual photos (some) and they were different photos. I copied a small subset of the photos over to a separate directory (in the interest of submitting a bug report) and lo and behold the md5 tags were unique - I went back to the original folder and re-entered the command and this time the duplicate md5 tags moved to a different range of photos. I performed the same test on my new MacBook pro (M4) 15.3 Sequoia and got the same bug.

It always seems to be the same identical MD5 hash ===>> 3ccbfb1c186cf9b14d4248a6b5912c11 while the photos have different images

The folder that I am testing in has 620 photos.

I have submitted a bug report, however, the text field did not allow enough space... I guess that's a bug too :)

MacBook Pro 16″, macOS 15.3

Posted on Apr 17, 2025 11:19 AM

Reply

Answer 1

Top-ranking reply

etresoft

Level 9

56,326 points

Apr 17, 2025 2:31 PM in response to hgtefla

Those are metadata files. Note the "._" prefix. You are using a foreign filesystem like MS-DOS or something. It can't support macOS metadata so it writes an hidden file with the metadata. But the metadata is likely identical for many of the files.

Reply

Answer 2

Barney-15E

Level 10

122,126 points

Apr 17, 2025 7:27 PM in response to hgtefla

The comment was in reference to your reply to Etresoft about the metadata files. It shouldn’t be writing those metadata files (._filename) to a macOS extended volume. It should only write those to one of the FAT formats. The only reason I could see them appearing there is if you copied the drive contents from a FAT formatted drive.

Again, those are not images and may very well contain exactly the same content. They are also so small that a hash collision is certainly possible.

There appears to be nothing wrong with the MD5 hash algorithm.

Reply

Answer 3

VikingOSX

Level 10

123,254 points

Apr 18, 2025 4:33 AM in response to hgtefla

The dot_clean(1) command will remove any dot_underbar file on the drive it is pointed at:

/usr/sbin/dot_clean -m /Volumes/mounted_drive_name

You can simplify your find syntax in this manner:

find -E "${pwd}" -type f -iregex '.*\.(jpe?g|png|gif|tiff|bmp|heic)' | xargs -P 0 -I{} md5 -r "{}"

That -P 0 will run as many processes as it can to complete the work and the trailing "{}" will handle filenames with spaces in their name. I omitted the redirect to a filename and you can add that.

Reply

Answer 4

Kurt Lang

Level 9

68,508 points

Apr 18, 2025 11:28 AM in response to hgtefla

I always use both the m and n switches

-m Always delete dot underbar files.

-n Delete dot underbar file if there is no matching native file.

The second just in case there's an orphaned ._ file on the drive for some reason.

You also don't need to type in the target path. Drag and drop is much easier, and the OS will automatically put in the proper breaks for target names with spaces in them.

So, I type in:

dot_clean -mn (with a space after the n), then drag and drop the icon of the flash drive on the desktop into the Terminal window. Press Enter. Done.

Reply

Answer 5

hgtefla Author

Level 1

10 points

Apr 17, 2025 2:21 PM in response to Barney-15E

Hi Barney-15E,

Many thanks for the response. I filed the bug report vis apple.com/feedback however, there was not enough space for the complete text to get it logged completely. However, I have been testing and noticed that there were .mov files in the folder along with the photos. I thought that the command ===>>

find "$(pwd)" -type f $ -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" -o -iname "*.gif" -o -iname "*.tiff" -o -iname "*.bmp" -o -iname "*.heic" $ -print0 | xargs -0 md5 -r 2>/dev/null > md5listing.txt

would have omitted the .mov files - it didn't

I have wanted to keep the .mov files with the photos so that they would be related in time and place (Travel etc.). I guess I will have to rethink my strategy.

Here's a snippet - I have checked that the photos are actually different images

teba@tebas-iMac 411_Misc_Photos % find "$(pwd)" -type f $ -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" -o -iname "*.gif" -o -iname "*.tiff" -o -iname "*.bmp" -o -iname "*.heic" $ -print0 | xargs -0 md5 -r 2>/dev/null > md5listing.txt

teba@tebas-iMac 411_Misc_Photos % cat md5listing.txt

3ccbfb1c186cf9b14d4248a6b5912c11 /Volumes/SDEXPRO/data/photos/411_Misc_Photos/._2007_03_411_Misc_Photos_07062.JPG

3ccbfb1c186cf9b14d4248a6b5912c11 /Volumes/SDEXPRO/data/photos/411_Misc_Photos/._2007_03_411_Misc_Photos_07063.JPG

3ccbfb1c186cf9b14d4248a6b5912c11 /Volumes/SDEXPRO/data/photos/411_Misc_Photos/._2007_03_411_Misc_Photos_07064.JPG

3ccbfb1c186cf9b14d4248a6b5912c11 /Volumes/SDEXPRO/data/photos/411_Misc_Photos/._2007_03_411_Misc_Photos_07065.JPG

I never hear back from apple so I thought I'd post here so that some of the helpful and knowledgeable people like you would be aware of this issue.

Cheers

Reply

Answer 6

hgtefla Author

Level 1

10 points

Apr 17, 2025 2:25 PM in response to hgtefla

A minor correction - the find command did not printout the .mov files in the md5listing.txt file, however, I still got the corruption. If I do this in a folder where I have removed the .mov files it works perfectly.

Reply

Answer 7

Barney-15E

Level 10

122,126 points

Apr 17, 2025 3:14 PM in response to hgtefla

hgtefla wrote:

Hi Etresoft,

Thanks for responding - No I'm using a Mac Mini (Monterey) and MacBook Pro (M4) Sequoia 15.3.

Cheers

Your computer isn’t the factor. Get info on the drive and what does it say for the format?

Reply

Answer 8

hgtefla Author

Level 1

10 points

Apr 17, 2025 6:43 PM in response to Barney-15E

Hi Barney-15e,

Many thanks for the clarification.

You are absolutely right. It works on my system drive disk utility shows it as APFS System Volume - APFS (macOS 13.4.1 (22F82)

My external Sandisk shows as USB External Physical Volume • Mac OS Extended (Journaled). Where it doesn't work.

What should my external disk be formatted as - all the doc tells me to use Mac OS Extended (Journaled)

I appreciate any direction you may offer.

Again thanks.

Reply

Answer 9

Barney-15E

Level 10

122,126 points

Apr 17, 2025 2:02 PM in response to hgtefla

Where did you file the bug report?

This is a user to user tech-support forum. We can’t do anything with a bug report, nor will Apple see it.

if you were talking about not being able to enter the full text in this forum, you have to use the additional text field, but it is still somewhat limited.

Reply

Answer 10

hgtefla Author

Level 1

10 points

Apr 18, 2025 2:51 AM in response to Barney-15E

Yes, in the past I used a Windows/Xcopy for backing up my mac.

Is it safe for me to remove all ._filenames?

I'm using rsync to backup now: rsync -auv /Volumes/SDEXPRO/data/* /Volumes/WD2T/data

is that correct?

Again many thanks for your help.

Reply

Answer 11

hgtefla Author

Level 1

10 points

Apr 18, 2025 3:41 AM in response to Barney-15E

Hi Barney-15E I found your post regarding ._files and in it you mention the dot_clean command. I have tried that command with -m I also tried it with -n. As far as I can tell the command is making no difference. I tried to delete a file:

rm ._411 on a cold and frosty morning_0566.jpg

rm: ._411: No such file or directory

How do I get rid of these files?

Thanks for your continued support

Reply

Answer 12

Barney-15E

Level 10

122,126 points

Apr 18, 2025 5:03 AM in response to hgtefla

rm: ._411: No such file or directory

How do I get rid of these files?

Thanks for your continued support

You didn’t escape the spaces. White space is a separator in Unix, so you passed it seven file names, not one. You can use quotes around the name or escape the space with \

rm ._411\ on\ a\ cold\ and\ frosty\ morning_0566.jpg

Without an example of what you tried with dot_clean I don’t know why it didn’t work, but VikingOSX gave a usage example.

Reply

Answer 13

hgtefla Author

Level 1

10 points

Apr 17, 2025 3:03 PM in response to etresoft

Hi Etresoft,

Thanks for responding - No I'm using a Mac Mini (Monterey) and MacBook Pro (M4) Sequoia 15.3.

Cheers

Reply

Answer 14

hgtefla Author

Level 1

10 points

Apr 18, 2025 4:53 AM in response to VikingOSX

Hi Viking,

Many thanks for the response. I was not using the full path I was just doing the dot_clean -m in the directory and it was failing. With the full path it works.

Many thanks.

Reply

Answer 15

hgtefla Author

Level 1

10 points

Apr 18, 2025 11:02 AM in response to hgtefla

Hi Etresoft, Barney-15E and Viking,

As always - many thanks for all your help. All of you knowledgeable contributors really help us neophytes get further on.

All the best.

Cheers

Reply