I tried to turn off iCloud Photos - Bad mistake, some photos missing.

I use iCloud Photo Library. Mac and iPhone were synced to the cloud library. On the iPhone I use the optimise iPhone storage option, on the Mac I do not. The library comes to about 350GB containing 40,000 photos. The other day I decided to save a bit of money I would just store the library on HDDs and use Time Machine to back up the system. I already did this anyway and not storing stuff in the cloud would save me the £8 a month.

To start I made a backup via Time Machine to a HDD, I then deleted my photos in the library and turned off iCloud photos. So far so good. The space used on iCloud reduced to almost nothing. (the HDD was disconnected during this process) I then copied back my photo library from the Time Machine backup. Now remember I never used the optimise storage option on my Mac, and the library appeared as 350GB on the Time Machine backup. Once I restored the library, I opened it to find all the photos were gone, and the library was actually only 4GB in size.

Luckily I had an additional backup from January, so I instead restored this version of my photo library to the Mac and this one was full, up until January. Again so far so good, lost a month or so of photos but nothing important, except when I opened photos today, the library had been modified again, all the albums are gone and some photos are missing which were there before. iCloud photo is turned off. On my iPhone again I turned off iCloud Photos but every time I try and send a photo it has to pause and download as if the photo is stored on the cloud.

Does anyone have any idea how to break the link to the cloud? to confirm iCloud Photos is turned off. If there is no way, I may have to abandon photos app and just store photos in folders.

Thanks.