Screen Share Mac mini on imac27”

The only lag-free way to do that, Target Display Mode (TDM), is not supported on 2017 iMacs nor any Mini made after 2019. The very restrictive hardware and operating system conditions that TDM required are laid out here:

Use your iMac as a display with target display mode - Apple Support

Apple started the depreciation of TDM 10 years ago and threw the last shovel of dirt in its face in early 2020.

There are third-party "solutions" but they are mostly laggy. One, Luna Display, goes as far as placing this disclaimer on its main support page:

The red is their emphasis, not mine.

That says to me that it is unsuitable for making an iMac a primary display.

"On another forum somebody suggested..."

I'm sure they did. We see that misinformation a lot about iMacs being the wunderkind of computers. It started with the work-from-home requirements forced by COVID. Web writers, lacking editorial assistance and, it seems, the research skill to take two minutes to look up the Apple support article linked above, wrote thousands of articles claiming that any iMac and TDM could dot everything form working with any non-Apple device to parting the Red Sea.

A decent free-standing eternal monitor will always work best without lag, use less power, and allow full display options.

Not sure what you mean by "only get basic jpeg images".

What are you doing to get these images exactly?

Or are you saying it's very slow with your windows PC and you just get still images instead of flowing video?

Anyway, the option to Screen Share between the iMac and Mac mini exists, but you cannot have it automatically connect to the Mac mini, and display its screen, you need to initiate the screen sharing through Finder manually on the iMac.

And you'll still need a screen and a keyboard and mouse to set up the Mac mini and set up sharing the first time. Once set up it can be used headless, but it won't just magically work out of the box. Also depending on your network the screen use may be a bit laggy in many cases. Specially if you need it for real time video editing or games.

The other option is to use AirPlay to Mac, but like Screen Sharing, you still need access to the Mac mini to set that up, and initiate the option each time from the Mac mini and will still introduce lag.

Stream video and audio with AirPlay - Apple Support

There really is no practical way of using the iMac's screen for the Mac mini unfortunately.

Being a fellow photographer, editing image using any remote desktop app (macOS Screen Sharing, LunaDisply, VPN) is not a great idea.

One the work over a network so your responsiveness is going to be limited.

Second, they all use some manor of lossy compression at not a very great bit depth so the image you edit with such a method will likely not look anything like the actual result you would get having it displayed directly on a monitor. While they may be more than acceptable for web browsing (could also be an issue since web browsing is using network bandwidth), text editing, spread sheets, etc., It is really not an acceptable solution for graphics intensive tasks.

Since you already know the limitations Microsoft option, there are not any other solutions that are not much better.

Best thing is to find a good home for the 2017 iMac and get another display. There are many 4K displays available that are just as good as the iMac and there are also 5K displays that will fit the bill as well.

The Remote Desktop likely reduces the color depth and image quality but does not actually convert the file you are working on, to a jpeg.

If the file is a raw file or a tiff or whatever it stays as that, its just that what you see through remote desktop is a lower quality representation of that but is not a jpeg.

The file is the same one, it's just what you see may be at less quality and color depth.

Same thing will happen with the Mac. Editing large his resolution images remotely as has been said, is not a good idea.

Get an actual display for that.

You can do that. However you will need an External Display or HDMI TV, keyboard and mouse to initially setup the Mac mini and Screen Sharing.