How do I connect a Mac Studio to an iMac through AirPlay

Your iMac 2017 predates the supported hardware for AirPlay to Mac:

Continuity features and requirements for Apple devices - Apple Support

It simply WILL NOT WORK! Please carefully read Target Display Mode and you will see what combinations will work. Your 2017 and Mac Studio are both too new to support Target Display Mode. If your Mac Studio needs a display, please buy one. This is:

• Simple setup
• No expensive cabling required
• Dead reliable
• Inexpensive, you can easily find 27" displays for <$100.

IF you are worried about the planet and don't want to toss the iMac in the trash then use one of Apple's free recycling programs as explained in Apple Recycling Programs

Your M1 MacBook Air is new enough to be an AirPlay Receiver for AirPlay to Mac. Your 2017 iMac isn't.

AirPlay from Macs to Apple TV set-top boxes was around long before the AirPlay to Mac feature came out. That is probably why your 2017 iMac can take advantage of your M1 MacBook Air's ability to be an AirPlay Receiver.

As for the replies mentioning Target Display Mode – you didn't ask about that, but the replies are correct in saying that your 2017 iMac does not support it.

bjoernar wrote:

I am aware of Target display Mode, but since it is not supported on my iMac, I cannot use it. Since the Mac Studio can share screen with the MacBook Air, it seems to me that the iMac 2017-model is too new to use Target Display, and too old to use Airplay?

That is correct. Your 2017 iMac is:

Too new to support Target Display Mode

Reference: Use your iMac as a display with target display mode - Apple Support

and

Too old to support being an AirPlay Receiver for AirPlay to Mac

Reference: Continuity features and requirements for Apple devices - Apple Support

The funny thing is that I can use Airplay with expanded display from my iMac to my MacBook Air and from MacBook Air to iMac and from iMac to Mac Studio.

Your Mac Studio is new enough to be an AirPlay Receiver for AirPlay to Mac. If you can AirPlay to your MacBook Air, it presumably is new enough, also. As for your iMac:

"You can AirPlay to Mac at lower video resolutions from older iPhone, iPad, and Mac models and operating systems if the AirPlay Receiver setting is turned on and the adjacent "Allow AirPlay for" setting is configured to allow AirPlay either for everyone or for anyone on the same network."

bjoernar wrote:

It DOES work with some combination, like using the iMac as an extended display for a MacBook Air. This shows that the iMac 2017-version is capable of working in the same was as a dedicated monitor would. The problem in my case is the fact that I cannot use extended display from my Mac Studio to the iMac. I still cannot see why.

None of that is AirPlay. AirPlay into an iMac 2017 is not supported.

Continuity features and requirements for Apple devices - Apple Support

I could be mistaken, but that doesn't look like the Screen Sharing user interface. That looks as though you're still trying to use AirPlay to Mac with the iMac as an AirPlay Receiver. Which we know won't work and isn't supported.

----------

A quick run-through of Screen Sharing controls:

Screen Sharing has

• A dedicated Screen Sharing application to control screen sharing sessions
• A set of System Settings > General > Sharing > Screen Sharing settings to control whether a Mac is willing to let other Macs remotely control it. (Many "hidden" behind the "lower-case i in a circle" next to the main switch.)

To use Screen Sharing, I believe that you would need to enable Screen Sharing on the Mac Studio. Then go onto the iMac and run the Screen Sharing application and connect to the Mac Studio.

If successful, this would result in you getting a window on the iMac that displayed a view onto the Mac Studio. It might be a mirrored view of whatever was on your real hardware monitor.

Please read the link I provided regarding TDM. We are asked your question almost daily and the answer is always the same. Target Display Mode died in 2014 so it has been over a decade Apple killed it off. To be honest Target Display Mode was never supposed to work, however when it was discovered it did (sort of, it was never very reliable) Apple eventually killed it when Retina display Macs appeared (2014) as it requires a lot of power.

If you want to continue trying you can of course, however I can say with some certainty that it simply won't work.

There'is a workaround product called Luna Display, by Astropad, that might let you reuse that 27" 5K Retina iMac as a second display.

However, even if you plug the Luna Display device into your Mac Studio, and run a separate cable between the Mac Studio and iMac to help carry video traffic, performance will not be the same as with a connection to something like an Apple 27" 5K Studio Display.

An article on the Astropad site acknowledges that the product has limitations – saying that it is not recommended for high-motion applications (like video editing or playing games).

consider just enabling MacOS screen sharing (NOT airplay)

it uses your existing Wi-Fi or Ethernet connections.

Turn Mac screen sharing on or off - Apple Support

Share the screen of another Mac - Apple Support

must first be enabled here:

settings > general > sharing > screen sharing

Note that while the 27" 5K Apple Studio Display goes for $1600+, there are now several other 27" 5120x2880 (5K) displays on the market, some with street prices as low as $800 – $950 USD.

Offhand, I believe these include monitors from LG, Samsung, BenQ, ViewSonic, and Asus.