Compressor 4.6.6 "cannot encode" error

First of all, just like with FCP, try deleting Compressor preferences. To delete preferences, hold down command and option while starting Compressor and click Delete Preferences.

If the problem persists, we will need to know more.

Please run Etrecheck and post its full report here. Use the "additional text" button and paste the report into the text box.

Here's the standard Compressor repair process, I have found it works most of the time.

First, go to Compressor's Settings > Advanced tab.

Click Reset Queue.

Cancel Jobs.

Do this again but this time select Repair Compressor.

Quit, reboot your Mac, launch Compressor, test.

If you have Droplets you've created in previous versions of Compressor, in that same Advanced pane, select Update Droplets. This should be done every time you update Compressor.

You can also launch Compressor while holding the Opt+Cmd keys together to delete the preference files, that often helps, also. Be aware that it will warn you that your custom settings will be deleted, but this does NOT include the custom presets you've created. If you wanted to be safe, user folder>Library>Application Support > Compressor, right-click that folder, from the pop-up menu, compress it. There, you have a zip backup copy in case something goes wrong, but I've never see custom presets disappear when deleting preferences files. But, Apple's stuff is all just weird and you never know.