Preventing installation of apps from outside the Store
In Windows 10 Creators Update, if you navigate to Settings > Apps > Apps & features, you can see app installing options. Here, you can choose from the following three options:

• Allow apps from anywhere
• Warn me before installing apps from outside the Store
• Allow apps from the Store only

This is the message the users are shown when using : “Allow apps from the Store only”

The Configure App Install Control can be configured with a Group Policy

Computer Configuration -> Administrative Templates -> Windows Components -> Windows Defender SmartScreen -> Explorer -> Configure App install Control

App Install Control is a feature of Windows Defender SmartScreen that helps protect PCs by allowing users to install apps only from the Store.  SmartScreen must be enabled for this feature to work properly.

If you enable this setting, users will only be able to install apps from the Store.

If you disable this setting, users will be able to install apps from anywhere, including files downloaded from the Internet.

If you do not configure this setting, users will be able to choose their own setting for App Install Control.

Then it is not possible for the end user to change the settings or install Application other places then Windows Store.

After the policy is applied I have tested if it possible to install a Appx and a MSI file.

When installing the Appx files from a elevated permissions powershell Promt it is possible to install a modern Application that is not from the Windows Store.

When installing the MSI files from a elevated permissions powershell Promt it is not possible to install a MSI file.