Linux
First change the authentication plugin in /etc/my.cnf file for Linux
[mysqld]
default_authentication_plugin=mysql_native_password
Then restart mysql service: systemctl restart mysqld
Windows
Docs:
DOCUMENTATION: Caching SHA-2 Pluggable Authentication
stackoverflow: plugin 'caching_sha2_password' cannot be loaded