Enhance SAC with Mixture-of-Expert and BEE Operator for Improved Stability and Performance

[XuGW-Kevin]

This PR introduces two plug-and-play enhancements to the Soft Actor-Critic (SAC) algorithm, inspired by recent advancements in reinforcement learning research.

The integration is based on insights from several papers:
https://arxiv.org/abs/2306.02865
https://arxiv.org/abs/2402.08609
https://arxiv.org/abs/2410.14972

To maintain robustness and practicality, I excluded experimental techniques (e.g., dormant ratio, perturbing network weights, and dynamic hyperparameter tuning) that have not yet stood the test of time. The retained methods are relatively well-validated and provide substantial improvements in both stability and performance.

Key Features:
(1) Mixture-of-Expert Network
(2) Blended Exploration and Exploitation (BEE) Operator

This enhancement aims to provide a more performant SAC-based RL baseline algorithm.
Feedback and suggestions are welcome!

[StoneT2000]

nice work! I'll review this next week probably (busy with icml/rss)

[XuGW-Kevin]

Thank you so much! Good luck with ICML/RSS!

[StoneT2000]

Hi @XuGW-Kevin I am beginning to review your PR. I have one recommendation first which is to make this a separate folder to the SAC baseline. I would like to keep the SAC baseline essentially the old and not so powerful anymore SAC (but you may modify the SAC README to point to better alternatives like your work). In this separate folder you can follow the structure of the PPO/SAC baselines and make sure to add bibtex citations to the relevant work that should be cited. You may also include the more experimental features in the separate folder.

The next thing is baseline testing. Can you write a script like the one in PPO: examples/baselines/ppo/baselines.sh

And split between state and RGB-based baseline scripts? Essentially we want something where I can just setup the environment, and copy paste a script and get it training successfully.

After that I can help use my compute to run this baseline and upload all results to wandb so they can be compared with other algorithms neatly.