PettingZoo和ParallelEnv的稳定-Baselines3 3的问题
PettingZoo和ParallelEnv的稳定-Baselines3 3的问题
提问于 2022-07-25 15:30:17
我在使用我用ParallelEnv编写的自定义PettingZoo时遇到了困难。我正在使用SuperSuit的ss.pettingzoo_env_to_vec_env_v1(env)作为包装器，对环境进行矢量化，并使其与稳定-底座3和文档化的这里一起工作。
您可以找到所附的代码最相关部分的摘要：
from typing import Optional
from gym import spaces
import random
import numpy as np
from pettingzoo import ParallelEnv
from pettingzoo.utils.conversions import parallel_wrapper_fn
import supersuit as ss
from gym.utils import EzPickle, seeding
def env(**kwargs):
env_ = parallel_env(**kwargs)
env_ = ss.pettingzoo_env_to_vec_env_v1(env_)
#env_ = ss.concat_vec_envs_v1(env_, 1)
return env_
petting_zoo = env
class parallel_env(ParallelEnv, EzPickle):
metadata = {'render_modes': ['ansi'], "name": "PlayerEnv-Multi-v0"}
def __init__(self, n_agents: int = 20, new_step_api: bool = True) -> None:
EzPickle.__init__(
self,
n_agents,
new_step_api
)
self._episode_ended = False
self.n_agents = n_agents
self.possible_agents = [
f"player_{idx}" for idx in range(n_agents)]
self.agents = self.possible_agents[:]
self.agent_name_mapping = dict(
zip(self.possible_agents, list(range(len(self.possible_agents))))
)
self.observation_spaces = spaces.Dict(
{agent: spaces.Box(shape=(len(self.agents),),
dtype=np.float64, low=0.0, high=1.0) for agent in self.possible_agents}
)
self.action_spaces = spaces.Dict(
{agent: spaces.Discrete(4) for agent in self.possible_agents}
)
self.current_step = 0
def seed(self, seed=None):
self.np_random, seed = seeding.np_random(seed)
def observation_space(self, agent):
return self.observation_spaces[agent]
def action_space(self, agent):
return self.action_spaces[agent]
def __calculate_observation(self, agent_id: int) -> np.ndarray:
return self.observation_space(agent_id).sample()
def __calculate_observations(self) -> np.ndarray:
observations = {
agent: self.__calculate_observation(
agent_id=agent)
for agent in self.agents
}
return observations
def observe(self, agent):
return self.__calculate_observation(agent_id=agent)
def step(self, actions):
if self._episode_ended:
return self.reset()
observations = self.__calculate_observations()
rewards = random.sample(range(100), self.n_agents)
self.current_step += 1
self._episode_ended = self.current_step >= 100
infos = {agent: {} for agent in self.agents}
dones = {agent: self._episode_ended for agent in self.agents}
rewards = {
self.agents[i]: rewards[i]
for i in range(len(self.agents))
}
if self._episode_ended:
self.agents = {} # To satisfy `set(par_env.agents) == live_agents`
return observations, rewards, dones, infos
def reset(self,
seed: Optional[int] = None,
return_info: bool = False,
options: Optional[dict] = None,):
self.agents = self.possible_agents[:]
self._episode_ended = False
self.current_step = 0
observations = self.__calculate_observations()
return observations
def render(self, mode="human"):
# TODO: IMPLEMENT
print("TO BE IMPLEMENTED")
def close(self):
pass不幸的是，当我尝试使用以下主要过程进行测试时：
from stable_baselines3 import DQN, PPO
from stable_baselines3.common.env_checker import check_env
from dummy_env import dummy
from pettingzoo.test import parallel_api_test
if __name__ == '__main__':
# Testing the parallel algorithm alone
env_parallel = dummy.parallel_env()
parallel_api_test(env_parallel) # This works!
# Testing the environment with the wrapper
env = dummy.petting_zoo()
# ERROR: AssertionError: The observation returned by the `reset()` method does not match the given observation space
check_env(env)
# Model initialization
model = PPO("MlpPolicy", env, verbose=1)
# ERROR: ValueError: could not broadcast input array from shape (20,20) into shape (20,)
model.learn(total_timesteps=10_000)我得到以下错误：
AssertionError: The observation returned by the `reset()` method does not match the given observation space如果跳过check_env()，就会得到以下内容：
ValueError: could not broadcast input array from shape (20,20) into shape (20,)ss.pettingzoo_env_to_vec_env_v1(env)似乎能够将并行环境拆分成多个矢量化的环境，但对于reset()函数则不然。
有人知道如何解决这个问题吗？
请找到Github储存库来重现问题。
回答 2
Stack Overflow用户
回答已采纳
发布于 2022-08-01 10:17:51
由于我在SuperSuit存储库的问题部分中的讨论，我能够发布问题的解决方案。多亏了Jj笋！
首先，有必要拥有最新的SuperSuit版本。为了达到这个目的，我需要使用Stable-Baseline3的指令这里来安装gym 0.24+，使之与gym 0.24+一起工作。
之后，以问题中的代码为例，有必要替换
def env(**kwargs):
env_ = parallel_env(**kwargs)
env_ = ss.pettingzoo_env_to_vec_env_v1(env_)
#env_ = ss.concat_vec_envs_v1(env_, 1)
return env_使用
def env(**kwargs):
env_ = parallel_env(**kwargs)
env_ = ss.pettingzoo_env_to_vec_env_v1(env_)
env_ = ss.concat_vec_envs_v1(env_, 1, base_class="stable_baselines3")
return env_其结果是：
- 结果1：将行保留为
check_env(env)，我得到了一个错误AssertionError: Your environment must inherit from the gym.Env class cf https://github.com/openai/gym/blob/master/gym/core.py - 结果2：用
check_env(env)删除行，代理就会成功地开始训练！
最后，我认为base_class="stable_baselines3"的论点产生了不同的效果。只有check_env上的小问题还有待报告，但我认为如果培训奏效，它可以被认为是微不足道的。
Stack Overflow用户
发布于 2022-07-25 19:14:32
您应该在PettingZoo中反复检查reset()函数。它将什么也不回，而不是像健身房这样的观察。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/73111772
复制相关文章
相似问题
腾讯云开发者
