Introduction
In this tutorial, we will show how to use PyTupli to set up an efficient pipeline for offline reinforcement learning (RL) for a custom environment. This includes
creating a benchmark and uploading it to an instance of a TupliStorage,
re-loading this benchmark from the storage,
recording RL tuples of (state, action, reward, done) for this benchmark and uploading them to the storage,
creating a dataset from the stored episodes, and
training an offline RL agent using d3rlpy.
You can skip the last part, but if you want to try that, you have to install the d3rlpy library using pip install d3rlpy.
[1]:
import io
import os
import tempfile
import math
from typing import Optional
import numpy as np
import pandas as pd
from gymnasium import spaces
from gymnasium.envs.classic_control import utils, MountainCarEnv
from gymnasium.wrappers import TimeLimit
import d3rlpy
from d3rlpy.algos import DiscreteCQLConfig
from d3rlpy.dataset import MDPDataset
from pytupli.benchmark import TupliEnvWrapper
from pytupli.storage import TupliAPIClient, TupliStorage, FileStorage
from gymnasium import Env
from pytupli.schema import ArtifactMetadata, FilterEQ, EpisodeMetadataCallback
from pytupli.dataset import TupliDataset, NumpyTupleParser
/home/hannah/anaconda3/envs/pytupli_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
2025-07-28 15:45.23 [info ] Register Shimmy environments.
PyTupli has two storage options: A local FileStorage and using MongoDB as a backend in the TupliAPIClient. You can run this notebook with both storage types by adjusting the flag below. If you want to use the TupliAPIClient, follow the instructions in the Readme to start the application.
[2]:
STORAGE_FLAG = 'api' # "api"
Creating a Custom Environment
We will use the MountainCar example from gymnasium with a small modification: The cart is slowed down by wind in the horizontal direction. We load the wind data from a csv file.
[3]:
class CustomMountainCarEnv(MountainCarEnv):
"""
## Description
The Mountain Car MDP is a deterministic MDP that consists of a car placed stochastically
at the bottom of a sinusoidal valley, with the only possible actions being the accelerations
that can be applied to the car in either direction. The goal of the MDP is to strategically
accelerate the car to reach the goal state on top of the right hill. There are two versions
of the mountain car domain in gymnasium: one with discrete actions and one with continuous.
This version is the one with discrete actions.
This MDP first appeared in [Andrew Moore's PhD Thesis (1990)](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-209.pdf)
```
@TECHREPORT{Moore90efficientmemory-based,
author = {Andrew William Moore},
title = {Efficient Memory-based Learning for Robot Control},
institution = {University of Cambridge},
year = {1990}
}
```
## Observation Space
The observation is a `ndarray` with shape `(2,)` where the elements correspond to the following:
| Num | Observation | Min | Max | Unit |
|-----|--------------------------------------|-------|------|--------------|
| 0 | position of the car along the x-axis | -1.2 | 0.6 | position (m) |
| 1 | velocity of the car | -0.07 | 0.07 | velocity (v) |
## Action Space
There are 3 discrete deterministic actions:
- 0: Accelerate to the left
- 1: Don't accelerate
- 2: Accelerate to the right
## Transition Dynamics:
Given an action, the mountain car follows the following transition dynamics:
*velocity<sub>t+1</sub> = velocity<sub>t</sub> + (action - 1) * force - cos(3 * position<sub>t</sub>) * gravity*
*position<sub>t+1</sub> = position<sub>t</sub> + velocity<sub>t+1</sub>*
where force = 0.001 and gravity = 0.0025. The collisions at either end are inelastic with the velocity set to 0
upon collision with the wall. The position is clipped to the range `[-1.2, 0.6]` and
velocity is clipped to the range `[-0.07, 0.07]`.
## Reward:
The goal is to reach the flag placed on top of the right hill as quickly as possible, as such the agent is
penalised with a reward of -1 for each timestep.
## Starting State
The position of the car is assigned a uniform random value in *[-0.6 , -0.4]*.
The starting velocity of the car is always assigned to 0.
## Episode End
The episode ends if either of the following happens:
1. Termination: The position of the car is greater than or equal to 0.5 (the goal position on top of the right hill)
2. Truncation: The length of the episode is 200.
## Arguments
Mountain Car has two parameters for `gymnasium.make` with `render_mode` and `goal_velocity`.
On reset, the `options` parameter allows the user to change the bounds used to determine the new random state.
```python
>>> import gymnasium as gym
>>> env = gym.make("MountainCar-v0", render_mode="rgb_array", goal_velocity=0.1) # default goal_velocity=0
>>> env
<TimeLimit<OrderEnforcing<PassiveEnvChecker<MountainCarEnv<MountainCar-v0>>>>>
>>> env.reset(seed=123, options={"x_init": np.pi/2, "y_init": 0.5}) # default x_init=np.pi, y_init=1.0
(array([-0.46352962, 0. ], dtype=float32), {})
```
## Version History
* v0: Initial versions release
"""
metadata = {
'render_modes': ['human', 'rgb_array'],
'render_fps': 30,
}
def __init__(self, data_path: str, render_mode: Optional[str] = None, goal_velocity=0):
self.min_position = -1.2
self.max_position = 0.6
self.max_speed = 0.07
self.goal_position = 0.5
self.goal_velocity = goal_velocity
self.current_step = 0
self.data = pd.read_csv(data_path, index_col=0, header=None) * 0.01
self.force = 0.001
self.gravity = 0.0025
self.low = np.array([self.min_position, -self.max_speed], dtype=np.float32)
self.high = np.array([self.max_position, self.max_speed], dtype=np.float32)
self.render_mode = render_mode
self.screen_width = 600
self.screen_height = 400
self.screen = None
self.clock = None
self.isopen = True
self.action_space = spaces.Discrete(3)
self.observation_space = spaces.Box(self.low, self.high, dtype=np.float32)
def step(self, action: int):
assert self.action_space.contains(action), f'{action!r} ({type(action)}) invalid'
position, velocity = self.state
velocity += (
(action - 1) * self.force
+ math.cos(3 * position) * (-self.gravity)
- self.data.loc[self.current_step].to_numpy().flatten()[0] * math.cos(position)
)
velocity = np.clip(velocity, -self.max_speed, self.max_speed)
position += velocity
position = np.clip(position, self.min_position, self.max_position)
if position == self.min_position and velocity < 0:
velocity = 0
terminated = bool(position >= self.goal_position and velocity >= self.goal_velocity)
reward = -1.0
self.current_step += 1
self.state = (position, velocity)
if self.render_mode == 'human':
self.render()
# truncation=False as the time limit is handled by the `TimeLimit` wrapper added during `make`
return np.array(self.state, dtype=np.float32), reward, terminated, False, {}
def reset(
self,
*,
seed: Optional[int] = None,
options: Optional[dict] = None,
):
super().reset(seed=seed)
# Note that if you use custom reset bounds, it may lead to out-of-bound
# state/observations.
self.current_step = 0
low, high = utils.maybe_parse_reset_bounds(options, -0.6, -0.4)
self.state = np.array([self.np_random.uniform(low=low, high=high), 0])
if self.render_mode == 'human':
self.render()
return np.array(self.state, dtype=np.float32), {}
Serialize Environment for Upload
As a next step, we want to upload our environment to our storage using PyTupli. For this, we will detach the csv file from the environment, upload it seperately, and replace the data attribute in the environment with the id of the stored object. This allows us to re-use artifacts such as csv files in multiple benchmarks. For example, consider a case where you only want to change one parameter within the environment, e.g., the maximum speed. You would have to create a new benchmark, but could re-use the csv file! PyTupli automatically recognizes such duplicates.
To separate the csv file, we have to subclass the TupliEnvWrapper class and overwrite the _serialize() and _deserialize() members. The TupliEnvWrapper is essentially a gymnasium wrapper that records RL tuples in the step() function, but it has a lot of extra functionalities for interacting with the storage.
[4]:
class MyTupliEnvWrapper(TupliEnvWrapper):
def _serialize(self, env) -> Env:
related_data_sources = []
ds = env.unwrapped.data
metadata = ArtifactMetadata(name='test')
data_kwargs = {'header': None}
try:
content = ds.to_csv(encoding='utf-8', **data_kwargs)
content = content.encode(encoding='utf-8')
except Exception as e:
raise ValueError(f'Failed to serialize data source: {e}')
ds_storage_metadata = self.storage.store_artifact(artifact=content, metadata=metadata)
related_data_sources.append(ds_storage_metadata.id)
setattr(env.unwrapped, 'data', ds_storage_metadata.id)
return env, related_data_sources
@classmethod
def _deserialize(cls, env: Env, storage: TupliStorage) -> Env:
data_kwargs = {'header': None, 'index_col': 0}
ds = storage.load_artifact(env.unwrapped.data)
ds = ds.decode('utf-8')
d = io.StringIO(ds)
df = pd.read_csv(d, **data_kwargs)
env.unwrapped.data = df
return env
[5]:
# which storag to use
if STORAGE_FLAG == 'api':
storage = TupliAPIClient()
elif STORAGE_FLAG == 'file':
storage = FileStorage()
else:
raise ValueError(f"Unknown storage flag: {STORAGE_FLAG}. Has to be 'api' or 'file'.")
[6]:
# instantiate the environment
max_eps_length = 999
data_path = '/home/hannah/Documents/Code/pytupli/docs/source/tutorials/data/wind_data.csv'
env = TimeLimit(
CustomMountainCarEnv(render_mode=None, data_path=data_path), max_episode_steps=max_eps_length
)
# Now we can create the benchmark
tupli_env = MyTupliEnvWrapper(env, storage=storage)
Uploading and Downloading Benchmarks
We will now upload the benchmark and download it again.
[7]:
tupli_env.store(name='mountain-car-v0', description='Mountain Car v0 benchmark')
Let us list the uploaded benchmarks:
[8]:
%system
!pytupli list_benchmarks
id created_by is_public created_at hash metadata
-------------------------------- ------------ ----------- -------------------------- ---------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------
9ab24bb4e96c4b38bf0c3577bdb4e4bb hannah ❌ 2025-07-10T13:37:28.852619 a23a5fdeafb580c2d170f19da7ce4969164d8e458c6eeef2494ad71ff5c340c1 {'name': 'test_env', 'description': '', 'difficulty': None, 'version': None, 'extra': {}}
411704fb2dd34f50982817141264840b hannah ❌ 2025-07-10T13:48:14.966254 6837628829fbb513e75eae4e0f33ad322f310780731c2308ab90fc4218fb8d47 {'name': 'second_test_env', 'description': '', 'difficulty': None, 'version': None, 'extra': {}}
989f723675fc4b85ab19c1621a96f9dc hannah ❌ 2025-07-26T13:27:28.985898 c8b39b0780165ed81f74676da754cdd8f0f3f75c2e24350b0df5c519f2ebc3c1 {'name': 'mountain-car-v0', 'description': 'Mountain Car v0 benchmark', 'difficulty': None, 'version': None, 'extra': {}}
c8d477bb5f3746669773114966112096 hannah ❌ 2025-07-28T15:18:23.126452 dc166e979ece91004094fc5e8e7ee44034d550a94681538c5ed541ff6e29326f {'name': 'mountain-car-v0', 'description': 'Mountain Car v0 benchmark', 'difficulty': None, 'version': None, 'extra': {}}
be5ddcefe39c49058e1dc88655c8a035 hannah ❌ 2025-07-28T15:23:15.751898 5cea9558ed1f80df903858d590e66349754a0d5ab2be718aa9cc549dd6be28d1 {'name': 'mountain-car-v0', 'description': 'Mountain Car v0 benchmark', 'difficulty': None, 'version': None, 'extra': {}}
7b334ed6b7724bec8872418ad32667a4 hannah ❌ 2025-07-28T15:36:14.354599 fe0113633bc4e52225040499e70ed3e7988698eb591620e6a15e9e6157922a59 {'name': 'mountain-car-v0', 'description': 'Mountain Car v0 benchmark', 'difficulty': None, 'version': None, 'extra': {}}
4b076cdedc4d433ea4da6d2801c05ca0 hannah ❌ 2025-07-28T15:45:23.936409 e1eaffecdc9a04ecb72ecb29e9ef329c5befa15ec7dc62cf1ba1c13d920022d4 {'name': 'mountain-car-v0', 'description': 'Mountain Car v0 benchmark', 'difficulty': None, 'version': None, 'extra': {}}
As a next step, we show how to download the benchmark. Note that this is only for demonstration purposes! When loading the benchmark, we can pass a callback that will later be used to add metadate to recorded episodes. We provide a simple example of such a function.
[9]:
class MyCallback(EpisodeMetadataCallback):
def __init__(self):
super().__init__()
# we will compute the cumulative reward for an episode
self.cum_reward = 0
# Furthermore, we want to store the fact that the episode was not an expert episode
self.is_expert = False
def reset(self):
# we will compute the cumulative reward for an episode
self.cum_reward = 0
def __call__(self, tuple):
self.cum_reward += tuple.reward
return {"cum_eps_reward": [self.cum_reward], "is_expert": self.is_expert}
[10]:
loaded_tupli_env = MyTupliEnvWrapper.load(storage=storage, benchmark_id=tupli_env.id, metadata_callback=MyCallback())
Recording Episodes for Offline RL Training
The TupliEnvWrapper wrapper allows us to record all interactions with the custom environment to the storage in form of tuples (state, action, reward, terminal, timeout). This can then be used for training an offline RL agent for this environment using any offline RL library. For simplicity, we will use a random policy to generate the data.
[11]:
# For reproducibility when generating episodes
np.random.seed(42)
obs, info = loaded_tupli_env.reset(seed=42)
for step in range(2000):
action = np.int64(np.random.randint(low=0, high=3))
obs, reward, done, truncated, info = loaded_tupli_env.step(action)
if done or truncated:
print(f'Episode finished after {step + 1} timesteps')
obs, info = loaded_tupli_env.reset()
Episode finished after 999 timesteps
Episode finished after 1998 timesteps
Downloading Episodes for a Benchmark
Next, let us download all episodes that have been recorded for our benchmark. For this, we create a TupliDataset using a filter with the id of the benchmark.
[12]:
# Create dataset
mdp_dataset = TupliDataset(storage=storage).with_benchmark_filter(
FilterEQ(key='id', value=loaded_tupli_env.id)
)
mdp_dataset.load()
We can show the contents of the dataset using the preview() method:
[13]:
mdp_dataset.preview()
[13]:
[EpisodeItem(id='8236cd52226b44cdb9ed453d180ce4fa', created_by='hannah', is_public=False, created_at='2025-07-28T15:45:24.757499', benchmark_id='4b076cdedc4d433ea4da6d2801c05ca0', metadata={'cum_eps_reward': [-999.0], 'is_expert': False}, n_tuples=999, terminated=False, timeout=True, tuples=[RLTuple(state=[-0.5113905072212219, 0.0008338160114362836], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5116723775863647, -0.0002818846551235765], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5110931396484375, 0.0005792493466287851], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5096522569656372, 0.0014408753486350179], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5093242526054382, 0.000327992340316996], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5101639032363892, -0.0008396401535719633], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5101851224899292, -2.124540878867265e-05], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5103291869163513, -0.0001440104970242828], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.509620726108551, 0.0007084066164679825], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5080977082252502, 0.0015230465214699507], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5057256817817688, 0.002372018527239561], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5025493502616882, 0.0031763564329594374], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.500569760799408, 0.0019795973785221577], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4977911710739136, 0.0027785508427768946], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4952910244464874, 0.002500171773135662], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4940441846847534, 0.001246814732439816], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4930240511894226, 0.0010201389668509364], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4923064112663269, 0.0007176604704000056], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49190738797187805, 0.00039900271804071963], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49182963371276855, 7.777586142765358e-05], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4930543899536133, -0.0012247838312759995], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4955599009990692, -0.002505498006939888], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49830546975135803, -0.0027455675881356], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5013017058372498, -0.002996222348883748], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5055070519447327, -0.004205367062240839], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.510878324508667, -0.005371248349547386], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5173525810241699, -0.006474245805293322], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5229570269584656, -0.005604492034763098], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5276248455047607, -0.004667783621698618], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5312625169754028, -0.003637672169134021], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5348554253578186, -0.0035929519217461348], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5374233722686768, -0.0025679245591163635], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5399391651153564, -0.002515781205147505], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5423727631568909, -0.0024336320348083973], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5437002182006836, -0.0013273990480229259], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5449283719062805, -0.0012281722156330943], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5450786352157593, -0.00015025046013761312], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.544078528881073, 0.0010000934125855565], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5440093278884888, 6.916956044733524e-05], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5428723692893982, 0.0011369569692760706], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5426400899887085, 0.00023230024089571089], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5412861108779907, 0.0013539785286411643], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5388557314872742, 0.002430384047329426], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5373508334159851, 0.0015049121575430036], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5367832183837891, 0.0005675791180692613], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5351337194442749, 0.0016495289746671915], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5334744453430176, 0.001659264788031578], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5327448844909668, 0.0007295446703210473], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5320248603820801, 0.0007200646796263754], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5313005447387695, 0.0007242888677865267], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5305811762809753, 0.0007193594938144088], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5308516025543213, -0.0002704005455598235], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5310946106910706, -0.00024304342514369637], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5323395729064941, -0.0012449665227904916], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5335495471954346, -0.0012099361047148705], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5337236523628235, -0.00017413827299606055], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5328255295753479, 0.0008981470600701869], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5328859686851501, -6.0470742027973756e-05], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5318993926048279, 0.0009865766623988748], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5298661589622498, 0.0020332620479166508], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5277872681617737, 0.0020788954570889473], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5267308354377747, 0.0010564220137894154], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5256749391555786, 0.00105590361636132], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5246210098266602, 0.001053902436979115], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5236331224441528, 0.0009878887794911861], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5226479768753052, 0.0009851761860772967], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5217301249504089, 0.0009178569889627397], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5208677649497986, 0.0008623460889793932], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5200468301773071, 0.0008209246443584561], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5202727913856506, -0.00022597238421440125], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5195737481117249, 0.0006990367546677589], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5189687013626099, 0.000605061708483845], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5184198617935181, 0.0005488255410455167], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5179292559623718, 0.0004906049580313265], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5175549387931824, 0.00037433297256939113], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.517245352268219, 0.000309614697471261], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5170034170150757, 0.0002419227676000446], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.515887975692749, 0.0011154473759233952], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5138584971427917, 0.0020294473506510258], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5119637846946716, 0.0018947572680190206], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5091778039932251, 0.0027859758120030165], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5075864791870117, 0.0015912812668830156], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5061923265457153, 0.0013941703364253044], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5060040950775146, 0.00018822852871380746], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5070022344589233, -0.0009981263428926468], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5081721544265747, -0.001169910072349012], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5085447430610657, -0.0003726317372638732], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5100646615028381, -0.0015199058689177036], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5117618441581726, -0.0016971792792901397], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5146201848983765, -0.002858339808881283], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5185853838920593, -0.003965205978602171], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.523647129535675, -0.005061722360551357], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5297783017158508, -0.0061311605386435986], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5348930358886719, -0.005114757921546698], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5409407019615173, -0.006047682370990515], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5479172468185425, -0.006976546719670296], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5557628870010376, -0.00784562062472105], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5624416470527649, -0.006678763311356306], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5699036121368408, -0.0074619329534471035], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5780211091041565, -0.008117523044347763], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5847542881965637, -0.006733175832778215], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5900659561157227, -0.005311679095029831], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5938848853111267, -0.003818926867097616], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5982547402381897, -0.00436982698738575], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6011183857917786, -0.0028636441566050053], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6024767756462097, -0.0013584232656285167], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.604319155216217, -0.0018423564033582807], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6045958995819092, -0.00027673502336256206], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6053089499473572, -0.0007130824378691614], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6054692268371582, -0.0001602756092324853], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6040581464767456, 0.001411092234775424], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6020604968070984, 0.0019976769108325243], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6005573868751526, 0.0015030945651233196], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5975644588470459, 0.0029929226730018854], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5950524210929871, 0.0025120568461716175], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5920725464820862, 0.002979846205562353], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5896499156951904, 0.0024226270616054535], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5858072638511658, 0.0038426443934440613], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5805745124816895, 0.005232765804976225], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5749903917312622, 0.0055841002613306046], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5700504183769226, 0.00493996636942029], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5637997388839722, 0.006250693462789059], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5573214292526245, 0.0064783054403960705], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5496217608451843, 0.007699707057327032], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5407589673995972, 0.008862749673426151], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5328400135040283, 0.007918983697891235], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5239173173904419, 0.008922661654651165], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5160330533981323, 0.007884295657277107], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5072849988937378, 0.008748052641749382], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49874258041381836, 0.00854241568595171], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48947420716285706, 0.009268364869058132], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4815200865268707, 0.007954118773341179], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47492921352386475, 0.006590870674699545], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4687783718109131, 0.00615086080506444], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46207600831985474, 0.006702371872961521], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45485618710517883, 0.007219808176159859], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44819074869155884, 0.006665448192507029], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44116243720054626, 0.007028302177786827], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4338301122188568, 0.007332322653383017], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4281710684299469, 0.005659053102135658], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4222997725009918, 0.005871289409697056], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4161776304244995, 0.006122150458395481], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4109116792678833, 0.005265939515084028], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4064755439758301, 0.004436126444488764], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4039517939090729, 0.002523773582652211], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40135127305984497, 0.0026004991959780455], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3986596167087555, 0.0026916645001620054], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39588677883148193, 0.002772834151983261], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3951275944709778, 0.0007591942558065057], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3963949978351593, -0.0012674005702137947], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39862439036369324, -0.0022293967194855213], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4028591811656952, -0.004234800580888987], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.407021164894104, -0.004161973483860493], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41105911135673523, -0.004037961363792419], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.416995644569397, -0.00593652855604887], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42280468344688416, -0.0058090281672775745], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.428392618894577, -0.005587928928434849], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43571794033050537, -0.007325336802750826], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4447041451931, -0.008986195549368858], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4533352553844452, -0.008631102740764618], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4615289270877838, -0.008193697780370712], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4691871106624603, -0.007658177521079779], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47730588912963867, -0.008118769153952599], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.485789954662323, -0.00848408229649067], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.494637131690979, -0.008847154676914215], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5047628283500671, -0.01012569759041071], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5150337219238281, -0.010270931757986546], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5263786911964417, -0.011344928294420242], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5387546420097351, -0.012375985272228718], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.551028311252594, -0.012273619882762432], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5631109476089478, -0.01208264660090208], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5749309062957764, -0.011819968931376934], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5854355096817017, -0.010504623875021935], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5955489873886108, -0.010113446041941643], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6041737794876099, -0.008624814450740814], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6132490038871765, -0.009075222536921501], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6207288503646851, -0.007479832042008638], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6275708675384521, -0.006842003203928471], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6347057819366455, -0.007134954910725355], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6420928239822388, -0.007387015037238598], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6496621370315552, -0.007569336332380772], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6553828716278076, -0.0057207439094781876], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6601834893226624, -0.004800579976290464], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6649965643882751, -0.004813105333596468], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6698190569877625, -0.004822499584406614], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6746217608451843, -0.004802709445357323], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6773593425750732, -0.0027375577483326197], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6780165433883667, -0.0006571953999809921], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6775906682014465, 0.0004258534754626453], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6750766634941101, 0.00251401262357831], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6725336313247681, 0.0025430740788578987], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6689592003822327, 0.003574422327801585], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6653852462768555, 0.003573961555957794], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6618319153785706, 0.0035532775800675154], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6562931537628174, 0.00553875882178545], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6498505473136902, 0.00644261809065938], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6415277719497681, 0.008322770707309246], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6313640475273132, 0.010163730010390282], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6204244494438171, 0.010939628817141056], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6098325848579407, 0.010591857135295868], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5996437072753906, 0.010188865475356579], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5888920426368713, 0.010751644149422646], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5786861181259155, 0.010205965489149094], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5681355595588684, 0.010550525970757008], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5573152899742126, 0.010820302180945873], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5452965497970581, 0.012018737383186817], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5331947803497314, 0.012101748958230019], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5200710296630859, 0.013123788870871067], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5080083012580872, 0.012062697671353817], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4970666170120239, 0.010941681452095509], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4873277544975281, 0.009738857857882977], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4788789451122284, 0.008448814041912556], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4698355495929718, 0.009043391793966293], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46225249767303467, 0.007583058904856443], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.455206960439682, 0.007045549340546131], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44868358969688416, 0.006523370277136564], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4427776634693146, 0.005905919708311558], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4365246295928955, 0.0062530189752578735], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43198490142822266, 0.004539727699011564], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4291447699069977, 0.002840145491063595], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4280281662940979, 0.0011166040785610676], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42665162682533264, 0.0013765476178377867], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4260137677192688, 0.0006378371617756784], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42610982060432434, -9.603546641301364e-05], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4279322028160095, -0.0018223950173705816], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43051135540008545, -0.002579153049737215], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43381431698799133, -0.0033029557671397924], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43686750531196594, -0.0030531769152730703], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43958407640457153, -0.002716567600145936], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44195911288261414, -0.0023750511463731527], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44401609897613525, -0.002056971425190568], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4477478265762329, -0.003731728531420231], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4510704576969147, -0.003322632983326912], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45493826270103455, -0.003867819206789136], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46038684248924255, -0.005448565352708101], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.466317743062973, -0.005930902902036905], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.472736120223999, -0.00641838600859046], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47955310344696045, -0.006816993001848459], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4857166111469269, -0.006163501180708408], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49120810627937317, -0.0054915002547204494], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4979897439479828, -0.006781619507819414], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5059962868690491, -0.008006556890904903], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5131844878196716, -0.007188220974057913], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5204573273658752, -0.00727279856801033], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5287824273109436, -0.008325126953423023], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5360691547393799, -0.00728671345859766], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5423032641410828, -0.006234143394976854], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5474693179130554, -0.005166038870811462], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5525184869766235, -0.005049147643148899], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5563540458679199, -0.003835570067167282], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5589885115623474, -0.0026344344951212406], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5603809356689453, -0.0013924705563113093], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5605099201202393, -0.000128936895634979], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5614379048347473, -0.0009280064259655774], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.563127338886261, -0.0016894546570256352], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5635663866996765, -0.0004390247049741447], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5637374520301819, -0.00017105104052461684], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5646784901618958, -0.0009410682250745595], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5643532276153564, 0.0003252687456551939], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5647416114807129, -0.0003884013567585498], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.565868079662323, -0.001126479241065681], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.56675785779953, -0.0008897639345377684], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5663831830024719, 0.0003746908623725176], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5647314190864563, 0.0016517604235559702], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5628189444541931, 0.0019124648533761501], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5606885552406311, 0.0021303952671587467], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5573334097862244, 0.0033551419619470835], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5528077483177185, 0.004525674507021904], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5481144785881042, 0.004693279508501291], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5443128943443298, 0.003801575629040599], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5413762927055359, 0.0029366170056164265], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5383337736129761, 0.0030425270088016987], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5362496972084045, 0.0020840547513216734], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5340766310691833, 0.002173063578084111], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.532871663570404, 0.00120497215539217], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5326224565505981, 0.00024917611153796315], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5313883423805237, 0.0012341293040663004], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.529147744178772, 0.0022405870258808136], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5279299020767212, 0.0012178769102320075], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5276857614517212, 0.00024413976643700153], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5264415144920349, 0.0012442480074241757], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.524176836013794, 0.002264658221974969], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5209717750549316, 0.0032050597947090864], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5178144574165344, 0.0031573267187923193], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5157871246337891, 0.002027310198172927], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.512904167175293, 0.0028829844668507576], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5111815929412842, 0.0017225535120815039], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5095811486244202, 0.0016004572389647365], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5080946683883667, 0.001486480701714754], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5057552456855774, 0.0023394019808620214], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5036264061927795, 0.0021288844291120768], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5026800632476807, 0.0009462923044338822], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5008928179740906, 0.001787240500561893], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5002910494804382, 0.0006017701234668493], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.500945508480072, -0.0006544392090290785], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5018519759178162, -0.0009064803016372025], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5029844641685486, -0.0011324947699904442], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5043456554412842, -0.0013611391186714172], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.505925178527832, -0.001579570583999157], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5076871514320374, -0.001761933439411223], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5085905194282532, -0.0009034107788465917], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5106569528579712, -0.0020664315670728683], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5138859152793884, -0.00322896521538496], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5182275772094727, -0.004341629799455404], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.523683488368988, -0.005455898120999336], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5291428565979004, -0.005459412466734648], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5355896353721619, -0.006446735467761755], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5410110354423523, -0.00542140519246459], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5453211665153503, -0.004310137592256069], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5504854321479797, -0.005164292640984058], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5565241575241089, -0.006038706284016371], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5613296627998352, -0.004805475007742643], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5649281144142151, -0.0035984658170491457], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5682640671730042, -0.003335972549393773], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5703202486038208, -0.0020561886485666037], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5720935463905334, -0.0017732627457007766], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.573533296585083, -0.0014397974591702223], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5746445059776306, -0.0011111772619187832], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5754483938217163, -0.000803919683676213], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5759277939796448, -0.0004793908155988902], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5770617723464966, -0.0011339574120938778], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5768375396728516, 0.00022420514142140746], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5762447118759155, 0.0005928356549702585], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5742772221565247, 0.0019675088115036488], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5709463357925415, 0.0033308714628219604], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5672945380210876, 0.003651793347671628], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5643849968910217, 0.002909551141783595], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5612449645996094, 0.0031400241423398256], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5588324666023254, 0.00241253268904984], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5551992654800415, 0.0036332046147435904], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5523994565010071, 0.0027997633442282677], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5504654049873352, 0.0019340969156473875], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5493890643119812, 0.0010763007448986173], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5471386313438416, 0.002250424586236477], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5437910556793213, 0.003347601043060422], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5413433909416199, 0.0024476435501128435], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5398003458976746, 0.0015430472558364272], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5381681323051453, 0.0016322402516379952], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5374951958656311, 0.0006729100714437664], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5367227792739868, 0.0007724046008661389], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5369045734405518, -0.00018177952733822167], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5370184779167175, -0.00011388959683245048], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5360886454582214, 0.0009298506774939597], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.536106526851654, -1.7911592294694856e-05], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5370672345161438, -0.000960722507443279], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5389707088470459, -0.001903428928926587], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5417616367340088, -0.002790922299027443], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5444688200950623, -0.002707231556996703], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5480523705482483, -0.0035835467278957367], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5505238771438599, -0.0024714788887649775], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5518034100532532, -0.0012795504881069064], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5539410710334778, -0.0021376339718699455], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5549137592315674, -0.0009727295255288482], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5566703677177429, -0.0017566145397722721], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.559230625629425, -0.0025602043606340885], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5605870485305786, -0.0013564479304477572], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5626757740974426, -0.002088740933686495], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.563530683517456, -0.0008548742043785751], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5641184449195862, -0.0005877804360352457], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5654233694076538, -0.0013049028348177671], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5664921402931213, -0.0010688015026971698], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5663163065910339, 0.00017581830616109073], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5658440589904785, 0.00047229931806214154], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5651035308837891, 0.0007405286305584013], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5631201267242432, 0.0019833804108202457], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5608629584312439, 0.0022571678273379803], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5583384037017822, 0.002524539828300476], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5546316504478455, 0.003706755582243204], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5507143139839172, 0.003917335532605648], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5456293821334839, 0.005084930919110775], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5404415130615234, 0.00518787419423461], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5341306328773499, 0.006310886237770319], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5268117189407349, 0.007318909280002117], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5195292234420776, 0.0072825015522539616], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5122973322868347, 0.007231913041323423], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5052002668380737, 0.00709706312045455], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49931231141090393, 0.005887946579605341], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4946846663951874, 0.004627641756087542], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49132585525512695, 0.0033587913494557142], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48726338148117065, 0.0040624914690852165], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.483501136302948, 0.003762244712561369], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4791053831577301, 0.004395757801830769], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47606050968170166, 0.003044854151085019], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4734346866607666, 0.002625837456434965], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47025546431541443, 0.003179208841174841], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4674873650074005, 0.0027681151404976845], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4641849994659424, 0.0033023571595549583], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4613819122314453, 0.00280309421941638], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46011844277381897, 0.0012634715531021357], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46036064624786377, -0.00024222211504820734], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46111780405044556, -0.0007571389432996511], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46140629053115845, -0.00028848336660303175], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4632420837879181, -0.0018358012894168496], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46655017137527466, -0.0033080868888646364], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47131696343421936, -0.004766811151057482], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4765413999557495, -0.005224415101110935], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4821544289588928, -0.005613031331449747], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4871441125869751, -0.004989690147340298], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49142035841941833, -0.0042762355878949165], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49494025111198425, -0.0035199050325900316], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4986891448497772, -0.0037488830275833607], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5026981830596924, -0.004009040538221598], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.505939781665802, -0.0032416144385933876], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5083450675010681, -0.0024052471853792667], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5109439492225647, -0.002598899882286787], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5126900672912598, -0.001746137160807848], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5155878067016602, -0.0028977561742067337], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5186110138893127, -0.003023165510967374], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.520710825920105, -0.002099822275340557], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.522884726524353, -0.00217388104647398], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5260918736457825, -0.0032071981113404036], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5293306112289429, -0.003238696837797761], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5325558185577393, -0.003225228050723672], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5357164740562439, -0.0031606468837708235], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5388666391372681, -0.0031501937191933393], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.54290771484375, -0.004041068255901337], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5468834042549133, -0.003975661937147379], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5507588386535645, -0.0038754267152398825], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5534729957580566, -0.002714182250201702], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5560405254364014, -0.0025675336364656687], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.557366132736206, -0.0013255986850708723], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5594791173934937, -0.002113001886755228], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5623839497566223, -0.002904785331338644], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5660111904144287, -0.003627269295975566], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5703759789466858, -0.0043647619895637035], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5734575986862183, -0.0030816704966127872], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5762405395507812, -0.002782912692055106], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5796511769294739, -0.0034106578677892685], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.582705557346344, -0.003054338973015547], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5843876004219055, -0.0016820761375129223], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5846678018569946, -0.0002801776572596282], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5835671424865723, 0.0011006336426362395], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5830495953559875, 0.0005175855476409197], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5821473598480225, 0.0009022310841828585], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5808501243591309, 0.001297186827287078], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5791434645652771, 0.0017066954169422388], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5760438442230225, 0.0030995907727628946], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5736358165740967, 0.0024080355651676655], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5698796510696411, 0.0037561815697699785], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5647990107536316, 0.0050806389190256596], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.559490978717804, 0.0053080045618116856], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5529444217681885, 0.006546576973050833], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5461798906326294, 0.006764536257833242], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5402976870536804, 0.005882205907255411], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5343128442764282, 0.005984836257994175], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5292903780937195, 0.005022480618208647], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5243026614189148, 0.004987719934433699], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5183708667755127, 0.00593177555128932], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5115000605583191, 0.006870820187032223], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5057628750801086, 0.005737191066145897], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5011658072471619, 0.004597056657075882], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49775704741477966, 0.003408747026696801], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4945470988750458, 0.003209945047274232], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49157923460006714, 0.0029678710270673037], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4899277687072754, 0.0016514729941263795], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48757898807525635, 0.002348772482946515], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4855206608772278, 0.002058326965197921], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48379576206207275, 0.0017249101074412465], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4824362099170685, 0.0013595526106655598], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48240017890930176, 3.6022913263877854e-05], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4836891293525696, -0.0012889317004010081], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4862784743309021, -0.0025893638376146555], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4902329742908478, -0.003954503685235977], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49549731612205505, -0.0052643269300460815], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5000534057617188, -0.0045560672879219055], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5038300156593323, -0.00377662549726665], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5088151097297668, -0.004985127132385969], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5139462947845459, -0.005131141282618046], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5202324986457825, -0.006286219228059053], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5255771279335022, -0.005344641860574484], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5299603343009949, -0.004383209627121687], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5343713760375977, -0.004411051515489817], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5377740263938904, -0.0034026484936475754], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5420942902565002, -0.004320275504142046], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5463172793388367, -0.004222932271659374], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5514503121376038, -0.00513307424262166], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5574296712875366, -0.005979323294013739], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.563185453414917, -0.0057558161206543446], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5697234869003296, -0.006538021378219128], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5749983787536621, -0.0052749160677194595], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.580920934677124, -0.005922546610236168], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5864444375038147, -0.0055235158652067184], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5915267467498779, -0.005082300398498774], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5961740612983704, -0.004647292196750641], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6012952327728271, -0.005121173337101936], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6058599352836609, -0.004564697854220867], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6098852753639221, -0.004025329370051622], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6143296360969543, -0.004444385413080454], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6171301603317261, -0.0028005039785057306], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6203039884567261, -0.003173856297507882], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.622818648815155, -0.0025146373081952333], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6246082186698914, -0.0017895614728331566], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6267284154891968, -0.0021202347707003355], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6271567344665527, -0.0004282885347492993], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6268823146820068, 0.00027440086705610156], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6259173154830933, 0.000965039012953639], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6242260932922363, 0.0016911891289055347], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6228296756744385, 0.0013964246027171612], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6207717061042786, 0.0020579535048455], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.617029070854187, 0.003742656670510769], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6126534938812256, 0.004375589545816183], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6076633334159851, 0.0049901558086276054], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6011157631874084, 0.006547566968947649], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.594017744064331, 0.007098023314028978], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5874762535095215, 0.006541471462696791], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5815160870552063, 0.00596013804897666], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5761650204658508, 0.005351074505597353], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.569464385509491, 0.00670067872852087], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5624321103096008, 0.007032271008938551], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5561196208000183, 0.006312476471066475], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5485939979553223, 0.007525609340518713], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5409052968025208, 0.007688732817769051], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5321528315544128, 0.008752446621656418], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.524359405040741, 0.007793405558913946], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5176243782043457, 0.006735033355653286], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.511975884437561, 0.005648498423397541], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5054791569709778, 0.00649674516171217], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4981759488582611, 0.007303196936845779], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4921310544013977, 0.0060449037700891495], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4853307902812958, 0.006800264585763216], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47986289858818054, 0.0054678949527442455], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47577473521232605, 0.004088155459612608], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4721180200576782, 0.003656709101051092], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46792685985565186, 0.004191162530332804], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46519553661346436, 0.00273133278824389], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46396538615226746, 0.001230129157193005], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46218737959861755, 0.0017780105117708445], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46093058586120605, 0.0012568082893267274], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46022215485572815, 0.0007084071403369308], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46103161573410034, -0.0008094456279650331], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4613919258117676, -0.00036030367482453585], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4622304439544678, -0.0008385169203393161], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.462568074464798, -0.0003376509412191808], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46240323781967163, 0.0001648421020945534], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4637228548526764, -0.0013196062063798308], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4655725061893463, -0.0018496562261134386], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46893778443336487, -0.003365286160260439], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47273576259613037, -0.0037979669868946075], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47695282101631165, -0.004217056557536125], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48152491450309753, -0.004572105128318071], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48645085096359253, -0.004925931338220835], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4916765093803406, -0.0052256532944738865], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49615922570228577, -0.004482726566493511], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5019213557243347, -0.005762119311839342], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5068929195404053, -0.0049715423956513405], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5120121836662292, -0.005119269248098135], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5182741284370422, -0.006261969450861216], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5246479511260986, -0.006373835727572441], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5320659875869751, -0.007418038789182901], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5384863018989563, -0.006420299410820007], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5458438396453857, -0.007357535418123007], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5540586709976196, -0.008214817382395267], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5631056427955627, -0.009047009982168674], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5729207396507263, -0.009815055876970291], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5834064483642578, -0.010485731065273285], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5935152173042297, -0.010108754970133305], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6041060090065002, -0.010590816847980022], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6151033639907837, -0.010997319594025612], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.624439001083374, -0.00933567713946104], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6341108679771423, -0.009671835228800774], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6420381665229797, -0.007927315309643745], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6491546630859375, -0.007116460241377354], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6563762426376343, -0.007221629377454519], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6616709232330322, -0.005294653121381998], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6650065183639526, -0.0033355746418237686], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6673784852027893, -0.002372015966102481], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6697537899017334, -0.0023752544075250626], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6721296906471252, -0.0023759205359965563], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6724555492401123, -0.0003258723299950361], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6727141737937927, -0.000258616782957688], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6709027290344238, 0.0018114495323970914], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6690531373023987, 0.001849588705226779], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.665203869342804, 0.003849282395094633], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6613727807998657, 0.003831090172752738], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.656609833240509, 0.004762944765388966], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6519381403923035, 0.004671653266996145], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6473859548568726, 0.004552238620817661], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.641964316368103, 0.005421589128673077], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6366990804672241, 0.005265254061669111], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6306599378585815, 0.006039144471287727], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6248966455459595, 0.005763296503573656], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6174634099006653, 0.007433258928358555], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.608390212059021, 0.009073177352547646], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5997479557991028, 0.00864227581769228], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5916116237640381, 0.008136332035064697], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5839830040931702, 0.007628574036061764], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5749471783638, 0.009035815484821796], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5665423274040222, 0.008404886350035667], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5568511486053467, 0.009691174142062664], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5469629764556885, 0.009888190776109695], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5359594821929932, 0.011003448627889156], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5239342451095581, 0.012025258503854275], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5119067430496216, 0.012027484364807606], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49899622797966003, 0.012910542078316212], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48727893829345703, 0.011717271991074085], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4768848121166229, 0.010394125245511532], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46592414379119873, 0.01096067950129509], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45646971464157104, 0.009454426355659962], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4465978443622589, 0.009871866554021835], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43835994601249695, 0.008237921632826328], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43176138401031494, 0.0065985508263111115], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4249296486377716, 0.006831740960478783], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41990453004837036, 0.005025110207498074], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41471511125564575, 0.00518942903727293], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.411363422870636, 0.0033516876865178347], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40985652804374695, 0.0015068941283971071], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4092411398887634, 0.0006153829745016992], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4095379114151001, -0.00029676780104637146], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40976011753082275, -0.00022221283870749176], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.410837858915329, -0.0010777516290545464], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4118162989616394, -0.0009784447029232979], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4136268198490143, -0.0018105192575603724], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4172731637954712, -0.0036463274154812098], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42071884870529175, -0.003445683280006051], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.423986554145813, -0.003267704276368022], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4290522336959839, -0.0050656795501708984], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4338932931423187, -0.004841061774641275], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43844762444496155, -0.004554328974336386], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44370952248573303, -0.0052618905901908875], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44860291481018066, -0.004893413744866848], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45510900020599365, -0.006506086327135563], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46218493580818176, -0.00707591837272048], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.469729483127594, -0.007544543128460646], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4787139892578125, -0.00898450706154108], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4890372157096863, -0.010323245078325272], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4996487498283386, -0.010611525736749172], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5094437003135681, -0.009794957935810089], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5193766951560974, -0.009932998567819595], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5304019451141357, -0.011025226674973965], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5414050221443176, -0.011003118008375168], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5532904267311096, -0.01188535988330841], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5649632811546326, -0.011672891676425934], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5754072666168213, -0.010443956591188908], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5854645371437073, -0.010057277046144009], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.595064640045166, -0.009600082412362099], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6051974296569824, -0.01013281848281622], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6157484650611877, -0.010551031678915024], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6266798377037048, -0.010931397788226604], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6369279623031616, -0.010248111560940742], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6453897356987, -0.008461764082312584], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6540260314941406, -0.008636277168989182], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6627405881881714, -0.008714554831385612], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6704939603805542, -0.007753359153866768], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6781898736953735, -0.0076959580183029175], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6837999820709229, -0.005610120948404074], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6872962713241577, -0.0034962547942996025], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6906664967536926, -0.0033702312503010035], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6938546895980835, -0.003188185626640916], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6958429217338562, -0.0019882475025951862], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6956221461296082, 0.00022078800247982144], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6932061314582825, 0.002415986265987158], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6905931830406189, 0.0026129670441150665], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6867866516113281, 0.003806540509685874], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.680873692035675, 0.0059129828587174416], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6728257536888123, 0.008047910407185555], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.663735568523407, 0.009090213105082512], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6536523103713989, 0.010083233937621117], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6416891813278198, 0.011963138356804848], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6278913617134094, 0.013797796331346035], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6133575439453125, 0.01453380286693573], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5972093939781189, 0.016148151829838753], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5815202593803406, 0.015689173713326454], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5664403438568115, 0.015079930424690247], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5510434508323669, 0.01539685484021902], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5354695320129395, 0.015573938377201557], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5198460817337036, 0.015623425133526325], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5053066611289978, 0.014539459720253944], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48997607827186584, 0.015330573543906212], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4759466350078583, 0.014029419980943203], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4613509476184845, 0.014595690183341503], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44822463393211365, 0.013126308098435402], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43469372391700745, 0.01353093609213829], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4228416085243225, 0.011852100491523743], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41179323196411133, 0.01104838028550148], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4026416838169098, 0.009151531383395195], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3934576213359833, 0.009184062480926514], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3862406313419342, 0.007217005360871553], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.38110411167144775, 0.005136516410857439], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37802451848983765, 0.0030795906204730272], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37608978152275085, 0.0019347519846633077], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.376275897026062, -0.00018612563144415617], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37758931517601013, -0.0013134113978594542], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3799859881401062, -0.002396687865257263], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3844859004020691, -0.004499909933656454], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.391023188829422, -0.006537280511111021], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3995897173881531, -0.008566536009311676], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40809670090675354, -0.008506979793310165], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41846513748168945, -0.01036843005567789], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42965954542160034, -0.01119441818445921], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4425753355026245, -0.012915773317217827], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4551335275173187, -0.012558194808661938], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46927693486213684, -0.014143422245979309], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48284268379211426, -0.01356574147939682], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49573031067848206, -0.012887642718851566], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5098758935928345, -0.01414557360112667], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5241631269454956, -0.014287253841757774], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5374845266342163, -0.013321395963430405], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5517358183860779, -0.014251254498958588], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5668113827705383, -0.015075573697686195], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.581641435623169, -0.014830038882791996], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5951007008552551, -0.01345925871282816], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6071028709411621, -0.012002185918390751], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6195659637451172, -0.012463082559406757], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6323779821395874, -0.012812059372663498], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6444051861763, -0.012027188204228878], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6555779576301575, -0.011172753758728504], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6648167371749878, -0.009238816797733307], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6720871329307556, -0.007270357105880976], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6792952418327332, -0.007208111695945263], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.684380829334259, -0.005085593555122614], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.688309907913208, -0.00392910884693265], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.69107586145401, -0.0027659237384796143], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6936991214752197, -0.0026232816744595766], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6951149702072144, -0.0014158504782244563], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.695302426815033, -0.00018745796114671975], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.694263219833374, 0.0010392206022515893], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6920142769813538, 0.00224892795085907], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6896190643310547, 0.002395226387307048], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6871051788330078, 0.002513863379135728], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.682488739490509, 0.00461648590862751], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6767570972442627, 0.00573161942884326], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6689606308937073, 0.007796445395797491], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6611605286598206, 0.007800129242241383], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6534262299537659, 0.007734276354312897], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6457879543304443, 0.007638269569724798], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6382679343223572, 0.007520031183958054], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6309633851051331, 0.007304565981030464], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.623930037021637, 0.007033334579318762], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6172245144844055, 0.006705543026328087], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6098425388336182, 0.007381973788142204], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6008661985397339, 0.008976349607110023], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5923908352851868, 0.008475332520902157], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5834048390388489, 0.008985989727079868], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5739762783050537, 0.009428556077182293], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5631902813911438, 0.010786032304167747], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5531371831893921, 0.010053073056042194], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5418825149536133, 0.011254682205617428], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5315516591072083, 0.010330884717404842], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5212188363075256, 0.010332804173231125], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5109627842903137, 0.010256021283566952], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5008219480514526, 0.010140886530280113], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4908835291862488, 0.009938403964042664], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48025569319725037, 0.010627823881804943], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47099706530570984, 0.009258619509637356], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46317294239997864, 0.007824134081602097], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45483896136283875, 0.008333978243172169], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44808441400527954, 0.006754559930413961], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.441953182220459, 0.006131226196885109], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43747642636299133, 0.004476745147258043], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.433697909116745, 0.0037785274907946587], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42963293194770813, 0.004064956679940224], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4272706210613251, 0.0023623211309313774], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4247116148471832, 0.0025590008590370417], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42391642928123474, 0.0007952019805088639], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42389750480651855, 1.892348336696159e-05], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4236239790916443, 0.0002735196612775326], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4250958561897278, -0.0014718850143253803], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4273810088634491, -0.0022851345129311085], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4293995797634125, -0.0020185860339552164], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4311494529247284, -0.0017498572124168277], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43464288115501404, -0.0034934361465275288], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4398445785045624, -0.005201704800128937], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44670039415359497, -0.006855817511677742], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45417797565460205, -0.007477583363652229], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4632214307785034, -0.009043456986546516], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.471725195646286, -0.008503751829266548], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47962725162506104, -0.007902048528194427], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4889170825481415, -0.00928984209895134], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4995116591453552, -0.01059458963572979], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5093461275100708, -0.00983448512852192], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5193578600883484, -0.010011715814471245], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.530423104763031, -0.011065240018069744], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.542492687702179, -0.01206954661756754], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5534803867340088, -0.010987735353410244], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5652887225151062, -0.011808336712419987], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5768511295318604, -0.011562422849237919], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5870924592018127, -0.01024128869175911], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5978679060935974, -0.010775484144687653], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6070998311042786, -0.009231885895133018], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6167905926704407, -0.009690770879387856], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.626807451248169, -0.010016866959631443], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6371209025382996, -0.010313465259969234], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6466594338417053, -0.009538501501083374], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6553665399551392, -0.008707142435014248], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6621190309524536, -0.006752470042556524], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6689125299453735, -0.006793494801968336], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6756657361984253, -0.006753238849341869], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6813359260559082, -0.005670188460499048], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6868806481361389, -0.0055446932092309], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6913250684738159, -0.004444430582225323], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6956409215927124, -0.004315853584557772], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6977431178092957, -0.002102165250107646], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6996490359306335, -0.0019059664336964488], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7013376355171204, -0.0016885794466361403], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7017611861228943, -0.00042355034383945167], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7019112706184387, -0.0001500823418609798], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.701808512210846, 0.00010274641681462526], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7004781365394592, 0.0013304156018421054], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6968847513198853, 0.0035933679901063442], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6911073923110962, 0.005777379963546991], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6841573715209961, 0.006950004491955042], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6761101484298706, 0.008047192357480526], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6669698357582092, 0.009140326641499996], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6568120718002319, 0.010157771408557892], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6457481980323792, 0.01106385700404644], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6328083872795105, 0.012939805164933205], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6181405186653137, 0.014667868614196777], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6028442978858948, 0.01529624778777361], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5880255699157715, 0.014818704687058926], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5727776885032654, 0.01524790283292532], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5581898093223572, 0.01458785030990839], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5433822274208069, 0.014807596802711487], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5284692645072937, 0.014912975020706654], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5125822424888611, 0.015887022018432617], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49782806634902954, 0.014754174277186394], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4843044877052307, 0.013523568399250507], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4701324701309204, 0.014172029681503773], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45542600750923157, 0.014706451445817947], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4402877390384674, 0.015138288959860802], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4248470067977905, 0.015440710820257664], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41114455461502075, 0.013702468015253544], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3993058502674103, 0.011838693171739578], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3884105384349823, 0.010895316489040852], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37855374813079834, 0.009856787510216236], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36884352564811707, 0.009710218757390976], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3592977523803711, 0.009545775130391121], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3499380350112915, 0.009359716437757015], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.34187135100364685, 0.008066685870289803], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33516669273376465, 0.00670465687289834], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3298436999320984, 0.0053230044431984425], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3259719908237457, 0.0038716995622962713], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3235418498516083, 0.00243013771250844], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3225719630718231, 0.0009698907961137593], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3230527639389038, -0.0004808007797691971], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.32395827770233154, -0.0009055103873834014], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.32527396082878113, -0.0013156856875866652], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3270091414451599, -0.0017351702554151416], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3291476368904114, -0.0021385038271546364], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33267033100128174, -0.003522680839523673], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3385743796825409, -0.005904067773371935], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3458311855792999, -0.00725678913295269], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3554081618785858, -0.009576971642673016], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36522138118743896, -0.009813242591917515], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3762608766555786, -0.011039480566978455], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.38741809129714966, -0.01115722768008709], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3996557593345642, -0.01223766803741455], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41188845038414, -0.012232692912220955], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4260327219963074, -0.014144258573651314], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44198891520500183, -0.015956204384565353], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4595973491668701, -0.017608441412448883], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.477752685546875, -0.018155336380004883], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49728038907051086, -0.019527703523635864], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5170096158981323, -0.019729238003492355], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5378297567367554, -0.0208201315253973], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5595858693122864, -0.021756097674369812], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5811392664909363, -0.02155340276658535], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6023310422897339, -0.021191800013184547], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6240003108978271, -0.02166924625635147], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6449528336524963, -0.020952492952346802], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6660266518592834, -0.02107381820678711], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6860679388046265, -0.020041311159729958], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7039881944656372, -0.017920251935720444], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.719685971736908, -0.015697790309786797], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7330267429351807, -0.013340791687369347], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7439622282981873, -0.01093545276671648], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7523941397666931, -0.008431901223957539], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7602511644363403, -0.007857019081711769], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7654819488525391, -0.005230822134763002], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7681103944778442, -0.002628448884934187], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7681325674057007, -2.216328175563831e-05], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7655192017555237, 0.0026133849751204252], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7613048553466797, 0.0042143273167312145], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7564687132835388, 0.004836132284253836], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7490378618240356, 0.007430858910083771], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7390862107276917, 0.009951657615602016], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7266337871551514, 0.012452425435185432], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7138216495513916, 0.01281213853508234], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6996731162071228, 0.014148540794849396], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6853371858596802, 0.014335950836539268], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6698561310768127, 0.015481043606996536], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6523776650428772, 0.017478466033935547], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6330317258834839, 0.019345909357070923], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6138885021209717, 0.019143259152770042], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5951533317565918, 0.018735161051154137], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5758986473083496, 0.019254671409726143], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5562707185745239, 0.019627956673502922], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5374215841293335, 0.01884910650551319], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5185425281524658, 0.018879059702157974], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5007860064506531, 0.017756493762135506], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4832504987716675, 0.01753551885485649], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4670874774456024, 0.016163049265742302], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4523734450340271, 0.014714024029672146], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4391998052597046, 0.013173647224903107], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4266754388809204, 0.012524344958364964], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41489115357398987, 0.011784299276769161], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40391966700553894, 0.01097146887332201], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3948328495025635, 0.009086823090910912], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.38672828674316406, 0.008104554377496243], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37969696521759033, 0.0070313275791704655], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3747478127479553, 0.004949149210005999], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3699158728122711, 0.004831952042877674], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36728718876838684, 0.0026286961510777473], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3658738434314728, 0.0014133407967165112], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36662745475769043, -0.0007536247721873224], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3675842583179474, -0.000956808973569423], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3707543611526489, -0.00317010167054832], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37511080503463745, -0.004356446675956249], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37956953048706055, -0.004458716604858637], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3851472735404968, -0.005577730946242809], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.390754759311676, -0.005607502069324255], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39634525775909424, -0.005590497050434351], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.403875470161438, -0.00753021938726306], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4133603870868683, -0.009484910406172276], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4237235486507416, -0.01036315131932497], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43590405583381653, -0.012180518358945847], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44978034496307373, -0.013876267708837986], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4632178544998169, -0.013437525369226933], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4781390428543091, -0.014921179972589016], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4924006760120392, -0.014261636883020401], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.505940854549408, -0.013540154322981834], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5186970829963684, -0.012756230309605598], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5315362215042114, -0.012839156202971935], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5443320870399475, -0.012795854359865189], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5570091009140015, -0.012677016668021679], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5684456825256348, -0.011436572298407555], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5795915126800537, -0.011145880445837975], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5903295278549194, -0.010737979784607887], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6015780568122864, -0.01124853827059269], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.612273097038269, -0.010695042088627815], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6223455667495728, -0.0100724957883358], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6306979656219482, -0.008352342061698437], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6373077034950256, -0.0066097392700612545], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6420817971229553, -0.004774104338139296], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6450402140617371, -0.0029584430158138275], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6461442112922668, -0.001103964983485639], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6454048752784729, 0.0007393205305561423], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6438308954238892, 0.0015739620430395007], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6403946280479431, 0.0034363034646958113], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6351686120033264, 0.00522600719705224], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6301481127738953, 0.00502051180228591], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6243968605995178, 0.005751202814280987], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.617953896522522, 0.006442998070269823], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6108292937278748, 0.007124566473066807], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6021121144294739, 0.008717180229723454], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5918826460838318, 0.010229472070932388], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5801675915718079, 0.011715083383023739], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5690932869911194, 0.01107428316026926], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5567342638969421, 0.01235902775079012], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5451797246932983, 0.011554558761417866], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5335444808006287, 0.01163520384579897], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5208942890167236, 0.012650198303163052], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5092874765396118, 0.011606806889176369], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4987988770008087, 0.010488610714673996], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48758313059806824, 0.0112157566472888], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4766516387462616, 0.01093147974461317], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4660720229148865, 0.01057962141931057], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4559739828109741, 0.010098057799041271], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4474039077758789, 0.008570067584514618], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4404352605342865, 0.006968642119318247], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4351687431335449, 0.005266535561531782], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43160882592201233, 0.003559897653758526], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42876362800598145, 0.002845209091901779], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4256289303302765, 0.0031347074545919895], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42429855465888977, 0.001330363331362605], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42472028732299805, -0.00042174916598014534], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.425878643989563, -0.001158347469754517], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4268037974834442, -0.0009251388255506754], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4274972975254059, -0.0006935125566087663], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42897772789001465, -0.0014804401434957981], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4312284290790558, -0.00225068349391222], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43317970633506775, -0.0019512861035764217], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43683844804763794, -0.003658744040876627], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4402036666870117, -0.003365210723131895], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44418904185295105, -0.00398536492139101], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44981321692466736, -0.005624187644571066], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45702266693115234, -0.007209440227597952], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4647813141345978, -0.007758655585348606], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47298958897590637, -0.008208267390727997], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4815932512283325, -0.0086036566644907], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48954829573631287, -0.007955070585012436], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4967966079711914, -0.007248306646943092], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5052783489227295, -0.00848174374550581], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5139791965484619, -0.008700834587216377], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5237712264060974, -0.009792032651603222], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.534641444683075, -0.010870205238461494], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5464503765106201, -0.011808955110609531], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5591621398925781, -0.012711775489151478], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5716877579689026, -0.01252557523548603], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5848554968833923, -0.01316777989268303], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5975918769836426, -0.012736343778669834], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6098501086235046, -0.012258226983249187], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6205460429191589, -0.010695967823266983], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6305891275405884, -0.010043100453913212], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6389150619506836, -0.008325918577611446], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6463968753814697, -0.007481801323592663], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6530277132987976, -0.006630809977650642], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6587209701538086, -0.005693314131349325], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.662455677986145, -0.0037346642930060625], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6641841530799866, -0.0017284934874624014], action=2, reward=-1.0, info={}, terminal=False, timeout=True)]),
EpisodeItem(id='d250f9c779ac4d6e8ff137f00b6ee6ae', created_by='hannah', is_public=False, created_at='2025-07-28T15:45:24.932031', benchmark_id='4b076cdedc4d433ea4da6d2801c05ca0', metadata={'cum_eps_reward': [-999.0], 'is_expert': False}, n_tuples=999, terminated=False, timeout=True, tuples=[RLTuple(state=[-0.4620797634124756, -0.0015533595578745008], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46311667561531067, -0.0010369353694841266], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4636554718017578, -0.0005387924029491842], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4636871814727783, -3.168750845361501e-05], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4631742835044861, 0.0005128714838065207], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4641742408275604, -0.0009999375324696302], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4647003710269928, -0.0005261305486783385], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46568867564201355, -0.000988305313512683], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46715834736824036, -0.0014696900034323335], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4681317210197449, -0.0009733674232847989], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4685547351837158, -0.00042301174835301936], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4694516062736511, -0.0008968732436187565], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4707925021648407, -0.0013408743543550372], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47355666756629944, -0.0027641840279102325], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47678086161613464, -0.0032242017332464457], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4803967773914337, -0.0036158922594040632], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4833410978317261, -0.002944346284493804], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48666054010391235, -0.00331941363401711], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4913409650325775, -0.004680447280406952], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4963468909263611, -0.005005928222090006], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.500621497631073, -0.004274584352970123], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5041199922561646, -0.0034985167440027], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5077944397926331, -0.0036744100507348776], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5106481313705444, -0.0028537169564515352], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.514640748500824, -0.003992646466940641], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5197306871414185, -0.005089929327368736], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.524857223033905, -0.005126517731696367], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5300573706626892, -0.00520012341439724], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5352672934532166, -0.00520992511883378], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5393897891044617, -0.004122511949390173], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5444065928459167, -0.0050168149173259735], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5503265261650085, -0.005919900722801685], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5550972819328308, -0.004770768340677023], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5606721043586731, -0.005574864335358143], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5660036206245422, -0.005331498570740223], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5710687041282654, -0.005065055564045906], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.575859546661377, -0.00479084812104702], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5812704563140869, -0.0054109361954033375], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5863335132598877, -0.005063020624220371], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.590011715888977, -0.0036782226525247097], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5922427773475647, -0.002231091493740678], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5949831008911133, -0.0027402909472584724], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5962463617324829, -0.0012632847065106034], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5960075259208679, 0.00023884265101514757], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5962688326835632, -0.00026131432969123125], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5960057377815247, 0.00026312435511499643], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5952776670455933, 0.0007280674763023853], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5940197706222534, 0.0012578937457874417], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5933127403259277, 0.0007070290157571435], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5911433696746826, 0.002169352490454912], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5885312557220459, 0.0026121053379029036], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5854758620262146, 0.003055403009057045], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5829851627349854, 0.0024906927719712257], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5801077485084534, 0.0028774505481123924], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.577838122844696, 0.002269615652039647], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5742012858390808, 0.0036368495784699917], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5711898803710938, 0.0030113623943179846], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5688499808311462, 0.0023399244528263807], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5671938061714172, 0.001656163134612143], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5662268400192261, 0.0009669645805843174], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5639423727989197, 0.0022844653576612473], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5604087710380554, 0.003533588722348213], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5566232204437256, 0.003785578301176429], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.551607608795166, 0.005015608388930559], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5454546809196472, 0.0061529241502285], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5391408801078796, 0.0063137938268482685], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5317701697349548, 0.00737070944160223], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.525379478931427, 0.006390734575688839], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5199962258338928, 0.005383205134421587], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5146602988243103, 0.005335927940905094], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.509441614151001, 0.005218690726906061], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5043932199478149, 0.005048389546573162], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5005105137825012, 0.003882738994434476], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49582037329673767, 0.0046901400201022625], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4924129843711853, 0.0034073791466653347], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4892588257789612, 0.003154163248836994], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4853821098804474, 0.0038767135702073574], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48086974024772644, 0.004512356594204903], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4777056872844696, 0.003164049470797181], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47494763135910034, 0.002758071990683675], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4735751152038574, 0.001372512779198587], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4716646075248718, 0.0019105055835098028], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47122058272361755, 0.0004440185730345547], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47224465012550354, -0.0010240724077448249], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47370782494544983, -0.00146315002348274], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4755919575691223, -0.0018841472920030355], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47692349553108215, -0.0013315515825524926], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4796389937400818, -0.002715469803661108], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4837602376937866, -0.004121263977140188], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48925310373306274, -0.00549285439774394], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49404317140579224, -0.004790080711245537], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49811434745788574, -0.004071182571351528], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5024473071098328, -0.004332936368882656], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.506969153881073, -0.00452187517657876], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5116336345672607, -0.004664445295929909], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5174476504325867, -0.005814045667648315], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5243600606918335, -0.006912407465279102], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5313421487808228, -0.006982103455811739], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5393414497375488, -0.007999276742339134], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5472245216369629, -0.007883044891059399], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5539529323577881, -0.006728441454470158], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5604893565177917, -0.006536389701068401], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5657520890235901, -0.005262777209281921], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5717750787734985, -0.006022996734827757], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5774879455566406, -0.0057128663174808025], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.583870530128479, -0.006382535211741924], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.588874340057373, -0.005003852769732475], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5924260020256042, -0.0035516477655619383], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5945032835006714, -0.00207728473469615], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5971071124076843, -0.0026038025971502066], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6002007722854614, -0.003093663603067398], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6037359833717346, -0.003535239025950432], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6077535152435303, -0.004017519764602184], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.610228419303894, -0.0024748824071139097], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6110916137695312, -0.0008632463868707418], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6113694906234741, -0.0002778734778985381], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6110629439353943, 0.0003065424971282482], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6111790537834167, -0.00011605852341745049], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6097182631492615, 0.0014607880730181932], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6066911816596985, 0.0030270610004663467], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6020748019218445, 0.0046163639053702354], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5969110727310181, 0.005163747351616621], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5912734866142273, 0.0056375740095973015], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5841622352600098, 0.007111242972314358], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5756305456161499, 0.008531737141311169], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5657809972763062, 0.009849541820585728], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5556802153587341, 0.010100752115249634], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5443792343139648, 0.011301005259156227], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5320006608963013, 0.012378562241792679], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5206412672996521, 0.01135940756648779], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5103907585144043, 0.01025049202144146], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5002973675727844, 0.010093367658555508], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4894266426563263, 0.010870737954974174], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47788745164871216, 0.011539188213646412], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46772903203964233, 0.010158422403037548], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45801135897636414, 0.009717668406665325], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44982481002807617, 0.008186559192836285], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44326338171958923, 0.006561423186212778], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4383825659751892, 0.004880816675722599], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43514204025268555, 0.0032405382953584194], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4316386878490448, 0.003503341693431139], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42781737446784973, 0.0038213245570659637], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42576864361763, 0.002048719208687544], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4244438409805298, 0.0013248197501525283], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.423903226852417, 0.0005405936390161514], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4241439402103424, -0.00024069903884083033], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42513179779052734, -0.0009878630517050624], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42785099148750305, -0.0027192039415240288], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4303564429283142, -0.002505444223061204], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4336375296115875, -0.0032810880802571774], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.438615620136261, -0.004978087730705738], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44431251287460327, -0.005696894600987434], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.450639009475708, -0.006326483562588692], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4585273265838623, -0.00788831152021885], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4659695029258728, -0.007442180532962084], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4729262888431549, -0.006956805475056171], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48129528760910034, -0.008368981070816517], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48901286721229553, -0.007717592641711235], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49799850583076477, -0.008985637687146664], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5082330703735352, -0.010234583169221878], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5196212530136108, -0.011388181708753109], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5300408601760864, -0.010419604368507862], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5414646863937378, -0.01142383087426424], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5527731776237488, -0.011308453977108002], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5629401803016663, -0.010167027823626995], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5718720555305481, -0.008931845426559448], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5794469714164734, -0.007574940100312233], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5856143832206726, -0.006167425774037838], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5913693904876709, -0.0057550170458853245], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5956305265426636, -0.004261126276105642], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5983697175979614, -0.0027391863986849785], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6015850305557251, -0.00321531156077981], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6032876968383789, -0.0017026611603796482], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6044674515724182, -0.0011797742918133736], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6050931811332703, -0.0006257205968722701], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6051626205444336, -6.940473394934088e-05], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6046956777572632, 0.0004669383342843503], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6027078032493591, 0.0019878901075571775], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5991933345794678, 0.0035144281573593616], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5951886177062988, 0.004004735965281725], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5907052755355835, 0.004483333323150873], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5847997665405273, 0.005905528552830219], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5784826874732971, 0.006317084655165672], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5727648138999939, 0.005717862397432327], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5667204856872559, 0.006044305860996246], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5603989362716675, 0.0063215866684913635], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5548335313796997, 0.0055653927847743034], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5500695109367371, 0.004763992503285408], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5441445112228394, 0.0059249852783977985], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.537094235420227, 0.0070502799935638905], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5310179591178894, 0.006076272577047348], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.52593994140625, 0.0050780451856553555], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5199065804481506, 0.006033382844179869], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5139585733413696, 0.0059479898773133755], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5091069340705872, 0.004851627629250288], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5034359097480774, 0.005671034567058086], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49796468019485474, 0.0054712132550776005], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4937131404876709, 0.004251543898135424], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4897054433822632, 0.004007700830698013], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4850206971168518, 0.004684746265411377], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4816705584526062, 0.003350131446495652], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47863689064979553, 0.003033681306988001], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47697383165359497, 0.0016630615573376417], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47573012113571167, 0.0012437158729881048], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4739108979701996, 0.0018192214192822576], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4715195894241333, 0.0023913164623081684], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4706002175807953, 0.0009193597361445427], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4701283276081085, 0.0004718777199741453], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4710913598537445, -0.0009630135027691722], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47345075011253357, -0.0023594049271196127], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4761889576911926, -0.0027381908148527145], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4793001711368561, -0.0031112227588891983], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48281365633010864, -0.0035134984645992517], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48569074273109436, -0.0028770864009857178], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4899305999279022, -0.004239832982420921], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49543511867523193, -0.005504521541297436], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5002095103263855, -0.0047743795439600945], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.505210280418396, -0.0050007994286715984], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5094142556190491, -0.004203954711556435], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5127448439598083, -0.003330587176606059], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5151803493499756, -0.0024355307687073946], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5187106132507324, -0.0035302808973938227], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5212984085083008, -0.002587751019746065], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5249152183532715, -0.003616807283833623], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5295274257659912, -0.004612228833138943], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5351417660713196, -0.005614311899989843], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5407023429870605, -0.005560589488595724], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5472149848937988, -0.006512636784464121], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.554569661617279, -0.0073547218926250935], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.561725378036499, -0.007155676372349262], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5696666240692139, -0.007941252551972866], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5763413906097412, -0.00667474465444684], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5816472172737122, -0.00530586251989007], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5875250101089478, -0.005877808202058077], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5929906368255615, -0.005465621594339609], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5969498753547668, -0.003959193825721741], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6004188656806946, -0.003469032235443592], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6033341288566589, -0.0029152201022952795], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.604672908782959, -0.001338787260465324], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6054509878158569, -0.0007781008607707918], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6046656966209412, 0.0007852792041376233], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6033093333244324, 0.0013563488610088825], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6004076600074768, 0.002901701023802161], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5979411005973816, 0.002466543111950159], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.593949019908905, 0.003992094658315182], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5884336829185486, 0.0055153691209852695], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5814746618270874, 0.006958963815122843], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5731542110443115, 0.008320506662130356], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5645243525505066, 0.008629810996353626], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5565913915634155, 0.007932956330478191], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5474555492401123, 0.009135870262980461], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5381640195846558, 0.009291518479585648], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5297752618789673, 0.008388788439333439], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.522416889667511, 0.007358354516327381], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5161131024360657, 0.006303796544671059], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5109120607376099, 0.0052010235376656055], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.504838228225708, 0.00607384042814374], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4979778230190277, 0.006860385183244944], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49135205149650574, 0.006625796668231487], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4849870800971985, 0.006364947650581598], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4789591133594513, 0.0060279713943600655], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4743484854698181, 0.004610637668520212], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4711673855781555, 0.0031810912769287825], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4674232602119446, 0.003744141198694706], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46514812111854553, 0.002275136997923255], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4643889367580414, 0.0007591701578348875], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4651273190975189, -0.0007383712218143046], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46638837456703186, -0.0012610669946298003], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4681301414966583, -0.0017417578492313623], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46936485171318054, -0.0012346983421593904], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4720255732536316, -0.0026607345789670944], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47609999775886536, -0.004074424505233765], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47960084676742554, -0.0035008369013667107], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4844367504119873, -0.004835901781916618], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4906136393547058, -0.00617691595107317], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4970634877681732, -0.006449843291193247], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5027965307235718, -0.005733045283704996], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5077382922172546, -0.004941771272569895], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5118641257286072, -0.004125795792788267], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5150841474533081, -0.0032200647983700037], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5183988809585571, -0.0033146825153380632], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5217534303665161, -0.0033545943442732096], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.526185929775238, -0.004432457499206066], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5316268801689148, -0.005441001150757074], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.538093626499176, -0.006466742139309645], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5455363392829895, -0.007442703004926443], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.552893340587616, -0.007357007823884487], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.559059202671051, -0.006165867205709219], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5649682879447937, -0.005909034051001072], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5715976357460022, -0.006629372481256723], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5779420733451843, -0.006344416178762913], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5829121470451355, -0.004970096983015537], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5864418745040894, -0.0035297481808811426], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5895176529884338, -0.003075747285038233], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5921794772148132, -0.002661819336935878], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.594408392906189, -0.002228911267593503], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5971697568893433, -0.0027613877318799496], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5994538068771362, -0.0022840453311800957], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6012437343597412, -0.001789923757314682], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6025035977363586, -0.0012598553439602256], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.60219806432724, 0.0003055022389162332], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6003561615943909, 0.0018419049447402358], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5990056395530701, 0.0013505668612197042], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5981320738792419, 0.0008735337178222835], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5957745909690857, 0.002357504330575466], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5918839573860168, 0.003890636609867215], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5865125060081482, 0.005371415987610817], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5807346105575562, 0.005777925252914429], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5745490193367004, 0.006185558624565601], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5669994950294495, 0.007549549452960491], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5592004060745239, 0.0077990698628127575], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5511481165885925, 0.008052289485931396], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5439652800559998, 0.007182827219367027], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5356771349906921, 0.00828818790614605], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.526353657245636, 0.009323452599346638], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5170776844024658, 0.009275969117879868], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5068805813789368, 0.010197118856012821], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4978550970554352, 0.009025479666888714], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49010005593299866, 0.0077550411224365234], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48366186022758484, 0.006438203155994415], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47657039761543274, 0.00709144352003932], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4688740670681, 0.007696330547332764], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46061715483665466, 0.008256909437477589], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4518496096134186, 0.008767555467784405], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44263237714767456, 0.009217227809131145], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43505188822746277, 0.007580501027405262], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42820191383361816, 0.006849963217973709], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4231380820274353, 0.005063826683908701], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4178263247013092, 0.005311776418238878], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4143410325050354, 0.0034852810204029083], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.410736083984375, 0.0036049429327249527], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4070492088794708, 0.0036868813913315535], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4052824079990387, 0.0017668086802586913], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4034050405025482, 0.0018773592310026288], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40249505639076233, 0.0009099919116124511], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40152865648269653, 0.0009663997334428132], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40149784088134766, 3.079755697399378e-05], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.402397096157074, -0.0008992458460852504], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40325894951820374, -0.000861835025716573], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40600886940956116, -0.002749936655163765], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40967878699302673, -0.0036699010524898767], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41522055864334106, -0.00554179260507226], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42162153124809265, -0.006400957237929106], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4278194010257721, -0.00619786512106657], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4337644875049591, -0.005945099052041769], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44042137265205383, -0.006656872574239969], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44669848680496216, -0.00627713929861784], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45360252261161804, -0.006904012057930231], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46006205677986145, -0.006459536496549845], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4680699408054352, -0.008007894270122051], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4755026400089264, -0.007432688493281603], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4843672215938568, -0.008864578790962696], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4925902187824249, -0.008223014883697033], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5020645260810852, -0.009474321268498898], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5107531547546387, -0.008688598871231079], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5206028819084167, -0.009849727153778076], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5304844379425049, -0.009881554171442986], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5403740406036377, -0.009889612905681133], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5511700510978699, -0.010795986279845238], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.560779869556427, -0.009609874337911606], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5711884498596191, -0.010408579371869564], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5803173184394836, -0.009128833189606667], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5890456438064575, -0.008728311397135258], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5963332653045654, -0.007287653163075447], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6031460165977478, -0.006812721025198698], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.608389139175415, -0.005243120715022087], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6120142936706543, -0.0036251996643841267], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6140592694282532, -0.0020449701696634293], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.616455078125, -0.0023958024103194475], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6191974878311157, -0.0027423882856965065], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6212918758392334, -0.002094410825520754], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.622667133808136, -0.001375243067741394], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6243771910667419, -0.0017100849654525518], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6253997087478638, -0.0010224975412711501], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6266892552375793, -0.0012895463733002543], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6262644529342651, 0.0004248041659593582], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6251477003097534, 0.0011167596094310284], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6233533620834351, 0.001794312964193523], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.621870219707489, 0.0014831381849944592], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6197110414505005, 0.002159183146432042], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6168670058250427, 0.002844033529981971], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6123937964439392, 0.0044732061214745045], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6072793006896973, 0.005114544648677111], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6006025671958923, 0.006676734425127506], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5944199562072754, 0.006182569079101086], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5877220630645752, 0.006697915494441986], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5795899033546448, 0.008132151328027248], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5720922350883484, 0.007497698999941349], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5643031597137451, 0.0077890390530228615], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5572402477264404, 0.007062949705868959], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.54996657371521, 0.007273650262504816], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5435577034950256, 0.006408871151506901], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5380790829658508, 0.005478589795529842], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5315131545066833, 0.006565964315086603], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5259196162223816, 0.005593517795205116], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5203738212585449, 0.005545794032514095], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5148882269859314, 0.005485619883984327], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5085333585739136, 0.006354848388582468], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5023049712181091, 0.006228374317288399], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4962388873100281, 0.006066089496016502], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4903920590877533, 0.005846826825290918], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4858677089214325, 0.0045243497006595135], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.482702374458313, 0.003165356582030654], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4798746109008789, 0.0028277563396841288], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47645434737205505, 0.003420249791815877], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4744398593902588, 0.002014514524489641], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4718639552593231, 0.0025758875999599695], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4697414040565491, 0.00212254305370152], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4680613577365875, 0.0016800708835944533], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4678497910499573, 0.00021155639842618257], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4680832028388977, -0.00023342019994743168], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46978288888931274, -0.0016996670747175813], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4719149172306061, -0.002132028341293335], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4744356870651245, -0.00252078240737319], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47840702533721924, -0.003971352241933346], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4837218225002289, -0.005314779933542013], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4904169738292694, -0.0066951666958630085], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4974372982978821, -0.007020309567451477], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5046971440315247, -0.007259872276335955], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5111780762672424, -0.006480897776782513], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5187531113624573, -0.0075750527903437614], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5264056921005249, -0.007652556989341974], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.534098744392395, -0.007693089544773102], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5427260398864746, -0.008627290837466717], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5502654910087585, -0.007539437618106604], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5586725473403931, -0.008407083339989185], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5658915638923645, -0.007219002116471529], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5728152990341187, -0.0069237155839800835], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5784335732460022, -0.005618303548544645], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5837113857269287, -0.005277805961668491], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5895923376083374, -0.0058809188194572926], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5960559248924255, -0.00646359845995903], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6030104756355286, -0.006954575423151255], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.608433187007904, -0.005422693677246571], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6132676601409912, -0.004834446590393782], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6184550523757935, -0.005187393166124821], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6239617466926575, -0.0055067227222025394], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6288076639175415, -0.004845907911658287], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6339023113250732, -0.005094646010547876], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6382052898406982, -0.0043029929511249065], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.641742467880249, -0.003537154057994485], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6444404125213623, -0.002697960939258337], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6462535262107849, -0.0018131077522411942], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6472154855728149, -0.0009619859047234058], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6462924480438232, 0.0009230475989170372], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6455094814300537, 0.0007830053800716996], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6429040431976318, 0.0026054177433252335], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6404794454574585, 0.0024246207904070616], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6362161040306091, 0.004263297654688358], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6311637759208679, 0.005052371881902218], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.626324474811554, 0.0048392475582659245], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6217455863952637, 0.0045789023861289024], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6154459714889526, 0.006299618631601334], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6074888706207275, 0.007957096211612225], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5999805927276611, 0.007508289068937302], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5909517407417297, 0.009028824977576733], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5804408192634583, 0.010510969907045364], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5685513615608215, 0.011889411136507988], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5563901662826538, 0.012161238119006157], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5439983010292053, 0.012391828931868076], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5304697751998901, 0.013528550043702126], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5158914923667908, 0.014578301459550858], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5014554262161255, 0.014436060562729836], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48724493384361267, 0.01421047281473875], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47438815236091614, 0.012856771238148212], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4629438519477844, 0.011444312520325184], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4520135819911957, 0.010930274613201618], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4406523108482361, 0.011361273005604744], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4289926290512085, 0.011659683659672737], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4170677661895752, 0.011924856342375278], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40498480200767517, 0.012082959525287151], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39485326409339905, 0.0101315313950181], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.38574114441871643, 0.00911213830113411], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3766595125198364, 0.009081636555492878], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.369689404964447, 0.006970102898776531], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36292076110839844, 0.006768625695258379], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.35737136006355286, 0.005549421533942223], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.35405048727989197, 0.0033208648674190044], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3520340919494629, 0.002016401616856456], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3513394594192505, 0.0006946252542547882], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.35291537642478943, -0.0015759352827444673], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3567485213279724, -0.0038331435061991215], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36281222105026245, -0.006063672713935375], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37111544609069824, -0.008303239941596985], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3815385103225708, -0.010423053987324238], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39401867985725403, -0.01248016394674778], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40852653980255127, -0.014507866464555264], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4229464530944824, -0.014419922605156898], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4381413459777832, -0.01519489660859108], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45504310727119446, -0.016901744529604912], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4715171456336975, -0.01647404581308365], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48738911747932434, -0.015871990472078323], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5026156902313232, -0.015226575545966625], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5180744528770447, -0.015458764508366585], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5346407294273376, -0.01656624674797058], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5502002835273743, -0.015559577383100986], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5655913352966309, -0.015391037799417973], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.580707848072052, -0.015116524882614613], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5954720377922058, -0.014764164574444294], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6087357401847839, -0.013263698667287827], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6224268078804016, -0.013691089116036892], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6344326138496399, -0.012005803175270557], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6446875333786011, -0.010254932567477226], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6550790667533875, -0.01039151195436716], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6655871272087097, -0.010508077219128609], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6761165261268616, -0.010529402643442154], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6855800747871399, -0.009463531896471977], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6949157118797302, -0.009335625916719437], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7020325660705566, -0.007116891909390688], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7088835835456848, -0.006851015146821737], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7134426236152649, -0.004559000488370657], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7166749238967896, -0.003232320537790656], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7175968885421753, -0.0009219487546943128], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7161648869514465, 0.0014320043846964836], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7144234776496887, 0.0017414102330803871], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7123640179634094, 0.0020594156812876463], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7080194354057312, 0.0043446095660328865], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7034112811088562, 0.004608160350471735], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6975778341293335, 0.005833426956087351], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6905058026313782, 0.007072017528116703], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6832737326622009, 0.0072320797480642796], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6749354004859924, 0.008338326588273048], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6665656566619873, 0.008369768969714642], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6562276482582092, 0.01033798884600401], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6439602375030518, 0.0122674023732543], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6308681964874268, 0.01309208944439888], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6179919242858887, 0.01287624891847372], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6054747104644775, 0.012517208233475685], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5924240946769714, 0.013050619512796402], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5779029726982117, 0.01452112477272749], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5630552768707275, 0.014847713522613049], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5469259023666382, 0.016129380092024803], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5316612720489502, 0.015264620073139668], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5153769254684448, 0.016284311190247536], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4991815686225891, 0.016195377334952354], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4842510223388672, 0.014930551871657372], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4696967303752899, 0.014554290100932121], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4555695056915283, 0.014127235859632492], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44098836183547974, 0.014581117779016495], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4270269572734833, 0.01396140456199646], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41281992197036743, 0.014207040891051292], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40045100450515747, 0.012368935160338879], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3880036175251007, 0.01244736835360527], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3776231110095978, 0.010380509309470654], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36735254526138306, 0.010270562022924423], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3582353889942169, 0.009117155335843563], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3493681848049164, 0.008867214433848858], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3408268094062805, 0.008541373535990715], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33464527130126953, 0.006181542761623859], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3298782706260681, 0.004767003934830427], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3255375921726227, 0.004340675659477711], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3226231038570404, 0.002914493205025792], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3201935291290283, 0.002429576823487878], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3182680010795593, 0.0019255157094448805], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.31783127784729004, 0.00043674063635990024], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.31792035698890686, -8.909193275030702e-05], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3194589614868164, -0.001538618584163487], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.32143986225128174, -0.0019808781798928976], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.325864315032959, -0.0044244700111448765], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33178091049194336, -0.005916579160839319], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33813926577568054, -0.006358354818075895], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.34488722681999207, -0.006747981999069452], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3539414703845978, -0.009054237976670265], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36426451802253723, -0.01032303087413311], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3747933506965637, -0.01052883267402649], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.38747894763946533, -0.012685599736869335], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40021443367004395, -0.01273548137396574], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4139271676540375, -0.01371276006102562], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4274798035621643, -0.013552634976804256], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4407578706741333, -0.01327804010361433], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45366397500038147, -0.012906118296086788], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46612638235092163, -0.012462400831282139], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48008251190185547, -0.01395613607019186], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49541884660720825, -0.015336339361965656], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5110468864440918, -0.015628015622496605], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5278379321098328, -0.016791081055998802], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.545660674571991, -0.017822707071900368], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5633586049079895, -0.017697935923933983], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5817859768867493, -0.018427401781082153], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6008396744728088, -0.019053664058446884], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6193856596946716, -0.018545987084507942], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.637301504611969, -0.017915872856974602], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6544350981712341, -0.01713356003165245], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6696704030036926, -0.015235304832458496], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6829133629798889, -0.013242983259260654], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6950198411941528, -0.012106481939554214], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7059363126754761, -0.010916488245129585], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7155656218528748, -0.009629311040043831], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7228646874427795, -0.007299051620066166], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.727802574634552, -0.004937894642353058], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7323552966117859, -0.004552681930363178], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7365039587020874, -0.004148663487285376], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7391624450683594, -0.002658500336110592], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7393393516540527, -0.00017694027337711304], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7370163798332214, 0.0023229853250086308], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7322428822517395, 0.004773502238094807], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.726075291633606, 0.0061676232144236565], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7195442318916321, 0.006531034130603075], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7116959095001221, 0.00784832239151001], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7035622596740723, 0.008133679628372192], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6951493620872498, 0.008412868715822697], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.685579240322113, 0.00957013014703989], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6759072542190552, 0.009671996347606182], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6661924719810486, 0.009714777581393719], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6544710993766785, 0.01172136515378952], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6428095102310181, 0.011661573313176632], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6303207278251648, 0.012488802894949913], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6171078085899353, 0.01321292296051979], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6042796969413757, 0.012828096747398376], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5898687839508057, 0.014410939998924732], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5740284323692322, 0.0158403180539608], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5588198900222778, 0.015208548866212368], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5443716645240784, 0.014448233880102634], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5297821164131165, 0.014589549042284489], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.516205370426178, 0.013576729223132133], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5017431974411011, 0.014462168328464031], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4885178804397583, 0.013225332833826542], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4756024479866028, 0.012915444560348988], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4631201922893524, 0.012482237070798874], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4511267840862274, 0.011993417516350746], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4397275149822235, 0.011399279348552227], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4290108382701874, 0.010716652497649193], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4200049042701721, 0.009005934000015259], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4128021001815796, 0.007202820852398872], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40741825103759766, 0.005383845418691635], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40391895174980164, 0.0034993106964975595], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4013022184371948, 0.0026167226023972034], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39861592650413513, 0.002686290303245187], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3979094922542572, 0.0007064350065775216], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39715445041656494, 0.0007550478912889957], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3983420729637146, -0.0011876259231939912], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40145790576934814, -0.0031158505007624626], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4045571982860565, -0.0030992808751761913], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4085308015346527, -0.003973587416112423], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4143548905849457, -0.0058241174556314945], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4220552146434784, -0.007700301706790924], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4315321147441864, -0.009476897306740284], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4407605528831482, -0.009228438138961792], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45069050788879395, -0.009929952211678028], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4602159261703491, -0.009525422006845474], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4692898392677307, -0.009073925204575062], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47880491614341736, -0.009515080600976944], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48771384358406067, -0.008908935822546482], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4969015419483185, -0.009187698364257812], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5073259472846985, -0.010424376465380192], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5179196000099182, -0.010593641549348831], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5296153426170349, -0.011695751920342445], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5402873754501343, -0.010672078467905521], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5498591065406799, -0.009571706876158714], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5602630376815796, -0.010403917171061039], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5694349408149719, -0.009171937592327595], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5782870650291443, -0.008852123282849789], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5877386331558228, -0.009451562538743019], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5957865715026855, -0.008047904819250107], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6042987108230591, -0.008512159809470177], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6122534871101379, -0.007954775355756283], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.619579553604126, -0.007326056715101004], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6252683997154236, -0.005688855424523354], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6302394866943359, -0.004971105605363846], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6354578137397766, -0.00521830515936017], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6399027705192566, -0.004444954916834831], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6444994211196899, -0.004596649669110775], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6492487192153931, -0.0047493199817836285], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6530811786651611, -0.0038324110209941864], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6549932956695557, -0.0019121168879792094], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.655981719493866, -0.000988428364507854], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6570541262626648, -0.0010724200401455164], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6582170724868774, -0.0011629696236923337], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.658441960811615, -0.00022485421504825354], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.656751811504364, 0.0016900997143238783], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6530940532684326, 0.0036577729042619467], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6495200395584106, 0.003574010916054249], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6460414528846741, 0.003478623228147626], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.640717089176178, 0.005324335768818855], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6345970034599304, 0.006120077334344387], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6287298202514648, 0.005867178551852703], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6231008768081665, 0.005628987681120634], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6158061027526855, 0.00729477871209383], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6078464984893799, 0.007959546521306038], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5983390212059021, 0.009507495909929276], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5883203744888306, 0.010018663480877876], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5778712034225464, 0.010449165478348732], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5670374035835266, 0.01083378680050373], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5569324493408203, 0.010104939341545105], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5455999970436096, 0.011332503519952297], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5331589579582214, 0.012441040948033333], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5206758379936218, 0.012483114376664162], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.509226381778717, 0.011449458077549934], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4989328980445862, 0.01029344741255045], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48784950375556946, 0.011083412915468216], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47807347774505615, 0.009776011109352112], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4697123169898987, 0.008361155167222023], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46177026629447937, 0.007942062802612782], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45530644059181213, 0.006463828030973673], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4494014382362366, 0.0059050158597528934], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4430883228778839, 0.0063130916096270084], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43841326236724854, 0.004675072617828846], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43340596556663513, 0.0050073047168552876], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4281042814254761, 0.0053016808815300465], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4245927929878235, 0.003511488903313875], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42288094758987427, 0.0017118266550824046], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4209957718849182, 0.0018851953791454434], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42095762491226196, 3.814747105934657e-05], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42174312472343445, -0.0007855045259930193], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4232994616031647, -0.0015563315246254206], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4266331195831299, -0.003333666594699025], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.429718554019928, -0.0030854311771690845], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43456727266311646, -0.004848729819059372], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44109317660331726, -0.0065258825197815895], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44723525643348694, -0.006142101250588894], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45294877886772156, -0.005713511258363724], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45921438932418823, -0.006265608128160238], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46503114700317383, -0.0058167497627437115], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.47130078077316284, -0.006269639823585749], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4769633114337921, -0.005662544164806604], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4839801490306854, -0.00701683945953846], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49131080508232117, -0.007330651395022869], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49895626306533813, -0.007645446341484785], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5068724751472473, -0.0079162223264575], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5159990191459656, -0.009126531891524792], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5262182950973511, -0.01021930668503046], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5374666452407837, -0.011248317547142506], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5496682524681091, -0.012201611883938313], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5627485513687134, -0.013080325908958912], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5755841732025146, -0.012835610657930374], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5890464782714844, -0.013462289236485958], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6020817160606384, -0.01303526945412159], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6155977249145508, -0.013515980914235115], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6275017857551575, -0.011904091574251652], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6386556625366211, -0.011153820902109146], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.648007869720459, -0.009352258406579494], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6565214395523071, -0.008513531647622585], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6630657911300659, -0.006544378120452166], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6695978045463562, -0.006532006431370974], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6760871410369873, -0.006489353720098734], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6824992895126343, -0.006412100978195667], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6877821683883667, -0.005282883998006582], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6909377574920654, -0.003155582118779421], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6939419507980347, -0.0030041993595659733], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6947757005691528, -0.0008337512263096869], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6933992505073547, 0.001376432366669178], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6918315291404724, 0.0015676982002332807], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.690112292766571, 0.0017192679224535823], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6862346529960632, 0.0038776532746851444], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6822211742401123, 0.004013477358967066], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6780965924263, 0.004124533850699663], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6739128232002258, 0.004183819983154535], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6686930060386658, 0.005219768267124891], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6614609956741333, 0.007232058327645063], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6522759795188904, 0.009184998460114002], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6421912312507629, 0.010084722191095352], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6302420496940613, 0.011949188075959682], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.61858731508255, 0.011654771864414215], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6052591800689697, 0.013328097760677338], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5923606157302856, 0.012898575514554977], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5789579153060913, 0.013402681797742844], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5661482214927673, 0.01280973944813013], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5520991086959839, 0.014049078337848186], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5368569493293762, 0.015242177061736584], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.52054762840271, 0.01630931720137596], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.504317581653595, 0.016230031847953796], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4882793724536896, 0.016038214787840843], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.471537321805954, 0.016742050647735596], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4562336802482605, 0.015303656458854675], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44148075580596924, 0.014752926304936409], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4263475239276886, 0.015133209526538849], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41094446182250977, 0.015403076075017452], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39643123745918274, 0.014513232745230198], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.38289591670036316, 0.013535317033529282], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37044757604599, 0.012448337860405445], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3601830005645752, 0.010264559648931026], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.35211896896362305, 0.008064037188887596], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.34434542059898376, 0.0077735427767038345], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33791816234588623, 0.006427264306694269], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3338618576526642, 0.004056314937770367], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33122673630714417, 0.002635112963616848], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3290420174598694, 0.0021847113966941833], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.32924360036849976, -0.00020156607206445187], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3318317234516144, -0.002588131697848439], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33687058091163635, -0.005038846284151077], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.34225624799728394, -0.005385673139244318], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.34900376200675964, -0.0067475116811692715], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3580719232559204, -0.009068163111805916], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3674141764640808, -0.009342269971966743], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.37789493799209595, -0.010480733588337898], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.38849321007728577, -0.010598300956189632], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.40009546279907227, -0.011602247133851051], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.411624550819397, -0.011529076844453812], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4239940941333771, -0.012369546107947826], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43818914890289307, -0.014195054769515991], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4531082510948181, -0.014919092878699303], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4675756096839905, -0.01446738000959158], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4815209209918976, -0.013945311307907104], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4958301782608032, -0.014309239573776722], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5113540291786194, -0.015523867681622505], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5279697775840759, -0.016615718603134155], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5445771813392639, -0.016607416793704033], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5610790252685547, -0.016501856967806816], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5763028860092163, -0.015223845839500427], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5921963453292847, -0.015893496572971344], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6076129078865051, -0.015416532754898071], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6234706044197083, -0.015857676044106483], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6386020183563232, -0.015131460502743721], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.652913510799408, -0.01431148312985897], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6673461198806763, -0.014432597905397415], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.680753231048584, -0.013407078571617603], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6930984854698181, -0.012345261871814728], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7032938003540039, -0.01019531860947609], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7122671008110046, -0.00897333212196827], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7199462056159973, -0.007679065223783255], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7262609601020813, -0.006314805708825588], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7301823496818542, -0.003921337891370058], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7336889505386353, -0.003506607376039028], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7347758412361145, -0.0010869352845475078], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7334256172180176, 0.0013502775691449642], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7316352128982544, 0.0017903681145980954], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7274322509765625, 0.00420294888317585], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7228496074676514, 0.004582647699862719], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7159104943275452, 0.006939156912267208], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7086737751960754, 0.0072367205284535885], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7001261711120605, 0.008547564037144184], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6893459558486938, 0.010780231095850468], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6764005422592163, 0.012945408001542091], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6613928079605103, 0.015007734298706055], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6444595456123352, 0.016933247447013855], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6266781687736511, 0.01778138056397438], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6071386337280273, 0.01953953504562378], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5870216488838196, 0.02011696994304657], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5664898753166199, 0.020531808957457542], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5456752181053162, 0.020814642310142517], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5247661471366882, 0.020909080281853676], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5048891305923462, 0.01987701654434204], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48619693517684937, 0.018692191690206528], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.46681517362594604, 0.019381750375032425], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44886642694473267, 0.01794877089560032], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.43247419595718384, 0.016392206773161888], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.41577446460723877, 0.01669975183904171], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39888182282447815, 0.016892625018954277], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3819098472595215, 0.016971994191408157], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3669947683811188, 0.014915072359144688], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3542463183403015, 0.012748434208333492], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3417631685733795, 0.012483151629567146], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3306049704551697, 0.011158197186887264], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3198983669281006, 0.010706625878810883], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3106755018234253, 0.009222856722772121], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.30303290486335754, 0.007642601616680622], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.29601630568504333, 0.007016605231910944], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.28967025876045227, 0.0063460408709943295], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.28403350710868835, 0.005636733025312424], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.28109192848205566, 0.0029415965545922518], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.27988773584365845, 0.0012041914742439985], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.2813848555088043, -0.0014971329364925623], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.28554767370224, -0.004162797704339027], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.2903936803340912, -0.004846025258302689], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.2978971302509308, -0.0075034331530332565], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3060421347618103, -0.008145001716911793], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.315782755613327, -0.009740634821355343], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.32605504989624023, -0.010272285901010036], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.33875277638435364, -0.012697742320597172], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.35378047823905945, -0.015027700923383236], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3690316081047058, -0.015251122415065765], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3844608962535858, -0.015429284423589706], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4019831717014313, -0.017522292211651802], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4204345643520355, -0.018451375886797905], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4397260546684265, -0.019291499629616737], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45867860317230225, -0.018952535465359688], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4781266152858734, -0.019448010250926018], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.49891865253448486, -0.020792046561837196], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5189646482467651, -0.02004600502550602], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5401268601417542, -0.021162208169698715], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5622108578681946, -0.02208399958908558], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5840743780136108, -0.02186349220573902], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6045025587081909, -0.02042817696928978], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6233447790145874, -0.018842224031686783], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6414988040924072, -0.018154041841626167], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6597924828529358, -0.018293695524334908], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6781675815582275, -0.018375081941485405], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6964390277862549, -0.018271425738930702], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7145442366600037, -0.018105221912264824], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7303147912025452, -0.015770575031638145], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7436939477920532, -0.013379121199250221], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7556087374687195, -0.011914811097085476], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7659285068511963, -0.010319755412638187], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.773654580116272, -0.007726110052317381], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.778687596321106, -0.005032968241721392], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7830034494400024, -0.00431587640196085], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7865839004516602, -0.003580466378480196], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7884508967399597, -0.0018670061836019158], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7876027226448059, 0.0008481730474159122], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7850154042243958, 0.002587367547675967], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7797186970710754, 0.0052967071533203125], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7737062573432922, 0.006012422498315573], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7659972906112671, 0.007708963006734848], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7566447257995605, 0.009352549910545349], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7447019219398499, 0.011942818760871887], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7322293519973755, 0.012472577393054962], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7182972431182861, 0.013932103291153908], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7040194272994995, 0.014277790673077106], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6885156035423279, 0.01550386007875204], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6728547215461731, 0.015660854056477547], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6551430225372314, 0.017711684107780457], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6365476250648499, 0.018595430999994278], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6171948909759521, 0.0193527489900589], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5981734991073608, 0.01902136020362377], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5796585083007812, 0.018514998257160187], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.56080162525177, 0.018856866285204887], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5407388210296631, 0.02006283961236477], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5215705633163452, 0.019168226048350334], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.503484845161438, 0.018085716292262077], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48656973242759705, 0.01691511832177639], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.469944030046463, 0.01662570610642433], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4537270665168762, 0.016216974705457687], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4381045401096344, 0.015622529201209545], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4231818914413452, 0.014922629110515118], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4080805480480194, 0.015101362019777298], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.39287450909614563, 0.015206040814518929], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3786363899707794, 0.014238100498914719], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.36448612809181213, 0.014150275848805904], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.35248497128486633, 0.012001140974462032], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.342759370803833, 0.009725620970129967], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3354145884513855, 0.007344760466367006], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3294576406478882, 0.005956971552222967], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3258935809135437, 0.0035640408750623465], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3227674067020416, 0.0031261842232197523], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.32106882333755493, 0.0016985831316560507], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3198477327823639, 0.0012210874119773507], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3210728168487549, -0.0012250767322257161], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3247338831424713, -0.003661075606942177], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3308294415473938, -0.006095568183809519], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3383316397666931, -0.007502185646444559], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.34616410732269287, -0.007832463830709457], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3553199768066406, -0.009155881591141224], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3656858801841736, -0.010365903377532959], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3782557249069214, -0.012569841928780079], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3919236660003662, -0.013667936436831951], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4066176116466522, -0.014693958684802055], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4232385456562042, -0.01662091724574566], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.44062361121177673, -0.01738506741821766], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.459701806306839, -0.01907818578183651], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.48028603196144104, -0.020584244281053543], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5022544860839844, -0.021968428045511246], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5254397988319397, -0.023185359314084053], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.548629105091095, -0.023189257830381393], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5726872086524963, -0.024058133363723755], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5974510312080383, -0.024763835594058037], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6216883659362793, -0.024237319827079773], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.645261824131012, -0.023573484271764755], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6669955253601074, -0.021733684465289116], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6877352595329285, -0.020739730447530746], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.707366406917572, -0.019631143659353256], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7247406840324402, -0.017374251037836075], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7407162189483643, -0.015975531190633774], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7551852464675903, -0.014469083398580551], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7681260108947754, -0.012940741144120693], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.779404878616333, -0.01127886027097702], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7879487872123718, -0.008543896488845348], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7947526574134827, -0.006803907919675112], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7997588515281677, -0.00500617315992713], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.8019497394561768, -0.0021909084171056747], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.8033525347709656, -0.0014027844881638885], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.8019341230392456, 0.0014184153405949473], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7996881008148193, 0.0022460310719907284], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7946069240570068, 0.005081195384263992], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7867714166641235, 0.007835503667593002], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7771750688552856, 0.009596336632966995], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7668596506118774, 0.010315393097698689], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7549125552177429, 0.01194709911942482], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7424086332321167, 0.012503966689109802], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7274405360221863, 0.014968070201575756], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.7100947499275208, 0.017345808446407318], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6924359202384949, 0.017658784985542297], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.673596203327179, 0.018839722499251366], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6537235379219055, 0.019872702658176422], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6329005360603333, 0.02082298882305622], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.6113160252571106, 0.02158450335264206], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5881142616271973, 0.023201745003461838], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5634810328483582, 0.024633264169096947], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5375595092773438, 0.02592148631811142], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.5105500221252441, 0.027009490877389908], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4826750159263611, 0.0278750229626894], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.45514458417892456, 0.027530424296855927], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.42815732955932617, 0.02698727324604988], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.4019611179828644, 0.026196204125881195], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3766772150993347, 0.02528388984501362], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.3515452444553375, 0.02513197623193264], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.32667064666748047, 0.024874597787857056], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.30227068066596985, 0.024399951100349426], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.28050151467323303, 0.021769186481833458], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.26040002703666687, 0.020101478323340416], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.24210317432880402, 0.0182968620210886], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.2247602641582489, 0.01734289713203907], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.20946234464645386, 0.015297923237085342], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.19626384973526, 0.013198498636484146], action=1, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.1842312514781952, 0.012032595463097095], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.17533086240291595, 0.008900394663214684], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.16965334117412567, 0.005677512381225824], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.16717015206813812, 0.0024831995833665133], action=0, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.16591306030750275, 0.0012570895487442613], action=2, reward=-1.0, info={}, terminal=False, timeout=False), RLTuple(state=[-0.16685937345027924, -0.000946320709772408], action=1, reward=-1.0, info={}, terminal=False, timeout=True)])]
Training an Offline RL Agent
Finally, we use d3rlpy to train an offline RL agent on this environment. Note that we do not train it to convergence, we only show how to get from a PyTupli dataset to actually doing offline RL! Our TupliDataset has a method for converting all episodes into numpy arrays for states, actions, rewards, terminals, and timeouts. This can be customized if other output formats are required. Using these arrays, we can then create an MDPDataset, which is the required input format for all d3rlpy
algorithms.
[14]:
obs, act, rew, terminal, truncated = mdp_dataset.convert_to_tensors(parser=NumpyTupleParser)
# create d3rlpy dataset
d3rlpy_dataset = MDPDataset(
observations=obs, actions=act, rewards=rew, terminals=terminal, timeouts=truncated
)
2025-07-28 15:45.25 [info ] Signatures have been automatically determined. action_signature=Signature(dtype=[dtype('float32')], shape=[(1,)]) observation_signature=Signature(dtype=[dtype('float32')], shape=[(2,)]) reward_signature=Signature(dtype=[dtype('float32')], shape=[(1,)])
2025-07-28 15:45.25 [info ] Action-space has been automatically determined. action_space=<ActionSpace.DISCRETE: 2>
2025-07-28 15:45.25 [info ] Action size has been automatically determined. action_size=3
Finally, let us show that training an agent using conservative Q-learning (CQL) works with this data:
[15]:
# algorithm for offline training: CQL from d3rlpy
d3rlpy.seed(1) # for reproducibility
algo = DiscreteCQLConfig(batch_size=64, alpha=2.0, target_update_interval=1000).create(device='cpu')
# train
algo.fit(dataset=d3rlpy_dataset, n_steps=10000, n_steps_per_epoch=100)
2025-07-28 15:45.25 [info ] dataset info dataset_info=DatasetInfo(observation_signature=Signature(dtype=[dtype('float32')], shape=[(2,)]), action_signature=Signature(dtype=[dtype('float32')], shape=[(1,)]), reward_signature=Signature(dtype=[dtype('float32')], shape=[(1,)]), action_space=<ActionSpace.DISCRETE: 2>, action_size=3)
2025-07-28 15:45.25 [debug ] Building models...
2025-07-28 15:45.25 [debug ] Models have been built.
2025-07-28 15:45.25 [info ] Directory is created at d3rlpy_logs/DiscreteCQL_20250728154525
2025-07-28 15:45.25 [info ] Parameters params={'observation_shape': [2], 'action_size': 3, 'config': {'type': 'discrete_cql', 'params': {'batch_size': 64, 'gamma': 0.99, 'observation_scaler': {'type': 'none', 'params': {}}, 'action_scaler': {'type': 'none', 'params': {}}, 'reward_scaler': {'type': 'none', 'params': {}}, 'compile_graph': False, 'learning_rate': 6.25e-05, 'optim_factory': {'type': 'adam', 'params': {'clip_grad_norm': None, 'lr_scheduler_factory': {'type': 'none', 'params': {}}, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False}}, 'encoder_factory': {'type': 'default', 'params': {'activation': 'relu', 'use_batch_norm': False, 'dropout_rate': None}}, 'q_func_factory': {'type': 'mean', 'params': {'share_encoder': False}}, 'n_critics': 1, 'target_update_interval': 1000, 'alpha': 2.0}}}
Epoch 1/100: 100%|██████████| 100/100 [00:00<00:00, 377.93it/s, loss=2.41, td_loss=0.213, conservative_loss=1.1]
2025-07-28 15:45.25 [info ] DiscreteCQL_20250728154525: epoch=1 step=100 epoch=1 metrics={'time_sample_batch': 0.0005567145347595215, 'time_algorithm_update': 0.0020180344581604004, 'loss': 2.3966327929496765, 'td_loss': 0.19853577110916376, 'conservative_loss': 1.09904851436615, 'time_step': 0.00263277530670166} step=100
2025-07-28 15:45.25 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_100.d3
Epoch 2/100: 100%|██████████| 100/100 [00:00<00:00, 439.61it/s, loss=2.21, td_loss=0.016, conservative_loss=1.1]
2025-07-28 15:45.25 [info ] DiscreteCQL_20250728154525: epoch=2 step=200 epoch=2 metrics={'time_sample_batch': 0.0004550933837890625, 'time_algorithm_update': 0.00175459623336792, 'loss': 2.212840859889984, 'td_loss': 0.01538949720794335, 'conservative_loss': 1.0987256860733032, 'time_step': 0.0022612261772155763} step=200
2025-07-28 15:45.25 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_200.d3
Epoch 3/100: 100%|██████████| 100/100 [00:00<00:00, 438.60it/s, loss=2.2, td_loss=0.00351, conservative_loss=1.1]
2025-07-28 15:45.25 [info ] DiscreteCQL_20250728154525: epoch=3 step=300 epoch=3 metrics={'time_sample_batch': 0.00045412778854370117, 'time_algorithm_update': 0.0017646265029907227, 'loss': 2.200702414512634, 'td_loss': 0.003590560882585123, 'conservative_loss': 1.0985559260845184, 'time_step': 0.002267448902130127} step=300
2025-07-28 15:45.25 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_300.d3
Epoch 4/100: 100%|██████████| 100/100 [00:00<00:00, 392.28it/s, loss=2.2, td_loss=0.00166, conservative_loss=1.1]
2025-07-28 15:45.26 [info ] DiscreteCQL_20250728154525: epoch=4 step=400 epoch=4 metrics={'time_sample_batch': 0.0005457544326782226, 'time_algorithm_update': 0.0019359111785888672, 'loss': 2.1985737800598146, 'td_loss': 0.001622819603071548, 'conservative_loss': 1.0984754824638367, 'time_step': 0.0025356578826904296} step=400
2025-07-28 15:45.26 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_400.d3
Epoch 5/100: 100%|██████████| 100/100 [00:00<00:00, 407.84it/s, loss=2.2, td_loss=0.00415, conservative_loss=1.1]
2025-07-28 15:45.26 [info ] DiscreteCQL_20250728154525: epoch=5 step=500 epoch=5 metrics={'time_sample_batch': 0.0004919314384460449, 'time_algorithm_update': 0.001887047290802002, 'loss': 2.19991708278656, 'td_loss': 0.004285427670693025, 'conservative_loss': 1.09781583070755, 'time_step': 0.0024393200874328613} step=500
2025-07-28 15:45.26 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_500.d3
Epoch 6/100: 100%|██████████| 100/100 [00:00<00:00, 432.74it/s, loss=2.2, td_loss=0.00218, conservative_loss=1.1]
2025-07-28 15:45.26 [info ] DiscreteCQL_20250728154525: epoch=6 step=600 epoch=6 metrics={'time_sample_batch': 0.0004746437072753906, 'time_algorithm_update': 0.0017737793922424317, 'loss': 2.198714690208435, 'td_loss': 0.0020858740305993704, 'conservative_loss': 1.0983144092559813, 'time_step': 0.002298996448516846} step=600
2025-07-28 15:45.26 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_600.d3
Epoch 7/100: 100%|██████████| 100/100 [00:00<00:00, 428.39it/s, loss=2.2, td_loss=0.00353, conservative_loss=1.1]
2025-07-28 15:45.26 [info ] DiscreteCQL_20250728154525: epoch=7 step=700 epoch=7 metrics={'time_sample_batch': 0.00046657800674438477, 'time_algorithm_update': 0.001801440715789795, 'loss': 2.2006174540519714, 'td_loss': 0.003812542131054215, 'conservative_loss': 1.098402464389801, 'time_step': 0.0023208904266357423} step=700
2025-07-28 15:45.26 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_700.d3
Epoch 8/100: 100%|██████████| 100/100 [00:00<00:00, 397.56it/s, loss=2.2, td_loss=0.00462, conservative_loss=1.1]
2025-07-28 15:45.27 [info ] DiscreteCQL_20250728154525: epoch=8 step=800 epoch=8 metrics={'time_sample_batch': 0.0004999542236328125, 'time_algorithm_update': 0.0019462394714355468, 'loss': 2.201106686592102, 'td_loss': 0.004484999722335487, 'conservative_loss': 1.098310842514038, 'time_step': 0.0025028419494628907} step=800
2025-07-28 15:45.27 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_800.d3
Epoch 9/100: 100%|██████████| 100/100 [00:00<00:00, 399.19it/s, loss=2.2, td_loss=0.00228, conservative_loss=1.1]
2025-07-28 15:45.27 [info ] DiscreteCQL_20250728154525: epoch=9 step=900 epoch=9 metrics={'time_sample_batch': 0.0004914283752441406, 'time_algorithm_update': 0.0019498515129089357, 'loss': 2.1986337685585022, 'td_loss': 0.00250896273937542, 'conservative_loss': 1.0980623996257781, 'time_step': 0.002493433952331543} step=900
2025-07-28 15:45.27 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_900.d3
Epoch 10/100: 100%|██████████| 100/100 [00:00<00:00, 374.70it/s, loss=2.2, td_loss=0.0028, conservative_loss=1.1]
2025-07-28 15:45.27 [info ] DiscreteCQL_20250728154525: epoch=10 step=1000 epoch=10 metrics={'time_sample_batch': 0.0005762290954589844, 'time_algorithm_update': 0.0020333194732666017, 'loss': 2.1994874477386475, 'td_loss': 0.002849989581736736, 'conservative_loss': 1.0983187305927276, 'time_step': 0.0026563167572021484} step=1000
2025-07-28 15:45.27 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1000.d3
Epoch 11/100: 100%|██████████| 100/100 [00:00<00:00, 360.13it/s, loss=2.3, td_loss=0.102, conservative_loss=1.1]
2025-07-28 15:45.27 [info ] DiscreteCQL_20250728154525: epoch=11 step=1100 epoch=11 metrics={'time_sample_batch': 0.0006136512756347657, 'time_algorithm_update': 0.0020970726013183594, 'loss': 2.28967072725296, 'td_loss': 0.0928150053115678, 'conservative_loss': 1.0984278655052184, 'time_step': 0.0027637600898742674} step=1100
2025-07-28 15:45.27 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1100.d3
Epoch 12/100: 100%|██████████| 100/100 [00:00<00:00, 354.22it/s, loss=2.2, td_loss=0.000653, conservative_loss=1.1]
2025-07-28 15:45.28 [info ] DiscreteCQL_20250728154525: epoch=12 step=1200 epoch=12 metrics={'time_sample_batch': 0.0006206274032592773, 'time_algorithm_update': 0.0021375727653503416, 'loss': 2.1974543070793153, 'td_loss': 0.0006476261564239394, 'conservative_loss': 1.0984033381938934, 'time_step': 0.0028111600875854494} step=1200
2025-07-28 15:45.28 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1200.d3
Epoch 13/100: 100%|██████████| 100/100 [00:00<00:00, 359.39it/s, loss=2.2, td_loss=0.000719, conservative_loss=1.1]
2025-07-28 15:45.28 [info ] DiscreteCQL_20250728154525: epoch=13 step=1300 epoch=13 metrics={'time_sample_batch': 0.0006099987030029297, 'time_algorithm_update': 0.002111635208129883, 'loss': 2.197365963459015, 'td_loss': 0.0008116933463315945, 'conservative_loss': 1.0982771360874175, 'time_step': 0.0027706408500671386} step=1300
2025-07-28 15:45.28 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1300.d3
Epoch 14/100: 100%|██████████| 100/100 [00:00<00:00, 363.63it/s, loss=2.2, td_loss=0.00086, conservative_loss=1.1]
2025-07-28 15:45.28 [info ] DiscreteCQL_20250728154525: epoch=14 step=1400 epoch=14 metrics={'time_sample_batch': 0.0006060028076171876, 'time_algorithm_update': 0.0020788097381591797, 'loss': 2.1977316999435423, 'td_loss': 0.000837066586536821, 'conservative_loss': 1.0984473168849944, 'time_step': 0.002737417221069336} step=1400
2025-07-28 15:45.28 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1400.d3
Epoch 15/100: 100%|██████████| 100/100 [00:00<00:00, 352.61it/s, loss=2.2, td_loss=0.000487, conservative_loss=1.1]
2025-07-28 15:45.28 [info ] DiscreteCQL_20250728154525: epoch=15 step=1500 epoch=15 metrics={'time_sample_batch': 0.0006205201148986817, 'time_algorithm_update': 0.0021503591537475586, 'loss': 2.197506539821625, 'td_loss': 0.000559124128849362, 'conservative_loss': 1.0984737122058867, 'time_step': 0.002822751998901367} step=1500
2025-07-28 15:45.28 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1500.d3
Epoch 16/100: 100%|██████████| 100/100 [00:00<00:00, 343.59it/s, loss=2.2, td_loss=0.000888, conservative_loss=1.1]
2025-07-28 15:45.29 [info ] DiscreteCQL_20250728154525: epoch=16 step=1600 epoch=16 metrics={'time_sample_batch': 0.0006248164176940918, 'time_algorithm_update': 0.002216043472290039, 'loss': 2.197997591495514, 'td_loss': 0.00094447792347637, 'conservative_loss': 1.0985265529155732, 'time_step': 0.0028961730003356934} step=1600
2025-07-28 15:45.29 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1600.d3
Epoch 17/100: 100%|██████████| 100/100 [00:00<00:00, 344.67it/s, loss=2.2, td_loss=0.000705, conservative_loss=1.1]
2025-07-28 15:45.29 [info ] DiscreteCQL_20250728154525: epoch=17 step=1700 epoch=17 metrics={'time_sample_batch': 0.0006282734870910645, 'time_algorithm_update': 0.0021921420097351075, 'loss': 2.1970662832260133, 'td_loss': 0.0007343449354812037, 'conservative_loss': 1.0981659662723542, 'time_step': 0.002886338233947754} step=1700
2025-07-28 15:45.29 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1700.d3
Epoch 18/100: 100%|██████████| 100/100 [00:00<00:00, 367.11it/s, loss=2.2, td_loss=0.000985, conservative_loss=1.1]
2025-07-28 15:45.29 [info ] DiscreteCQL_20250728154525: epoch=18 step=1800 epoch=18 metrics={'time_sample_batch': 0.0005885624885559082, 'time_algorithm_update': 0.0020654106140136717, 'loss': 2.194919857978821, 'td_loss': 0.0009621448916732334, 'conservative_loss': 1.0969788503646851, 'time_step': 0.0027103424072265625} step=1800
2025-07-28 15:45.29 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1800.d3
Epoch 19/100: 100%|██████████| 100/100 [00:00<00:00, 391.83it/s, loss=2.2, td_loss=0.000917, conservative_loss=1.1]
2025-07-28 15:45.30 [info ] DiscreteCQL_20250728154525: epoch=19 step=1900 epoch=19 metrics={'time_sample_batch': 0.0005143189430236816, 'time_algorithm_update': 0.001968593597412109, 'loss': 2.1962599110603334, 'td_loss': 0.0009440037869353546, 'conservative_loss': 1.0976579535007476, 'time_step': 0.002539358139038086} step=1900
2025-07-28 15:45.30 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_1900.d3
Epoch 20/100: 100%|██████████| 100/100 [00:00<00:00, 413.44it/s, loss=2.2, td_loss=0.000863, conservative_loss=1.1]
2025-07-28 15:45.30 [info ] DiscreteCQL_20250728154525: epoch=20 step=2000 epoch=20 metrics={'time_sample_batch': 0.0004642653465270996, 'time_algorithm_update': 0.0018935203552246094, 'loss': 2.1974512267112734, 'td_loss': 0.0008664641072391533, 'conservative_loss': 1.0982923746109008, 'time_step': 0.002407989501953125} step=2000
2025-07-28 15:45.30 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2000.d3
Epoch 21/100: 100%|██████████| 100/100 [00:00<00:00, 388.95it/s, loss=2.27, td_loss=0.0712, conservative_loss=1.1]
2025-07-28 15:45.30 [info ] DiscreteCQL_20250728154525: epoch=21 step=2100 epoch=21 metrics={'time_sample_batch': 0.0005271315574645996, 'time_algorithm_update': 0.0019734764099121095, 'loss': 2.261094753742218, 'td_loss': 0.0650152921580593, 'conservative_loss': 1.0980397248268128, 'time_step': 0.002556498050689697} step=2100
2025-07-28 15:45.30 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2100.d3
Epoch 22/100: 100%|██████████| 100/100 [00:00<00:00, 385.53it/s, loss=2.2, td_loss=0.00115, conservative_loss=1.1]
2025-07-28 15:45.30 [info ] DiscreteCQL_20250728154525: epoch=22 step=2200 epoch=22 metrics={'time_sample_batch': 0.000541989803314209, 'time_algorithm_update': 0.001987617015838623, 'loss': 2.197791721820831, 'td_loss': 0.0011231240205233917, 'conservative_loss': 1.098334299325943, 'time_step': 0.002582676410675049} step=2200
2025-07-28 15:45.30 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2200.d3
Epoch 23/100: 100%|██████████| 100/100 [00:00<00:00, 413.56it/s, loss=2.19, td_loss=0.00109, conservative_loss=1.1]
2025-07-28 15:45.31 [info ] DiscreteCQL_20250728154525: epoch=23 step=2300 epoch=23 metrics={'time_sample_batch': 0.0004741644859313965, 'time_algorithm_update': 0.001876215934753418, 'loss': 2.194902892112732, 'td_loss': 0.0010723149092518724, 'conservative_loss': 1.0969152915477753, 'time_step': 0.0024067497253417967} step=2300
2025-07-28 15:45.31 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2300.d3
Epoch 24/100: 100%|██████████| 100/100 [00:00<00:00, 424.30it/s, loss=2.2, td_loss=0.000945, conservative_loss=1.1]
2025-07-28 15:45.31 [info ] DiscreteCQL_20250728154525: epoch=24 step=2400 epoch=24 metrics={'time_sample_batch': 0.0004554891586303711, 'time_algorithm_update': 0.0018364500999450684, 'loss': 2.1958152437210083, 'td_loss': 0.0009611728510935791, 'conservative_loss': 1.0974270331859588, 'time_step': 0.0023450636863708496} step=2400
2025-07-28 15:45.31 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2400.d3
Epoch 25/100: 100%|██████████| 100/100 [00:00<00:00, 428.66it/s, loss=2.2, td_loss=0.000974, conservative_loss=1.1]
2025-07-28 15:45.31 [info ] DiscreteCQL_20250728154525: epoch=25 step=2500 epoch=25 metrics={'time_sample_batch': 0.00045788049697875977, 'time_algorithm_update': 0.0018152213096618653, 'loss': 2.1973110389709474, 'td_loss': 0.0009258302650414407, 'conservative_loss': 1.0981926012039185, 'time_step': 0.002320990562438965} step=2500
2025-07-28 15:45.31 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2500.d3
Epoch 26/100: 100%|██████████| 100/100 [00:00<00:00, 417.96it/s, loss=2.2, td_loss=0.000771, conservative_loss=1.1]
2025-07-28 15:45.31 [info ] DiscreteCQL_20250728154525: epoch=26 step=2600 epoch=26 metrics={'time_sample_batch': 0.0004761838912963867, 'time_algorithm_update': 0.0018532466888427734, 'loss': 2.196728525161743, 'td_loss': 0.0007932424181490205, 'conservative_loss': 1.0979676389694213, 'time_step': 0.0023805713653564453} step=2600
2025-07-28 15:45.31 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2600.d3
Epoch 27/100: 100%|██████████| 100/100 [00:00<00:00, 384.42it/s, loss=2.2, td_loss=0.00115, conservative_loss=1.1]
2025-07-28 15:45.32 [info ] DiscreteCQL_20250728154525: epoch=27 step=2700 epoch=27 metrics={'time_sample_batch': 0.0005347943305969239, 'time_algorithm_update': 0.001997036933898926, 'loss': 2.197197813987732, 'td_loss': 0.0011636634092428721, 'conservative_loss': 1.0980170750617981, 'time_step': 0.0025886750221252443} step=2700
2025-07-28 15:45.32 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2700.d3
Epoch 28/100: 100%|██████████| 100/100 [00:00<00:00, 362.13it/s, loss=2.2, td_loss=0.000858, conservative_loss=1.1]
2025-07-28 15:45.32 [info ] DiscreteCQL_20250728154525: epoch=28 step=2800 epoch=28 metrics={'time_sample_batch': 0.0005988383293151856, 'time_algorithm_update': 0.002096092700958252, 'loss': 2.1970012974739075, 'td_loss': 0.0008431548319640569, 'conservative_loss': 1.0980790734291077, 'time_step': 0.002748849391937256} step=2800
2025-07-28 15:45.32 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2800.d3
Epoch 29/100: 100%|██████████| 100/100 [00:00<00:00, 410.93it/s, loss=2.2, td_loss=0.00127, conservative_loss=1.1]
2025-07-28 15:45.32 [info ] DiscreteCQL_20250728154525: epoch=29 step=2900 epoch=29 metrics={'time_sample_batch': 0.0004908180236816406, 'time_algorithm_update': 0.0018802642822265624, 'loss': 2.1956692242622378, 'td_loss': 0.0013493687222944572, 'conservative_loss': 1.0971599233150482, 'time_step': 0.0024227428436279295} step=2900
2025-07-28 15:45.32 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_2900.d3
Epoch 30/100: 100%|██████████| 100/100 [00:00<00:00, 374.62it/s, loss=2.2, td_loss=0.00183, conservative_loss=1.1]
2025-07-28 15:45.32 [info ] DiscreteCQL_20250728154525: epoch=30 step=3000 epoch=30 metrics={'time_sample_batch': 0.0005670523643493652, 'time_algorithm_update': 0.0020356225967407228, 'loss': 2.196509358882904, 'td_loss': 0.001714693953981623, 'conservative_loss': 1.097397335767746, 'time_step': 0.0026561284065246583} step=3000
2025-07-28 15:45.32 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3000.d3
Epoch 31/100: 100%|██████████| 100/100 [00:00<00:00, 418.41it/s, loss=2.26, td_loss=0.0625, conservative_loss=1.1]
2025-07-28 15:45.33 [info ] DiscreteCQL_20250728154525: epoch=31 step=3100 epoch=31 metrics={'time_sample_batch': 0.00046095371246337893, 'time_algorithm_update': 0.0018667006492614747, 'loss': 2.2531930565834046, 'td_loss': 0.05699506776465569, 'conservative_loss': 1.0980989944934845, 'time_step': 0.0023780179023742674} step=3100
2025-07-28 15:45.33 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3100.d3
Epoch 32/100: 100%|██████████| 100/100 [00:00<00:00, 407.62it/s, loss=2.2, td_loss=0.00104, conservative_loss=1.1]
2025-07-28 15:45.33 [info ] DiscreteCQL_20250728154525: epoch=32 step=3200 epoch=32 metrics={'time_sample_batch': 0.0004948973655700684, 'time_algorithm_update': 0.0018903970718383788, 'loss': 2.196646854877472, 'td_loss': 0.0009938515130488669, 'conservative_loss': 1.0978265023231506, 'time_step': 0.0024421000480651855} step=3200
2025-07-28 15:45.33 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3200.d3
Epoch 33/100: 100%|██████████| 100/100 [00:00<00:00, 430.93it/s, loss=2.2, td_loss=0.000858, conservative_loss=1.1]
2025-07-28 15:45.33 [info ] DiscreteCQL_20250728154525: epoch=33 step=3300 epoch=33 metrics={'time_sample_batch': 0.0004577851295471191, 'time_algorithm_update': 0.0018019366264343261, 'loss': 2.1971650981903075, 'td_loss': 0.0008624040643917397, 'conservative_loss': 1.098151340484619, 'time_step': 0.002308976650238037} step=3300
2025-07-28 15:45.33 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3300.d3
Epoch 34/100: 100%|██████████| 100/100 [00:00<00:00, 412.23it/s, loss=2.2, td_loss=0.000943, conservative_loss=1.1]
2025-07-28 15:45.33 [info ] DiscreteCQL_20250728154525: epoch=34 step=3400 epoch=34 metrics={'time_sample_batch': 0.00048842191696167, 'time_algorithm_update': 0.0018771505355834961, 'loss': 2.19635671377182, 'td_loss': 0.0009327739142463542, 'conservative_loss': 1.0977119731903076, 'time_step': 0.0024143147468566896} step=3400
2025-07-28 15:45.33 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3400.d3
Epoch 35/100: 100%|██████████| 100/100 [00:00<00:00, 416.87it/s, loss=2.2, td_loss=0.00114, conservative_loss=1.1]
2025-07-28 15:45.34 [info ] DiscreteCQL_20250728154525: epoch=35 step=3500 epoch=35 metrics={'time_sample_batch': 0.0004731059074401855, 'time_algorithm_update': 0.001864159107208252, 'loss': 2.1976770973205566, 'td_loss': 0.0011044965320616028, 'conservative_loss': 1.0982862997055054, 'time_step': 0.0023872184753417967} step=3500
2025-07-28 15:45.34 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3500.d3
Epoch 36/100: 100%|██████████| 100/100 [00:00<00:00, 370.42it/s, loss=2.2, td_loss=0.000734, conservative_loss=1.1]
2025-07-28 15:45.34 [info ] DiscreteCQL_20250728154525: epoch=36 step=3600 epoch=36 metrics={'time_sample_batch': 0.0005660319328308106, 'time_algorithm_update': 0.0020635390281677247, 'loss': 2.1967822265625, 'td_loss': 0.0007547940395306795, 'conservative_loss': 1.0980137073993683, 'time_step': 0.002687985897064209} step=3600
2025-07-28 15:45.34 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3600.d3
Epoch 37/100: 100%|██████████| 100/100 [00:00<00:00, 403.28it/s, loss=2.19, td_loss=0.00114, conservative_loss=1.1]
2025-07-28 15:45.34 [info ] DiscreteCQL_20250728154525: epoch=37 step=3700 epoch=37 metrics={'time_sample_batch': 0.0005019855499267578, 'time_algorithm_update': 0.0019105076789855957, 'loss': 2.194888060092926, 'td_loss': 0.0011231697170296683, 'conservative_loss': 1.0968824434280395, 'time_step': 0.0024664974212646484} step=3700
2025-07-28 15:45.34 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3700.d3
Epoch 38/100: 100%|██████████| 100/100 [00:00<00:00, 390.65it/s, loss=2.2, td_loss=0.00107, conservative_loss=1.1]
2025-07-28 15:45.34 [info ] DiscreteCQL_20250728154525: epoch=38 step=3800 epoch=38 metrics={'time_sample_batch': 0.0005265402793884277, 'time_algorithm_update': 0.0019679880142211914, 'loss': 2.1961218523979187, 'td_loss': 0.001076404427876696, 'conservative_loss': 1.0975227224826813, 'time_step': 0.0025478482246398928} step=3800
2025-07-28 15:45.34 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3800.d3
Epoch 39/100: 100%|██████████| 100/100 [00:00<00:00, 410.59it/s, loss=2.2, td_loss=0.00105, conservative_loss=1.1]
2025-07-28 15:45.35 [info ] DiscreteCQL_20250728154525: epoch=39 step=3900 epoch=39 metrics={'time_sample_batch': 0.0004711127281188965, 'time_algorithm_update': 0.0019014263153076172, 'loss': 2.1965386128425597, 'td_loss': 0.0010531386702496092, 'conservative_loss': 1.0977427423000337, 'time_step': 0.0024236011505126952} step=3900
2025-07-28 15:45.35 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_3900.d3
Epoch 40/100: 100%|██████████| 100/100 [00:00<00:00, 418.40it/s, loss=2.2, td_loss=0.000764, conservative_loss=1.1]
2025-07-28 15:45.35 [info ] DiscreteCQL_20250728154525: epoch=40 step=4000 epoch=40 metrics={'time_sample_batch': 0.0004645562171936035, 'time_algorithm_update': 0.0018659543991088867, 'loss': 2.1961905717849732, 'td_loss': 0.0007454320683609694, 'conservative_loss': 1.09772256731987, 'time_step': 0.002378692626953125} step=4000
2025-07-28 15:45.35 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4000.d3
Epoch 41/100: 100%|██████████| 100/100 [00:00<00:00, 426.11it/s, loss=2.25, td_loss=0.054, conservative_loss=1.1]
2025-07-28 15:45.35 [info ] DiscreteCQL_20250728154525: epoch=41 step=4100 epoch=41 metrics={'time_sample_batch': 0.00045660495758056643, 'time_algorithm_update': 0.0018251967430114747, 'loss': 2.2444769239425657, 'td_loss': 0.04920598843396874, 'conservative_loss': 1.0976354658603669, 'time_step': 0.002335050106048584} step=4100
2025-07-28 15:45.35 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4100.d3
Epoch 42/100: 100%|██████████| 100/100 [00:00<00:00, 417.92it/s, loss=2.2, td_loss=0.00118, conservative_loss=1.1]
2025-07-28 15:45.35 [info ] DiscreteCQL_20250728154525: epoch=42 step=4200 epoch=42 metrics={'time_sample_batch': 0.0004755043983459473, 'time_algorithm_update': 0.0018530845642089843, 'loss': 2.197680506706238, 'td_loss': 0.0011245051398873329, 'conservative_loss': 1.0982780075073242, 'time_step': 0.002380106449127197} step=4200
2025-07-28 15:45.35 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4200.d3
Epoch 43/100: 100%|██████████| 100/100 [00:00<00:00, 422.72it/s, loss=2.2, td_loss=0.000876, conservative_loss=1.1]
2025-07-28 15:45.36 [info ] DiscreteCQL_20250728154525: epoch=43 step=4300 epoch=43 metrics={'time_sample_batch': 0.00046542406082153323, 'time_algorithm_update': 0.0018411803245544434, 'loss': 2.19569308757782, 'td_loss': 0.0008939958154223859, 'conservative_loss': 1.0973995530605316, 'time_step': 0.00235353946685791} step=4300
2025-07-28 15:45.36 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4300.d3
Epoch 44/100: 100%|██████████| 100/100 [00:00<00:00, 424.83it/s, loss=2.2, td_loss=0.000986, conservative_loss=1.1]
2025-07-28 15:45.36 [info ] DiscreteCQL_20250728154525: epoch=44 step=4400 epoch=44 metrics={'time_sample_batch': 0.0004541301727294922, 'time_algorithm_update': 0.0018338370323181153, 'loss': 2.197810397148132, 'td_loss': 0.0010860397884971463, 'conservative_loss': 1.0983621823787688, 'time_step': 0.0023408055305480955} step=4400
2025-07-28 15:45.36 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4400.d3
Epoch 45/100: 100%|██████████| 100/100 [00:00<00:00, 427.27it/s, loss=2.2, td_loss=0.00112, conservative_loss=1.1]
2025-07-28 15:45.36 [info ] DiscreteCQL_20250728154525: epoch=45 step=4500 epoch=45 metrics={'time_sample_batch': 0.00045701980590820315, 'time_algorithm_update': 0.001823270320892334, 'loss': 2.1963061475753785, 'td_loss': 0.0011160826054401696, 'conservative_loss': 1.0975950312614442, 'time_step': 0.002328629493713379} step=4500
2025-07-28 15:45.36 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4500.d3
Epoch 46/100: 100%|██████████| 100/100 [00:00<00:00, 423.77it/s, loss=2.2, td_loss=0.000778, conservative_loss=1.1]
2025-07-28 15:45.36 [info ] DiscreteCQL_20250728154525: epoch=46 step=4600 epoch=46 metrics={'time_sample_batch': 0.0004597926139831543, 'time_algorithm_update': 0.0018334841728210448, 'loss': 2.1983505201339724, 'td_loss': 0.0007569296075962484, 'conservative_loss': 1.0987967932224274, 'time_step': 0.002347161769866943} step=4600
2025-07-28 15:45.36 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4600.d3
Epoch 47/100: 100%|██████████| 100/100 [00:00<00:00, 422.02it/s, loss=2.2, td_loss=0.000766, conservative_loss=1.1]
2025-07-28 15:45.37 [info ] DiscreteCQL_20250728154525: epoch=47 step=4700 epoch=47 metrics={'time_sample_batch': 0.00045778989791870115, 'time_algorithm_update': 0.0018497180938720703, 'loss': 2.1964164352416993, 'td_loss': 0.0007753977939137257, 'conservative_loss': 1.097820520401001, 'time_step': 0.0023577427864074707} step=4700
2025-07-28 15:45.37 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4700.d3
Epoch 48/100: 100%|██████████| 100/100 [00:00<00:00, 393.45it/s, loss=2.2, td_loss=0.00124, conservative_loss=1.1]
2025-07-28 15:45.37 [info ] DiscreteCQL_20250728154525: epoch=48 step=4800 epoch=48 metrics={'time_sample_batch': 0.0005004525184631348, 'time_algorithm_update': 0.001968719959259033, 'loss': 2.1953913831710814, 'td_loss': 0.0011732379256864079, 'conservative_loss': 1.097109078168869, 'time_step': 0.002527334690093994} step=4800
2025-07-28 15:45.37 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4800.d3
Epoch 49/100: 100%|██████████| 100/100 [00:00<00:00, 385.98it/s, loss=2.19, td_loss=0.00118, conservative_loss=1.1]
2025-07-28 15:45.37 [info ] DiscreteCQL_20250728154525: epoch=49 step=4900 epoch=49 metrics={'time_sample_batch': 0.0005402588844299317, 'time_algorithm_update': 0.0019860291481018067, 'loss': 2.195286548137665, 'td_loss': 0.0011594361462630332, 'conservative_loss': 1.0970635497570038, 'time_step': 0.002579333782196045} step=4900
2025-07-28 15:45.37 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_4900.d3
Epoch 50/100: 100%|██████████| 100/100 [00:00<00:00, 407.55it/s, loss=2.19, td_loss=0.000825, conservative_loss=1.1]
2025-07-28 15:45.37 [info ] DiscreteCQL_20250728154525: epoch=50 step=5000 epoch=50 metrics={'time_sample_batch': 0.0005037856101989747, 'time_algorithm_update': 0.0018882203102111817, 'loss': 2.194396321773529, 'td_loss': 0.0008400411676848307, 'conservative_loss': 1.0967781388759612, 'time_step': 0.0024425601959228516} step=5000
2025-07-28 15:45.37 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5000.d3
Epoch 51/100: 100%|██████████| 100/100 [00:00<00:00, 363.68it/s, loss=2.24, td_loss=0.0451, conservative_loss=1.1]
2025-07-28 15:45.38 [info ] DiscreteCQL_20250728154525: epoch=51 step=5100 epoch=51 metrics={'time_sample_batch': 0.0005719494819641113, 'time_algorithm_update': 0.0021027255058288573, 'loss': 2.2378094482421873, 'td_loss': 0.041239810792612845, 'conservative_loss': 1.0982848227024078, 'time_step': 0.002734043598175049} step=5100
2025-07-28 15:45.38 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5100.d3
Epoch 52/100: 100%|██████████| 100/100 [00:00<00:00, 358.04it/s, loss=2.2, td_loss=0.00234, conservative_loss=1.1]
2025-07-28 15:45.38 [info ] DiscreteCQL_20250728154525: epoch=52 step=5200 epoch=52 metrics={'time_sample_batch': 0.0005881166458129882, 'time_algorithm_update': 0.002130899429321289, 'loss': 2.1982424426078797, 'td_loss': 0.002603997699916363, 'conservative_loss': 1.0978192222118377, 'time_step': 0.0027790045738220214} step=5200
2025-07-28 15:45.38 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5200.d3
Epoch 53/100: 100%|██████████| 100/100 [00:00<00:00, 387.23it/s, loss=2.2, td_loss=0.00196, conservative_loss=1.1]
2025-07-28 15:45.38 [info ] DiscreteCQL_20250728154525: epoch=53 step=5300 epoch=53 metrics={'time_sample_batch': 0.0005287861824035644, 'time_algorithm_update': 0.001987776756286621, 'loss': 2.1960549640655516, 'td_loss': 0.001880011151661165, 'conservative_loss': 1.0970874762535094, 'time_step': 0.0025689053535461427} step=5300
2025-07-28 15:45.38 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5300.d3
Epoch 54/100: 100%|██████████| 100/100 [00:00<00:00, 297.48it/s, loss=2.2, td_loss=0.000848, conservative_loss=1.1]
2025-07-28 15:45.39 [info ] DiscreteCQL_20250728154525: epoch=54 step=5400 epoch=54 metrics={'time_sample_batch': 0.0005863070487976074, 'time_algorithm_update': 0.0026813626289367678, 'loss': 2.1959192824363707, 'td_loss': 0.0009607326565310359, 'conservative_loss': 1.0974792742729187, 'time_step': 0.003337249755859375} step=5400
2025-07-28 15:45.39 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5400.d3
Epoch 55/100: 100%|██████████| 100/100 [00:00<00:00, 415.45it/s, loss=2.2, td_loss=0.00116, conservative_loss=1.1]
2025-07-28 15:45.39 [info ] DiscreteCQL_20250728154525: epoch=55 step=5500 epoch=55 metrics={'time_sample_batch': 0.000460207462310791, 'time_algorithm_update': 0.0018834710121154786, 'loss': 2.196174066066742, 'td_loss': 0.0012187362852273508, 'conservative_loss': 1.0974776697158815, 'time_step': 0.0023953890800476074} step=5500
2025-07-28 15:45.39 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5500.d3
Epoch 56/100: 100%|██████████| 100/100 [00:00<00:00, 386.78it/s, loss=2.2, td_loss=0.0025, conservative_loss=1.1]
2025-07-28 15:45.39 [info ] DiscreteCQL_20250728154525: epoch=56 step=5600 epoch=56 metrics={'time_sample_batch': 0.0005307507514953614, 'time_algorithm_update': 0.001985924243927002, 'loss': 2.197875699996948, 'td_loss': 0.0023507915827212854, 'conservative_loss': 1.0977624487876891, 'time_step': 0.00257326602935791} step=5600
2025-07-28 15:45.39 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5600.d3
Epoch 57/100: 100%|██████████| 100/100 [00:00<00:00, 386.88it/s, loss=2.2, td_loss=0.0011, conservative_loss=1.1]
2025-07-28 15:45.39 [info ] DiscreteCQL_20250728154525: epoch=57 step=5700 epoch=57 metrics={'time_sample_batch': 0.0005351758003234863, 'time_algorithm_update': 0.0019792938232421876, 'loss': 2.1951411747932434, 'td_loss': 0.0010955849025049247, 'conservative_loss': 1.0970227921009064, 'time_step': 0.0025716471672058107} step=5700
2025-07-28 15:45.39 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5700.d3
Epoch 58/100: 100%|██████████| 100/100 [00:00<00:00, 416.13it/s, loss=2.2, td_loss=0.00129, conservative_loss=1.1]
2025-07-28 15:45.40 [info ] DiscreteCQL_20250728154525: epoch=58 step=5800 epoch=58 metrics={'time_sample_batch': 0.00047307252883911134, 'time_algorithm_update': 0.0018701672554016114, 'loss': 2.1969187307357787, 'td_loss': 0.0012697618824313395, 'conservative_loss': 1.097824476957321, 'time_step': 0.002392282485961914} step=5800
2025-07-28 15:45.40 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5800.d3
Epoch 59/100: 100%|██████████| 100/100 [00:00<00:00, 408.85it/s, loss=2.19, td_loss=0.00188, conservative_loss=1.1]
2025-07-28 15:45.40 [info ] DiscreteCQL_20250728154525: epoch=59 step=5900 epoch=59 metrics={'time_sample_batch': 0.00047320127487182617, 'time_algorithm_update': 0.0019121074676513672, 'loss': 2.1937217545509338, 'td_loss': 0.0018052738095866517, 'conservative_loss': 1.0959582471847533, 'time_step': 0.0024355602264404296} step=5900
2025-07-28 15:45.40 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_5900.d3
Epoch 60/100: 100%|██████████| 100/100 [00:00<00:00, 418.71it/s, loss=2.2, td_loss=0.00119, conservative_loss=1.1]
2025-07-28 15:45.40 [info ] DiscreteCQL_20250728154525: epoch=60 step=6000 epoch=60 metrics={'time_sample_batch': 0.0004694795608520508, 'time_algorithm_update': 0.001861875057220459, 'loss': 2.1961928606033325, 'td_loss': 0.0012504387358785608, 'conservative_loss': 1.0974712038040162, 'time_step': 0.002376384735107422} step=6000
2025-07-28 15:45.40 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6000.d3
Epoch 61/100: 100%|██████████| 100/100 [00:00<00:00, 410.58it/s, loss=2.24, td_loss=0.0434, conservative_loss=1.1]
2025-07-28 15:45.40 [info ] DiscreteCQL_20250728154525: epoch=61 step=6100 epoch=61 metrics={'time_sample_batch': 0.00047635555267333985, 'time_algorithm_update': 0.0018982386589050293, 'loss': 2.236229724884033, 'td_loss': 0.03967914224718697, 'conservative_loss': 1.0982752883434295, 'time_step': 0.0024237799644470214} step=6100
2025-07-28 15:45.40 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6100.d3
Epoch 62/100: 100%|██████████| 100/100 [00:00<00:00, 404.97it/s, loss=2.2, td_loss=0.00187, conservative_loss=1.1]
2025-07-28 15:45.41 [info ] DiscreteCQL_20250728154525: epoch=62 step=6200 epoch=62 metrics={'time_sample_batch': 0.0004819440841674805, 'time_algorithm_update': 0.0019266748428344728, 'loss': 2.197395284175873, 'td_loss': 0.0018316685914760455, 'conservative_loss': 1.0977818071842194, 'time_step': 0.0024573302268981936} step=6200
2025-07-28 15:45.41 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6200.d3
Epoch 63/100: 100%|██████████| 100/100 [00:00<00:00, 405.50it/s, loss=2.2, td_loss=0.00162, conservative_loss=1.1]
2025-07-28 15:45.41 [info ] DiscreteCQL_20250728154525: epoch=63 step=6300 epoch=63 metrics={'time_sample_batch': 0.0004749917984008789, 'time_algorithm_update': 0.001930844783782959, 'loss': 2.1965748119354247, 'td_loss': 0.001626943639712408, 'conservative_loss': 1.0974739360809327, 'time_step': 0.002455434799194336} step=6300
2025-07-28 15:45.41 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6300.d3
Epoch 64/100: 100%|██████████| 100/100 [00:00<00:00, 410.23it/s, loss=2.2, td_loss=0.00117, conservative_loss=1.1]
2025-07-28 15:45.41 [info ] DiscreteCQL_20250728154525: epoch=64 step=6400 epoch=64 metrics={'time_sample_batch': 0.0004700160026550293, 'time_algorithm_update': 0.0019088816642761231, 'loss': 2.197249164581299, 'td_loss': 0.0011479717545444146, 'conservative_loss': 1.0980505990982055, 'time_step': 0.0024258971214294435} step=6400
2025-07-28 15:45.41 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6400.d3
Epoch 65/100: 100%|██████████| 100/100 [00:00<00:00, 412.26it/s, loss=2.2, td_loss=0.0015, conservative_loss=1.1]
2025-07-28 15:45.41 [info ] DiscreteCQL_20250728154525: epoch=65 step=6500 epoch=65 metrics={'time_sample_batch': 0.0004769134521484375, 'time_algorithm_update': 0.0018893003463745118, 'loss': 2.1957614421844482, 'td_loss': 0.0014366957996389828, 'conservative_loss': 1.0971623718738557, 'time_step': 0.0024144744873046877} step=6500
2025-07-28 15:45.41 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6500.d3
Epoch 66/100: 100%|██████████| 100/100 [00:00<00:00, 406.36it/s, loss=2.2, td_loss=0.00151, conservative_loss=1.1]
2025-07-28 15:45.42 [info ] DiscreteCQL_20250728154525: epoch=66 step=6600 epoch=66 metrics={'time_sample_batch': 0.0004748296737670898, 'time_algorithm_update': 0.001923069953918457, 'loss': 2.1966270327568056, 'td_loss': 0.0014970313938101755, 'conservative_loss': 1.0975650000572204, 'time_step': 0.0024482226371765137} step=6600
2025-07-28 15:45.42 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6600.d3
Epoch 67/100: 100%|██████████| 100/100 [00:00<00:00, 324.99it/s, loss=2.2, td_loss=0.00169, conservative_loss=1.1]
2025-07-28 15:45.42 [info ] DiscreteCQL_20250728154525: epoch=67 step=6700 epoch=67 metrics={'time_sample_batch': 0.0006723475456237793, 'time_algorithm_update': 0.0023181843757629395, 'loss': 2.1983171677589417, 'td_loss': 0.0017722686848719604, 'conservative_loss': 1.0982724475860595, 'time_step': 0.003056778907775879} step=6700
2025-07-28 15:45.42 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6700.d3
Epoch 68/100: 100%|██████████| 100/100 [00:00<00:00, 419.24it/s, loss=2.2, td_loss=0.00114, conservative_loss=1.1]
2025-07-28 15:45.42 [info ] DiscreteCQL_20250728154525: epoch=68 step=6800 epoch=68 metrics={'time_sample_batch': 0.00045365333557128906, 'time_algorithm_update': 0.0018648409843444824, 'loss': 2.1956448483467104, 'td_loss': 0.0012274235920631327, 'conservative_loss': 1.0972087144851685, 'time_step': 0.002373466491699219} step=6800
2025-07-28 15:45.42 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6800.d3
Epoch 69/100: 100%|██████████| 100/100 [00:00<00:00, 413.47it/s, loss=2.2, td_loss=0.00108, conservative_loss=1.1]
2025-07-28 15:45.42 [info ] DiscreteCQL_20250728154525: epoch=69 step=6900 epoch=69 metrics={'time_sample_batch': 0.00047734737396240237, 'time_algorithm_update': 0.0018783402442932129, 'loss': 2.1954086685180663, 'td_loss': 0.0010653546734829434, 'conservative_loss': 1.0971716606616975, 'time_step': 0.0024070024490356447} step=6900
2025-07-28 15:45.42 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_6900.d3
Epoch 70/100: 100%|██████████| 100/100 [00:00<00:00, 421.44it/s, loss=2.2, td_loss=0.00129, conservative_loss=1.1]
2025-07-28 15:45.43 [info ] DiscreteCQL_20250728154525: epoch=70 step=7000 epoch=70 metrics={'time_sample_batch': 0.0004603123664855957, 'time_algorithm_update': 0.0018489694595336915, 'loss': 2.1972744131088255, 'td_loss': 0.0013059828532277606, 'conservative_loss': 1.0979842114448548, 'time_step': 0.002361128330230713} step=7000
2025-07-28 15:45.43 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7000.d3
Epoch 71/100: 100%|██████████| 100/100 [00:00<00:00, 385.32it/s, loss=2.24, td_loss=0.0424, conservative_loss=1.1]
2025-07-28 15:45.43 [info ] DiscreteCQL_20250728154525: epoch=71 step=7100 epoch=71 metrics={'time_sample_batch': 0.0005326128005981445, 'time_algorithm_update': 0.0019896578788757323, 'loss': 2.2326128482818604, 'td_loss': 0.03883167172083631, 'conservative_loss': 1.0968905925750732, 'time_step': 0.0025800132751464845} step=7100
2025-07-28 15:45.43 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7100.d3
Epoch 72/100: 100%|██████████| 100/100 [00:00<00:00, 404.99it/s, loss=2.2, td_loss=0.00172, conservative_loss=1.1]
2025-07-28 15:45.43 [info ] DiscreteCQL_20250728154525: epoch=72 step=7200 epoch=72 metrics={'time_sample_batch': 0.000496530532836914, 'time_algorithm_update': 0.0019109487533569337, 'loss': 2.1950860810279846, 'td_loss': 0.001654102717875503, 'conservative_loss': 1.0967159831523896, 'time_step': 0.0024579763412475586} step=7200
2025-07-28 15:45.43 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7200.d3
Epoch 73/100: 100%|██████████| 100/100 [00:00<00:00, 404.06it/s, loss=2.2, td_loss=0.00152, conservative_loss=1.1]
2025-07-28 15:45.43 [info ] DiscreteCQL_20250728154525: epoch=73 step=7300 epoch=73 metrics={'time_sample_batch': 0.0005002140998840332, 'time_algorithm_update': 0.0019114255905151368, 'loss': 2.1955630135536195, 'td_loss': 0.0014652931969612838, 'conservative_loss': 1.0970488595962524, 'time_step': 0.0024619674682617187} step=7300
2025-07-28 15:45.43 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7300.d3
Epoch 74/100: 100%|██████████| 100/100 [00:00<00:00, 384.02it/s, loss=2.2, td_loss=0.0012, conservative_loss=1.1]
2025-07-28 15:45.44 [info ] DiscreteCQL_20250728154525: epoch=74 step=7400 epoch=74 metrics={'time_sample_batch': 0.0005357074737548828, 'time_algorithm_update': 0.0019951415061950683, 'loss': 2.1952951455116274, 'td_loss': 0.0011524688403005711, 'conservative_loss': 1.0970713353157044, 'time_step': 0.0025900816917419434} step=7400
2025-07-28 15:45.44 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7400.d3
Epoch 75/100: 100%|██████████| 100/100 [00:00<00:00, 359.23it/s, loss=2.2, td_loss=0.00105, conservative_loss=1.1]
2025-07-28 15:45.44 [info ] DiscreteCQL_20250728154525: epoch=75 step=7500 epoch=75 metrics={'time_sample_batch': 0.000604555606842041, 'time_algorithm_update': 0.002111508846282959, 'loss': 2.196189987659454, 'td_loss': 0.0011116434112773278, 'conservative_loss': 1.0975391697883605, 'time_step': 0.002771108150482178} step=7500
2025-07-28 15:45.44 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7500.d3
Epoch 76/100: 100%|██████████| 100/100 [00:00<00:00, 406.81it/s, loss=2.19, td_loss=0.00148, conservative_loss=1.1]
2025-07-28 15:45.44 [info ] DiscreteCQL_20250728154525: epoch=76 step=7600 epoch=76 metrics={'time_sample_batch': 0.0004975366592407227, 'time_algorithm_update': 0.0018982982635498047, 'loss': 2.1946453285217284, 'td_loss': 0.0015036492938816082, 'conservative_loss': 1.0965708434581756, 'time_step': 0.002446579933166504} step=7600
2025-07-28 15:45.44 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7600.d3
Epoch 77/100: 100%|██████████| 100/100 [00:00<00:00, 423.70it/s, loss=2.2, td_loss=0.00129, conservative_loss=1.1]
2025-07-28 15:45.44 [info ] DiscreteCQL_20250728154525: epoch=77 step=7700 epoch=77 metrics={'time_sample_batch': 0.00047012090682983397, 'time_algorithm_update': 0.0018267321586608886, 'loss': 2.196987895965576, 'td_loss': 0.0012641044548945502, 'conservative_loss': 1.0978618907928466, 'time_step': 0.002348353862762451} step=7700
2025-07-28 15:45.44 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7700.d3
Epoch 78/100: 100%|██████████| 100/100 [00:00<00:00, 371.20it/s, loss=2.2, td_loss=0.00112, conservative_loss=1.1]
2025-07-28 15:45.45 [info ] DiscreteCQL_20250728154525: epoch=78 step=7800 epoch=78 metrics={'time_sample_batch': 0.0005690073966979981, 'time_algorithm_update': 0.002055826187133789, 'loss': 2.195310065746307, 'td_loss': 0.0011137244015117175, 'conservative_loss': 1.0970981705188751, 'time_step': 0.002682504653930664} step=7800
2025-07-28 15:45.45 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7800.d3
Epoch 79/100: 100%|██████████| 100/100 [00:00<00:00, 403.94it/s, loss=2.2, td_loss=0.00116, conservative_loss=1.1]
2025-07-28 15:45.45 [info ] DiscreteCQL_20250728154525: epoch=79 step=7900 epoch=79 metrics={'time_sample_batch': 0.0005019116401672363, 'time_algorithm_update': 0.0019088006019592286, 'loss': 2.195496542453766, 'td_loss': 0.001138850417046342, 'conservative_loss': 1.0971788382530212, 'time_step': 0.0024642467498779295} step=7900
2025-07-28 15:45.45 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_7900.d3
Epoch 80/100: 100%|██████████| 100/100 [00:00<00:00, 369.64it/s, loss=2.2, td_loss=0.00111, conservative_loss=1.1]
2025-07-28 15:45.45 [info ] DiscreteCQL_20250728154525: epoch=80 step=8000 epoch=80 metrics={'time_sample_batch': 0.0005827975273132324, 'time_algorithm_update': 0.0020485162734985352, 'loss': 2.1947550106048586, 'td_loss': 0.001167198322364129, 'conservative_loss': 1.0967939066886903, 'time_step': 0.002692546844482422} step=8000
2025-07-28 15:45.45 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8000.d3
Epoch 81/100: 100%|██████████| 100/100 [00:00<00:00, 419.20it/s, loss=2.23, td_loss=0.0371, conservative_loss=1.1]
2025-07-28 15:45.45 [info ] DiscreteCQL_20250728154525: epoch=81 step=8100 epoch=81 metrics={'time_sample_batch': 0.00047295331954956056, 'time_algorithm_update': 0.0018500924110412599, 'loss': 2.227931935787201, 'td_loss': 0.033952856429386884, 'conservative_loss': 1.0969895398616791, 'time_step': 0.0023734259605407717} step=8100
2025-07-28 15:45.45 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8100.d3
Epoch 82/100: 100%|██████████| 100/100 [00:00<00:00, 421.24it/s, loss=2.2, td_loss=0.00183, conservative_loss=1.1]
2025-07-28 15:45.46 [info ] DiscreteCQL_20250728154525: epoch=82 step=8200 epoch=82 metrics={'time_sample_batch': 0.0004636073112487793, 'time_algorithm_update': 0.0018477320671081543, 'loss': 2.19747983455658, 'td_loss': 0.0023927097214618697, 'conservative_loss': 1.0975435674190521, 'time_step': 0.002362477779388428} step=8200
2025-07-28 15:45.46 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8200.d3
Epoch 83/100: 100%|██████████| 100/100 [00:00<00:00, 386.21it/s, loss=2.2, td_loss=0.00325, conservative_loss=1.1]
2025-07-28 15:45.46 [info ] DiscreteCQL_20250728154525: epoch=83 step=8300 epoch=83 metrics={'time_sample_batch': 0.000538039207458496, 'time_algorithm_update': 0.001985733509063721, 'loss': 2.2000172615051268, 'td_loss': 0.0034504746797028927, 'conservative_loss': 1.098283394575119, 'time_step': 0.0025764894485473633} step=8300
2025-07-28 15:45.46 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8300.d3
Epoch 84/100: 100%|██████████| 100/100 [00:00<00:00, 419.34it/s, loss=2.2, td_loss=0.00298, conservative_loss=1.1]
2025-07-28 15:45.46 [info ] DiscreteCQL_20250728154525: epoch=84 step=8400 epoch=84 metrics={'time_sample_batch': 0.0004709506034851074, 'time_algorithm_update': 0.0018519878387451172, 'loss': 2.1969930720329285, 'td_loss': 0.0028046430594986303, 'conservative_loss': 1.097094213962555, 'time_step': 0.002372426986694336} step=8400
2025-07-28 15:45.46 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8400.d3
Epoch 85/100: 100%|██████████| 100/100 [00:00<00:00, 409.10it/s, loss=2.2, td_loss=0.00295, conservative_loss=1.1]
2025-07-28 15:45.46 [info ] DiscreteCQL_20250728154525: epoch=85 step=8500 epoch=85 metrics={'time_sample_batch': 0.0004938220977783203, 'time_algorithm_update': 0.0018848967552185058, 'loss': 2.197575035095215, 'td_loss': 0.0028086735983379185, 'conservative_loss': 1.0973831844329833, 'time_step': 0.0024326634407043457} step=8500
2025-07-28 15:45.46 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8500.d3
Epoch 86/100: 100%|██████████| 100/100 [00:00<00:00, 413.77it/s, loss=2.2, td_loss=0.00271, conservative_loss=1.1]
2025-07-28 15:45.47 [info ] DiscreteCQL_20250728154525: epoch=86 step=8600 epoch=86 metrics={'time_sample_batch': 0.0004716086387634277, 'time_algorithm_update': 0.0018826532363891602, 'loss': 2.197786946296692, 'td_loss': 0.0026098875643219797, 'conservative_loss': 1.0975885307788849, 'time_step': 0.002405276298522949} step=8600
2025-07-28 15:45.47 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8600.d3
Epoch 87/100: 100%|██████████| 100/100 [00:00<00:00, 385.00it/s, loss=2.2, td_loss=0.00274, conservative_loss=1.1]
2025-07-28 15:45.47 [info ] DiscreteCQL_20250728154525: epoch=87 step=8700 epoch=87 metrics={'time_sample_batch': 0.0005232834815979004, 'time_algorithm_update': 0.002008681297302246, 'loss': 2.197643308639526, 'td_loss': 0.002755700311390683, 'conservative_loss': 1.0974437963962556, 'time_step': 0.002584958076477051} step=8700
2025-07-28 15:45.47 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8700.d3
Epoch 88/100: 100%|██████████| 100/100 [00:00<00:00, 405.34it/s, loss=2.2, td_loss=0.00213, conservative_loss=1.1]
2025-07-28 15:45.47 [info ] DiscreteCQL_20250728154525: epoch=88 step=8800 epoch=88 metrics={'time_sample_batch': 0.0004944634437561035, 'time_algorithm_update': 0.001909637451171875, 'loss': 2.1980531549453737, 'td_loss': 0.0023269034427357838, 'conservative_loss': 1.0978631222248076, 'time_step': 0.002456667423248291} step=8800
2025-07-28 15:45.47 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8800.d3
Epoch 89/100: 100%|██████████| 100/100 [00:00<00:00, 398.64it/s, loss=2.2, td_loss=0.00367, conservative_loss=1.1]
2025-07-28 15:45.47 [info ] DiscreteCQL_20250728154525: epoch=89 step=8900 epoch=89 metrics={'time_sample_batch': 0.0005027151107788086, 'time_algorithm_update': 0.0019388103485107422, 'loss': 2.198425705432892, 'td_loss': 0.0036549880728125573, 'conservative_loss': 1.0973853635787965, 'time_step': 0.002494988441467285} step=8900
2025-07-28 15:45.47 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_8900.d3
Epoch 90/100: 100%|██████████| 100/100 [00:00<00:00, 415.54it/s, loss=2.2, td_loss=0.00237, conservative_loss=1.1]
2025-07-28 15:45.48 [info ] DiscreteCQL_20250728154525: epoch=90 step=9000 epoch=90 metrics={'time_sample_batch': 0.0004624009132385254, 'time_algorithm_update': 0.001880943775177002, 'loss': 2.1986197805404664, 'td_loss': 0.0026663483370793985, 'conservative_loss': 1.0979767179489135, 'time_step': 0.002394850254058838} step=9000
2025-07-28 15:45.48 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9000.d3
Epoch 91/100: 100%|██████████| 100/100 [00:00<00:00, 391.35it/s, loss=2.23, td_loss=0.0362, conservative_loss=1.1]
2025-07-28 15:45.48 [info ] DiscreteCQL_20250728154525: epoch=91 step=9100 epoch=91 metrics={'time_sample_batch': 0.0005315661430358887, 'time_algorithm_update': 0.001959683895111084, 'loss': 2.228097147941589, 'td_loss': 0.033212613491923546, 'conservative_loss': 1.0974422657489777, 'time_step': 0.002542593479156494} step=9100
2025-07-28 15:45.48 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9100.d3
Epoch 92/100: 100%|██████████| 100/100 [00:00<00:00, 427.37it/s, loss=2.2, td_loss=0.00235, conservative_loss=1.1]
2025-07-28 15:45.48 [info ] DiscreteCQL_20250728154525: epoch=92 step=9200 epoch=92 metrics={'time_sample_batch': 0.00045597076416015625, 'time_algorithm_update': 0.0018234753608703613, 'loss': 2.1968222665786743, 'td_loss': 0.0023351298249326647, 'conservative_loss': 1.0972435677051544, 'time_step': 0.0023288178443908692} step=9200
2025-07-28 15:45.48 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9200.d3
Epoch 93/100: 100%|██████████| 100/100 [00:00<00:00, 426.30it/s, loss=2.2, td_loss=0.00227, conservative_loss=1.1]
2025-07-28 15:45.48 [info ] DiscreteCQL_20250728154525: epoch=93 step=9300 epoch=93 metrics={'time_sample_batch': 0.00046092748641967773, 'time_algorithm_update': 0.0018229055404663085, 'loss': 2.1981692504882813, 'td_loss': 0.002199953671079129, 'conservative_loss': 1.0979846489429474, 'time_step': 0.0023334813117980957} step=9300
2025-07-28 15:45.48 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9300.d3
Epoch 94/100: 100%|██████████| 100/100 [00:00<00:00, 384.56it/s, loss=2.2, td_loss=0.00219, conservative_loss=1.1]
2025-07-28 15:45.49 [info ] DiscreteCQL_20250728154525: epoch=94 step=9400 epoch=94 metrics={'time_sample_batch': 0.0005285000801086426, 'time_algorithm_update': 0.002005882263183594, 'loss': 2.197166223526001, 'td_loss': 0.0024109763628803195, 'conservative_loss': 1.097377623319626, 'time_step': 0.0025871992111206055} step=9400
2025-07-28 15:45.49 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9400.d3
Epoch 95/100: 100%|██████████| 100/100 [00:00<00:00, 398.11it/s, loss=2.2, td_loss=0.00302, conservative_loss=1.1]
2025-07-28 15:45.49 [info ] DiscreteCQL_20250728154525: epoch=95 step=9500 epoch=95 metrics={'time_sample_batch': 0.0005094480514526368, 'time_algorithm_update': 0.0019360923767089845, 'loss': 2.1988195276260374, 'td_loss': 0.0029319437482627107, 'conservative_loss': 1.0979437935352325, 'time_step': 0.0025005745887756348} step=9500
2025-07-28 15:45.49 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9500.d3
Epoch 96/100: 100%|██████████| 100/100 [00:00<00:00, 404.66it/s, loss=2.2, td_loss=0.00149, conservative_loss=1.1]
2025-07-28 15:45.49 [info ] DiscreteCQL_20250728154525: epoch=96 step=9600 epoch=96 metrics={'time_sample_batch': 0.0005018281936645508, 'time_algorithm_update': 0.0019066476821899415, 'loss': 2.1952530360221862, 'td_loss': 0.0014993083794252015, 'conservative_loss': 1.0968768572807313, 'time_step': 0.002458498477935791} step=9600
2025-07-28 15:45.49 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9600.d3
Epoch 97/100: 100%|██████████| 100/100 [00:00<00:00, 414.73it/s, loss=2.19, td_loss=0.00232, conservative_loss=1.1]
2025-07-28 15:45.49 [info ] DiscreteCQL_20250728154525: epoch=97 step=9700 epoch=97 metrics={'time_sample_batch': 0.00048806190490722655, 'time_algorithm_update': 0.0018617892265319824, 'loss': 2.1928140926361084, 'td_loss': 0.002345390365226194, 'conservative_loss': 1.095234352350235, 'time_step': 0.0023996424674987793} step=9700
2025-07-28 15:45.49 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9700.d3
Epoch 98/100: 100%|██████████| 100/100 [00:00<00:00, 416.84it/s, loss=2.2, td_loss=0.002, conservative_loss=1.1]
2025-07-28 15:45.50 [info ] DiscreteCQL_20250728154525: epoch=98 step=9800 epoch=98 metrics={'time_sample_batch': 0.00047081470489501953, 'time_algorithm_update': 0.0018648099899291992, 'loss': 2.19599750995636, 'td_loss': 0.001943185601849109, 'conservative_loss': 1.0970271587371827, 'time_step': 0.00238771915435791} step=9800
2025-07-28 15:45.50 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9800.d3
Epoch 99/100: 100%|██████████| 100/100 [00:00<00:00, 415.27it/s, loss=2.2, td_loss=0.00217, conservative_loss=1.1]
2025-07-28 15:45.50 [info ] DiscreteCQL_20250728154525: epoch=99 step=9900 epoch=99 metrics={'time_sample_batch': 0.00047873735427856446, 'time_algorithm_update': 0.001869077682495117, 'loss': 2.195319347381592, 'td_loss': 0.0021575475920690224, 'conservative_loss': 1.0965808951854705, 'time_step': 0.002396836280822754} step=9900
2025-07-28 15:45.50 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_9900.d3
Epoch 100/100: 100%|██████████| 100/100 [00:00<00:00, 321.70it/s, loss=2.2, td_loss=0.00222, conservative_loss=1.1]
2025-07-28 15:45.50 [info ] DiscreteCQL_20250728154525: epoch=100 step=10000 epoch=100 metrics={'time_sample_batch': 0.00057830810546875, 'time_algorithm_update': 0.002337493896484375, 'loss': 2.1974499917030332, 'td_loss': 0.002233420611009933, 'conservative_loss': 1.0976082861423493, 'time_step': 0.002981810569763184} step=10000
2025-07-28 15:45.50 [info ] Model parameters are saved to d3rlpy_logs/DiscreteCQL_20250728154525/model_10000.d3
[15]:
[(1,
{'time_sample_batch': 0.0005567145347595215,
'time_algorithm_update': 0.0020180344581604004,
'loss': 2.3966327929496765,
'td_loss': 0.19853577110916376,
'conservative_loss': 1.09904851436615,
'time_step': 0.00263277530670166}),
(2,
{'time_sample_batch': 0.0004550933837890625,
'time_algorithm_update': 0.00175459623336792,
'loss': 2.212840859889984,
'td_loss': 0.01538949720794335,
'conservative_loss': 1.0987256860733032,
'time_step': 0.0022612261772155763}),
(3,
{'time_sample_batch': 0.00045412778854370117,
'time_algorithm_update': 0.0017646265029907227,
'loss': 2.200702414512634,
'td_loss': 0.003590560882585123,
'conservative_loss': 1.0985559260845184,
'time_step': 0.002267448902130127}),
(4,
{'time_sample_batch': 0.0005457544326782226,
'time_algorithm_update': 0.0019359111785888672,
'loss': 2.1985737800598146,
'td_loss': 0.001622819603071548,
'conservative_loss': 1.0984754824638367,
'time_step': 0.0025356578826904296}),
(5,
{'time_sample_batch': 0.0004919314384460449,
'time_algorithm_update': 0.001887047290802002,
'loss': 2.19991708278656,
'td_loss': 0.004285427670693025,
'conservative_loss': 1.09781583070755,
'time_step': 0.0024393200874328613}),
(6,
{'time_sample_batch': 0.0004746437072753906,
'time_algorithm_update': 0.0017737793922424317,
'loss': 2.198714690208435,
'td_loss': 0.0020858740305993704,
'conservative_loss': 1.0983144092559813,
'time_step': 0.002298996448516846}),
(7,
{'time_sample_batch': 0.00046657800674438477,
'time_algorithm_update': 0.001801440715789795,
'loss': 2.2006174540519714,
'td_loss': 0.003812542131054215,
'conservative_loss': 1.098402464389801,
'time_step': 0.0023208904266357423}),
(8,
{'time_sample_batch': 0.0004999542236328125,
'time_algorithm_update': 0.0019462394714355468,
'loss': 2.201106686592102,
'td_loss': 0.004484999722335487,
'conservative_loss': 1.098310842514038,
'time_step': 0.0025028419494628907}),
(9,
{'time_sample_batch': 0.0004914283752441406,
'time_algorithm_update': 0.0019498515129089357,
'loss': 2.1986337685585022,
'td_loss': 0.00250896273937542,
'conservative_loss': 1.0980623996257781,
'time_step': 0.002493433952331543}),
(10,
{'time_sample_batch': 0.0005762290954589844,
'time_algorithm_update': 0.0020333194732666017,
'loss': 2.1994874477386475,
'td_loss': 0.002849989581736736,
'conservative_loss': 1.0983187305927276,
'time_step': 0.0026563167572021484}),
(11,
{'time_sample_batch': 0.0006136512756347657,
'time_algorithm_update': 0.0020970726013183594,
'loss': 2.28967072725296,
'td_loss': 0.0928150053115678,
'conservative_loss': 1.0984278655052184,
'time_step': 0.0027637600898742674}),
(12,
{'time_sample_batch': 0.0006206274032592773,
'time_algorithm_update': 0.0021375727653503416,
'loss': 2.1974543070793153,
'td_loss': 0.0006476261564239394,
'conservative_loss': 1.0984033381938934,
'time_step': 0.0028111600875854494}),
(13,
{'time_sample_batch': 0.0006099987030029297,
'time_algorithm_update': 0.002111635208129883,
'loss': 2.197365963459015,
'td_loss': 0.0008116933463315945,
'conservative_loss': 1.0982771360874175,
'time_step': 0.0027706408500671386}),
(14,
{'time_sample_batch': 0.0006060028076171876,
'time_algorithm_update': 0.0020788097381591797,
'loss': 2.1977316999435423,
'td_loss': 0.000837066586536821,
'conservative_loss': 1.0984473168849944,
'time_step': 0.002737417221069336}),
(15,
{'time_sample_batch': 0.0006205201148986817,
'time_algorithm_update': 0.0021503591537475586,
'loss': 2.197506539821625,
'td_loss': 0.000559124128849362,
'conservative_loss': 1.0984737122058867,
'time_step': 0.002822751998901367}),
(16,
{'time_sample_batch': 0.0006248164176940918,
'time_algorithm_update': 0.002216043472290039,
'loss': 2.197997591495514,
'td_loss': 0.00094447792347637,
'conservative_loss': 1.0985265529155732,
'time_step': 0.0028961730003356934}),
(17,
{'time_sample_batch': 0.0006282734870910645,
'time_algorithm_update': 0.0021921420097351075,
'loss': 2.1970662832260133,
'td_loss': 0.0007343449354812037,
'conservative_loss': 1.0981659662723542,
'time_step': 0.002886338233947754}),
(18,
{'time_sample_batch': 0.0005885624885559082,
'time_algorithm_update': 0.0020654106140136717,
'loss': 2.194919857978821,
'td_loss': 0.0009621448916732334,
'conservative_loss': 1.0969788503646851,
'time_step': 0.0027103424072265625}),
(19,
{'time_sample_batch': 0.0005143189430236816,
'time_algorithm_update': 0.001968593597412109,
'loss': 2.1962599110603334,
'td_loss': 0.0009440037869353546,
'conservative_loss': 1.0976579535007476,
'time_step': 0.002539358139038086}),
(20,
{'time_sample_batch': 0.0004642653465270996,
'time_algorithm_update': 0.0018935203552246094,
'loss': 2.1974512267112734,
'td_loss': 0.0008664641072391533,
'conservative_loss': 1.0982923746109008,
'time_step': 0.002407989501953125}),
(21,
{'time_sample_batch': 0.0005271315574645996,
'time_algorithm_update': 0.0019734764099121095,
'loss': 2.261094753742218,
'td_loss': 0.0650152921580593,
'conservative_loss': 1.0980397248268128,
'time_step': 0.002556498050689697}),
(22,
{'time_sample_batch': 0.000541989803314209,
'time_algorithm_update': 0.001987617015838623,
'loss': 2.197791721820831,
'td_loss': 0.0011231240205233917,
'conservative_loss': 1.098334299325943,
'time_step': 0.002582676410675049}),
(23,
{'time_sample_batch': 0.0004741644859313965,
'time_algorithm_update': 0.001876215934753418,
'loss': 2.194902892112732,
'td_loss': 0.0010723149092518724,
'conservative_loss': 1.0969152915477753,
'time_step': 0.0024067497253417967}),
(24,
{'time_sample_batch': 0.0004554891586303711,
'time_algorithm_update': 0.0018364500999450684,
'loss': 2.1958152437210083,
'td_loss': 0.0009611728510935791,
'conservative_loss': 1.0974270331859588,
'time_step': 0.0023450636863708496}),
(25,
{'time_sample_batch': 0.00045788049697875977,
'time_algorithm_update': 0.0018152213096618653,
'loss': 2.1973110389709474,
'td_loss': 0.0009258302650414407,
'conservative_loss': 1.0981926012039185,
'time_step': 0.002320990562438965}),
(26,
{'time_sample_batch': 0.0004761838912963867,
'time_algorithm_update': 0.0018532466888427734,
'loss': 2.196728525161743,
'td_loss': 0.0007932424181490205,
'conservative_loss': 1.0979676389694213,
'time_step': 0.0023805713653564453}),
(27,
{'time_sample_batch': 0.0005347943305969239,
'time_algorithm_update': 0.001997036933898926,
'loss': 2.197197813987732,
'td_loss': 0.0011636634092428721,
'conservative_loss': 1.0980170750617981,
'time_step': 0.0025886750221252443}),
(28,
{'time_sample_batch': 0.0005988383293151856,
'time_algorithm_update': 0.002096092700958252,
'loss': 2.1970012974739075,
'td_loss': 0.0008431548319640569,
'conservative_loss': 1.0980790734291077,
'time_step': 0.002748849391937256}),
(29,
{'time_sample_batch': 0.0004908180236816406,
'time_algorithm_update': 0.0018802642822265624,
'loss': 2.1956692242622378,
'td_loss': 0.0013493687222944572,
'conservative_loss': 1.0971599233150482,
'time_step': 0.0024227428436279295}),
(30,
{'time_sample_batch': 0.0005670523643493652,
'time_algorithm_update': 0.0020356225967407228,
'loss': 2.196509358882904,
'td_loss': 0.001714693953981623,
'conservative_loss': 1.097397335767746,
'time_step': 0.0026561284065246583}),
(31,
{'time_sample_batch': 0.00046095371246337893,
'time_algorithm_update': 0.0018667006492614747,
'loss': 2.2531930565834046,
'td_loss': 0.05699506776465569,
'conservative_loss': 1.0980989944934845,
'time_step': 0.0023780179023742674}),
(32,
{'time_sample_batch': 0.0004948973655700684,
'time_algorithm_update': 0.0018903970718383788,
'loss': 2.196646854877472,
'td_loss': 0.0009938515130488669,
'conservative_loss': 1.0978265023231506,
'time_step': 0.0024421000480651855}),
(33,
{'time_sample_batch': 0.0004577851295471191,
'time_algorithm_update': 0.0018019366264343261,
'loss': 2.1971650981903075,
'td_loss': 0.0008624040643917397,
'conservative_loss': 1.098151340484619,
'time_step': 0.002308976650238037}),
(34,
{'time_sample_batch': 0.00048842191696167,
'time_algorithm_update': 0.0018771505355834961,
'loss': 2.19635671377182,
'td_loss': 0.0009327739142463542,
'conservative_loss': 1.0977119731903076,
'time_step': 0.0024143147468566896}),
(35,
{'time_sample_batch': 0.0004731059074401855,
'time_algorithm_update': 0.001864159107208252,
'loss': 2.1976770973205566,
'td_loss': 0.0011044965320616028,
'conservative_loss': 1.0982862997055054,
'time_step': 0.0023872184753417967}),
(36,
{'time_sample_batch': 0.0005660319328308106,
'time_algorithm_update': 0.0020635390281677247,
'loss': 2.1967822265625,
'td_loss': 0.0007547940395306795,
'conservative_loss': 1.0980137073993683,
'time_step': 0.002687985897064209}),
(37,
{'time_sample_batch': 0.0005019855499267578,
'time_algorithm_update': 0.0019105076789855957,
'loss': 2.194888060092926,
'td_loss': 0.0011231697170296683,
'conservative_loss': 1.0968824434280395,
'time_step': 0.0024664974212646484}),
(38,
{'time_sample_batch': 0.0005265402793884277,
'time_algorithm_update': 0.0019679880142211914,
'loss': 2.1961218523979187,
'td_loss': 0.001076404427876696,
'conservative_loss': 1.0975227224826813,
'time_step': 0.0025478482246398928}),
(39,
{'time_sample_batch': 0.0004711127281188965,
'time_algorithm_update': 0.0019014263153076172,
'loss': 2.1965386128425597,
'td_loss': 0.0010531386702496092,
'conservative_loss': 1.0977427423000337,
'time_step': 0.0024236011505126952}),
(40,
{'time_sample_batch': 0.0004645562171936035,
'time_algorithm_update': 0.0018659543991088867,
'loss': 2.1961905717849732,
'td_loss': 0.0007454320683609694,
'conservative_loss': 1.09772256731987,
'time_step': 0.002378692626953125}),
(41,
{'time_sample_batch': 0.00045660495758056643,
'time_algorithm_update': 0.0018251967430114747,
'loss': 2.2444769239425657,
'td_loss': 0.04920598843396874,
'conservative_loss': 1.0976354658603669,
'time_step': 0.002335050106048584}),
(42,
{'time_sample_batch': 0.0004755043983459473,
'time_algorithm_update': 0.0018530845642089843,
'loss': 2.197680506706238,
'td_loss': 0.0011245051398873329,
'conservative_loss': 1.0982780075073242,
'time_step': 0.002380106449127197}),
(43,
{'time_sample_batch': 0.00046542406082153323,
'time_algorithm_update': 0.0018411803245544434,
'loss': 2.19569308757782,
'td_loss': 0.0008939958154223859,
'conservative_loss': 1.0973995530605316,
'time_step': 0.00235353946685791}),
(44,
{'time_sample_batch': 0.0004541301727294922,
'time_algorithm_update': 0.0018338370323181153,
'loss': 2.197810397148132,
'td_loss': 0.0010860397884971463,
'conservative_loss': 1.0983621823787688,
'time_step': 0.0023408055305480955}),
(45,
{'time_sample_batch': 0.00045701980590820315,
'time_algorithm_update': 0.001823270320892334,
'loss': 2.1963061475753785,
'td_loss': 0.0011160826054401696,
'conservative_loss': 1.0975950312614442,
'time_step': 0.002328629493713379}),
(46,
{'time_sample_batch': 0.0004597926139831543,
'time_algorithm_update': 0.0018334841728210448,
'loss': 2.1983505201339724,
'td_loss': 0.0007569296075962484,
'conservative_loss': 1.0987967932224274,
'time_step': 0.002347161769866943}),
(47,
{'time_sample_batch': 0.00045778989791870115,
'time_algorithm_update': 0.0018497180938720703,
'loss': 2.1964164352416993,
'td_loss': 0.0007753977939137257,
'conservative_loss': 1.097820520401001,
'time_step': 0.0023577427864074707}),
(48,
{'time_sample_batch': 0.0005004525184631348,
'time_algorithm_update': 0.001968719959259033,
'loss': 2.1953913831710814,
'td_loss': 0.0011732379256864079,
'conservative_loss': 1.097109078168869,
'time_step': 0.002527334690093994}),
(49,
{'time_sample_batch': 0.0005402588844299317,
'time_algorithm_update': 0.0019860291481018067,
'loss': 2.195286548137665,
'td_loss': 0.0011594361462630332,
'conservative_loss': 1.0970635497570038,
'time_step': 0.002579333782196045}),
(50,
{'time_sample_batch': 0.0005037856101989747,
'time_algorithm_update': 0.0018882203102111817,
'loss': 2.194396321773529,
'td_loss': 0.0008400411676848307,
'conservative_loss': 1.0967781388759612,
'time_step': 0.0024425601959228516}),
(51,
{'time_sample_batch': 0.0005719494819641113,
'time_algorithm_update': 0.0021027255058288573,
'loss': 2.2378094482421873,
'td_loss': 0.041239810792612845,
'conservative_loss': 1.0982848227024078,
'time_step': 0.002734043598175049}),
(52,
{'time_sample_batch': 0.0005881166458129882,
'time_algorithm_update': 0.002130899429321289,
'loss': 2.1982424426078797,
'td_loss': 0.002603997699916363,
'conservative_loss': 1.0978192222118377,
'time_step': 0.0027790045738220214}),
(53,
{'time_sample_batch': 0.0005287861824035644,
'time_algorithm_update': 0.001987776756286621,
'loss': 2.1960549640655516,
'td_loss': 0.001880011151661165,
'conservative_loss': 1.0970874762535094,
'time_step': 0.0025689053535461427}),
(54,
{'time_sample_batch': 0.0005863070487976074,
'time_algorithm_update': 0.0026813626289367678,
'loss': 2.1959192824363707,
'td_loss': 0.0009607326565310359,
'conservative_loss': 1.0974792742729187,
'time_step': 0.003337249755859375}),
(55,
{'time_sample_batch': 0.000460207462310791,
'time_algorithm_update': 0.0018834710121154786,
'loss': 2.196174066066742,
'td_loss': 0.0012187362852273508,
'conservative_loss': 1.0974776697158815,
'time_step': 0.0023953890800476074}),
(56,
{'time_sample_batch': 0.0005307507514953614,
'time_algorithm_update': 0.001985924243927002,
'loss': 2.197875699996948,
'td_loss': 0.0023507915827212854,
'conservative_loss': 1.0977624487876891,
'time_step': 0.00257326602935791}),
(57,
{'time_sample_batch': 0.0005351758003234863,
'time_algorithm_update': 0.0019792938232421876,
'loss': 2.1951411747932434,
'td_loss': 0.0010955849025049247,
'conservative_loss': 1.0970227921009064,
'time_step': 0.0025716471672058107}),
(58,
{'time_sample_batch': 0.00047307252883911134,
'time_algorithm_update': 0.0018701672554016114,
'loss': 2.1969187307357787,
'td_loss': 0.0012697618824313395,
'conservative_loss': 1.097824476957321,
'time_step': 0.002392282485961914}),
(59,
{'time_sample_batch': 0.00047320127487182617,
'time_algorithm_update': 0.0019121074676513672,
'loss': 2.1937217545509338,
'td_loss': 0.0018052738095866517,
'conservative_loss': 1.0959582471847533,
'time_step': 0.0024355602264404296}),
(60,
{'time_sample_batch': 0.0004694795608520508,
'time_algorithm_update': 0.001861875057220459,
'loss': 2.1961928606033325,
'td_loss': 0.0012504387358785608,
'conservative_loss': 1.0974712038040162,
'time_step': 0.002376384735107422}),
(61,
{'time_sample_batch': 0.00047635555267333985,
'time_algorithm_update': 0.0018982386589050293,
'loss': 2.236229724884033,
'td_loss': 0.03967914224718697,
'conservative_loss': 1.0982752883434295,
'time_step': 0.0024237799644470214}),
(62,
{'time_sample_batch': 0.0004819440841674805,
'time_algorithm_update': 0.0019266748428344728,
'loss': 2.197395284175873,
'td_loss': 0.0018316685914760455,
'conservative_loss': 1.0977818071842194,
'time_step': 0.0024573302268981936}),
(63,
{'time_sample_batch': 0.0004749917984008789,
'time_algorithm_update': 0.001930844783782959,
'loss': 2.1965748119354247,
'td_loss': 0.001626943639712408,
'conservative_loss': 1.0974739360809327,
'time_step': 0.002455434799194336}),
(64,
{'time_sample_batch': 0.0004700160026550293,
'time_algorithm_update': 0.0019088816642761231,
'loss': 2.197249164581299,
'td_loss': 0.0011479717545444146,
'conservative_loss': 1.0980505990982055,
'time_step': 0.0024258971214294435}),
(65,
{'time_sample_batch': 0.0004769134521484375,
'time_algorithm_update': 0.0018893003463745118,
'loss': 2.1957614421844482,
'td_loss': 0.0014366957996389828,
'conservative_loss': 1.0971623718738557,
'time_step': 0.0024144744873046877}),
(66,
{'time_sample_batch': 0.0004748296737670898,
'time_algorithm_update': 0.001923069953918457,
'loss': 2.1966270327568056,
'td_loss': 0.0014970313938101755,
'conservative_loss': 1.0975650000572204,
'time_step': 0.0024482226371765137}),
(67,
{'time_sample_batch': 0.0006723475456237793,
'time_algorithm_update': 0.0023181843757629395,
'loss': 2.1983171677589417,
'td_loss': 0.0017722686848719604,
'conservative_loss': 1.0982724475860595,
'time_step': 0.003056778907775879}),
(68,
{'time_sample_batch': 0.00045365333557128906,
'time_algorithm_update': 0.0018648409843444824,
'loss': 2.1956448483467104,
'td_loss': 0.0012274235920631327,
'conservative_loss': 1.0972087144851685,
'time_step': 0.002373466491699219}),
(69,
{'time_sample_batch': 0.00047734737396240237,
'time_algorithm_update': 0.0018783402442932129,
'loss': 2.1954086685180663,
'td_loss': 0.0010653546734829434,
'conservative_loss': 1.0971716606616975,
'time_step': 0.0024070024490356447}),
(70,
{'time_sample_batch': 0.0004603123664855957,
'time_algorithm_update': 0.0018489694595336915,
'loss': 2.1972744131088255,
'td_loss': 0.0013059828532277606,
'conservative_loss': 1.0979842114448548,
'time_step': 0.002361128330230713}),
(71,
{'time_sample_batch': 0.0005326128005981445,
'time_algorithm_update': 0.0019896578788757323,
'loss': 2.2326128482818604,
'td_loss': 0.03883167172083631,
'conservative_loss': 1.0968905925750732,
'time_step': 0.0025800132751464845}),
(72,
{'time_sample_batch': 0.000496530532836914,
'time_algorithm_update': 0.0019109487533569337,
'loss': 2.1950860810279846,
'td_loss': 0.001654102717875503,
'conservative_loss': 1.0967159831523896,
'time_step': 0.0024579763412475586}),
(73,
{'time_sample_batch': 0.0005002140998840332,
'time_algorithm_update': 0.0019114255905151368,
'loss': 2.1955630135536195,
'td_loss': 0.0014652931969612838,
'conservative_loss': 1.0970488595962524,
'time_step': 0.0024619674682617187}),
(74,
{'time_sample_batch': 0.0005357074737548828,
'time_algorithm_update': 0.0019951415061950683,
'loss': 2.1952951455116274,
'td_loss': 0.0011524688403005711,
'conservative_loss': 1.0970713353157044,
'time_step': 0.0025900816917419434}),
(75,
{'time_sample_batch': 0.000604555606842041,
'time_algorithm_update': 0.002111508846282959,
'loss': 2.196189987659454,
'td_loss': 0.0011116434112773278,
'conservative_loss': 1.0975391697883605,
'time_step': 0.002771108150482178}),
(76,
{'time_sample_batch': 0.0004975366592407227,
'time_algorithm_update': 0.0018982982635498047,
'loss': 2.1946453285217284,
'td_loss': 0.0015036492938816082,
'conservative_loss': 1.0965708434581756,
'time_step': 0.002446579933166504}),
(77,
{'time_sample_batch': 0.00047012090682983397,
'time_algorithm_update': 0.0018267321586608886,
'loss': 2.196987895965576,
'td_loss': 0.0012641044548945502,
'conservative_loss': 1.0978618907928466,
'time_step': 0.002348353862762451}),
(78,
{'time_sample_batch': 0.0005690073966979981,
'time_algorithm_update': 0.002055826187133789,
'loss': 2.195310065746307,
'td_loss': 0.0011137244015117175,
'conservative_loss': 1.0970981705188751,
'time_step': 0.002682504653930664}),
(79,
{'time_sample_batch': 0.0005019116401672363,
'time_algorithm_update': 0.0019088006019592286,
'loss': 2.195496542453766,
'td_loss': 0.001138850417046342,
'conservative_loss': 1.0971788382530212,
'time_step': 0.0024642467498779295}),
(80,
{'time_sample_batch': 0.0005827975273132324,
'time_algorithm_update': 0.0020485162734985352,
'loss': 2.1947550106048586,
'td_loss': 0.001167198322364129,
'conservative_loss': 1.0967939066886903,
'time_step': 0.002692546844482422}),
(81,
{'time_sample_batch': 0.00047295331954956056,
'time_algorithm_update': 0.0018500924110412599,
'loss': 2.227931935787201,
'td_loss': 0.033952856429386884,
'conservative_loss': 1.0969895398616791,
'time_step': 0.0023734259605407717}),
(82,
{'time_sample_batch': 0.0004636073112487793,
'time_algorithm_update': 0.0018477320671081543,
'loss': 2.19747983455658,
'td_loss': 0.0023927097214618697,
'conservative_loss': 1.0975435674190521,
'time_step': 0.002362477779388428}),
(83,
{'time_sample_batch': 0.000538039207458496,
'time_algorithm_update': 0.001985733509063721,
'loss': 2.2000172615051268,
'td_loss': 0.0034504746797028927,
'conservative_loss': 1.098283394575119,
'time_step': 0.0025764894485473633}),
(84,
{'time_sample_batch': 0.0004709506034851074,
'time_algorithm_update': 0.0018519878387451172,
'loss': 2.1969930720329285,
'td_loss': 0.0028046430594986303,
'conservative_loss': 1.097094213962555,
'time_step': 0.002372426986694336}),
(85,
{'time_sample_batch': 0.0004938220977783203,
'time_algorithm_update': 0.0018848967552185058,
'loss': 2.197575035095215,
'td_loss': 0.0028086735983379185,
'conservative_loss': 1.0973831844329833,
'time_step': 0.0024326634407043457}),
(86,
{'time_sample_batch': 0.0004716086387634277,
'time_algorithm_update': 0.0018826532363891602,
'loss': 2.197786946296692,
'td_loss': 0.0026098875643219797,
'conservative_loss': 1.0975885307788849,
'time_step': 0.002405276298522949}),
(87,
{'time_sample_batch': 0.0005232834815979004,
'time_algorithm_update': 0.002008681297302246,
'loss': 2.197643308639526,
'td_loss': 0.002755700311390683,
'conservative_loss': 1.0974437963962556,
'time_step': 0.002584958076477051}),
(88,
{'time_sample_batch': 0.0004944634437561035,
'time_algorithm_update': 0.001909637451171875,
'loss': 2.1980531549453737,
'td_loss': 0.0023269034427357838,
'conservative_loss': 1.0978631222248076,
'time_step': 0.002456667423248291}),
(89,
{'time_sample_batch': 0.0005027151107788086,
'time_algorithm_update': 0.0019388103485107422,
'loss': 2.198425705432892,
'td_loss': 0.0036549880728125573,
'conservative_loss': 1.0973853635787965,
'time_step': 0.002494988441467285}),
(90,
{'time_sample_batch': 0.0004624009132385254,
'time_algorithm_update': 0.001880943775177002,
'loss': 2.1986197805404664,
'td_loss': 0.0026663483370793985,
'conservative_loss': 1.0979767179489135,
'time_step': 0.002394850254058838}),
(91,
{'time_sample_batch': 0.0005315661430358887,
'time_algorithm_update': 0.001959683895111084,
'loss': 2.228097147941589,
'td_loss': 0.033212613491923546,
'conservative_loss': 1.0974422657489777,
'time_step': 0.002542593479156494}),
(92,
{'time_sample_batch': 0.00045597076416015625,
'time_algorithm_update': 0.0018234753608703613,
'loss': 2.1968222665786743,
'td_loss': 0.0023351298249326647,
'conservative_loss': 1.0972435677051544,
'time_step': 0.0023288178443908692}),
(93,
{'time_sample_batch': 0.00046092748641967773,
'time_algorithm_update': 0.0018229055404663085,
'loss': 2.1981692504882813,
'td_loss': 0.002199953671079129,
'conservative_loss': 1.0979846489429474,
'time_step': 0.0023334813117980957}),
(94,
{'time_sample_batch': 0.0005285000801086426,
'time_algorithm_update': 0.002005882263183594,
'loss': 2.197166223526001,
'td_loss': 0.0024109763628803195,
'conservative_loss': 1.097377623319626,
'time_step': 0.0025871992111206055}),
(95,
{'time_sample_batch': 0.0005094480514526368,
'time_algorithm_update': 0.0019360923767089845,
'loss': 2.1988195276260374,
'td_loss': 0.0029319437482627107,
'conservative_loss': 1.0979437935352325,
'time_step': 0.0025005745887756348}),
(96,
{'time_sample_batch': 0.0005018281936645508,
'time_algorithm_update': 0.0019066476821899415,
'loss': 2.1952530360221862,
'td_loss': 0.0014993083794252015,
'conservative_loss': 1.0968768572807313,
'time_step': 0.002458498477935791}),
(97,
{'time_sample_batch': 0.00048806190490722655,
'time_algorithm_update': 0.0018617892265319824,
'loss': 2.1928140926361084,
'td_loss': 0.002345390365226194,
'conservative_loss': 1.095234352350235,
'time_step': 0.0023996424674987793}),
(98,
{'time_sample_batch': 0.00047081470489501953,
'time_algorithm_update': 0.0018648099899291992,
'loss': 2.19599750995636,
'td_loss': 0.001943185601849109,
'conservative_loss': 1.0970271587371827,
'time_step': 0.00238771915435791}),
(99,
{'time_sample_batch': 0.00047873735427856446,
'time_algorithm_update': 0.001869077682495117,
'loss': 2.195319347381592,
'td_loss': 0.0021575475920690224,
'conservative_loss': 1.0965808951854705,
'time_step': 0.002396836280822754}),
(100,
{'time_sample_batch': 0.00057830810546875,
'time_algorithm_update': 0.002337493896484375,
'loss': 2.1974499917030332,
'td_loss': 0.002233420611009933,
'conservative_loss': 1.0976082861423493,
'time_step': 0.002981810569763184})]
Storing and Retrieving Policy Parameters as Artifacts
Now let’s store the trained policy parameters as an artifact associated with our benchmark. This demonstrates how to link artifacts to specific benchmarks for better organization and retrieval.
[16]:
# Save the model to a temporary file
with tempfile.NamedTemporaryFile(suffix='.pt', delete=False) as temp_file:
temp_path = temp_file.name
# Save the model using d3rlpy's save_model method
algo.save_model(temp_path)
# Read the file content as bytes
with open(temp_path, 'rb') as f:
policy_artifact = f.read()
# Clean up the temporary file
os.unlink(temp_path)
# Create metadata for the artifact, linking it to our benchmark
policy_metadata = ArtifactMetadata(
name='trained_cql_policy',
description='Trained CQL policy parameters for MountainCar environment',
benchmark_id=loaded_tupli_env.id # Link artifact to the benchmark
)
# Store the artifact
stored_policy = storage.store_artifact(artifact=policy_artifact, metadata=policy_metadata)
print(f"Stored policy artifact with ID: {stored_policy.id}")
print(f"Artifact linked to benchmark: {stored_policy.benchmark_id}")
Stored policy artifact with ID: 48ecc6b86e9a4cdeafd000e293511a6b
Artifact linked to benchmark: 7b334ed6b7724bec8872418ad32667a4
Another collaborator could now retrieve this artifact by listing all artifacts associated to our benchmark and downloading it.
[17]:
# Create a filter to find artifacts associated with our benchmark
benchmark_filter = FilterEQ(key='benchmark_id', value=loaded_tupli_env.id)
# List all artifacts associated with this benchmark
benchmark_artifacts = storage.list_artifacts(filter=benchmark_filter)
print(f"Found {len(benchmark_artifacts)} artifacts associated with benchmark {loaded_tupli_env.id}:")
for artifact in benchmark_artifacts:
print(f" - ID: {artifact.id}")
print(f" Name: {artifact.name}")
print(f" Description: {artifact.description}")
print(f" Benchmark ID: {artifact.benchmark_id}")
print(f" Created: {artifact.created_at}")
print()
Found 0 artifacts associated with benchmark 4b076cdedc4d433ea4da6d2801c05ca0:
Finally, we demonstrate deserialization of the stored policy.
[18]:
# Load the policy artifact
loaded_policy_artifact = storage.load_artifact(stored_policy.id)
# Write the bytes to a temporary file
with tempfile.NamedTemporaryFile(suffix='.pt', delete=False) as temp_file:
temp_path = temp_file.name
temp_file.write(loaded_policy_artifact)
# Create a new algorithm instance and load the model
loaded_algo = DiscreteCQLConfig().create(device='cpu')
loaded_algo.build_with_env(loaded_tupli_env)
loaded_algo.load_model(temp_path)
print("Successfully loaded trained CQL policy!")
Successfully loaded trained CQL policy!
Testing the Trained Policy
Now, let us test the trained (and loaded) policy:
[19]:
# activate rendering
setattr(loaded_tupli_env.unwrapped, 'render_mode', 'human')
# deactivate recording of episodes
loaded_tupli_env.deactivate_recording()
# run the environment
np.random.seed(seed=42)
obs, info = loaded_tupli_env.reset(seed=42)
for step in range(800):
action = np.int64(loaded_algo.predict(np.expand_dims(obs, axis=0))[0])
obs, reward, done, truncated, info = loaded_tupli_env.step(action)
if done or truncated:
print(f'Episode finished after {step + 1} timesteps')
obs, info = loaded_tupli_env.reset()
# deactivate rendering
loaded_tupli_env.close()
Episode finished after 375 timesteps
Episode finished after 694 timesteps
The trained policy manages to reach the flag even though it has only learned from random actions!
Deleting Benchmarks
To clean up our storage, we now delete the benchmark and all related artifacts. Episodes will automatically be deleted, too.
[20]:
# Clean up: First delete the policy artifact we created
print(f"Deleting policy artifact: {stored_policy.id}")
storage.delete_artifact(stored_policy.id)
# Then delete the benchmark and all remaining related artifacts
# Episodes will automatically be deleted too
print(f"Deleting benchmark: {loaded_tupli_env.id}")
loaded_tupli_env.delete(delete_artifacts=True)
print("Cleanup completed!")
Deleting policy artifact: 48ecc6b86e9a4cdeafd000e293511a6b
Deleting benchmark: 4b076cdedc4d433ea4da6d2801c05ca0
Cleanup completed!
[ ]: